Efficient federated segmentation learning method and device based on clustering in Internet of Vehicles environment

Abstract

The present application relates to the field of federated learning technology, and to an efficient federated segmentation learning method and device based on clustering in a vehicle networking environment. The car side collects local data, and the local data is a labeled road information picture. The car side organizes the label information of the collected local data into a category vector and sends it to the cloud server; the cloud server uses a clustering algorithm to cluster and group the local data of all car sides, and calculates the cosine similarity matrix between each group; the similarity between each group is judged, and if the similarity between each group is less than a specified threshold, personalized federated learning is performed, otherwise non-personalized federated learning is performed. The present invention solves the problem of decreased convergence speed and convergence accuracy due to non-independent and identically distributed data in federated segmentation learning in the vehicle networking scenario, as well as the problem of insufficient server resources due to the excessive number of car sides.

Description

Technical Field

[0001] The present application relates to the field of federated learning technology, and more particularly to an efficient clustering-based federated segmentation learning method and device in a connected vehicle environment. Background Art

[0002] At present, with the great success and progress of big models in recent years, people's daily lives have changed greatly. Big models can achieve intelligent decision-making in scenarios such as text generation, machine translation, disease diagnosis and intelligent transportation. Data is the basis and driving force of model training. The power of big models comes from the mining of hidden patterns and laws in massive data. Where does massive data come from? With the rapid popularization of Internet of Things technology, a large amount of data is stored in the sensors on cars. How to use these naturally distributed data? One way is to send them to centralized nodes for training. However, this method has the risk of privacy leakage and high communication cost.

[0003] Federated Learning (FL), as a distributed machine learning method, provides a new idea for solving the above problems. Without exchanging data samples, the training algorithm spans multiple distributed cars or servers, and a global model is trained based on distributed data. Unlike traditional centralized machine learning technology, there is no need to upload the local data of distributed cars to a centralized node, which can effectively protect user privacy. The typical framework of federated learning in the Internet of Vehicles scenario consists of a server and multiple car terminals. In each round of training, the server sends the current global model to each car terminal, and the car terminal uses its own data to train the model locally, and then only uploads the updated part of the model to the server information exchange area, and the server aggregates the models of all car terminals to form a new global model. This process iterates multiple rounds until the model reaches the preset accuracy or number of training rounds. The aggregation method usually adopts FedAvg (Federated Averaging), which defines the weight as the proportion of data volume, that is, the larger the weight of the car terminal with a larger data volume, the higher the contribution to the global model. However, the main disadvantage of FL is that each car needs to run a complete machine learning model, which is usually difficult to run on a resource-constrained car. In particular, when a deep machine learning model is used, the car cannot run it.

[0004] Split Learning (SL) provides an effective solution to the above problems. In split learning in the IoV scenario, the complete machine learning model is divided into multiple smaller network parts, and only a part of the smaller network is trained on the resource-constrained vehicle side, which greatly reduces the resource consumption on the vehicle side. In a split learning between two parties (the vehicle side and the server), the complete machine learning model is Divided into two parts and , and complete the initialization on the car side and the server respectively, where It is the car-side model, including the input layer To the segmented layer The continuous network layers between them; It is a server-side model, including the network layer after the segmentation layer To the output layer The car side uses local data In the local model Perform forward propagation on the intermediate result , and the intermediate results With data labels The server uses the intermediate results in the local model Perform forward propagation on the output of the entire model The server outputs and data labels Calculating Losses ,exist Back propagation calculation right Gradient and For intermediate results Gradient The server will For intermediate results Gradient Sent to the car side, the car side Back propagation calculation right Gradient In the above scenario, The output result is the same as The output result is the same, so the loss are the same, according to the chain rule for differentiation, and The same is true for the training process, so this training process will get the same results as the usual training method. However, due to the relay-based training across multiple cars, the training efficiency of split learning is much lower than that of federated learning.

[0005] Federated segmented learning combines federated learning and segmented learning, alleviating their inherent shortcomings. In federated segmented learning in the IoV scenario, the complete machine learning model is divided into two parts: the vehicle-side model and the server-side model. In the neural network, the vehicle-side model is a model composed of several consecutive layers from the input layer to the segmentation layer. The server-side model is a model consisting of several consecutive layers from the segmentation layer to the output layer. . Before starting training, multiple distributed car terminals hold local data and car terminal side models, and the server holds a server-side model for each car terminal. After starting training, all car terminals perform forward propagation in parallel on their car terminal side models and pass the intermediate results to the server. Then, the server processes the forward propagation and back propagation of each car terminal in parallel on its server-side model and updates the server-side model. The server then transmits the gradient of the loss function to the intermediate result back to the respective car terminals and performs back propagation on the car terminal side model and updates the model. After a certain period of training, the car terminal sends the local car terminal side model to the server, and the server helps to complete the aggregation of the car terminal side model. Since the server-side model is concentrated on the server, the server directly completes the aggregation. This training and aggregation process is iterated many times until the model training reaches the preset accuracy or the preset training time.

[0006] In the IoV object recognition, the nodes involved in federated segmentation learning are multiple IoV car terminals and cloud servers, all of which have computing power and communication capabilities. Among them, the cloud server node has high computing power and high communication capabilities, so the cloud server serves as the server for federated segmentation learning. Multiple IoV car terminals are multiple distributed nodes. The car terminal communicates with the cloud server through 2G / 3G / 4G / 5G, NB-IoT, and Wifi to complete the sending and receiving of information. In such a scenario, an important problem is that the local data of different car terminals are often non-independent and identically distributed (non-IID, non-Independent and Identically Distributed), (for example, some vehicles are engaged in long-distance transportation, so the data collected by these vehicles may be mainly mountains, trees, and some vehicles often travel in downtown areas, so the data collected by these vehicles may be mainly pedestrians, buildings, etc.) This will reduce the convergence speed of traditional federated segmentation learning using the FedAvg aggregation model. In addition, a server-side model is maintained for each car terminal on the server side, which is also a huge challenge for the server's computing power and storage resources. Therefore, how to solve the problems of decreased convergence speed and accuracy caused by non-IID data and insufficient server resources caused by an excessive number of vehicles in the federated segmentation learning scenario of vehicle network object recognition, and further improve user experience, has become an urgent problem to be solved.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0008] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0009] The disclosed embodiments provide an efficient clustering-based federated segmentation learning method and device in a vehicle networking environment, which solves the problems of decreased convergence speed and convergence accuracy in federated segmentation learning in a vehicle networking scenario due to non-independent and identically distributed data, and insufficient server resources due to an excessive number of vehicle terminals.

[0010] In some embodiments, the method comprises:

[0011] The car end collects local data, which is a marked road information picture. The car end organizes the label information of the collected local data into a category vector and sends it to the cloud server;

[0012] The cloud server uses a clustering algorithm to cluster all the local data on the car side and calculate the cosine similarity matrix between each group;

[0013] Determine the similarity between groups. If the similarity between groups is less than the specified threshold, perform personalized federated learning. Otherwise, perform non-personalized federated learning.

[0014] The non-personalized federated learning trains a global model according to the grouping results, and the global model includes a global model on the vehicle side and a global model on the server side; each group aggregates the trained vehicle side models within the group, and the aggregated models within the group are further aggregated between groups into a global model on the vehicle side, and the server aggregates the trained server side models into a global model on the server side; the personalized federated learning trains a group model for each group according to the grouping results, and each group model includes a vehicle side group model and a server side group model trained with the group data, and the objective function of the group model training adds a second normal form of the difference between the parameters to be trained and the parameters of the benchmark model;

[0015] The images to be predicted are collected and clustered, and the road conditions are predicted using a global model or a group model based on the similarities of each group.

[0016] Preferably, the clustering algorithm is Kmeans clustering, using cosine similarity as a similarity metric.

[0017] Preferably, the non-personalized federated learning process is as follows:

[0018] S3011: The server initializes the server-side group model and sends the global model on the vehicle side to all vehicle-side nodes. The vehicle-side nodes receive the model and initialize the local vehicle-side model.

[0019] S3012: The vehicle-end node samples some local data and performs forward propagation in the local model to calculate the intermediate result, where the intermediate result is the output of the local model, and sends the intermediate result and the data label to the server;

[0020] S3013: The server receives the intermediate result from the vehicle-side node, performs forward propagation on the server-side model of the corresponding group of the vehicle-side node to calculate the data label predicted by the model, calculates all losses, and performs back propagation to update the parameters of the server-side model; the server calculates the gradient of the intermediate result and sends it back to the vehicle-side node;

[0021] S3014: After receiving the intermediate result gradient, the vehicle-side node performs back propagation to update the parameters of the vehicle-side model; if the vehicle-side node still has untrained data, it continues to sample some data and performs forward propagation on the updated vehicle-side model, and sends the intermediate result to the server, iterating S3013-S3014 until there is no remaining untrained data on the vehicle-side node;

[0022] S3015: The vehicle-side node sends the updated vehicle-side model to the server. The server aggregates the vehicle-side model within the group to form an intra-group model, and aggregates the intra-group model across groups into a vehicle-side global model. The server aggregates the classifier layer and other layers of the server-side model respectively to generate a server-side global model.

[0023] Preferably, the server aggregates the model on the vehicle side into an intra-group model using FedAvg within the group, and aggregates the intra-group model into an automotive side global model using FedAvg across groups;

[0024] The server-side model uses energy ratio-based aggregation for the classifier layer of the model and FedAvg aggregation for the other layers of the model.

[0025] Preferably, after the server-side model aggregates the classifier layer of the model using an energy ratio-based aggregation method, the classifier parameter matrix No. Column parameters The calculation formula is as follows:

[0026] ,

[0027] ,

[0028] in, represents the number of groups for local data clustering, Indicates The first parameter matrix of the model classifier layer of the group The energy ratio of the columns, Indicates The first parameter matrix of the model classifier layer of the group The energy ratio of the columns, Indicates The first parameter matrix of the model classifier layer of the group List, Indicates The first parameter matrix of the model classifier layer of the group Line Column parameters, Indicates The first parameter matrix of the model classifier layer of the group Line Column parameters, and represents the dimension of the model parameter matrix of the classifier, is the absolute value function;

[0029] Aggregated classifier parameter vector No. Dimensional parameters The calculation formula is as follows:

[0030] ,

[0031] ,

[0032] in, Indicates The first parameter vector of the model classifier layer of the group The energy ratio of the dimension, Indicates The first parameter vector of the model classifier layer of the group The energy ratio of the dimension, Indicates The first parameter vector of the model classifier layer of the group dimension, Indicates The first parameter vector of the model classifier layer of the group dimension, and represents the dimension of the model parameter matrix of the classifier, is the absolute value function.

[0033] Preferably, the personalized federated learning process is as follows:

[0034] S3021: The server generates a reference model for each group, and sends the vehicle-side model and the corresponding vehicle-side reference model to all vehicle-side nodes belonging to the group;

[0035] S3022: The vehicle-side node initializes the local vehicle-side model, samples some data from the local data for forward propagation, records the intermediate results, and sends the intermediate results and data labels to the server;

[0036] S3023: The server receives the intermediate result from the vehicle-side node, performs forward propagation on the server-side model of the group corresponding to the vehicle-side node to calculate the data label predicted by the model, calculates all losses, and performs back propagation to update the parameters of the server-side model; the server calculates the gradient of the intermediate result and sends it back to the vehicle-side node;

[0037] S3024: After receiving the intermediate result gradient, the vehicle-side node performs back propagation to update the parameters of the vehicle-side model; if the vehicle-side node still has untrained data, it continues to sample some data and performs forward propagation on the updated vehicle-side model, and sends the intermediate result to the server, iterating S3023-S3024 until there is no remaining untrained data on the vehicle-side node;

[0038] S3025: The vehicle-end node sends the vehicle-end model to the server. The server aggregates the vehicle-end model within the group to obtain a vehicle-end group model, and assigns the aggregation result to each vehicle node in the group. The server-side group model is obtained from the training in S3023.

[0039] Preferably, the benchmark model generation formula is as follows:

[0040] ,

[0041] ,

[0042] in, Representation Group The vehicle side group benchmark model, Representation Group The server-side group baseline model, represents the number of groups for local data clustering, Representation Group and Group Cosine similarity between category vectors, Representation Group The car side group model, Representation Group The server-side group model.

[0043] Preferably, the objective function of each group is as follows:

[0044] ,

[0045] in, Representation Group The group model, Indicates The objective function of the group, Representation Group The baseline model, represents the prediction error, Represents the prediction result of the model, represents the true label, Representation Group data, It is a hyperparameter, which indicates the proportion of regularization loss to the overall loss.

[0046] Preferably, the intermediate results in the personalized federated learning include the output of the local vehicle-side model, the label of the local data, and the second normal form of the difference between the vehicle-side model parameters to be trained and the vehicle-side model parameters of the benchmark model;

[0047] The second normal form of the difference between the parameters of the vehicle-side model to be trained and the parameters of the vehicle-side model of the baseline model is calculated as follows:

[0048] ,

[0049] in, represents the local car end-side model, represents the vehicle end-side benchmark model;

[0050] Personalized Federated Learning Total Loss The formula is as follows:

[0051] ,

[0052] in, represents the data label predicted by the model, represents the real data label, The second normal form of the difference between the trained server-side model parameters and the baseline model's server-side model parameters is calculated as follows:

[0053] ,

[0054] in, It is a super parameter.

[0055] In some embodiments, the device includes: a processor and a memory storing program instructions, and the processor is configured to execute the efficient clustering-based federated segmentation learning method in the Internet of Vehicles environment when running the program instructions.

[0056] The efficient federated segmentation learning method and device based on clustering in the Internet of Vehicles environment provided by the embodiments of the present disclosure can achieve the following technical effects:

[0057] In the vehicle network object recognition scenario, each car end encrypts the category information of local data and sends it to the server before training. The server uses a clustering algorithm to cluster all car ends into several classes based on these category information, that is, they are divided into several groups. The server maintains a server-side model for each group. In this way, maintaining a server-side model for each group can effectively deal with the problems caused by non-independent and identically distributed data while reducing the computing power and storage pressure of the server. Since the car ends in the same group share a server-side model, their use of the server-side model is mutually exclusive, that is, only one car end in the same group can use the server-side model of the group at the same time. Since the car ends between different groups do not share the server-side model, they can use the server-side model of their respective groups at the same time. Although this method of serial within the group and parallel between groups reduces the degree of parallelism to a certain extent, it is beneficial to improve the convergence speed of model training because the data distribution within the group is close.

[0058] On this basis, the degree of data heterogeneity of the entire system is estimated through the clustering results, and the system adaptively selects non-personalized clustering federated segmentation learning or personalized clustering federated segmentation learning.

[0059] Non-personalized clustering federated segmentation learning runs when the data heterogeneity of the entire system is low, and it generates a global model shared by all cars; personalized clustering federated segmentation learning runs when the data heterogeneity of the entire system is high. When the clustering results show that the similarity between groups is low, it is difficult for a global model to efficiently process such heterogeneous data, so it is a better choice to generate a model suitable for each group.

[0060] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0062] Figure 1 It is a schematic diagram of the process of the present invention;

[0063] Figure 2 The present invention performs Kmeans clustering process based on LabelVector;

[0064] Figure 3 The present invention performs a training process according to the group relationship;

[0065] Figure 4The present invention performs a model aggregation process according to group relationships;

[0066] Figure 5 This is a flow chart of the server side of the present invention;

[0067] Figure 6 This is a flow chart of the automobile end of the present invention;

[0068] Figure 7 A schematic diagram showing the comparison between the non-personalized federated segmentation learning and the federated averaging algorithm of the present invention;

[0069] Figure 8 A schematic diagram showing the comparison between the personalized federated segmentation learning and the federated averaging algorithm of the present invention;

[0070] Fig. 9 It is a schematic diagram of a device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0071] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0072] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0073] Combination Figure 1 , an efficient clustering-based federated segmentation learning method in the Internet of Vehicles environment, including:

[0074] S1: The car collects local data, which is a marked road information picture. The car organizes the label information of the collected local data into a category vector and sends it to the cloud server.

[0075] S2: The cloud server uses a clustering algorithm to cluster all the local data of the car and calculate the cosine similarity matrix between each group.

[0076] S3: Determine the similarity between groups. If the similarity between groups is less than the specified threshold, personalized federated learning is performed; otherwise, non-personalized federated learning is performed.

[0077] The non-personalized federated learning trains a global model according to the grouping results, and the global model includes a global model on the vehicle side and a global model on the server side; each group aggregates the trained vehicle side models within the group, and the aggregated models within the group are further aggregated between groups into a global model on the vehicle side, and the server aggregates the trained server side models into a global model on the server side; the personalized federated learning trains a group model for each group according to the grouping results, and each group model includes a vehicle side group model and a server side group model trained with the group data; the second normal form of the difference between the parameters to be trained and the parameters of the baseline model is added to the objective function of the group model training.

[0078] S4: Collect the images to be predicted and cluster them into groups, and use the global model or group model to predict the road conditions according to the similarity of each group.

[0079] As a refinement of the above embodiment, in step S1 Each car end encrypts the local data category related information and sends it to the cloud server. Specifically, assuming that the entire system coexists If there are multiple categories of data, each car terminal will convert the local data category into one It is represented in the form of a dimensional category vector, where the value of each dimension represents the number of samples of that category locally.

[0080] In this embodiment, the data set adopts the traffic sign data set CCTSDB2021 (https: / / mp.weixin.qq.com / s / Vu-ghupx0Gr3nhRvf6a7ew).

[0081] As a refinement of the above embodiment, Figure 2 As shown in the figure, in step S2, after receiving the category vector of each car, the server uses the Kmeans clustering method to cluster the indivual dimensional vector clustering The vector similarity in Kmeans is measured by cosine similarity. and The cosine similarity is calculated as follows:

[0082] ,

[0083] in, and They are vectors and The second paradigm, is a vector and The specific number of groups It is a hyperparameter. The more groups there are, the greater the consumption of server computing power and storage resources. However, the distribution of car-side data within the group is more similar, and the effect of model training will be better.

[0084] According to the clustering results, calculate Cosine similarity matrix between groups ,in Yes Group and Group The cosine similarity between the category vectors. The category vector of a group is the sum of the category vectors of all the cars belonging to the group. If The similarity between groups is less than a threshold , the system performs personalized federated learning, otherwise it enters non-personalized federated learning. The choice of threshold depends on the needs. The threshold is a hyperparameter. The larger the threshold, the lower the tolerance for data heterogeneity between car ends, and the more inclined to perform personalized federated learning. Taking a simple 10-class problem as an example, a threshold of 0.75 can well distinguish two different data heterogeneity scenarios.

[0085] As a refinement of the above embodiment, the non-personalized federated learning goal in step S3 is to train a global model, which consists of two parts: the global model on the vehicle side and the server-side global model The process is as follows:

[0086] S3011: The server converts the global model of the vehicle end into Send to all cars and Group Server Side Group Model Server-side group model initialization.

[0087] S3012: Automotive End Node After receiving the vehicle end-side model, the local vehicle end-side model by Initialize and sample some local data in the car side model Perform forward propagation on and Sent to the server, where is the output of the local car-side model, is the true label of the local data.

[0088] S3013: The server receives the vehicle end node Intermediate results sent and ,Will In the corresponding group of the car end node Server-side model The data labels predicted by the forward propagation calculation model are , calculate the loss Back-propagation updates the server-side model The server will Gradients of intermediate results Send back to the car end node. If the intermediate results of multiple car end nodes in a group arrive at the same time, this step is performed in the order of arrival. That is, the training of the server-side model in the same group is serial, while the training between groups is parallel.

[0089] S3014: Automotive End Node After receiving the gradient of the intermediate result sent by the server, back propagate on the vehicle-side model and update the vehicle-side model If the car end node If there is still data that has not been trained, then another part of the data will be sampled and used in the car-side model. Perform forward propagation on and The data is sent to the server and iterates S3013-S3014 until there is no remaining data that has not been trained at the vehicle end node.

[0090] S3015: The vehicle-side node sends the updated vehicle-side model to the server. The server aggregates the vehicle-side model within the group to form an intra-group model, and aggregates the intra-group model across groups into a vehicle-side global model. The server aggregates the classifier layer and other layers of the server-side model respectively to generate a server-side global model.

[0091] Automotive End Node The car side model Sent to the server, the server uses the car-side model within the group Aggregate and assign the result to ,Right now

[0092] ,

[0093] in ,

[0094] On this basis, use Will The vehicle side group model Aggregate into a global model on the vehicle side ,Right now:

[0095] ,

[0096] in .

[0097] For the server-side model, the classifier layer of the model is aggregated using the energy ratio-based aggregation method, and the other layers of the model are aggregated using Aggregation. The classifier widely used in neural networks today is a fully connected layer. The input of this fully connected layer is the feature vector extracted by the neural network layer before the classifier for the classified entity. Let its dimension be , the output is a vector whose dimension is the same as the number of categories in the whole system Similarly, the value of each dimension corresponds to the probability that the classified entity belongs to the category corresponding to that dimension. In this case, the model parameters of the classifier are a matrix and a vector , each column parameter of the matrix and each dimension parameter of the vector are highly correlated with the category. Therefore, the parameters of the classifier are aggregated between groups based on the energy ratio.

[0098] Specifically, for the Classifier parameters for groups and , respectively maintain a dimensional energy ratio vector and , where the vector The values ​​of the dimensions are the parameter matrices No. Column pair for the entire parameter matrix The energy ratio and the parameter vector dimension for the entire parameter vector The energy ratio of them is given by the following formulas: and For the matrix Dimensions of:

[0099] ,

[0100] ,

[0101] In fact, the energy ratio here can be considered as the result of the one-paradigm normalization of the parameters. In order to effectively realize the one-paradigm representation of the energy ratio of the parameters, the parameters of the classifier are normalized during training. and Apply regularization terms to shrink the low-energy parameters as much as possible .

[0102] In the energy ratio-based aggregation method, the classifier parameter matrix after aggregation is No. The parameters of the columns are given by the following formula:

[0103] ,

[0104] in, represents the number of groups for local data clustering, Indicates The first parameter matrix of the model classifier layer of the group The energy ratio of the column. The energy ratio is a measure of the parameter energy. The larger its value, the greater the impact of the parameter on the model output. Indicates The first parameter matrix of the model classifier layer of the group The energy ratio of the column. Indicates The first parameter matrix of the model classifier layer of the group List. Indicates The first parameter matrix of the model classifier layer of the group Line Column parameters, Indicates The first parameter matrix of the model classifier layer of the group Line Column parameters, and represents the dimension of the model parameter matrix of the classifier, is the absolute value function.

[0105] Aggregated classifier parameter vector No. The parameter of dimension is given by the following formula:

[0106] ,

[0107] in, Indicates The first parameter vector of the model classifier layer of the group The energy ratio of the dimension is a measure of the parameter energy. The larger its value is, the greater the influence of the parameter on the model output is. Indicates The first parameter vector of the model classifier layer of the group The energy ratio of the dimension, Indicates The first parameter vector of the model classifier layer of the group dimension, Indicates The first parameter vector of the model classifier layer of the group dimension, and represents the dimension of the model parameter matrix of the classifier, is the absolute value function.

[0108] Aggregation produces a server-side global model In this way, one round of training is completed. If the training reaches the preset number of rounds or the model reaches the preset accuracy, the training is stopped. Otherwise, continue to execute S3011.

[0109] As a refinement of the above embodiment, the goal of the personalized clustering federated segmentation learning in step S3 is to train a group model for each group, and the model of group k It consists of two parts: the vehicle side group model and server-side group model In addition, by Generate a baseline model enhancement group The generalization ability of the model.

[0110] The personalized federated learning process is as follows:

[0111] S3021: The server generates a baseline model for each group. The server sends the vehicle-side model and the automotive end-side benchmark model Sent to all cars belonging to group k together.

[0112] S3022: Automotive End Node After receiving the vehicle-side model and the vehicle-side benchmark model, the local vehicle-side model by Initialize and sample some local data in the car side model Perform forward propagation on , and Sent to the server, where is the output of the local car-side model, is the true label of the local data, The second normal form representing the difference between the trained car-side model parameters and the baseline model's car-side model parameters.

[0113] S3023: The server receives the vehicle end node Intermediate results sent , and ,Will The server-side model corresponding to group k at the car end node The data labels predicted by the forward propagation calculation model are , the second normal form of calculating the difference between the trained server-side model parameters and the baseline model's server-side model parameters , the specific formula is as follows:

[0114] ,

[0115] And calculate the total loss:

[0116] ,

[0117] Then backpropagate to update the server-side model ; The server calculates the gradient of the intermediate result and sends it back to the car end node. If the intermediate results of multiple car end nodes in a group arrive at the same time, this step is performed in the order of arrival. That is, the training of the server-side model in the same group is serial, while the training between groups is parallel.

[0118] S3024: Automotive End Node After receiving the gradient of the intermediate result sent by the server, back propagate on the vehicle-side model and update the vehicle-side model Parameters; if the car end node If there is still data that has not been trained, then another part of the data will be sampled and used in the car-side model. Perform forward propagation on , and Send to the server, iterate S3023-S3025 until there is no remaining data that has not been trained on the vehicle end node;

[0119] S3026: Automotive End Node The car side model Sent to the server, the server uses the car-side model within the group Aggregate and assign the result to ,Right now

[0120] ,

[0121] in .

[0122] In this way, one round of training is completed. If the training reaches the preset number of rounds or the model reaches the preset accuracy, the training is stopped. Otherwise, continue to execute S3021.

[0123] Optionally, the benchmark model generation formula is as follows:

[0124] ,

[0125] ,

[0126] in, represents the vehicle side group benchmark model, represents the server-side group baseline model, represents the number of groups for local data clustering, Representation Group and Group Cosine similarity between category vectors, Representation Group The car side group model, Representation Group The server-side group model.

[0127] Each group model is trained only based on the data of that group, without using the data and model information of other groups. This will result in each group model performing well on the data within the group, but its model generalization ability is poor. In order to make each group model more generalizable, some knowledge of other group models is obtained through the benchmark model.

[0128] Optionally, the objective function for each group is as follows:

[0129] ,

[0130] in, Representation Group The group model, Indicates The objective function of the group, represents the baseline model for group k, represents the prediction error (e.g., usually the cross entropy for classification problems), Represents the prediction result of the model, represents the true label, Representation Group data, It is a hyperparameter, which indicates the proportion of regularization loss to the overall loss.

[0131] By adding the second normal form of the difference between the parameters to be trained and the parameters of the baseline model to the objective function, the group model can be trained to minimize the prediction error while taking into account the difference with the baseline model to a certain extent. This modification enables the group model to obtain global knowledge, thereby enhancing the generalization ability of each group model.

[0132] As a refinement of the above embodiment, the specific use process of the model in step S4 is as follows:

[0133] If the similarity between the groups is greater than or equal to the specified threshold, non-personalized federated learning is performed. For non-personalized federated segmentation learning, the server sends the server-side global model to each car terminal, so that each car terminal has not only the car-side global model, but also the server-side global model. The car-side global model and the server-side global model together form a complete global model. This complete global model can then be used on the car side: the car side first forward-propagates the road image collected in real time on the car-side global model, and after obtaining the intermediate result, it continues to forward-propagate on the server-side global model to output the prediction result. This prediction result can evaluate the state of the road to meet user needs.

[0134] If the similarity between the groups is less than the specified threshold, personalized federated learning is performed, otherwise non-personalized federated learning is performed. For personalized federated segmentation learning, the server sends the server-side group model to all car terminals belonging to the group, so that each car terminal has a corresponding server-side group model in addition to the car-side group model. The car-side group model and the server-side group model together form a complete group model. This complete group model can then be used on the car side: the car side will first forward propagate the road image collected in real time on the car-side group model, and after obtaining the intermediate result, continue to forward propagate on the server-side group model to output the prediction result. This prediction result can evaluate the state of the road to meet user needs.

[0135] In order to verify the effectiveness of the design of the present invention, experiments were carried out on a very broad class of image recognition problems.

[0136] There are two non-independent and identically distributed data settings in the experiment, which correspond to the non-personalized federated segmentation learning and personalized federated segmentation learning in the design of the present invention respectively.

[0137] In the experiment of verifying the non-personalized federated segmentation learning, the non-personalized federated segmentation learning of the present invention is compared with the federated average algorithm. The results are as follows: Figure 7 The experimental results show that the non-personalized federated segmentation learning of the present invention improves the model convergence speed and the accuracy of model convergence compared with the federated average algorithm.

[0138] In the experiment of verifying personalized federated segmentation learning, the personalized federated segmentation learning of the present invention is compared with the federated average algorithm. The results are as follows: Figure 8 The experimental results show that compared with the federated average algorithm, the personalized federated segmentation learning of the present invention improves the model convergence speed and also significantly improves the model convergence accuracy.

[0139] Combination Fig. 9As shown, the embodiment of the present disclosure provides an efficient federated segmentation learning device 300 based on clustering in a vehicle networking environment, including a processor (processor) 304 and a memory (memory) 301. Optionally, the device may also include a communication interface (CommunicationInterface) 302 and a bus 303. Among them, the processor 304, the communication interface 302, and the memory 301 can communicate with each other through the bus 303. The communication interface 302 can be used for information transmission. The processor 304 can call the logical instructions in the memory 301 to execute the efficient federated segmentation learning method based on clustering in the vehicle networking environment of the above embodiment.

[0140] In addition, the logic instructions in the memory 301 described above can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0141] The memory 301 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 304 executes the function application and data processing by running the program instructions / modules stored in the memory 301, that is, to implement the efficient federated segmentation learning method based on clustering in the vehicle networking environment in the above embodiment.

[0142] The memory 301 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 301 may include a high-speed random access memory and may also include a non-volatile memory.

[0143] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0144] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method or device including the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0145] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. The technicians may clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

Claims

1. An efficient clustering-based federated segmentation learning method in a connected vehicle environment, characterized in that: include: The car end collects local data, which is a marked road information picture. The car end organizes the label information of the collected local data into a category vector and sends it to the cloud server; The cloud server uses a clustering algorithm to cluster all the local data on the car side and calculate the cosine similarity matrix between each group; Determine the similarity between groups. If the similarity between groups is less than the specified threshold, perform personalized federated learning. Otherwise, perform non-personalized federated learning. The non-personalized federated learning trains a global model according to the grouping results, and the global model includes a global model on the vehicle side and a global model on the server side. The vehicle-side node sends the updated vehicle-side model to the server, and the server aggregates the vehicle-side model within the group to form an intra-group model, and aggregates the intra-group model across groups into a global model on the vehicle side. The server aggregates the classifier layer and other layers of the server-side model respectively to generate a server-side global model, and all vehicle terminals share the server-side global model. The training of the server-side model within the same group is serial, while the training between groups is parallel. The personalized federated learning trains a group model for each group according to the grouping results. Each group model includes a car-side group model and a server-side group model trained with the group data. The server generates a benchmark model for each group. The server sends the car-side model and the corresponding car-side benchmark model to all car-side nodes belonging to the group. The car-side node sends the trained and updated car-side model to the server. The server aggregates the car-side model within the group to obtain the car-side group model, and assigns the aggregation result to each car node in the group. The objective function of the group model training adds the second normal form of the difference between the parameters to be trained and the parameters of the benchmark model. The server maintains a server-side group model for each group, and the cars in the same group share a server-side group model. The training of the server-side model in the same group is serial, while the training between groups is parallel. The images to be predicted are collected and clustered, and the road conditions are predicted using a global model or a group model based on the similarities of each group.

2. The efficient clustering-based federated segmentation learning method in the Internet of Vehicles environment according to claim 1 is characterized in that: The clustering algorithm is Kmeans clustering, and cosine similarity is used as the similarity metric.

3. The efficient clustering-based federated segmentation learning method in the Internet of Vehicles environment according to claim 1 is characterized in that: The non-personalized federated learning process is as follows: S3011: The server initializes the server-side group model and sends the global model on the vehicle side to all vehicle-side nodes. The vehicle-side nodes receive the model and initialize the local vehicle-side model. S3012: The vehicle-end node samples some local data and performs forward propagation in the local model to calculate the intermediate result, where the intermediate result is the output of the local model, and sends the intermediate result and the data label to the server; S3013: The server receives the intermediate result from the vehicle-side node, performs forward propagation on the server-side model of the corresponding group of the vehicle-side node to calculate the data label predicted by the model, calculates all losses, and performs back propagation to update the parameters of the server-side model; the server calculates the gradient of the intermediate result and sends it back to the vehicle-side node; S3014: After receiving the intermediate result gradient, the vehicle-side node performs back propagation to update the parameters of the vehicle-side model; if the vehicle-side node still has untrained data, it continues to sample some data and performs forward propagation on the updated vehicle-side model, and sends the intermediate result to the server, iterating S3013-S3014 until there is no remaining untrained data on the vehicle-side node; S3015: The vehicle-side node sends the trained and updated vehicle-side model to the server. The server aggregates the vehicle-side model within the group to form an intra-group model, and aggregates the intra-group model across groups into a vehicle-side global model. The server aggregates the classifier layer and other layers of the server-side model respectively to generate a server-side global model.

4. The efficient clustering-based federated segmentation learning method in the Internet of Vehicles environment according to claim 3 is characterized in that: The server aggregates the car-side model using FedAvg within the group into an intra-group model, and aggregates the intra-group model across groups using FedAvg into a car-side global model. The server-side model uses energy ratio-based aggregation for the classifier layer of the model and FedAvg aggregation for the other layers of the model.

5. The efficient clustering-based federated segmentation learning method in the Internet of Vehicles environment according to claim 4 is characterized in that: After the server-side model aggregates the classifier layer of the model using the energy ratio-based aggregation method, the classifier parameter matrix No. Column parameters The calculation formula is as follows: ， ， in, represents the number of groups for local data clustering, Indicates The first parameter matrix of the model classifier layer of the group The energy ratio of the columns, Indicates The first parameter matrix of the model classifier layer of the group The energy ratio of the columns, Indicates The first parameter matrix of the model classifier layer of the group List, Indicates The first parameter matrix of the model classifier layer of the group Line Column parameters, Indicates The first parameter matrix of the model classifier layer of the group Line Column parameters, and represents the dimension of the model parameter matrix of the classifier, is the absolute value function; Aggregated classifier parameter vector No. Dimensional parameters The calculation formula is as follows: ， ， in, Indicates The first parameter vector of the model classifier layer of the group The energy ratio of the dimension, Indicates The first parameter vector of the model classifier layer of the group The energy ratio of the dimension, Indicates The first parameter vector of the model classifier layer of the group dimension, Indicates The first parameter vector of the model classifier layer of the group dimension, and represents the dimension of the model parameter matrix of the classifier, is the absolute value function.

6. The efficient clustering-based federated segmentation learning method in the Internet of Vehicles environment according to claim 2 is characterized in that: The personalized federated learning process is as follows: S3021: The server generates a reference model for each group, and sends the vehicle-side model and the corresponding vehicle-side reference model to all vehicle-side nodes belonging to the group; S3022: The vehicle-side node initializes the local vehicle-side model, samples some data from the local data for forward propagation, records the intermediate results, and sends the intermediate results and data labels to the server; S3023: The server receives the intermediate result from the vehicle-side node, performs forward propagation on the server-side model of the group corresponding to the vehicle-side node to calculate the data label predicted by the model, calculates all losses, and performs back propagation to update the parameters of the server-side model; the server calculates the gradient of the intermediate result and sends it back to the vehicle-side node; S3024: After receiving the intermediate result gradient, the vehicle-side node performs back propagation to update the parameters of the vehicle-side model; if the vehicle-side node still has untrained data, it continues to sample some data and performs forward propagation on the updated vehicle-side model, and sends the intermediate result to the server, iterating S3023-S3024 until there is no remaining untrained data on the vehicle-side node; S3025: The vehicle-side node sends the trained and updated vehicle-side model to the server. The server aggregates the vehicle-side model within the group to obtain a vehicle-side group model, and assigns the aggregation result to each vehicle node in the group. The server-side group model is obtained from the training in S3023.

7. The efficient clustering-based federated segmentation learning method in the Internet of Vehicles environment according to claim 6 is characterized in that: The benchmark model generation formula is as follows: ， ， in, Representation Group The vehicle side group benchmark model, Representation Group The server-side group baseline model, represents the number of groups for local data clustering, Representation Group and Group Cosine similarity between category vectors, Representation Group The car side group model, Representation Group The server-side group model.

8. The efficient clustering-based federated segmentation learning method in the Internet of Vehicles environment according to claim 7 is characterized in that: The objective function for each group is as follows: ， in, Representation Group The group model, Indicates The objective function of the group, Representation Group The baseline model, represents the prediction error, Represents the prediction result of the model, represents the true label, Representation Group data, It is a hyperparameter, which indicates the proportion of regularization loss to the overall loss.

9. The efficient clustering-based federated segmentation learning method in the Internet of Vehicles environment according to claim 8 is characterized in that: The intermediate results in the personalized federated learning include the output of the local vehicle-side model, the label of the local data, and the second normal form of the difference between the vehicle-side model parameters to be trained and the vehicle-side model parameters of the benchmark model; The second normal form of the difference between the parameters of the vehicle-side model to be trained and the parameters of the vehicle-side model of the baseline model is calculated as follows: ， in, represents the local car end-side model, represents the vehicle end-side benchmark model; Personalized Federated Learning Total Loss The formula is as follows: ， in, represents the data label predicted by the model, represents the real data label, The second normal form of the difference between the trained server-side model parameters and the baseline model's server-side model parameters is calculated as follows: ， in, It is a super parameter.

10. An efficient federated segmentation learning device based on clustering in a vehicle networking environment, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the efficient federated segmentation learning method based on clustering in a vehicle network environment as described in any one of claims 1 to 9 when running the program instructions.

Images