Federated learning methods, equipment, media and products based on dual clustering

Abstract

The present application discloses a federated learning method, electronic device, and storage medium based on a dual clustering approach. The method comprises: obtaining a connection weight parameter vector capable of characterizing a road condition image training set in a smart driving client, and performing dual clustering based on the parameter vector to obtain multiple smart driving client sets with similar road condition image training sets; then, performing federated learning based on the smart driving client sets to obtain shared model parameters and local model parameters applicable to the smart driving client sets until the training is completed. Implementing this method can accurately and effectively perform intra-class model clustering training based on the sample-based dual clustering method of the federated learning method, thereby improving the overall performance of the clustering model.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent driving image recognition, and specifically to a federated learning method, electronic device, readable storage medium and computer program product based on a dual clustering approach. Background Art

[0002] Currently, training road image recognition models often requires a large amount of training data. However, most smart driving companies only possess localized road image data. This data inevitably contains differences in road image features, such as road signs, text types, road environments, and architectural styles. If each smart driving company uses only localized road image data for road image recognition model training, the resulting local model will be applicable to a narrow geographical range, hindering the reuse and promotion of road image recognition systems. If all smart driving companies' road image data is aggregated for model training, the resulting unified model will have the best average fit for road images across all regions, but may not necessarily be the best fit for road images in each specific region, reducing the actual recognition performance of the road image recognition system in that region. Furthermore, road image data from different countries or regions may involve national or regional security concerns. Relevant policies and regulations restrict the cross-border or cross-regional sharing of road image data. Simply aggregating all road image data to train a unified model would pose legal risks. In fact, some smart driving companies may have similar road image data. These smart driving companies jointly train a model to achieve a balance between the contradiction between applicable geographical scope and adaptability to specific regions. Summary of the Invention

[0003] In view of the above problems, the present application provides a federated learning method, electronic device, readable storage medium and computer program product based on a dual clustering method, which can solve the problems of the narrow geographical scope of applicability of local models, the low adaptability of unified models to specific regions, the difficulty in evaluating the correlation between different local domain road condition image data sets under data privacy protection conditions, how to use the correlation of road condition image data of certain smart driving companies to jointly train models, and the excessive adaptation of models obtained based on traditional personalized federated learning training to specific regions.

[0004] In a first aspect, the present application provides a federated learning method based on a dual clustering approach, the method being applied to a road condition image federated learning center, the method comprising:

[0005] Receive a connection weight parameter vector set uploaded by multiple intelligent driving clients; the connection weight parameter vector set includes multiple connection weight parameter vectors, the connection weight parameter vector set is extracted from a pre-trained model obtained by the intelligent driving client after completing pre-training, and the extraction location is the convolution layer closest to the input layer; wherein each connection weight parameter vector corresponds to a convolution kernel in the convolution layer;

[0006] Calculating a distance metric value between every two connection weight parameter vectors in all connection weight parameter vector sets, and clustering all connection weight parameter vectors based on multiple distance metric values ​​to obtain multiple parameter vector cluster sets;

[0007] Clustering all intelligent driving clients based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets to obtain multiple client cluster sets;

[0008] Send the initialization model parameters of the road condition image recognition model to all intelligent driving clients, so that all intelligent driving clients can upload the local model parameters of the local model after completing local training to obtain the local model;

[0009] Calculate the common model parameters of each client cluster set based on multiple local model parameters and the number of images in multiple local traffic image training sets;

[0010] The corresponding shared model parameters are issued to all intelligent driving clients, so that after completing local training to obtain a local model, all intelligent driving clients upload the local model update parameters of the local model, and trigger the execution of the step of calculating the shared model parameters of each client cluster set based on multiple local model parameters and the number of images in the multiple local road condition image training sets;

[0011] Among them, when the preset training termination condition is reached, the intelligent driving client stops uploading the local model update parameters and determines the local model update parameters as the final parameters of the local model.

[0012] In the above technical solution, the method can construct a highly matched client clustering set based on a dual clustering strategy, thereby using a large number of similar samples to train a better clustering model, thereby improving the clustering model's recognition effect on road condition images.

[0013] In some embodiments, the method further comprises:

[0014] Build a road condition image recognition model based on CNN architecture;

[0015] Initializing parameters of the road condition image recognition model to obtain initialized model parameters;

[0016] The model architecture and the initialization parameters of the road condition image recognition model are sent to multiple intelligent driving clients, so that the multiple intelligent driving clients feedback a connection weight parameter vector set.

[0017] In the above technical solution, this method can first obtain the set of connection weight parameter vectors in the intelligent driving client through the pre-training process, thereby providing a data basis for the subsequent calculation of the correlation between samples in each intelligent driving client.

[0018] In some embodiments, calculating the distance metric between every two connection weight parameter vectors in the set of all connection weight parameter vectors includes:

[0019] Determine two connection weight parameter vectors in the set of all connection weight parameter vectors as a first connection weight parameter vector and a second connection weight parameter vector;

[0020] extracting a first connection weight parameter set from the first connection weight parameter vector, and extracting a second connection weight parameter set from the second connection weight parameter vector;

[0021] Calculating a first mean of the first connection weight parameter set, a first standard deviation of the first connection weight parameter set, a second mean of the second connection weight parameter set, and a second standard deviation of the second connection weight parameter set, and calculating a covariance between the first connection weight parameter set and the second connection weight parameter set;

[0022] Calculating based on a similarity index calculation formula, the first mean, the first standard deviation, the second mean, the second standard deviation, and the covariance to obtain a similarity index between the first connection weight parameter vector and the second connection weight parameter vector;

[0023] determining the complement of the similarity index as a distance metric between the first connection weight parameter vector and the second connection weight parameter vector;

[0024] Repeat the above steps until the distance metric between every two connection weight parameter vectors in all connection weight parameter vector sets is calculated.

[0025] In the above technical solution, this method can use the complement of the similarity index as a distance metric, thereby more accurately quantifying the differences between connection weight parameter vectors, thereby providing data guarantee for subsequent clustering operations.

[0026] In some embodiments, clustering all intelligent driving clients based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets to obtain multiple client cluster sets includes:

[0027] Based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets, calculate the feature vector of the local road condition image training set in each intelligent driving client;

[0028] Based on multiple feature vectors, the correlation distance between every two smart driving clients is determined, and all smart driving clients are clustered based on the multiple correlation distances to obtain multiple client cluster sets.

[0029] In the above technical solution, this method can accurately classify intelligent driving clients with similar road condition image features, thereby providing effective support for customized model optimization.

[0030] In some embodiments, a correlation distance between every two intelligent driving clients is determined based on multiple feature vectors, and all intelligent driving clients are clustered based on the multiple correlation distances to obtain multiple client cluster sets, including:

[0031] The Euclidean distance between the feature vectors of each two intelligent driving clients is determined as the corresponding correlation distance;

[0032] Based on multiple correlation distances, K-means algorithm and silhouette coefficient method, all intelligent driving clients are clustered to obtain multiple client cluster sets.

[0033] In the above technical solution, this method can accurately characterize the correlation differences between smart driving clients, and thereby improve the rationality and effectiveness of smart driving client clustering.

[0034] In some embodiments, the step of calculating the shared model parameters of each client cluster set based on the multiple local model parameters and the number of images in the multiple local traffic image training sets includes:

[0035] Selecting a target client cluster set from a plurality of intelligent driving clients; the target client cluster set includes the target client and other clients;

[0036] Calculating the total number of images in all local traffic image training sets in the target client cluster set;

[0037] Determine the ratio of the number of images in the local traffic image training set in the target client to the total number of images as the weight coefficient of the target client;

[0038] Calculating target weighted model parameters based on the weight coefficient of the target client and the corresponding local model parameters;

[0039] Calculating the sum of all target weighted model parameters in the target client cluster set to obtain the common model parameters of the target client cluster set;

[0040] Repeat the above steps until the common model parameters of all target client cluster sets are calculated.

[0041] In the above technical solution, this method can more reasonably fuse model parameters based on the contribution of data of different scales, thereby improving the representativeness and generalization ability of shared model parameters.

[0042] In a second aspect, the present application provides a road condition image recognition method, the method comprising:

[0043] Obtaining a road condition image to be identified;

[0044] The road condition image to be identified is input into the local model in the intelligent driving client, so that the local model outputs the road condition information corresponding to the road condition image to be identified; wherein, the model parameters of the local model are determined according to the federated learning method based on the dual clustering method described in any one of the first aspects.

[0045] In the above technical solution, the method can utilize the local model optimized by federated learning to accurately and efficiently identify the road condition information in the road condition image to be identified, thereby improving the accuracy and real-time performance of road condition identification.

[0046] In a third aspect, the present application provides a federated learning device based on a dual clustering method, which is applied to a road condition image federated learning center, and includes:

[0047] A receiving unit, configured to receive a set of connection weight parameter vectors uploaded by multiple intelligent driving clients; the set of connection weight parameter vectors includes multiple connection weight parameter vectors, the set of connection weight parameter vectors is extracted from a pre-trained model obtained by the intelligent driving client after completing pre-training, and the extraction location is the convolution layer closest to the input layer; wherein each connection weight parameter vector corresponds to a convolution kernel in the convolution layer;

[0048] A distance measurement unit, used to calculate the distance measurement value between every two connection weight parameter vectors in the set of all connection weight parameter vectors;

[0049] A parameter clustering unit, configured to cluster all connection weight parameter vectors based on multiple distance metrics to obtain multiple parameter vector cluster sets;

[0050] A client clustering unit, configured to cluster all intelligent driving clients based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets to obtain multiple client cluster sets;

[0051] A sending unit is used to send the initialization model parameters of the road condition image recognition model to all intelligent driving clients, so that all intelligent driving clients upload the local model parameters of the local model after completing local training to obtain the local model;

[0052] A parameter calculation unit, configured to calculate the common model parameters of each client cluster set based on multiple local model parameters and the number of images in multiple local traffic image training sets;

[0053] The issuing unit is further configured to issue corresponding shared model parameters to all intelligent driving clients, so that after completing local training to obtain a local model, all intelligent driving clients upload the local model update parameters of the local model, and trigger the parameter calculation unit to perform the operation of calculating the shared model parameters of each client cluster set based on the multiple local model parameters and the number of images in the multiple local road condition image training sets;

[0054] Among them, when the preset training termination condition is reached, the intelligent driving client stops uploading the local model update parameters and determines the local model update parameters as the final parameters of the local model.

[0055] In a fourth aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the federated learning method based on dual clustering described in any one of the first aspects.

[0056] In a fifth aspect, the present application provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, it executes the federated learning method based on dual clustering described in any one of the first aspects.

[0057] In a sixth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it executes the federated learning method based on dual clustering described in any one of the first aspects.

[0058] The beneficial effect of the present application is that the method can quantitatively evaluate the correlation between various intelligent driving clients based on the correlation of training data, thereby achieving a more accurate and reasonable client clustering effect.

[0059] This method can also perform intra-class federated learning based on the correlation between intelligent driving clients, so that each intelligent driving client in the class can be assigned a more reasonable aggregation weight, thereby optimizing the performance of the intelligent driving client clustering model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] To more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly describes the drawings required for use in the embodiments of the present application. The following drawings illustrate only certain embodiments of the present invention and should not be construed as limiting the scope of the present invention. The same reference numerals are used throughout the drawings to represent the same content.

[0061] Figure 1 This is a flowchart of a federated learning method based on dual clustering in some embodiments of the present application;

[0062] Figure 2 This is a flowchart of a federated learning method based on dual clustering in some embodiments of the present application;

[0063] Figure 3 A flowchart of a federated learning method based on dual clustering as an example for this application;

[0064] Figure 4 Schematic diagram of the flow of a road condition image recognition method in some embodiments of the present application;

[0065] Figure 5 Schematic diagram of the structure of a federated learning device based on a dual clustering approach in some embodiments of the present application;

[0066] Figure 6 This is a schematic diagram of the structure of an electronic device in some embodiments of the present application. DETAILED DESCRIPTION

[0067] The following embodiments of the technical solution of the present application will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application and are therefore only examples and are not intended to limit the scope of protection of the present application.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0069] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood as indicating or implying relative importance or implicitly indicating the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is two or more (including two).

[0070] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0071] At present, the local models trained by local training methods are applicable to a narrow geographical range, and the unified models trained by federated learning methods are often not suitable for smart driving companies in different regions. The models trained based on traditional personalized federated learning have the problem of over-adaptation to specific regions.

[0072] In response to the above technical problems, an embodiment of the present application provides a federated learning method based on a dual clustering approach. This method performs dual clustering processing based on the road condition image training set in each smart driving client to obtain clustering results of multiple smart driving clients, and then trains the corresponding road condition image recognition model in the clustering results, so that the road condition image recognition model can achieve excellent performance effects in each smart driving client in the clustering results.

[0073] In the above technical solution, the method can construct a highly matched client cluster set based on a dual clustering strategy, thereby using a large number of similar samples to train a better local model, thereby improving the recognition effect of the local model on road condition images.

[0074] Example 1

[0075] like Figure 1 As shown, some embodiments of the present application provide a dual clustering-based federated learning method, which is applied to a road condition image federated learning center, including:

[0076] S101. Receive a set of connection weight parameter vectors uploaded by multiple intelligent driving clients; the set of connection weight parameter vectors includes multiple connection weight parameter vectors, and the set of connection weight parameter vectors is extracted from the pre-training model obtained by the intelligent driving client after completing the pre-training, and the extraction position is the convolution layer closest to the input layer; wherein each connection weight parameter vector corresponds to a convolution kernel in the convolution layer.

[0077] S102 , calculating a distance metric value between every two connection weight parameter vectors in all connection weight parameter vector sets, and clustering all connection weight parameter vectors based on multiple distance metric values ​​to obtain multiple parameter vector cluster sets.

[0078] S103. Cluster all intelligent driving clients based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets to obtain multiple client cluster sets.

[0079] S104. Send the initialization model parameters of the road condition image recognition model to all intelligent driving clients, so that all intelligent driving clients upload the local model parameters of the local model after completing local training to obtain the local model.

[0080] S105 : Calculate the common model parameters of each client cluster set based on the multiple local model parameters and the number of images in the multiple local road condition image training sets.

[0081] S106. Send the corresponding shared model parameters to all smart driving clients, so that all smart driving clients upload the local model update parameters of the local model after completing local training to obtain the local model, and trigger the execution of step S105.

[0082] S107. When the preset training termination condition is reached, the intelligent driving client stops uploading the local model update parameters and determines the local model update parameters as the final parameters of the local model.

[0083] In some embodiments, the intelligent driving client refers to a client node in federated learning.

[0084] In some embodiments, the convolutional layer closest to the input layer is closest to the local traffic image training set, so the connection weight parameter vector in its convolution kernel can be considered to have a strong correlation with the local traffic image training set. A convolutional layer has K convolution kernels, and each convolution kernel has one connection weight parameter vector.

[0085] In some embodiments, the set of connection weight parameter vectors is i is the ID of the Zhijia client;

[0086] The connection weight parameter vector is That is, any one of the connection weight parameter vector set; wherein the connection weight parameter vector generally includes multiple connection weight parameters, and the multiple connection weight parameters can constitute a connection weight parameter set;

[0087] The distance metric is That is, the distance metric between the kth vector of Zhijia client i and the lth vector of Zhijia client j;

[0088] The parameter vector clustering set is Multiple parameter vector clustering sets correspond to C1 is a positive integer;

[0089] The eigenvector is f i, which is the feature vector of intelligent driving client i;

[0090] The client cluster set is C2 is a positive integer;

[0091] Initialize the model parameters as That is, the parameters determined when the traffic image federated learning center initializes the aggregation model;

[0092] The local model parameters are That is, the local model parameters obtained by the intelligent driving client i after the rth round of training;

[0093] The shared model parameters are That is, the shared parameters of the client cluster set obtained by the traffic image federated learning center after federated learning based on the local model parameters.

[0094] For example, the method first obtains Then calculate At this time, clustering is performed to obtain A1, A2, ..., A C Then, the method is based on and The relationship between them is used to determine the eigenvector corresponding to each smart driving client, and the Euclidean distance S(i,j) between the two smart driving clients is calculated based on the eigenvector, and the smart driving clients are further clustered based on S(i,j).

[0095] After that, this method allows the intelligent driving client to Training Then let the traffic image federated learning center be based on Calculate And re-issue it, and repeat the cycle until the training is completed.

[0096] For example, in the local model parameters Convergence or number of training rounds r ≥ R end , then terminate the client clustering set E m During the training process of the Zhongzhijia client, It is the client cluster set E m The model parameters of the final clustering model. Not converged and the number of training rounds r is less than R end When the traffic image is uploaded to the federal learning center S

[0097] Among them, R end is the preset training cycle threshold constant, R end is a positive integer greater than 1.

[0098] In these embodiments, the method can generate a clustered road condition image recognition model adapted to each smart driving client based on the correlation of local road condition image training sets of different smart driving clients using a federated learning method, thereby solving problems such as the narrow geographical scope of applicability of local models, the low adaptability of unified models to specific regions, the difficulty in evaluating the correlation between different local domain road condition image data sets under data privacy protection conditions, how to use the correlation of road condition image data of certain smart driving companies to jointly train models, and the excessive adaptation of models obtained based on traditional personalized federated learning training to specific regions.

[0099] In order to provide initial data for federated learning, in some embodiments, the method further includes:

[0100] Build a road condition image recognition model based on CNN architecture;

[0101] Initializing the parameters of the road condition image recognition model to obtain the initialized model parameters;

[0102] The model architecture and initialization parameters of the road condition image recognition model are sent to multiple intelligent driving clients so that the multiple intelligent driving clients can feedback the connection weight parameter vector set.

[0103] For example, when the traffic image federated learning center initializes the parameters of the traffic image recognition model based on the CNN architecture, the initialization parameters are Among them, the initialization is the road image federated learning center to N c The parameters to be sent to each client are the same for each client.

[0104] For example, after the initialization, the traffic image federated learning center S sets the parameter value It is sent to all intelligent driving clients, and the value of r is initialized to 1.

[0105] That is, at this time r = 1, the intelligent driving client initializes the parameters of the local model to

[0106] For example, the number of road condition image federated learning centers is S, and the total number of intelligent driving clients (federated learning client nodes) is N. c .

[0107] The image federated learning center S initializes the parameters of the road condition image recognition model using the CNN architecture, and sends the architecture and initialization parameters of the CNN road condition image recognition model to all intelligent driving clients.

[0108] For example, each smart driving client trains the parameters of the CNN road condition image recognition model received from S based on the local image sample training set. After the training is completed, each smart driving client sets the connection weight parameter vector set of the first convolution layer closest to the input layer in the local model. Upload to S.

[0109] Among them, i is the serial number of the intelligent driving client, K is the number of convolution kernels in the first convolution layer of the model, Represents the vector of connection weight parameters of the kth convolution kernel of the first convolution layer of the i-th intelligent driving client. 1≤i≤N c , 1≤k≤K, i, k, N c , K is a positive integer.

[0110] In these embodiments, the method can first obtain the set of connection weight parameter vectors in the intelligent driving client through a pre-training process, thereby providing a data basis for the subsequent calculation of the correlation between samples in each intelligent driving client.

[0111] In order to achieve high-precision clustering model training effects, in some embodiments, the pre-training and local training of the intelligent driving client are both based on the model architecture of the local road condition image training set, the gradient descent algorithm, and the road condition image recognition model issued in the road condition image federated learning.

[0112] For example, in the rth round of training, the i-th intelligent driving client receives the aggregated model parameters sent from S m is the client cluster set number to which the intelligent driving client i belongs.

[0113] Among them, the intelligent driving client uses the gradient descent algorithm to train the local model based on the local road image training set, and to and the original local model parameters Generate new local model parameters

[0114] In these embodiments, the method can utilize a unified model architecture, combine a local road image training set with a gradient descent algorithm, and achieve efficient clustering training of the intelligent driving client model while maintaining data privacy and model consistency under the federated learning framework.

[0115] In order to more accurately determine the distance between the traffic data training sets, in some embodiments, calculating the distance metric between every two connection weight parameter vectors in all connection weight parameter vector sets includes:

[0116] Determine two connection weight parameter vectors in the set of all connection weight parameter vectors as a first connection weight parameter vector and a second connection weight parameter vector;

[0117] Extracting a first connection weight parameter set from the first connection weight parameter vector, and extracting a second connection weight parameter set from the second connection weight parameter vector;

[0118] Calculating a first mean of the first connection weight parameter set, a first standard deviation of the first connection weight parameter set, a second mean of the second connection weight parameter set, a second standard deviation of the second connection weight parameter set, and calculating a covariance between the first connection weight parameter set and the second connection weight parameter set;

[0119] Calculating based on a similarity index calculation formula, a first mean, a first standard deviation, a second mean, a second standard deviation, and a covariance to obtain a similarity index between the first connection weight parameter vector and the second connection weight parameter vector;

[0120] determining the complement of the similarity index as a distance measure between the first connection weight parameter vector and the second connection weight parameter vector;

[0121] Repeat the above steps until the distance metric between every two connection weight parameter vectors in all connection weight parameter vector sets is calculated.

[0122] For example, the traffic image federated learning center S receives N c N uploaded by intelligent driving clients c ×K connection weight parameter vectors, and calculate the distance metric between the above connection weight parameter vectors Where 1≤j≤N c , 1≤l≤K, j, l are positive integers. If i=j, then k≠l.

[0123] The similarity index's complement calculation formula is as follows:

[0124]

[0125] in, Take [0, 2], the smaller the value, the more similar it is.

[0126] In the above formula, Considering the brightness of the image perception scene (given by μ i 、μ j ), contrast (denoted by σ i , σ j denoted by ), structure (denoted by σ ij Therefore, compared with the traditional cosine distance and Euclidean distance, this method has better measurement accuracy.

[0127] Among them, c1 and c2 are constants, and the recommended value is: c1 = (0.01 × L) 2, c2=(0.03×L) 2 ;

[0128] L is the difference between the maximum and minimum values ​​of all components of all parameter vectors;

[0129] μ i , σ i is the mean (i.e., the first mean) and standard deviation (i.e., the first standard deviation) of all components of the k-th parameter vector of the i-th client;

[0130] μ j , σ j is the mean (i.e., the second mean) and standard deviation (i.e., the second standard deviation) of all components of the l-th parameter vector of the j-th client;

[0131] σ ij is the covariance of the components of the kth parameter vector of the ith client and the lth parameter vector of the jth client. At this time, all components of a parameter vector can be regarded as different values ​​of a random variable, and the covariance of the two random variables can be calculated.

[0132] In these embodiments, the method can use the complement of the similarity index as a distance metric, thereby more accurately quantifying the differences between connection weight parameter vectors, thereby providing data guarantee for subsequent clustering operations.

[0133] In order to more effectively cluster the connection weight parameter vectors, in some embodiments, all connection weight parameter vectors are clustered based on multiple distance metrics to obtain multiple parameter vector cluster sets, including:

[0134] Based on multiple distance metrics, K-means algorithm and silhouette coefficient method, all connection weight parameter vectors are clustered to obtain multiple parameter vector cluster sets.

[0135] For example, the traffic image federated learning center S uses the K-means clustering algorithm to cluster the N c ×K connection weight parameter vectors are clustered, and the number of cluster centers C1 is determined by the silhouette coefficient method. Finally, the clustering results in a set of C1 parameter vectors. Wherein, C1 is a positive integer.

[0136] In these embodiments, the method can adaptively determine the best clustering set, thereby improving the effect of parameter clustering.

[0137] In order to more effectively determine multiple client cluster sets, in some embodiments, all intelligent driving clients are clustered based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets to obtain multiple client cluster sets, including:

[0138] Based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets, calculate the feature vector of the local road condition image training set in each intelligent driving client;

[0139] Based on multiple feature vectors, the correlation distance between every two smart driving clients is determined, and all smart driving clients are clustered based on the multiple correlation distances to obtain multiple client cluster sets.

[0140] In these embodiments, the method can accurately classify intelligent driving clients with similar road condition image features, thereby providing effective support for customized model optimization.

[0141] In order to more effectively calculate accurate feature vectors, in some embodiments, based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets, the feature vectors of the local road condition image training set in each intelligent driving client are calculated, including:

[0142] Based on multiple parameter vector clustering sets, the connection weight parameter vectors corresponding to the intelligent driving client are classified to obtain the connection weight parameter vector classification results;

[0143] Based on the classification results of the connection weight parameter vectors, the proportion of the connection weight parameter vectors in each category of results is determined, and these proportions are determined as the feature vectors of the local road condition image training set corresponding to the intelligent driving client.

[0144] For example, the connection weight parameter vector classification result of the k connection weight parameter vectors in the i-th intelligent driving client is The proportion of connection weight parameter vectors in one type of result is Indicates that the parameter vector of the i-th client belongs to the set A j The number of i is the number of parameter vectors of the i-th intelligent driving client.

[0145] For example, the feature vector is represented as follows:

[0146]

[0147] For example: when K=5, i=1, If there is Then we can get

[0148] In these embodiments, the method can determine the feature vector of the local road condition image training set in the smart driving client by the attribution of the connection weight parameter vector corresponding to the smart driving client in the cluster set, thereby providing a key basis for model aggregation and parameter adjustment in cluster federated learning.

[0149] In order to achieve clustering of intelligent driving clients, in some embodiments, the correlation distance between every two intelligent driving clients is determined based on multiple feature vectors, and all intelligent driving clients are clustered based on the multiple correlation distances to obtain multiple client cluster sets, including:

[0150] The Euclidean distance of the feature vectors between each two intelligent driving clients is determined as the corresponding correlation distance;

[0151] Based on multiple correlation distances, K-means algorithm and silhouette coefficient method, all intelligent driving clients are clustered to obtain multiple client cluster sets.

[0152] For example, the traffic image federated learning center S is based on the feature vector f of the traffic image training set of two intelligent driving clients. i and f j Calculate the Euclidean distance and obtain the distance S(i,j) between the i-th smart driving client and the j-th smart driving client.

[0153] For example, the method uses the K-means clustering algorithm to cluster all intelligent driving clients. The number of cluster centers C2 is determined by the silhouette coefficient method. Finally, the clustering results in C2 sets of client nodes. C2 is a positive integer.

[0154] In these embodiments, the method can accurately characterize the correlation differences between smart driving clients, and thereby improve the rationality and effectiveness of smart driving client clustering.

[0155] In order to accurately calculate the common model parameters of the client cluster set, in some embodiments, the common model parameters of each client cluster set are calculated based on multiple local model parameters and the number of images in multiple local traffic image training sets, including:

[0156] A target client cluster set is selected from multiple intelligent driving clients; the target client cluster set includes the target client and other clients;

[0157] Calculate the total number of images in the target client cluster set, including the number of images in the local traffic image training set;

[0158] The ratio of the number of images in the local traffic image training set of the target client to the total number of images is determined as the weight coefficient of the target client;

[0159] Calculate target weighted model parameters based on the target client's weight coefficient and the corresponding local model parameters;

[0160] Calculate the sum of all target weighted model parameters in the target client cluster set to obtain the shared model parameters of the target client cluster set;

[0161] Repeat the above steps until the common model parameters of all target client cluster sets are calculated.

[0162] In some embodiments, the method may synchronously perform the above-mentioned calculation process of the common model parameters on all target client cluster sets.

[0163] In some embodiments, after performing the above-mentioned common model parameter calculation process once, the method may repeat the above-mentioned common model parameter calculation process for a new target client cluster set until all target client cluster sets have undergone the above-mentioned calculation process.

[0164] For example, in the rth round of training, the traffic image federated learning center S receives the local model parameters uploaded by each client node According to the result of the second clustering, the local model parameters of the intelligent driving clients belonging to the same client cluster set are weightedly aggregated to generate an aggregation model specific to the client cluster set (the aggregation model includes shared model parameters).

[0165] For example, for any client cluster set E m , E m The elements in are the serial numbers of the client nodes, and the following formula is used to generate the client cluster set E m Dedicated aggregation model parameters The formula is as follows:

[0166]

[0167] Among them, D j is the local road condition image training set of the j-th intelligent driving client;

[0168] |D j | is set D j The cardinality of

[0169] 1≤m≤C2, p, q, m are positive integers.

[0170] In these embodiments, the method can accurately quantify the association strength and data contribution between clients, thereby improving the representativeness and adaptability of shared model parameters, and further optimizing the training effect of the road condition image recognition model under the federated learning framework.

[0171] Example 2

[0172] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be described clearly and completely below. In some embodiments, Figure 2 As shown in FIG, the dual clustering-based federated learning method is applied to the road condition image federated learning center, including:

[0173] S201. Construct a road condition image recognition model based on CNN architecture.

[0174] S202: Initialize the parameters of the road condition image recognition model to obtain initialized model parameters.

[0175] S203: Send the model architecture and initialization parameters of the road condition image recognition model to multiple intelligent driving clients, so that the multiple intelligent driving clients can feedback a set of connection weight parameter vectors.

[0176] S204. Receive a set of connection weight parameter vectors uploaded by multiple intelligent driving clients; the set of connection weight parameter vectors includes multiple connection weight parameter vectors, and the set of connection weight parameter vectors is extracted from the pre-training model obtained by the intelligent driving client after completing the pre-training, and the extraction location is the convolution layer closest to the input layer; wherein, each connection weight parameter vector corresponds to a convolution kernel in the convolution layer.

[0177] S205 : Calculate the similarity index between every two connection weight parameter vectors, and determine the complement of the similarity index as the distance metric value.

[0178] S206 : Clustering all connection weight parameter vectors based on multiple distance metrics to obtain multiple parameter vector cluster sets.

[0179] S207. Based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets, calculate the feature vector of the local road condition image training set in each intelligent driving client.

[0180] S208. Determine the Euclidean distance between the feature vectors of each two intelligent driving clients as the corresponding correlation distance.

[0181] S209. Based on multiple correlation distances, K-means algorithm and silhouette coefficient method, all intelligent driving clients are clustered to obtain multiple client cluster sets.

[0182] S210. Send the initialization model parameters of the road condition image recognition model to all smart driving clients, so that all smart driving clients upload the local model parameters of the local model after completing local training to obtain the local model.

[0183] S211. Select a target client cluster set from multiple intelligent driving clients; the target client cluster set includes the target client and other clients.

[0184] S212: Calculate the total number of images in all local road condition image training sets in the target client cluster set.

[0185] S213: Determine the ratio of the number of images in the local road condition image training set to the total number of images in the target client as the weight coefficient of the target client.

[0186] S214: Calculate target weighted model parameters based on the target client's weight coefficient and the corresponding local model parameters.

[0187] S215: Calculate the sum of all target weighted model parameters in the target client cluster set to obtain the shared model parameters of the target client cluster set.

[0188] S216. Send the corresponding shared model parameters to all smart driving clients, so that all smart driving clients upload the local model update parameters of the local model after completing local training to obtain the local model, and trigger the execution of step S211.

[0189] S217. When the preset training termination condition is reached, the intelligent driving client stops uploading the local model update parameters and determines the local model update parameters as the final parameters of the local model.

[0190] For example, Figure 3 As shown, Figure 3 An example process of this method is shown as follows:

[0191] (1) The third-party international credible organization S (Road Condition Image Federated Learning Center) builds a road condition image recognition model based on the CNN architecture and initializes the model parameters. S sends the structure and initialization parameters of the road condition image recognition model to smart driving companies (smart driving clients) in different countries or regions. Each smart driving company uses the road condition image recognition model received from S to train the parameters of the above model based on the local road condition image sample training set. After training, each smart driving company sets the parameters of the first convolutional layer closest to the input layer in the local model. Upload to S.

[0192] (2) S receives N c The parameter vectors uploaded by the intelligent driving companies are then compared and the distance metric between each pair of these parameter vectors is calculated. Based on these distance metrics, S clusters these parameter vectors using the K-means clustering algorithm. The number of cluster centers, C1, is determined using the silhouette coefficient method, resulting in a cluster of C1 parameter vectors. S calculates the feature vectors of the training data for each intelligent driving company.

[0193] (3) S uses the Euclidean distance of the feature vectors to measure the distance between the training data of two intelligent driving companies. The K-means clustering algorithm is used to cluster the training data of all intelligent driving companies. The number of cluster centers C2 is determined by the silhouette coefficient method. Finally, the clustering results in C2 sets consisting of intelligent driving companies.

[0194] (4) S reinitializes the parameters of the road condition image recognition model. The initialization parameters are S sets the parameter value Issued to all smart driving companies.

[0195] (5) In the rth round of training, the i-th intelligent driving company receives the aggregated model parameter value received from S If r=1, the smart driving company initializes the parameters of the local model to Otherwise, the smart driving company updates the parameters of the local model to

[0196] The smart driving company uses the gradient descent algorithm to train the local model based on the local image sample training set, and updates the local model parameters to Upload to S

[0197] (6) In the rth round of training, the parameter server S receives the local model parameters uploaded by each intelligent driving company. Based on the results of the second clustering, the local model parameters of the smart driving companies belonging to the same smart driving company set are weightedly aggregated to generate an aggregation model specific to the smart driving company set.

[0198] For any set of smart driving companies E m , generate the smart driving company set E m Dedicated aggregation model parameters

[0199] if Convergence or r ≥ R end =100, then the set E is terminated m The model training process in which Chinese intelligent driving companies participate, It is the set E m The final model of the intelligent driving company, otherwise let r = r + 1, S will Send to set E m and repeat step (5).

[0200] In these embodiments, the method can quantitatively evaluate the correlation between various intelligent driving clients based on the correlation of training data, thereby achieving a more accurate and reasonable client clustering effect.

[0201] In these embodiments, the method can also perform intra-class federated learning based on the correlation between intelligent driving clients, so that each intelligent driving client in the class can be assigned a more reasonable aggregation weight, thereby optimizing the performance of the intelligent driving client clustering model.

[0202] Example 3

[0203] like Figure 4 As shown, some embodiments of the present application provide a road condition image recognition method, which includes:

[0204] S301: Obtain a road condition image to be identified.

[0205] S302. Input the road condition image to be identified into the local model in the intelligent driving client, so that the local model outputs road condition information corresponding to the road condition image to be identified; wherein, the model parameters of the local model are determined according to the federated learning method based on the dual clustering method of any one of the first aspects.

[0206] In these embodiments, the method can use the local model optimized by federated learning to accurately and efficiently identify the road condition information in the road condition image to be identified, thereby improving the accuracy and real-time performance of road condition identification.

[0207] Example 4

[0208] Figure 5 The schematic diagram of the structure of a federated learning device based on dual clustering is shown. It should be understood that the device is Figure 1 The method executed in the embodiment corresponds to the embodiment, and the steps involved in the aforementioned method can be executed. The specific functions and effects of the device can be found in the description above. To avoid repetition, detailed description is appropriately omitted here.

[0209] The dual clustering-based federated learning device is applied to the road condition image federated learning center, including:

[0210] Receiving unit 410 is used to receive a connection weight parameter vector set uploaded by multiple intelligent driving clients; the connection weight parameter vector set includes multiple connection weight parameter vectors, and the connection weight parameter vector set is extracted from the pre-trained model obtained by the intelligent driving client after completing pre-training, and the extraction location is the convolution layer closest to the input layer; wherein each connection weight parameter vector corresponds to a convolution kernel in the convolution layer;

[0211] A distance measurement unit 420 is configured to calculate a distance measurement value between every two connection weight parameter vectors in the set of all connection weight parameter vectors;

[0212] A parameter clustering unit 430 is configured to cluster all connection weight parameter vectors based on multiple distance metrics to obtain multiple parameter vector cluster sets;

[0213] The client clustering unit 440 is configured to cluster all intelligent driving clients based on the multiple connection weight parameter vector sets and the multiple parameter vector cluster sets to obtain multiple client cluster sets;

[0214] The sending unit 450 is used to send the initialization model parameters of the road condition image recognition model to all intelligent driving clients, so that all intelligent driving clients upload the local model parameters of the local model after completing local training to obtain the local model;

[0215] A parameter calculation unit 460 is configured to calculate the common model parameters of each client cluster set based on the multiple local model parameters and the number of images in the multiple local traffic image training sets;

[0216] The sending unit 450 is further configured to send the corresponding shared model parameters to all intelligent driving clients, so that after completing local training to obtain a local model, all intelligent driving clients upload the local model update parameters of the local model, and trigger the parameter calculation unit 460 to perform an operation of calculating the shared model parameters of each client cluster set based on multiple local model parameters and the number of images in the multiple local road condition image training sets;

[0217] Among them, when the preset training termination conditions are reached, the intelligent driving client stops uploading the local model update parameters and determines the local model update parameters as the final parameters of the local model.

[0218] In some embodiments, the dual clustering-based federated learning apparatus further includes:

[0219] A construction unit 470 is used to construct a road condition image recognition model based on a CNN architecture;

[0220] Initialization unit 480, used to initialize the parameters of the road condition image recognition model to obtain initialized model parameters;

[0221] The sending unit 450 is used to send the model architecture and initialization parameters of the road condition image recognition model to multiple intelligent driving clients, so that the multiple intelligent driving clients can feedback the connection weight parameter vector set.

[0222] In some embodiments, the distance measurement unit 420 includes:

[0223] A first determining subunit 421 is configured to determine two connection weight parameter vectors in the set of all connection weight parameter vectors as a first connection weight parameter vector and a second connection weight parameter vector;

[0224] An extraction subunit 422 is configured to extract a first connection weight parameter set from the first connection weight parameter vector and a second connection weight parameter set from the second connection weight parameter vector;

[0225] a first calculation subunit 423, configured to calculate a first mean of the first connection weight parameter set, a first standard deviation of the first connection weight parameter set, a second mean of the second connection weight parameter set, a second standard deviation of the second connection weight parameter set, and calculate a covariance between the first connection weight parameter set and the second connection weight parameter set;

[0226] The first calculation subunit 423 is further configured to calculate based on the similarity index calculation formula, the first mean, the first standard deviation, the second mean, the second standard deviation, and the covariance to obtain a similarity index between the first connection weight parameter vector and the second connection weight parameter vector;

[0227] The second determining subunit 424 is further configured to determine the complement of the similarity index as a distance metric between the first connection weight parameter vector and the second connection weight parameter vector;

[0228] The above operation is repeated until the distance metric value between every two connection weight parameter vectors in all connection weight parameter vector sets is calculated.

[0229] In some embodiments, the client clustering unit 440 includes:

[0230] A second calculation subunit 441 is configured to calculate a feature vector of a local road condition image training set in each intelligent driving client based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets;

[0231] The clustering subunit 442 is used to determine the correlation distance between every two smart driving clients based on multiple feature vectors, and cluster all smart driving clients based on the multiple correlation distances to obtain multiple client cluster sets.

[0232] In some embodiments, the clustering subunit 442 is specifically configured to determine the Euclidean distance between the feature vectors of each two intelligent driving clients as the corresponding correlation distance;

[0233] The clustering subunit 442 is specifically used to cluster all intelligent driving clients based on multiple correlation distances, K-means algorithm and silhouette coefficient method to obtain multiple client cluster sets.

[0234] In some embodiments, the parameter calculation unit 460 includes:

[0235] The selection subunit 461 is configured to select a target client cluster set from a plurality of intelligent driving clients; the target client cluster set includes the target client and other clients;

[0236] The third calculation subunit 462 is used to calculate the total number of images in all local traffic image training sets in the target client cluster set;

[0237] The third determining subunit 463 is configured to determine the ratio of the number of images in the local traffic image training set to the total number of images in the target client as the weight coefficient of the target client;

[0238] The third calculation subunit 462 is further configured to calculate target weighted model parameters based on the weight coefficient of the target client and the corresponding local model parameters;

[0239] The third calculation subunit 462 is further configured to calculate the sum of all target weighted model parameters in the target client cluster set to obtain the common model parameters of the target client cluster set;

[0240] The above operations are repeated until the common model parameters of all target client cluster sets are calculated.

[0241] like Figure 6 As shown, the present application provides an electronic device 500, which includes a processor 501 and a memory 502. The processor 501 and the memory 502 are interconnected and communicate with each other through a communication bus 503 and / or other forms of connection mechanisms (not shown). The memory 502 stores a computer program executable by the processor 501. When the computing device is running, the processor 501 executes the computer program to perform the method in any of the aforementioned optional implementations.

[0242] The present application provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the method in any of the aforementioned optional implementations is executed.

[0243] Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0244] The present application provides a computer program product, which includes computer programmability. When the computer program is executed by a processor, the method in any of the aforementioned optional implementations is executed.

[0245] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A federated learning method based on dual clustering, characterized in that: The method is applied to a road condition image federated learning center, and the method includes: Receive a connection weight parameter vector set uploaded by multiple intelligent driving clients; the connection weight parameter vector set includes multiple connection weight parameter vectors, the connection weight parameter vector set is extracted from a pre-trained model obtained by the intelligent driving client after completing pre-training, and the extraction location is the convolution layer closest to the input layer; wherein each connection weight parameter vector corresponds to a convolution kernel in the convolution layer; Calculating a distance metric value between every two connection weight parameter vectors in all connection weight parameter vector sets, and clustering all connection weight parameter vectors based on multiple distance metric values ​​to obtain multiple parameter vector cluster sets; Clustering all intelligent driving clients based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets to obtain multiple client cluster sets; Send the initialization model parameters of the road condition image recognition model to all intelligent driving clients, so that all intelligent driving clients can upload the local model parameters of the local model after completing local training to obtain the local model; Calculate the common model parameters of each client cluster set based on multiple local model parameters and the number of images in multiple local traffic image training sets; The corresponding shared model parameters are issued to all intelligent driving clients, so that after completing local training to obtain a local model, all intelligent driving clients upload the local model update parameters of the local model, and trigger the execution of the step of calculating the shared model parameters of each client cluster set based on multiple local model parameters and the number of images in the multiple local road condition image training sets; Among them, when the preset training termination condition is reached, the intelligent driving client stops uploading the local model update parameters and determines the local model update parameters as the final parameters of the local model.

2. The federated learning method based on dual clustering according to claim 1, characterized in that: The method further comprises: Build a road condition image recognition model based on CNN architecture; Initializing parameters of the road condition image recognition model to obtain initialized model parameters; The model architecture and the initialization parameters of the road condition image recognition model are sent to multiple intelligent driving clients, so that the multiple intelligent driving clients feedback a connection weight parameter vector set.

3. The federated learning method based on dual clustering according to claim 1, characterized in that: The calculating of the distance metric between every two connection weight parameter vectors in the set of all connection weight parameter vectors includes: Determine two connection weight parameter vectors in the set of all connection weight parameter vectors as a first connection weight parameter vector and a second connection weight parameter vector; extracting a first connection weight parameter set from the first connection weight parameter vector, and extracting a second connection weight parameter set from the second connection weight parameter vector; Calculating a first mean of the first connection weight parameter set, a first standard deviation of the first connection weight parameter set, a second mean of the second connection weight parameter set, and a second standard deviation of the second connection weight parameter set, and calculating a covariance between the first connection weight parameter set and the second connection weight parameter set; Calculating based on a similarity index calculation formula, the first mean, the first standard deviation, the second mean, the second standard deviation, and the covariance to obtain a similarity index between the first connection weight parameter vector and the second connection weight parameter vector; determining the complement of the similarity index as a distance metric between the first connection weight parameter vector and the second connection weight parameter vector; Repeat the above steps until the distance metric between every two connection weight parameter vectors in all connection weight parameter vector sets is calculated.

4. The federated learning method based on dual clustering according to claim 1, characterized in that: The method clusters all intelligent driving clients based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets to obtain multiple client cluster sets, including: Based on multiple connection weight parameter vector sets and multiple parameter vector cluster sets, calculate the feature vector of the local road condition image training set in each intelligent driving client; Based on multiple feature vectors, the correlation distance between every two smart driving clients is determined, and all smart driving clients are clustered based on the multiple correlation distances to obtain multiple client cluster sets.

5. The federated learning method based on dual clustering according to claim 4, characterized in that: The method determines the correlation distance between each two intelligent driving clients based on multiple feature vectors, and clusters all intelligent driving clients based on the multiple correlation distances to obtain multiple client cluster sets, including: The Euclidean distance between the feature vectors of each two intelligent driving clients is determined as the corresponding correlation distance; Based on multiple correlation distances, K-means algorithm and silhouette coefficient method, all intelligent driving clients are clustered to obtain multiple client cluster sets.

6. The federated learning method based on dual clustering according to claim 1, characterized in that: The method of calculating the common model parameters of each client cluster set based on the multiple local model parameters and the number of images in the multiple local traffic image training sets includes: Selecting a target client cluster set from a plurality of intelligent driving clients; the target client cluster set includes the target client and other clients; Calculating the total number of images in all local traffic image training sets in the target client cluster set; Determine the ratio of the number of images in the local traffic image training set in the target client to the total number of images as the weight coefficient of the target client; Calculating target weighted model parameters based on the weight coefficient of the target client and the corresponding local model parameters; Calculating the sum of all target weighted model parameters in the target client cluster set to obtain the common model parameters of the target client cluster set; Repeat the above steps until the common model parameters of all target client cluster sets are calculated.

7. A road condition image recognition method, characterized in that: The method comprises: Obtaining a road condition image to be identified; The road condition image to be identified is input into a local model in the intelligent driving client, so that the local model outputs road condition information corresponding to the road condition image to be identified; wherein, the model parameters of the local model are determined by the federated learning method based on the dual clustering method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the federated learning method based on dual clustering according to any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the federated learning method based on dual clustering according to any one of claims 1 to 6 is executed.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the computer program executes the federated learning method based on dual clustering according to any one of claims 1 to 6.

Images