Communication-efficient federal personalized feature selection method and system

Abstract

The invention provides a federal personalized feature selection method and system with efficient communication, and the method comprises the steps: S1, carrying out the feature analysis on local equipment of a plurality of data owners, and transmitting a preliminary analysis result to a central server; s2, the central server receives the preliminary analysis result and generates a personalized feature selection strategy for the plurality of local devices; s3, selecting a strategy training model by each local device according to the received personalized features, and sending updated model parameters to a central server; s4, the central server summarizes the model parameters sent by all the devices, and the global model is updated. According to the method, a personalized feature selection mechanism adapted to local data features of all data owners is introduced, and the defects of a traditional global feature selection method in processing heterogeneous data with different feature dimensions, data distribution and sample sizes are overcome, so that the overall performance and local adaptability of a federated learning model are improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and big data analysis, and in particular, to a method and system for selecting federated personalized features with efficient communication. Background Art

[0002] With the rapid development of big data and artificial intelligence, data-driven machine learning models have been widely used in various fields. However, the issue of data privacy protection has become increasingly important, especially in fields involving sensitive information such as medicine and finance. Federated Learning, as a new distributed learning paradigm, enables multi-party collaborative machine learning models while protecting data privacy by allowing each data owner (such as an institution or individual) to train the model locally and only share the model parameters.

[0003] However, a major challenge faced by federated learning in practical applications is that the data sets held by different data owners may have significant heterogeneity, which is manifested in many aspects such as feature dimension, data distribution and sample size. Traditional feature selection methods often rely on a unified data set and are difficult to effectively deal with the challenges of heterogeneous data in a federated learning environment.

[0004] At present, some studies on federated learning have proposed global feature selection methods, that is, sharing a unified feature selection scheme among all participants. However, this global scheme may not fully consider the uniqueness of each participant's data, resulting in poor performance of the model on local data. Therefore, it is necessary to design a method that can perform personalized feature selection based on the data characteristics of each participant to improve the personalized performance of the federated learning model.

[0005] In addition, in federated learning, each round of iteration involves the transmission of model parameters between the global server and multiple local clients. Communication-efficient federated learning can significantly reduce network communication costs by reducing the frequency of model updates, compressing the amount of transmitted data, or adopting more efficient encoding and transmission technologies. This is particularly beneficial for clients that need to connect through low-bandwidth networks, such as devices in remote areas or IoT devices. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a method and system for selecting federated personalized features with high communication efficiency.

[0007] A communication efficient federated personalized feature selection method provided by the present invention includes:

[0008] Step S1: Perform feature analysis on the local devices of multiple data owners to obtain preliminary analysis results, and send the results to the central server;

[0009] Step S2: The central server receives the preliminary analysis results, generates personalized feature selection strategies for multiple local devices, and sends them to the corresponding local devices;

[0010] Step S3: Each local device selects a strategy training model according to the received personalized feature, and sends the updated model parameters to the central server;

[0011] Step S4: The central server aggregates the model parameters sent by each device and updates the global model.

[0012] Preferably, it also includes:

[0013] Step S5: The central server re-evaluates the effectiveness of the personalized feature selection strategy periodically or according to specific conditions, and makes corresponding adjustments.

[0014] Preferably, the preliminary analysis results do not contain raw data information, and the preliminary analysis results include importance scores of statistical features, correlation analysis or other feature engineering operations.

[0015] Preferably, the personalized feature selection strategy is customized based on a heterogeneity measure of the data sets of the participants; the heterogeneity measure includes data distribution differences, feature dimension differences or sample size differences.

[0016] Preferably, based on the local available resources r of the current client i Adaptively adjust the compression rate when downloading and uploading the model; sort the model parameters according to the absolute value, filter out some model parameters whose absolute values ​​exceed the preset value according to the compression rate, and perform unstructured compression on the model.

[0017] Preferably, the global model is adjusted and generated based on a federal coordination mechanism; the federal coordination mechanism includes:

[0018] The global server calculates the number of local clients C participating in round t. j ,j∈Π t The uploaded model parameters dynamically adjust the aggregation parameters of the global model; through the weighted average model fusion strategy, the knowledge contained in the local model is aggregated to generate a global model:

[0019]

[0020] Among them, n j Represents the number of samples contained in the j-th client.

[0021] Preferably, the personalized feature selection strategy includes:

[0022] Each local client w iSplit into feature extractor part f i , and the classifier part c i ; Train an additional local personalized feature selection module q for each client i , the personalization module does not perform upload and aggregation operations, reads any input x, and outputs the corresponding decision vector d; the dimension of d is the same as the dimension of the feature z output by the feature extractor, and each value in d belongs to {0,1}, indicating "select" or "not select" the feature at the corresponding position, and:

[0023] z=f i (x)

[0024] d=q i (x)

[0025]

[0026] Preferably, the local client dynamically adjusts the feature selection scheme through an adaptive adjustment mechanism during the federated learning process; the local model is connected to the personalized feature selection module q i Finally, in the testing phase, it can make adaptive adjustments based on the input test samples.

[0027] Preferably, a personalized accuracy evaluation index is used for performance evaluation, and the accuracy of each client model on the local test set is collected and weighted averaged to obtain the final accuracy index:

[0028]

[0029] Where I is the indicator function, n ′ represents the number of test set samples, y k Represents the label value of the kth test sample.

[0030] A federated personalized feature selection system with efficient communication provided by the present invention includes:

[0031] Multiple local devices are used to perform feature analysis and obtain preliminary feature analysis results;

[0032] The central server is used to receive the preliminary analysis results, generate personalized feature selection strategies to enable local devices to adjust model parameters and train models;

[0033] The local device sends the adjusted parameters to the central server, which summarizes and uses them to update the global model.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The present invention overcomes the shortcomings of traditional global feature selection methods in processing heterogeneous data with different feature dimensions, data distributions and sample sizes by introducing a personalized feature selection mechanism that adapts to the local data characteristics of each data owner, thereby improving the overall performance and local adaptability of the federated learning model.

[0036] 2. The present invention pays attention to the problem of communication efficiency in the process of federated learning, and proposes a series of optimization measures such as adaptive quantization compression mechanism, federated coordination mechanism, personalized feature selection strategy, adaptive adjustment mechanism and performance evaluation to reduce the network resources required for global model updating.

[0037] Other beneficial effects of the present invention will be explained in the specific implementation manner through the introduction of specific technical features and technical solutions. Through the introduction of these technical features and technical solutions, those skilled in the art should be able to understand the beneficial technical effects brought about by the technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 The figure is a flow chart of the method of the present invention.

[0040] Figure 2 Schematic diagram of the structure of the federated learning system of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0042] Reference Figure 1 As shown, a communication efficient federated personalized feature selection method includes:

[0043] a) Perform preliminary feature analysis on each of the data sets on the local devices of multiple data owners; the preliminary feature analysis includes but is not limited to importance scoring of statistical features, correlation analysis or other feature engineering operations.

[0044] b) sending the preliminary analysis results to a central server, wherein the analysis results do not contain raw data information; the central server is also configured with an optimization algorithm to minimize the communication cost between the global server and multiple local clients during the federated learning process. The optimization algorithm includes but is not limited to reducing the frequency of model updates, compressing the amount of transmitted data, or using efficient encoding and transmission technology.

[0045] c) The central server determines one or more shared features based on the received preliminary analysis results and generates a personalized feature selection strategy for each data owner; the personalized feature selection strategy is customized based on the heterogeneity measure of the data sets of the participants, and the heterogeneity measure includes but is not limited to data distribution differences, feature dimension differences or sample size differences.

[0046] d) Sending the personalized feature selection strategy to the corresponding local device;

[0047] e) Each local device adjusts the feature set used when training its model according to the received personalized feature selection strategy;

[0048] f) training the model on the local device based on the adjusted feature set and sending the updated model parameters to the central server;

[0049] g) The central server aggregates the received model parameters to update the global model.

[0050] h) The central server re-evaluates the effectiveness of the personalized feature selection strategy periodically or according to specific conditions and makes adjustments accordingly.

[0051] Reference Figure 2 As shown, a communication efficient federated personalized feature selection system includes:

[0052] a plurality of local devices for performing preliminary feature analysis and training models according to personalized feature selection strategies thereon;

[0053] A central server that receives preliminary analysis results, determines shared features, generates personalized feature selection strategies, and aggregates model parameters to update the global model.

[0054] The federated personalized feature selection module of the present invention includes a global server S and multiple local clients C i ,i∈[N], where N is the total number of clients participating in federated learning. The global server is responsible for coordinating and managing the federated learning process, while the local clients perform local feature selection and local model w i ,i∈[N]’s training.

[0055] Adaptive quantization compression mechanism: The present invention uses the local available resources r of the current client to iAdaptively adjust the compression rate when downloading and uploading the model. The specific compression method is: first sort the model parameters according to the absolute value, then filter the model parameters with larger absolute values ​​according to the compression rate, and then perform unstructured compression on the model.

[0056] Federation coordination mechanism: The global server is based on the local client C participating in the tth round. j ,j∈Π t The uploaded model parameters dynamically adjust the aggregation parameters of the global model to take into account the different sample numbers of each participant. Through the weighted average model fusion strategy, the knowledge contained in the local model is aggregated to generate a global model:

[0057]

[0058] Among them, n j Represents the number of samples contained in the j-th client.

[0059] Personalized feature selection strategy: Each local client dynamically and independently executes the feature selection algorithm according to the characteristics of its local data. i Split into two parts, one part is the "feature extractor" f i , the other part is the "classifier" c i In addition, for each client, an additional local personalized feature selection module q is trained i , where s i is the trainable model parameter. The personalized module does not perform upload and aggregation operations. The personalized feature selection module reads any input x and outputs the corresponding decision vector d. The dimension of d is the same as the dimension of the feature z output by the feature extractor. Each value in d belongs to {0,1}, indicating "select" or "not select" the feature at the corresponding position. Specifically,

[0060] z=f i (x),

[0061] d=q i (x),

[0062]

[0063] Adaptive adjustment mechanism: The present invention also proposes an adaptive learning mechanism that allows the local client to dynamically adjust the feature selection scheme during the federated learning process to cope with changes in data distribution. i Each output of q considers different input samples, which can well handle the problem of inconsistent data distribution on different client devices. iAfterwards, during the testing phase, adaptive adjustments can also be made based on the input test samples.

[0064] Performance evaluation: This invention is a personalized federated learning method, which is applicable to personalized accuracy evaluation indicators. The accuracy of each client model on the local test set is collected, and then the final accuracy indicator is obtained by weighted average, that is:

[0065]

[0066] Where I is the indicator function, n ′ represents the number of test set samples, y k represents the label value of the kth test sample. In addition to the accuracy index, the present invention also evaluates the average computational cost cost of each local client c and the communication cost between the server and the client t , and its calculation process is similar to the weighted average mentioned above.

[0067] The present invention overcomes the shortcomings of traditional global feature selection methods in processing heterogeneous data with different feature dimensions, data distributions and sample sizes by introducing a personalized feature selection mechanism that adapts to the local data characteristics of each data owner, thereby improving the overall performance and local adaptability of the federated learning model.

[0068] The above is a basic embodiment of the present invention. The technical solution of the present invention is further described below through a preferred embodiment.

[0069] Example 1

[0070] A personalized feature selection based on federated learning mainly includes: global model adaptive compression download, local personalized feature selection, local model adaptive compression upload, and global model aggregation.

[0071] These include:

[0072] 1. Global model adaptive compression download

[0073] In the tth round of federated learning, the server obtains the global model w through the aggregation of the previous round t-1 The parameters of the global model are sorted according to their absolute values ​​to obtain an ordered parameter list U t-1 .

[0074] Afterwards, the server queries the historical record sequence of valid resource counts corresponding to each client based on the valid resource lists of the client uploaded in round t-1 and before. If the client does not participate in training in some rounds, the default is to fill in the early valid resource records with data. If the client has never participated in training, the default is to use the full model, that is, the default valid resources are infinite. After that, the server trains a logistic regression model for each client. i , used to fit the effective resource number history sequence seq i , and through the model l i Predict the effective resource amount of client i in round t Then, the server receives the t-1 Select the front Parameters, where is the maximum available resource of client i, which is sent to client i:

[0075]

[0076] 2. Local Personalized Feature Selection

[0077] Client i receives After that, according to the model structure, it is reconstructed into a local model This model has the same model architecture as the global model. Subsequently, client i calculates the local loss function as:

[0078]

[0079] Among them, D i For local datasets, is the loss function (such as the cross entropy loss function). Then, the model parameters and the personalized feature selection module are updated as follows:

[0080]

[0081] 3. Local model adaptive compression upload

[0082] After the local model is updated, in order to reduce communication costs, the model needs to be quantized and compressed before uploading. At this time, since it is uploaded from the client, the local effective resource volume can be easily obtained. The client directly obtains the effective resource volume from the system And for local models Perform topK sampling and then perform unstructured compression to obtain parameters Upload to server S.

[0083] 4. Global Model Aggregation

[0084] The server receives After that, we reconstruct the model for each client and get And calculate

[0085]

[0086] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0087] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A communication efficient federated personalized feature selection method, characterized in that: include: Step S1: Perform feature analysis on the local devices of multiple data owners to obtain preliminary analysis results, and send the results to the central server; Step S2: The central server receives the preliminary analysis results, generates personalized feature selection strategies for multiple local devices, and sends them to the corresponding local devices; Step S3: Each local device selects a strategy training model according to the received personalized feature, and sends the updated model parameters to the central server; Step S4: The central server aggregates the model parameters sent by each device and updates the global model.

2. The communication efficient federated personalized feature selection method according to claim 1, characterized in that: Also includes: Step S5: The central server re-evaluates the effectiveness of the personalized feature selection strategy periodically or according to specific conditions, and makes corresponding adjustments.

3. The communication efficient federated personalized feature selection method according to claim 1, characterized in that: The preliminary analysis results do not contain original data information, and the preliminary analysis results include importance scores of statistical features, correlation analysis or other feature engineering operations.

4. The communication efficient federated personalized feature selection method according to claim 1, characterized in that: The personalized feature selection strategy is customized based on the heterogeneity measure of the participant's data set; the heterogeneity measure includes data distribution difference, feature dimension difference or sample size difference.

5. The communication efficient federated personalized feature selection method according to claim 1, characterized in that: Based on the local available resources of the current client i Adaptively adjust the compression rate when downloading and uploading the model; sort the model parameters according to the absolute value, filter out some model parameters whose absolute values ​​exceed the preset value according to the compression rate, and perform unstructured compression on the model.

6. The communication efficient federated personalized feature selection method according to claim 1, characterized in that: The global model is adjusted and generated based on a federal coordination mechanism; the federal coordination mechanism includes: The global server calculates the number of local clients C participating in round t. j ,j∈Π t The uploaded model parameters dynamically adjust the aggregation parameters of the global model; through the weighted average model fusion strategy, the knowledge contained in the local model is aggregated to generate a global model: Among them, n j Represents the number of samples contained in the j-th client.

7. The communication efficient federated personalized feature selection method according to claim 4, characterized in that: The personalized feature selection strategy includes: Each local client w u Split into feature extractor part f u , and the classifier part c i ; Train an additional local personalized feature selection module q for each client i , the personalization module does not perform upload and aggregation operations, reads any input x, and outputs the corresponding decision vector d; the dimension of d is the same as the dimension of the feature z output by the feature extractor, and each value in d belongs to {0,1}, indicating "select" or "not select" the feature at the corresponding position, and: z=f i (x) d=q i (x) 8. The communication efficient federated personalized feature selection method according to claim 7, characterized in that: The local client dynamically adjusts the feature selection scheme through an adaptive adjustment mechanism during the federated learning process; the local model is connected to the personalized feature selection module q i Finally, in the testing phase, it can make adaptive adjustments based on the input test samples.

9. The communication efficient federated personalized feature selection method according to claim 1, characterized in that: Use personalized accuracy evaluation indicators to evaluate performance, collect the accuracy of each client model on the local test set and then take the weighted average to get the final accuracy indicator: Where I is the indicator function, n ′ represents the number of test set samples, y k Represents the label value of the kth test sample.

10. A communication efficient federated personalized feature selection system, based on the communication efficient federated personalized feature selection method according to any one of claims 1 to 9, characterized in that: include: Multiple local devices are used to perform feature analysis and obtain preliminary feature analysis results; The central server is used to receive the preliminary analysis results, generate personalized feature selection strategies to enable local devices to adjust model parameters and train models; The local device sends the adjusted parameters to the central server, which summarizes and uses them to update the global model.

Images