Federated learning method, system and device

Abstract

The application provides a transverse federated learning method, system and device, the method comprising: each participating device of the federated learning uses an ensemble learning method to perform T rounds of data sampling, and each round of data sampling obtains a group of sampling sets with close distribution; wherein each group of sampling sets comprises a sampling set corresponding to each participating device respectively; each participating device uses each group of sampling sets to perform transverse federated learning respectively, and generates T base learners; T is an integer greater than 1; since each group of sampling sets has close distribution, the problem of non-independent and identically distributed original data is overcome, so that the model precision of the trained base learners can be improved; at the same time, the distributions of different groups of sampling sets are also different, so that the base learners trained by different groups of sampling sets will be different. According to the ensemble learning theory, the base learner set meeting the two conditions has higher model precision when performing ensemble learning.

Description

Technical Field

[0001] This invention relates to the fields of federated learning and big data analytics, and in particular to a horizontal federated learning method, system, and device. Background Technology

[0002] In recent years, artificial intelligence has entered the era of big data, dominated by deep learning, with big data being a fundamental element of current AI applications. However, in practical applications, we face the following challenges: data is small-scale, fragmented, lacks data labels, is scattered, and suffers from data silos caused by user privacy and data security concerns. The traditional approach is to aggregate scattered data into a data center for modeling. However, with the continuous expansion of application areas and the ongoing improvement of laws and regulations protecting user privacy, this traditional method of data aggregation followed by modeling is becoming increasingly difficult to implement. Therefore, federated learning has emerged. The basic idea of ​​federated learning is to build a high-performance model shared by multiple participants while ensuring user privacy and data security, aiming to solve the problems of privacy protection and data silos. Specifically, it uses data from multiple parties for joint modeling. During the modeling process, each party's data remains local and is not exposed to other participating parties, thus protecting user privacy and data security while simultaneously building a high-performance model using the data from all parties.

[0003] Horizontal federated learning is an important component of federated learning, primarily applied when multiple participants have similar data structures but insufficient data volume, aiming to obtain high-performance models through joint modeling without exposing data privacy. In horizontal federated learning, each participant's algorithm model and data features are consistent. Each participant trains a local model using local data and transmits the parameters to the coordinator. The coordinator merges the uploaded model parameters into a global model, which is then transmitted to each participant for local updates. The basic process is as follows: ① Local training task configuration; ② Local model training and parameter extraction; ③ Encrypted transmission of model parameters; ④ Merging of parameterized models; ⑤ Distribution of merged model parameters; ⑥ Local updates. This is the currently common horizontal federated learning process in the industry. However, in practical applications, problems such as poor model performance arise, with the non-independent and identically distributed data among nodes being a significant contributing factor. Currently, no mature method has been found to address these issues. Summary of the Invention

[0004] The purpose of this invention is to provide a horizontal federated learning method, system, and device to solve the problem of inaccurate prediction results of the base learners generated by existing horizontal federated learning methods.

[0005] To address the above problems, embodiments of the present invention provide a horizontal federated learning method, including:

[0006] Each participating device in federated learning uses an ensemble learning approach to perform T rounds of data sampling. Each round of data sampling yields a set of samples with similar distributions. Each set of samples includes the sample sets corresponding to each participating device.

[0007] Each participating device performs lateral federated learning using each sample set to generate T base learners; T is an integer greater than 1.

[0008] In this process, each participating device in the federated learning uses an ensemble learning approach to perform one round of data sampling, resulting in a set of samples with similar distributions, including:

[0009] The leading participating device randomly samples from the local dataset to obtain the first sample set;

[0010] Other participating devices sample from their local datasets based on the sample distribution vector of the first sample set to obtain a second sample set whose distribution is similar to that of the first sample set.

[0011] The set of sample sets with similar distributions includes the first sample set and at least one second sample set.

[0012] In this context, all participating devices in the federated learning process are sequentially designated as the leading participating device.

[0013] Before the participating devices in the federated learning process perform T rounds of data sampling using ensemble learning, the method further includes:

[0014] Each participating device receives the data value range division scheme for each numerical feature and the value points for each categorical feature sent by the coordinating device;

[0015] Each participating device performs one-hot encoding on the local dataset according to the data value range division scheme of each numerical feature and the value points of each categorical feature, transforming each sample of the local dataset into a k-dimensional 0-1 vector, where k is an integer greater than 1.

[0016] Wherein, after the leading participating device obtains the first sample set by randomly sampling from the local dataset, the method further includes:

[0017] The leading participating device calculates the sample distribution vector of the first sampling set based on the k-dimensional 0-1 vector of the samples included in each label of the first sampling set; the sample distribution vector is used to indicate the proportion of the number of samples whose values ​​of each numerical feature fall within the sub-interval sent by the coordination device to the total number of samples of the label to which the numerical feature belongs, and / or the proportion of the number of samples whose values ​​of each categorical feature fall within the category sent by the coordination device to the total number of samples of the label to which the categorical feature belongs.

[0018] The leading participating device sends the sample distribution vector of the first sampling set to the other participating devices.

[0019] Among them, other participating devices sample from their local datasets based on the sample distribution vector of the first sampling set to obtain a second sampling set whose distribution is similar to that of the first sampling set, including:

[0020] The other participating devices perform random sampling with replacement in the local dataset, and determine whether to include the sample in the second sampling set based on the Manhattan distance between the 0-1 vector of the sample and the sample distribution vector of the first sampling set, until the number of samples in the second sampling set reaches a set value, at which point sampling stops.

[0021] The other participating devices determine whether to include the sample in the second sample set based on the Manhattan distance between the 0-1 vector of the sample and the sample distribution vector of the first sample set, including:

[0022] The other participating devices calculate the Manhattan distance between the sampled sample and the sample distribution vector of the first sample set;

[0023] If the Manhattan distance is less than the first threshold, the other participating devices determine that the sampled sample is a valid sample and include it in the second sample set; otherwise, the sampled sample is determined to be an invalid sample.

[0024] The method further includes:

[0025] Each participating device determines the value range of each numerical feature and the value point of each categorical feature in its local dataset;

[0026] Each participating device adds perturbations to the value range of each numerical feature and the value point of each categorical feature, and sends the perturbated value range of each numerical feature and the value point of each categorical feature to the coordinating device; so that the coordinating device can determine the data value range division scheme of each numerical feature and the value point of each categorical feature.

[0027] This invention also provides a horizontal federated learning system, comprising:

[0028] The sampling module is used by the participating devices in federated learning to perform T rounds of data sampling using ensemble learning methods. Each round of data sampling yields a set of samples with similar distributions; each set of samples includes the sample sets corresponding to each participating device.

[0029] The generation module is used by each participating device to perform lateral federated learning on each sample set to generate T base learners; T is an integer greater than 1.

[0030] The sampling module includes:

[0031] The first sampling module is used to randomly sample the first sample set from the local dataset by the leading participating device;

[0032] The second sampling module is used by other participating devices to sample from the local dataset based on the sample distribution vector of the first sampling set, so as to obtain a second sampling set that is close to the distribution of the first sampling set.

[0033] The set of sample sets with similar distributions includes the first sample set and at least one second sample set.

[0034] In this context, all participating devices in the federated learning process are sequentially designated as the leading participating device.

[0035] The system also includes:

[0036] The first receiving module is used by each participating device to receive the data value range division scheme of each numerical feature and the value points of each categorical feature sent by the coordinating device;

[0037] The preprocessing module is used by the participating devices to perform one-hot encoding on the local dataset according to the data value range division scheme of each numerical feature and the value points of each categorical feature, transforming each sample of the local dataset into a k-dimensional 0-1 vector, where k is an integer greater than 1.

[0038] The system also includes:

[0039] The calculation module is used by the leading participating device to calculate the sample distribution vector of the first sampling set based on the k-dimensional 0-1 vector of the samples included in each label of the first sampling set; the sample distribution vector is used to indicate the proportion of the number of samples whose values ​​of each numerical feature fall within the sub-interval sent by the coordination device to the total number of samples of the label to which the numerical feature belongs, and / or the proportion of the number of samples whose values ​​of each categorical feature fall within the category sent by the coordination device to the total number of samples of the label to which the categorical feature belongs;

[0040] The sending module is used to send the sample distribution vector of the first sampling set from the leading participating device to other participating devices.

[0041] The second sampling module includes:

[0042] The sampling unit is used by the other participating devices to perform random sampling with replacement in the local dataset, and to determine whether to include the sampled sample in the second sampling set based on the Manhattan distance between the 0-1 vector of the sampled sample and the sample distribution vector of the first sampling set, until the number of samples in the second sampling set reaches a set value and then sampling stops.

[0043] The sampling unit includes:

[0044] A computational subunit is used by the other participating devices to calculate the Manhattan distance between the sampled sample and the sample distribution vector of the first sample set;

[0045] The determination subunit is used to determine that if the Manhattan distance is less than a first threshold, the other participating devices determine that the sampled sample is a valid sample and include the sampled sample in the second sample set; otherwise, the sampled sample is determined to be an invalid sample.

[0046] The system also includes:

[0047] The determination module is used by each participating device to determine the value range of each numerical feature and the value point of each categorical feature in the local dataset;

[0048] The reporting module is used to have each participating device add perturbations to the value range of each numerical feature and the value point of each categorical feature, and send the perturbated value range of each numerical feature and the value point of each categorical feature to the coordinating device; so that the coordinating device can determine the data value range division scheme of each numerical feature and the value point of each categorical feature.

[0049] This invention also provides a participating device for horizontal federated learning, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the horizontal federated learning method as described above.

[0050] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the horizontal federated learning method described above.

[0051] The above-described technical solution of the present invention has at least the following beneficial effects:

[0052] The horizontal federated learning method, system, and participating devices of this invention involve each participating device performing T rounds of data sampling using an ensemble learning method. Each round of sampling yields a set of samples with similar distributions. These sets are then federated to train a base learner. Because each set of samples has a similar distribution, the problem of non-independent and identically distributed original data is overcome, thus improving the model accuracy of the trained base learner. However, since different sets of samples have different distributions, the base learners trained from different sets will differ. According to ensemble learning theory, a set of base learners that satisfies these two conditions will achieve higher model accuracy through ensemble learning. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the steps of the horizontal federated learning method provided in this embodiment of the invention.

[0054] Figure 2 An example diagram illustrating the horizontal federated learning method provided in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram illustrating the structure of the horizontal federated learning system provided in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram showing the structure of the participating device for horizontal federated learning provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0058] like Figure 1 As shown, this embodiment of the invention provides a horizontal federated learning method, including:

[0059] Step 101: Each participating device in the federated learning uses an ensemble learning method to perform T rounds of data sampling. Each round of data sampling yields a set of samples with similar distributions. Each set of samples includes the sample sets corresponding to each participating device.

[0060] Step 102: Each participating device performs horizontal federated learning using each sample set to generate T base learners; T is an integer greater than 1.

[0061] In this embodiment of the invention, the above process generates a total of T base learners, which can be used to complete prediction or regression tasks; for example, a simple voting method is used for classification problems, and an averaging method is used for regression problems.

[0062] This invention introduces the concept of ensemble learning into federated learning, constructing multiple base learners through federated ensemble learning. Its main feature is the application of Bagging (guided aggregation algorithm) in ensemble learning. Data is sampled from the datasets of each participating party in the modeling process, and through a specific sampling mechanism, the distribution of the sample sets of each node is made similar, resulting in a set of sample sets. Multiple base learners are then trained using federated learning with these similarly distributed sample sets, ultimately outputting a set of base learners. During prediction, data is input into this set of base learners, and the results of each base learner are fused to obtain the final prediction result. This federated learning method based on ensemble learning can improve the accuracy and efficiency of federated modeling. Specifically, because each set of sample sets has a similar distribution, the problem of non-independent and identically distributed original data is overcome, thus ensuring the accuracy of the trained base learners. At the same time, the distributions of different sets of sample sets are also different, resulting in differences in the trained base learners. According to ensemble learning theory, a set of base learners that meets these two conditions will result in higher model accuracy when ensemble learning is performed.

[0063] In at least one embodiment of the present invention, in step 101, each participating device in the federated learning performs a round of data sampling using an ensemble learning method to obtain a set of sampled data with similar distributions, including:

[0064] The leading participating device randomly samples from the local dataset to obtain the first sample set;

[0065] Other participating devices sample from their local datasets based on the sample distribution vector of the first sample set to obtain a second sample set whose distribution is similar to that of the first sample set.

[0066] The set of sample sets with similar distributions includes the first sample set and at least one second sample set.

[0067] In this context, all participating devices in the federated learning process are sequentially designated as the leading participating device.

[0068] For example, the participating devices in federated learning include node A and node B. The aforementioned T rounds of data sampling include: round T1 data sampling with node A as the leading participating device, and round T2 data sampling with node B as the leading participating device, i.e., T = T1 + T2. Optionally, T1 and T2 are both integers greater than or equal to 1, and T1 and T2 can be the same or different; no specific restrictions are imposed here.

[0069] This invention introduces the concept of ensemble learning into federated learning, but it does not merely transplant the ensemble learning process into federated learning. Compared to traditional ensemble learning, a data sampling method is designed based on the actual modeling process of federated learning. This includes the following:

[0070] 1. The sampling method consists of two stages. The first stage is that the leading participating device randomly samples with replacement from the local dataset. The second stage is that all other participating devices sample according to the distribution of the sampling set of the leading participating device.

[0071] 2. All participating devices in the modeling process take turns being the leading participating device;

[0072] The above scheme can ensure the consistency of data distribution among nodes participating in the training of base learners, thus guaranteeing the accuracy of each base learner. On the other hand, all participating modeling devices take turns being the leading participating device, ensuring that each base learner has certain differences, and that the data features of each participating device can be fully reflected in the base learner, thus guaranteeing the prediction accuracy of the entire ensemble learning.

[0073] In at least one embodiment of the present invention, before each participating device in the federated learning performs T rounds of data sampling using an ensemble learning method, each participating device preprocesses its local dataset for ease of subsequent sampling; that is, the method further includes:

[0074] Each participating device receives the data value range division scheme for each numerical feature and the value points for each categorical feature sent by the coordinating device;

[0075] Each participating device performs one-hot encoding on the local dataset according to the data value range division scheme of each numerical feature and the value points of each categorical feature, transforming each sample of the local dataset into a k-dimensional 0-1 vector, where k is an integer greater than 1.

[0076] In this embodiment of the invention, each participating device performs one-hot encoding on its local dataset according to the data value range division scheme for each numerical feature and the value points for each categorical feature published by the coordinating device. This can also be called one-bit effective encoding. For example, suppose that after the above one-hot encoding preprocessing, a certain sample is transformed into a k-dimensional 0-1 vector. Suppose that the sample has n features, and the dimension of the 0-1 vector corresponding to each feature is... ,but .

[0077] For example, suppose a sample has two features. After preprocessing, the sample becomes a 6-dimensional 0-1 vector, with each feature corresponding to a 3-dimensional 0-1 vector. A local dataset with a certain label contains 5 samples. The preprocessed samples are shown below:

[0078]

[0079] Wherein, X1 is a 6-dimensional 0-1 vector after preprocessing sample 1; X2 is a 6-dimensional 0-1 vector after preprocessing sample 2; X3 is a 6-dimensional 0-1 vector after preprocessing sample 3; X4 is a 6-dimensional 0-1 vector after preprocessing sample 4; and X5 is a 6-dimensional 0-1 vector after preprocessing sample 5.

[0080] Accordingly, in this embodiment of the invention, after the dominant participating device obtains the first sampling set by randomly sampling from the local dataset, the method further includes:

[0081] The leading participating device calculates the sample distribution vector of the first sampling set based on the k-dimensional 0-1 vector of the samples included in each label of the first sampling set; the sample distribution vector is used to indicate the proportion of the number of samples whose values ​​of each numerical feature fall within the sub-interval sent by the coordination device to the total number of samples of the label to which the numerical feature belongs, and / or the proportion of the number of samples whose values ​​of each categorical feature fall within the category sent by the coordination device to the total number of samples of the label to which the categorical feature belongs.

[0082] The leading participating device sends the sample distribution vector of the first sampling set to the other participating devices.

[0083] In short, after preprocessing, all samples in the first sampling set are k-dimensional 0-1 vectors. The percentage of samples with each dimension set to 1 is counted to obtain the sample distribution vector of the first sampling set. ;in n is the number of features. Let be the dimension of the 0-1 vector for each feature.

[0084] For example, suppose there are 5 samples in a certain label in the first sampling set, and the preprocessed samples are as follows:

[0085] ,but The leading participating device will It is sent to other participating devices as a basis for their sampling.

[0086] In at least one embodiment of the present invention, other participating devices sample from a local dataset based on the sample distribution vector of the first sampling set to obtain a second sampling set whose distribution is close to that of the first sampling set, including:

[0087] The other participating devices perform random sampling with replacement in the local dataset, and determine whether to include the sample in the second sampling set based on the Manhattan distance between the 0-1 vector of the sample and the sample distribution vector of the first sampling set, until the number of samples in the second sampling set reaches a set value, at which point sampling stops.

[0088] The other participating devices determine whether to include the sample in the second sample set based on the Manhattan distance between the 0-1 vector of the sample and the sample distribution vector of the first sample set, including:

[0089] The other participating devices calculate the Manhattan distance between the sampled sample and the sample distribution vector of the first sample set;

[0090] If the Manhattan distance is less than the first threshold, the other participating devices determine that the sampled sample is a valid sample and include it in the second sample set; otherwise, the sampled sample is determined to be an invalid sample.

[0091] For example, by performing random sampling with replacement on a preprocessed dataset, a sample is obtained. ;calculate and Manhattan distance ,like Then it is believed It is effective sampling, The second sampling set included other participating devices Otherwise, it is considered This is an invalid sample. Repeat the above sampling and Manhattan distance comparison steps until the sample size in the second sampling set reaches the preset value.

[0092] It should be noted that, according to the principles of ensemble learning, the accuracy of each base learner needs to be sufficiently high. When the data of each node is not independent and identically distributed, the data distributions of the nodes participating in the modeling differ. If the conventional bagging method of ensemble learning is used, with random sampling with replacement at each node, the data distribution of the sample sets of each node will vary significantly, and the accuracy of the model obtained through federated learning cannot be guaranteed. To solve this problem, this invention divides the sampling process into two steps: first, the leading participating device (e.g., node A) performs random sampling with replacement; then, other participating devices (e.g., node B) sample according to the distribution of node A. In this way, the distributions of the sample sets of the two nodes are relatively consistent. The specific principle is as follows:

[0093] 1. After normalizing the numerical data at each node using a uniform maximum-minimum value, one-hot encoding is performed using a uniform partitioning interval (numerical data) and a uniform classification scheme (categorical data). This allows the preprocessed data at each node to be directly compared.

[0094] 2. After random sampling with replacement at node A, calculate the proportion of nodes with a value of 1 in each dimension to obtain the data distribution vector of the sample set. Taking the communication fee dimension as an example, suppose that after normalization and one-hot encoding, the communication fee is converted into a 5-dimensional 0-1 vector. After preprocessing, in any sample, the 5-dimensional vector corresponding to its communication fee has one dimension of 1 and all other dimensions of 0. Let the part corresponding to the communication fee dimension in the distribution vector of node A's sample set be... , , , , The larger the value, the more communication charge values ​​in the sample set of node A will fall within the corresponding range. When sampling from node B, it is desirable that the distribution of communication charge values ​​in the sampled data is consistent with that of node A. The larger the value of a sub-dimension, the more samples with a value of 1 will be in the sampling set of node B, and vice versa.

[0095] Assume that the portion of the communication cost corresponding to a sample randomly selected by node B in a single sampling is... ,calculate and The Manhattan distance (the reason for using the Manhattan distance is because of the vector) It is a 0-1 sequence, and (This is a distribution vector, which is easier to calculate using Manhattan distance), resulting in:

[0096]

[0097] Obviously, here and The distance to Manhattan is actually only... It depends on the value it takes. The larger, The smaller the value, the smaller the distance between the sample and the sampling set distribution; thus, the distance between the sample and the sampling set distribution is related to the sample set... The data was linked to the proportion of data that were consistent with or similar to that sample; the closer the distance, the higher the proportion. The higher the proportion of relevant samples in the middle, the more of the collected samples should be retained. In other words, it can be used. and The distance is used to reflect the sampling of node B. and The closer the distributions are, the greater the distance. exist The smaller the proportion in the middle, the less it meets the sampling requirements; the smaller the distance, the less it meets the sampling requirements. exist The larger the proportion of the sample, the better it meets the sampling requirements. Therefore, we should consider sampling from node B. and Distribution vector The distance between them is used as the basis for determining whether a sample is qualified.

[0098] Extend to all samples The Manhattan distance between the sampled sample and the distribution vector of the sampling set of node A is a dimension. The sum of the Manhattan distances between each dimension vector and its corresponding distribution vector segment, as analyzed above, generally speaking, the smaller the distance, the more the current sample meets the sampling requirements; the larger the distance, the less the current sample meets the sampling requirements.

[0099] Therefore, in this embodiment of the invention, the sampling vector of node B is used The sample distribution vector of the sampling set of node A Manhattan distance As a criterion The basis for determining whether the sampling requirements are met. After sampling at node A is completed, when sampling at node B, for each sample drawn, the following calculation is performed. ,when If the sample size is less than a certain threshold, the sample is considered to meet the sampling distribution requirements and is retained in the sampling set of node B; otherwise, the sample is considered to not meet the sampling distribution requirements and is deleted.

[0100] Based on the calculations above, the Manhattan distance between a single original dimension and the corresponding distribution is:

[0101] ;in The distribution vector takes values ​​where the current sample value is 1. Clearly,

[0102] ;

[0103] ;

[0104] ;

[0105] Therefore, the threshold for selecting the sample can be set as follows: Specific thresholds can also be given based on empirical data.

[0106] In at least one embodiment of the present invention, the method further includes:

[0107] Each participating device determines the value range of each numerical feature and the value point of each categorical feature in its local dataset;

[0108] Each participating device adds perturbations to the value range of each numerical feature and the value point of each categorical feature, and sends the perturbated value range of each numerical feature and the value point of each categorical feature to the coordinating device; so that the coordinating device can determine the data value range division scheme of each numerical feature and the value point of each categorical feature.

[0109] In this embodiment of the invention, before each participating device performs data sampling, preparatory work is also included, specifically divided into:

[0110] a) Statistics: Each participating device statistically analyzes the values ​​of each feature data in its local samples. Numerical features are analyzed to determine the range of values, while categorical features are analyzed to determine all distinct value points.

[0111] b) Perturbation: Add random perturbation to the statistical values.

[0112] Numerical characteristics: Let the range of values ​​be... Then generate random numbers. , By adding a perturbation to the range of values, we obtain ;

[0113] Categorical features: Based on the actual significance of the feature in the relevant business, if not all possible categories are obtained, some or all missing categories can be added to the original category range.

[0114] c) Summary:

[0115] i. Each participating device sends the perturbed sample value range to the coordination node. The coordination node then summarizes the data and calculates the union to obtain the overall value range of each feature.

[0116] ii. The coordination equipment divides the value range of each feature according to the specific business situation. For numerical features, the value range is directly divided into several adjacent sub-intervals. For categorical features, the original data category can be directly used or the original category can be re-integrated.

[0117] d) Release: The coordinating device releases the overall value range of each feature segmented and reorganized to all participating devices.

[0118] To more clearly describe the horizontal federated learning method provided in the embodiments of the present invention, an example is given below.

[0119] Suppose that the participating devices in the federated learning are nodes A and B, and the data distributions of the two nodes are different (for the same category, the distributions of the corresponding features of the samples are different). Nodes A and B need to build a telecom fraud identification model through horizontal federated learning. The data features include whether the call was initiated, the proportion of roaming calls, the number of days the account has been open, the average daily call duration within 7 days, the number of calls, number segment information, gender, age, etc. Due to the different data distributions of the two parties, the federated model built directly through federated learning is not effective. To improve the model accuracy, a federated ensemble learning method is adopted. To ensure that the sampling sets of all parties are consistent, the data of each participant must first be preprocessed, and then the ensemble learning process is completed within the federated learning framework. The specific implementation process is as follows:

[0120] 1. Preparatory work:

[0121] a) Statistics: Nodes A and B respectively count the value range of each feature in their local datasets. For numerical features such as roaming call percentage, number of days since account opening, and average daily call duration within 7 days, the maximum and minimum values ​​are counted. For categorical features such as number segment information and gender, all value points are counted.

[0122] b) Disturbance:

[0123] Let the range of values ​​for the numerical feature be... Then generate random numbers. , This adds a perturbation to the range of values. , Obey The mean, Gaussian distribution with standard deviation , Positive integers Adding the perturbation to the original range of values ​​yields... ;

[0124] Categorical features: Based on the actual significance of the feature in the relevant business, if not all possible categories are obtained, some or all missing categories can be added to the original category range.

[0125] c) Each node sends its perturbed local data value range to the coordinating node of the federated learning process. The coordinating node then takes the union of all received data value ranges from each node. For numerical features, it calculates the lower and upper bounds of the union; for categorical data, it merges the values ​​from all nodes.

[0126] d) Since subsequent sampling methods require one-hot encoding of the data, it is necessary to coordinate nodes to divide the data value ranges for numerical features and classify the categories of categorical data. For example, based on experience, most fraudulent numbers have a high proportion of roaming calls; therefore, the roaming call percentage range is divided into {[0,50%],[50%,80%],[80%,100%]}. Fraudulent number users generally have accounts opened for less than 425 days; therefore, by comparing the statistically calculated account opening days range, 425 days is used as one of the dividing points. After merging and classifying the categorical data, a new category segmentation method is obtained. For example, fraudulent number segments are often 170 / 171; therefore, these two number segments can be grouped into one category. Then, the divided data value ranges and all classification methods are sent to each node.

[0127] 2. Data Preprocessing: To facilitate subsequent sampling, both A and B will first preprocess their local data. Specific methods:

[0128] Encoding: Based on the data value ranges, partitioning schemes, and classification methods published by the coordinating node, one-hot encoding is performed on all node data; assuming the original data has... After the above preprocessing, each sample is transformed into a k-dimensional 0-1 vector. Let the dimension of the 0-1 vector corresponding to each feature be k. , .

[0129] 3. Data sampling: such as Figure 2 As shown, node A is sampled first, and after node B is sampled, its distribution is compared with that of the samples in node A. Samples that are close to the distribution are retained, and those that are deleted are removed. Let the sample sizes of nodes A and B be respectively... , Specifically:

[0130] Node A:

[0131] i. First, perform random sampling with replacement to obtain samples containing... A sample set of samples ;

[0132] ii. After preprocessing, all samples are k-dimensional 0-1 vectors. The percentage of samples with a value of 1 for each category and each dimension is calculated. Medium sample distribution vector ,in .

[0133] For example, numerical features: based on the segmented data value range of the relevant features published by the coordinating node, calculate the proportion of samples for each category and each feature value falling within each segmented sub-interval out of the total number of samples in the relevant category; categorical features: based on the reorganized data value range of the relevant features published by the coordinating node, calculate the proportion of samples for each category and each feature value falling within each integrated category out of the total number of samples in the relevant category; the above distributions are merged to form The sample distribution vector;

[0134] Regarding the sample distribution vector mentioned above, an example is shown below: Suppose a sample has two features, which, after preprocessing, become a 6-dimensional 0-1 vector, with each feature corresponding to a 3-dimensional 0-1 vector. In a dataset with 5 samples in a certain category, the preprocessed samples and their distribution vectors are shown below.

[0135]

[0136] Will It is sent to node B as the basis for node B's sampling.

[0137] Node B:

[0138] i. Perform random sampling with replacement on the preprocessed dataset to obtain a sample. ;

[0139] ii. Calculation and Manhattan distance ,like Then it is believed It is effective sampling, The sampling set included in node B Otherwise, it is considered This is an invalid sample;

[0140] calculate Number of elements ,like If the condition is met, return i; otherwise, end sampling at node B.

[0141] 4. Joint modeling: In node A With node B Joint modeling is performed following a standard horizontal federated learning process to obtain a base learner dominated by node A's data. ;

[0142] 5. Return to step 3, and repeat steps 3 and 4 a total of n1 times to construct a base learner dominated by n1 nodes A;

[0143] 6. Return to step 3 and repeat steps 3 and 4, but with the roles of node A and node B reversed. Node B first performs random sampling without replacement, and node A samples according to the distribution of node B's sample set. After sampling, both nodes jointly model the model through lateral federated learning to obtain a base learner dominated by node B's data. Repeat the above process to train a total of n² node B-dominated base learners;

[0144] Through the above federated ensemble learning process, a total of n1+n2 base learners were trained to identify telecommunications fraud.

[0145] In summary, this invention applies the concept of ensemble learning to federated learning, which can significantly improve modeling accuracy. Combined with the federated learning model training process, a step is added to the basic ensemble learning process where the leading node first randomly samples, and other nodes perform precise sampling based on the distribution of the leading node's sampling set. This ensures the consistency of data distribution among all nodes participating in the training of the base learners, thus guaranteeing the accuracy of the base learners. A method for precise sampling based on the sampling set distribution is designed. In the sampling process of ensemble learning, a process is designed in which all participating modeling nodes take turns leading the sampling distribution, ensuring the diversity of each base learner, thereby effectively improving the prediction accuracy of ensemble learning.

[0146] like Figure 3 As shown, this embodiment of the invention also provides a horizontal federated learning system, including:

[0147] The sampling module 301 is used to perform T rounds of data sampling by the participating devices in federated learning using an ensemble learning method. Each round of data sampling yields a set of sampled data with similar distributions. Each set of sampled data includes the sampled data corresponding to each participating device.

[0148] The generation module 302 is used by each participating device to perform horizontal federated learning using each set of samples to generate T base learners; T is an integer greater than 1.

[0149] As an optional embodiment, the sampling module includes:

[0150] The first sampling module is used to randomly sample the first sample set from the local dataset by the leading participating device;

[0151] The second sampling module is used by other participating devices to sample from the local dataset based on the sample distribution vector of the first sampling set, so as to obtain a second sampling set that is close to the distribution of the first sampling set.

[0152] The set of sample sets with similar distributions includes the first sample set and at least one second sample set.

[0153] As an optional embodiment, all participating devices in federated learning are sequentially designated as the leading participating device.

[0154] As an optional embodiment, the system further includes:

[0155] The first receiving module is used by each participating device to receive the data value range division scheme of each numerical feature and the value points of each categorical feature sent by the coordinating device;

[0156] The preprocessing module is used by the participating devices to perform one-hot encoding on the local dataset according to the data value range division scheme of each numerical feature and the value points of each categorical feature, transforming each sample of the local dataset into a k-dimensional 0-1 vector, where k is an integer greater than 1.

[0157] As an optional embodiment, the system further includes:

[0158] The calculation module is used by the leading participating device to calculate the sample distribution vector of the first sampling set based on the k-dimensional 0-1 vector of the samples included in each label of the first sampling set; the sample distribution vector is used to indicate the proportion of the number of samples whose values ​​of each numerical feature fall within the sub-interval sent by the coordination device to the total number of samples of the label to which the numerical feature belongs, and / or the proportion of the number of samples whose values ​​of each categorical feature fall within the category sent by the coordination device to the total number of samples of the label to which the categorical feature belongs;

[0159] The sending module is used to send the sample distribution vector of the first sampling set from the leading participating device to other participating devices.

[0160] As an optional embodiment, the second sampling module includes:

[0161] The sampling unit is used by the other participating devices to perform random sampling with replacement in the local dataset, and to determine whether to include the sampled sample in the second sampling set based on the Manhattan distance between the 0-1 vector of the sampled sample and the sample distribution vector of the first sampling set, until the number of samples in the second sampling set reaches a set value and then sampling stops.

[0162] As an optional embodiment, the sampling unit includes:

[0163] A computational subunit is used by the other participating devices to calculate the Manhattan distance between the sampled sample and the sample distribution vector of the first sample set;

[0164] The determination subunit is used to determine that if the Manhattan distance is less than a first threshold, the other participating devices determine that the sampled sample is a valid sample and include the sampled sample in the second sample set; otherwise, the sampled sample is determined to be an invalid sample.

[0165] As an optional embodiment, the system further includes:

[0166] The determination module is used by each participating device to determine the value range of each numerical feature and the value point of each categorical feature in the local dataset;

[0167] The reporting module is used to have each participating device add perturbations to the value range of each numerical feature and the value point of each categorical feature, and send the perturbated value range of each numerical feature and the value point of each categorical feature to the coordinating device; so that the coordinating device can determine the data value range division scheme of each numerical feature and the value point of each categorical feature.

[0168] The horizontal federated learning method, system, and participating devices of this invention involve each participating device performing T rounds of data sampling using an ensemble learning method. Each round of sampling yields a set of samples with similar distributions. These sets are then federated to train a base learner. Because each set of samples has a similar distribution, the problem of non-independent and identically distributed original data is overcome, thus improving the model accuracy of the trained base learner. However, since different sets of samples have different distributions, the base learners trained from different sets will differ. According to ensemble learning theory, a set of base learners that satisfies these two conditions will achieve higher model accuracy through ensemble learning.

[0169] It should be noted that the horizontal federated learning system provided in the embodiments of the present invention is a system capable of executing the above-described horizontal federated learning method. Therefore, all embodiments of the above-described horizontal federated learning method are applicable to this system and can achieve the same or similar beneficial effects.

[0170] like Figure 4 As shown, this embodiment of the invention also provides a participating device for horizontal federated learning, including a memory 410, a processor 400, and a computer program stored in the memory 410 and executable on the processor 400. When the processor 400 executes the program, it implements the various processes in the horizontal federated learning method embodiment described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0171] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this program implements the various processes described above in the embodiments of the horizontal federated learning method, achieving the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0172] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0173] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 A device for one or more processes and / or the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce a paper article including an instruction means, the instruction means being implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment, causing the computer or other programmable equipment to perform a series of operational steps to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for identifying telecommunications fraud, characterized in that, include: The data is input into T base learners, the outputs of the T base learners are obtained, and the outputs of the T base learners are fused to obtain the prediction result; T is an integer greater than 1; the T base learners form a telecommunications fraud identification model, and the prediction result is used to indicate whether the input data is telecommunications fraud. The data features include: whether the call was initiated proactively, the percentage of roaming calls, the number of days since the account was opened, the average daily call duration within 7 days, the number of calls, number segment information, gender, and age; The T base learners are generated by the participating devices in the federated learning process through horizontal federated learning using multiple sets of sample data. Each participating device in the federated learning process performs T rounds of data sampling, and each round of data sampling yields a set of sample data with a similar distribution. Each set of sample data includes the sample data corresponding to each participating device. In this process, each participating device in the federated learning process performs a round of data sampling to obtain a set of samples with similar distributions, including: The leading participating device randomly samples from the local dataset to obtain the first sample set; Other participating devices sample from their local datasets based on the sample distribution vector of the first sample set to obtain a second sample set whose distribution is close to that of the first sample set; wherein, the set of sample sets whose distributions are close includes the first sample set and at least one second sample set; Before each participating device in the federated learning process performs T rounds of data sampling, the method further includes: Each participating device receives the data value range division scheme for each numerical feature and the value points for each categorical feature sent by the coordinating device; Each participating device performs one-hot encoding on the local dataset according to the data value range division scheme of each numerical feature and the value points of each categorical feature, transforming each sample of the local dataset into a k-dimensional 0-1 vector, where k is an integer greater than 1. Wherein, after the leading participating device obtains the first sample set by randomly sampling from the local dataset, the method further includes: The dominant participating device calculates the sample distribution vector of the first sampling set based on the k-dimensional 0-1 vector of the samples included in each label of the first sampling set; the sample distribution vector is used to indicate the proportion of the number of samples whose values ​​of each numerical feature fall within the sub-interval sent by the coordination device to the total number of samples of the label to which the numerical feature belongs, and / or the proportion of the number of samples whose values ​​of each categorical feature fall within the category sent by the coordination device to the total number of samples of the label to which the categorical feature belongs. The leading participating device sends the sample distribution vector of the first sampling set to the other participating devices.

2. The method according to claim 1, characterized in that, All participating devices in federated learning are sequentially designated as the leading participating device.

3. The method according to claim 1, characterized in that, Other participating devices sample from their local datasets based on the sample distribution vector of the first sample set to obtain a second sample set whose distribution is similar to that of the first sample set, including: The other participating devices perform random sampling with replacement in the local dataset, and determine whether to include the sample in the second sampling set based on the Manhattan distance between the 0-1 vector of the sample and the sample distribution vector of the first sampling set, until the number of samples in the second sampling set reaches a set value, at which point sampling stops.

4. The method according to claim 3, characterized in that, The other participating devices determine whether to include the sample in the second sample set based on the Manhattan distance between the 0-1 vector of the sample and the sample distribution vector of the first sample set, including: The other participating devices calculate the Manhattan distance between the sampled sample and the sample distribution vector of the first sample set; If the Manhattan distance is less than the first threshold, the other participating devices determine that the sampled sample is a valid sample and include it in the second sample set; otherwise, the sampled sample is determined to be an invalid sample.

5. The method according to claim 1, characterized in that, The method further includes: Each participating device determines the value range of each numerical feature and the value point of each categorical feature in its local dataset; Each participating device adds perturbations to the value range of each numerical feature and the value point of each categorical feature, and sends the perturbated value range of each numerical feature and the value point of each categorical feature to the coordinating device; so that the coordinating device can determine the data value range division scheme of each numerical feature and the value point of each categorical feature.

6. A telecommunications fraud identification system, characterized in that, include: The prediction module is used to input data into T base learners, obtain the output results of the T base learners, and fuse the output results of the T base learners to obtain a prediction result; T is an integer greater than 1; the T base learners form a telecommunications fraud identification model, and the prediction result is used to indicate whether the input data is telecommunications fraud. The data features include: whether the call was initiated proactively, the percentage of roaming calls, the number of days since the account was opened, the average daily call duration within 7 days, the number of calls, number segment information, gender, and age; The T base learners are generated by the participating devices in the federated learning process through horizontal federated learning using multiple sets of sample data. Each participating device in the federated learning process performs T rounds of data sampling, and each round of data sampling yields a set of sample data with a similar distribution. Each set of sample data includes the sample data corresponding to each participating device. In this process, each participating device in the federated learning performs T rounds of data sampling, with each round yielding a set of samples with similar distributions, including: The first sample set is obtained by randomly sampling from the local dataset by the leading participating device; Other participating devices sample from the local dataset based on the sample distribution vector of the first sample set to obtain a second sample set whose distribution is close to that of the first sample set; Wherein, the set of sample sets with similar distributions includes the first sample set and at least one second sample set; The system also includes: The first receiving module is used by each participating device to receive the data value range division scheme of each numerical feature and the value points of each categorical feature sent by the coordinating device; The preprocessing module is used by the participating devices to perform one-hot encoding on the local dataset according to the data value range division scheme of each numerical feature and the value points of each categorical feature, and to transform each sample of the local dataset into a k-dimensional 0-1 vector, where k is an integer greater than 1. The system also includes: The calculation module is used by the leading participating device to calculate the sample distribution vector of the first sampling set based on the k-dimensional 0-1 vector of the samples included in each label of the first sampling set; the sample distribution vector is used to indicate the proportion of the number of samples whose values ​​of each numerical feature fall within the sub-interval sent by the coordination device to the total number of samples of the label to which the numerical feature belongs, and / or the proportion of the number of samples whose values ​​of each categorical feature fall within the category sent by the coordination device to the total number of samples of the label to which the categorical feature belongs; The sending module is used to send the sample distribution vector of the first sampling set from the leading participating device to other participating devices.

7. The system according to claim 6, characterized in that, All participating devices in federated learning are sequentially designated as the leading participating device.

8. The system according to claim 6, characterized in that, The system also includes: The sampling unit is used by the other participating devices to perform random sampling with replacement in the local dataset, and to determine whether to include the sampled sample in the second sampling set based on the Manhattan distance between the 0-1 vector of the sampled sample and the sample distribution vector of the first sampling set, until the number of samples in the second sampling set reaches a set value and then sampling stops.

9. The system according to claim 8, characterized in that, The sampling unit includes: A computational subunit is used by the other participating devices to calculate the Manhattan distance between the sampled sample and the sample distribution vector of the first sample set; The determination subunit is used to determine that if the Manhattan distance is less than a first threshold, the other participating devices determine that the sampled sample is a valid sample and include the sampled sample in the second sample set; otherwise, the sampled sample is determined to be an invalid sample.

10. The system according to claim 6, characterized in that, The system also includes: The determination module is used by each participating device to determine the value range of each numerical feature and the value point of each categorical feature in the local dataset; The reporting module is used to have each participating device add perturbations to the value range of each numerical feature and the value point of each categorical feature, and send the perturbated value range of each numerical feature and the value point of each categorical feature to the coordinating device; so that the coordinating device can determine the data value range division scheme of each numerical feature and the value point of each categorical feature.

11. An apparatus comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that, When the processor executes the program, it implements the telecommunications fraud identification method as described in any one of claims 1-5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the telecommunications fraud identification method as described in any one of claims 1-5.

Images