A data set construction system and method for non-independent and identically distributed federated learning

Abstract

The present invention provides a data set construction system and method for non-independent and identically distributed federated learning, the system includes an initial module, a selection module and a state sequence group extraction module; wherein the initial module is used to receive a data set, and generate an initial state sequence by sampling according to an initial probability distribution matrix and an initial probability transfer matrix; the selection module is used to receive an initial state sequence group, and generate a data set for non-independent and identically distributed federated learning; the state sequence group extraction module is used to generate a state sequence group according to the received state sequence and the number of groups. Through the present invention, a real data set is divided into a plurality of smaller subsets to form a distributed non-independent and identically distributed data set, which is easy to set different numbers of participants, and is convenient for quantifying the balance of local data, and constructing a non-independent and identically distributed scenario for research use.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data set construction system and method for non-independent and identically distributed federated learning. Background Art

[0002] Federated learning provides a modeling method to ensure data security in order to solve the problem of data islands and privacy protection. It is divided into horizontal federation, vertical federation and federated transfer learning according to the same characteristics of different users, different characteristics of the same user and different characteristics of different users. It helps the participants to cooperate to achieve the overall goal and is regarded as a technology with great application prospects. For example, the Google project Gboard based on federated learning focuses on the application of the federated average algorithm on mobile phones to monitor the statistical data of large-scale cluster devices.

[0003] Generally, the training data used for federated learning is assumed to follow independent and identical distribution, and common algorithms such as neural networks and deep learning are studied based on this. However, for application scenarios that do not meet the independent and identical distribution, it will lead to the negative effects of low accuracy of the trained model and failure of the global model to converge. Therefore, the local data set of any specific user cannot represent the overall distribution. With the integration of business between industries, more scenarios meet the situation where there are more overlapping user features. Taking smart retail business as an example, machine learning technology is used to provide users with high-quality product recommendations and sales services. The data features involved in smart retail business include user purchasing power, user personal preferences, and product features. In real life, the three data features may be scattered in three or more different departments or enterprises. For example, banks have user purchase features, shopping platforms and social networks have customer preference features, and shopping websites have product features. For the protection of user privacy and the supervision of laws and regulations, it is impossible to directly aggregate the above data scattered in different locations in one party for modeling. In the telecommunications field, operators have deployed a large number of smart devices in the network, generating a large amount of data every moment. The data is stored locally and does not need to be uploaded to the central server. However, the amount of data collected by each device and the distribution of data features vary greatly, and the generalization ability of the trained model is very poor.

[0004] The federated learning architecture consists of a central server and K decentralized clients. The K clients have datasets D1,…,D K The goal is to jointly train the global model. The central server sends the modeling task and seeks the participation of the client. The client proposes a joint modeling response according to its own needs. The central server publishes the initial parameters w to each participating client. t, each client trains the received initial parameters on the local data set, and the client uploads the weight update to the central server. The central server securely aggregates the uploaded updated weights. The above steps are to update the global model once, distribute the aggregated parameters to the client again, start the next local calculation, and iterate multiple times according to the above steps until the model converges, and the joint modeling is completed. During the whole process, the data of each participant is retained in the local training, and all participating clients jointly model. The client and the server have no right to obtain and control the information of other participants, and there is no need to share data resources to achieve the goal of training the overall model. At present, federated learning has been applied in image classification, linear regression and other aspects.

[0005] During the experimental phase of federated learning, it is very important to sample the training data independently and identically in order to ensure that the stochastic gradient is an unbiased estimate of the full gradient. In other words, having independent and identically distributed data on the client means that each mini-batch of data used for local updates on the client is statistically identical to a sample uniformly drawn from the entire training dataset (i.e., the union of all local datasets on the client). In practice, it is unrealistic to assume that the local data on each edge device is always independent and identically distributed. More specifically:

[0006] Independence violation: If the order in which the data is processed is not random enough (e.g., sorted by device collection or by time), then independence is violated. Also, devices in the same geographic location may have correlated data.

[0007] Identity violation: Because devices are bound to specific geographical regions, the distribution of tags varies between different partitions. In addition, different devices (partitions) can hold a lot of different data.

[0008] therefore,

[0009] The data of each node is generated by a unique distribution;

[0010] The amount of data on each node may also vary greatly;

[0011] There may be an underlying structure that captures the relationships between nodes and their associated distribution.

[0012] Most empirical studies on synthesizing non-IID datasets focus on skewed label distributions, i.e., forming a non-IID dataset by partitioning a flat existing dataset based on its labels.

[0013] Equally important, the distribution may change over time, which introduces another dimension of non-IID.

[0014] Recent works have shown that most decentralized learning algorithms suffer from model quality loss (or even divergence) when running on non-IID data partitions. Although several solutions have been proposed to deal with highly skewed non-IID data (such as data sharing and model migration), they are all unsatisfactory. For example, some existing works have proposed heuristic-based methods by sharing local device data or creating some server-side proxy data. However, these methods may be unrealistic: in addition to burdening network bandwidth, sending local data to the server violates the key privacy assumption of federated learning, and sending globally shared proxy data to all devices requires careful generation or collection of such auxiliary data. Summary of the invention

[0015] In view of the shortcomings of the existing technology and to meet the needs of non-independent and identically distributed federated learning experiments, the present invention adjusts the correlation probability between data in the data set and uses a quantity regulator to control the number and balance of samples allocated to the client, thereby constructing a non-independent and identically distributed data set with multiple rules and mutual correlation between data suitable for federated learning.

[0016] To achieve the above objectives, the present invention is implemented through the following technical solutions.

[0017] A data set construction system for non-independent and identically distributed federated learning, including an initial module, a selection module and a state sequence group extraction module; wherein,

[0018] The initial module is used to receive the data set and generate the initial state sequence by sampling according to the initial probability distribution matrix and the initial probability transfer matrix;

[0019] The selection module is used to receive the initial state sequence group and generate a dataset for non-IID federated learning:

[0020] The state sequence group extraction module is used to receive the state sequence and the group number and generate the state sequence group.

[0021] Furthermore, in the initial module, the initial probability distribution matrix π=(π i ), where π i = p(i j =q i ),j=1,2,…,T,T represents the number of samples contained in the sampled state sequence; p represents i j =q i The probability of 0 <p<1;

[0022] The initial state sequence is extracted according to the initial probability distribution matrix and the initial state transition probability matrix.

[0023] Furthermore, in the initial module, the initial state sequence, initial state transition probability matrix and group number are transmitted to the state sequence group extraction module, and then the state sequence group returned by the state sequence group extraction module is received as the initial state sequence group, and the initial state sequence group is transmitted to the selection module.

[0024] Furthermore, in the initial module, the initial state transition probability matrix is

[0025] A0=(p ij ) T×n ,i=1,…,T,j=1,…,n,p ij =P(i t+1 =q j |i t =q i ), i=1,2,…,n; j=1,2,…,n,p ij represents the transition probability, n is the number of labels in the data set, ∑ j＝1 p ij =1,…,∑ j＝n p ij =1, i=1,…,T.

[0026] Furthermore, in the state sequence group extraction module, according to the received state sequence, transition probability matrix and group number, a state sequence group {I1, I2, ..., I S}, the specific steps are as follows: For the state sequence I1={i1,i2,…,i T} for each state value i j , obtain the transition probability distribution {p 1j ,…,p Tj}, take the label corresponding to the maximum probability, and then store the label corresponding to the maximum value in the corresponding position of the next state sequence, so as to obtain the next state sequence I2 with a fixed length. Repeat this method until I is obtained. S , thus generating a state sequence group {I1, I2,…, I S}, and then return the state sequence group.

[0027] Furthermore, in the selection module, the steps of generating a data set for non-IID federated learning are as follows:

[0028] (1) Setting the threshold N for group replacement to adjust the uniformity of the data;

[0029] (2) Set the transition probability matrix A, value p ij =P(i t+1 =q j |i t =qi ), where \(i = 1, 2, \ldots, n\); \(j = 1, 2, \ldots, n, p\) ij represents the transition probability, and \(\sum\) j＝1 p ij = 1, \(\ldots\), \(\sum\) j＝n p ij = 1, \(i = 1, \ldots, T\)

[0030]

[0031] where \(m=\lceil W / N\rceil\), \(W\) is the number of samples in the dataset;

[0032] (3) If the number of state sequence groups \(Z\) is in the range \([(k - 1)N, kN)\), \(1\lt k\leq m\), then the last sequence of the state sequence group, the transition probability \(A\) k-1 and the number \(kN - Z\) are sent to the state sequence group extraction module to generate a state sequence group \(\{I\) Z+1 , \(I\) Z+2 , \(\ldots\), \(I\) kN \}, and together with the state sequence group to form a new state sequence group \(\{I_1, I_2, \ldots, I\) kN \}; if \(k\gt m\), then stop execution;

[0033] (4) Repeat step (3) until the entire dataset is traversed;

[0034] The finally formed state sequence group is a non - independent and identically distributed dataset.

[0035] Furthermore, it also includes an allocation module, connected to the selection module, for allocating data samples according to the nodes of federated learning; first, set the quantity regulator \(C=(C_1, C_2, C_3, \ldots, C\) K ), where \(K\) represents the number of nodes; the final state sequence group is evenly distributed to \(K\) nodes in turn, and each node is allocated samples; when \(K\) nodes are allocated different proportions of data sample amounts, \(C=(C_1, C_2, C_3, \ldots, C\) ), K and the number of samples allocated to each node is \(C_1W\), \(C_2W\), \(C_3W\), \(\ldots\), \(C\) W, the number of samples allocated to node 1 is: K The number of samples allocated to node 2 is: The number of samples allocated to node 3 is: The number of samples allocated to node \(K\) is:

[0036] Furthermore, the method includes:

[0037] S1, passing the data set into the initialization module, sampling the data set according to the initial distribution probability matrix, and generating the initial state sequence group according to the initial transition probability matrix;

[0038] S2. Pass the initial state sequence group and the transition probability matrix into the selection module to generate a data set for non-independent and identically distributed (NIID) datasets.

[0039] Furthermore, the initial state transition probability matrix A0=(p ij ) T×n ,i=1,…,T,j=1,…,n,p ij =P(i t+1 =q j |i t =q i ), i=1, 2,…, n; j=1, 2,…, n, p ij represents the transition probability, n is the number of labels in the data set, ∑ j＝1 p ij =1,…,∑ j＝n p ij =1,i=1,…,T.

[0040] In the present invention, a real data set is divided into multiple smaller subsets to form a distributed non-independent and identically distributed data set, which makes it easy to set different numbers of participants and facilitates quantification of the balance of local data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a schematic diagram of the structure of a data set construction system according to an embodiment of the present invention;

[0043] Figure 2 It is a flowchart of a selection module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The present invention proposes a dataset construction system for non - independent and identically - distributed federated learning, as Figure 1 shown, which includes an initial module, a selection module, and a state sequence group extraction module.

[0046] Initial module: Receive a dataset, the total number of samples in which is W, and convert the dataset into a matrix. Each row of the matrix represents a sample data, and each column represents the features of the sample data. The sample data features X = {x0, x1, …, x m} and the label data Q = {q0, q1, q2, …, q n} form the complete training data (X, Q), as shown in Table 1.

[0047] Table 1 Training data matrix structure

[0048]

[0049] Set the initial state distribution probability matrix π=(π i ), where π i = p(i j = q i ), j = 1, 2, …, T, and T can be set manually, and its size is the number of samples included in the state sequence. p is the probability of i j = q i (0 < p < 1). According to the initial state distribution probability π i =(p1, …, p n ), sample the initial state i j , i j ∈ {q1, q2, …, q n}. Through the serial number and i j , the corresponding sample data can be found from the training data, and will not be described later, that is, i j can also represent the corresponding sample data.

[0050] Set the initial state transition probability matrix A0=(p ij ) T×n , i = 1, …, T, j = 1, …, n, p ij = p(i t+1 = q j |i t = q i ), i = 1, 2, …, n; j = 1, 2, …, n, p ij represents the transition probability. The transition probability matrix controls the balance of the data sample distribution and the correlation between data, and can be centralized or dispersed, and can be set by itself, but needs to satisfy: ∑ j＝1 p ij=1,…,∑ j＝n p ij =1,i=1,…,T, according to the initial state transition probability matrix A0, we get the initial state sequence I={i1,i2,…,i T},i j ∈{q1,q2,…,q n},j=0,1,2,…,T.

[0051]

[0052] In the initial module, the initial distribution state sequence, the initial state transition probability matrix and the number of groups are transmitted to the state sequence group extraction module, and then the state sequence group generated by it is received as the initial state sequence group, and the initial state sequence group is transmitted to the selection module.

[0053] The state sequence group extraction module extracts the state sequence group {I1, I2, …, I S The specific process is as follows: For the state sequence I1={i1,i2,…,i T} for every value i j , obtain its transition probability distribution {p 1j ,…,p Tj}, take the label corresponding to the maximum probability value, and then store the label corresponding to the maximum value in the corresponding position of the next state sequence, so as to obtain the next state sequence I2 with a fixed length. Repeat this method until I is obtained. S , thus generating a state sequence group {I1,I2,…,I S}, and then return the state sequence group.

[0054] Selection module: receives the state sequence group sent by the initial module and Figure 2 As shown, the data set is divided into the final state sequence groups as follows:

[0055] (1) Set the threshold N for group replacement. N is set manually to adjust the uniformity of the data.

[0056] (2) Set the transition probability matrix A in the same way as A0, with a value of p ij = p(i t+1 =q j |i t =q i ),i=1,2,…,n; j=1,2,…,n,p ij Represents the transition probability, which can be set by yourself, but needs to satisfy:∑ j＝1 p ij =1,…,∑j＝ n p ij = 1, i = 1, …, T.

[0057]

[0058] Where m = ceiling(W / N).

[0059] (3) If the number of state sequence groups Z is between [(k - 1)N, kN), 1 < k ≤ m, then the last sequence of the state sequence group, the transition probability A k-1 and the number kN - Z are sent to the state sequence group extraction module to generate a state sequence group {I Z+1 , I Z+2 , …, I kN}, and together with the state sequence group to form a new state sequence group {I1, I2, …, I kN}; if k > m, stop execution.

[0060] (4) Repeat step (3) until the entire data set is traversed.

[0061] By the above method, the conditional probability correlation between the labeled sample data is adjusted, that is, the transition probability matrix A = (p ij ), T×n , i = 1, …, T, j = 1, …, n. The idea of the whole process is that after every N state sequence groups are generated, the transition probability matrix is transformed to continue generating N state sequence groups, so as to obtain the final state sequence group {I1, I2, …, I mN}, and this state sequence group is the non-independent and identically distributed data set.

[0062] The role of the transition probability matrix is that after the previous state of the state sequence is determined, the probability of the next state is determined accordingly, that is, the previous state will affect the next state, so as to continuously adjust the correlation between the data;

[0063] p ij = p(i t+1 = q j |i t = q i ), i = 1, 2, …, n; j = 1, 2, …, n

[0064] In the selection module, the final state sequence group {I1, I2, …, I mN} can also be re-sorted. As shown in Table 2, the sorted data set still satisfies non-independent and identically distributed, and can allocate data to the nodes of federated learning more clearly.

[0065] Table 2 The mixed data set

[0066]

[0067] In one embodiment, the system further includes an allocation module, which is connected to the selection module and allocates data samples according to the number of nodes in the federated learning. First, set the number regulator C = (C1, C2, C3, ..., C K ), K represents the number of nodes; the final state sequence group is proportional Distribute them evenly to K nodes in turn, and each node is assigned samples, the data set assigned to node 1 is: The data set allocated to node 2 is: The data set assigned to node K is: When K nodes are assigned different proportions of data samples, C=(C1,C2,C3,…,C K ), The number of samples allocated to each node is C1W, C2W, C3W, …, C K W, the data set allocated to node 1: Dataset allocated to node 2: Dataset allocated to node K: The quantity regulator C is set manually.

[0068] According to another aspect of the present invention, a method for constructing a dataset for non-IID federated learning is proposed. The method uses the above-mentioned dataset construction system, including:

[0069] S1, passing the data set into the initialization module, sampling the data set according to the initial distribution probability matrix, and generating the initial state sequence group according to the initial probability transfer matrix;

[0070] S2. Pass the initial state sequence group and the initial transfer matrix into the selection module to generate a data set for non-independent and identically distributed (NIID) datasets.

[0071] Initial state transition probability matrix A0=(p ij ) T×n ,i=1,…,T,j=1,…,n,p ij = p(i t+1 =q j |i t =q i ),i=1,2,…,n; j=1,2,…,n,p ij represents the transition probability, n is the number of data set labels, ∑ j＝1 p ij =1,…,∑ j＝n p ij =1,i=1,…,T.

[0072] Example 1

[0073] Take the MNIST dataset as an example and construct it as a non-independent and identically distributed dataset. The MNIST dataset is composed of 10 numbers handwritten by 250 different people, namely 0, 1, 2, 3, ..., 9, 50% of which are written by high school students and 50% by staff of the Census Bureau. The labels are marked. Each image in the MNIST dataset consists of 28x28 pixels, and each pixel is represented by a grayscale value. The 28x28 pixels are expanded into a one-dimensional row vector. These row vectors are the rows in the image array. The class labels of the handwritten numbers (integers 0-9), the training dataset contains 60,000 samples, and the test dataset contains 10,000 samples. Figure 2 , the specific method comprises the following steps:

[0074] (1) The original dataset MNIST is passed into the initialization module. Let Q be the set of all possible initialization states, and the label set Q = {q1, q2, …, q N},q i ={0,1,2,3,4,5,6,7,8,9},i=1,…,N, the number of samples corresponding to each label is different, the initial distribution probability matrix π=(π i ), where π i =P(i j =q i ),i j =0,1,…,9, let the initial distribution probability matrix be π=(0.2,0.4,0.1,0.04,0.05,0.05,0.06,0.01,0,0.09), generate the initial state i1=1, according to the initial transition probability matrix A0=(p ij ) T×n ,T=20,n=10, extract i2=1, and the state sequence is recorded as I={i1,i2,…,i T},i j =0,1,2,3,…,9, generating the initial state sequence I1={1,1,1,4,2,1,0,5,6,1,2,1,3,4,5,6,1,2,3,4}, T=20;

[0075] (2) The number of state sequence groups generated is S, denoted by I1,…,I S , passed to the selection module. In the selection module, the initial transfer matrix A0 is set. The function of the transfer matrix is ​​to adjust the correlation between samples so that the distribution between samples satisfies non-independence and the sum of each column of the transfer matrix is ​​1. According to the Markov chain transfer process, the probability transfer matrix A=(p ij ) 20×10. If the number of state sequence groups obtained does not meet the conditions of the replacement threshold, the initial transition probability matrix A0 is still used to sample the initial state sequence. If the number of state sequence groups S reaches the set replacement (group number) threshold N, the setting of the threshold N is set according to the number of state sequences. The larger the threshold N, the more state sequence groups S are included. Similarly, the smaller the threshold N, the fewer state sequence groups S are included. That is to say, when S>N, the replacement transition probability matrix A=(p ij ) T×n , the transfer probability matrix A satisfies the matrix whose column probability sum is 1, T=20, n=10.

[0076]

[0077] In the selection module, according to the set number of state sequence groups (here N=30), when S<30, the initial transition probability matrix A0 is continued to be used, and the initial state sequence group {I1,…,I S}, when S ≥ 30 groups, replace the transition probability matrix with A, and generate the number of state sequence groups {I N+1 ,…,I 2N ,I 2N+1 ,…,I 3N ,…,I (m-1)N ,…,I mN}

[0078] Such as: I 35 ={4,1,2,3,4,5,6,8,9,0,2,1,2,3,5,6,7,3,5,1}, the transition probability matrix is ​​replaced every N groups, sampling once will affect the sampling probability of the next step, dynamically adjust the conditional probability of the sample data, traverse the entire data set according to this strategy, and obtain the state sequence group {I1,…,I mN}.

[0079] The state sequence group I1,…,I mN Reorder the sequence number.

[0080] (3) In the allocation module, set the quantity adjustment parameter C = (C1, C2, C3, ..., C K ), K represents the number of nodes, and the quantity adjustment parameter C is used to adjust the number of samples allocated to each node. The sample set generated by the hybrid module is divided into The proportions are evenly distributed to each node in turn. For example, when the node K=10, each node gets 6000 samples, and the data set allocated to node 1 is: The data set allocated to node 2 is: The data set allocated to node 10 is: When the node K=5, each node is allocated a different proportion of data samples, and the quantity regulator C=(C1,C2,C3,C4,C5)=(0.1,0.1,0.4,0.2,0.2). Each node is allocated a different number of samples, which are 6000,6000,24000,12000,12000 respectively. The sample set after the mixing module is allocated to 5 nodes according to the regulator. The allocation results are: The data set allocated to node 1 is: The data set allocated to node 2 is: The data set allocated to node 5 is: The classification performance is tested using the federated learning framework using the federated average algorithm.

[0081] Example 2

[0082] Taking the credit card default dataset of the banking industry as an example, a non-independent and identically distributed dataset is constructed. The credit card default dataset has a total of 30,000 samples, 23 features and 1 label, including the credit card limit of consumers and families x1, cardholder gender x2, education level x3, marital status x4, age x5, historical payment records x6~x7. 11 (April-September 2005), bill amount (April-September 2005) x 12 ~x 17 , last payment amount x 18 ~x 23 , the label is Y, the goal of processing data is to estimate the probability of default, rather than simply predicting whether the user is trustworthy or not. The specific steps are as follows:

[0083] (1) The original data set is passed into the initial module. In the initial module, let X be the set of all possible variables, the label set Y = {0,1}, the number of samples under each label is different, the minimum credit card limit x1 in the data set is 10,000, and the maximum is 1,000,000. Sort by credit card limit, [10,000-100,000], [100,000-200,000], …, [900,000-1,000,000], a total of 10 intervals. Assume that the initial distribution probability matrix is ​​π = (0.3, 0.1, 0.1, 0.04, 0.05, 0.05, 0.06, 0.1, 0.1, 0.1), generate the initial state i1 = 20,000, according to the initial distribution probability matrix π and the initial state transition probability matrix A0, the initial state sequence is the sampling sequence denoted as I = {i1, i2, …, i T},i j=0,1,2,3,…,9, for example, I1={20000,50000,140000,20000,70000,630000,320000,50000,130000,20000,380000,60000,240000,30000,250000,80000,10000,20000,150000,10000}, T=20.

[0084] (2) The initial distribution state sequence, the initial state transition probability matrix and the number of groups are transmitted to the state sequence group extraction module, and then the state sequence group generated by it is received as the initial state sequence group, and the initial state sequence S group is recorded as I1,…,I S , transmitted to the selection module. If the number of state sequence groups S does not reach the threshold N, (here set to N = 30), the initial transfer probability matrix A0 is still used. If it reaches the set group number threshold N, that is, when S>N, the transfer probability matrix A is replaced. The transfer probability matrix A satisfies the column probability sum of the matrix to be 1, T = 20, n = 10.

[0085] (3) According to the selection strategy of step (2), a state sequence group is obtained, and a state sequence group I is obtained by sampling, for example, I 45 =(20000,120000,240000,30000,50000,70000,320000,70000,40000,150000,20000,310000,210000,40000,240000,130000,

[0086] 30000, 450000, 670000, 20000), the transfer probability matrix is ​​changed every 30 groups, and the transfer probability matrix is ​​changed once for 600 samples, for a total of 50 rounds, the transfer probability matrix is ​​adjusted to dynamically adjust the data distribution;

[0087]

[0088] It is also possible to calculate all state sequence groups I1, ..., I 1500 Merge and reorder the numbers;

[0089] (4) In the allocation module, the number of clients is K, and the number adjuster C is set to (C1, C2, C3, ..., C K ), Group all state sequences into The proportion of is evenly distributed to each client, and the credit card default data set after the hybrid module is distributed to each node. For example, when the number of nodes is K = 10, the data set allocated to node 1 is: The data set allocated to node 2 is: The data set allocated to node 10 is: Or when the number of nodes is 5, set the quantity adjuster C = (C1, C2, C3, C4, C5) = (0.1, 0.3, 0.2, 0.3, 0.1) to allocate different proportions of state sequence groups to each node. The data set allocated to node 1 is: The data set allocated to node 2 is: The data set allocated to node 5 is: The federated average algorithm is used in the federated learning framework to predict the probability of default of users using credit cards.

[0090] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dataset construction system for non-IID federated learning, characterized in that: It includes an initial module, a selection module, a state sequence group extraction module and an allocation module; among which, The initial module is used to receive the data set and generate the initial state sequence by sampling according to the initial probability distribution matrix and the initial probability transfer matrix; The selection module is used to receive the initial state sequence group and generate a dataset for non-IID federated learning: A state sequence group extraction module, used for receiving a state sequence and a group number, and generating a state sequence group; An allocation module, connected to the selection module, for allocating data samples according to the nodes of the federated learning; In the selection module, the steps for generating a dataset for non-IID federated learning are as follows: (1) Setting the threshold N for group replacement to adjust the uniformity of the data; (2) Set the transition probability matrix A, value p ij =P(i t+1 =q j |i t =q i ),i=1,2,…,n; j=1,2,…,n,p ij represents the transition probability, and ∑ j＝1 p ij =1,…,∑ j＝n p ij =1,i=1,…,T, Wherein, m=W / N rounded up, W is the number of samples in the data set; (3) If the number of state sequence groups Z is between [(k - 1)N, kN), where 1 < k ≤ m, then the last sequence of the state sequence group, the transition probability A k-1 and the number kN - Z are sent to the state sequence group extraction module to generate the state sequence group {I Z+1 , I Z+2 , …, I kN}, and together with the state sequence group, form a new state sequence group {I1, I2, …, I kN}; if k > m, then stop execution; (4) Repeat step (3) until the entire data set is traversed; The state sequence group finally formed is a non-independent and identically distributed data set; In the allocation module, first set the quantity regulator C corresponding to the node = (C1, C2, C3, ..., C K ), K represents the number of nodes; the final state sequence group is Distribute them evenly to K nodes in turn, and each node is assigned samples; when K nodes are assigned different proportions of data samples, The number of samples allocated to each node is C1W, C2W, C3W, …, C K W, the number of samples allocated to node 1 is: The number of samples allocated to node 2 is: The number of samples allocated to node K is:

2. The data set construction system according to claim 1, characterized in that: In the initial module, the initial probability distribution matrix π=(π i ), where π i = p(i j =q i ),j=1,2,…,T,T represents the number of samples contained in the sampled state sequence; p represents i j =q i The probability of 0 <p<1; The initial state sequence is extracted according to the initial probability distribution matrix and the initial state transition probability matrix.

3. The data set construction system according to claim 2, characterized in that: In the initial module, the initial state sequence, initial state transition probability matrix and group number are transmitted to the state sequence group extraction module, and then the state sequence group returned by the state sequence group extraction module is received as the initial state sequence group, and the initial state sequence group is transmitted to the selection module.

4. The data set construction system according to claim 1, characterized in that: In the initial module, the initial state transition probability matrix is ​​A0=(p ij ) T×n ,i=1,…,T,j=1,…,n,p ij =P(i t+1 =q j |i t =q i ),i=1,2,…,n; j=1,2,…,n,p ij represents the transition probability, n is the number of labels in the data set, ∑ j＝1 p ij =1,…,∑ j＝n p ij =1,i=1,…,T.

5. The data set construction system according to claim 1, characterized in that: In the state sequence group extraction module, according to the received state sequence, transition probability matrix and group number, a state sequence group {I1, I2, ..., I S }, the specific steps are as follows: For the state sequence I1={i1,i2,…,i T } for each state value i j , obtain the transition probability distribution {p 1j ,…,p Tj }, take the label corresponding to the maximum probability, and then store the label corresponding to the maximum value in the corresponding position of the next state sequence, so as to obtain the next state sequence I2 with a fixed length. Repeat this method until I is obtained. S , thus generating a state sequence group {I1,I2,…,I S }, and then return the state sequence group.

6. The method for constructing a data set by the data set construction system according to any one of claims 1 to 5, characterized in that: The method comprises: S1, passing the data set into the initialization module, sampling the data set according to the initial distribution probability matrix, and generating the initial state sequence group according to the initial transition probability matrix; S2. Pass the initial state sequence group and the transition probability matrix into the selection module to generate a data set for non-independent and identically distributed (NIID) datasets.

7. The method for constructing a data set according to claim 6, characterized in that: The initial state transition probability matrix A0=(p ij ) T×n ,i=1,…,T,j=1,…,n,p ij =P(i t+1 =q j |i t =q i ),i=1,2,…,n; j=1,2,…,n,p ij represents the transition probability, n is the number of labels in the data set, ∑ j＝1 p ij =1,…,∑ j＝n p ij =1,i=1,…,T.

Images