Information classification system, method and application for low-latency federated learning

Abstract

The present application relates to the technical field of distributed machine learning and data prediction, and particularly relates to a low-delay longitudinal federated learning information classification system, method and application. In the low-delay longitudinal federated learning information classification system provided by the present application, the similarity value of the class label predicted by the independent model and the class label predicted by the longitudinal federated model of the to-be-tested sample is judged by the discriminator. If the similarity value is high, it means that the class label predicted by the independent model and the class label predicted by the longitudinal federated model of the to-be-tested sample are highly likely to coincide. At this time, the system directly uses the independent model to predict the to-be-tested sample, thereby avoiding the communication overhead required by the longitudinal federated model prediction, greatly reducing the communication overhead and delay, and improving the service efficiency and quality. Moreover, for the to-be-tested sample with low similarity, the longitudinal federated model is specified to perform prediction, thereby ensuring the prediction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed machine learning and data prediction, and particularly relates to a low-latency longitudinal federated learning information classification system, method and application. BACKGROUND

[0002] Longitudinal federated learning enables multiple participants with different data characteristics and the same sample space to collaboratively train models in a privacy-preserving manner. However, high computational and communication overhead hinders the application of longitudinal federated learning in many resource-constrained or delay-sensitive applications and systems, such as mobile computing and online advertising.

[0003] In a typical longitudinal federated system, there is usually one active party and one or more passive parties, where the active party has labels and part of the features, and the passive party has the remaining features. Since the samples and models are divided and distributed to multiple participants, they have to transmit a large amount of intermediate results (mainly feature representations) during federated inference. For some resource-limited systems, such a large communication overhead and delay is unbearable, and can cause a large delay for many delay-sensitive applications, seriously affecting the prediction service quality of the model. For example, in an online advertising system, an advertising company (active party) can cooperate with an Internet company (passive party) to train a more accurate advertising recommendation model using longitudinal federated learning. However, millions of queries per second will be generated during peak hours, and the overall delay of each query must be implemented in milliseconds to provide better service experience to users, including audience identification, federated model inference, and advertising display, etc. Tests in actual systems have found that about 90% of the prediction delay in longitudinal linear models and longitudinal neural network models comes from communication delay, and for longitudinal tree models, communication delay accounts for 99%. Therefore, reducing the communication overhead and inference delay of longitudinal federated learning models is a strong demand and a very challenging problem. SUMMARY

[0004] In order to overcome the defects that the communication overhead and inference delay of the longitudinal federated learning model in the prior art limit its application in many scenarios, the present application provides a low-latency longitudinal federated learning information classification system and method, which can reduce the communication overhead and delay of the prediction and inference stage of the longitudinal federated learning system and improve the service quality.

[0005] The low-latency longitudinal federated learning information classification system of the present application comprises: a plurality of participants and a server; the participants share an information sample space, and at least part of the participants store a class label set of information samples, the participants who have the class label set are active parties, and the class label sets of different active parties are the same or different; the active parties are built-in discriminators, independent models and longitudinal federated models;

[0006] The input of the independent model and the longitudinal federated model is information samples, and the output of the independent model and the longitudinal federated model is the category label of the information samples; each active party trains an independent model based on a local training sample set XB, and each active party trains a longitudinal federated model in cooperation with the server and other participating parties in a longitudinal federated learning manner based on the local training sample set XB; the local training sample set XB is used to store labeled samples, and the labeled samples include a category label belonging to a category label set of the active party and a data sample with the same dimension as the information sample; and the discriminator is used to judge the similarity of the category judgment of the independent model and the longitudinal federated model of the active party for the information sample.

[0007] The active party inputs the obtained to-be-tested sample into the discriminator to obtain a similarity value of the to-be-tested sample; when the similarity value is greater than or equal to a set similarity threshold, the active party predicts the category label of the to-be-tested sample through the independent model; and when the similarity value is less than the set similarity threshold, the active party predicts the category label of the to-be-tested sample through the longitudinal federated model.

[0008] The information classification method of low-latency longitudinal federated learning provided in the application is suitable for the information classification system of low-latency longitudinal federated learning and includes the following steps:

[0009] S1, an active party sharing an information sample space is obtained, and an active party having a category label set is taken as an active party; the active party is provided with a local training sample set XB used to store labeled samples, and the labeled samples include a category label belonging to a category label set of the active party and a data sample with the same dimension as the information sample; the active party performs longitudinal federated learning based on the local training sample set XB to obtain a longitudinal federated model; and the active party obtains an independent model through machine learning based on the local training sample set XB;

[0010] S2, a discriminator is constructed by a machine learning method, and the discriminator is used to judge the similarity of the category judgment of the independent model and the longitudinal federated model of the active party for the information sample; the input of the discriminator is an information sample, and the output is a predicted similarity value;

[0011] S3, a to-be-tested sample is obtained, and the to-be-tested sample is input into the discriminator; and the discriminator outputs a similarity value of the to-be-tested sample.

[0012] S4, whether the similarity value of the to-be-tested sample is greater than or equal to a set similarity threshold is judged; if yes, the category label of the to-be-tested sample is labeled through the independent model; and if no, the category label of the to-be-tested sample is labeled through longitudinal federated learning.

[0013] Preferably, the construction of the discriminator in S2 includes the following steps:

[0014] S21, information samples are selected from an information sample space to construct a learning data set;

[0015] S22, select part of the information samples in the learning dataset as training samples, predict the similarity value of the training samples by the discriminator as a predicted value, predict the class label of the training samples by the independent model and the longitudinal federated model respectively, and construct the similarity true value of the training samples; the similarity true value is a binary number, and the similarity true value is 1 when the class label predicted by the independent model is the same as the class label predicted by the federated learning; otherwise, it is 0;

[0016] S23, calculate the loss function combining the similarity true value and the predicted value of the training samples, and update the discriminator according to the loss function;

[0017] S24, judge whether the discriminator meets the set convergence condition; if yes, fix the parameters of the discriminator; if not, return to step S22.

[0018] Preferably, the setting method of the similarity threshold is that after the discriminator is fixed, the discriminator traverses the learning dataset to obtain the maximum similarity value as the similarity threshold;

[0019] Alternatively, the setting method of the similarity threshold is that each information sample in the learning dataset is associated with a class label, and after the discriminator is fixed, the discriminator predicts the similarity value corresponding to each information sample in the learning dataset; set multiple thresholds to form a threshold set, calculate the gain G of each threshold as the similarity threshold, and obtain the threshold with the maximum gain G as the similarity threshold;

[0020] The calculation method of the gain G corresponding to the threshold is:

[0021] SA1, let the threshold be the similarity threshold, predict the class label of the information sample in the learning dataset with a similarity greater than or equal to the similarity threshold by the independent model, and calculate the prediction accuracy of the independent model; predict the class label of the information sample in the learning dataset with a similarity less than the similarity threshold by the longitudinal federated model, and calculate the prediction accuracy of the longitudinal federated model;

[0022] SA2, calculate the value of the prediction accuracy gain P by subtracting the prediction accuracy of the independent model from the prediction accuracy of the longitudinal federated model, and calculate the communication frequency T between the active party and the server in the prediction process of the longitudinal federated model, and calculate the gain G combining the following model:

[0023] G = p1 x P - p2 x T

[0024] Wherein, p1 is the accuracy coefficient of the active party, p2 is the communication overhead coefficient of the active party, and p1 and p2 are both set coefficients of the active party.

[0025] The low-delay longitudinal federated learning information classification method provided by the application is suitable for image recognition, population census, financial risk control, film and television evaluation, advertisement recommendation and the like.

[0026] The image recognition device adopting the low-delay longitudinal federated learning information classification method provided by the application is the active party of the longitudinal federated learning, and is used for executing the low-delay longitudinal federated learning information classification method to obtain the category of the image to be recognized.

[0027] When the image recognition device is used for recognizing handwritings, the information sample is a handwritten symbol, and the set category label is a standard symbol; the image recognition device executes the low-delay longitudinal federated learning information classification method to obtain the standard symbol corresponding to the handwritten symbol.

[0028] When the image recognition device is used for recognizing articles, the information sample is an article image, and the set category label is an article category; the image recognition device executes the low-delay longitudinal federated learning information classification method to obtain the article category corresponding to the article image.

[0029] The population census device adopting the low-delay longitudinal federated learning information classification method provided by the application is the active party of the longitudinal federated learning, and is used for executing the low-delay longitudinal federated learning information classification method to obtain the category of the article under different classification indexes.

[0030] The film and television emotion classification device adopting the low-delay longitudinal federated learning information classification method provided by the application is the active party of the longitudinal federated learning, and is used for executing the low-delay longitudinal federated learning information classification method to obtain the emotion category of the film and television information to be tested.

[0031] The application provides a working condition diagnosis device adopting the low-delay longitudinal federal learning information classification method, information samples are collected data of a diagnosis object, and the information samples include at least one of an image, a vibration signal and an electrical signal; the fault diagnosis device is an active party of longitudinal federal learning, and category labels set by the fault diagnosis device include a plurality of working condition categories; the fault diagnosis device executes the low-delay longitudinal federal learning information classification method to determine whether the diagnosis object is faulty at a moment when the information sample is collected; when the diagnosis object is a motor, the working condition category is used for indicating at least one of speed, load torque and load force of the diagnosis object; and when the diagnosis object is a mechanical transmission component, the working condition category includes normal and at least one fault category.

[0032] The application provides a credit risk control device adopting the low-delay longitudinal federal learning information classification method, information samples are credit card score data of a resident, and the information samples include at least one of age, bank card overdue times and income; the credit risk control device is an active party of longitudinal federal learning, and category labels set by the credit risk control device are used for indicating financial conditions of the resident in a future specified time; the credit risk control device executes the low-delay longitudinal federal learning information classification method to obtain the category labels corresponding to the credit card score data of the resident, so that whether the resident is in financial difficulties in the future specified time is determined.

[0033] The application provides an advertisement recommendation platform adopting the low-delay longitudinal federal learning information classification method, and information samples include an advertisement and a customer; the advertisement recommendation platform is an active party of longitudinal federal learning, and category labels set by the advertisement recommendation platform include compliance and non-compliance; the advertisement recommendation platform executes the low-delay longitudinal federal learning information classification method to obtain information samples predicted as compliant as target samples, and pushes the advertisement in the target samples to the corresponding customer.

[0034] The application has the following advantages:

[0035] (1) The low-delay longitudinal federal learning information classification system provided by the application, the similarity value of the category label predicted by the independent model and the category label predicted by the longitudinal federal model is judged by the discriminator, if the similarity value is high, it means that the category label predicted by the independent model and the category label predicted by the longitudinal federal model for the to-be-tested sample are highly likely to coincide, at this time the system directly uses the independent model to predict the to-be-tested sample, thereby avoiding the communication overhead required by the longitudinal federal model prediction, greatly reducing the communication overhead and delay, and improving the service efficiency and quality. And for the to-be-tested sample with low similarity, the longitudinal federal model is specified for prediction, thereby ensuring the prediction accuracy. It can be seen that, in the system, the data shunting method is used to reduce the calculation amount of the longitudinal federal model, thereby reducing the communication overhead and delay; the similarity value is used as the data shunting threshold to ensure the prediction accuracy of the independent model for the to-be-tested sample, thereby ensuring the overall prediction accuracy and service quality while reducing the calculation amount of the longitudinal federal model.

[0036] (2) The low-delay longitudinal federal learning information classification method provided by the application, the discriminator is trained in combination with the pre-trained independent model and the longitudinal federal model, and then the prediction model of the to-be-tested sample is specified in combination with the similarity value predicted by the discriminator. The application is equivalent to realizing a tunable bus federal learning framework (TVFL), which can greatly reduce the communication overhead and delay while reducing the prediction accuracy as little as possible, so as to improve the quality of the prediction service in the longitudinal federal learning.

[0037] (3) In the application, the similarity threshold is selected to balance the prediction accuracy and the communication overhead, so as to accurately determine the prediction model of the to-be-tested sample. In the application, a method for selecting the similarity threshold according to the maximum gain is also provided, first, the benefits of the active party in the longitudinal federal system are defined, that is, the trade-off between the prediction performance improvement and the communication frequency, then the threshold that maximizes the gain on the learning data set is selected as the similarity threshold, in this way, the accuracy coefficient ρ1 and the communication overhead coefficient ρ2 can be set to realize flexible adjustment of the similarity threshold in the corresponding scene, so that the low-delay longitudinal federal learning information classification method can flexibly adjust whether to focus on prediction accuracy or save communication overhead according to the scene needs, improve the scene adaptability, and facilitate the popularization and application of the method. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 It is a low-delay longitudinal federal learning information classification method flowchart;

[0039] Figure 2 It is a discriminator training flowchart;

[0040] Figure 3(a) is a comparison of the communication overhead of each model when the base model in examples 3 to 7 adopts a logistic regression model;

[0041] Figure 3(b) is a comparison of the communication delay of each model when the base model in Examples 3 to 7 adopts a logistic regression model;

[0042] Figure 4(a) is a comparison of the communication overhead of each model when the base model in Examples 1 to 7 adopts a neural network model;

[0043] Figure 4(b) is a comparison of the communication delay of each model when the base model in Examples 1 to 7 adopts a neural network model;

[0044] Figure 5(a) is a comparison of the communication overhead of each model when the base model in Examples 1 to 7 adopts a tree model;

[0045] Figure 5(b) is a comparison of the communication delay of each model when the base model in Examples 1 to 7 adopts a tree model;

[0046] Figure 6 Figure 6 is a trend of the communication overhead on the data set as the active party occupies the importance of the feature. C Figure 6 is a trend of the communication overhead on the data set as the active party occupies the importance of the feature. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0048] A low-latency longitudinal federated learning information classification method

[0049] Reference Figure 1 , Figure 2 The low-latency longitudinal federated learning information classification method proposed in the present embodiment includes the following steps S1-S4.

[0050] S1, obtain the participants of the shared information sample space, let the participants with a class label set as the active party, the active party is provided with a local training sample set XB for storing labeled samples, the labeled samples include class labels belonging to the class label set of the active party and data samples with the same dimension as the information sample; the active party performs longitudinal federated learning based on the local training sample set XB to obtain a longitudinal federated model; the active party obtains an independent model through machine learning based on the local training sample set XB.

[0051] Thus, the input of the independent model and the longitudinal federated learning is the information sample, the output is the category label of the information sample, and the category label belongs to the category label set of the active party.

[0052] In this step, the longitudinal federated learning model is a model pre-trained by the active party through the server to jointly implement the longitudinal federated learning algorithm with other participating parties; when it is necessary to predict the information sample through the longitudinal federated learning model, the active party communicates with the server to obtain the features of the information sample to be predicted provided by other participating parties. It is worth noting that the longitudinal federated learning is a prior art, and therefore the model training and application of the longitudinal federated learning are not described in detail.

[0053] In this step, the independent model can be a linear model or a neural network model, and the independent model fixes the model parameters by learning the local training sample set XB autonomously.

[0054] S2, the active party constructs a discriminator by steps S21-S24 using a machine learning method, the discriminator is used to judge the similarity of the category judgment of the independent model and the longitudinal federated model for the information sample; the input of the discriminator is the information sample, and the output is the predicted similarity value.

[0055] Specifically, the higher the similarity value, the greater the possibility that the discriminator considers that the category judgment results of the independent model and the longitudinal federated model for the information sample are consistent, and the similarity value is a probability value between 0 and 1.

[0056] S21, select information samples from the information sample space to construct a learning data set.

[0057] S22, select part of the information samples in the learning data set as training samples, predict the similarity value of the training samples by the discriminator as the predicted value; predict the category label of the training samples by the independent model and the longitudinal federated model respectively, and construct the similarity true value of the training samples; the similarity true value is a binary number, the similarity true value is 1 when the category label predicted by the independent model is the same as the category label predicted by the federated learning; otherwise, it is 0.

[0058] In this step, the similarity of the training sample is calculated as the similarity ground truth according to the category label predicted by the independent model and the category label predicted by the longitudinal federated model; specifically, the similarity ground truth is 1, indicating that the category label predicted by the independent model is consistent with the category label predicted by the longitudinal federated model; otherwise, the similarity ground truth is 0, indicating that the category label predicted by the independent model is different from the category label predicted by the longitudinal federated model. In specific implementation, the outputs of the independent model and the longitudinal federated model can be set as category probability distributions, i.e., probabilities of the information sample belonging to each category in the category label set of the active party, and the sum of the probabilities is 1, wherein the category corresponding to the maximum probability value in the category probability distribution output by the independent model is the category label predicted by the independent model for the information sample, and the category corresponding to the maximum probability value in the category probability distribution output by the longitudinal federated model is the category label predicted by the longitudinal federated model for the information sample.

[0059] The similarity value output by the discriminator is a probability value in the interval (0, 1).

[0060] S23, a loss function is calculated in combination with the similarity ground truth and the predicted value of the training sample, and the discriminator is updated according to the loss function. Specifically, in this step, the mean square error loss of the similarity ground truth and the predicted value of the training sample is taken as the loss function, and the parameters of the discriminator are updated in reverse according to the loss function.

[0061] S24, whether the discriminator meets the set convergence condition is judged; if yes, the parameters of the discriminator are fixed; if no, step S22 is returned.

[0062] Specifically, the convergence condition is that the change of the loss is less than the set change threshold; or the number of updates of the discriminator reaches the set threshold; or the loss of the discriminator reaches the set loss threshold.

[0063] S3, a to-be-tested sample is obtained, the to-be-tested sample is input into the discriminator, and the discriminator outputs a similarity value of the to-be-tested sample. The data structure of the to-be-tested sample is the same as that of the information sample, i.e., the dimensions of the to-be-tested sample and the information sample are the same.

[0064] S4, whether the similarity value of the to-be-tested sample is greater than or equal to a set similarity threshold is judged; if yes, the category label of the to-be-tested sample is labeled by the independent model; if no, the category label of the to-be-tested sample is labeled by the longitudinal federated learning.

[0065] In this embodiment, two setting methods of the similarity threshold are also given.

[0066] The first similarity threshold setting method is that after the discriminator is fixed, the discriminator traverses the learning data set, the maximum value in the similarity values of each information sample in the learning data set is obtained, and the maximum similarity value is taken as the similarity threshold.

[0067] The second similarity threshold setting method is to predict the similarity values corresponding to each information sample in the learning data set through the discriminator after the discriminator is fixed; a plurality of threshold values are set to form a threshold set, and the gain G of each threshold value is calculated when the threshold value is used as the similarity threshold value, and the threshold value with the maximum gain G is obtained as the similarity threshold value.

[0068] Specifically, in the second similarity threshold setting method, the gain G corresponding to the threshold value is calculated by the following steps SA1-SA2.

[0069] SA1, the information samples in the learning data set with a similarity greater than or equal to the similarity threshold value are predicted by the independent model to obtain the class labels, and the prediction accuracy of the independent model is calculated; the information samples in the learning data set with a similarity less than the similarity threshold value are predicted by the vertical federated model to obtain the class labels, and the prediction accuracy of the vertical federated model is calculated.

[0070] The prediction accuracy of the model = the number of samples predicted correctly by the model / the total number of samples predicted by the model.

[0071] The model prediction is correct, which means that the class label predicted by the model is consistent with the real class associated with the information sample. As can be seen, when the second similarity threshold setting method is used, the information samples in the learning data set should be associated with real classes, and the real classes belong to the class label set of the active party.

[0072] That is, when the second similarity threshold setting method is used, the information samples with class labels should be selected from the information sample space to construct the learning data set in S21. In specific implementation, a specified number of information samples can be selected from the information sample space in S21, and the real classes of the information samples are labeled by manual labeling or other existing labeling methods, so as to construct the learning data set; or a specified number of information samples with class labels are directly selected from the information samples to construct the learning data set.

[0073] SA2, the value obtained by subtracting the prediction accuracy of the independent model from the prediction accuracy of the vertical federated model is denoted as the prediction accuracy gain P, and the number of communications T between the active party and the server in the prediction process of the vertical federated model is calculated, and the gain G is calculated by the following model:

[0074] G = p1 x P - p2 x T

[0075] Wherein, p1 is the accuracy coefficient of the active party, p2 is the communication overhead coefficient of the active party, and p1 and p2 are both set coefficients of the active party.

[0076] A low-latency vertical federated learning information classification system

[0077] The embodiment also provides an information classification system of low-latency longitudinal federated learning, which is used for implementing the information classification method of low-latency longitudinal federated learning.

[0078] The category label sets of different active parties are the same or different.

[0079] The inputs of the independent model and the longitudinal federated model are information samples, and the outputs of the independent model and the longitudinal federated model are category labels of information samples.

[0080] The active party inputs the to-be-tested sample into the discriminator to obtain a similarity value of the to-be-tested sample.

[0081] It should be emphasized that the information classification method of low-latency longitudinal federated learning provided by the embodiment is suitable for various scenarios and is suitable for active parties in a federated learning scenario composed of two or more parties.

[0082] The following embodiments are further explanations of the present application, rather than limitations, and the application of the information classification method of low-latency longitudinal federated learning provided by the present application to other scenarios without creative labor also falls within the protection scope of the present application.

[0083] In the following embodiments, the following model symbols are defined for convenience of expression.

[0084] θ alone : independent model, i.e., a model trained based on the local training sample set XB of the active party;

[0085] VFL: traditional vertical federated learning model, i.e. the active party

[0086] TVFL(Basic): the information classification method of low-latency vertical federated learning provided by the present application, and the similarity threshold adopts the first similarity threshold setting method;

[0087] TVFL(MaxGain): the information classification method of low-latency vertical federated learning provided by the present application, and the similarity threshold adopts the second similarity threshold setting method.

[0088] Embodiment 1: Handwriting recognition device

[0089] The information sample is a handwritten symbol image; the vertical federated system in which the handwriting recognition device is located includes two participating parties, the handwriting recognition device is one of the participating parties, and is the active party of the vertical federated learning, and the category label set by the handwriting recognition device is a standard symbol; the other participating party is the passive party.

[0090] In this embodiment, in the vertical federated learning, the active party extracts the image features of the upper half of the handwritten symbol image, and the passive party extracts the image features of the other half of the handwritten symbol image, and the vertical federated model of the active party predicts the standard symbol corresponding to the handwritten symbol image based on the image features extracted by the active party and the image features extracted by the passive party. In this embodiment, handwritten digits are used for testing, i.e. the standard symbols are: 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. The handwriting recognition device executes the information classification method of low-latency vertical federated learning to obtain the standard symbol corresponding to the handwritten symbol image, thereby realizing the recognition of handwritten digits.

[0091] In this embodiment, independent models θ alone , a traditional vertical federated learning model VFL, a TVFL(Basic) classification method, and a TVFL(MaxGain) classification method are respectively constructed based on a neural network model.

[0092] In this embodiment, model training and testing are performed based on a data set MNIST dataset. The data set MNIST dataset is abbreviated as D M , which includes 60,000 training examples and 10,000 testing examples, and each example is a handwritten digit image labeled with a standard character.

[0093] In this embodiment, based on the 60,000 training examples, the θ alone model and the VFL model are respectively trained; then the discriminator is trained in combination with the 60,000 training examples and the θ alone model and the VFL model to realize the TVFL(Basic) model and the TVFL(MaxGain) model.

[0094] In this embodiment, the accuracy coefficient p1=1 and the communication overhead coefficient p2=1 in the process of training and applying the TVFL(MaxGain) model.

[0095] In this embodiment, the image features of the upper half of the image are extracted by the active party and the image features of the other half of the image are extracted by the passive party. alone The test samples are predicted by the model, the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model, the classification accuracies of the four models are compared, and the communication overhead and prediction delay of the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model are compared. In this embodiment, the communication overhead and delay of the three models are shown in FIG. 4(a) and FIG. 4(b), and the prediction accuracies of the four models are shown in Table 2.

[0096] Embodiment 2: Image recognition device

[0097] The Fashion MNIST dataset is a dataset containing 10 categories, each category containing 7000 28*28 pixel grayscale images, which can be used for image classification, image recognition and other tasks.

[0098] In this embodiment, the Fashion MNIST dataset is used for clothing classification, and the 10 categories defined include T-shirt, pants, pullover, dress, coat, sandal, shirt, sneaker, bag and ankle boot.

[0099] In this embodiment, the information sample is a clothing image; the vertical federal system in which the image recognition device is located includes two participants, the image recognition device is one of the participants and is the active party of the vertical federal learning, and the category label set by the image recognition device is the standard symbol; the other participant is the passive party.

[0100] In this embodiment, in the vertical federal learning, the active party extracts the image features of the upper half of the clothing image, the passive party extracts the image features of the other half of the clothing image, and the vertical federal model of the active party predicts the category corresponding to the clothing image based on the image features extracted by the active party and the image features extracted by the passive party. The image recognition device executes the low-delay vertical federal learning information classification method to obtain the clothing category corresponding to the clothing image, thereby realizing the recognition of the clothing image.

[0101] In this embodiment, the independent model θ alone , the traditional vertical federal learning model VFL, the TVFL(Basic) classification method and the TVFL(MaxGain) classification method are constructed based on the neural network model.

[0102] In this embodiment, the model is trained and tested based on the Fashion MNIST dataset. The Fashion MNIST dataset is abbreviated as D FM In this embodiment, 60000 clothing images are extracted from the Fashion MNIST dataset as the local learning dataset XB, and the remaining 10000 clothing images in the Fashion MNIST dataset are used as test samples.

[0103] In this embodiment, the local learning dataset XB is used to train the model θ alone The model, the VFL model; then combined with the local learning dataset XB and θ alone The discriminator is trained to realize the TVFL(Basic) model and the TVFL(MaxGain) model.

[0104] In this embodiment, during the training and application of the TVFL(MaxGain) model, the accuracy coefficient ρ1 = 1 and the communication overhead coefficient ρ2 = 1.

[0105] In this embodiment, the test samples are predicted by the model θ alone In this embodiment, the test samples are predicted by the model θ

[0106] In this embodiment, the communication overhead and delay of the three models are shown in Figures 4(a) and 4(b), and the prediction accuracy of the four models is shown in Table 2.

[0107] Embodiment 3: Population census device

[0108] The population census device provided in this embodiment judges the income of residents based on population census data, and specifically sets two class labels: personal annual input below 50,000 and personal annual input above 50,000.

[0109] The information sample is population census data; the vertical federal system in which the population census device is located includes two participants, the population census device is one of the participants, and is the active party of vertical federal learning, and the class label set by the population census device is the standard symbol; the other participant is the passive party.

[0110] In this embodiment, when performing longitudinal federated learning, the active party extracts 50% of the data features of the census data, and the passive party extracts the other half of the data features of the census data, that is, the dimension of the feature vector extracted by the active party is equal to the number of dimensions of the feature vector extracted by the passive party. The longitudinal federated model of the active party predicts the annual income level of the residents based on the data features extracted by the active party and the data features extracted by the passive party.

[0111] In this embodiment, the base model is first determined. The base model can be a neural network model, a logistic regression model, or a tree model, or other existing model structures.

[0112] In this embodiment, the independent model θ alone , the traditional longitudinal federated learning model VFL, the TVFL(Basic) classification method, and the TVFL(MaxGain) classification method are constructed based on the base model.

[0113] In this embodiment, the model training and testing are performed based on the census income dataset (Census income dataset, referred to as D C ). C In this embodiment, 20% of the census income dataset D C is extracted as a test sample, and the remaining data of D alone is used as a local learning dataset XB. In this embodiment, the θ alone model and the VFL model are trained based on the local learning dataset XB; then the discriminator is trained in combination with the local learning dataset XB and the θ alone model and the VFL model to implement the TVFL(Basic) model and the TVFL(MaxGain) model.

[0114] In this embodiment, during the training and application of the TVFL(MaxGain) model, the accuracy coefficient ρ1=1 and the communication overhead coefficient ρ2=1.

[0115] In this embodiment, the test sample is predicted by the θ alone model, the VFL model, the TVFL(Basic) model, and the TVFL(MaxGain) model, respectively. The classification accuracies of the four models are compared, and the communication overhead and prediction delay of the VFL model, the TVFL(Basic) model, and the TVFL(MaxGain) model are compared.

[0116] In this embodiment, when the base model adopts a longitudinal logistic regression model, the communication overhead and delay of the VFL model, the TVFL(Basic) model, and the TVFL(MaxGain) model are shown in FIGS. 3(a) and 3(b), respectively, and the θ aloneThe prediction accuracy of the model, the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model is shown in Table 1.

[0117] When the base model adopts a neural network model, the communication overhead and delay of the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model are shown in FIG. 4(a) and FIG. 4(b), respectively, θ alone The prediction accuracy of the model, the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model is shown in Table 2.

[0118] When the base model adopts a tree model, the communication overhead and delay of the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model are shown in FIG. 5(a) and FIG. 5(b), respectively, θ alone The prediction accuracy of the model, the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model is shown in Table 3.

[0119] Embodiment 4: Movie review sentiment classification device

[0120] The movie review sentiment classification device judges whether the sentiment attribute of a movie or television program is positive or negative based on movie or television evaluation information. In this embodiment, the information sample is movie or television evaluation information. The vertical federated system in which the movie review sentiment classification device is located includes two participants. The movie review sentiment classification device is one of the participants and is the active party of the vertical federated learning. The category labels set by the active party include positive and negative. The other participant is the passive party.

[0121] In this embodiment, during vertical federated learning, the active party extracts 50% of the data features of the movie or television evaluation information, and the passive party extracts the other half of the data features of the movie or television evaluation information.

[0122] In this embodiment, a base model is first determined, and then independent models θ alone , a traditional vertical federated learning model VFL, a TVFL(Basic) classification method and a TVFL(MaxGain) classification method are constructed based on the base model.

[0123] In this embodiment, model training and testing are performed based on a large movie review dataset (Large Movie Review dataset, referred to as D L ). L In this embodiment, 20% of D L is extracted as a test sample, and the remaining data in D aloneModel, VFL model; then combine the local learning dataset XB and θ alone Model, VFL model, and TVFL (Basic) model and TVFL (MaxGain) model.

[0124] In this embodiment, during the training and application of the TVFL (MaxGain) model, the accuracy coefficient ρ1 = 1 and the communication overhead coefficient ρ2 = 1.

[0125] In this embodiment, the θ alone Model, VFL model, TVFL (Basic) model, and TVFL (MaxGain) model are used to predict the test samples, and the classification accuracies of the four models are compared, and the communication overhead and prediction delay of the VFL model, TVFL (Basic) model, and TVFL (MaxGain) model are compared.

[0126] In this embodiment, when the base model uses a longitudinal logistic regression model, the communication overhead and delay of the VFL model, TVFL (Basic) model, and TVFL (MaxGain) model are shown in FIGS. 3(a) and 3(b), respectively, and θ alone The prediction accuracies of the model, VFL model, TVFL (Basic) model, and TVFL (MaxGain) model are shown in Table 1.

[0127] When the base model uses a neural network model, the communication overhead and delay of the VFL model, TVFL (Basic) model, and TVFL (MaxGain) model are shown in FIGS. 4(a) and 4(b), respectively, and θ alone The prediction accuracies of the model, VFL model, TVFL (Basic) model, and TVFL (MaxGain) model are shown in Table 2.

[0128] When the base model uses a tree model, the communication overhead and delay of the VFL model, TVFL (Basic) model, and TVFL (MaxGain) model are shown in FIGS. 5(a) and 5(b), respectively, and θ alone The prediction accuracies of the model, VFL model, TVFL (Basic) model, and TVFL (MaxGain) model are shown in Table 3.

[0129] Embodiment 5: Motor diagnosis device

[0130] The motor diagnosis device is used to determine the working state of the motor according to the current signal generated in the motor operation. In this embodiment, the information sample is the current signal of the motor; the longitudinal federal system in which the motor diagnosis device is located includes two participants, the Internet of Things data diagnosis device is one of the participants, and is the active party of the longitudinal federal learning. The active party sets a category label according to the motor operating state. Specifically, the category label can be used to identify the working condition, including speed, load torque and load force, etc.

[0131] In this embodiment, in the longitudinal federal learning, the active party extracts 50% of the data features of the information sample, and the passive party extracts the other half of the data features of the information sample. In this embodiment, a basic model is first determined, and then independent models θ alone , a traditional longitudinal federal learning model VFL, a TVFL(Basic) classification method and a TVFL(MaxGain) classification method are constructed based on the basic model.

[0132] In this embodiment, the sensorless drive diagnosis dataset (Sensorless drive diagnosis dataset, referred to as D S ) is used for model training and testing. In this embodiment, 20% of D S is extracted as a test sample, and the remaining data in D S is used as a local learning dataset XB. In this embodiment, θ alone model and VFL model are trained based on the local learning dataset XB; then the discriminator is trained in combination with the local learning dataset XB and the θ alone model and the VFL model to realize the TVFL(Basic) model and the TVFL(MaxGain) model.

[0133] In this embodiment, during the training and application of the TVFL(MaxGain) model, the accuracy coefficient ρ1=1 and the communication overhead coefficient ρ2=1.

[0134] In this embodiment, the test sample is predicted by the θ alone model, the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model respectively, the classification accuracy of the four models is compared, and the communication overhead and prediction delay of the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model are compared.

[0135] Embodiment 6: Credit risk control device

[0136] This credit risk control device is used to predict residents' financial risk within a specified future period based on credit card scoring data. In this embodiment, the information sample is residents' credit card scoring data, including age, number of bank card delinquencies, and income, etc. The vertical federated system in which the credit risk control device is located includes two participants. The IoT data diagnostic device is one of the participants and is the active party in the vertical federated learning. The active party sets category labels based on the information sample. Specifically, the category labels include two types: residents are likely to fall into financial distress within a specified future period, and residents are not likely to fall into financial distress within a specified future period.

[0137] In this embodiment, during vertical federated learning, the active party extracts 50% of the data features from the information samples, while the passive party extracts the other half. In this embodiment, a base model is first determined, and then independent models θ are constructed based on the base model. alone Traditional longitudinal federated learning models include VFL, TVFL (Basic) classification method, and TVFL (MaxGain) classification method.

[0138] This embodiment is based on the credit card scoring dataset (Give me some credit DGtaset, abbreviated as DGtaset). G ) to train and test the model, D G The category labels associated with credit card scoring data represent the likelihood of residents facing financial hardship within the next two years. In this embodiment, 20% of the DG (Debt Collection Group) is selected as a test sample. G The remaining data is used as a local learning dataset XB. In this embodiment, θ is trained based on the local learning dataset XB. alone Model, VFL model; then combined with the local learning datasets XB and θ alone The discriminator is trained using the model and the VFL model to implement the TVFL (Basic) model and the TVFL (MaxGain) model.

[0139] In this embodiment, during the training and application of the TVFL (MaxGain) model, the accuracy coefficient ρ1 = 1 and the communication overhead coefficient ρ2 = 1.

[0140] In this embodiment, respectively through θ alone The model, VFL model, TVFL (Basic) model and TVFL (MaxGain) model are used to predict test samples. The classification accuracy of the four models is compared. The communication overhead and prediction latency of VFL model, TVFL (Basic) model and TVFL (MaxGain) model are also compared.

[0141] Example 7: Advertising Recommendation Platform

[0142] This advertising recommendation platform is used to determine users' interests and then select ads to recommend to them, thereby improving the effectiveness of ad delivery.

[0143] In this embodiment, the information samples include advertisements and customers; the vertical federated system in which the advertisement recommendation platform is located includes two participants, the advertisement recommendation platform being one of the participants and the active party in the vertical federated learning. The category labels set by the active party include "matching" and "not matching". "Matching" means that the advertisement in the information sample matches the customer's interests, and "not matching" means that the advertisement in the information sample does not match the customer's interests; the other participant is the passive party.

[0144] In this embodiment, the features of the information samples are manually divided to separate the data features extracted by the active party and the data features extracted by the passive party during longitudinal federated learning.

[0145] In this embodiment, a basic model is first determined, and then independent models θ are constructed based on the basic model. alone Traditional longitudinal federated learning models include VFL, TVFL (Basic) classification method, and TVFL (MaxGain) classification method.

[0146] In this embodiment, a CTR benchmark dataset (Criteo, abbreviated as D) containing 600,000 information samples is constructed based on historical data. Cr Model training and testing are performed. In this embodiment, from D... Cr 20% of the sample was selected as the test sample, and D was used as the test sample. Cr The remaining data is used as a local learning dataset XB. In this embodiment, θ is trained based on the local learning dataset XB. alone Model, VFL model; then combined with the local learning datasets XB and θ alone The discriminator is trained using the model and the VFL model to implement the TVFL (Basic) model and the TVFL (MaxGain) model.

[0147] In this embodiment, during the training and application of the TVFL (MaxGain) model, the accuracy coefficient ρ1 = 1 and the communication overhead coefficient ρ2 = 1.

[0148] In this embodiment, respectively through θ alone The model, VFL model, TVFL (Basic) model and TVFL (MaxGain) model are used to predict test samples. The classification accuracy of the four models is compared. The communication overhead and prediction latency of VFL model, TVFL (Basic) model and TVFL (MaxGain) model are also compared.

[0149] The communication overhead and delay of the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model when the base model is a longitudinal logistic regression model are shown in FIG. 3(a) and FIG. 3(b), respectively, θ alone The prediction accuracy of the base model, the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model is shown in Table 1.

[0150] The communication overhead and delay of the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model when the base model is a neural network model are shown in FIG. 4(a) and FIG. 4(b), respectively, θ alone The prediction accuracy of the base model, the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model is shown in Table 2.

[0151] The communication overhead and delay of the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model when the base model is a tree model are shown in FIG. 5(a) and FIG. 5(b), respectively, θ alone The prediction accuracy of the base model, the VFL model, the TVFL(Basic) model and the TVFL(MaxGain) model is shown in Table 3.

[0152] Table 1: Accuracy of the four models when a longitudinal logistic regression model is used in Examples 3-7

[0153]

[0154] Table 2: Accuracy of the four models when a longitudinal logistic regression model is used in Examples 1-7

[0155]

[0156] Table 3: Accuracy of the four models when a longitudinal logistic regression model is used in Examples 3-7

[0157]

[0158] In combination with Examples 1-7, it can be seen that the two TVFL models (TVFL(Basic) and TVFL(MaxGain)) significantly save the communication overhead and reduce the prediction delay in the prediction stage.

[0159] As shown in Table 1, FIG. 3(a), FIG. 3(b), the prediction performance of TVFL(basic) and TVFL(MaxGain) is very close to the traditional longitudinal federal model VFL, much greater than the independent model θ alone . For example, in the D SOn the dataset D

[0160] Table 2, Figure 4(a) and 4(b) indicate that TVFL can significantly save communication overhead and reduce prediction delay on vertical federated learning model. As shown in Table 4, the prediction performance of TVFL(basic) model and TVFL(MaxGain) model is very close to VFL model, much greater than independent model θ alone . For example, on the dataset D FM , TVFL(MaxGain) saves the communication overhead from 48.8MB to 16.4MB and reduces the prediction delay from 201.6ms to 57.6ms, while the performance only decreases from 88.6%±0.1% to 86.8%±0.2%.

[0161] Table 3, Figure 5(a) and 5(b) indicate that on the vertical tree model, TVFL can significantly reduce the prediction communication overhead and delay, while the prediction accuracy decreases slightly. Specifically, compared with VFL, TVFL(MaxGain) only reduces the prediction accuracy by 2.2% on average, while reduces the communication overhead and prediction delay by 47.3% respectively.

[0162] Embodiment 8

[0163] In this embodiment, on the basis of Embodiment 3, the influence of the feature importance owned by the active party on the communication overhead is further verified.

[0164] Specifically, SHAP (a game theory method for measuring the influence of features on the output of machine learning models) is used to calculate the importance of the data features in D C , the features are randomly assigned to the active party according to the feature importance, and then the communication overhead saved by the TVFL(MaxGain) model relative to the independent model θ alone is calculated.

[0165] In this embodiment, three experiments are performed on D C , respectively denoted as seed=0, seed=1, seed=2, and the statistical results of the three experiments are shown in Table 6. Figure 6 ​

[0166] Figure 6 It can be seen that in the three experiments, the saved communication overhead is basically consistent with the rising trend of the feature importance occupied by the active party, proving the reliability of the experiment.

[0167] At the same time, combined with Figure 6 It can be seen that the higher the feature importance occupied by the active party, the more communication overhead saved by the TVFL(MaxGain) model, and thus the higher the feature importance occupied by the active party, the more communication overhead saved by the information classification method of low-delay vertical federated learning provided by the present application.

[0168] Of course, the present application is not limited to the details of the above exemplary embodiments, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims involved.

[0169] In addition, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

[0170] The technologies, shapes, and structural parts not described in detail in the present application are well-known technologies.

Claims

1. A low-latency federated learning information classification system, comprising: The application relates to a method for training a federated learning model. The application comprises the following steps: S1, obtaining a participant of a shared information sample space, and taking a participant with a category label set as an active participant; the active participant is provided with a local training sample set XB for storing labeled samples, wherein the labeled samples comprise category labels belonging to the category label set of the active participant and data samples with the same dimension as the information samples; the active participant performs federated learning based on the local training sample set XB to obtain a longitudinal federated model; and the active participant obtains an independent model through machine learning based on the local training sample set XB; S2, constructing a discriminator through machine learning by the active participant, wherein the discriminator is used to judge the similarity of the category judgment of the independent model and the longitudinal federated model of the active participant; the input of the discriminator is an information sample, and the output is a predicted similarity value; S3, obtaining a to-be-tested sample, and inputting the to-be-tested sample into the discriminator; the discriminator outputs a similarity value of the to-be-tested sample; 2. The information classification method of low-latency vertical federated learning, applied to the information classification system of low-latency vertical federated learning according to claim 1, characterized in that, S4, judging whether the similarity value of the to-be-tested sample is greater than or equal to a set similarity threshold value; if yes, the category label of the to-be-tested sample is labeled through the independent model; and if no, the category label of the to-be-tested sample is labeled through the longitudinal federated learning. The construction of the discriminator in S2 comprises the following steps: S21, selecting an information sample from an information sample space to construct a learning data set; S22, selecting part of the information samples in the learning data set as training samples, and predicting the similarity value of the training samples through the discriminator as a predicted value; the category label of the training samples is predicted through the independent model and the longitudinal federated model respectively, and the similarity true value of the training samples is constructed; the similarity true value is a binary number; when the category label predicted by the independent model is the same as the category label predicted by the federated learning, the similarity true value is 1; otherwise, the similarity true value is 0. ​ ​ 3.The method of claim 2, wherein, ​ ​ ​ S23, calculate a loss function according to the similarity true value and the predicted value of the training sample, and update the discriminator according to the loss function; S24, judge whether the discriminator meets the set convergence condition; if yes, fix the parameters of the discriminator; if no, return to step S22. 4.The method of claim 2, wherein The setting mode of the similarity threshold value is that after the discriminator is fixed, the discriminator traverses the learning data set to obtain the maximum similarity value as the similarity threshold value; Alternatively, the setting mode of the similarity threshold value is that each information sample in the learning data set is associated with a category label, and after the discriminator is fixed, the discriminator is used to predict the similarity values corresponding to the information samples in the learning data set; a plurality of threshold values are set to form a threshold value set, and the gain G of each threshold value is calculated when the threshold value is used as the similarity threshold value, and the threshold value with the maximum gain G is obtained as the similarity threshold value; The calculation mode of the gain G corresponding to the threshold value is: SA1, let the threshold value be used as the similarity threshold value, predict the category label of the information sample in the learning data set with a similarity greater than or equal to the similarity threshold value through an independent model, and calculate the prediction accuracy of the independent model; predict the category label of the information sample in the learning data set with a similarity less than the similarity threshold value through a vertical federated model, and calculate the prediction accuracy of the vertical federated model; SA2, calculate the value obtained by subtracting the prediction accuracy of the independent model from the prediction accuracy of the vertical federated model as the prediction accuracy gain P, and calculate the number T of communications between the active party and the server in the prediction process of the vertical federated model, and calculate the gain G according to the following model: G=ρ1×P-ρ2×T Wherein, ρ1 is the accuracy coefficient of the active party, ρ2 is the communication overhead coefficient of the active party, and ρ1 and ρ2 are both set coefficients of the active party.

5. An image recognition device using an information classification method of low-latency federated learning according to any one of claims 2-4, characterized in that, The image recognition device is the active party of the vertical federated learning, and is used to execute the information classification method of the low-latency vertical federated learning to obtain the category of the image to be recognized. When the image recognition device is used to recognize handwritings, the information sample is a handwritten symbol, and the set category label is a standard symbol; the image recognition device executes the information classification method of the low-latency vertical federated learning to obtain the standard symbol corresponding to the handwritten symbol. When the image recognition device is used to recognize an article, the information sample is an article image, and the set category label is an article category; the image recognition device executes the information classification method of the low-latency vertical federated learning to obtain the article category corresponding to the article image.

6. A census device employing an information classification method with low-latency federated learning according to any one of claims 2-4, characterized in that, The information sample is population census data; the population census device is the active party of the vertical federated learning, and sets the category label as a population classification index; the population census device executes the information classification method of the low-latency vertical federated learning to obtain the category of the residents under different classification indexes. The classification index includes at least one of age, income, education, gender and health status.

7. A movie and television emotion classification device using an information classification method of low-latency federated learning according to any one of claims 2-4. The information sample is film and television evaluation information; the film and television emotion classification device is the active party of the vertical federated learning, and sets the category label to include positive and negative; the film and television emotion classification device executes the information classification method of the low-latency vertical federated learning to obtain the emotion category of the to-be-tested film and television information, which is positive or negative.

8. A failure diagnosis apparatus that adopts an information classification method of low-latency federated learning according to any one of claims 2 to 4, characterized by, The information sample is collected data of a diagnosis object, and the information sample includes at least one of an image, a vibration signal, and an electrical signal; the fault diagnosis device is an active party of longitudinal federated learning, and the category label set by the fault diagnosis device includes multiple working condition categories; the fault diagnosis device executes the information classification method of low-latency longitudinal federated learning to determine whether the diagnosis object is faulty at the time when the information sample is collected; when the diagnosis object is a motor, the working condition category is used to indicate at least one of a speed, a load torque, and a load force of the diagnosis object; when the diagnosis object is a mechanical transmission component, the working condition category includes normal and at least one fault category.

9. A credit risk control device using an information classification method of low-latency federated learning according to any one of claims 2 to 4, characterized by, The information sample is credit card score data of a resident, and the credit card score data includes at least one of an age, a bank card overdue times, and an income; the credit risk control device is an active party of longitudinal federated learning, and the category label set by the credit risk control device is used to indicate a financial situation of the resident in a future specified time; the credit risk control device executes the information classification method of low-latency longitudinal federated learning to obtain a category label corresponding to the credit card score data of the resident, so as to determine whether the resident is in a financial difficulty in the future specified time.

10. An advertisement recommendation platform employing the information classification method of low-latency federated learning according to any one of claims 2-4, characterized in that, The information sample includes an advertisement and a customer; the advertisement recommendation platform is an active party of longitudinal federated learning, and the category label set by the advertisement recommendation platform includes compliance and non-compliance; the advertisement recommendation platform executes the information classification method of low-latency longitudinal federated learning to obtain information samples predicted as compliant as target samples, and pushes the advertisement in the target samples to the corresponding customer.

Images