A hierarchical multi-label scene federated learning method

Abstract

The hierarchical multi-label scene federated learning method disclosed in the present application constructs a directed acyclic graph through the label types obtained in the hierarchical multi-label scene, records the level of the directed acyclic graph where each client label type is located, constructs a feature processing model according to the characteristics of the sample data and the directed acyclic graph, the feature processing model can obtain activation values of different levels, and each client can obtain the activation values of the corresponding level through the level where the client is located, thereby supporting the subsequent calculation process; then the pseudo-label cross-entropy loss of the unlabeled activation value is calculated, the gradient back propagation of the labeled and unlabeled data in the training data is ensured, thereby protecting the data privacy, in addition, the mean square error loss of the correlation graph between adjacent levels is calculated, the low-level client can guide the parameter optimization of the high-level client, the federated learning effect is achieved, and the security and usability of the federated learning in the hierarchical multi-label scene are improved.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a federated learning method for a hierarchical, multi-label scenario. Background Technology

[0002] In the internet age, data has permeated every industry and business function, becoming a crucial factor of production. Because massive amounts of data are often scattered across various enterprises, departments, or business systems within an organization, data owners need to collaborate to achieve better information mining and utilization. On the other hand, due to concerns about protecting data value and privacy breaches, data owners are often reluctant to directly transfer their data to others for processing.

[0003] Federated learning is a type of machine learning method that leverages the non-interpretive nature of deep learning models by replacing raw data interaction with model parameters, activation values, or gradient transfer. Federated learning enables collaborative modeling among partners without requiring the raw data to remain within its domain, achieving a balance between data privacy and learning effectiveness to some extent. In existing federated learning implementations, when partners possess data with differing dimensions, sample alignment is necessary. This is achieved by sharing the intersection of data IDs from all parties to determine the training batch and location of the intersection, enabling model training and parameter optimization.

[0004] However, sample alignment schemes still have the following problems in hierarchical multi-label training scenarios: In hierarchical multi-label scenarios, different partners have different methods for data segmentation, and the data is classified into multiple labels of different granularities, with these labels exhibiting a progressive hierarchical structure. In this scenario, low-level labels and high-level labels are correlated, and may even be causally related. Sample alignment schemes share the intersection of data IDs, and partners with high-level labels can deduce low-level labels in the intersection through the correlation between labels, leading to the leakage of label information of partners with low-level labels in the intersection. Summary of the Invention

[0005] The purpose of this invention is to provide a federated learning method that can guarantee the data privacy of collaborators in hierarchical multi-label scenarios, thereby improving the security and usability of federated learning in hierarchical multi-label scenarios.

[0006] The federated learning method for hierarchical multi-label scenarios described in this invention includes the following steps:

[0007] S1. The server with sample data obtains the types of tags owned by each client;

[0008] S2. The server constructs a directed acyclic graph (DAG) based on the obtained label types and records the level of the DAG where each client's label type is located. It constructs a feature processing model based on the features of the sample data and the DAG. Each client constructs a semi-supervised prediction model based on its own label type. The parameters of the feature processing model and the semi-supervised prediction model are randomly initialized.

[0009] S3. The server shuffles the sample data and uses a hash or privacy information retrieval algorithm based on the enumeration attribute of the sample data ID to enable the client to determine the activation value type corresponding to the tag data. The activation value type corresponds to tagged data and untagged data.

[0010] S4. The server inputs the sample data into the feature processing model and sends the activation values ​​of the sample data to the corresponding client. The client calculates the cross-entropy loss for labeled activation values ​​and the pseudo-label cross-entropy loss for unlabeled activation values ​​based on the received activation values ​​and the activation value type determined in S3. It also calculates the correlation graph of the activation values ​​and calculates the mean squared error loss with the correlation graph of the previous level. The client updates the parameters of the semi-supervised prediction model and sends the activation value gradient back to the server to update the parameters of the feature processing model.

[0011] S5. Repeat step S4 until the parameters of the feature processing model and the semi-supervised prediction model converge to obtain a federated learning model for sample prediction.

[0012] The federated learning method for hierarchical multi-label scenarios described in this invention utilizes a directed acyclic graph (DAG) structure. Therefore, when constructing the feature processing model, a DAG is built based on the acquired label types, and the hierarchy of the DAG for each client's label type is recorded. The feature processing model is constructed based on the features of the sample data and the DAG, enabling the acquisition of activation values ​​at different levels. Each client can obtain the activation value of its corresponding level, thus supporting subsequent calculations of labeled and unlabeled data. Furthermore, by calculating the pseudo-label cross-entropy loss of the unlabeled activation values, gradient backpropagation is ensured for both labeled and unlabeled training data, protecting data privacy. Additionally, by calculating the mean squared error loss of the correlation graph between adjacent levels, lower-level clients can guide parameter optimization for higher-level clients, achieving the effect of federated learning and improving the security and usability of federated learning in hierarchical multi-label scenarios.

[0013] As a preferred embodiment of the present invention, the directed acyclic graph is a multi-label sequence for classification, where the multi-label refers to multiple labels for samples classified according to different levels, different granularities, or different subdivision methods.

[0014] As a preferred embodiment of the present invention, the sample data includes voice, text, images, and other manually defined feature data.

[0015] As a preferred embodiment of the present invention, the server-side construction of a feature processing model based on the characteristics of the sample data and the directed acyclic graph specifically includes:

[0016] The server selects different embedding forms, such as fully connected networks, convolutional neural networks, or recurrent neural networks, to construct the feature embedding layer based on the characteristics of the sample data.

[0017] The feature embedding layer is constructed based on the residual multilayer perceptron with the maximum depth of the directed acyclic graph, and the number of feature embedding layers is constructed to finally build the feature processing model.

[0018] As a preferred embodiment of the present invention, the enumeration attribute of the sample data ID includes both non-enumerable and enumerable sample data ID;

[0019] Since the sample data IDs are not enumerable, the server uses a hash algorithm to hash the unenumerable sample data IDs and sends the activation value corresponding to the hashed tag data to the corresponding client.

[0020] The sample data IDs are enumerable. The server uses a privacy information retrieval algorithm to process and encrypt the enumerable sample data IDs. The client uses the encrypted ID as the key to query the server and obtain the activation value corresponding to the tagged data. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the federated learning method for multi-label scenarios in this invention.

[0022] Figure 2 This is a schematic diagram of the model structure of the federated learning method for multi-label scenarios in this invention. Detailed Implementation

[0023] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This application provides a federated learning method for a hierarchical multi-label scenario, such as... Figure 1 and Figure 2 As shown,

[0025] Includes the following steps:

[0026] S1. The server with sample data obtains the types of tags owned by each client; the sample data includes voice, text, images and other manually defined feature data.

[0027] S2. The server constructs a directed acyclic graph (DAG) based on the obtained label types and records the level of the DAG where each client's label type is located. It constructs a feature processing model based on the features of the sample data and the DAG. Each client constructs a semi-supervised prediction model based on its own label type. The parameters of the feature processing model and the semi-supervised prediction model are randomly initialized.

[0028] In this context, hierarchical multi-label refers to a sequence of classification multi-labels that can be represented by a directed acyclic graph structure. Classification multi-label refers to multiple labels for samples classified according to different levels, granularities, or subdivision methods. In this embodiment, the federated learning task is a Chinese book classification prediction task, with Chinese books classified hierarchically according to the *Chinese Library Classification*. Sample data consists of the titles, keywords, and abstracts of Chinese books; the sample ID is the title of the Chinese book; and the hierarchical labels are the basic category labels and classification number labels of the Chinese book. Assume that partner A has sample data X, which is a feature set represented by unique codes after word segmentation of the titles and keywords of Chinese books; partner B has basic category labels Y. B Partner C owns category number tag Y C Generally speaking, the number of sample data is greater than the number of sample labels, i.e., |X| > |Y. B |,|X|>|Y C You can select partner A as the server and partners B and C as clients.

[0029] According to the *Chinese Library Classification*, a directed acyclic graph is represented as root → basic category → classification number label. Therefore, client B has a depth of 1, and client C has a depth of 2. The server can then construct a feature embedding layer using different embedding methods—fully connected networks, convolutional neural networks, or recurrent neural networks—based on the characteristics of the sample data. The feature embedding layer is a Chinese pre-trained word embedding layer f. emb (Proposed in "Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings"), used to transform sample data (titles and keywords) into word embeddings Z. (0) :

[0030] Z (0) :=f emb (X)

[0031] The feature embedding layers are constructed based on the maximum depth of the directed acyclic graph (DAG) residual multilayer perceptron, resulting in a feature processing model. Since the maximum depth is 2, the multilayer perceptron consists of two fully connected residual networks. Composition. Define the activation value Z of the i-th layer network. (i) :

[0032]

[0033] Among them W i ,b i These are the parameters of the i-th layer network, and ReLU(·) is the linear rectified function.

[0034] The semi-supervised prediction model built by the client can employ a logistic regression model, a deep learning model, or other models that support gradient backpropagation. In this embodiment, both clients B and C build single-layer logistic regression models. Used to generate prediction P(Z) k ):

[0035]

[0036] Where softmax(·) is the normalized exponential function, and k is the depth of the client. In this embodiment, k = 1 for client B and k = 2 for client C.

[0037] S3. The server shuffles the sample data and uses a hash or privacy information retrieval algorithm based on the enumeration attribute of the sample data ID to enable the client to determine the activation value type corresponding to the tag data. The activation value type corresponds to tagged data and untagged data.

[0038] The enumerable attribute of the sample data ID includes non-enumerable and enumerable sample data IDs. If the sample data ID is non-enumerable, the server uses a hash algorithm to hash the non-enumerable sample data ID and sends the activation value corresponding to the hashed tag data to the corresponding client. If the sample data ID is enumerable, the server uses a privacy information retrieval algorithm to process and encrypt the enumerable sample data ID. The client uses the encrypted ID as the key to query the server and obtain the activation value corresponding to its own tag data.

[0039] S4. The server inputs the sample data into the feature processing model and sends the activation values ​​of the sample data to the corresponding client. The client calculates the cross-entropy loss for labeled activation values ​​and the pseudo-label cross-entropy loss for unlabeled activation values ​​based on the received activation values ​​and the activation value type determined in S3. It also calculates the correlation graph of the activation values ​​and calculates the mean squared error loss with the correlation graph of the previous level. The client updates the parameters of the semi-supervised prediction model and sends the activation value gradient back to the server to update the parameters of the feature processing model.

[0040] The client divides the activation values ​​into two groups based on the activation value type determined in step 3: one group consists of tagged activation values. One group is unlabeled activation values Calculate cross-entropy loss for labeled activation values. Calculate pseudo-label cross-entropy loss for unlabeled activation values. Calculate the correlation plot of activation values ​​and the correlation plot loss between adjacent levels.

[0041] Specifically, and Input prediction model Obtain predictions with labeled activation values Prediction of unlabeled activation values Loss function with labeled activation values Defined as:

[0042]

[0043] Where CE(·,·) is the cross-entropy loss, and Y k This is a client-side tag; in this embodiment, Y1 = Y. B and Y2=Y C .

[0044] Unlabeled activation values ​​are calculated using pseudo-labeled cross-entropy loss, defined as:

[0045]

[0046] Here, argmax(·) refers to the index of the maximum value.

[0047] Calculate the correlation plot of activation values:

[0048]

[0049] And calculate the correlation graph loss between two adjacent levels. Defined as:

[0050]

[0051] Where MSE(·,·) is the mean squared error loss.

[0052] Finally, the loss function of the prediction model is:

[0053]

[0054] The activation gradient can be obtained through backpropagation.

[0055] S5. Repeat step S4 until the parameters of the feature processing model and the semi-supervised prediction model converge to obtain a federated learning model for sample prediction.

[0056] In this embodiment, when labeled data accounts for 20% to 100% of all data, the federated learning method proposed in this paper, while protecting the privacy of client B, can achieve an average 2% improvement in prediction accuracy compared to training using only the classification labels owned by client C, as shown in the table below:

[0057]

[0058] The above embodiments are only used to illustrate the detailed solutions of the present invention. The present invention is not limited to the above detailed solutions, that is, it does not mean that the present invention must rely on the above detailed solutions to be implemented. Those skilled in the art should understand that any improvements to the present invention, equivalent substitutions of the raw materials of the product of the present invention, addition of auxiliary components, selection of specific methods, etc., all fall within the protection scope and disclosure scope of the present invention.

Claims

1. A federated learning method for hierarchical multi-label scenarios, characterized in that, Includes the following steps: S1. The server with sample data obtains the types of tags owned by each client; S2. The server constructs a directed acyclic graph (DAG) based on the obtained label types and records the level of the DAG where each client's label type is located. It constructs a feature processing model based on the features of the sample data and the DAG. Each client constructs a semi-supervised prediction model based on its own label type. The parameters of the feature processing model and the semi-supervised prediction model are randomly initialized. Here, hierarchical multi-label refers to a classification multi-label sequence that can be represented by a DAG structure. Classification multi-label refers to multiple labels of samples that are classified according to different levels, different granularities, or different subdivision methods. S3. The server shuffles the sample data and uses a hash or privacy information retrieval algorithm based on the enumeration attribute of the sample data ID to enable the client to determine the activation value type corresponding to the tag data. The activation value type corresponds to tagged data and untagged data. The enumeration attribute of the sample data ID includes sample data ID that is not enumerable and enumerable. Since the sample data IDs are not enumerable, the server uses a hash algorithm to hash the unenumerable sample data IDs and sends the activation value corresponding to the hashed tag data to the corresponding client. The sample data IDs are enumerable. The server uses a privacy information retrieval algorithm to process and encrypt the enumerable sample data IDs. The client uses the encrypted ID as the key to query the server and obtain the activation value corresponding to its tagged data. S4. The server inputs the sample data into the feature processing model and sends the activation values ​​of the sample data to the corresponding client. The client, based on the received activation values ​​and the activation value type determined in S3, calculates the cross-entropy loss for labeled activation values ​​and the pseudo-label cross-entropy loss for unlabeled activation values. It also calculates the correlation graph of the activation values ​​and calculates the mean squared error loss with the correlation graph of the previous level, updates the parameters of the semi-supervised prediction model, and sends the activation value gradient back to the server to update the parameters of the feature processing model; the correlation graph G of the activation values ​​is then calculated. K : G K =P(Z k ) T P(Z k ); Where k represents the depth of the client, P (Z k ) indicates that the semi-supervised prediction model is for the activation value Z. k The output predicted value; P(Z k ):= (WITH k )=softmax(W k WITH k +b k ) Where := represents a definition, This represents a single-layer logistic regression model built by the client, and softmax() represents the normalized exponential function. S5. Repeat step S4 until the parameters of the feature processing model and the semi-supervised prediction model converge to obtain a federated learning model for sample prediction.

2. The federated learning method for hierarchical multi-label scenarios according to claim 1, characterized in that, The sample data includes voice, text, images, and other manually defined feature data.

3. The federated learning method for hierarchical multi-label scenarios according to claim 1, characterized in that, The server-side construction of a feature processing model based on the characteristics of the sample data and the directed acyclic graph specifically includes: The server selects different embedding forms, such as fully connected networks, convolutional neural networks, or recurrent neural networks, to construct the feature embedding layer based on the characteristics of the sample data. The feature embedding layer is constructed based on the residual multilayer perceptron with the maximum depth of the directed acyclic graph, and the number of feature embedding layers is constructed to finally build the feature processing model.

Images