Machine forgetting learning method for longitudinal federal learning framework and application
By using machine forgetting learning methods in the vertical federated learning framework, the nodes and features of the gradient-enhancing decision tree are updated, and data forgetting problems in vertical federated learning are solved, efficient and privacy-protected forgetting learning is achieved, and model performance is maintained.
Patent Information
- Application Number
- CN202510015582.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-06
- Publication Date
- 2025-05-13
AI Technical Summary
In vertical federated learning, it is difficult for the prior art to efficiently implement forgetful learning of data instances or features, especially when multiple data parties do not share the original data, resulting in excessive resource consumption.
Using a machine forgetting learning method oriented towards a vertical federated learning framework, we use initialization gradient to improve the decision tree, respond to instance and feature deletion requests, update the splitting method and robust feature replacement of the decision tree nodes, and realize local updates without retraining the entire model.
It realizes efficient forgetting learning ability, significantly reduces computing resource consumption, ensures data privacy protection, and maintains the stability of model performance.
Smart Images
Figure CN119990364A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed model training, and in particular to a machine forgetting learning method and application for a vertical federated learning framework. Background Art
[0002] As privacy protection becomes more important around the world, companies are required to support users’ requests to delete personal data. This makes it particularly important for machine learning models to quickly and efficiently complete the forgetting learning operation of data when some data needs to be deleted.
[0003] In the existing technology, traditional forgetting learning methods mostly rely on retraining the model, that is, after removing the user data, the entire model is retrained using the remaining data. However, this method is very computationally intensive, especially in the vertical federated learning (VFL) scenario, where multiple data providers participate in the training and the data cannot be directly shared, making data deletion more complicated.
[0004] In summary, when deleting data instances or features, the federated learning model must retrain the model, resulting in excessive resource consumption. Especially in vertical federated learning, how to support efficient instance and feature forgetting learning when multiple data parties do not share original data is a difficult problem that has not been solved in existing technologies. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a machine forgetting learning method and application for a vertical federated learning framework to achieve feature-level forgetting learning.
[0006] The purpose of the present invention can be achieved by the following technical solutions:
[0007] One aspect of the present invention provides a machine forgetting learning method for a vertical federated learning framework, comprising:
[0008] Initializing a gradient boosting decision tree involving a first data cube and at least one second data cube, and constructing a vertical federated learning framework, wherein the first data cube includes a training data set with labels, and the second data cube includes a training data set lacking labels;
[0009] In response to receiving an instance deletion request, identifying a target instance to be deleted, and adjusting each subtree by updating a splitting method of a gradient boosting decision tree node;
[0010] In response to receiving a feature deletion request, a node corresponding to a target feature to be deleted is identified, the target feature is replaced with a robust feature, and the nodes and subtrees associated with the target feature are trained.
[0011] As a preferred technical solution, the construction process of the gradient boosting decision tree includes:
[0012] The first data party establishes a connection with the second data party, and assigns a homomorphic encryption key to each second data party to achieve privacy-protected data transmission and calculation;
[0013] Each data cube builds a binary tree in the instance space to bucket samples based on the characteristics of its own data set. During the construction process, if the number of instances contained in a leaf node exceeds the preset threshold, it is split at the intermediate feature value until the number of instances of all leaf nodes does not exceed the preset maximum value.
[0014] The first data cube calculates the split gain of the gradient boosting decision tree in the buckets of its independent instance space according to all features, and selects the current best split feature;
[0015] The first data party notifies the data party having the best split feature to complete the construction of the corresponding node of the gradient boosting decision tree.
[0016] As a preferred technical solution, the process of splitting at the intermediate eigenvalues includes:
[0017] The first data entity obtains the encrypted gradient information of each second data entity based on the label data using the homomorphic encryption key;
[0018] For each second data cube, the sum of the prefixes of all splits and feature losses is calculated through gradient information and sent to the first data cube. The first data cube decrypts and calculates the corresponding gain, selects robust features, splits adjacent features and sends them to each second data cube. The second data cube calculates the optimal split point to form a lookup table. The first data cube randomly selects robust features and builds a response record table based on the lookup table.
[0019] As a preferred technical solution, the process of adjusting each subtree by updating the instance space and split gain of the gradient boosting decision tree node includes:
[0020] The first data cube identifies a loss of the target instance, and updates a split neighbor feature of a node based on the loss;
[0021] The second data cube selects a split point with a maximum gain from splitting adjacent features, and updates the instance space of the subtree.
[0022] As a preferred technical solution, the process of replacing the target feature with the robust feature and retraining the nodes and subtrees associated with the target feature includes:
[0023] The first data cube traverses each decision tree and identifies a first node that matches the target feature;
[0024] The first data cube randomly selects robust features to replace the target features of the decision tree, transfers the ownership of the node, and retrains the subtree.
[0025] Another aspect of the present invention provides a machine forgetting learning method for the aforementioned vertical federated learning framework, wherein the first data party is the central control platform of the manufacturing enterprise, and the second data party is the distributed factory terminal. The machine forgetting learning method is used to implement forgetting training of the equipment maintenance prediction model, and the input of the equipment maintenance prediction model includes equipment operation data and sensor fault data, and the output of the equipment maintenance prediction model includes equipment health status score, fault warning signal and maintenance scheduling recommendation.
[0026] Another aspect of the present invention provides a machine forgetting learning system for a vertical federated learning framework, which is used to implement the aforementioned machine forgetting learning method for a vertical federated learning framework, and the system includes:
[0027] A gradient boosting decision tree construction module, used to initialize a gradient boosting decision tree involving a first data cube and at least one second data cube, and construct a vertical federated learning framework, wherein the first data cube includes a training data set with labels, and the second data cube includes a training data set lacking labels;
[0028] An instance deletion forgetting learning module is used to identify a target instance to be deleted in response to receiving an instance deletion request, and to adjust each subtree by updating the instance space and split gain of the gradient boosting decision tree node;
[0029] The feature deletion forgetting learning module is used to respond to receiving a feature deletion request, identify the node corresponding to the target feature to be deleted, replace the target feature with a robust feature, and train the nodes and words associated with the target feature.
[0030] As a preferred technical solution, the gradient boosting decision tree construction module is used to achieve:
[0031] The first data party establishes a connection with the second data party, and assigns a homomorphic encryption key to each second data party to achieve privacy-protected data transmission and calculation;
[0032] Each data cube builds a binary tree in the instance space to bucket samples based on the characteristics of its own data set. During the construction process, if the number of instances contained in a leaf node exceeds the preset threshold, it is split at the intermediate feature value until the number of instances of all leaf nodes does not exceed the preset maximum value.
[0033] The first data cube calculates the split gain of the gradient boosting decision tree in the buckets of its independent instance space according to all features, and selects the current best split feature;
[0034] The first data party notifies the data party having the best split feature to complete the construction of the corresponding node of the gradient boosting decision tree.
[0035] As a preferred technical solution, the example deletion and forgetting learning module is used to achieve:
[0036] The first data cube identifies gradient information of the target instance, and updates the split gain of the node based on the gradient information;
[0037] The second data cube selects a split point with a maximum gain from splitting adjacent features, and updates the instance space of the subtree.
[0038] As a preferred technical solution, the feature deletion and forgetting learning module is used to achieve:
[0039] The first data cube traverses each decision tree and identifies a first node that matches the target feature;
[0040] The first data cube randomly selects robust features to replace the target features of the decision tree, transfers the ownership of the node, and retrains the subtree.
[0041] Compared with the prior art, the present invention has at least one of the following beneficial effects:
[0042] (1) Efficient forgetting learning capability: There is no need to retrain the entire model, only the affected parts are locally updated, which significantly reduces computing resource consumption.
[0043] (2) Privacy protection: Under the vertical federated learning framework, participants do not need to share original data and can complete the collaborative training and forgetting learning operations of the model while ensuring data privacy.
[0044] (3) Stable model performance: Through a robust splitting method, the impact of the forgetting learning operation on the model performance is minimized, maintaining high model accuracy and effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 It is a flow chart of a machine forgetting learning method for a vertical federated learning framework in an embodiment;
[0046] Figure 2 A schematic diagram of a machine forgetting learning system for a vertical federated learning framework in an embodiment;
[0047] Figure 3 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0049] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0050] Example 1
[0051] Regarding the problems existing in the above prior art, see Figure 1 This embodiment provides a machine forgetting learning method for a vertical federated learning framework, which aims to achieve efficient instance and feature forgetting learning under the vertical federated learning framework while maintaining model performance, including the following steps:
[0052] Step S1, initializing a gradient boosting decision tree involving a first data entity and at least one second data entity, and constructing a vertical federated learning framework, wherein the first data entity includes a labeled training data set with labels, and the second data entity includes a labeled training data set with labels.
[0053] In this step, a gradient boosting decision tree model is used, which uses a secure binary tree bucketing and robust splitting method to achieve forgetting learning of instances and features without full retraining. This model is suitable for the vertical federated learning framework, where multiple participants can collaborate on training through a secure gradient boosting decision tree model without sharing the original data, and supports efficient forgetting learning operations based on user privacy requests after model training is completed.
[0054] Preferably, the construction process includes steps S101-S103.
[0055] Step S101, initialization construction.
[0056] The initialization phase involves the first data party (owning the labels) collaborating with m second data parties (owning the training datasets) and training a global boosted tree model on the training datasets owned by each second data party.
[0057] When the first data party and the second data party establish a connection and the protocol is ready, the first data party creates a homomorphic encryption key set.
[0058] Step S102, safe binary tree bucketing.
[0059] Each second data cube has its own training data set, and each second data cube constructs a binary tree.
[0060] In the process of building a binary tree, if the number of instances of a leaf node exceeds the preset maximum bucket size, the middle feature value will be split. It should be noted that the binary tree bucketing of the instance is applied to all terminal nodes.
[0061] Step S103: robust splitting.
[0062] After multiple binary tree bucketing, the number of instances of all leaf nodes does not exceed the preset maximum value, and the global model is obtained. The first data party can fork the instance to ensure that the depth of the tree can reach the preset depth. Step S103 includes.
[0063] Robust split point search. The first data cube will continue to identify each robust split point. The adjacent features of the node represent the set of candidate split points in the feature. The robust feature represents the set of features whose gain and maximum gain are less than a preset value. Preferably, the gain is the gain of the gradient boosting decision tree.
[0064] Preferably, the process of finding a robust split point may include:
[0065] Step 1: The first data party uses the homomorphic encryption key based on the label data to obtain the encrypted losses of each second data party;
[0066] Step 2: For each second data cube, the sum of all split and feature loss prefixes is calculated and sent to the first data cube. The first data cube decrypts and calculates the corresponding gain, selects robust features, splits adjacent features and sends them to each second data cube. The second data cube calculates the optimal split point and forms a lookup table. The first data cube randomly selects robust features and builds a response record table based on the lookup table.
[0067] Tree weight calculation. Calculate the weight of each leaf node and aggregate the regression tree.
[0068] Step S2, in response to receiving an instance deletion request, identifying a target instance to be deleted, and adjusting each subtree by updating the instance space and split gain of the gradient boosting decision tree node.
[0069] The first data cube identifies the loss corresponding to the target instance to be removed, and then updates the node's split-neighboring features based on the loss. Finally, the second data cube selects the split point with the maximum gain from the split-neighboring features and updates the instance space of the subtree.
[0070] In the actual execution process, the first data party first identifies the gradient and Hessian values of the instances to be deleted, and adjusts each subtree by updating the splitting method of the decision tree nodes so that the deleted instances no longer affect the model's decision. When users request to delete certain specific instances, these instances are marked in the data of multiple participants. The operation of instance forgetting involves deleting these marked instances and adjusting the model so that these instances no longer affect the training and prediction results.
[0071] Step S3, in response to receiving a feature deletion request, identifying the node corresponding to the target feature to be deleted, replacing the target feature with a robust feature, and training the nodes and subtrees associated with the target feature.
[0072] First, the first data cube traverses each decision tree and identifies the first node that matches the target feature. Then, the first data cube randomly selects a robust feature to replace the target feature of the decision tree, transfers the ownership of the node, and retrains the subtree.
[0073] In the actual execution process, the first executor identifies the nodes corresponding to the features to be deleted, selects a new robust feature as a replacement, and retrains the relevant nodes and subtrees to reduce the model performance degradation caused by feature deletion. Some features are marked as needing to be deleted in different data cubes. In feature forgetting learning, the model will identify the relevant features and delete them, and then adjust the model structure to ensure that the deleted features no longer affect the model performance. Throughout the process, the boundaries between data cubes are not breached, ensuring the data privacy of the participants.
[0074] This method has the following beneficial effects:
[0075] (1) Efficient forgetting learning capability: There is no need to retrain the entire model, only the affected parts are locally updated, which significantly reduces computing resource consumption.
[0076] (2) Privacy protection: Under the vertical federated learning framework, participants do not need to share original data and can complete the collaborative training and forgetting learning operations of the model while ensuring data privacy.
[0077] (3) Stable model performance: Through a robust splitting method, the impact of the forgetting learning operation on the model performance is minimized, maintaining high model accuracy and effectiveness.
[0078] Example 2
[0079] Based on Example 1, this example provides an application of the aforementioned machine forgetting learning method for the vertical federated learning framework. Specifically, the first data party is the central control platform of the manufacturing enterprise, and the second data party is the distributed factory terminal. The machine forgetting learning method is used to implement forgetting training of the equipment maintenance prediction model. The input of the equipment maintenance prediction model includes equipment operation data and sensor fault data, and the output of the equipment maintenance prediction model includes equipment health status score, fault warning signal and maintenance scheduling suggestion.
[0080] Among them, forgotten scenarios include the elimination of old model equipment. By receiving deletion requests for eliminated equipment, identifying historical operation data and fault data related to the equipment, and removing the impact of old model equipment on prediction accuracy from the model, the model is ensured to adapt to the updated equipment group and maintain stable prediction performance.
[0081] Example 3
[0082] Based on the above embodiments, see Figure 2 This embodiment provides a machine forgetting learning system for a vertical federated learning framework, which is used to implement the aforementioned machine forgetting learning method for a vertical federated learning framework. The system includes:
[0083] (1) A gradient boosting decision tree construction module, used to initialize a gradient boosting decision tree involving a first data cube and at least one second data cube, and to construct a vertical federated learning framework, wherein the first data cube includes a training data set with labels and the second data cube includes a training data set without labels.
[0084] Specifically, this module is used to establish a connection between the first data party and the second data party, and assign a homomorphic encryption key to each second data party to achieve privacy-protected data transmission and calculation; each data party constructs a binary tree in the instance space to bucket samples based on the characteristics of its own data set. During the construction process, if the number of instances contained in a leaf node exceeds a preset threshold, it is split at the intermediate feature value until the number of instances of all leaf nodes does not exceed the preset maximum value; the first data party calculates the split gain of the gradient boosting decision tree in the bucket of its independent instance space based on all features, and selects the current best split feature; the first data party notifies the data party with the best split feature to complete the construction of the corresponding node of the gradient boosting decision tree.
[0085] (2) An instance deletion forgetting learning module is used to respond to an instance deletion request, identify the target instance to be deleted, and adjust each subtree by updating the instance space and split gain of the gradient boosting decision tree node.
[0086] Specifically, this module is used to enable the first data party to identify the gradient information of the target instance, update the split gain of the node based on the gradient information, and the second data party selects the split point with the maximum gain from the split adjacent features to update the instance space of the subtree.
[0087] (3) A feature deletion forgetting learning module is used to respond to receiving a feature deletion request, identify the node corresponding to the target feature to be deleted, replace the target feature with a robust feature, and train the nodes and words associated with the target feature.
[0088] Specifically, this module is used to enable the first data party to traverse each decision tree, identify the first node that matches the target feature, randomly select robust features to replace the target feature of the decision tree, transfer the ownership of the node, and retrain the subtree.
[0089] Example 4
[0090] Based on the above embodiments, see Figure 3 This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned machine forgetting learning method for the vertical federated learning framework.
[0091] like Figure 3 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The machine forgetting learning method for the vertical federated learning framework. Of course, in addition to the software implementation, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0092] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0093] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0094] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A machine forgetting learning method for a vertical federated learning framework, characterized in that: include: Initializing a gradient boosting decision tree involving a first data cube and at least one second data cube, and constructing a vertical federated learning framework, wherein the first data cube includes a training data set with labels, and the second data cube includes a training data set lacking labels; In response to receiving an instance deletion request, identifying a target instance to be deleted, and adjusting each subtree by updating an instance space and a split gain of a gradient boosting decision tree node; In response to receiving a feature deletion request, a node corresponding to a target feature to be deleted is identified, the target feature is replaced with a robust feature, and a subtree of the node associated with the target feature is retrained.
2. According to the machine forgetting learning method for a vertical federated learning framework according to claim 1, it is characterized in that: The construction process of the gradient boosting decision tree includes: The first data party establishes a connection with the second data party, and assigns a homomorphic encryption key to each second data party to achieve privacy-protected data transmission and calculation; Each data cube builds a binary tree in the instance space to bucket samples based on the characteristics of its own data set. During the construction process, if the number of instances contained in a leaf node exceeds the preset threshold, it is split at the intermediate feature value until the number of instances of all leaf nodes does not exceed the preset maximum value. The first data cube calculates the split gain of the gradient boosting decision tree in the buckets of its independent instance space according to all features, and selects the current best split feature; The first data party notifies the data party having the best split feature to complete the construction of the corresponding node of the gradient boosting decision tree.
3. According to claim 2, a machine forgetting learning method for a vertical federated learning framework is characterized in that: The process of splitting at the intermediate eigenvalues includes: The first data entity obtains the encrypted gradient information of each second data entity based on the label using the homomorphic encryption key; For each second data cube, the sum of the prefixes of all splits and feature losses is calculated through gradient information and sent to the first data cube. The first data cube decrypts and calculates the corresponding gain, selects robust features, splits adjacent features and sends them to each second data cube. The second data cube calculates the optimal split point to form a lookup table. The first data cube randomly selects robust features and builds a response record table based on the lookup table.
4. According to claim 3, a machine forgetting learning method for a vertical federated learning framework is characterized in that: The process of adjusting each subtree by updating the instance space and split gain of the decision tree node by gradient boosting includes: The first data cube identifies gradient information of the target instance, and updates the split gain of the node based on the gradient information; The second data cube selects a split point with a maximum gain from splitting adjacent features, and updates the instance space of the subtree.
5. According to claim 3, a machine forgetting learning method for a vertical federated learning framework is characterized in that: The process of replacing the target feature with the robust feature and retraining the subtree of the node associated with the target feature includes: The first data cube traverses each decision tree and identifies a first node that matches the target feature; The first data cube randomly selects robust features to replace the target features of the decision tree, transfers the ownership of the node, and retrains the subtree.
6. An application of the machine forgetting learning method for a vertical federated learning framework as claimed in any one of claims 1 to 5, characterized in that: The first data party is the central control platform of the manufacturing enterprise, the second data party is the distributed factory terminal, and the machine forgetting learning method is used to realize the forgetting training of the equipment maintenance prediction model. The input of the equipment maintenance prediction model includes equipment operation data and sensor fault data, and the output of the equipment maintenance prediction model includes equipment health status score, fault warning signal and maintenance scheduling suggestion.
7. A machine forgetting learning system for a vertical federated learning framework, characterized in that: A system for implementing a machine forgetting learning method for a vertical federated learning framework as described in any one of claims 1 to 5, the system comprising: A gradient boosting decision tree construction module, used to initialize a gradient boosting decision tree involving a first data cube and at least one second data cube, and construct a vertical federated learning framework, wherein the first data cube includes a training data set with labels, and the second data cube includes a training data set lacking labels; An instance deletion forgetting learning module is used to identify a target instance to be deleted in response to receiving an instance deletion request, and to adjust each subtree by updating the instance space and split gain of the gradient boosting decision tree node; The feature deletion forgetting learning module is used to respond to receiving a feature deletion request, identify the node corresponding to the target feature to be deleted, replace the target feature with a robust feature, and train the nodes and words associated with the target feature.
8. The machine forgetting learning system for a vertical federated learning framework according to claim 7, characterized in that: The gradient boosting decision tree building module is used to achieve: The first data party establishes a connection with the second data party, and assigns a homomorphic encryption key to each second data party to achieve privacy-protected data transmission and calculation; Each data cube builds a binary tree in the instance space to bucket samples based on the characteristics of its own data set. During the construction process, if the number of instances contained in a leaf node exceeds the preset threshold, it is split at the intermediate feature value until the number of instances of all leaf nodes does not exceed the preset maximum value. The first data cube calculates the split gain of the gradient boosting decision tree in the buckets of its independent instance space according to all features, and selects the current best split feature; The first data party notifies the data party having the best split feature to complete the construction of the corresponding node of the gradient boosting decision tree.
9. The machine forgetting learning system for a vertical federated learning framework according to claim 8, characterized in that: The instance deletion forgetting learning module is used to achieve: The first data cube identifies gradient information of the target instance, and updates the split gain of the node based on the gradient information; The second data cube selects a split point with a maximum gain from splitting adjacent features, and updates the instance space of the subtree.
10. The machine forgetting learning system for a vertical federated learning framework according to claim 8, characterized in that: The feature deletion and forgetting learning module is used to achieve: The first data cube traverses each decision tree and identifies a first node that matches the target feature; The first data cube randomly selects robust features to replace the target features of the decision tree, transfers the ownership of the node, and retrains the subtree.