Abnormity detection method and system based on federated learning and isolated forest algorithm

Abstract

The invention relates to an anomaly detection method and system based on federated learning and an isolated forest algorithm. The method comprises the following steps: issuing an initial isolated forest model through a federated learning server; receiving a plurality of first isolated forest models and private data packets; acquiring a corresponding first isolated forest model and corresponding private data packets fed back by other users according to associated information in the private data packets; combining the plurality of associated private data packets to obtain a joint data packet, and training the corresponding first isolated forest model again through the joint data packet to obtain a second isolated forest model; the second isolated forest model is fed back to the corresponding user, and abnormal data detection is carried out according to the second isolated forest model; according to the method, cross-user combined data can be completely restored, privacy of user data is reserved, the training problem of an isolated forest model in the scene can be well solved, and the scheme is simple in structure and very easy to design.

Description

Technical Field

[0001] The present invention relates to the technical field of financial fraud detection, and more specifically, to an anomaly detection method and system based on federated learning and isolation forest algorithm. Background Art

[0002] In the field of financial fraud detection, accurately identifying and locating abnormal behaviors is crucial to ensuring the security and stability of the system. Isolation Forest is an anomaly detection algorithm based on ensemble learning. It identifies abnormal samples by building a forest. The specific steps are as follows:

[0003] 1. Randomly select a feature and a split point to divide the data set into two subsets;

[0004] 2. Recursively repeat step 1 until each subset contains only one sample or reaches the predetermined tree height;

[0005] 3. Repeat steps 1 and 2 to construct multiple random binary trees (i.e., forests);

[0006] 4. For new samples, determine whether they are abnormal by calculating the average path length in each tree;

[0007] However, for some abnormal situations involving the need to unite data across different enterprises to build an isolated forest, for example, the abnormal data is composed of A+B, but data A and data B are held by different enterprises. Due to privacy protection issues between enterprises, they cannot be used jointly, which ultimately leads to the inability of the constructed isolated forest to identify abnormal data A+B. The existing Chinese patent, application number 202111040857.2, provides an isolated forest training and prediction method and system based on vertical federation. The enterprise data is processed based on the vertical federation method, deprivatized, and then the isolated forest is jointly constructed. Although some problems can be solved, taking data A as an example, A' obtained after deprivatization cannot fully represent A, that is, the isolated forest constructed by A'+B will still have the defect of not being able to identify abnormal data A+B to a certain extent. A more reasonable anomaly detection method and system based on federated learning and isolation forest algorithm is needed. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide an anomaly detection method based on federated learning and isolation forest algorithm, and also provide an anomaly detection system based on federated learning and isolation forest algorithm in view of the above-mentioned defects of the prior art.

[0009] The technical solution adopted by the present invention to solve its technical problem is:

[0010] Construct an anomaly detection method based on federated learning and isolation forest algorithm, which includes the following steps:

[0011] The federated learning server publishes the initial isolation forest model to multiple users participating in the federated learning;

[0012] Receiving a trained first isolation forest model of multiple user feedbacks and a private data packet associated with other user data;

[0013] According to the association information in the private data packet, obtain the first isolation forest model fed back by other users and the private data packet fed back by other users;

[0014] Combining multiple associated private data packets to obtain a joint data packet, and retraining the corresponding first isolation forest model through the joint data packet to obtain a second isolation forest model;

[0015] The second isolation forest model is fed back to the corresponding user, and abnormal data detection is performed based on the second isolation forest model.

[0016] The anomaly detection method based on federated learning and isolation forest algorithm of the present invention, wherein the server of federated learning publishes the initial isolation forest model to multiple users participating in federated learning, comprises the following steps:

[0017] The server receives requests from multiple users for jointly training an isolation forest model;

[0018] Obtain information about technical fields that multiple users are in common, and confirm the technical field information with the users;

[0019] Recalling the stored historical isolation forest model associated with the confirmed technical field information;

[0020] Identify the historical isolated forest models needed;

[0021] The historical isolation forest model is processed by removing the data retention framework to obtain the initial isolation forest model.

[0022] In the anomaly detection method based on federated learning and isolation forest algorithm of the present invention, the private data packet includes the following content:

[0023] The private data that needs to be jointly trained and the joint information of the private data confirmed by the user.

[0024] The anomaly detection method based on federated learning and isolation forest algorithm of the present invention, wherein the step of combining multiple associated private data packets to obtain a joint data packet comprises the following steps:

[0025] Get the joint information in the private data packet;

[0026] Find the private data that needs to be joined based on the joint information;

[0027] Private data that is not involved in the federation is marked and fed back to the corresponding users.

[0028] The anomaly detection method based on federated learning and isolation forest algorithm of the present invention, wherein the step of retraining the corresponding first isolation forest model by combining the data packets to obtain the second isolation forest model comprises the following steps:

[0029] According to the association relationship between the private data and the user, query the first isolation forest model sent by the user;

[0030] According to the association relationship between the private data and the joint data package, the joint data package is matched with the first isolation forest model;

[0031] The server trains the first isolation forest model again by combining the data packets to obtain a second isolation forest model.

[0032] The anomaly detection method based on federated learning and isolation forest algorithm of the present invention further comprises the steps of:

[0033] If the feedback received from the server by the user only contains the second isolation forest model, the second training will not be initiated;

[0034] If the feedback received from the server by the user includes the second isolation forest model and the marked private data not involved in the joint training, the user will negotiate with other users to revise the private data packet and initiate a joint training request to the server again.

[0035] The anomaly detection method based on federated learning and isolation forest algorithm described in the present invention, wherein the technical field information includes the technical field of financial fraud detection.

[0036] An anomaly detection system based on federated learning and isolation forest algorithm, according to the above-mentioned anomaly detection method based on federated learning and isolation forest algorithm, wherein the system includes a federated learning server and a plurality of user-end devices;

[0037] The federated learning server is used to publish an initial isolation forest model to multiple users participating in the federated learning; receive the trained first isolation forest model fed back by multiple user-end devices, and a private data packet associated with other user data; obtain the first isolation forest model fed back by the corresponding other user-end devices and the private data packet fed back by the corresponding other user-end devices based on the associated information in the private data packet; combine multiple associated private data packets to obtain a joint data packet, retrain each first isolation forest model through the joint data packet to obtain a second isolation forest model; feed back the second isolation forest model to the corresponding user-end device, and the user-end device performs abnormal data detection based on the second isolation forest model.

[0038] The anomaly detection system based on federated learning and isolation forest algorithm of the present invention, wherein the server of federated learning publishes the initial isolation forest model to multiple users participating in federated learning, comprises the following steps:

[0039] The server receives requests from multiple users for jointly training an isolation forest model;

[0040] Obtain information about technical fields that multiple users are in common, and confirm the technical field information with the users;

[0041] Recalling the stored historical isolation forest model associated with the confirmed technical field information;

[0042] Identify the historical isolated forest models needed;

[0043] The historical isolation forest model is processed by removing the data retention framework to obtain the initial isolation forest model;

[0044] The private data package includes:

[0045] Private data confirmed by the user that needs to be jointly trained and joint information of the private data;

[0046] Combining multiple associated private data packets to obtain a joint data packet includes the following steps:

[0047] Get the joint information in the private data packet;

[0048] Find the private data that needs to be joined based on the joint information;

[0049] Private data that is not involved in the federation is marked and fed back to the corresponding users.

[0050] The anomaly detection system based on federated learning and isolation forest algorithm of the present invention, wherein retraining the corresponding first isolation forest model by combining data packets to obtain the second isolation forest model comprises the following steps:

[0051] According to the association relationship between the private data and the user, query the first isolation forest model sent by the user;

[0052] According to the association relationship between the private data and the joint data package, the joint data package is matched with the first isolation forest model;

[0053] The server trains the first isolation forest model again by combining the data packets to obtain a second isolation forest model;

[0054] If the feedback received from the server by the user only contains the second isolation forest model, the second training will not be initiated;

[0055] If the feedback received from the server by the user includes the second isolation forest model and the marked private data not involved in the joint training, the user will negotiate with other users to revise the private data packet and initiate a joint training request to the server again.

[0056] The beneficial effects of the present invention are: applying the method of the present application, relying on the federated learning server to publish the isolation forest basic model, and then receiving the feedback of the trained first isolation forest model and private data package, integrating the private data package and re-training and feedback of multiple first isolation forest models, the combined data across users can be completely restored and the privacy of user data is retained, which can well solve the training problem of the isolation forest model in this scenario, the solution structure is simple, and the design is also very easy. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work:

[0058] Figure 1 It is a flow chart of an anomaly detection method based on federated learning and isolation forest algorithm according to a preferred embodiment of the present invention;

[0059] Figure 2 It is a principle block diagram of an anomaly detection system based on federated learning and isolation forest algorithm in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be described clearly and completely in combination with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are partial embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0061] The anomaly detection method based on federated learning and isolation forest algorithm in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0062] S01: The federated learning server publishes the initial isolation forest model to multiple users participating in the federated learning;

[0063] S02: receiving a trained first isolation forest model fed back by multiple users, and a private data packet associated with other user data;

[0064] S03: acquiring the first isolation forest model fed back by other users and the private data packets fed back by other users according to the associated information in the private data packets;

[0065] S04: combining multiple associated private data packets to obtain a joint data packet, and retraining the corresponding first isolation forest model through the joint data packet to obtain a second isolation forest model;

[0066] S05: Feedback the second isolation forest model to the corresponding user, and perform abnormal data detection based on the second isolation forest model;

[0067] By applying the method of the present application, the isolated forest basic model is published by relying on the federated learning server, and then the trained first isolated forest model and private data packet are fed back. After the private data packet is integrated, multiple first isolated forest models are retrained and fed back. The combined data across users can be completely restored and the privacy of user data can be retained. The training problem of the isolated forest model in this scenario can be well solved. The solution structure is simple and the design is very easy.

[0068] Preferably, publishing the initial isolation forest model to multiple users participating in the federated learning through the federated learning server includes the following steps:

[0069] The server receives requests from multiple users for jointly training an isolation forest model;

[0070] Obtain information about technical fields that multiple users are in common, and confirm the technical field information with the users;

[0071] Recalling the stored historical isolation forest model associated with the confirmed technical field information;

[0072] Identify the historical isolated forest models needed;

[0073] The historical isolation forest model is processed by removing the data retention framework to obtain the initial isolation forest model;

[0074] Applying this method can reduce the occupancy rate of server resources and the workload of redesigning model algorithms. With the help of this architecture, users who need to conduct joint analysis have more autonomy in choosing, and it has stronger applicability and flexibility.

[0075] Preferably, the private data packet includes the following contents:

[0076] The private data that needs to be jointly trained and the joint information of the private data confirmed by the user.

[0077] Preferably, combining multiple associated private data packets to obtain a joint data packet comprises the steps of:

[0078] Get the joint information in the private data packet;

[0079] Find the private data that needs to be joined based on the joint information;

[0080] Mark the private data that is not involved in the federation and feedback it to the corresponding users;

[0081] The application of this automatic combination mechanism is intelligent and does not require much human intervention. It only needs to agree on the combination protocol information in the early stage. For the uncombined private data, as described below, it can participate in the next repeated training, which reduces manual operations and improves the flexibility of combined applications.

[0082] Preferably, retraining the corresponding first isolation forest model by combining the data packets to obtain the second isolation forest model comprises the following steps:

[0083] According to the association relationship between the private data and the user, query the first isolation forest model sent by the user;

[0084] According to the association relationship between the private data and the joint data package, the joint data package is matched with the first isolation forest model;

[0085] The server trains the first isolation forest model again by combining the data packets to obtain a second isolation forest model;

[0086] It is convenient to perform automatic correspondence and retraining operations; wherein, the retraining can adopt the existing method without making any improvement thereto.

[0087] Preferably, the method further comprises the steps of:

[0088] If the feedback received from the server by the user only contains the second isolation forest model, the second training will not be initiated;

[0089] If the feedback received from the server includes the second isolation forest model and the marked private data not involved in the joint training, the user will negotiate with other users to revise the private data packet and initiate a joint training request to the server again;

[0090] Uncombined private data can be used in the next repeated training, which reduces manual operations and improves the flexibility of combined applications.

[0091] Preferably, the technical field information includes the technical field of financial fraud detection, which can be expanded or narrowed as needed and can be determined based on user consultation; of course, it is understandable that this method can also be extended to other fields such as network security.

[0092] An anomaly detection system based on federated learning and isolation forest algorithm, according to the above-mentioned anomaly detection method based on federated learning and isolation forest algorithm, such as Figure 2 As shown, the system includes a federated learning server 1 and multiple user terminal devices 2;

[0093] A federated learning server 1 is used to publish an initial isolation forest model to multiple users participating in federated learning; receive a trained first isolation forest model fed back by multiple user-end devices 2, and a private data packet associated with other user data; obtain the first isolation forest model fed back by the corresponding other user-end devices and the private data packet fed back by the corresponding other user-end devices 2 according to the associated information in the private data packet; combine multiple associated private data packets to obtain a joint data packet, and retrain each first isolation forest model through the joint data packet to obtain a second isolation forest model; feed back the second isolation forest model to the corresponding user-end device 2, and the user-end device 2 performs abnormal data detection based on the second isolation forest model;

[0094] The system of the present application relies on a federated learning server to publish an isolation forest basic model, and then receives feedback of the trained first isolation forest model and private data package. After integrating the private data package, multiple first isolation forest models are retrained and fed back. The combined data across users can be completely restored and the privacy of user data can be retained. The training problem of the isolation forest model in this scenario can be well solved. The solution structure is simple and the design is also very easy.

[0095] For the specific implementation method, please refer to the above description and will not be repeated here.

[0096] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. An anomaly detection method based on federated learning and isolation forest algorithm, characterized in that: The following steps are involved: The federated learning server publishes the initial isolation forest model to multiple users participating in the federated learning; Receiving a trained first isolation forest model of multiple user feedbacks and a private data packet associated with other user data; According to the association information in the private data packet, obtain the first isolation forest model fed back by other users and the private data packet fed back by other users; Combining multiple associated private data packets to obtain a joint data packet, and retraining the corresponding first isolation forest model through the joint data packet to obtain a second isolation forest model; The second isolation forest model is fed back to the corresponding user, and abnormal data detection is performed based on the second isolation forest model.

2. The anomaly detection method based on federated learning and isolation forest algorithm according to claim 1, characterized in that: The method of publishing the initial isolation forest model to multiple users participating in the federated learning through the federated learning server includes the following steps: The server receives requests from multiple users for jointly training an isolation forest model; Obtain information about technical fields that multiple users are in common, and confirm the technical field information with the users; Recalling the stored historical isolation forest model associated with the confirmed technical field information; Identify the historical isolated forest models needed; The historical isolation forest model is processed by removing the data retention framework to obtain the initial isolation forest model.

3. The anomaly detection method based on federated learning and isolation forest algorithm according to claim 2, characterized in that: The private data package includes the following contents: The private data that needs to be jointly trained and the joint information of the private data confirmed by the user.

4. The anomaly detection method based on federated learning and isolation forest algorithm according to claim 3, characterized in that: The step of combining multiple associated private data packets to obtain a joint data packet comprises the following steps: Get the joint information in the private data packet; Find the private data that needs to be joined based on the joint information; Private data that is not involved in the federation is marked and fed back to the corresponding users.

5. The anomaly detection method based on federated learning and isolation forest algorithm according to claim 4, characterized in that: The method of retraining the corresponding first isolation forest model by combining the data packets to obtain the second isolation forest model comprises the following steps: According to the association relationship between the private data and the user, query the first isolation forest model sent by the user; According to the association relationship between the private data and the joint data package, the joint data package is matched with the first isolation forest model; The server trains the first isolation forest model again by combining the data packets to obtain a second isolation forest model.

6. The anomaly detection method based on federated learning and isolation forest algorithm according to claim 4, characterized in that: The method further comprises the steps of: If the feedback received from the server by the user only contains the second isolation forest model, the second training will not be initiated; If the feedback received from the server by the user includes the second isolation forest model and the marked private data not involved in the joint training, the user will negotiate with other users to revise the private data packet and initiate a joint training request to the server again.

7. The anomaly detection method based on federated learning and isolation forest algorithm according to claim 1, characterized in that: The technical field information includes the technical field of financial fraud detection.

8. An anomaly detection system based on federated learning and isolation forest algorithm, according to any one of claims 1-7, characterized in that: The system includes a federated learning server and a plurality of user-end devices; The federated learning server is used to publish an initial isolation forest model to multiple users participating in the federated learning; receive the trained first isolation forest model fed back by multiple user-end devices, and a private data packet associated with other user data; obtain the first isolation forest model fed back by the corresponding other user-end devices and the private data packet fed back by the corresponding other user-end devices based on the associated information in the private data packet; combine multiple associated private data packets to obtain a joint data packet, retrain each first isolation forest model through the joint data packet to obtain a second isolation forest model; feed back the second isolation forest model to the corresponding user-end device, and the user-end device performs abnormal data detection based on the second isolation forest model.

9. The anomaly detection system based on federated learning and isolation forest algorithm according to claim 8, characterized in that: Publishing the initial isolation forest model to multiple users participating in federated learning through the federated learning server includes the following steps: The server receives requests from multiple users for jointly training an isolation forest model; Obtain information about technical fields that multiple users are in common, and confirm the technical field information with the users; Recalling the stored historical isolation forest model associated with the confirmed technical field information; Identify the historical isolated forest models needed; The historical isolation forest model is processed by removing the data retention framework to obtain the initial isolation forest model; The private data package includes: Private data confirmed by the user that needs to be jointly trained and joint information of the private data; Combining multiple associated private data packets to obtain a joint data packet includes the following steps: Get the joint information in the private data packet; Find the private data that needs to be joined based on the joint information; Private data that is not involved in the federation is marked and fed back to the corresponding users.

10. The anomaly detection system based on federated learning and isolation forest algorithm according to claim 9, characterized in that: Retraining the corresponding first isolation forest model by combining the data packets to obtain the second isolation forest model includes the following steps: According to the association relationship between the private data and the user, query the first isolation forest model sent by the user; According to the association relationship between the private data and the joint data package, the joint data package is matched with the first isolation forest model; The server trains the first isolation forest model again by combining the data packets to obtain a second isolation forest model; If the feedback received from the server by the user only contains the second isolation forest model, the second training will not be initiated; If the feedback received from the server by the user includes the second isolation forest model and the marked private data not involved in the joint training, the user will negotiate with other users to revise the private data packet and initiate a joint training request to the server again.

Images