Integrated model generation method, integrated model generation apparatus, and integrated model generation program

Abstract

In an integrated model generation device (10), an evaluation unit (15b) evaluates each model trained by each participant by using a predetermined evaluation index. A calculation unit (15c) calculates a weight according to the evaluation index for each model. An integrating unit (15d) integrates each model in accordance with the calculated weight.

Description

Technical Field

[0001] The present invention relates to an integrated model generation method, an integrated model generation device, and an integrated model generation program.

Background Art

[0002] Conventionally, in a system for detecting a Command & Control server (hereinafter, malicious server), which is a commander in a large-scale attack infrastructure, using only traffic data collected by a single telecommunications carrier has limitations in the number of malicious servers detected worldwide. On the other hand, it is difficult from the perspective of the privacy of each telecommunications carrier to have multiple telecommunications carriers cooperate to exchange data in order to improve the number of detected malicious servers. Therefore, there is known a technique called Federated Learning in which, without exchanging data, participating telecommunications carriers (hereinafter, participants) exchange and integrate only the prediction models generated by each of them to generate a model with better performance (see Non-Patent Documents 1 and 2).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when integrating the models of each participant using conventional technologies, it can be difficult to improve the accuracy of detecting malicious servers. For example, the amount of data related to malicious servers contained in actual traffic data (hereinafter referred to as malicious data) is overwhelmingly small compared to other benign data, and there are large differences in the performance of the predictive models that can be generated depending on the scale of each participant. Therefore, when integrating models in Federated Learning, poorly performing models can have a negative impact, making it difficult to detect malicious data without causing false positives or missed detections, which is particularly important in the field of cybersecurity.

[0005] The present invention has been made in view of the above, and aims to suppress the influence of poorly performing models when integrating the models of each participant, thereby enabling highly accurate detection of data related to small amounts of malicious servers. [Means for solving the problem]

[0006] To solve the above-mentioned problems and achieve the objective, the integrated model generation method according to the present invention is an integrated model generation method executed by an integrated model generation device, characterized in that it includes: an evaluation step of evaluating each model learned by each participant using a predetermined evaluation index; a calculation step of calculating weights for each model according to the evaluation index; and an integration step of integrating each model according to the calculated weights. [Effects of the Invention]

[0007] According to the present invention, when integrating the models of each participant, it is possible to suppress the influence of poorly performing models and detect small amounts of data related to malicious servers with high accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a diagram illustrating the overview of the integrated model generation system. [Figure 2] Figure 2 is a diagram illustrating the overview of the integrated model generation system. [Figure 3] Figure 3 is a diagram illustrating the overview of the integrated model generation device. [Figure 4] Figure 4 is a schematic diagram illustrating the general configuration of an integrated model generation device. [Figure 5] Figure 5 is a flowchart showing the integrated model generation process. [Figure 6] Figure 6 is a diagram illustrating an integrated model generation apparatus of another embodiment. [Figure 7] Figure 7 illustrates a computer running an integrated model generation program. [Modes for carrying out the invention]

[0009] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to this embodiment. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0010] [Overview of the Integrated Model Generation System] Figures 1 to 3 illustrate the overview of the integrated model generation device. First, as illustrated in Figure 1, each telecommunications carrier participating in the generation of the integrated model (hereinafter referred to as "participant") prepares its own traffic dataset for training. For example, each participant collects network flow data from network equipment on the network they manage using a flow collection function, and extracts the features of each host from the collected data. Then, based on the public list of malicious hosts and past detection results, they assign either a benign or malicious label to the feature data of each host to create the traffic dataset for training.

[0011] Participants will share the types of features and the form of the model to be trained in advance. Then, each participant will be given an initial model with a predetermined form and randomized parameters. Each participant will use their own traffic dataset to train their initial model and determine the parameters so that it outputs a predicted probability of whether a host is benign or malignant for the input host features. The integrated model generator will then combine the models trained by each participant to generate an integrated model.

[0012] Here, Figure 2 illustrates an integrated model generated by conventional Federated Learning. Figure 3 also illustrates an integrated model generated by the integrated model generation device of this embodiment. In conventional Federated Learning, as illustrated in Figure 2, the parameters of the integrated model were simply the average of the parameters of the models learned by each participant. In the example shown in Figure 2, the average of the parameters of the three models m1, m2, and m3 is used as the parameter w11 of the integrated model.

[0013] In contrast, the integrated model generation device of this embodiment, as illustrated in Figure 3, uses the predicted probability of each model as an evaluation metric and determines the parameters of the integrated model by weighting according to the evaluation metric. In the example shown in Figure 3, the parameter w11 is determined according to the average of the predicted probabilities of each model m1, m2, and m3.

[0014] Specifically, the integrated model generator first extracts features from a public list of malicious servers or from malicious servers previously detected collaboratively by participants, and prepares a malicious dataset for evaluation. Then, the integrated model generator evaluates each model using this malicious dataset and calculates the parameter weights for the integrated model. For example, the integrated model generator determines the weights of each model using a weight determination function based on the average of the predicted probabilities of the malicious data for each model. The weights of each model represent the degree to which high-performing models are emphasized and the degree to which low-performing models are emphasized, and can be adjusted using the parameters of the weight determination function.

[0015] The integrated model generation device generates an integrated model by combining each model based on the determined weights. The integrated model generation device generates an integrated model by repeating the above process a predetermined number of times with the integrated model generated here as the initial model for the next round.

[0016] As described above, according to the integrated model generation device of the present embodiment, it is possible to suppress the influence of models with poor performance and integrate the models of each participant. Therefore, it is possible to accurately detect data related to a small number of malicious servers.

[0017] [Configuration of Integrated Model Generation Device] FIG. 4 is a schematic diagram illustrating a schematic configuration of the integrated model generation device. As illustrated in FIG. 4, the integrated model generation device 10 is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.

[0018] The input unit 11 is realized using an input device such as a keyboard or a mouse, and inputs various instruction information such as a processing start to the control unit 15 in response to an input operation by an operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, or the like.

[0019] The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between an external device such as a server via a network and the control unit 15. For example, the communication control unit 13 controls communication between a server or the like that manages models and traffic data of each participant used in the integrated model generation process described later and the control unit 15.

[0020] The memory unit 14 is implemented using semiconductor memory elements such as RAM (Random Access Memory) or flash memory, or storage devices such as hard disks or optical discs. The memory unit 14 pre-stores processing programs for operating the integrated model generation device 10, as well as data used during the execution of the processing programs, or temporarily stores them each time processing is performed. The memory unit 14 may also be configured to communicate with the control unit 15 via the communication control unit 13.

[0021] In this embodiment, the storage unit 14 stores a malicious dataset 14a used in the integrated model generation process described later. The malicious dataset 14a consists of data relating to each known malicious server, and is a predetermined feature of each malicious server. Here, a known malicious server means a malicious server on a publicly available list, or a malicious server that participants have previously detected in cooperation with each other.

[0022] The control unit 15 is implemented using a CPU (Central Processing Unit) or the like, and executes a processing program stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, an evaluation unit 15b, a calculation unit 15c, an integration unit 15d, and a distribution unit 15e, as illustrated in Figure 4, and executes the integrated model generation process described later. Note that these functional units may each be implemented on different hardware. For example, the acquisition unit 15a and the distribution unit 15e may be implemented on a separate device from the other functional units. Furthermore, the control unit 15 may also include other functional units.

[0023] The collection unit 15a collects the models learned by each participant. Specifically, the collection unit 15a collects the models learned by each participant via the input unit 11 or via the communication control unit 13 from each participant's server.

[0024] Furthermore, as will be described later, the collection unit 15a repeatedly collects the models learned by each participant and transfers them to the evaluation unit 15b. On the first attempt, as illustrated in Figure 1, an initial model in a predetermined form with randomized parameters is distributed to each participant, and the models learned by each participant using their own traffic dataset are collected. From the second attempt onward, an integrated model generated by the integration unit 15d (described later) is distributed to each participant, and, as with the first attempt, the models learned by each participant using their own traffic dataset are collected.

[0025] The collection unit 15a may immediately transfer the collected models of each participant to the evaluation unit 15b, or it may collect them in advance prior to the processing described later and store them in the storage unit 14.

[0026] Returning to the explanation of Figure 4, the evaluation unit 15b evaluates each model learned by each participant using predetermined evaluation metrics. For example, the evaluation unit 15b calculates the predicted probability of malignancy using the malignancy dataset 14a, which is information on known malicious servers, as an evaluation metric. In the example shown in Figure 1, the average predicted probabilities of malignancy p1, p2, and p3 are calculated.

[0027] Returning to the explanation of Figure 4, the calculation unit 15c calculates weights for each model according to the evaluation index. For example, the calculation unit 15c calculates weights such that they increase as the predicted probability of malignancy increases. Specifically, the calculation unit 15c uses the weight determination function f(p) shown in equation (1) or equation (2) below. i The weights for each model are calculated using ).

[0028]

number

number

[0029] Equation (2) above is called the temperature-sensitive softmax function, and it allows for a greater influence of the parameter T than equation (1). For example, when T < 1, the closer T approaches 0, the more weighted and emphasized the models of participants with a high probability of malignancy. Also, when T > 1, the larger T is, the more easily models of participants with a low probability of malignancy are emphasized.

[0030] The integration unit 15d integrates each model according to the calculated weights. For example, as illustrated in Figure 1, f(p i By using this method to weight and integrate each model, an integrated model is generated. As a result, the integrated model generator 10 generates an integrated model that is weighted to emphasize models with high accuracy in predicting maliciousness and good performance. Therefore, the integrated model generator 10 can suppress the influence of poorly performing participant models and generate an integrated model that can detect malicious servers with high accuracy.

[0031] Returning to the explanation of Figure 4, the distribution unit 15e distributes the integrated model to each participant. For example, as illustrated in Figure 1, the distribution unit 15e distributes the integrated model to each participant as the initial model for the next round. In this way, the integrated model generation device 10 can improve the accuracy of the integrated model by repeating the processing of the collection unit 15a to the distribution unit 15e a predetermined number of times.

[0032] [Integrated Model Generation Process] Next, with reference to Figure 5, the integrated model generation process by the integrated model generation device 10 according to this embodiment will be described. Figure 5 is a flowchart of the integrated model generation process procedure. The flowchart in Figure 5 starts, for example, when an operation input instructing the start of the integrated model generation process is received.

[0033] First, the collection unit 15a collects the trained models that each participant has trained using their own traffic dataset from each participant's server, etc. (Step S1).

[0034] Next, the evaluation unit 15b calculates predetermined evaluation metrics for each model learned by each participant (step S2). For example, the evaluation unit 15b calculates the predicted probability of malignancy using the malignancy dataset 14a, which is information on known malicious servers, as an evaluation metric.

[0035] Furthermore, the calculation unit 15c calculates weights for each model according to the evaluation index (step S3). For example, the calculation unit 15c uses a weight determination function to calculate weights for each model such that the higher the predicted probability of malignancy, the more emphasis is placed on it.

[0036] Then, the integration unit 15d integrates each model according to the calculated weights (step S4). For example, the integration unit 15d generates an integrated model that is weighted to emphasize the higher the accuracy of the malignancy prediction and the better the performance.

[0037] Furthermore, the distribution unit 15e distributes the integrated model to each participant (step S5) and returns the process to step S1. This repeats the series of integrated model generation processes.

[0038] [Other embodiments] Figure 6 is a diagram illustrating an integrated model generation device of another embodiment. In the above embodiment, as shown in Figure 6(a), an integrated model is generated with each of the multiple telecommunications carriers as a participant. In this case, it is possible to increase the number of malicious servers detected by coordinating with other telecommunications carriers.

[0039] However, the present invention is not limited thereto. For example, as illustrated in Figure 6(b), an integrated model may be generated with multiple regions as participants, targeting edge computing within a single telecommunications carrier. Global telecommunications carriers need to aggregate traffic data from various locations to one place, and the overhead for aggregation may increase significantly with future increases in traffic volume. Furthermore, even within the same organization, regional legal regulations may prevent traffic data from being directly transferred outside the region, making it difficult to aggregate it to one place. In such cases, a single large-scale telecommunications carrier can generate an integrated model with each region as a participant, enabling highly accurate detection of malicious servers.

[0040] [effect] As explained above, in the integrated model generation device 10, the evaluation unit 15b evaluates each model learned by each participant using predetermined evaluation indicators. The calculation unit 15c calculates weights for each model according to the evaluation indicators. The integration unit 15d integrates each model according to the calculated weights.

[0041] Specifically, the evaluation unit 15b calculates the predicted probability of malignancy using information on known malicious servers as an evaluation index. In this case, the calculation unit 15c calculates weights such that the higher the predicted probability of malignancy, the larger the weight.

[0042] Thus, the integrated model generation device 10 of this embodiment makes it possible to integrate models from various telecommunications carriers by, for example, emphasizing models with a high probability of predicting maliciousness and good performance in detecting malicious servers, while suppressing the influence of models with poor performance. Therefore, it becomes possible to detect small amounts of data on malicious servers with high accuracy.

[0043] Furthermore, the distribution unit 15e distributes the integrated model to each participant. This allows each participant to train their own distributed integrated model, and by repeatedly integrating the trained models, the accuracy of detecting malicious servers can be further improved.

[0044] [program] It is also possible to create a program that describes the processing performed by the integrated model generation device 10 according to the above embodiment in a language that can be executed by a computer. In one embodiment, the integrated model generation device 10 can be implemented by installing an integrated model generation program that performs the above integrated model generation processing as packaged software or online software on a desired computer. For example, by having an information processing device execute the above integrated model generation program, the information processing device can be made to function as the integrated model generation device 10. In addition, the information processing device includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDA (Personal Digital Assistant). Furthermore, the functions of the integrated model generation device 10 may be implemented on a cloud server.

[0045] Figure 7 shows an example of a computer running an integrated model generation program. Computer 1000 includes, for example, memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070. These components are connected by bus 1080.

[0046] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1031. The disk drive interface 1040 is connected to the disk drive 1041. A removable storage medium, such as a magnetic disk or optical disk, is inserted into the disk drive 1041. A serial port interface 1050 is connected to, for example, a mouse 1051 and a keyboard 1052. A video adapter 1060 is connected to, for example, a display 1061.

[0047] Here, the hard disk drive 1031 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. The information described in the above embodiment is stored, for example, in the hard disk drive 1031 or the memory 1010.

[0048] Furthermore, the integrated model generation program is stored in the hard disk drive 1031 as a program module 1093 containing instructions to be executed by the computer 1000, for example. Specifically, the program module 1093 containing instructions for each process executed by the integrated model generation device 10 described in the above embodiment is stored in the hard disk drive 1031.

[0049] Furthermore, the data used for information processing by the integrated model generation program is stored as program data 1094, for example, in the hard disk drive 1031. The CPU 1020 then reads the program module 1093 and program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as needed and executes the procedures described above.

[0050] Furthermore, the program module 1093 and program data 1094 related to the integrated model generation program are not limited to being stored on the hard disk drive 1031; for example, they may be stored on a removable storage medium and read by the CPU 1020 via a disk drive 1041 or the like. Alternatively, the program module 1093 and program data 1094 related to the integrated model generation program may be stored on another computer connected via a network such as a LAN (Local Area Network) or WAN (Wide Area Network) and read by the CPU 1020 via a network interface 1070.

[0051] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in this embodiment. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are included in the scope of the present invention. [Explanation of Symbols]

[0052] 10. Integrated Model Generation Device 11 Input section 12 Output section 13 Communication Control Unit 14 Storage section 14a Malignant dataset 15 Control Unit 15a Collection Department 15b Evaluation Department 15c Calculation part 15d Integration Department 15e Distribution Department

Claims

1. An integrated model generation method performed by an integrated model generation device, The evaluation process involves evaluating each model learned by each participant using predetermined evaluation metrics. For each of the aforementioned models, a calculation process is performed to calculate weights corresponding to the evaluation indicators, An integration step in which each of the models is integrated according to the calculated weights, Includes, The evaluation step involves, for each model that outputs a prediction probability of whether a server is benign or malicious based on the input server features, calculating the prediction probability when the features of a known malicious server are input to the model. The calculation step involves calculating the weights such that they increase as the predicted probability of malignancy increases. A method for generating an integrated model characterized by the following features.

2. The method for generating an integrated model according to claim 1, further comprising a distribution step of distributing the integrated model to each of the aforementioned participants.

3. An evaluation unit evaluates each model learned by each participant using predetermined evaluation metrics, For each of the aforementioned models, a calculation unit calculates a weight corresponding to the evaluation index, An integration unit that integrates each of the models according to the calculated weights, It has, The evaluation unit calculates the predicted probability for each model that outputs a predicted probability of whether a server is benign or malicious based on the input server features, when the features of a known malicious server are input to the model. The calculation unit calculates the weights such that they increase as the predicted probability of malignancy increases. An integrated model generation device characterized by the following features.

4. An evaluation step in which each model learned by each participant is evaluated using predetermined evaluation metrics, For each of the aforementioned models, a calculation step is performed to calculate weights according to the evaluation indicators, A merging step in which each of the models is integrated according to the calculated weights, Have the computer run it, The evaluation step involves, for each model that outputs a prediction probability of whether a server is benign or malicious based on the input server features, calculating the prediction probability when the features of a known malicious server are input to the model. The calculation step is an integrated model generation program that calculates the weights such that they increase as the predicted probability of malignancy increases.

Images