Determination device, determination method, and determination program

Abstract

A determination device according to the present invention obtains discrimination results regarding whether or not the input data indicates malignancy, using a discriminatory model for discrimination of whether or not input data indicates malignancy. Also, the determination device uses an explanatory model that outputs grounds for the discrimination by the discriminatory model to obtain grounds for the discrimination of the input data by the discriminatory model (explanation results). Further, the determination device takes the results of the discrimination and the explanation results regarding the discrimination as input, and obtains a probability of the results of the discrimination by the discriminatory model being erroneous, using an objection judgment model that outputs the probability that the results of the discrimination are erroneous. The determination device then extracts data that a user is to verify, on the basis of height of the probability that the results of the discrimination are erroneous.

Description

[Technical field]

[0001] The present invention relates to a determination device, a determination method, and a determination program. [Background technology]

[0002] Conventionally, there are technologies that use machine learning models to detect malicious activities in cyberspace. However, detection by models is not perfect, and there is a possibility of false positives or overlooking malicious activities. For this reason, in practical situations, it is essential that humans verify the detection results of the models.

[0003] One way to assist with the above human verification is to use Explainable AI (XAI), which outputs the reasons for the model's detection results. By having humans check the output of this XAI, the detection results of the above model can be verified. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] SCSundaramurthy, et al., A Human Capital Model for Mitigating Security Analyst Burnout, Proc. SOUPS, 2015. [Non-Patent Document 2] Ponemon Institute, Improving the Effectiveness of the Security Operations Center, 2019. [Non-Patent Document 3] D. Chiba, et al., DomainProfiler: Discovering Domain Names Abused in Future, in Proc. IEEE / IFIP DSN, 2016. [Non-Patent Document 4] N. Fukushi, et al., Exploration into Gray Area: Toward Efficient Labeling for Detecting Malicious Domain Names, IEICE Trans. Communications, 2020. [Non-Patent Document 5] Adadi, et al., Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI), IEEE Access, 2018. [Non-Patent Document 6] Ministry of Internal Affairs and Communications, AI Network Society Promotion Council Report 2018, [Retrieved September 22, 2021], Internet<URL:http: / / www.soumu.go.jp / menu_news / s-news / 01iicp01_02000072.html> [Non-Patent Document 7] Ribeiro, et al., “Why Should I Trust You?”Explaining the Predictions of Any Classifier, Proc. ACM KDD, 2016. [Non-Patent Document 8] Lundberg, et al., A Unified Approach to Interpreting Model Predictions, Proc. NIPS, 2017. [Non-Patent Document 9] I.Lage, et al., Human-in-the-Loop Interpretability Prior, Proc. NeurIPS, 2018. Summary of the Invention [Problem to be solved by the invention]

[0005] However, when the number of detection results to be verified is huge, it is difficult for humans to check all of the grounds for the detection results (the output results of XAI). In addition, interpreting the output results of XAI itself can be difficult. Therefore, the objective of the present invention is to reduce the effort required for humans to verify the detection results of a model. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present invention is characterized in that it comprises an identification unit that identifies whether input data is malignant or not using an identification model that identifies whether the input data is malignant or not, a basis output unit that outputs a basis for identification of input data by the identification model using an explanatory model that outputs a basis for identification by the identification model, and a judgment unit that outputs the probability that the result of the identification is incorrect by inputting the result of the identification and the basis for the identification into an objection judgment model that receives as input the result of the identification and the basis for the identification, and outputs the probability that the result of the identification is incorrect. Effect of the Invention

[0007] According to the present invention, it is possible to reduce the effort required by humans to verify the detection results of a model. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining an outline of the determination device. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of the determination device. [Diagram 3] FIG. 3 is a diagram showing an example of a training data set for the objection determination model. [Figure 4] FIG. 4 is a flowchart showing an example of a learning procedure of the objection determination model in the first embodiment. [Diagram 5] FIG. 5 is a flowchart showing an example of a determination process using the objection determination model in the first embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of a learning procedure for the objection determination model in the second embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the configuration of a computer that executes a determination program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a first embodiment and a second embodiment of the present invention will be described with reference to the drawings, but the present invention is not limited to each of the embodiments.

[0010] [First embodiment] First, an overview of a determination device according to a first embodiment will be described with reference to Fig. 1. The determination device includes an identification model, an explanatory model, and an objection determination model.

[0011] A discriminative model is a model that outputs a discrimination result as to whether or not input data is malignant. For example, a discriminative model outputs the probability that input data is malignant as a discrimination result.

[0012] The explanatory model is a model that outputs the basis for classification by the discriminative model. For example, when the explanatory model receives a discriminative model and input data to the discriminative model, it outputs the basis for classification by the discriminative model (explanation result). The explanation result is, for example, information indicating the importance of each feature used by the discriminative model.

[0013] The objection judgment model is a model that takes as input the classification result of input data by the classification model and the explanation result by the explanation model, and outputs a value indicating the plausibility of the classification result of the input data (e.g., the probability that the classification result is incorrect) as the judgment result.

[0014] This objection determination model is trained based on the identification result from the identification model, the explanation result from the explanation model, and a label (label indicating a true value) indicating whether the identification result is correct or not.

[0015] A user of the judgment device (for example, an analyst) can know the plausibility of the judgment result of the input data by the discriminative model by checking the judgment result output by the objection judgment model.

[0016] This allows the user of the determination device to know, for example, which of the identification results of the input data is most likely to be incorrect. Therefore, for example, the user can know which identification result should be given priority in verification, and can efficiently verify the identification results. For example, the user can reduce the effort required for verification by narrowing down the identification results to those that are most likely to be incorrect.

[0017] [composition] Next, a configuration example of the determination device 10 will be described with reference to Fig. 2. The determination device 10 includes an input / output unit 11, a storage unit 12, and a control unit 13.

[0018] The input / output unit 11 is an input / output interface that receives input of various data used by the control unit 13 and outputs the results of processing by the control unit 13 .

[0019] The storage unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. The storage unit 12 may be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM) or a flash memory. The storage unit 12 stores an operating system (OS) and various programs executed by the determination device 10. Furthermore, the storage unit 12 stores various information used in the execution of the programs.

[0020] For example, the storage unit 12 stores a discriminative model, an explanatory model, a data set used to train the discriminative model, etc. In addition, when the control unit 13 trains an objection determination model, the storage unit 12 stores the objection determination model.

[0021] As described above, the discrimination model is a model that outputs a discrimination result as to whether input data is malicious or not (for example, a probability value that the input data is malicious). This discrimination model is realized, for example, by DomainProfiler (a model that performs binary classification of domain names as malicious / benign by machine learning; see Non-Patent Document 3).

[0022] The explanatory model is a model that receives input data and a discriminative model, and outputs the basis for discriminating the input data by the discriminative model (explanation result). This explanatory model is realized, for example, by LIME (see Non-Patent Document 7), SHAP (see Non-Patent Document 8), etc.

[0023] The dataset is a dataset used to train a discrimination model. This dataset is, for example, data (feature amounts) that correspond to the true discrimination labels (malignant / benign) of the data.

[0024] The control unit 13 controls the entire determination device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 13 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory. The control unit 13 also functions as various processing units by running various programs.

[0025] The control unit 13 includes a learning unit 131, a data receiving unit 132, a discrimination unit 133, a reason output unit 134, a judgment unit 135, an extraction unit 136, and a data output unit 137. A tendency analysis unit 138 and a learning data selection unit 139 may or may not be provided, and the case where they are provided will be described later.

[0026] The learning unit 131 learns the objection determination model by using a learning dataset including, for example, a result of input data classification output by a discrimination model, an explanation result of the classification output by an explanation model, and a label indicating whether the result of the classification is correct or not.

[0027] An example of learning of the objection adjudication model by the learning unit 131 will be described with reference to Fig. 3. First, a procedure for creating a learning dataset Q for the objection adjudication model by the learning unit 131 will be described.

[0028] First, the learning unit 131 learns a data set P={(x i ,y i *)}, i = 1, 2, . . . , n. This dataset P is, for example, an m-dimensional feature vector x i =(x i,1 ,x i,2 ,x i,m ) and its true identification label y i *∈{malignant, benign} is a set of n samples.

[0029] The learning unit 131 generates a learning data set Q by i is input to the discriminant model f(·) and explanatory model g(·), and the discriminant result y 1,i =f(x i )∈{malignant, benign} and explanation result y 2,i =g(x i ,f(·)) is obtained. 2,i is the feature vector x i The result of this explanation is y 2,i Each element of, for example, the feature vector x i The feature x corresponding to each element of i,j This is the importance of the feature x i,j The importance of may be a value calculated using any XAI.

[0030] Next, the learning unit 131 calculates the classification result y 1,i The true identification label y i*Compare with objection label y 3,i *=I(y 1,i ≠y i 3 is an indicator function that is 1 (objection) if the value in the parentheses is true and 0 (no objection) if the value is false. That is, the learning unit 131 generates the classification result y 1,i and the true identification label y i If the * does not match, the label "Dispute (classification result is incorrect)" is generated; if they do match, the label "No Dispute (classification result is correct)" is generated.

[0031] Through the above procedure, the learning unit 131 creates a learning dataset Q={(y 1,i ,y 2,i ),y 3,i *}, i = 1, 2, · · · , n.

[0032] The learning unit 131 uses a learning dataset Q to calculate the feature quantity (y 1,i ,y 2,i ) the output of the objection judgment model h(y 1,i ,y 2,i ) is the objection label y 3,i A challenge adjudication model is trained to match *. Any supervised machine learning algorithm can be used to train this challenge adjudication model.

[0033] In order to improve the judgment accuracy of the objection judgment model, the learning unit 131 may learn the objection judgment model using samples that are highly likely to have an objection judgment (that is, samples that are highly likely to have a false positive or oversight in the classification result) among the samples included in the learning dataset Q. An embodiment in this case will be described in the second embodiment.

[0034] Returning to the explanation of Fig. 2, the data accepting unit 132 accepts input of data to be classified by the discriminative model (input data). The discriminative unit 133 uses the discriminative model to classify whether the input data is malicious or not. The basis output unit 134 uses the explanatory model to output the basis (explanation result) of the classification of the input data by the discriminative model.

[0035] The judgment unit 135 inputs the result of the classification output by the classification model and the grounds for the classification output by the explanatory model to the trained objection judgment model, and outputs a value indicating the plausibility of the result of the classification (a judgment result). The judgment result is, for example, the probability that the result of the classification is incorrect.

[0036] The extraction unit 136 extracts data to be verified by the user of the determination device 10 based on the determination result by the determination unit 135. For example, the extraction unit 136 extracts data (e.g., the result of classification, the input data that was the subject of the classification, and the explanation result of the classification) up to a predetermined rank in descending order of the probability of being incorrect from the storage unit 12. The extraction unit 136 may also extract data from the storage unit 12 whose probability of being incorrect is equal to or greater than a predetermined threshold. The data output unit 137 outputs the data extracted by the extraction unit 136 via the input / output unit 11.

[0037] [Example of processing procedure] Next, an example of a processing procedure of the determination device 10 will be described with reference to Fig. 4 and Fig. 5. First, a procedure in which the determination device 10 learns an objection determination model will be described with reference to Fig. 4.

[0038] The data accepting unit 132 of the determination device 10 accepts input data (data set) (S1). Next, the identification unit 133 uses an identification model to identify whether the input data accepted in S1 is malicious or not (S2). In addition, the basis output unit 134 uses an explanation model to output a basis (explanation result) for the result of the identification in S2 (identification result) (S3).

[0039] The learning unit 131 creates a training dataset for the objection determination model using the input data classification result obtained by the process of S2, the explanation result for the classification obtained by the process of S3, and a label indicating whether the classification result is correct or not (S4).Then, the learning unit 131 uses the training dataset created in S4 to train the objection determination model (S5).

[0040] Next, an example of a judgment process using the objection judgment model will be described with reference to Fig. 5. The data receiving unit 132 of the judgment device 10 receives input data to be judged (S11). Next, the identification unit 133 uses the identification model to identify whether the input data received in S11 is malicious or not (S12). After that, the basis output unit 134 uses the explanation model to output the basis (explanation result) of the result of the classification in S12 (classification result) (S13).

[0041] Thereafter, the judgment unit 135 judges whether or not the result of the classification of the input data by the classification model is incorrect using the objection judgment model. For example, the judgment unit 135 inputs the result of the classification of the input data in S12 and the explanation result in S13 to the objection judgment model, and outputs the probability that the result of the classification of the input data is incorrect (judgment result) (S14).

[0042] Then, the extraction unit 136 preferentially extracts data with a high probability of the classification result being incorrect (S15). For example, the extraction unit 136 extracts input data, classification results, and explanation results for the classification, whose probability of the classification result being incorrect is up to the above-mentioned predetermined rank. After that, the data output unit 137 outputs the data extracted in S15 (S16).

[0043] Through the above process, the determination device 10 can output data (= data that should be verified by humans) that is highly likely to be an erroneous result of classification by a classification model. As a result, humans (e.g., analysts) can efficiently verify the results of classification. In addition, the effort required by humans to verify the results of classification by a classification model can be reduced.

[0044] [Second embodiment] As described above, in order to improve the judgment accuracy of the objection judgment model, the learning unit 131 of the judgment device 10 may learn the objection judgment model using samples that are likely to have an erroneous classification result (i.e., are likely to have a false positive or oversight) among the samples included in the learning dataset Q. An embodiment in this case will be described as the second embodiment. The same components as those in the first embodiment are denoted by the same reference numerals and description thereof will be omitted.

[0045] The determination device 10 of the second embodiment further includes a tendency analysis unit 138 and a learning data selection unit 139 shown in Fig. 2. The tendency analysis unit 138 analyzes whether the discrimination model has a tendency to discriminate malignant input data as not malignant (i.e., a tendency to overlook malignancy) or to discriminate non-malignant input data as malignant (i.e., a tendency to make false positives).

[0046] For example, the trend analysis unit 138 compares the classification results obtained by inputting the data set used to train the classification model into the classification model with the correct label indicating whether each piece of data in the data set is malignant or not, thereby analyzing which of the above tendencies the classification model has.

[0047] The learning data selection unit 139 selects the range of data to be used for learning the objection determination model, based on the analysis result of the trend of the discriminative model by the trend analysis unit 138.

[0048] For example, if the trend analysis unit 138 analyzes that the discrimination model "tends to overlook malignancy," the training data selection unit 139 selects data that the discrimination model has identified as not malignant despite being malignant, as data to be used in training the objection determination model.

[0049] On the other hand, if the trend analysis unit 138 analyzes that the discrimination model "tends to have a high number of false positives," the training data selection unit 139 selects data that has been identified as malignant by the discrimination model despite not being malignant as data to be used in training the objection determination model.

[0050] By the determination device 10 selecting data to be used for learning the objection determination model in the above-described manner, the objection determination model can be learned efficiently.

[0051] An example of a learning procedure of the objection determination model in the determination device 10 of the second embodiment will be described with reference to Fig. 6. The tendency analysis unit 138 of the determination device 10 inputs the data set used for learning the discrimination model to the discrimination model and acquires a discrimination result (S21). After that, the tendency analysis unit 138 analyzes the tendency of the discrimination model to make mistakes (high tendency to overlook malignancy / high tendency to make false positives) based on the difference between the discrimination result acquired in S21 and the correct answer label included in the above data set (S22).

[0052] In S22, when the tendency analysis unit 138 analyzes that the discrimination model has a strong tendency to overlook malignancy (S23 -> strong tendency to overlook malignancy), the learning data selection unit 139 selects a non-malignant prediction region and a region with a low prediction probability as a region to be trained by the objection determination model (S24). For example, the learning data selection unit 139 selects, from among the samples included in the training dataset Q, samples whose probability of being classified as malignant is a predetermined value or less (e.g., 0.1 or more and 0.5 or less) as samples to be trained by the objection determination model.

[0053] On the other hand, if the trend analysis unit 138 has analyzed in S22 that the discrimination model has a strong tendency for false positives (S23 -> strong tendency for false positives), the learning data selection unit 139 selects, from the learning dataset, a region predicted as malignant and having a low prediction probability, as a region from which the objection determination model should learn (S25). For example, the learning data selection unit 139 selects, from the samples included in the learning dataset Q, samples whose probability of being classified as malignant is equal to or lower than a predetermined value (for example, 0.5 or more and 0.9 or less), as samples from which the objection determination model should learn. Then, the process proceeds to S26.

[0054] The learning unit 131 uses the data in the area selected by the learning data selection unit 139 in S24 or S25 to learn the objection adjudication model (S26).

[0055] In this way, the determination device 10 can efficiently learn the objection determination model based on the characteristics of the discriminative model.

[0056] [others] The determination device 10 may output data for which the discrimination result by the discriminative model is suspicious (=data to be verified by a human), and after the human verifies the data, the determination device 10 may re-learn the discriminative model using the verification result by the human. In this way, the determination device 10 can improve the discrimination accuracy of the discriminative model. As a result, the effort of a human verifying the discrimination result of the discriminative model can be further reduced.

[0057] [Experimental Results] Next, the evaluation result of the objection determination model trained by the determination device 10 described in the second embodiment will be shown below.

[0058] Here, we explain the evaluation results when the discrimination model to be judged by the objection judgment model is a model trained with the DomainProfiler dataset. From this DomainProfiler dataset, 80 samples that were discriminated as malicious domain names by the discrimination model were used as test data for the objection judgment model. All of these 80 samples were discriminated as malicious domain names by the discrimination model, but among them, 12 were actually benign domain names.

[0059] The effectiveness of the objection judgment model was evaluated using the above test data. For samples with a confidence level of 0.5 or more at which the objection judgment model judged the classification result of the classification model to be incorrect (objection), the objection judgment model judged the sample as "objection". For samples with a confidence level of less than 0.5, the objection judgment model judged the sample as "no objection".

[0060] Here, when a human analyst verified 31 samples that were determined to be "objective" by the objection determination model, 11 of them were correct. In other words, the correct answer rate for "objective" was 0.35 (= 11 / 31). In addition, when a human analyst verified 49 samples that were determined to be "no objection" by the objection determination model, 3 of them were incorrect. In other words, the incorrect answer rate for "no objection" was 0.06 (= 3 / 49).

[0061] We will also explain the evaluation results when the classification model to be judged by the objection judgment model is a model trained on the EMBER (Endgame Malware BEnchmark for Research) dataset. From this EMBER dataset, 81 samples that were classified as benign PE files (benign portable executable files) by the classification model were used as test data for the objection judgment model. All of these 81 samples were classified as benign PE files by the classification model, but 18 of them were actually malicious PE files.

[0062] The effectiveness of the objection judgment model was evaluated using the above test data. Here, too, samples with a confidence level of 0.5 or higher for which the objection judgment model judged the classification result of the identification model to be incorrect (objection) were considered to be samples that the objection judgment model judged to be "objectionable." In addition, samples with a confidence level of less than 0.5 were considered to be samples that the objection judgment model judged to be "no objection."

[0063] Here, when an analyst verified 52 samples that were determined to be "objective" by the objection determination model, 15 of them were correct. In other words, the correct answer rate for "objective" was 0.28 (= 15 / 52). In addition, when an analyst verified 29 samples that were determined to be "no objection" by the objection determination model, 3 of them were incorrect. In other words, the incorrect answer rate for "no objection" was 0.10 (= 3 / 29).

[0064] From the above, it was confirmed that samples judged as disputed by the objection judgment model (samples for which the identification results by the identification model are likely to be incorrect) are suitable samples for human verification.

[0065] [System configuration, etc.] In addition, each component of each unit shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in part by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0066] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0067] [program] The above-mentioned determination device 10 can be implemented by installing a program (determination program) as package software or online software on a desired computer. For example, by executing the above-mentioned program on an information processing device, the information processing device can function as the determination device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and further terminals such as PDA (Personal Digital Assistant), etc.

[0068] 7 is a diagram showing an example of a computer that executes a determination program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0069] The memory 1010 includes a read only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0070] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process executed by the above-mentioned determination device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing a process similar to the functional configuration of the determination device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0071] Data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0072] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0073] 10 Judgment device 11 Input / output section 12 Storage section 13 Control section 131 Learning Department 132 Data Reception Department 133 Identification Unit 134 Basis Output Unit 135 Judgment section 136 Extraction part 137 Data output section 138 Trend Analysis Department 139 Learning Data Selection Unit

Claims

1. a discrimination unit that discriminates whether input data is malignant or not by using a discrimination model that discriminates whether input data is malignant or not; a reason output unit that outputs a reason for classification of input data by the discriminative model using an explanatory model that outputs a reason for classification by the discriminative model; a determination unit that receives the result of the identification and a reason for the identification as an input to an objection determination model that outputs a probability that the result of the identification is incorrect, and outputs the probability that the result of the identification is incorrect by inputting the result of the identification and the reason for the identification; A determination device comprising:

2. A learning unit that learns the objection determination model using a data set including the result of the identification, a reason for the identification, and a label indicating whether the result of the identification is correct or not. The determination device according to claim 1 , further comprising:

3. The learning unit is If the discriminative model has a tendency to classify malignant data as non-malignant, the objection determination model is trained using data from the data set that is actually malignant but is classified as non-malignant by the discriminative model.

3. The determination device according to claim 2.

4. The learning unit is If the discriminative model has a tendency to classify non-malicious data as malignant, the objection determination model is trained using data from the data set that is not malignant but is classified as malignant by the discriminative model.

3. The determination device according to claim 2.

5. a tendency analysis unit that analyzes whether the discriminant model has a tendency to classify malignant input data as non-malignant or a tendency to classify non-malignant input data as malignant, based on a classification result obtained by inputting data used for training the discriminant model into the discriminant model and a correct answer label indicating whether the data is malignant or not; The determination device according to claim 3 or 4, further comprising:

6. an extraction unit for extracting data to be verified by a user based on the probability that the result of the identification is incorrect; The determination device according to claim 1 , further comprising:

7. A determination method executed by a determination device, identifying whether the input data is malicious or not using a discriminative model that identifies whether the input data is malicious or not; outputting a reason for classification of input data by the discriminative model using an explanatory model that outputs a reason for classification by the discriminative model; a step of inputting the result of the identification and the grounds for the identification into an objection determination model that receives as input the result of the identification and the grounds for the identification, and outputs the probability that the result of the identification is incorrect, thereby outputting the probability that the result of the identification is incorrect; A method for determining whether or not a particular

8. identifying whether the input data is malicious or not using a discriminative model that identifies whether the input data is malicious or not; outputting a reason for classification of input data by the discriminative model using an explanatory model that outputs a reason for classification by the discriminative model; a step of inputting the result of the identification and the grounds for the identification into an objection determination model that receives as input the result of the identification and the grounds for the identification, and outputs the probability that the result of the identification is incorrect, thereby outputting the probability that the result of the identification is incorrect; A judgment program for causing a computer to execute the above.

Images