Abnormality judgment device

Abstract

The present disclosure relates to an abnormality-determining device, in which, with respect to a test information group and a request information group output from an inspection device, on the basis of a response information group output by a device to be tested, an unclassified information group output by a plurality of monitoring functions of the inspection device is received, an abnormality classification process and an abnormality detection process are performed on the unclassified information group and a detection result is output, the abnormality classification process classifies the unclassified information group as normal and abnormal by a machine learning algorithm, the information that is classified as abnormal is considered an abnormality information group, the abnormality detection process identifies relationships between each of the pieces of abnormal information of the abnormal information group and the plurality of monitoring functions of the inspection device, weighs the identified monitoring functions to produce abnormality identification values, compares the abnormality identification values with a predetermined threshold value, determines an abnormality if the abnormality identification value is equal to or greater than the threshold value, and outputs, as the detection result, the test information group used in the test when the identified monitoring function is operating.

Description

[Technical field]

[0001] The present disclosure relates to an abnormality determination device for a device, and more particularly to an abnormality determination device for a device that is a target of a black-box type security test. [Background technology]

[0002] As an example of a conventional equipment abnormality determination device, Patent Document 1 discloses a device that detects abnormalities in the tools of a machine tool, and discloses technology that learns the normal state and determines an abnormality when the normal state deviates from that value.

[0003] Furthermore, Patent Document 2 discloses a technique for learning measurement data of an abnormal state of a device, comparing normal and abnormal values ​​using a neural network, and judging an abnormality.

[0004] Furthermore, Patent Document 3 discloses a technology that uses machine learning to learn measurement data of an abnormal state of a device, compares normal and abnormal values, and determines whether the device is abnormal.

[0005] Furthermore, Patent Document 4 discloses a technique for identifying the cause of a transmission line abnormality without requiring a special circuit, and for example, can distinguish between a break, a short circuit, and a malfunction caused by noise. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-24055 A [Patent Document 2] Japanese Patent Application Publication No. 10-198420 [Patent Document 3] JP 2019-124984 A [Patent Document 4] JP 2016-20185 A Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, normal and abnormal states are classified using machine learning such as OCSVM (One Class Support Vector Machine), and the classified abnormal states are further classified using a multi-class SVM. The method is to assign an abnormal state to the classification that is finally output, and does not specify a method for additionally implementing information output by machine learning.

[0008] In Patent Documents 2 and 3, there is a part that learns the measurement values ​​of abnormal states and a part that learns the normal states, and weights can be calculated by learning each of them. The weights are used to associate causes with vectors (nodes). However, the association between causes and vectors is done manually or using a table created in advance, and the method of creating the table is not specified.

[0009] In Patent Document 4, the cause of an abnormality is determined from the receive count and transmit count values ​​using an error count, which is a standard function of CAN (Controller Area Network), but the classification of errors other than those related to communication protocol functions is not clearly stated.

[0010] As such, conventional equipment anomaly detection devices have limited uses and functions and are not compatible with black-box security testing, which tests the robustness of today's control systems.

[0011] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an abnormality determination device capable of determining an abnormality in equipment that is the subject of black-box type security testing. [Means for solving the problem]

[0012] The abnormality judgment device according to the present disclosure is an abnormality judgment device for a test target device that is the subject of testing by an inspection device, and receives a pre-classified information group output by a plurality of monitoring functions of the inspection device based on a response information group output by the test target device in response to a test information group and a request information group output from the inspection device, performs an abnormality classification process and an abnormality detection process on the pre-classified information group, and outputs a detection result, wherein the abnormality classification process classifies information of the pre-classified information group into normal and abnormal using a machine learning algorithm, and sets the information classified as abnormal as an abnormality information group, and the abnormality detection process identifies a relationship between each abnormality information of the abnormality information group and the plurality of monitoring functions of the inspection device, weights the identified monitoring functions to set an abnormality identification value, compares the abnormality identification value with a predetermined threshold value, and judges an abnormality to be present if the value is equal to or greater than the threshold value, and outputs the test information group used in the test when the identified monitoring function is operating as the detection result. Effect of the Invention

[0013] According to the abnormality determination device of the present disclosure, a test information group that causes the test target device to respond in an unusual way when tested by the inspection device is identified from a group of pre-classification information output by multiple monitoring functions of the inspection device, and the monitoring function identified from among that group is weighted to determine an abnormality identification value. This makes it possible to narrow down a huge group of test information to a group of test information that may be abnormal, as in black-box security testing, thereby efficiently determining an abnormality in the test target device. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration of a system for judging an abnormality in a test target device using an abnormality judging device according to a first embodiment. [Diagram 2] FIG. 11 is a diagram showing an example of a pre-classification information group. [Diagram 3] FIG. 11 is a diagram showing an example of a pre-classification information group. [Figure 4] FIG. 11 is a diagram showing an example of a pre-classification information group. [Diagram 5] FIG. 11 is a diagram showing an example of a pre-classification information group. [Figure 6] 4 is a flowchart illustrating an abnormality classification process in the abnormality determination device according to the first embodiment. [Figure 7] 4 is a flowchart illustrating a test item classification process in the abnormality judging device according to the first embodiment. [Figure 8] FIG. 13 is a diagram showing a configuration in which a lamp is provided in a test target device. [Figure 9] FIG. 4 is a waveform diagram showing blinking of a lamp under normal conditions. [Figure 10] FIG. 4 is a waveform diagram showing the blinking of a lamp when an abnormality occurs. [Figure 11] 4 is a diagram showing an example of a weighting table of the abnormality determination device according to the first embodiment; FIG. [Figure 12] 4 is a flowchart illustrating an abnormality determination process in the abnormality determination device according to the first embodiment. [Figure 13] 11 is a block diagram showing a configuration of a system for judging an abnormality in a test target device using an abnormality judging device according to a second embodiment. FIG. [Figure 14] 10 is a flowchart illustrating training for abnormality classification in the abnormality determination device according to the second embodiment. [Figure 15] 13 is a diagram showing an example of template information stored in a determination table of the abnormality determining device according to the second embodiment. FIG. [Figure 16] 13 is a diagram showing an example of template information stored in a determination table of the abnormality determining device according to the second embodiment. FIG. [Figure 17] 13 is a diagram showing an example of template information stored in a determination table of the abnormality determining device according to the second embodiment. FIG. [Figure 18] FIG. 2 is a diagram illustrating a hardware configuration for implementing the abnormality determination device according to the first and second embodiments. [Figure 19] FIG. 2 is a diagram illustrating a hardware configuration for implementing the abnormality determination device according to the first and second embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] <Embodiment 1> FIG. 1 is a block diagram showing a configuration of a system for determining an abnormality in a test target device 202 using an abnormality determining device 100 according to a first embodiment of the present disclosure.

[0016] A test target device 202 shown in FIG. 1 is a device that is the subject of a black-box type security test by a security inspection device 201, and a test information group 211 and a request information group 212 are input from the security inspection device 201.

[0017] The test information group 211 may be, for example, grammatically incorrect information of a communication protocol such as Ethernet (registered trademark) or information such as normal information of a communication protocol such as Ethernet, which is sent at short intervals. Other examples include information of a communication protocol such as TCP (Transmission Control Protocol) and input information of software such as a program or an argument of a function. Here, one of the above information is referred to as single test information t n Let the set of single test information be the set of single test information {t 1 , t 2 , t n}. Then, the single test information group is expressed as T n Then, the set of single test information is {T 1 , T 2 , ,T n}This is expressed as:

[0018] The security inspection device 201 holds in advance or mechanically creates a set of the above-mentioned single test information groups, and inputs them to the test target device 202 as a test information group 211 .

[0019] Examples of the security testing device 201 include Achilles Test Platform (registered trademark) and Defensics (registered trademark), which have a large amount of test information. Achilles Test Platform is a tool that performs robustness verification by fuzzing specifically for control devices, and Defensics is a black-box type tool that detects software security vulnerabilities.

[0020] As a method for mechanically creating the test information group, for example, the "length" field is provided in the IEEE802.3 frame of the Ethernet communication packet, but there is a method of increasing this "length" to make the test packet actually a large communication packet. Note that such a test packet is prepared by the above-mentioned tool.

[0021] The request information group 212 includes, for example, request information such as an ARP (Address Resolution Protocol) request and an ICMP (Internet Control Message Protocol) request. Other examples include requests of other communication protocols, which are request information that returns a response to a request. Taking a personal computer (PC) as an example, the request information group 212 includes PING, and if the PC is running, it responds to the PING and is assumed to be normal, and if there is no response to the PING, it is determined to be abnormal. The request information group 212 is the PING sent to the PC, and the PING returned from the PC becomes the response information group 213.

[0022] Examples of the test target device 202 include embedded devices connected to inter-controller networks such as a distributed control system (DCS) and a programmable logic controller (PLC), PCs, and devices that run application software, but are not limited to these, and also include devices that operate independently or in conjunction with each other.

[0023] The test target device 202 outputs a response information group 213 in response to the input test information group 211 and request information group 212, and the output response information group 213 is input to the security inspection device 201. More specifically, the response information group 213 is an information group output by the test target device 202 when the test information group 211 is input to the test target device 202 and the request information group 212 is additionally input, or an information group output by the test target device 202 when the request information group 212 is not input. The response information group 213 includes, for example, ARP and ICMP response information, and output information such as analog output and digital output.

[0024] The security inspection device 201 outputs a pre-classification information group 214 based on the input response information group 213. Here, the pre-classification information group 214 is a measurement value obtained by measuring a delay between when the security inspection device 201 inputs the request information group 212 to the test target device 202 and when the response information group 213 is output. When the load on the test target device 202 increases, the output of the response information group 213 is delayed, causing a delay. Alternatively, when the load on the test target device 202 increases, the test target device 202 cannot receive the request information group 212, causing a delay.

[0025] The security inspection device 201 has a monitoring function f that measures this delay, etc. 1 , f 2 , f n The monitoring functions include, for example, an ARP monitoring function, an ICMP monitoring function, a physical LINK monitoring function, an analog output monitoring function, a digital output monitoring function, and a lamp ON / OFF monitoring function. An example of the pre-classification information group 214 by each monitoring function is shown in Figs. 2 to 5. In addition, the pre-classification information group 214 includes a single test information group T constituting the test information group 211. n The information {t1, t2, ..., tn} is given as the result.

[0026] Fig. 2 is an example of pre-classification information group 214 by the ARP monitoring function and the ICMP monitoring function, and shows two-dimensional information, time t on the X-axis and response time (delay time) from request to response on the Y-axis. Fig. 2 shows that the response time for responses to ARP requests and ICMP requests increases as time passes, and it can be inferred that the equipment is under load.

[0027] Fig. 3 is an example of the pre-classification information group 214 by the digital output monitoring function, and shows two-dimensional information, time t on the X axis and digital output information on the Y axis. In Fig. 3, a predetermined digital waveform is shown, but the waveform may become dull or missing due to an increase in load or the introduction of a bug.

[0028] Fig. 4 is an example of the pre-classification information group 214 by the analog output monitoring function, and shows two-dimensional information, time t on the X axis and analog output information on the Y axis. In Fig. 4, a predetermined analog waveform is shown, but the waveform may become dull or missing due to an increase in load or the introduction of a bug.

[0029] Regarding the analog output information and the digital output information, if the analog output and the digital output, which originally change every second, are fixed, these outputs may change due to the influence of the test information group 211. It is also possible to monitor the degree of the influence of the test information group 211 based on whether the change is large or small.

[0030] Fig. 5 is an example of pre-classification information group 214 by the physical link monitoring function, and shows two-dimensional information, time t on the X-axis and binary information of information when the link is ON, i.e., 1, and information when the link is OFF, i.e., 0, on the Y-axis. Fig. 5 shows that the device alternates between ON, which indicates that it is connected to the physical layer of the network, or more precisely, the data link layer, and OFF, which indicates that it is not connected. Fig. 5 also shows an example of pre-classification information group 214 by the lamp ON / OFF monitoring function.

[0031] 1, the configuration of the anomaly determination device 100 will now be described. The anomaly determination device 100 includes a processing information acquisition unit 101, an anomaly classification unit 102, a test item classification unit 103, an anomaly detection unit 104, a determination table 111, a weight table 112, and a threshold value table 113, and receives a pre-classification information group 214 output by a security inspection device 201 as input, performs an anomaly classification process and an anomaly detection process on the pre-classification information group 214, and detects a single test information group T n is output as a single test information group 217.

[0032] The processing information acquisition unit 101 receives the pre-classification information group 214 and outputs the pre-classification information group 214. The received pre-classification information group 214 is, for example, a monitoring function f of the security inspection device 201. n 2 to 5, it is two-dimensional information on the X axis and the Y axis. That is, when all the tests based on the test information group 211 are completed, the monitoring function f that is turned on in the security inspection device 201 is 1 , f 2 , f n For each (n = number of functions turned ON), two-dimensional information on the X and Y axes can be obtained. In this case, the X axis is the time axis, which is the time from the start of the test to the end of the test.

[0033] The anomaly classifier 102 classifies the two-dimensional information group of the pre-classification information group 214, that is, the information group represented by time and delay time (latency) {(Time 1 , latency 1 ), (Time 2 , latency 2 ), ... (Time N , latency N )}(N=t n ) is subjected to anomaly classification processing. This anomaly classification processing will be explained with reference to FIG.

[0034] 6 is a flowchart illustrating the anomaly classification process in the anomaly classification unit 102. The anomaly classification unit 102 first receives input information of a two-dimensional information group (step S1).

[0035] Next, the parameter values ​​stored in the decision table 111 are obtained (step S2). Specifically, when a machine learning algorithm such as an One Class Support Vector Machine (OCSVM) is used for the anomaly classification process, the parameter values ​​such as kernel, degree, gamma, coef0, tol, and nu stored in the decision table 111 are obtained. OCSVM is a known technique that learns data patterns by unsupervised learning and detects data that deviates from the pattern as an anomaly. Note that the classification of anomalies is not limited to OCSVM, and other algorithms, for example, k-means clustering, which is a clustering algorithm, can also be used. Clustering is a known technique that determines a threshold value and classifies based on the value. The decision table 111 stores parameter values ​​suitable for the algorithm to be used.

[0036] The anomaly classifier 102 classifies the pre-classification information group 214 using the acquired parameter values ​​(step S3). For example, when OCSVM is used, the OCSVM algorithm is executed by setting nu=0.05, kernel=rbf, and gamma=scale, and the pre-classification information group 214 is classified. Information determined to be abnormal among the classification results is set as an abnormality information group 215, and is output as the classification result (step S4). For example, the classification result of OCSVM is output as 1 for normal and -1 for abnormal.

[0037] In the above description, the classification of anomalies has been mentioned, but training has not been mentioned. The classification of anomalies in this disclosure is not to classify normal and abnormal as in fuzzing testing, but to identify rare abnormal values ​​among those classified as abnormal by fuzzing testing. In other words, the purpose is not to perform accurate classification through accurate training, but to perform rough classification and then identify test items that exhibit anomalies that rarely occur.

[0038] The anomaly information group 215 output from the anomaly classifying unit 102 is input to the test item classifying unit 103 (FIG. 1), which identifies the relationship between the anomaly information group 215 and the monitoring functions of the security inspection device 201, and then classifies the test items. This test item classifying process will be described with reference to FIG.

[0039] 7 is a flowchart for explaining the test item classification process in the test item classification unit 103. The test item classification unit 103 first receives the input information of the abnormality information group 215 (step S11).

[0040] Next, the relationship between the anomaly information group 215 and the monitoring function of the security inspection device 201 is identified (step S12). That is, the monitoring function f is used to identify which of the anomaly information groups 215 is generated by the monitoring function. 1 , f 2 , f n or by function names such as ARP monitoring function, ICMP monitoring function, etc. Taking the information by the physical link monitoring function shown in Fig. 5 as an example, the abnormality information group 215 is data that repeats ON and OFF, so it is identified as information by the physical link monitoring function. Note that the number of identified monitoring functions is not limited to one, and multiple functions may be identified.

[0041] Furthermore, lamp information given to the test target device 202, which indicates the device status, can also be used as monitoring information. Fig. 8 shows a configuration in which lamp A and lamp B are provided in the test target device 202. Lamp A and lamp B are lamps that indicate the device status by their blinking state, and can notify the outside by their blinking state when the test target device 202 is operating normally (normal) and when it is operating abnormally (abnormal). Figs. 9 and 10 respectively show the behavior of the lamps in normal and abnormal cases using waveform diagrams of signals input to the lamps, and in both cases, the horizontal axis indicates time (t) and the vertical axis indicates output.

[0042] As shown in the left and right diagrams of Fig. 9, under normal circumstances, Lamp A and Lamp B repeatedly turn on and off at different cycles, but as shown in the left and right diagrams of Fig. 10, under abnormal circumstances, Lamp A turns on and off at a different cycle than normal, and Lamp B does not turn on and is off. In this way, lamp behavior that is not seen under normal circumstances can be used as this monitoring function. The ON and OFF states of the lamps can also be obtained as video information using a separately installed video device.

[0043] Next, the monitoring functions identified in step S12 are weighted using the weighting table 112 (FIG. 1) to calculate abnormal values ​​(step S13).

[0044] An example of the weighting table 112 is shown in FIG. 11. In the weighting table 112 shown in FIG. 11, the analog output monitoring function and the digital output monitoring (analog-digital output monitoring) function f 1 Weight "3", lamp ON / OFF monitoring function f 2 Weight is "3", LINK ON / OFF monitoring function f 3 has a weight of 3, and the ARP monitoring function f 4 has a weight of 2, and the ICMP monitoring function f 5 The weight is set to "1", and the TCP Port monitoring function f 6 is assigned a weight of "1".

[0045] For example, the monitoring function identified in step S12 is the LINK ON / OFF monitoring function f 3 If so, the weight is "3" according to the weight table 112, so the abnormal value is 3. Also, the monitoring function identified in step S12 is the ARP monitoring function f 4 and ICMP monitoring function 5 If so, the weights are "1" and "2" according to weighting table 112, respectively, so the abnormal value is calculated by multiplying the two together to obtain 1×2=2. The value weighted using weighting table 112 for the identified monitoring function in this way is determined as the abnormal value, and if multiple monitoring functions are identified, the total product value multiplied by the weights is determined as the abnormal value.

[0046] Next, the abnormal value obtained in step S13 is output as an abnormality identification value 216 (step S14).

[0047] In the example shown in the weight table 112 in FIG. 11, the weight value is set high for physical information such as analog / digital output, which is information for fuzzing testing and is unlikely to interfere with communication, i.e., information that is not directly related to communication. Also, in the case of low-layer information such as ARP communication, the weight value is set high compared to high-layer information. The lowest weight value is for high-layer information such as ICMP and TCP Port, i.e., information that is directly related to communication. In this way, the test item classification unit 103 makes it possible to classify information useful for fuzzing testing, etc., by weighting information obtained by a monitoring function that is unlikely to be directly related to communication errors, i.e., information that is not usually classified as abnormal in OCSVM, etc., so as to determine it as abnormal.

[0048] In many fuzzing test results, errors determined to be errors in the higher layers are often caused by load phenomena such as DoS (Denial of Service), and due to the nature of the phenomenon, it is difficult to classify them using black box testing alone. As a result, it is difficult to trace the cause of the failure of the tested device, and the information obtained is often not useful, but the method disclosed here can efficiently discover such information.

[0049] The abnormality identification value 216 output from the test item classification section 103 is input to the abnormality detection section 104 (FIG. 1), which performs abnormality determination. This abnormality determination process will be described with reference to FIG.

[0050] 12 is a flowchart illustrating the abnormality determination process in the abnormality detection unit 104. The abnormality detection unit 104 first receives the input information on the abnormality identification value 216 (step S21).

[0051] Next, the abnormality identification value 216 is compared with the threshold value table 113 (FIG. 1), and if it is equal to or greater than the threshold value stored in the threshold value table 113, it is determined to be abnormal (step S22). n is output as the single test information group 217 (step S23).

[0052] For example, if the threshold value in threshold value table 113 is set to 3 or more, information in higher layers of communication will not be recognized as abnormal, and information in lower layers of communication will be defined as abnormal. If information not related to communication is not recognized as abnormal, the threshold value is set to 5 or more. The test item classification process and the abnormality determination process are collectively referred to as the abnormality detection process.

[0053] As described above, the abnormality determination device 100 of embodiment 1 identifies a test information group from a group of pre-classification information of abnormalities monitored by the security inspection device 201, which causes the test target device 202 to respond in an unusual manner during a security test, and further assigns weights to physical information, communication information, etc. from among the test information groups, thereby narrowing down a large number of test information groups to single test information groups that may be abnormal.

[0054] <Embodiment 2> Fig. 13 is a block diagram showing the configuration of a system that uses an abnormality determination device 100A according to the second embodiment of the present disclosure to determine an abnormality in a test target device 202. Note that in Fig. 13, the same components as those in the system of the first embodiment shown in Fig. 1 are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0055] The anomaly determination device 100A is similar to the anomaly determination device 100 shown in FIG. 1 in that it receives the pre-classification information group 214 output by the security inspection device 201 as input and outputs the single test information group 217, but it includes an anomaly training unit 105 that receives the pre-classification information group 214 output by the processing information acquisition unit 101 as input and uses sample template information stored in the determination table 111 together with parameter values ​​for anomaly classification processing to train the anomaly classification in the anomaly classification unit 102. By including the anomaly training unit 105, the accuracy of classification between anomalies and normalities is improved. Note that the configuration of the anomaly determination device 100A is the same as that of the anomaly determination device 100 shown in FIG. 1 except for the anomaly training unit 105, and the same configuration is assigned the same reference numeral, and duplicated explanations will be omitted.

[0056] The processing information acquisition unit 101 receives the pre-classification information group 214 and outputs the pre-classification information group 214 to the abnormality training unit 105. The pre-classification information group 214 may include, for example, a monitoring function f of the security inspection device 201. n 2 to 5, it is two-dimensional information on the X axis and the Y axis. That is, when all the tests based on the test information group 211 are completed, the monitoring function f that is turned on in the security inspection device 201 is 1 , f 2 , f n For each (n = number of functions turned ON), two-dimensional information on the X and Y axes can be obtained. In this case, the X axis is the time axis, which is the time from the start of the test to the end of the test.

[0057] The abnormality training unit 105 receives the pre-classification information group 214 as an input and performs training of the abnormality classification unit 102 for abnormality classification using the sample template information stored in the decision table 111. This abnormality classification training will be described with reference to FIG.

[0058] 14 is a flowchart explaining the training of anomaly classification in the anomaly training unit 105. The anomaly training unit 105 first receives the input two-dimensional information group of the pre-classification information group 214 (step S31). The received two-dimensional information group of the pre-classification information group 214 is input to the anomaly classification unit 102.

[0059] Next, the abnormality training unit 105 acquires the template information stored in the judgment table 111 (step S32). An example of this template information is shown in Figs.

[0060] Figure 15 shows the physical link monitoring function 1 This is an example of template information corresponding to the above, and is a group of information in which the X-axis, which is two-dimensional information, indicates time t, and the Y-axis indicates that a connection to the LINK is possible as ON, and that a connection to the LINK is not possible as OFF. In the example of Figure 15, it can be seen that a physical connection to the LINK is possible.

[0061] Figure 16 shows the ICMP monitoring function 2 16 is an example of template information corresponding to the above, and the X-axis of the two-dimensional information is time t, and the Y-axis is the delay from when the security inspection device 201 sends an ICMP request to the test target device 202 until the test target device 202 returns an ICMP response, expressed as a numerical value from 0% to 100%. The delay is specified by a predetermined time, and if the time is 1 second, a delay of 1 second or more is 100%. In the example of FIG. 16, the time t 1 It can be seen that there is a 100% delay in

[0062] Figure 17 shows the ARP monitoring function 3 17 is an example of template information corresponding to the above, and the X-axis of the two-dimensional information is time t, and the Y-axis is the delay from when the security inspection device 201 sends an ARP request to the test target device 202 until the test target device 202 returns an ARP response, expressed as a numerical value from 0% to 100%. The delay is specified by a predetermined time, and if the time is 1 second, a delay of 1 second or more is 100%. In the example of FIG. 17, it can be seen that delays of about 30% to 50% are repeated.

[0063] The purpose of using templates such as those shown in Figures 15 to 17 will be described. 1 ~f 315, the contents that can be determined using the above are that although no event such as a power cut has occurred in the test target device 202, as shown in FIG. 16 and FIG. 17, a communication abnormality has occurred in the layer above IP (Internet Protocol), and further, as shown in FIG. 16, the abnormality is proportional to time by the test. This is because, as shown in FIG. 16, at time t 0 From time t 1 This can be seen from the fact that the delay increases in proportion to time between 0 and 1. For example, if the value on the Y axis exceeds 50% it is judged to be abnormal, and if it is not, it is not abnormal, then if there is no load on ICMP, then there will not be such a proportional relationship, and the value will transition over time with a maximum value of around 50%, as shown in Figure 17. If the delay reaches 100%, as in Figure 16, then there is no ICMP response at all to the ICMP request, and it can be judged to be abnormal.

[0064] These phenomena are judged as abnormal on the security inspection device 201, but the possibility that this abnormality is caused by an error occurring in a specific communication packet is low, and it is highly likely that a more definite judgment cannot be made by black box testing alone. Therefore, such tests are not appropriate as test items for investigating the cause when correcting a defect in the test target device 202, so it is desired that the abnormality classification unit 102 judges it as normal.

[0065] Therefore, the abnormality training unit 105 changes the parameter values ​​stored in the judgment table 111 so that the abnormality classification unit 102 classifies the abnormality as normal. In this manner, changing the parameter values ​​to correct the abnormality classification unit 102 to classify the abnormality as normal is referred to as a "change process" or "training" in the present disclosure.

[0066] Specifically, when OCSVM is used for the anomaly classification process, the parameter values ​​such as kernel, degree, gamma, coef0, tol, and nu stored in the judgment table 111 are changed to values ​​such that the template information shown in Figs. 15 to 17 is classified normally, and set in the judgment table 111 (step S33). When an algorithm other than OCSVM, for example, k-means clustering, is used for anomaly classification, the judgment table 111 stores parameter values ​​suitable for the algorithm to be used, and the stored parameter values ​​are similarly changed and trained to classify normally. The template information is not limited to the information shown in Figs. 15 to 17, and may be stored in the judgment table 111 according to the phenomenon to be judged as normal in the anomaly classification unit 102.

[0067] The anomaly classification unit 102 acquires the changed parameter values ​​stored in the judgment table 111, and classifies the template information by the acquired parameter values ​​(step S34). For example, when using OCSVM, the OCSVM algorithm is executed by setting nu=0.05, kernel=rbf, and gamma=scale, but the template information is judged as normal by changing the parameter values ​​such as degree, coef0, and tol in step S33. If the template information is judged as abnormal even with the changed parameter values, the process returns to step S33, and the anomaly training unit 105 again changes the parameter values ​​stored in the judgment table 111. After the change, the parameter values ​​are set in the judgment table 111 again, and step S34 is repeated.

[0068] On the other hand, if the template is classified normally in step S34, the anomaly classification unit 102 acquires the parameter values ​​stored in the judgment table 111 (step S35). Specifically, when OCSVM is used for the anomaly classification process, the anomaly classification unit 102 acquires the parameter values ​​such as kernel, degree, gamma, coef0, tol, and nu stored in the judgment table 111. The parameter values ​​include the parameter values ​​changed in step S33.

[0069] Next, the anomaly classification unit 102 classifies the two-dimensional information group of the pre-classification information group 214 input from the anomaly training unit 105 (step S36). For example, when OCSVM is used, the OCSVM algorithm is executed by setting nu=0.05, kernel=rbf, and gamma=scale to classify the pre-classification information group 214, and information determined to be abnormal in the classification result is output as the abnormality information group 215 as the classification result (step S37). For example, the classification result of OCSVM is output as 1 for normal and -1 for abnormal.

[0070] Here, if the pre-classification information group 214 contains information similar to the template information shown in Figures 15 to 17, the information is classified normally by the parameter values ​​changed in step S33. The subsequent processing for the abnormality information group 215 is the same as in the first embodiment.

[0071] As described above, anomaly determination device 100A of embodiment 2 has the same effects as anomaly determination device 100 of embodiment 1, and in addition thereto, for specific anomalies that should not be determined as anomalies by anomaly classifier 102 among pre-classification information group 214 by the monitoring function of security inspection device 201, anomaly classifier 102 is trained using pre-prepared template information, so that if pre-classification information group 214 contains information similar to the template information, anomalies caused by that information are classified normally. As a result, specific anomalies are not detected, and target information can be acquired efficiently.

[0072] <Hardware configuration> Each component of the abnormality determination devices 100 and 100A according to the first and second embodiments described above can be configured using a computer, and is realized by the computer executing a program. That is, the abnormality determination devices 100 and 100A are realized by, for example, a processing circuit 50 shown in Fig. 18. A processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) is applied to the processing circuit 50, and the function of each part is realized by executing a program stored in a storage device.

[0073] Dedicated hardware may be applied to the processing circuit 50. When the processing circuit 50 is dedicated hardware, the processing circuit 50 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

[0074] In the abnormality determination devices 100 and 100A, the functions of the respective components may be realized by individual processing circuits, or the functions may be realized collectively by a single processing circuit.

[0075] 19 shows a hardware configuration in the case where the processing circuit 50 is configured using a processor. In this case, the functions of each part of the abnormality determination devices 100 and 100A are realized by a combination of software, etc. (software, firmware, or software and firmware). The software, etc. are written as a program and stored in the memory 52. ​​The processor 51 functioning as the processing circuit 50 realizes the functions of each part by reading and executing the program stored in the memory 52 (storage device). In other words, it can be said that this program causes a computer to execute the procedure and method of operation of the components of the abnormality determination devices 100 and 100A.

[0076] Here, the memory 52 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc) and its drive device, or any storage medium to be used in the future.

[0077] The above describes a configuration in which the functions of the components of the abnormality determination devices 100 and 100A are realized by either hardware or software, etc. However, the present invention is not limited to this, and some of the components of the abnormality determination devices 100 and 100A may be realized by dedicated hardware, and other components may be realized by software, etc. For example, it is possible to realize the functions of some of the components by the processing circuit 50 as dedicated hardware, and realize the functions of other components by the processing circuit 50 as the processor 51 reading and executing a program stored in the memory 52.

[0078] As described above, the abnormality determination devices 100 and 100A can realize the above-mentioned functions by hardware, software, etc., or a combination of these.

[0079] Although the present disclosure has been described in detail, the above description is illustrative in all respects and does not limit the present disclosure. It is understood that countless variations not illustrated can be assumed without departing from the scope of the present disclosure.

[0080] In addition, within the scope of the present disclosure, it is possible to freely combine the respective embodiments, and to appropriately modify or omit the respective embodiments.

Claims

1. An abnormality determination device for a test target device that is a target of testing by an inspection device, receiving a pre-classification information group output by a plurality of monitoring functions of the inspection device based on a response information group output by the test target device in response to the test information group and the request information group output by the inspection device, performing an anomaly classification process and an anomaly detection process on the pre-classification information group, and outputting a detection result; The abnormality classification process classifies the information of the pre-classification information group into normal and abnormal information by a machine learning algorithm, and sets the information classified as abnormal as an abnormality information group; The abnormality detection process includes: An abnormality determination device that identifies the relationship between each abnormality information in the abnormality information group and the multiple monitoring functions of the inspection device, weights the identified monitoring functions to determine an abnormality identification value, compares the abnormality identification value with a predetermined threshold value, and determines an abnormality if the value is equal to or greater than the threshold value, and outputs the test information group used in the test when the identified monitoring function is operating as the detection result.

2. The anomaly determination device according to claim 1 , wherein the machine learning algorithm is a one-class support vector machine or a k-means method.

3. The test includes a test for a communication anomaly, The response information group includes: The abnormality determination device according to claim 1 , further comprising information related to the communication and information unrelated to the communication.

4. The weighting of the identified monitoring functions is 4. The abnormality determination device according to claim 3, wherein the information less related to the communication is weighted higher, and the information more related to the communication is weighted lower.

5. The abnormality classification process includes: The abnormality determination device of claim 1 , further comprising a change process for changing values ​​of parameters used in the machine learning algorithm so that, when classifying information of the pre-classification information group into normal and abnormal, a specific abnormality is classified as normal in the machine learning algorithm.

6. The change process includes: changing values ​​of the parameters such that, when a plurality of pieces of template information respectively corresponding to the plurality of monitoring functions of the inspection device are classified by the machine learning algorithm, the plurality of pieces of template information are classified as normal; The abnormality determination device according to claim 5 , further comprising: a machine learning algorithm for classifying the information in the pre-classification information group using the changed parameter value.

Images