Detection device, detection method, and detection program
The detection device uses BERT to compare similar normal packets with target packets, enhancing anomaly detection accuracy for industrial and building network control systems by addressing the challenges of long packet lengths and false positives in existing systems.
Patent Information
- Application Number
- PCT/JP2023/019586
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-28
AI Technical Summary
Anomaly detection systems for industrial and building network control systems face challenges in accurately detecting unauthorized changes in packet payload content, particularly for protocols with long packet lengths, leading to potential false positives and decreased detection accuracy.
A detection device and method that utilizes a natural language processing model like BERT to select similar normal packets and calculate an anomaly score by comparing them with the evaluation target packet data, enabling precise detection of abnormalities even in protocols with long packet lengths.
The solution effectively detects anomalies with high accuracy for protocols with long packet lengths, reducing false positives and improving overall detection performance.
Smart Images

Figure JP2023019586_28112024_PF_FP_ABST
Abstract
Description
Detection device, detection method, and detection program
[0001] The present invention relates to a detection device, a detection method, and a detection program.
[0002] Attention is being focused on anomaly detection systems and intrusion detection systems (OT-IDS: Operational Technology-Intrusion Detection System) for communications used in industrial and building network control systems. For example, even a one-digit change in the temperature setting can lead to a serious accident. In other words, even a one-byte difference in the communication content (payload) can lead to a serious accident.
[0003] For this reason, an anomaly detection system for industrial and building network control systems must be able to perform detailed analysis of payload contents. In other words, the anomaly detection system must be able to detect even a single byte of unauthorized rewriting of the payload.
[0004] Furthermore, various protocols are used in industrial and building networks, so anomaly detection systems are required to be able to analyze payload content for general-purpose protocols.
[0005] One anomaly detection method involves applying natural language processing techniques such as BERT (Bidirectional Encoder Representations from Transformers) to packet analysis to extract information from the payload of any protocol and perform anomaly detection (Patent Document 1).
[0006] International Publication No. 2022 / 059209
[0007] Yutian Chen, Max Welling, Alex Smola, “Super-Samples from Kernel Herding,” [Retrieved April 26, 2023], Internet <URL: https: / / arxiv.org / ftp / arxiv / papers / 1203 / 1203.3472.pdf> Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi, “BERTSCORE: EVALUATING TEXT GENERATION WITH BERT,” [Retrieved April 26, 2023], Internet <URL: https: / / arxiv.org / abs / 1904.09675.pdf>
[0008] However, in the method described in Patent Document 1, since an averaging process is performed in the packet direction, the detection accuracy may be degraded for protocols with long packet lengths. Also, in the method described in Patent Document 1, there is a problem that false positives may occur if a timestamp or a serial number portion is included in a packet and the learning data does not include these comprehensively.
[0009] The present invention has been made in consideration of the above, and aims to provide a detection device, a detection method, and a detection program that can detect abnormalities that occur in the contents of packets with high accuracy, even for protocols with long packet lengths.
[0010] In order to solve the above-mentioned problems and achieve the objectives, the detection device is characterized by having an extraction unit that extracts a predetermined number of similar normal packet data that have a relatively high similarity to packet data to be evaluated from a plurality of normal packet data based on a natural language processing model, and a detection unit that detects whether the packet data to be evaluated is abnormal based on an anomaly score calculated by comparing the predetermined number of similar normal packet data extracted by the extraction unit with the packet data to be evaluated.
[0011] According to the present invention, even for protocols with long packet lengths, it is possible to detect abnormalities occurring in packet contents with high accuracy.
[0012] FIG. 1 is a diagram illustrating an example of the configuration of a detection system according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of a learning device illustrated in FIG. 1. FIG. 3 is a diagram illustrating a processing procedure for learning processing performed by the learning device illustrated in FIG. 2. FIG. 4 is a diagram illustrating an example of the configuration of the detection device illustrated in FIG. 1. FIG. 5 is a diagram illustrating the flow of processing by the detection device illustrated in FIG. 4. FIG. 6 is a diagram illustrating processing by the detection unit illustrated in FIG. 4. FIG. 7 is a diagram illustrating processing by the detection unit illustrated in FIG. 4. FIG. 8 is a diagram illustrating first calculation processing. FIG. 9 is a diagram illustrating processing procedures for detection processing according to an embodiment. FIG. 10 is a flowchart illustrating processing procedures for the first calculation processing illustrated in FIG. 9. FIG. 11 is a flowchart illustrating processing procedures for the second calculation processing illustrated in FIG. 9. FIG. 12 is a diagram illustrating results of an evaluation experiment. FIG. 13 is a diagram illustrating results of the evaluation experiment. FIG. 14 is a diagram illustrating an example of a computer that implements a learning device and a detection device by executing a program.
[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A detailed description of embodiments of a detection device, a detection method, and a detection program according to the present application will be given below with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.
[0014] [Embodiment] In the embodiment, a detection device that detects anomalies in communication data (payload) will be described.
[0015] A detection device according to an embodiment extracts data similar to packet data to be evaluated (similar normal packet data) from normal communication data by natural language processing using BERT (Bidirectional Encoder Representations from Transformers). The detection device then calculates an anomaly score indicating the difference between the extracted similar normal packet data and the packet data to be evaluated. If the anomaly score exceeds a predetermined threshold, the detection device detects that the packet data to be evaluated is abnormal and outputs an alert.
[0016] [Detection System] Next, a detection system according to an embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of the detection system according to an embodiment.
[0017] As shown in FIG. 1, the detection system 1 includes a learning device 10 that performs learning of a natural language processing model used by the detection device 20, and a detection device 20 that detects anomalies in communication data (payload) using the natural language processing model trained by the learning device 10.
[0018] [Learning Device] Fig. 2 is a diagram showing an example of the configuration of the learning device 10 shown in Fig. 1. Fig. 3 is a diagram showing the processing procedure of the learning process executed by the learning device 10 shown in Fig. 2.
[0019] The learning device 10 is implemented by loading a predetermined program into a computer or the like including, for example, a read-only memory (ROM), a random access memory (RAM), a central processing unit (CPU), etc., and having the CPU execute the predetermined program. The learning device 10 also has a communication interface for transmitting and receiving various types of information to and from other devices connected via a network or the like.
[0020] The learning device 10 includes a learning data collection unit 11 , a conversion unit 12 having a conversion model 121 , a learning control unit 13 , a comparison normal data generation unit 14 , and a storage unit 15 .
[0021] The learning data collection unit 11 collects learning packet data for the conversion model 121 (step S1 in FIG. 3).
[0022] The learning control unit 13 performs conversion model learning control to cause the conversion model 121 to learn packet conversion rules using the learning packet data (step S2 in FIG. 3). The model parameters of the conversion model 121 optimized in step S2 are applied to a conversion model 231 (described later) of the detection device 20.
[0023] The conversion unit 12 converts packet data into vector data using a conversion model 121. The vector data is data in which each byte of packet data is associated with a vector that represents the characteristics of the value of each byte. The learning control unit 154 causes the conversion model 121 to learn the value of each byte of the learning packet data.
[0024] The conversion model 121 is, for example, a natural language processing model using BERT. The conversion model 121 converts the value of each byte of each communication packet, which is learning packet data, into a 784-dimensional vector. In step S2 of FIG. 3, the conversion model 121 learns a conversion rule for vector data that associates the position of each byte of the communication packet with the 784-dimensional vector converted from that byte.
[0025] The learning control unit 13 may also control the system to perform preliminary learning by solving auxiliary tasks such as MLM (Masked Language Model) or NSP (Next Sentence Prediction) to determine the characteristics of the value of each byte in the learning packet data. MLM predicts the values of missing bytes in a packet that is missing multiple bytes. NSP determines whether two packet data are consecutive packets. The learning control unit 13 uses these auxiliary tasks to determine the validity of the data in the packet and the validity of consecutive packets, and generates a model that identifies the learning packet data. The auxiliary tasks listed here are merely examples, and the learning control unit 13 may also control the system to learn by solving other auxiliary tasks.
[0026] The comparison normal data generation unit 14 also uses the learning packet data to generate a comparison normal data group 151 (step S3 in FIG. 3). The comparison normal data generation unit 14 sets a packet data group sampled from the learning packet data as the comparison normal data group 151.
[0027] At this time, the comparison normal data generation unit 14 performs sampling using a sampling method such as kernel herding. In this case, random sampling is not performed to prevent performance degradation. Although sampling is desirable from the perspective of processing speed, all learning packet data may be used as the comparison normal data group 151. Furthermore, the greater the number of comparison normal packet data in the comparison normal data group 151, the higher the detection accuracy of the detection device 20, which will be described later.
[0028] The normal comparison data group 151 generated in step S3 of FIG. 3 is stored in the storage unit 15 and is also output to the detection device 20.
[0029] [Detection Device] Next, a description will be given of the detection device 20. Fig. 4 is a diagram showing an example of the configuration of the detection device 20 shown in Fig. 1. Fig. 5 is a diagram explaining the processing flow of the detection device 20 shown in Fig. 4.
[0030] The detection device 20 is implemented by, for example, loading a predetermined program into a computer or the like including a ROM, a RAM, a CPU, etc., and causing the CPU to execute the predetermined program. The detection device 20 also has a communication interface for transmitting and receiving various types of information to and from other devices connected via a network, etc.
[0031] The detection device 20 includes a storage unit 21 that stores a comparison normal data group 151 , an evaluation target data acquisition unit 22 , a conversion unit 23 that has a conversion model 231 , an extraction unit 24 , and a detection unit 25 .
[0032] The evaluation target data acquisition unit 22 acquires one piece of evaluation target packet data (step S11 in FIG. 5).
[0033] The conversion unit 23 converts the evaluation target packet data into vector data (hereinafter referred to as evaluation target vector data) using the conversion model 231 (step S12 in FIG. 5). The conversion model 231 is a natural language processing model using BERT. The conversion model 231 is applied with model parameters of the conversion model 121, which has learned vector conversion rules in the learning device 10.
[0034] The extraction unit 24 extracts M pieces of similar normal packet data that have a relatively high degree of similarity to the packet data to be evaluated from the comparison normal data group 151 (plurality of normal packet data) based on the natural language processing model (step S13 in FIG. 5 ). M is a hyperparameter that is set in advance. For example, M is 100.
[0035] The extraction unit 24 converts each piece of comparison normal packet data in the comparison normal data group 151 into vector data (comparison normal vector data) using the conversion model 231. For each piece of comparison normal vector data, the extraction unit 24 calculates the similarity between the comparison normal vector data and the vector data to be evaluated.
[0036] The extraction unit 24 calculates, for example, a BERTScore as the similarity.
[0037] The extraction unit 24 may also calculate the similarity between the vectors of the evaluation target vector data and the vectors of the comparison normal vector data for each byte of the evaluation target vector data, and calculate the similarity between the evaluation target vector data and the comparison normal vector data from the similarity calculated for each byte. Cosine similarity may be used to calculate the similarity between the vectors of each byte. The similarity between the evaluation target vector data and the comparison normal vector data is, for example, the average of the similarities calculated for each byte. In this case, if the number of vectors in the evaluation target vector data differs from the number of vectors in the comparison normal vector data, the similarity may be calculated based on the smaller number of vectors. The number of vectors in each vector data is the number of bytes of packet data before conversion.
[0038] The extraction unit 24 may calculate the similarity with some of the comparison normal vector data in the comparison normal vector data group. For example, the some comparison normal vector data is obtained by extracting multiple representative packet data from multiple normal packet data using MMD (Maximum Mean Discrepancy)-Critic and converting each of the extracted representative packet data using a model. Alternatively, the some comparison normal vector data is multiple comparison normal vector data obtained by extracting comparison normal packet data having the same packet length as the evaluation target packet data from the comparison normal data group 151 and converting each of the extracted comparison normal packet data using a model.
[0039] The extraction unit 24 extracts M pieces of comparative normal packet data with a relatively high degree of similarity. Hereinafter, the comparative normal packet data extracted by the extraction unit 24 will be referred to as similar normal packet data.
[0040] A relatively high similarity means that the similarity between the evaluation target vector data and a certain comparison normal vector data is higher than the similarity between the evaluation target vector data and another comparison normal vector data. The extraction unit 24 extracts a predetermined number of comparison normal packet data in descending order of similarity, starting with the comparison normal packet data with the highest similarity, as similar normal packet data. Alternatively, the extraction unit 24 may extract M comparison normal packet data from the comparison normal packet data with similarity higher than a threshold value as similar normal packet data.
[0041] The detection unit 25 detects whether the packet data to be evaluated is abnormal based on the anomaly score calculated by comparing the M pieces of similar normal packet data extracted by the extraction unit 24 with the packet data to be evaluated.
[0042] The detection unit 25 extracts, from the similar normal packet data, identical-length packet data having the same packet length as the evaluation target packet data. The detection unit 25 compares the evaluation target packet data with the identical-length packet data byte by byte to calculate a first anomaly score. Alternatively, if the packet length of the similar normal packet data is different from that of the evaluation target packet data, the detection unit 25 calculates a second anomaly score based on the edit distance between the evaluation target packet data and the similar normal packet data.
[0043] 6 and 7 are diagrams illustrating the processing of the detection unit 25 shown in Fig. 4. The detection unit 25 performs calculations by comparing the M pieces of similar normal packet data extracted by the extraction unit 24 with the packet data to be evaluated. The detection unit 25 includes a length comparison unit 251, a first calculation unit 252, and a second calculation unit 253.
[0044] The length comparison unit 251 compares the lengths of the M pieces of similar normal packet data with the evaluation target packet data. The length comparison unit 251 determines whether the number of similar normal packet data having the same length as the evaluation target packet data among the M pieces of similar normal packet data is equal to or greater than a predetermined threshold (step S14 in FIG. 5). Note that similar normal packet data having the same length as the evaluation target packet data is referred to as same-length normal packet data.
[0045] If the length comparison unit 251 determines that the number of normal packet data of the same length as the packet data to be evaluated is greater than or equal to a predetermined threshold (step S14 in Figure 5: Yes), the first calculation unit 252 performs a first calculation process (step S15 in Figure 5).
[0046] In the first calculation process, the first calculation unit 252 performs one-dimensional detection based on a comparison of the numerical values for each byte position between the evaluation target packet data and the same normal packet data, and calculates an anomaly score for each byte position ( FIG. 6 ). The first calculation unit 252 compares each byte (int) (0 to 255) one by one, detects any violations, and calculates an anomaly score. The first calculation unit 252 outputs the sum of the anomaly scores for all byte positions as the first anomaly score.
[0047] If the length comparison unit 251 determines that the number of similar normal packet data having the same length as the packet data to be evaluated is less than a predetermined threshold (step S14 in Figure 6: No), the second calculation unit 253 performs a second calculation process (step S16 in Figure 5).
[0048] As the second calculation process, the second calculation unit 253 calculates the edit distance between the evaluation target packet data and each similar normal packet data ( FIG. 7 ). The second calculation unit 253 outputs the smallest edit distance among the calculated edit distances as the second anomaly score. At the same time, the second calculation unit 253 estimates and detects insertion byte locations suspected of insertion or deletion byte locations suspected of deletion based on the calculated edit distance.
[0049] If the first anomaly score or the second anomaly score exceeds a predetermined threshold, the detection unit 25 detects that the evaluation target packet data is abnormal and issues an alert.
[0050] [First Calculation Process] The first calculation process will be described with reference to Fig. 8.
[0051] Specifically, the first calculation unit 252 directly treats the value of the first byte of the similar normal packet data extracted for comparison as a number between "0" and "255." The first calculation unit 252 calculates the anomaly score by applying one-dimensional anomaly detection using, for example, KDE (Kernel Density Estimator) (Reference 3). The first calculation unit 252 similarly calculates the anomaly scores for the second byte and subsequent bytes. The first calculation unit 252 performs one-dimensional detection based on a comparison of the numerical values for each byte position to calculate the anomaly score for each byte position. Reference 3: Duda, R. and Hart, P. (1973), "PATTERN CLASSIFICATION AND SCENE ANALYSIS," John Wiley & Sons. ISBN 0-471-22361-1.
[0052] The first calculation unit outputs the anomaly score in the range of a minimum of "0" to a maximum of "1." For example, a case where byte position L1 is the detection target will be described ((1) in FIG. 8).
[0053] The first calculation unit 252 calculates a probability density distribution of the numerical value of the byte appearing at byte position L1 of the similar normal packet data for each similar normal packet data. In other words, the first calculation unit 252 calculates a distribution centered on the number of bytes appearing in the similar normal packet and calculates the sum of these distributions ((2) in FIG. 8). For example, among the M similar normal packet data, the probability density distribution D1 of the first similar normal packet data, the probability density distribution D2 of the second similar normal packet data, and the probability density distribution D3 of the third similar normal packet data are calculated as the probability density distribution of the numerical value of the byte appearing at byte position L1.
[0054] The first calculation unit 252 obtains a probability density distribution Dt by taking the sum of each probability density distribution (e.g., probability density distributions D1 to D3) of the similar normal packet data. The first calculation unit 252 observes the probability density of the byte value of the evaluation target packet data on the probability density distribution Dt, and performs a normal / abnormal determination of byte position L1 of the evaluation target packet data ((3) in FIG. 8).
[0055] Specifically, the first calculation unit 252 compares a predetermined threshold (for example, 1 / 1024) with the probability density of the byte value of the packet data to be evaluated, and calculates the anomaly score.
[0056] If the probability density of the byte value of the packet data to be evaluated is greater than or equal to a threshold value (for example, byte values "17" or "21"), the first calculation unit 252 determines that this byte position L1 is normal and assigns an anomaly score of 0.
[0057] On the other hand, if the probability density of the byte value of the packet data to be evaluated is less than the threshold value, i.e., approximately "0" (for example, the byte value is "ff"), the first calculation unit 252 determines that this byte position L1 is abnormal and assigns an anomaly score of 1.
[0058] Furthermore, when the probability density of the evaluation target packet data is equal to or greater than the threshold but is close to the threshold (for example, the byte value "30"), the first calculation unit 252 may interpolate the anomaly score between 0 and 1 according to a predetermined rule. For example, the first calculation unit 252 assigns an anomaly score of 0.3 to the byte value "30".
[0059] In this way, the first calculation unit 252 determines whether there is no violation (normal) or a violation (abnormal) for each byte position based on the probability density distribution Dt ((3) in FIG. 8). Therefore, even if the byte value changes randomly at a byte position, this does not lead to overdetection.
[0060] The first calculation unit 252 then calculates an anomaly score for each byte position and outputs the sum of these anomaly scores as the final anomaly score (first anomaly score) for the packet data to be evaluated.
[0061] The first calculation unit 252 may use any anomaly detection method that can handle one-dimensional data, such as a non-parametric method or a naive Bayes method.
[0062] [Second Calculation Process] Next, the second calculation process will be described. In the second calculation process, the second calculation unit 253 calculates the edit distance between the evaluation target packet data and each similar normal packet data by using dynamic programming.
[0063] The second calculation unit 253 directly outputs the calculated edit distance as the anomaly score. Specifically, the second calculation unit 253 sets the smallest edit distance among the calculated edit distances as the second anomaly score.
[0064] Furthermore, by calculating the edit distance, the second calculation unit 253 can identify the location of, for example, an abnormal byte insertion and / or deletion, and directly identify how many packets have been inserted. Furthermore, if there is no similar normal packet data, the edit distance is considered to be large.
[0065] The second calculation unit 253 acquires the evaluation target packet data and M pieces of similar normal packet data from the length comparison unit 251. Then, the second calculation unit 253 compares the acquired similar normal packet data with the evaluation target packet data in order to identify inserted byte positions or deleted byte positions in the evaluation target packet data.
[0066] Specifically, the second calculation unit 253 calculates the edit distance between the packet data to be evaluated and each of the similar normal packet data by using dynamic programming. By calculating the edit distance, the second calculation unit 253 can identify an insertion byte location where insertion is suspected or a deletion byte location where deletion is suspected.
[0067] If there is similar normal packet data whose edit distance is less than the certain distance, the second calculation unit 253 selects the similar normal packet data whose edit distance is shortest from among the similar normal packet data whose edit distance is less than the certain distance as the final similar normal packet data.The second calculation unit 253 then estimates the insertion / deletion byte locations using the edit distance of the selected final similar normal packet data.
[0068] On the other hand, if the edit distances of all the acquired similar normal packet data are equal to or greater than the certain distance, the second calculation unit 253 processes the packet data to be evaluated as not having the final similar normal packet data corresponding to the packet data to be evaluated. Here, the certain distance is a parameter that can specify, for example, approximately 1 / 3 to 1 / 2 of the packet length.
[0069] [Procedure of Detection Processing] FIG. 9 is a diagram showing a procedure of detection processing according to the embodiment.
[0070] The detection device 20 acquires one piece of evaluation target packet data (step S21). The conversion unit 23 converts the evaluation target packet data into evaluation target vector data using the conversion model 231 (step S22).
[0071] The extraction unit 24 reads each comparison normal packet data from the comparison normal data group 151 (step S23), and converts each comparison normal packet data into comparison normal vector data using the conversion model 231 (step S24).
[0072] The extraction unit 24 extracts M pieces of normal vector data for comparison that have a relatively high degree of similarity with the vector data to be evaluated as normal vector data for similarity comparison (step S25).
[0073] The extraction unit 24 calculates the similarity between each of the comparison normal vector data and the evaluation target vector data, and extracts M similar normal packet data with relatively high similarity based on the calculated similarity (step S25).
[0074] The length comparison unit 251 compares the lengths of the M pieces of similar normal packet data with the evaluation target packet data, and determines whether the number of similar normal packet data pieces having the same length as the evaluation target packet data among the M pieces of similar normal packet data is equal to or greater than a predetermined threshold (step S26).
[0075] If the number of similar normal packet data having the same length as the packet data to be evaluated is greater than or equal to a predetermined threshold (step S26: Yes), the first calculation unit 252 performs a first calculation process (step S27) and outputs a first anomaly score.
[0076] If the number of quasi-normal packet data having the same length as the packet data to be evaluated is less than a predetermined threshold (step S26: No), the second calculation unit 253 performs a second calculation process (step S28) and outputs a second anomaly score.
[0077] The detection unit 25 determines whether the first anomaly score or the second anomaly score exceeds a predetermined threshold value (step S29).
[0078] If the first anomaly score or the second anomaly score exceeds the predetermined threshold (step S29: Yes), the detection unit 25 detects an abnormality in the evaluation target packet data and issues an alert (step S30).
[0079] If the first anomaly score or the second anomaly score is equal to or less than the predetermined threshold (step S29: No), the detection device 20 ends the detection process for this packet data to be evaluated.
[0080] [First Calculation Process] FIG. 10 is a flowchart showing the processing procedure of the first calculation process shown in FIG.
[0081] The first calculation unit 252 sets a parameter n, which indicates the position of the byte to be compared, to n=1 (step S31).
[0082] The first calculation unit 252 calculates a probability density distribution Dt by summing the values at byte position n of the same normal packet data (step S32). The first calculation unit 252 compares the probability density distribution Dt with the value at byte position n of the evaluation target packet data (step S33). The first calculation unit 252 then compares the value at byte position n of the evaluation target packet data with a predetermined threshold (e.g., 1 / 1024) and determines whether the value at byte position n of the evaluation target packet data is equal to or greater than the threshold (step S34).
[0083] If the numerical value at byte position n of the evaluation target packet data is equal to or greater than the threshold value (step S34: Yes), the first calculation unit 252 assigns 0 as the anomaly score for this byte position n (step S35). The first calculation unit 252 may interpolate the anomaly score between 0 and 1 according to a predetermined rule.
[0084] If the numerical value at byte position n of the evaluation target packet data is less than the threshold value (step S34: No), the first calculation unit 252 assigns 1 as the anomaly score for this byte position n (step S36).
[0085] If byte position n is not the last position (step S37: No), the first calculation unit 252 sets n = n + 1 (step S38), performs the processing from step S32 onwards for the numerical value of the next byte position, and calculates the anomaly score.
[0086] If byte position n is the last position (step S37: Yes), the first calculation unit 252 adds up the anomaly scores of all byte positions (step S39) and outputs the sum of the anomaly scores as the first anomaly score for the packet data to be evaluated.
[0087] [Second Calculation Process] FIG. 11 is a flowchart showing the processing procedure of the second calculation process shown in FIG.
[0088] The second calculation unit 253 selects one unselected piece of similar normal packet data from the predetermined number of similar normal packet data (step S41).
[0089] Next, the second calculation unit 253 calculates the edit distance between the evaluation target packet data and the selected similar normal packet data using dynamic programming (step S42).
[0090] Next, the second calculation unit 253 determines whether or not the selection of all of the predetermined number of similar normal packet data has been completed (step S43). If unselected normal packet data remains among the predetermined number of similar normal packet data (step S43: No), the second calculation unit 253 returns to step S41.
[0091] On the other hand, if selection of all of the predetermined number of similar normal packet data has been completed (step S43: Yes), the second calculation unit 253 determines whether there is similar normal packet data whose edit distance is less than a certain distance (step S44).
[0092] If similar normal packet data whose edit distance is less than the certain distance exists (step S44: Yes), the second calculation unit 253 executes the following process. In this case, the second calculation unit 253 selects the similar normal packet data whose edit distance is shortest among the similar normal packet data whose edit distance is less than the certain distance as the final similar normal packet data. Then, the second calculation unit 253 estimates the insertion / deletion byte location 19 using the edit distance between the selected final similar normal packet data and the evaluation target packet data (step S45).
[0093] On the other hand, if there is no similar normal packet data whose edit distance is less than the certain distance (step S44: No), the second calculation unit 253 determines that there is no final similar normal packet data (step S46).
[0094] The second calculation unit 253 then outputs the smallest edit distance among the calculated edit distances as the second anomaly score (step S47). The second calculation unit 253 also outputs the results of estimation and detection of insertion byte locations where insertion or deletion byte locations where deletion is suspected.
[0095] [Evaluation Experiment] The results of the evaluation experiment are shown below. 451 pieces of training packet data and 439 pieces of normal packet data for comparison were extracted from BACnet protocol communications (average packet length 93.5, maximum packet length 816) acquired in a small-scale experimental environment. The evaluation target packet data was artificially generated by rewriting random positions of 1 to 5 bytes in the normal packet data for comparison with random bytes. BERT's training data itself was also used as the normal packet data for comparison.
[0096] 12 and 13 are diagrams showing the results of the evaluation experiment. FIG. 12 shows the detection results obtained by the detection device 20. FIG. 13 shows the detection results obtained using the method described in Patent Document 1. The evaluation was performed using AUROC. For detection accuracy, an AUC score of 0.95 or higher is desirable.
[0097] 12 and 13, the detection accuracy of the detection device 20 was higher than that of the method described in Patent Document 1. Furthermore, as shown in Fig. 12, when detected by the detection device 20, the AUC score was significantly higher than 0.95 for data in which three or more bytes had been altered, meaning that detection was almost complete.
[0098] [Effects of the Embodiment] As shown in this evaluation experiment, according to the embodiment, it is possible to detect anomalies occurring in the contents of packets with high accuracy for any communication protocol with long packets.
[0099] [Application Example] This embodiment can be applied to an anomaly detection system for IoT devices, for example.
[0100] Specifically, a network sensor is placed on the IoT network and packets are captured. The captured packets are then used to train the encoding unit (BERT). After training is complete, normal packet data for comparison is set in the detection device 20. Packets acquired by the network sensor are then input into the detection device 20, and an anomaly score is calculated for each packet. This anomaly score is used to determine whether the packet to be evaluated is similar to the packet acquired during training (i.e., whether it is a normal packet).
[0101] [System Configuration of the Embodiment] The components of the learning device 10 and the detection device 20 are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of the functions of the learning device 10 and the detection device 20 is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.
[0102] Furthermore, all or any part of the processes performed by the learning device 10 and the detection device 20 may be realized by a CPU and a program analyzed and executed by the CPU. Furthermore, each process performed by the learning device 10 and the detection device 20 may be realized as hardware using wired logic.
[0103] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated can be changed as appropriate unless otherwise specified.
[0104] 14 is a diagram showing an example of a computer in which the learning device 10 and the detection device 20 are realized by executing a program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0105] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.
[0106] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes of the learning device 10 and the detection device 20 are implemented as program modules 1093 in which code executable by the computer 1000 is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to the functional configurations of the learning device 10 and the detection device 20 are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).
[0107] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes them.
[0108] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.
[0109] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.
[0110] REFERENCE SIGNS LIST 10 Learning device 11 Learning data collection unit 12, 23 Conversion unit 13 Learning control unit 14 Comparison normal data generation unit 15, 21 Storage unit 20 Detection device 22 Evaluation target data acquisition unit 24 Extraction unit 25 Detection unit 121, 231 Conversion model 151 Comparison normal data group 251 Length comparison unit 252 First calculation unit 253 Second calculation unit
Claims
1. A detection device comprising: an extraction unit that extracts a predetermined number of similar normal packet data that have a relatively high similarity to packet data to be evaluated from among a plurality of normal packet data based on a natural language processing model; and a detection unit that detects whether the packet data to be evaluated is abnormal based on an anomaly score calculated by comparing the predetermined number of similar normal packet data extracted by the extraction unit with the packet data to be evaluated.
2. The detection device according to claim 1, characterized in that the detection unit has: a first calculation unit that extracts same-length packet data having the same packet length as the packet data to be evaluated from the similar normal packet data, and compares the packet data to be evaluated with the same-length packet data for each byte to calculate a first anomaly score; and a second calculation unit that, when the packet length of the similar normal packet data is different from that of the packet data to be evaluated, calculates a second anomaly score based on an edit distance between the packet data to be evaluated and the similar normal packet data.
3. The detection device described in claim 2, characterized in that the first calculation unit performs one-dimensional detection based on a comparison of numerical values for each byte position between the packet data to be evaluated and the same-length packet data, calculates an anomaly score for each byte position, and sets the total value of the anomaly scores for all byte positions as the first anomaly score.
4. The detection device described in claim 2, characterized in that the second calculation unit calculates the edit distance between the packet data to be evaluated and the similar normal packet data, and sets the smallest edit distance among the calculated edit distances as the second anomaly score.
5. The detection device according to claim 1, wherein the extraction unit uses Bidirectional Encoder Representations from Transformers (BERT) as the natural language processing model.
6. A detection method executed by a detection device, comprising: a step of extracting a predetermined number of similar normal packet data having a relatively high similarity to packet data to be evaluated from among a plurality of normal packet data based on a natural language processing model; and a step of detecting whether the packet data to be evaluated is abnormal based on an anomaly score calculated by comparing the predetermined number of similar normal packet data extracted in the extraction step with the packet data to be evaluated.
7. A detection program for causing a computer to execute the steps of: extracting a predetermined number of similar normal packet data from among a plurality of normal packet data, the similarity of which is relatively high with packet data to be evaluated, based on a natural language processing model; and detecting whether the packet data to be evaluated is abnormal or not, based on an anomaly score calculated by comparing the predetermined number of similar normal packet data extracted in the extraction step with the packet data to be evaluated.
Citation Information
Patent Citations
Device and program for detecting improper access
JP2005250802A
