Data generation device, data generation method, and program

Abstract

According to an embodiment of the present invention, this data generation device is equipped with: a storage part that stores log data on text bases collected from a network in a target environment; and a processor. The processor comprises a unique extraction processing unit, a word list generating unit, a word vector generating unit, a clustering processing unit, and a data generating unit. The unique extraction processing unit extracts a unique log from log data and generates post-extraction log data. The word list generating unit divides the post-extraction log data into a plurality of words by means of morphological analysis and generates a word list. The word vector generating unit vectorizes the word list and generates a plurality of word vectors in a vector space of multiple dimensions. The clustering processing unit clusters the plurality of word vectors on the basis of a distance index between the plurality of word vectors in the vector space.

Description

[Technical Field]

[0001] One aspect of the present invention relates to a data generation device, a data generation method, and a program that generate training data for a machine learning model that detects an abnormality such as a failure. [Background technology]

[0002] Systems are developed with the ultimate goal of being applied to a target production environment. However, before the product is released, it is essential to conduct testing in a verification environment or staging environment that mimics the production environment. The same is true for AI (Artificial Intelligence) development, which has increasingly been discussed in conjunction with MLOps (Machine Learning Operations).

[0003] In recent years, attempts have been made to operate networks by using AI models trained on learning data related to network operations. To train an AI model, it is necessary to provide a large amount of data on the network's normal operating state (normal state data) and data on when a failure occurs (failure data) to the untrained model. However, since reproducing a failure in a production environment involves significant risks, it is difficult to collect a large amount of this data. Therefore, a technology has been proposed in which a mirror environment of the production environment is constructed and both the normal state data and the abnormal state data required for machine learning are generated in the mirror environment (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] D. Li, K. Akashi, H. Nozue and K. Tayama, "A Mirror Environment to Produce Artificial Intelligence Training Data," in IEEE Access, vol. 10, pp. 24578-24586, 2022, doi: 10.1109 / ACCESS.2022.3154825. [Non-patent document 2] L. Han, F. Gao, Z. Li and OA Dobre, "Low Complexity Automatic Modulation Classification Based on Order-Statistics," in IEEE Transactions on Wireless Communications, vol. 16, no. 1, pp. 400-411, Jan. 2017, doi: 10.1109 / TWC.2016.2623716. Summary of the Invention [Problem to be solved by the invention]

[0005] Log data collected from a mirror environment that mimics the target environment and the log data from the target environment are often text-based. By comparing the log data acquired in each environment and clustering or rewriting alarms with the same meaning, events for the target environment can be generated from the learned events created in the mirror environment. Attribute information, including response methods, can also be used as is.

[0006] However, the number and variety of network logs collected from the target environment is enormous, and as the number of words increases, so does the time required for clustering processing and learning. Furthermore, as the operation period progresses, the volume of logs collected from the target network increases unilaterally. Ultimately, the cumulative processing time for clustering between the target environment and the mirror environment becomes slower. If this problem is left unaddressed, log analysis will take an extremely long time, and a solution is desired.

[0007] The present invention has been made in light of the above circumstances, and aims to provide a technique that can reduce the time required for log analysis. [Means for solving the problem]

[0008] According to one aspect of the present invention, a data generation device includes a memory unit that stores text-based log data collected from a network of a target environment, and a processor. The processor includes a unique extraction processing unit, a word list generation unit, a word vector generation unit, a clustering processing unit, and a data generation unit. The unique extraction processing unit extracts unique logs from the log data to generate extracted log data. The word list generation unit divides the extracted log data into multiple words using morphological analysis to generate a word list. The word vector generation unit vectorizes the word list to generate multiple word vectors in a multi-dimensional vector space. The clustering processing unit clusters the multiple word vectors based on a distance index between the multiple word vectors in the vector space. The data generation unit generates events for the target environment from learned events created in a mirror environment of the target environment based on the clustering results. [Effects of the Invention]

[0009] According to one aspect of the present invention, a technique can be provided that can reduce the time required for log analysis. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a system to which a data generating device according to an embodiment of the present invention is applied. [Figure 2] FIG. 2 is a block diagram showing an example of the data generating device 3 shown in FIG. [Figure 3] FIG. 3 is a functional block diagram showing an example of the data generating device 3 shown in FIG. [Figure 4] FIG. 4 is a flowchart showing an example of a clustering processing procedure according to the embodiment. [Figure 5] FIG. 5 is a flowchart illustrating an example of a processing procedure in the cleansing process. [Figure 6] FIG. 6 is a flowchart illustrating an example of a processing procedure in the cluster determination processing. [Figure 7] FIG. 7 is a flowchart showing an example of the processing procedure in [File Reading and Preprocessing (Step S72 in FIG. 6)]. [Figure 8] FIG. 8 is a flowchart showing an example of the processing procedure of the clustering determination process (step S74 in FIG. 6). [Figure 9] FIG. 9 is a diagram showing the results of clustering in relation to whether cleansing has been performed. [Figure 10] FIG. 10 is a graph showing an example of processing time in the case of only the unique extraction process. [Figure 11] FIG. 11 is a graph showing an example of the processing time required for clustering including the unique extraction process and the clustering process. [Figure 12] FIG. 12 is a diagram showing another example of the result of clustering performed without cleansing. [Figure 13] FIG. 13 is a flowchart showing an example of a procedure for an existing hierarchical clustering process. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [One embodiment] <Configuration> Fig. 1 is a diagram showing an example of a system to which a data generation device according to an embodiment of the present invention is applied. The system shown in Fig. 1 is a system that operates a target system 1, such as a network system, by using an AI model 2. The system further includes a data generation device 3. The data generation device 3 generates training data for training the AI ​​model 2.

[0013] 2 is a block diagram showing an example of the data generating device 3 shown in FIG. The data generating device 3 is a computer including a processor 10, a memory 20, a storage 30, an input / output interface (I / F) 40, and a bus 45 interconnecting these components. The input / output I / F 40 establishes communication links between the data generating device 3 and the target system 1 and the AI ​​model 2, and transmits and receives various types of data. The processor 10 is a computing device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and realizes the processing functions of the embodiment in accordance with a program loaded from the storage 30 to the memory 20.

[0014] Fig. 3 is a functional block diagram showing an example of the data generating device 3 shown in Fig. 2. In Fig. 3, a memory 20 is a semiconductor memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory).

[0015] The storage 30 is a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD), and stores a program 30g for implementing the processing according to the embodiment in addition to basic software such as an operating system (OS). That is, the program 30g can be installed in the data generation device 3.

[0016] The storage 30 also stores log data 30a, extracted log data 30b, a word list 30c, a regenerated word list 30d, word vectors 30e, and clustering results 30f. Of these, the log data 30a is text-based log data collected from the network of the target environment.

[0017] The processor 50 includes, as processing functions according to one embodiment of the present invention, a unique extraction processing unit 50a, a word list generation unit 50b, a cleansing processing unit 50c, a word vector generation unit 50d, a clustering processing unit 50e, and a data generation unit 50f.

[0018] The unique extraction processing unit 50a, the word list generation unit 50b, the cleansing processing unit 50c, the word vector generation unit 50d, the clustering processing unit 50e, and the data generation unit 50f are realized by the processor 10 executing a program loaded into the memory 20. In other words, the program 30g includes instructions for causing the processor 50 to function as the unique extraction processing unit 50a, instructions for causing the processor 50 to function as the word list generation unit 50b, instructions for causing the processor 50 to function as the cleansing processing unit 50c, instructions for causing the processor 50 to function as the word vector generation unit 50d, instructions for causing the processor 50 to function as the clustering processing unit 50e, and instructions for causing the processor 50 to function as the data generation unit 50f.

[0019] The unique extraction processor 50a extracts unique logs from the log data to generate extracted log data, which is sent to the storage 30 and stored as extracted log data 30b.

[0020] The word list generator 50b generates a word list by dividing the post-extraction log data 30b into a plurality of words, for example, by morphological analysis. The word list is sent to the storage 30 and stored as a word list 30c.

[0021] The cleansing processing unit 50c performs cleansing processing to remove unnecessary words from the word list 30c, and generates a regenerated word list. The regenerated word list is sent to the storage 30 and stored as a regenerated word list 30d.

[0022] The word vector generation unit 50d vectorizes the regenerated word list 30d to generate a plurality of word vectors in a multi-dimensional vector space. The word vectors are sent to the storage 30 and stored as word vectors 30e.

[0023] The clustering processor clusters the word vectors based on the distance index between the word vectors in the vector space. The clustering result is sent to the storage 30 and stored as the clustering result 30f.

[0024] The data generation unit 50f generates events for the target environment from the learned events created in the mirror environment of the target environment based on the clustering result 30f.

[0025] <effect> Next, the operation of the above configuration will be described. 4 is a flowchart showing an example of a clustering processing procedure according to an embodiment. In FIG. 4, the processor 50 reads the log data 30a into the memory 20 (step S1) and performs preprocessing on the data (step S2). The preprocessing in step S2 may be general processing such as shaping and extraction for utilizing text data.

[0026] Next, the processor 50 performs a unique extraction process, for example, by deleting duplicate lines on a line-by-line basis (step S3), to generate post-extraction log data 30b. Next, the processor 50 performs a word segmentation process on the post-extraction log data 30b using morphological analysis (step S4), thereby generating a word list 30c.

[0027] Next, the processor 50 performs a cleansing process (step S5) to remove words that have no meaning for clustering from the divided word list (word list 30c). For example, words to be removed include numeric numbers for identifying network devices, or serial numbers or numeric numbers for simple counting. The processor 50 removes such words that reflect characteristics unique to the network from the word list 30c to generate a regenerated word list 30d.

[0028] Furthermore, the processor 50 generates a word vector 30e from the regenerated word list 30d through a vectorization process (step S6). Finally, the processor 50 calculates a distance index between vectors in a vector space from the word vector 30e using a method such as the shortest distance method, the longest distance method, or the group average method.

[0029] Clustering is a method for grouping data based on the similarity between the data. Clustering methods are broadly divided into hierarchical methods and partition optimization methods.

[0030] When clustering network logs, there is a method to obtain a hierarchical structure by repeatedly merging two clusters that are close to each other based on a distance function between clusters. On the other hand, there is also a non-hierarchical method that groups data without creating a hierarchy.

[0031] Distance indices in hierarchical clustering can be calculated using the shortest distance method, the longest distance method, or the group average method, etc. These indices can be calculated using Euclidean distance or cosine similarity, and the values ​​can be discriminated using a predetermined threshold to perform clustering.

[0032] For example, the Euclidean distance between n-dimensional vectors (x, y) is defined by equation (1).

[0033]

number

[0034] Euclidean distance is the most common distance measure between data in clustering.

[0035] 5 is a flowchart showing an example of the processing steps in the cleansing process (step S5 in FIG. 4). In FIG. 5, the processor 50 generates a word list 30c (step S51), and then performs data cleansing processing on the word list (step S52). Furthermore, the processor 50 regenerates the word list from the remaining words to generate a regenerated word list 30d (step S53).

[0036] 6 is a flowchart showing an example of the processing procedure of the cluster determination process (step S7 in FIG. 4). In FIG. 6, the processor 50 reads the cluster history (step S71), and performs file reading and preprocessing (step S72). Next, the processor 50 recalculates the center coordinates of the cluster (step S73), and performs clustering determination processing (step S74). Then, the processor 50 outputs the cluster history obtained by the clustering determination processing (step S75).

[0037] Fig. 7 is a flowchart showing an example of the processing procedure for "File Reading and Preprocessing (Step S72 in Fig. 6)." In Fig. 7, the processor 50 reads the input file (Step S21) and performs preprocessing (Step S22). Next, the processor 50 combines the cluster history obtained in the procedure in Fig. 6 with the input file to create a word list (Step S23), and creates one-hot data from the word list (Step S24).

[0038] 8 is a flowchart showing an example of the processing procedure in the clustering determination process (step S74 in FIG. 6). In FIG. 8, the processor 50 reads the one-hot data line by line (step S41) and checks whether or not there is data (step S42). If there is no data (No), the processing ends.

[0039] If the data can be read (Yes in step S42), the processor 50 calculates the Euclidean distance from the existing cluster (step S43) and determines whether there is a cluster within the threshold (step S44). If the Euclidean distance is equal to or less than the threshold (Yes in step S44), the processor 50 adds the word vector to the nearest cluster (step S45). If the Euclidean distance is greater than the threshold (No in step S44), the processor 50 registers the word vector in a new cluster (step S46). After step S45 or step S46, the processor updates the center coordinates of the cluster (step S47), and the processing procedure returns to step S41.

[0040] Figure 9 shows the clustering results with and without cleansing. In Figure 9, for example, seven log data files were integrated as sample data, providing 850,000 lines of log data. From this, data corresponding to messages was extracted and used for clustering. When processed on a PC server with the same specifications, the case without cleansing took 13 hours and 15 minutes. In contrast, the case with cleansing completed processing in just 35 minutes. In this case, the number of words was reduced by 45% compared to the case without cleansing, and the execution time was reduced by 96%.

[0041] Figure 10 is a graph showing an example of the processing time for unique extraction processing only. The horizontal axis is in seconds. The results of the distance calculation to the cluster generally increase steadily.

[0042] Figure 11 is a graph showing an example of the processing time required for clustering, including unique extraction processing and clustering processing. The horizontal axis is in seconds. The update of the results of distance calculations between clusters increases exponentially. This indicates that the update process of the results of distance calculations between clusters is a bottleneck. For example, the graph shows that it took 60,000 seconds (16 hours) to process approximately 10,000 rows.

[0043] This process calculates the Euclidean distance between all clusters and new records (rows) and adds the results to a list, so the list grows larger as the number of clusters increases. Moreover, since the sample data before extracting unique logs contains approximately 850,000 rows, processing takes several days.

[0044] Figure 12 shows another example of the results of clustering performed without cleansing. The CPU specifications were slightly lower than those in Figure 9. As you can see, the execution time exceeded 35 hours.

[0045] Fig. 13 is a flowchart showing an example of the processing procedure of an existing hierarchical clustering. In the existing technology, unlike the flowchart in Fig. 4, neither the unique extraction process (step S3) nor the cleansing process (step S5) is performed. Therefore, as the amount of log data collected from the network to be managed increases, the processing time required for clustering the logs becomes monotonically longer.

[0046] <Effects> As described above, in the embodiment, data that does not affect the clustering process is deleted from the log data in advance to reduce the clustering process time. Here, examples of data to be deleted include duplicate lines in the log and IDs for identifying devices and information.

[0047] That is, in the embodiment, before processing such as clustering and learning is performed on log data collected from a target network, unnecessary data is not provided to subsequent processing (cluster determination processing and learning processing). In other words, unique extraction of words that affect the message classification judgment results and duplicate data that takes up calculation time, and data cleansing are performed. This makes it possible to significantly reduce duplicate data such as numbers that simply identify devices, the number of pieces of equipment, and numbers that increase over time, thereby reducing the number of data to be processed. Because the number of data to be processed is reduced, the overall learning time can also be significantly reduced.

[0048] The amount of collected log data increases over time. As the amount of data increases, the subsequent processing and learning time also increases. The collected log data contains data that is meaningless in subsequent processing, such as large amounts of numbers and duplicated text. Pre-processing removes this data, reducing the amount of data to be processed and significantly shortening processing time, thereby increasing speed.

[0049] In particular, when sequentially processing large amounts of text data, the larger the amount of target data, the longer the processing time. By deleting duplicate data and unnecessary feature data, it is possible to process only data with the necessary features. This significantly reduces processing time and promotes speed.

[0050] Clustering of network logs involves dividing the logs into words using morphological analysis to create a word list. One possible approach is to vectorize the word list using a method such as one-hot, and then calculate the Euclidean distance for each vector to separate them into clusters. The processing time required to calculate the distance increases as the number of words resulting from morphological analysis increases.

[0051] Network logs are automatically generated by each device based on set specifications. As a result, there are many duplicate logs. Therefore, clustering all logs, including duplicates, would waste computer resources and processing time.

[0052] Therefore, in the embodiment, unique logs are extracted and data cleansing is performed to reduce the amount of data provided to the subsequent stage as much as possible, thereby facilitating faster processing.

[0053] Logs collected from a network contain many numerical numbers used simply for differentiation, such as numbers for identifying devices that make up the network and Syslog record numbers output by the OS. If these numerical numbers were used directly as clustering targets to generate clusters, they would be useless for clustering message texts and would require a considerable amount of processing time. As explained in the embodiments, by preprocessing the word list to perform word cleansing before clustering, the amount of data to be clustered can be significantly reduced and processing time can be accelerated.

[0054] As a result, according to the embodiment, it is possible to reduce the time required for log analysis, thereby shortening the time required for training an AI model related to network operation and speeding up the total processing time. Furthermore, in network management work, it is possible to improve the efficiency of analyzing the status of the managed network by collecting alarm information from the network to be managed and comparing and analyzing it with learned alarm information.

[0055] The present invention is not limited to the above-described embodiments. For example, in FIG. 1, the target system 1 may be, for example, a server device, a communication device, an office device, a medical device, an in-vehicle device, or various types of home appliances. The AI ​​model 2 and the data generation device may be included in the target system 1 or may be connected to each other via a communication line. The data generation device may be included in the AI ​​model 2 or may be connected to each other via a communication line.

[0056] In addition, any type of AI model can be applied, such as DNN (Deep Neural Network), CNN (Convolutional Neural Network), or RNN (Recurrent Neural Network).

[0057] 4, the purpose can be achieved by omitting the cleansing process (step S5) and executing the unique extraction process (step S3), thereby reducing the amount of data required for clustering and shortening the processing time. In this configuration, the word vector generation unit 50d vectorizes the word list 30c generated by the word list generation unit 50b to generate multiple word vectors 30e.

[0058] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0059] 1. Target system 2. AI model 3...Data generation device 10...Processor 20...Memory 30…Storage 30a...Log data 30b...Extracted log data 30c...Word list 30d…Regenerated word list 30e...word vector 30f…Clustering results 30g...Program 40...Input / output interface 45...Bus 50...processor 50a...Unique extraction processing section 50b...Word list generation section 50c...Cleansing treatment section 50d...Word vector generation section 50e...Clustering processing section 50f...Data generation section.

Claims

1. a storage unit for storing text-based log data collected from the network of the target environment; a processor; The processor: a unique extraction processing unit that extracts a unique log from the log data by deleting at least duplicated lines in the log data and IDs for identifying devices and information, and generates extracted log data; a word list generation unit that divides the extracted log data into a plurality of words by morphological analysis to generate a word list; a word vector generation unit that vectorizes the word list to generate a plurality of word vectors in a multi-dimensional vector space; a clustering processing unit that clusters the plurality of word vectors based on a distance index between the plurality of word vectors in the vector space; a data generation unit that generates events for the target environment from learned events created in a mirror environment of the target environment based on the results of the clustering.

2. The processor: a cleansing processing unit that removes unnecessary words from the word list by a cleansing process to generate a regenerated word list; The data generating device according to claim 1 , wherein the word vector generating unit generates the word vectors by vectorizing the regenerated word list.

3. The data generating device according to claim 2 , wherein the cleansing processing unit generates the regenerated word list by removing words that reflect characteristics unique to the network from the word list.

4. The data generating device according to claim 1 , wherein the distance measure is a Euclidean distance in the vector space.

5. A data generation method by a computer including a memory unit that stores text-based log data collected from a network of a target environment and a processor, the method comprising: a step in which the processor extracts, from the log data, a unique log in which at least duplicate lines in the log data and IDs for identifying devices and information have been deleted, and generates extracted log data; a step of the processor dividing the extracted log data into a plurality of words by morphological analysis to generate a word list; the processor vectorizing the word list to generate a plurality of word vectors in a multi-dimensional vector space; the processor clustering the plurality of word vectors based on a distance metric between the plurality of word vectors in the vector space; and generating events for the target environment from learned events created in a mirror environment of the target environment based on the results of the clustering by the processor.

6. A program installable on a computer having a processor and a storage unit that stores text-based log data collected from a network of a target environment, the program comprising: The processor, an instruction to function as a unique extraction processing unit that extracts a unique log from the log data by deleting at least duplicate lines in the log data and IDs for identifying devices and information, and generates extracted log data; an instruction to function as a word list generation unit that divides the extracted log data into a plurality of words by morphological analysis and generates a word list; an instruction to function as a word vector generator that vectorizes the word list and generates a plurality of word vectors in a multi-dimensional vector space; an instruction to function as a clustering processor that clusters the plurality of word vectors based on a distance index between the plurality of word vectors in the vector space; and instructions to cause the program to function as a data generation unit that generates events for the target environment from learned events created in a mirror environment of the target environment based on the results of the clustering.

Images