Data transfer device, data transfer method, and computer program
The data transfer device efficiently extracts necessary data from existing environments by clustering data from target and similar networks, addressing inefficiencies in hierarchical clustering for AI network failure response, and enhancing AI learning model data supplementation.
Patent Information
- Application Number
- PCT/JP2023/018150
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-21
AI Technical Summary
Hierarchical clustering methods for AI network failure response are inefficient due to increased processing time with large data volumes and potential mixing of unusable data, necessitating a method to extract necessary data from existing environments effectively.
A data transfer device and method that acquires and clusters data from both target and similar environments, determining similarity and generating clusters to extract relevant data for AI learning models, reducing processing time and maintaining data accuracy.
Enables efficient extraction of necessary data for AI learning models by reducing the number of objects in data classification and calculation, while maintaining accuracy, thereby supplementing data for AI network failure response.
Smart Images

Figure JP2023018150_21112024_PF_FP_ABST
Abstract
Description
Data transfer device, data transfer method and computer program
[0001] The present invention relates to a data transfer device, a data transfer method, and a computer program.
[0002] One method of responding to network failures is to classify the vast amount of alarm and log information collected from the network using a clustering technique, compare the information collected from the network with rules created from past experience, and use artificial intelligence (AI) to identify and respond to the failure.In order to accurately detect network failures using AI and respond quickly to failure recovery in the above-mentioned network failure response work, it is necessary to collect a large amount of failure patterns and response methods in advance and have the AI learn from the large amount of collected data.
[0003] A known method is to use past data as training data to train AI on large amounts of data. This method combines and classifies data acquired from a target environment network and existing networks of similar environments using hierarchical clustering.
[0004] Di LI, Haruhisa NOZUE, Norio YAMAMOTO and Kenichi TAYAMA, "Study on data clustering efficiency for the shortage of training datafor network failure response AI," The Institution of Electronics, Information and Communication Engineers Society Conference, 2021-09. D. Li, K. Akashi, H. Nozue and K. Tayama, "A Mirror Environment to Produce Artificial Intelligence Training Data," in IEEE Access, vol. 10, pp. 24578-24586, 2022, doi: 10.1109 / ACCESS.2022.3154825.
[0005] However, with hierarchical clustering, the larger the amount of data, the longer the clustering process takes. Also, simply clustering the data can result in some unusable data being mixed in.
[0006] The present invention has been made in consideration of the above circumstances, and has as its object to provide a data transfer device, a data transfer method, and a computer program that are capable of extracting necessary data from an existing environment.
[0007] A data transfer device according to a first aspect of the present invention comprises a data management unit that acquires first data from a first environment and acquires second data from a second environment; a similar data determination unit that determines the similarity between the first data and the second data; a data classification unit that generates a plurality of first data clusters from clustering the first data and generates a plurality of second data clusters from clustering the second data; and a transfer determination unit that generates a plurality of third data clusters by combining and clustering a plurality of the first data clusters and a plurality of the second data clusters, and extracts the third data cluster that includes at least one of the first data clusters.
[0008] A data transfer method according to a second aspect of the present invention includes acquiring first data from a first environment, acquiring second data from a second environment, determining similarity between the first data and the second data, generating a plurality of first data clusters from clustering the first data, generating a plurality of second data clusters from clustering the second data, generating a plurality of third data clusters by combining and clustering the plurality of first data clusters and the plurality of second data clusters, and extracting the third data cluster including at least one of the first data clusters.
[0009] A computer program according to a third aspect of the present invention is a computer program that causes a computer to function as the data transfer device according to the first aspect.
[0010] According to the present invention, it is possible to provide a data transfer device, a data transfer method, and a computer program that are capable of extracting necessary data from an existing environment.
[0011] Fig. 1 is a block diagram showing an outline of an example configuration of a data transfer device according to an embodiment. Fig. 2 is a flowchart for explaining an example of the operation of a data transfer process by a data transfer device according to an embodiment. Fig. 3 is a diagram for explaining an example of the operation of a data transfer process by a data transfer device according to an embodiment.
[0012] Hereinafter, a data transfer device, a data transfer method, and a computer program according to an embodiment of the present invention will be described with reference to the drawings. In the following embodiments, parts with the same numbers perform similar operations, and redundant explanations will be omitted.
[0013] 1 is a block diagram illustrating an example configuration of a data transfer device 1 according to an embodiment. The data transfer device 1 according to this embodiment extracts data that can be used to train an AI model (hereinafter referred to as a learning model) that outputs a method for dealing with network failures in a target network environment (hereinafter referred to as a production environment (first environment)).
[0014] Furthermore, the data transfer device 1 of this embodiment extracts data from an existing environment similar to the production environment (hereinafter referred to as the "existing environment (second environment)") that can be used for training the learning model from multiple data contained in the existing environment. By supplementing the data from the production environment with the extracted data from the existing environment, it is possible to obtain a sufficient amount of data necessary for training the learning model.
[0015] The data transfer device 1 comprises a control unit 2, an input unit 7, a storage unit 8, a communication unit 9, and an output unit 10. The control unit 2, input unit 7, storage unit 8, communication unit 9, and output unit 10 are each connected via a bus communication line BUS and transmit and receive data to and from each other. Note that the control unit 2, input unit 7, storage unit 8, communication unit 9, and output unit 10 may also be configured to transmit and receive data to and from each other via wireless communication or the like.
[0016] The input unit 7 may include, for example, a user interface such as a mouse or keyboard for inputting changes to the values of various settings of the data transfer device 1, a microphone, a touch panel, a camera, various sensors, and the like.
[0017] The input unit 7 acquires multiple pieces of data necessary for the learning model to learn from the production environment and inputs (transmits) the data to the control unit 2. Similarly, the input unit 7 acquires multiple pieces of corresponding data in the existing environment and inputs (transmits) the data to the control unit 2.
[0018] For example, in this embodiment, since a network failure response task is assumed, the data necessary for the learning model to learn may be network log data expressed in text including natural language, symbols, etc. The function of the input unit 7 may be included in the data management unit 3 described later, in which case the data management unit 3 may be configured to acquire a data set of the production environment and a data set of the existing environment, and then transmit the data set of the production environment and the data set of the existing environment to the similar data determination unit 4 described later.
[0019] The storage unit 8 includes, for example, a main storage unit and an auxiliary storage unit. The main storage unit may include, for example, a read-only memory (ROM) and a random-access memory (RAM). The ROM is a non-volatile memory used exclusively for reading data, and can store data and various setting values used by the control unit 2 when performing various processes. The RAM may be used as a so-called work area for temporarily storing data when the control unit 2 performs various processes. The main storage unit in this embodiment is, for example, a RAM, and is used as a memory.
[0020] The auxiliary storage unit is a non-transitory computer-readable storage medium of a computer centered around the control unit 2. The auxiliary storage unit is, for example, an EEPROM (registered trademark) (electric erasable programmable read-only memory), a HDD (hard disk drive), or an SSD (solid state drive).
[0021] The auxiliary storage unit can store data used by the control unit 2 when performing various processes, data generated by the processes in the control unit 2, various setting values, etc. The auxiliary storage unit is a memory that stores various information, and stores information such as a production environment data set (first data), an existing environment data set (second data), a production environment data cluster (first data cluster), an existing environment data cluster (second data cluster), a reclassified data cluster (third data cluster), etc., which will be described later.
[0022] The communication unit 9 can transmit and receive various data, including data necessary for the learning model to learn (hereinafter referred to as supplementary data), between the data transfer device 1 and an external device (e.g., a learning model). The function of the communication unit 9 may also be included in the data management unit 3, in which case the data management unit 3 may be configured to transmit the supplementary data to the learning model.
[0023] The output unit 10 may include, for example, a display means such as a monitor that allows the processing results of the control unit 2 to be visually recognized, or an audio output means that allows the results to be audibly recognized by a speaker, etc. The output unit 10 may be configured to output image information and audio information to an externally connected display means or audio output means.
[0024] The control unit 2 includes at least one CPU (Central Process Unit), MPU (micro processing unit), GPU (Graphics Processing Unit), FPGA (field-programmable gate array), etc., and can realize various functions of the data transfer device 1 using programs such as system software, application software, or firmware stored in the storage unit 8.
[0025] The control unit 2 includes a data management unit 3, a similar data determination unit 4, a data classification unit 5, and a transfer determination unit 6. The data management unit 3 may include the functions of the input unit 7 as described above, and acquires data of the production environment and data of the existing environment, and transmits the data sets of the production environment and the existing environment to the similar data determination unit 4.
[0026] The data management unit 3 further has the function of managing the data of the entire data transfer device 1. That is, the data management unit 3 manages various information stored in the storage unit 8, and may also have the function of the communication unit 9 as described above, and transmits supplementary data to the learning model.
[0027] The similar data determination unit 4 determines the similarity between the data set of the production environment and the data set of the existing environment. For example, the similar data determination unit 4 performs morphological analysis on the data set of the production environment and the data set of the existing environment.
[0028] More specifically, the similar data determination unit 4 converts each of the data sets in the production environment and the existing environment into a set of morphemes. The similar data determination unit 4 calculates the set match rate for the converted morpheme sets using a set match rate judgment index such as the Jaccard index, the Dice index, or the Simpson index.
[0029] If the calculated matching rate between the morpheme sets is equal to or greater than a predetermined threshold, the similar data determination unit 4 determines that there is similarity between the data set of the production environment and the data set of the existing environment. On the other hand, if the calculated matching rate between the morpheme sets is less than the predetermined threshold, the similar data determination unit 4 determines that there is no similarity between the data set of the production environment and the data set of the existing environment.
[0030] The similar data determination unit 4 transmits the similarity determination result to the data management unit 3. The data management unit 3 determines the next process depending on the received determination result. For example, when the data management unit 3 receives a determination result indicating that there is similarity between the data set of the production environment and the data set of the existing environment, it transmits the data set of the production environment to the data classification unit 5 and the transfer determination unit 6, and transmits the data set of the existing environment to the data classification unit 5.
[0031] The data classification unit 5 classifies the data set of the production environment and the data set of the existing environment (first clustering process). That is, the data classification unit 5 performs clustering on the data set of the production environment to generate multiple production environment data clusters (first data clusters). The data classification unit 5 also performs clustering on the data set of the existing environment to generate multiple existing environment data clusters (second data clusters). The data classification unit 5 transmits the multiple production environment data clusters and the multiple existing environment data clusters to the transfer determination unit 6.
[0032] For example, the data classification unit 5 performs hierarchical clustering on each of the data sets of the production environment and the existing environment using a method such as the shortest distance method, the longest distance method, the group average method, Ward's method, etc. Note that the clustering method is not limited to the above, and for example, the data classification unit 5 may be configured to perform non-hierarchical clustering on each of the data sets of the production environment and the existing environment using a method such as the k-means method.
[0033] The transfer determination unit 6 performs clustering on a plurality of data clusters that combine the plurality of production environment data clusters generated by the first clustering process with the plurality of existing environment data clusters (second clustering process). The transfer determination unit 6 generates a plurality of reclassified data clusters (third data clusters) by combining the plurality of production environment data clusters with the plurality of existing environment data clusters and performing clustering.
[0034] The transfer determination unit 6 performs hierarchical clustering on multiple data clusters that combine multiple production environment data clusters and multiple existing environment data clusters using a method such as the shortest distance method, the longest distance method, the group average method, Ward's method, etc. Note that the clustering method is not limited to the above, and for example, the transfer determination unit 6 may be configured to perform non-hierarchical clustering on multiple data clusters that combine multiple production environment data clusters and multiple existing environment data clusters using a method such as the k-means method.
[0035] The transfer determination unit 6 extracts reclassified data clusters including at least one production environment data cluster from the generated reclassified data clusters. The transfer determination unit 6 transmits data on the existing environment in the existing environment data cluster included in the extracted reclassified data cluster to the data management unit 3. The data management unit 3 outputs the data on the existing environment included in the extracted reclassified data cluster as data to be supplemented to the learning model.
[0036] Next, a description will be given of the data transfer process including clustering and data extraction performed by the data transfer device 1 of this embodiment. Figure 2 is a flowchart for explaining an example of the operation of the data transfer process performed by the data transfer device 1 according to one embodiment. Figure 3 is a diagram for explaining an example of the operation of the data transfer process performed by the data transfer device 1 according to one embodiment.
[0037] In this embodiment, for example, a start command based on a user operation or a start command from a higher-level control device can be used as a trigger to start the process.
[0038] The data management unit 3 identifies a learning model to be trained (step S1). In this embodiment, the data transfer device 1 identifies a learning model to be used for network failure response work as a target to be trained.
[0039] Furthermore, the data management unit 3 selects the type of data necessary for the learning model to learn and sets the type of data as an initial value (step S2). For example, the data management unit 3 selects network log data as the type of data necessary for the learning model to learn and sets the network log data as an initial value.
[0040] The data management unit 3 acquires data from the production environment based on the set initial values (step S3). The data management unit 3 acquires data from the existing environment based on the set initial values (step S4). Note that the order of steps S3 and S4 may be reversed, or they may be acquired simultaneously.
[0041] If the similar data determination unit 4 determines that there is similarity between the data set of the production environment and the data set of the existing environment (step S5, YES), the data classification unit 5 performs clustering on the data set of the production environment to generate multiple production environment data clusters (step S6).The data classification unit 5 further performs clustering on the data set of the existing environment to generate multiple existing environment data clusters (step S7).The order of steps S6 and S7 may be reversed, or they may be performed simultaneously.
[0042] The transfer determination unit 6 generates multiple reclassified data clusters by combining the multiple production environment data clusters generated in step S6 and the multiple existing environment data clusters generated in step S7 and performing clustering (step S8).
[0043] The transfer determination unit 6 extracts data of the existing environment from the reclassified data clusters including at least one production environment data cluster from among the plurality of reclassified data clusters in step S8 (step S9).
[0044] The data management unit 3 treats the extracted data from the existing environment as supplementary data necessary for the learning model to learn, and outputs it to the learning model, thereby completing the process (step S10).On the other hand, if the similar data determination unit 4 determines that there is no similarity between the data set of the production environment and the data set of the existing environment (step S5, NO), the data transfer device 1 completes the process.
[0045] By the data transfer device 1 repeatedly performing the above steps S1 to S10 in multiple different existing environments, the learning model can obtain the production environment data necessary for learning and supplementary data for multiple different existing environments.
[0046] According to the data transfer device 1 of the embodiment, by performing the data clustering process twice, the number of data classification / extraction targets is reduced, and efficient processing is realized. Furthermore, by performing clustering processing on data within the same environment in the first clustering process by the data transfer device 1, the number of targets for the second clustering process is reduced, and it is possible to reduce the amount of calculation required for processing while maintaining the accuracy of data classification.
[0047] That is, according to this embodiment, it is possible to provide a data transfer device, a data transfer method, and a computer program that are capable of extracting necessary data.
[0048] In the above embodiment, the data management unit 3, similar data determination unit 4, data classification unit 5, transfer determination unit 6, and storage unit 8 may be mounted on different devices. In this case, the multiple devices included in the data transfer device 1 may share various information stored in the storage unit 8.
[0049] The program according to the embodiment may be transferred in a state where it is stored in an electronic device, or may be transferred in a state where it is not stored in an electronic device. In the latter case, the program may be transferred via a network, or may be transferred in a state where it is stored in a storage medium. The storage medium is a non-transitory tangible medium. The storage medium is a computer-readable medium. The storage medium may be in any form, such as a CD-ROM or a memory card, as long as it is capable of storing the program and is computer-readable.
[0050] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.
[0051] DESCRIPTION OF SYMBOLS 1... Data transfer device 2... Control unit 3... Data management unit 4... Similar data determination unit 5... Data classification unit 6... Transfer determination unit 7... Input unit 8... Storage unit 9... Communication unit 10... Output unit
Claims
1. A data transfer device comprising: a data management unit that acquires first data from a first environment and second data from a second environment; a similar data determination unit that determines the similarity between the first data and the second data; a data classification unit that generates a plurality of first data clusters from clustering of the first data and generates a plurality of second data clusters from clustering of the second data; and a transfer determination unit that generates a plurality of third data clusters by combining and clustering a plurality of the first data clusters and a plurality of the second data clusters, and extracts the third data cluster including at least one of the first data clusters.
2. The data transfer device according to claim 1, wherein the first data and the second data are data expressed as text, and the similar data determination unit generates morpheme sets from morphological analysis of each of the first data and the second data, and determines that there is similarity if the matching rate between the morpheme sets is equal to or greater than a predetermined threshold.
3. The data transfer device according to claim 1, wherein the data classification unit, when the similar data determination unit determines that there is similarity between the first data and the second data, generates a plurality of first data clusters from clustering the first data cluster, and generates a plurality of second data clusters from clustering the second data cluster.
4. The data transfer device according to claim 1, wherein the data management unit acquires the first data and the second data contained in the extracted third data cluster, and outputs them to a learning model.
5. A data transfer method comprising: acquiring first data from a first environment and acquiring second data from a second environment; determining a similarity between the first data and the second data; generating a plurality of first data clusters from clustering the first data and generating a plurality of second data clusters from clustering the second data; generating a plurality of third data clusters by combining and clustering the plurality of first data clusters and the plurality of second data clusters; and extracting the third data cluster including at least one of the first data clusters.
6. A computer program causing a computer to function as the data transfer device according to any one of claims 1 to 4.
Citation Information
Patent Citations
Systems, methods and media for testing computer-executable code involving sensitive-information domains
US10860466B1
Predictive build quality assessment
US20220300280A1
