Label management system

Abstract

To distinguish the behavior of data from the occurrence of a failure to recovery into data at the time of the failure and data during response to the failure after it has occurred, and perform labeling appropriately.SOLUTION: A label management system that manages labels indicating events that occur on a network includes a trouble ticket DB that stores information on faults that occur on a network, an operation data DB that stores operation data collected from network devices that make up the network, and a label assignment unit that automatically provides labels indicating at least fault information obtained from the trouble ticket DB among events that occur on the network to the operation data DB.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a label management system and a label management apparatus. [Background technology]

[0002] In recent years, networks such as LTE (Long Term Evolution), 5G (5th generation mobile communication system), and FTTH (Fiber To The Home) have been widely used around the world. These networks provided by telecommunications carriers are used not only by humans but also by various devices such as sensors and automobiles, and their importance as social infrastructure is increasing day by day. In such network infrastructure, it is extremely important to prevent network failures, and appropriate network operation is essential to maintain network quality and provide stable services.

[0003] The scale of these networks is growing year by year, and human network operation is reaching its limits. As a solution, automated network operation using systems is being widely used. In particular, in recent years, with the evolution of AI / ML (Artificial Intelligence / Machine learning) technology, network operation using these technologies has attracted attention.

[0004] Generally, AI / ML technology is divided into two types: "unsupervised learning" and "supervised learning," with the latter requiring labeled data as training data (Figure 10). In an AI / ML system built using "supervised learning," it is possible to build a more accurate learning model by having it learn from labeled data as training data.

[0005] Non-Patent Document 1 discloses technology related to network operation using AI / ML technologies such as Autoencoder, SVM (Support Vector Machine), and LSTM (Long Short Term Memory), and technology related to handling network failures. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] A Survey of AIOps Methods for Failure Management Summary of the Invention [Problem to be solved by the invention]

[0007] However, Non-Patent Document 1 does not disclose a technology for labeling unlabeled data. As described above, in existing operation systems, although there is a system database that accumulates operation data, there is no system for labeling such data, and human operators and developers manually label data from various data sources, which takes a lot of time.

[0008] In addition, when a network failure occurs, the period from when the failure occurs to when it is restored is included, but the behavior of the data when the failure occurs and the behavior of the data when the failure is responded to after it occurs are significantly different (Figure 11). Therefore, if the period from when the failure occurs to when it is restored is learned as a single label (e.g., a failure), a significant degradation of accuracy will occur.

[0009] In addition, in existing network operations, it is common for there to be a database that stores records (also referred to as trouble tickets in this specification) that already describe the details of the faults, and a database that stores records (also referred to as work information in this specification) that describe the details of the work carried out on the network. The information stored as these records is the source of the above-mentioned labels, but database systems are often introduced on a large scale and are difficult to modify.

[0010] The present invention has been made in consideration of the above circumstances, and aims to provide a technology that distinguishes the behavior of data from the occurrence of a failure to recovery into data at the time of the failure and data during response to the failure after the failure, and appropriately labels the data. [Means for solving the problem]

[0011] (1) In order to achieve the above object, the present invention provides the following means: That is, the label management system of the present invention is a label management system that manages labels indicating events occurring on a network, and is characterized by comprising: a trouble ticket DB that stores information on faults occurring on the network, an operation data DB that stores operation data collected from network devices that constitute the network, and a label assignment unit that automatically provides, to the operation data DB, labels indicating at least fault information acquired from the trouble ticket DB among events occurring on the network.

[0012] (2) Furthermore, the label management system of the present invention further includes a work management DB that stores the details of work performed on the network, and the label assignment unit automatically provides a label indicating the work information obtained from the work management DB to the operation data DB.

[0013] (3) Furthermore, in the label management system of the present invention, the label assignment unit obtains information indicating the start time and end time of the work from the work management DB, and obtains information indicating a fault that occurred between the start time and the end time of the work from the trouble ticket DB, and provides the operation data DB label by distinguishing between a period during which a fault occurred and a period during which the work was being performed.

[0014] (4) In the label management system of the present invention, the operation data DB records the label provided by the label allocation unit in the operation data.

[0015] (5) Furthermore, the label management system of the present invention is characterized in that it further comprises an operations unit that detects failures and network faults occurring in network devices that make up the network, and provides fault information to at least the trouble ticket DB.

[0016] (6) Furthermore, a label management device of the present invention is a label management device that manages labels indicating events occurring on a network, and is characterized in that it acquires fault information occurring on the network from a trouble ticket DB, and provides at least labels indicating the fault information of the events occurring on the network to an operation data DB that stores operation data collected from network devices that constitute the network.

[0017] (7) Furthermore, the label management device of the present invention is characterized in that it obtains information indicating the start time and end time of work performed on a network from a work management DB that stores the details of the work, and obtains information indicating a fault that occurred between the start time and end time of the work from the trouble ticket DB, and provides a label to the operation data DB by distinguishing between a period during which a fault occurred and a period during which the work was performed. Effect of the Invention

[0018] According to the present invention, by distinguishing the behavior of data from the occurrence of a failure to recovery into data at the time of the failure and data during response to the failure after the failure and labeling the data appropriately, it is possible to provide better training data, and as a result, it is possible to build a more accurate learning model for an AI / ML system. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a label management system. [Diagram 2] FIG. 1 illustrates an example of a configuration of a trouble ticket. [Diagram 3] 13A and 13B are diagrams illustrating an example of a configuration of work information. [Figure 4] 13A and 13B are diagrams illustrating an example of a configuration of operation data. [Diagram 5] 13A and 13B are diagrams illustrating an example of operational data to which labels have been added. [Figure 6] FIG. 11 is a sequence diagram showing a labeling procedure when an operator inputs data into a trouble ticket DB. [Figure 7] 13 is a flowchart showing an example of a method for allocating labels to be applied. [Figure 8] FIG. 11 is a sequence diagram showing a labeling procedure when an operator inputs data into a trouble ticket DB. [Figure 9] 13 is a flowchart showing an example of a method for allocating labels to be applied. [Figure 10] FIG. 13 is a diagram showing an example of teacher data to which labels are assigned. [Figure 11] FIG. 13 is a diagram illustrating behavior of data after a network failure occurs. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The inventors noticed that the behavior of data when a failure occurs is significantly different from the behavior of data when the failure is responded to after the failure occurs, and that if the data from the failure occurrence to recovery is learned as a single label, the accuracy of the constructed learning model is low. They discovered that better training data can be provided by distinguishing the behavior of data from the failure occurrence to recovery into data at the time of the failure occurrence and data when the failure is responded to after the failure occurs and labeling them appropriately, which led to the present invention.

[0021] That is, the present invention is a label management system that manages labels indicating events occurring on a network, and is characterized by comprising: a trouble ticket DB that stores information on faults occurring on the network; an operation data DB that stores operation data collected from network devices that make up the network; and a label assignment unit that automatically provides labels indicating at least fault information obtained from the trouble ticket DB among events occurring on the network to the operation data DB.

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In order to facilitate understanding of the description, the same reference numerals are used to refer to the same components in the drawings of the respective embodiments, and duplicated descriptions will be omitted.

[0023] [1] Outline of the label management system 1 is a diagram showing a schematic configuration of a label management system. The label management system 1 includes a label allocation unit and an operation unit (including an operator 5), and is configured to be connectable to a trouble ticket DB 100, a work management DB 200, an operation data DB 300, a network 3, and an AI / ML operation system 7.

[0024] Network 3 is a network that is built and operated by a telecommunications carrier for the purpose of providing telecommunications services, and is a network to which AI / ML-based network operation is applied. AI / ML-based network operation refers to, but is not limited to, operations that use AI / ML to detect network failures and identify their causes.

[0025] In addition, the type of network may be one that is capable of generating pseudo-faults in NW nodes, which are components of the network, and that is capable of acquiring statistical information from the NW nodes. Examples of such networks include, but are not limited to, mobile networks composed of base stations such as 5G and mobile core equipment, and IP networks composed of routers such as FTTH.

[0026] The operator 5 is an element that operates the network, and is a human being or an automated system using a program or the like. The operator 5 detects failures that occur in the network devices that make up the network 3 and the network failures caused by them, identifies the cause of the detected network failure, and repairs the identified cause. In addition to dealing with network failures, network operation also includes maintaining communication services and opening up new services and users, and the operator operates the network devices to ensure healthy and stable network operation.

[0027] The trouble ticket DB100 is a database for storing detailed information about network failures that occur on the network 3. An operator 5 inputs detailed information about a network failure while or after responding to the network failure and stores the information in the trouble ticket DB100. FIG. 2 is a diagram showing an example of the configuration of a record (trouble ticket) stored in the trouble ticket DB100. Each record in the trouble ticket DB100 contains detailed information about the network failure, such as the time the network failure occurred, the time it was restored, the node (target node) where the network failure occurred, the failure type, the restoration method, and the impact on users.

[0028] The work management DB200 is a database for storing detailed information on work performed by the operator 5 on the network. Work is an action performed by the operator 5 on the network devices constituting the network 3, and includes, for example, changing the configuration, inserting a new line card, and restarting the device. The operator 5 enters detailed information on the work while or after responding to the work. FIGS. 3(a) and 3(b) are diagrams showing an example of the configuration of a record (single work information) stored in the work management DB200. Each record in the work management DB200 contains detailed information on the work performed by the operator 5 on the network 3, such as the work start time, work end time, node on which the work was performed (work node), and work content.

[0029] The operation data DB300 is a database that stores data indicating the performance of the network devices that constitute the network 3, which data is collected from the network devices. The data indicating the performance of the devices includes information related to the resources of the network devices and information related to events that have occurred in the network devices. The information related to the resources of the network devices indicates, for example, the CPU utilization rate and the number of user requests. FIGS. 4(a) and (b) are diagrams showing an example of the configuration of records (operation data) stored in the operation data DB300. The operation data includes information related to the resources of the network devices (time-series data) that is collected periodically, and event data that is collected when an event occurs in the network devices.

[0030] The label management system 1 is a system that acquires information from the above-mentioned trouble ticket DB 100, work management DB 200, and operation data DB 300, and performs labeling and label management for operation data.

[0031] The label management system 1 includes at least a label allocation unit, which acquires data from the trouble ticket DB 100 and the work management DB 200, determines the attribute of the data, such as whether the data is information indicating a fault or information indicating work, at the time recorded in each record, assigns a label using meta information indicating the determined attribute as label information, and provides the label to the operation data DB 300. Then, the operation data DB 300 assigns the provided label to the operation data. The label assignment procedure will be described in detail later. FIGS. 5(a) and (b) are diagrams showing an example of operation data to which a label has been assigned. The label management system 1 may further include an operation unit that performs the work of the operator described above.

[0032] The AI / ML operation system 7 is a system that obtains labeled operation data from the operation data DB300, trains an AI / ML model for network operation, and performs automatic network operation using the trained model.

[0033] [2] Labeling procedure Next, the label assignment procedure in the label management system will be described. The label management system according to the present invention starts operation when an operator registers information in at least one of the trouble ticket DB and the work management DB. Therefore, the label assignment procedure when the operator inputs data into the trouble ticket DB and the label assignment procedure when the operator inputs data into the work DB will be described.

[0034] (1) Labeling procedure when data is entered into the trouble ticket DB FIG. 6 is a sequence diagram showing a labeling procedure when an operator inputs data into the trouble ticket DB.

[0035] First, an operator inputs a trouble ticket as information (trouble information) about a network fault that has occurred, and stores the information in the trouble ticket DB (step S1).

[0036] As shown in Figure 2, a trouble ticket includes the time of occurrence (t fail_start ), recovery time (t fail_end ), target node (n fail ), fault type (f type ), recovery methods, and user impact. Trouble tickets can be entered by the operator through a GUI provided by the trouble ticket DB, or through an API provided by the trouble ticket DB.

[0037] Next, the trouble ticket DB notifies the label management system that new trouble information has been input (step S2). The notification to the label management system (step S2) is automatically performed when the new trouble information is input, and the label management system provides an API for receiving the trouble information, and the trouble ticket DB notifies the API using a protocol such as HTTP.

[0038] Next, the label management system checks the notified failure information, and confirms the failure impact time (occurrence time, recovery time) and the target node. After that, the label management system requests the work management DB to check whether there was any work performed during the failure impact time in the target node (step S3). Note that "work performed during the failure impact time" refers to work whose start time is after the failure occurrence time and before the failure recovery time.

[0039] Next, the work management DB checks whether there is work information that started at the specified time based on the request from the label management system, and collects the data (step S4). As shown in FIG. 3, the work information stored in the work management DB includes the work start time (t work_start ), and the end time of the work (t work_end ), working nodes (n work ), Work content (W type ) is included.

[0040] Next, if the work management DB contains a work that started at the specified time, the corresponding work information is sent to the label management system (step S5). If the work does not exist, a response is sent to the label management system to inform the label management system that the work does not exist.

[0041] Next, the label management system assigns a label to be attached to each piece of operational data for each time based on the information obtained in steps S2 and S5 (step S6).

[0042] Here, a method for allocating the labels to be applied will be specifically described. Fig. 7 is a flow chart showing an example of a method for allocating the labels to be applied. Although it is not necessary to follow this chart, the label management system needs to determine the attributes of the data, such as whether the data is information indicating a fault or information indicating an operation, and to distinguish between "fault" and "operation" and assign the label.

[0043] Therefore, first, in step S2, the label management system receives the trouble ticket from the trouble ticket DB and calculates the time (t fail_start ), (recovery time t fail_end )), target node (n fail ), fault content (fault type f type ) (step S6-1).

[0044] Next, in step S5, the label management system checks the results received from the work management DB, and checks whether there is any work that started during the failure affected time (step S6-2).

[0045] If no work is required in step S6-2, the fault is regarded as having been recovered from without work being performed, and a label is assigned as "fault" for the entire time period from the fault start time to the fault end time (step S6-3).

[0046] On the other hand, in step S6-2, if an operation exists, that is, the operation occurs at time t fail_start <start time t work_start<Recovery time t fail_end If there is data corresponding to the operation time (start time (t work_start ), end time (t work_end )), Work content (W type ) (step S6-4).

[0047] Based on the confirmation result of step S6-4, a label is assigned to the period from the problem start time to the work start time as "problem" and to the period from the work start time to the problem end time as "work" (step S6-5).

[0048] In this way, by assigning labels such that the period from the start time of the failure to the start time of the work is designated as "failure" and the period from the start time of the work to the end time of the failure is designated as "work," the behavior of the operational data from the occurrence of the network failure to recovery can be accurately distinguished between the behavior of the data at the time the failure occurred and the behavior of the data during response to the failure after it occurred, making it possible to provide appropriate training data.

[0049] The label information at each time and node assigned based on steps S6-1 to S6-5 is transmitted to the operation data DB (step S7).

[0050] The operation data DB checks the label information received from the label management system, and assigns the received label to the operation data at the corresponding time and node (step S8).

[0051] (2) Labeling procedure when data is entered into the work management database FIG. 8 is a sequence diagram showing a labeling procedure when an operator inputs data into the trouble ticket DB.

[0052] First, data on work carried out on the operator network is entered as work information and stored in the work management DB (step T1).

[0053] As shown in FIG. 3, the work information includes the work start time (t work_start ), and the end time of the work (t work_end ), working nodes (nwork ), Work content (W type The work information may be input by the operator through a GUI provided by the work management DB, or may be input through an API provided by the work management DB.

[0054] Next, the work management DB notifies the label management system that new work information has been input (step T2). The notification to the label management system (step T2) is automatically performed when the new work information is input, and the label management system provides an API for receiving the work information, and the work management DB notifies the API using a protocol such as HTTP.

[0055] Next, the label management system checks the notified work information, and confirms the work time (start time, end time) and the target node. After that, the label management system requests the trouble ticket DB to check whether any faults occurred during the work time in the target node (step T3). A "fault that occurred during the work time" is a fault where the work start time is after the fault occurrence time and before the fault recovery time.

[0056] Next, the trouble ticket DB checks whether there is a fault that occurred at the specified time based on a request from the label management system, and collects data (step T4). As shown in FIG. 2, the fault information stored in the trouble ticket DB is sorted by the time of occurrence (t fail_start ), recovery time (t fail_end ), target node (n fail ), fault type (f type ), recovery methods, and user impact.

[0057] If a fault occurs at the specified time, the corresponding fault information is sent to the label management system (step T5). If no fault exists, a reply is sent to the label management system to inform the label management system that no fault exists.

[0058] Next, the label management system assigns a label to be attached to each piece of operational data for each time based on the information obtained in steps T2 and T5 (step T6).

[0059] Here, a method for allocating the labels to be applied will be specifically described. Fig. 9 is a flowchart showing an example of a method for allocating the labels to be applied. It is not necessary to follow this chart, but the label management system needs to assign labels separately for failures and tasks.

[0060] Therefore, first, the label management system calculates the work time (work start time (t work_start ), and the end time of the work (t work_end )), check the node and work content (step T6-1).

[0061] Next, in step T5, the label management system checks the result received from the trouble ticket DB to check whether or not there is a fault that occurred at the time of the work (step T6-2).

[0062] If no fault is present in step T6-2, the task is deemed not to be fault-related, and a label is assigned as "task" for the entire time period from the task start time to the task end time (step T6-3).

[0063] On the other hand, if a fault exists in step T6-2, that is, at the occurrence time t fail_start <start time t work_start <Recovery time t fail_end If there is data corresponding to the failure, the failure impact time (occurrence time (t fail_start ), (recovery time t fail_end )) and the fault type (f type ) (Step T6-4).

[0064] Based on the confirmation result of step T6-4, a label is assigned to the period from the problem start time to the work start time as "problem" and to the period from the work start time to the problem end time as "work" (step T6-5).

[0065] In this way, by assigning labels such that the period from the start time of the failure to the start time of the work is designated as a "failure" and the period from the start time of the work to the end time of the failure is designated as a "work", the behavior of operational data from the occurrence of a network failure to recovery can be accurately classified into the behavior of the data at the time the failure occurred and the behavior of the data when responding to the failure after it occurred, making it possible to provide appropriate training data.

[0066] The label information for each time and node assigned based on steps T6-1 to T6-5 is transmitted to the operation data DB (step T7).

[0067] The operation data DB checks the label information received from the label management system, and assigns the received label to the operation data at the corresponding time and node (step T8).

[0068] As described above, according to the above embodiment, by appropriately labeling the data behavior from the occurrence of a failure to recovery, data at the time of the failure occurrence and data at the time of responding to the failure after the failure occurrence, appropriate training data can be provided, and as a result, a more accurate learning model can be constructed for an AI / ML system. [Explanation of symbols]

[0069] 1. Label Management System 3. Network 5 Operator 7 AI / ML Operational System 100 Trouble ticket DB 200 Work management DB 300 Operational Data DB

Claims

1. A label management system for managing labels indicating events occurring on a network, A trouble ticket database that stores information on troubles that occur on the network; an operation data DB for storing operation data collected from network devices constituting the network; a label assignment unit that automatically provides, to the operation data DB, a label indicating at least fault information obtained from the trouble ticket DB among events occurring on the network.

2. A task management DB for storing the details of tasks performed on the network is further provided.

2. The label management system according to claim 1, wherein the label assignment unit automatically provides the label indicating the work information acquired from the work management DB to the operation data DB.

3. 3. The label management system according to claim 2, wherein the label assignment unit obtains information indicating a start time and an end time of the work from the work management DB, and obtains information indicating a fault that occurred between the start time and the end time of the work from the trouble ticket DB, and provides labels to the operation data DB by distinguishing between a period during which a fault occurs and a period during which the work is being performed.

4. 4. The label management system according to claim 1, wherein the operation data DB records the label provided from the label allocation unit in the operation data.

5. 2. The label management system according to claim 1, further comprising an operation unit that detects failures and network troubles occurring in network devices that constitute the network, and provides failure information to at least said trouble ticket DB.

6. A label management device that manages labels indicating events occurring on a network, A label management device characterized by obtaining fault information occurring on a network from a trouble ticket DB, and providing a label indicating at least the fault information of an event occurring on the network to an operation data DB that stores operation data collected from network devices that constitute the network.

7. 7. The label management device according to claim 6, further comprising: a work management DB that stores the details of work performed on a network, and obtains information indicating the start and end times of the work; and a trouble ticket DB that obtains information indicating a failure that occurred between the start and end times of the work; and provides a label to the operation data DB by distinguishing between a period during which a failure occurred and a period during which the work was performed.

Images