Network operation system, method, and program using ai model

Abstract

To reduce capacity of a database for AI model deduction by calculating importance of operation data to be used for deduction in an AI model and not using operation data having low importance for the deduction.SOLUTION: A system for operating a network by using a deduction result obtained by applying operation data collected from the network to an AI model acquires operation data for each information term from the network, stores the acquired operation data in a database, trains the AI model using the operation data, calculates importance for each term of the operation data using which the AI model is trained, deletes operation data of a term whose importance does not satisfy a predetermined upper condition from the database, re-trains the AI model using operation data of a term whose importance satisfies the predetermined upper condition, and applies the operation data of the term whose importance satisfies the predetermined upper condition to the re-trained AI model to output a deduction result.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a network operation system, method, and program that uses an AI model, and in particular to a network operation system, method, and program that can reduce the capacity of an inference database used for inference using an AI model. [Background technology]

[0002] The scale of networks such as LTE, 5G, and FTTH provided by telecommunications carriers is expanding year by year, and human network operation is reaching its limits. As a solution to this problem, automation of network operation using systems is being widely used. In particular, in recent years, with the advancement of AI / ML (artificial intelligence / machine learning) technology, network operation utilizing these technologies has attracted attention. Non-Patent Document 1 discloses technological trends related to network operation using AI / ML technology, particularly methods for responding to failures. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] A Survey of AIOps Methods for Failure Management (https: / / dl.acm.org / doi / abs / 10.1145 / 3483424) Summary of the Invention [Problem to be solved by the invention]

[0004] In AI-based network operations, operational data is essential for model learning (training) and inference. Operational data includes various items such as information about the hardware of each system, detailed process information, and application / service information.

[0005] To detect failures quickly, operational data needs to be stored in a real-time database. However, databases with high real-time performance are generally expensive, so the larger the amount of operational data, the higher the operational costs.

[0006] Therefore, it is desirable to reduce the amount of data, especially in real-time databases used for inference. However, deleting too much operational data could reduce the amount of input data for the AI ​​model, which could degrade the accuracy of the AI ​​model and result in a deterioration in the quality of network operations.

[0007] Furthermore, as the scale of the network expands and operational procedures increase, the number of AI models increases, which may result in an increase in the amount of data required for inference. This may necessitate additional reduction of operational data after the initial construction of the AI ​​model.

[0008] The object of the present invention is to solve the above technical problems and to provide a network operation system, method, and program using an AI model that can reduce the capacity of the inference database while suppressing deterioration in the accuracy of the AI ​​model by calculating the importance of operational data used for inference in the AI ​​model and not using operational data with low importance for inference. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the present invention provides a system for operating a network using inference results obtained by applying operational data collected from the network to an AI model, which system acquires operational data for each information item from the network and stores it in a database, trains an AI model using the stored operational data, calculates the importance of each item of operational data used to train the AI ​​model, deletes from the database operational data for items whose importance does not satisfy a predetermined higher-level condition, retrains the AI ​​model using operational data for items whose importance satisfies the predetermined higher-level condition, and applies the operational data for items whose importance satisfies the predetermined higher-level condition to the retrained AI model to output inference results. [Effects of the Invention]

[0010] According to the present invention, the operational data stored in the real-time inference database can be reduced while minimizing the deterioration of the accuracy of the AI ​​model, thereby reducing the operational costs of the network. In addition, by determining the capacity limit of the inference database and reducing the operational data with less importance, the frequency of deterioration of the accuracy of the AI ​​model can be minimized. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a functional block diagram showing the configuration of a main part of a network operation system to which the present invention is applied. [Figure 2] FIG. 10 is a diagram illustrating an example of collection of operational data. [Figure 3] This is a block diagram showing the functional configuration of an AI / ML management system. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of a management device. [Figure 5] FIG. 2 is a sequence diagram showing the operation of the present embodiment. [Figure 6] 10 is a chart showing an example of the results of calculating the importance of operational data for each item. [Figure 7] 1 is a diagram showing an example of information regarding operational data for inference transmitted from an AI / ML management system to a management device. [Figure 8] FIG. 10 is a sequence diagram showing a procedure for saving operational data in a database. [Figure 9] FIG. 10 is a sequence diagram showing the procedure for determining and deleting operational data to be deleted from the inference DB. [Figure 10] FIG. 10 is a diagram showing an example of list information acquired from an inference data information storage DB. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a functional block diagram showing the configuration of the main parts of a network operation system using an AI model to which the present invention is applied, and the main components are a management device 1, an AI / ML operation system 2, a learning database (DB) 3, and an inference DB 4.

[0013] The operation target network 5 is a network that is constructed and operated by a telecommunications carrier for communication services. There are no restrictions on the type of network, and it can be applied to, for example, a mobile network consisting of base stations such as 5G and mobile core facilities, or an IP network consisting of routers such as FTTH.

[0014] The management device 1 manages the network 5 under management, and performs configuration settings for the devices that make up the network 5 under management based on input from the operator, and collects operation data sent from the devices.

[0015] 2 is a diagram showing an example of operational data collected by the management device 1, and operational data relating to a plurality of items is collected using collection time and node information (node ​​identifier) ​​as indexes. In this embodiment, CPU utilization rate and memory utilization rate are collected as items relating to hardware, the number of TCP retransmissions and TCP RTT are collected as items relating to processes, and the number of sessions and the number of handovers are collected as items relating to applications / services.

[0016] The learning DB3 is a database used by the AI / ML operation system 2 to train the AI / ML models used in network operation, and stores operational data, including past data, for all models that make up the operation target network 5. Because training of AI / ML models is not performed frequently, real-time performance is not required for the learning DB3. Therefore, the learning DB3 can be built in an inexpensive environment such as a cloud with low real-time performance.

[0017] The inference DB4 is a database used by the AI / ML operation system 2 to apply operational data to an AI / ML model to perform inference related to network operation, and stores the latest operational data for the models that make up the network 5 to be operated.

[0018] To improve network quality, inference using AI / ML models must be performed quickly, so the inference DB4 must have high real-time capabilities. For this reason, inference DB4 is generally built using dedicated database software in an on-premise environment within a telecommunications carrier, which makes the cost per data unit expensive.

[0019] The AI / ML operation system 2 obtains operation data on the target network 5 from the learning DB 3 and trains the AI / ML model to be used for network operation. The AI / ML operation system 2 further obtains real-time operation data on the target network 5 from the inference DB 4 and applies the data to the trained AI / ML model for inference, thereby performing automatic network operation. Network operation using the AI / ML model includes detecting network failures, identifying their causes, and considering recovery measures.

[0020] In this embodiment, the importance of operational data used in the AI / ML model in the AI / ML operation system 2 is calculated for each item, and based on the calculation results, the operational data to be stored in the inference DB 4 is narrowed down to a small amount of highly important data. As a result, the amount of operational data that is expensive per data unit because it is stored in the inference DB 4, which requires real-time performance, can be reduced, thereby reducing network operation costs while suppressing deterioration in the accuracy of the AI ​​model.

[0021] FIG. 3 is a block diagram showing the functional configuration of the internal components of the AI / ML management system 2.

[0022] The operation data acquisition unit 201 acquires operation data from the learning DB 3 and the inference DB 4. The AI ​​model training unit 202 trains an AI / ML model using the operation data acquired by the operation data acquisition unit 201. The AI ​​model storage unit 203 stores the AI / ML model trained by the AI ​​model training unit 202.

[0023] The AI ​​model inference unit 204 performs inference related to network operation by applying the operational data acquired by the operational data acquisition unit 201 to the AI / ML model. The inference result transmission unit 205 transmits the inference result of the AI ​​model inference unit 204 to the network 5 to be operated.

[0024] The operational data importance calculation unit 206 calculates the importance of each operational data item in the AI / ML model trained by the AI ​​model training unit 202. The inference data storage instruction unit 207 selects operational data with high importance to be used by the AI / ML model for training and inference based on the importance calculated by the operational data importance calculation unit 206, and transmits the selected operational data items to the management device 1.

[0025] FIG. 4 is a block diagram showing the functional configuration of the internal components of the management device 1. As shown in FIG.

[0026] The inference data information management unit 101 receives the operational data items with high importance sent by the operational data importance calculation unit 206 of the AI / ML operation system 2, and determines the operational data to be stored in the inference DB 4. The result of the determination is stored in the inference data information storage DB 102 as inference data information.

[0027] The operation data acquisition unit 103 acquires operation data from the operation target network 5. The operation data transmission unit 104 refers to the operation data information for inference stored in the inference data information storage DB 102, and checks whether or not to transmit the operation data of various items acquired by the operation data acquisition unit 103 to the inference DB 4, and transmits it to the inference DB 4 and the learning DB 3 if necessary.

[0028] The database management unit 105 checks the status of the inference DB 4, such as the remaining data storage capacity and operation costs (amount), and notifies the inference data information management unit 101 when each status exceeds the operation limit.

[0029] The management device 1 and the AI / ML management system 2 can be configured by installing applications (programs) that realize the functions described in detail below on a general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc., or on a portable smartphone or tablet terminal. Alternatively, they can be configured as dedicated or single-function machines in which part of the application is implemented as hardware or software.

[0030] Next, the operation of this embodiment will be described in detail with reference to the sequence diagram in Fig. 5. In step S101, the AI / ML operation system 2 requests the learning DB 3 to collect learning operation data to be used in training the AI / ML model. There are no restrictions on the method of request, but for example, a request message including specifications such as a target node and a target period (e.g., 2023 / 04 / 01 to 2023 / 09 / 30) is sent.

[0031] In step S102, the learning DB3 collects operation data in response to a request from the AI / ML operation system 2 and transmits it to the AI / ML operation system 2. Since training of the AI / ML model in the AI / ML operation system 2 does not need to be performed in real time, the learning DB3 prioritizes economy over real-time performance and can be built in an inexpensive environment such as the cloud.

[0032] In step S103, the operation data acquisition unit 201 of the AI / ML operation system 2 trains the AI / ML model using the operation data for learning acquired from the learning DB 3. In this embodiment, the AI ​​model training unit 202 trains the AI / ML model using a known method such as machine learning or deep learning.

[0033] In step S104, the operational data importance calculation unit 206 of the AI / ML operation system 2 calculates the importance of each operational data item in the trained AI / ML model. There are no restrictions on the algorithm for calculating the importance, and any known algorithm such as SHAP (SHapley Additive exPlanations) can be applied.

[0034] Figure 6 is a chart showing an example of the results of calculating the importance of operational data for each item. The importance of each item, such as CPU utilization, memory utilization, number of TCP retransmissions, TCP RTT, number of sessions, and number of handovers, is obtained along with node information and AI model type.

[0035] Furthermore, in step S104, the operational data importance calculation unit 206, based on the calculation results of the importance, identifies operational data items that are low in importance, are predicted to have little impact on the estimation accuracy of the AI / ML model, and do not satisfy the upper-level conditions, and determines other operational data that are high in importance and satisfy the upper-level conditions as operational data items for inference.

[0036] In this embodiment, operational data with low importance is determined by, for example, determining whether the importance is equal to or less than a predetermined threshold, or by using a method that determines the minimum number of operational data that satisfies a predetermined estimation accuracy.

[0037] In step S105, information about the operational data for inference that has high importance determined in step S104 is transmitted from the inference data storage instruction unit 207 to the management device 1.

[0038] Figure 7 is a diagram showing an example of inference data information sent from the inference data storage instruction unit 207 to the management device 1, and stores the items, node information, and importance of the operational data determined in step S104, as well as information regarding the period of operational data to be used when inferring the AI / ML model trained in step S103 (e.g., inference is performed using the past hour), and information regarding the model to be used in the AI / ML model (e.g., LSTM).

[0039] Returning to Figure 5, in step S106, the inference data information management unit 101 of the management device 1 checks the items of the inference operation data with high importance received from the AI / ML operation system 2 and stores them in the inference data information storage DB102.

[0040] Through the above sequence, information regarding operational data to be stored in the inference DB 4 can be automatically set in the management device 1 from the construction of the AI / ML model.

[0041] Next, the procedure for storing the operation data transmitted from the operation target network 5 in each of the DBs 3 and 4 will be described with reference to the sequence diagram of FIG.

[0042] In step S201, operational data is transmitted from various network devices (e.g., base stations, routers, servers) that make up the network 5 to be operated to the management device 1. There are no particular restrictions on the transmission procedure, and possible methods include transmitting the data from the network device to the management device 1 periodically (e.g., every 5 minutes), or having the management device 1 request the operational data from each network device.

[0043] In step S202, the operational data acquisition unit 103 of the management device 1 that received the operational data checks, for each operational data item, whether or not it should be stored in the inference DB 4. Specifically, the operational data transmission unit 104 acquires the operational data items to be stored in the inference DB 4 from the inference data information storage DB 102, and determines that only the operational data corresponding to the items should be stored in the inference DB 4.

[0044] Furthermore, for operational data that has been decided to be stored in the inference DB4, the retention period for storing the data in the inference DB4 is also determined by referring to the period information used during inference in the AI / ML model. Since the purpose is to reduce the amount of data stored in the inference DB4, the retention period must also be kept to the minimum necessary, and by setting the past period used in one inference as the retention period, the minimum necessary period can be achieved.

[0045] In step S203, the operational data transmission unit 104 of the management device 1 transmits all of the received operational data to the learning DB 3 regardless of the determination result in step S202. In step S204, the operational data transmission unit 104 of the management device 1 transmits to the inference DB 4 only the operational data for the items determined in step S202 to be stored in the inference DB 4 from the operational data received in step S201.

[0046] Figure 9 is a sequence diagram showing the procedure by which the management device 1 determines the items and period of operational data to be deleted from the inference DB 4 and deletes them when the amount of operational data stored in the inference DB 4 exceeds a predetermined limit value.

[0047] As AI-based network operations become more widespread, there is a concern that numerous AI / ML models will be continuously constructed, resulting in an increase in the amount of operational data to be stored in the inference DB 4. Therefore, in this embodiment, as will be described in detail below, when the amount of operational data to be stored in the inference DB 4 increases, the number of operational data items used for inference is reduced, and the AI / ML model is retrained in accordance with the reduction in operational data items.

[0048] In step S301, the inference DB 4 transmits, for example, the amount of stored data, as information about the current database to the database management unit 105 of the management device 1. In step S302, the database management unit 105 checks the information about the inference DB 4 received and confirms whether any data exceeding the operational limit has been stored.

[0049] There are no restrictions on the procedure for determining the limit, but if the amount of data stored exceeds the allowable operational cost (e.g., database license fee) or if the amount of data stored does not meet the allowable performance (e.g., response time to a request), it is determined that the limit has been exceeded.

[0050] If it is confirmed that data exceeding the operational limit value is stored, the process proceeds to step S303, where the inference data information management unit 101 of the management device 1 determines the operational data to be deleted from the inference DB 4. In this embodiment, the inference data information storage DB 102 checks the list of operational data to be currently stored in the inference DB 4, and determines the operational data items to be deleted from that list.

[0051] 10 is a diagram showing an example of list information that can be acquired from the inference data information storage DB 102. In this embodiment, operational data with a smaller value obtained by dividing the importance by the inference usage period is prioritized for deletion. In the example shown, the number of packet losses = 40 / 2 = 20, and the received optical intensity = 30 / 2 = 15, which are significantly lower than the values ​​of other operational data items, so "number of packet losses" and "received optical intensity" are determined to be deleted.

[0052] In step S304, the management device 1 notifies the AI / ML operation system 2 of the operation data items determined to be deleted and requests that the AI / ML model be updated. In step S305, the AI / ML operation system 2 updates and reconstructs the AI / ML model so that the notified operation data items are not used as input values ​​for the AI / ML model.

[0053] When the update and reconstruction of the AI / ML model is completed, in step S306 the AI / ML operation system 2 notifies the management device 1 of the completion of the update of the AI / ML model. Since the update of the AI / ML model may also update the importance of each item of the operation data, it is desirable to also send the updated importance.

[0054] In step S307, the management device 1, which has received notification of the completion of the update of the AI / ML model from the AI / ML operation system 2, stores the information including the updated importance received in step S306 in its own inference data information storage DB 102, and from then on, reflects this in the operation data sent from the operation target network 5 so that the items of the operation data that have been updated this time are not sent to the inference DB 4.

[0055] In step S308, the management device 1 notifies the inference DB 4 of the operational data items determined to be deleted in step S303, and the inference DB 4 deletes the past data of the operational data in question.

[0056] According to the above embodiment, the operational data stored in the real-time inference database can be reduced while minimizing the deterioration of the accuracy of the AI ​​model, thereby reducing the network operation costs. In addition, by determining the capacity limit of the inference database and reducing the operational data with lower importance, the frequency of deterioration of the accuracy of the AI ​​model can be minimized.

[0057] Therefore, it will be possible to contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), "Build resilient infrastructure and promote inclusive and sustainable industrialization," and Goal 11, "Make cities inclusive, safe, resilient and sustainable." [Explanation of symbols]

[0058] 1... management device, 2... AI / ML operation system, 3... learning DB, 4... inference DB, 5... operation target network, 101... inference data information management unit, 102... inference data information storage DB, 103... operation data acquisition unit, 104... operation data transmission unit, 105... database management unit, 201... operation data acquisition unit, 202... AI model training unit, 203... AI model storage unit, 20... AI model inference unit, 205... inference result transmission unit, 206... operation data importance calculation unit, 207... inference data storage instruction unit

Claims

1. In a system that operates a network using inference results obtained by applying operational data collected from the network to an AI model, a means for acquiring operational data for each item of information from the network; a database for storing the operational data; means for training an AI model using the operational data; A means for calculating the importance of each item of operational data used to train the AI ​​model; means for deleting from the database operational data of items whose importance does not satisfy a predetermined higher-level condition; The system includes a means for retraining the AI ​​model using operational data of items whose importance satisfies a predetermined higher-level condition, A network operation system using an AI model, characterized by applying operational data of items whose importance satisfies a predetermined higher-level condition to the retrained AI model and outputting inference results.

2. means for determining whether the capacity of the database has exceeded a predetermined upper limit; 2. A network operation system using an AI model as described in claim 1, characterized in that when the capacity of the database exceeds a predetermined upper limit, the deleting means deletes the operation data and the AI ​​model retraining means retrains the AI ​​model.

3. The means for calculating the importance identifies, for each item, a period during which each operational data item is used to train the AI ​​model; 3. A network operation system using an AI model according to claim 1, wherein the deleting means also deletes from the database operation data whose period of use has expired.

4. A network operation system using an AI model as described in claim 1 or 2, characterized in that the database is an inference database that stores operational data for inference and is placed in an environment that satisfies specified real-time performance.

5. A method for operating a network using inference results obtained by applying operational data collected from a network by a computer to an AI model, Obtain operational data for each information item from the network, storing the operational data in a database; training an AI model using the operational data; Calculate the importance of each item of the operational data used to train the AI ​​model; delete from the database the operational data of items whose importance does not satisfy a predetermined higher-level condition; Retraining the AI ​​model using operational data of items whose importance satisfies a predetermined top condition; A network operation method using an AI model, characterized by applying operational data of items whose importance satisfies a predetermined higher-level condition to the retrained AI model and outputting inference results.

6. determining whether the capacity of the database has exceeded a predetermined upper limit; A network operation method using an AI model as described in claim 5, characterized in that when the capacity of the database exceeds a predetermined upper limit, the operation data is deleted and the AI ​​model is retrained.

7. A program that operates a network using the inference results obtained by applying operational data collected from the network to an AI model. The procedure for obtaining operational data for each item of information from the network; storing the operational data in a database; training an AI model using the operational data; A procedure for calculating the importance of each item of operational data used to train the AI ​​model; a step of deleting from the database operational data of items whose importance does not satisfy a predetermined higher-level condition; and retraining the AI ​​model using operational data of items whose importance satisfies a predetermined higher-level condition. A network operation program using an AI model, characterized by applying operational data of items whose importance satisfies a predetermined higher-level condition to the retrained AI model and outputting inference results.

8. The method further includes a step of determining whether the capacity of the database has exceeded a predetermined upper limit value; 8. A network operation program using an AI model as described in claim 7, characterized in that when the capacity of the database exceeds a predetermined upper limit, the deleting step deletes the operation data, and the retraining step retrains the AI ​​model.

Images