Determining the machine learning model to be used for a given prediction objective related to a communication system

Abstract

The present invention makes it possible to accurately determine a machine learning model suitable for a communication system, from among a plurality of machine learning models used for a given predictive purpose for the communication system. An AI unit (70) enters, into each of a plurality of trained machine learning models, input data corresponding to the trained machine learning model, and acquires a predicted value as the output of the trained machine learning model. The AI unit (70) evaluates the prediction accuracy of each machine learning model with respect to the predictive purpose, on the basis of the acquired predicted value and a portion of the test data that indicates an actual value corresponding to the predicted value. The AI unit (70) determines at least one of the plurality of trained machine learning models on the basis of the accuracy evaluation results.

Description

Technical Field

[0001] The present invention relates to determining a machine learning model used for a given prediction purpose related to a communication system.

Background Art

[0002] There is a technology for predicting performance index values of a communication system. As an example of such a technology, Patent Document 1 describes estimating throughput based on the number of terminals existing in mesh i, the number of terminals existing in mesh i and in communication, and the like.

[0003] In recent years, predictions using machine learning have been actively carried out.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] For example, it is conceivable to predict the performance index value of a communication system based on the actual value of the performance index value of the communication system using a trained machine learning model.

[0006] Here, for the same prediction purpose related to a communication system, a plurality of machine learning models with different types of input data may be prepared. However, in the prior art, it has not been possible to accurately determine a machine learning model suitable for the communication system of interest from among these machine learning models.

[0007] The present invention has been made in view of the above circumstances, and one of its objects is to accurately determine a machine learning model suitable for the communication system from among a plurality of machine learning models used for a given prediction purpose related to the communication system.

Means for Solving the Problems

[0008] In order to solve the above problems, a model determination system according to the present disclosure includes: test data acquisition means for acquiring test data indicating a time series of actual values of a plurality of types of performance index values related to a communication system; input data indicating the actual values at at least one time point of at least one of the types, which is a part of the test data, are different from each other. For each of a plurality of trained machine learning models used for a given prediction purpose related to the communication system, the input data corresponding to the machine learning model is input, and a predicted value at a prediction time point after any of the time points is acquired as an output of the machine learning model; prediction value acquisition means; for each of the plurality of trained machine learning models, based on the acquired predicted value and a part of the test data indicating the actual value at the prediction time point of at least one of the types corresponding to the predicted value, prediction accuracy evaluation means for evaluating the accuracy of the prediction related to the prediction purpose by the machine learning model; and model determination means for determining at least one of the plurality of trained machine learning models based on the evaluation result of the accuracy.

[0009] In addition, the model determination method according to the present disclosure includes obtaining test data showing a time series of actual values of a plurality of types of performance index values related to a communication system, and at least one actual value at at least one time point of at least one of the types, which is a part of the test data and is used for a given prediction purpose related to the communication system, is different for each of a plurality of trained machine learning models. For each of the trained machine learning models, inputting the input data corresponding to the machine learning model, and obtaining, as an output of the machine learning model, a predicted value at a prediction time point after any of the time points. For each of the plurality of trained machine learning models, based on the obtained predicted value and a part of the test data showing the actual value at the prediction time point of at least one of the types corresponding to the predicted value, evaluating the accuracy of the prediction related to the prediction purpose by the machine learning model, and determining at least one of the plurality of trained machine learning models based on the evaluation result of the accuracy.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

MODE FOR CARRYING OUT THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0012] FIGS. 1 and 2 are diagrams showing an example of a communication system 1 according to an embodiment of the present invention. FIG. 1 focuses on the locations of the data center group included in the communication system 1. FIG. 2 focuses on the various computer systems installed in the data center group included in the communication system 1.

[0013] As shown in FIG. 1, the data center group included in the communication system 1 is classified into a central data center 10, a regional data center 12, and an edge data center 14.

[0014] The central data centers 10 are, for example, several distributed within the area covered by the communication system 1 (for example, within Japan).

[0015] Regional data centers 12 are, for example, distributed and dozens of them are arranged within the area covered by the communication system 1. For example, when the area covered by the communication system 1 is the entire territory of Japan, one or two regional data centers 12 may be arranged in each prefecture.

[0016] Edge data centers 14 are, for example, distributed and thousands of them are arranged within the area covered by the communication system 1. Also, each of the edge data centers 14 can communicate with communication facilities 18 equipped with antennas 16. As shown in FIG. 1 here, one edge data center 14 may be able to communicate with several communication facilities 18. The communication facilities 18 may include a computer such as a server computer. The communication facilities 18 according to the present embodiment perform wireless communication with a UE (User Equipment) 20 via the antenna 16. For example, an RU (Radio Unit) described later is provided in the communication facilities 18 equipped with the antenna 16.

[0017] A plurality of servers are arranged in the central data center 10, the regional data centers 12, and the edge data centers 14 according to the present embodiment, respectively.

[0018] In the present embodiment, for example, the central data center 10, the regional data centers 12, and the edge data centers 14 can communicate with each other. Also, the central data centers 10 can communicate with each other, the regional data centers 12 can communicate with each other, and the edge data centers 14 can communicate with each other.

[0019] As shown in FIG. 2, the communication system 1 according to the present embodiment includes a platform system 30, a plurality of radio access networks (RANs) 32, a plurality of core network systems 34, and a plurality of UEs 20. The core network system 34, the RAN 32, and the UE 20 cooperate with each other to realize a mobile communication network.

[0020] RAN32 is a computer system with 16 antennas, corresponding to eNB (eNodeB) in the 4th generation mobile communication system (hereinafter referred to as 4G) and gNB (NR base station) in the 5th generation mobile communication system (hereinafter referred to as 5G). RAN32 according to this embodiment is mainly implemented by a server group and communication facilities 18 arranged in the edge data center 14. Note that a part of RAN32 (for example, DU (Distributed Unit), CU (Central Unit), vDU (virtual Distributed Unit), vCU (virtual Central Unit)) may be implemented not in the edge data center 14 but in the central data center 10 or the regional data center 12.

[0021] The core network system 34 is a system corresponding to EPC (Evolved Packet Core) in 4G and 5G core (5GC) in 5G. The core network system 34 according to this embodiment is mainly implemented by a server group arranged in the central data center 10 and the regional data center 12.

[0022] The platform system 30 according to this embodiment is configured, for example, on a cloud infrastructure. As shown in FIG. 2, it includes a processor 30a, a storage unit 30b, and a communication unit 30c. The processor 30a is a program control device such as a microprocessor that operates according to a program installed in the platform system 30. The storage unit 30b is, for example, a storage element such as a ROM or a RAM, a solid state drive (SSD), a hard disk drive (HDD), etc. Programs executed by the processor 30a and the like are stored in the storage unit 30b. The communication unit 30c is a communication interface such as, for example, a NIC (Network Interface Controller) or a wireless LAN (Local Area Network) module. Note that SDN (Software-Defined Networking) may be implemented in the communication unit 30c. The communication unit 30c exchanges data with the RAN 32 and the core network system 34.

[0023] In this embodiment, the platform system 30 is implemented by a server group arranged in the central data center 10. Note that the platform system 30 may be implemented by a server group arranged in the regional data center 12.

[0024] In this embodiment, for example, in response to a purchase request for a network service (NS) by a purchaser, the purchased network service is constructed in the RAN 32 and the core network system 34. Then, the constructed network service is provided to the purchaser.

[0025] For example, network services such as voice communication services and data communication services are provided to a purchaser who is an MVNO (Mobile Virtual Network Operator). The voice communication service and data communication service provided by this embodiment will ultimately be provided to a customer (end user) for the purchaser (MVNO in the above example) who uses the UE20 shown in FIGS. 1 and 2. The end user can perform voice communication and data communication with other users via the RAN32 and the core network system 34. Also, the UE20 of the end user can access a data network such as the Internet via the RAN32 and the core network system 34.

[0026] Also, in this embodiment, an IoT (Internet of Things) service may be provided to an end user who uses a robotic arm, a connected car, or the like. And in this case, for example, an end user who uses a robotic arm, a connected car, or the like may be a purchaser of the network service according to this embodiment.

[0027] In this embodiment, container-type virtualized application execution environments such as Docker (registered trademark) are installed on the servers arranged in the central data center 10, the regional data center 12, and the edge data center 14, and containers can be deployed and operated on these servers. In these servers, a cluster composed of one or more containers generated by such virtualization technology may be constructed. For example, a Kubernetes cluster managed by a container management tool such as Kubernetes (registered trademark) may be constructed. And the processor on the constructed cluster may execute a container-type application.

[0028] In this embodiment, the network service provided to the purchaser is composed of one or more functional units (for example, network functions (NFs)). In this embodiment, the functional unit is implemented by an NF realized by virtualization technology. The NF realized by virtualization technology is called a VNF (Virtualized Network Function). Note that it does not matter by what virtualization technology it is virtualized. For example, a CNF (Containerized Network Function) realized by container-type virtualization technology is also included in the VNF in this description. In this embodiment, the network service will be described as being implemented by one or more CNFs. Also, the functional unit according to this embodiment may correspond to a network node.

[0029] FIG. 3 is a diagram schematically showing an example of an operating network service. The network service shown in FIG. 3 includes NFs such as a plurality of RUs 40, a plurality of DUs 42, a plurality of CUs 44 (CU-CP (Central Unit - Control Plane) 44a and CU-UP (Central Unit - User Plane) 44b), a plurality of AMFs (Access and Mobility Management Function) 46, a plurality of SMFs (Session Management Function) 48, and a plurality of UPFs (User Plane Function) 50 as software elements.

[0030] In the example of FIG. 3, the RUs 40, DUs 42, CU-CP 44a, AMFs 46, and SMFs 48 correspond to elements of the control plane (C-Plane), and the RUs 40, DUs 42, CU-UP 44b, and UPFs 50 correspond to elements of the user plane (U-Plane).

[0031] Note that other types of NFs may be included as software elements in the network service. Also, the network service is implemented on computer resources (hardware elements) such as a plurality of servers.

[0032] And in this embodiment, for example, a communication service in a certain area is provided by the network service shown in FIG. 3.

[0033] And in this embodiment, it is assumed that a plurality of RUs 40, a plurality of DUs 42, a plurality of CU-UPs 44b, and a plurality of UPFs 50 shown in FIG. 3 belong to one end-to-end network slice.

[0034] FIG. 4 is a diagram schematically showing an example of the association between elements constructed in the communication system 1 in this embodiment. Note that the symbols M and N shown in FIG. 4 represent arbitrary integers of 1 or more, and show the relationship of the number of elements connected by a link. When both ends of the link are a combination of M and N, the elements connected by the link have a many-to-many relationship, and when both ends of the link are a combination of 1 and N or a combination of 1 and M, the elements connected by the link have a one-to-many relationship.

[0035] As shown in FIG. 4, the network service (NS), the network function (NF), the CNFC (Containerized Network Function Component), the pod, and the container have a hierarchical structure.

[0036] The NS corresponds to, for example, a network service composed of a plurality of NFs. Here, the NS may correspond to elements such as 5GC, EPC, 5G RAN (gNB), 4G RAN (eNB), etc.

[0037] In 5G, NF corresponds to elements at a granularity level such as RU, DU, CU-CP, CU-UP, AMF, SMF, UPF, etc. Also, in 4G, NF corresponds to elements at a granularity level such as MME (Mobility Management Entity), HSS (Home Subscriber Server), S-GW (Serving Gateway), vDU, vCU, etc. In this embodiment, for example, one NS contains one or more NFs. That is, one or more NFs are under the jurisdiction of one NS.

[0038] CNFC corresponds to elements at a granularity level such as DU mgmt and DU Processing. CNFC may be a microservice deployed on a server as one or more containers. For example, a certain CNFC may be a microservice that provides some of the functions of DU, CU-CP, CU-UP, etc. Also, a certain CNFC may be a microservice that provides some of the functions of UPF, AMF, SMF, etc. In this embodiment, for example, one NF contains one or more CNFCs. That is, one or more CNFCs are under the jurisdiction of one NF.

[0039] A pod refers to, for example, the minimum unit for managing Docker containers in Kubernetes. In this embodiment, for example, one CNFC contains one or more pods. That is, one or more pods are under the jurisdiction of one CNFC.

[0040] And in this embodiment, for example, one pod contains one or more containers. That is, one or more containers are under the jurisdiction of one pod.

[0041] Also, as shown in FIG. 4, the network slice (NSI) and the network slice subnet instance (NSSI) have a hierarchical structure.

[0042] NSI can also be said to be an end-to-end virtual circuit spanning multiple domains (for example, from RAN 32 to core network system 34). NSI may be a slice for high-speed large-capacity communication (for example, for eMBB: enhanced Mobile Broadband), a slice for high-reliability and low-latency communication (for example, for URLLC: Ultra-Reliable and Low Latency Communications), or a slice for connection of a large number of terminals (for example, for mMTC: massive Machine Type Communication). NSSI can also be said to be a single-domain virtual circuit obtained by splitting NSI. NSSI may be a slice of the RAN domain, a slice of a transport domain such as the MBH (Mobile Back Haul) domain, or a slice of the core network domain.

[0043] In this embodiment, for example, one NSI includes one or more NSSIs. That is, one or more NSSIs are subordinate to one NSI. Note that in this embodiment, multiple NSIs may share the same NSSI.

[0044] Also, as shown in FIG. 4, generally, NSSI and NS have a many-to-many relationship.

[0045] Also, in this embodiment, for example, one NF can belong to one or more network slices. Specifically, for example, one NF can be set with an NSSAI (Network Slice Selection Assistance Information) including one or more S-NSSAIs (Sub Network Slice Selection Assist Information). Here, S-NSSAI is information associated with a network slice. Note that the NF may not belong to a network slice.

[0046] FIG. 5 is a functional block diagram showing an example of functions implemented in the platform system 30 according to the present embodiment. Note that not all of the functions shown in FIG. 5 need to be implemented in the platform system 30 according to the present embodiment, and functions other than those shown in FIG. 5 may be implemented.

[0047] As shown in FIG. 5, the platform system 30 according to the present embodiment functionally includes, for example, an operation support system (OSS) unit 60, an end-to-end orchestration (E2EO) unit 62, a service catalog storage unit 64, a big data platform unit 66, a data bus unit 68, an artificial intelligence (AI) unit 70, a monitoring function unit 72, an SDN controller 74, a configuration management unit 76, a container management unit 78, and a repository unit 80. The OSS unit 60 includes an inventory database 82, a ticket management unit 84, a fault management unit 86, and a performance management unit 88. The E2EO unit 62 includes a policy manager unit 90, a slice manager unit 92, and a lifecycle management unit 94. These elements are mainly implemented by a processor 30a, a storage unit 30b, and a communication unit 30c.

[0048] The functions shown in FIG. 5 may be installed in the platform system 30, which is one or more computers, and implemented by the processor 30a executing a program including instructions corresponding to the functions. This program may be supplied to the platform system 30 via a computer-readable information storage medium such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, a flash memory, or the like, or via the Internet or the like. Also, the functions shown in FIG. 5 may be implemented by circuit blocks, memories, and other LSIs. It is understood by those skilled in the art that the functions shown in FIG. 5 can be realized in various forms such as only hardware, only software, or a combination thereof.

[0049] The container management unit 78 executes the life cycle management of containers. For example, processes related to the construction of containers, such as the deployment and setting of containers, are included in the life cycle management.

[0050] Here, the platform system 30 according to the present embodiment may include a plurality of container management units 78. And for each of the plurality of container management units 78, a container management tool such as Kubernetes and a package manager such as Helm may be installed. And each of the plurality of container management units 78 may execute the construction of containers, such as the deployment of containers, for the server group (for example, a Kubernetes cluster) associated with the container management unit 78.

[0051] Note that the container management unit 78 does not necessarily have to be included in the platform system 30. The container management unit 78 may be provided, for example, in the server managed by the container management unit 78 (that is, RAN 32 or core network system 34), or may be provided in another server co-located with the server managed by the container management unit 78.

[0052] In the present embodiment, the repository unit 80 stores, for example, the container images of the containers included in the function unit group (for example, NF group) that realizes the network service.

[0053] The inventory database 82 is a database in which inventory information is stored. The inventory information includes, for example, information about the servers arranged in RAN 32 or core network system 34 and managed by the platform system 30.

[0054] Also, in this embodiment, inventory data is stored in the inventory database 82. The inventory data indicates the configuration of the element group included in the communication system 1 and the current status of the association between the elements. Further, the inventory data indicates the status of the resources (for example, the usage status of the resources) managed by the platform system 30. The inventory data may be physical inventory data or logical inventory data. Physical inventory data and logical inventory data will be described later.

[0055] FIG. 6 is a diagram showing an example of the data structure of physical inventory data. The physical inventory data shown in FIG. 6 is associated with one server. The physical inventory data shown in FIG. 6 includes, for example, a server ID, location data, building data, floor number data, rack data, specification data, network data, a list of operating container IDs, a cluster ID, and the like.

[0056] The server ID included in the physical inventory data is, for example, an identifier of the server associated with the physical inventory data.

[0057] The location data included in the physical inventory data is, for example, data indicating the location (for example, the address of the location) of the server associated with the physical inventory data.

[0058] The building data included in the physical inventory data is, for example, data indicating the building (for example, the building name) in which the server associated with the physical inventory data is located.

[0059] The floor number data included in the physical inventory data is, for example, data indicating the floor on which the server associated with the physical inventory data is located.

[0060] The rack data included in the physical inventory data is, for example, an identifier of the rack in which the server associated with the physical inventory data is located.

[0061] The spec data included in the physical inventory data is, for example, data indicating the specs of the server associated with the physical inventory data, and the spec data indicates, for example, the number of cores, memory capacity, hard disk capacity, etc.

[0062] The network data included in the physical inventory data is, for example, data indicating information related to the network of the server associated with the physical inventory data, and the network data indicates, for example, the NICs equipped in the server, the number of ports equipped in the NIC, the port IDs of the ports, etc.

[0063] The list of running container IDs included in the physical inventory data is, for example, data indicating information related to one or more containers running on the server associated with the physical inventory data, and the list of running container IDs indicates, for example, a list of identifiers (container IDs) of instances of the containers.

[0064] The cluster ID included in the physical inventory data is, for example, an identifier of the cluster (e.g., Kubernetes cluster) to which the server associated with the physical inventory data belongs.

[0065] The logical inventory data includes topology data indicating the current status of associations between a plurality of elements included in the communication system 1, as shown in FIG. 4. For example, the logical inventory data includes topology data including an identifier of a certain NS and identifiers of one or more NFs under the NS. Also, for example, the logical inventory data includes topology data including an identifier of a certain network slice and identifiers of one or more NFs belonging to the network slice.

[0066] Also, the inventory data may include data indicating the current situation such as the geographical relationship or topological relationship between the elements included in the communication system 1. As described above, the inventory data includes location data indicating the location where the elements included in the communication system 1 are operating, that is, the current location of the elements included in the communication system 1. From this, it can be said that the inventory data shows the current situation of the geographical relationship (for example, geographical proximity) between the elements.

[0067] Also, the logical inventory data may include NSI data indicating information related to network slices. The NSI data indicates, for example, the identifier of an instance of a network slice and attributes such as the type of the network slice. Also, the logical inventory data may include NSSI data indicating information related to network slice subnets. The NSSI data indicates, for example, the identifier of an instance of a network slice subnet and attributes such as the type of the network slice subnet.

[0068] In addition, the logical inventory data may include NS data indicating information about the NS. The NS data indicates, for example, the identifier of an instance of the NS and attributes such as the type of the NS. Further, the logical inventory data may include NF data indicating information about the NF. The NF data indicates, for example, the identifier of an instance of the NF and attributes such as the type of the NF. Also, the logical inventory data may include CNFC data indicating information about the CNFC. The CNFC data indicates, for example, the identifier of an instance and attributes such as the type of the CNFC. Additionally, the logical inventory data may include pod data indicating information about the pods included in the CNFC. The pod data indicates, for example, the identifier of an instance of the pod and attributes such as the type of the pod. Moreover, the logical inventory data may include container data indicating information about the containers included in the pod. The container data indicates, for example, the container ID of an instance of the container and attributes such as the type of the container.

[0069] The container ID of the container data included in the logical inventory data and the container ID included in the list of operating container IDs included in the physical inventory data will associate the instance of the container with the server on which the instance of the container is operating.

[0070] Also, data indicating various attributes such as a host name and an IP address may be included in the above-mentioned data included in the logical inventory data. For example, the container data may include data indicating the IP address of the container corresponding to the container data. Further, for example, the NF data may include data indicating the IP address and the host name of the NF indicated by the NF data.

[0071] In addition, the logical inventory data may include data indicating an NSSAI including one or more S-NSSAIs set for each NF.

[0072] In addition, the inventory database 82 is capable of appropriately grasping the status of resources in cooperation with the container management unit 78. Then, based on the latest status of the resources, the inventory database 82 appropriately updates the inventory data stored in the inventory database 82.

[0073] Also, for example, in response to actions such as the construction of new elements included in the communication system 1, the configuration change of elements included in the communication system 1, the scaling of elements included in the communication system 1, and the replacement of elements included in the communication system 1 being executed, the inventory database 82 updates the inventory data stored in the inventory database 82.

[0074] The service catalog storage unit 64 stores service catalog data. The service catalog data may include, for example, service template data indicating logic used by the life cycle management unit 94. This service template data includes information necessary for constructing a network service. For example, the service template data includes information defining NS, NF, and CNFC, and information indicating the correspondence relationship of NS-NF-CNFC. Also, for example, the service template data includes a workflow script for constructing a network service.

[0075] As an example of the service template data, NSD (NS Descriptor) can be mentioned. NSD is associated with a network service and indicates the types of a plurality of functional units (for example, a plurality of CNFs) included in the network service. Note that the number of each type of functional unit such as CNF included in the network service may be indicated in the NSD. Also, the file name of the CNFD, which will be described later, related to the CNF included in the network service may be indicated in the NSD.

[0076] Also, as an example of service template data, CNFD (CNF Descriptor) can be cited. In the CNFD, computer resources (such as CPU, memory, hard disk, etc.) required by the CNF may be indicated. For example, in the CNFD, for each of a plurality of containers included in the CNF, computer resources (CPU, memory, hard disk, etc.) required by the container may be indicated.

[0077] Also, the service catalog data may include information regarding a threshold value (for example, a threshold value for anomaly detection) that is used by the policy manager unit 90 and compared with the calculated performance index value. The performance index value will be described later.

[0078] Also, the service catalog data may include, for example, slice template data. The slice template data includes information necessary to execute the instantiation of a network slice, and includes, for example, logic used by the slice manager unit 92.

[0079] The slice template data includes information of "Generic Network Slice Template" defined by GSMA (GSM Association) (where "GSM" is a registered trademark). Specifically, the slice template data includes template data of a network slice (NST), template data of a network slice subnet (NSST), and template data of a network service. Also, the slice template data includes information indicating the hierarchical structure of these elements as shown in FIG. 4.

[0080] In this embodiment, for example, in response to a purchase request for an NS by a purchaser, the lifecycle management unit 94 constructs a new network service for which the purchase request has been made.

[0081] The lifecycle management unit 94 may, for example, execute a workflow script associated with the network service to be purchased in response to a purchase request. By executing this workflow script, the lifecycle management unit 94 may instruct the container management unit 78 to deploy the containers included in the newly purchased network service. Then, the container management unit 78 may obtain the container image of the container from the repository unit 80 and deploy the container corresponding to the container image to the server.

[0082] Also, in this embodiment, the lifecycle management unit 94, for example, performs scaling and replacement of the elements included in the communication system 1. Here, the lifecycle management unit 94 may output a container deployment instruction or a deletion instruction to the container management unit 78. Then, the container management unit 78 may execute processes such as container deployment and container deletion according to the instruction. In this embodiment, the lifecycle management unit 94 can execute scaling and replacement that cannot be handled by tools such as Kubernetes of the container management unit 78.

[0083] Also, the lifecycle management unit 94 may output a communication path creation instruction to the SDN controller 74. For example, the lifecycle management unit 94 presents two IP addresses at both ends of the communication path to be created to the SDN controller 74, and the SDN controller 74 creates a communication path connecting these two IP addresses. The created communication path may be managed in association with these two IP addresses.

[0084] Also, the lifecycle management unit 94 may output a communication path creation instruction to the SDN controller 74 for the communication path between two IP addresses associated with the two IP addresses.

[0085] In this embodiment, for example, the slice manager unit 92 executes the instantiation of network slices. In this embodiment, for example, the slice manager unit 92 executes the instantiation of network slices by executing the logic indicated by the slice template stored in the service catalog storage unit 64.

[0086] The slice manager unit 92 is configured to include, for example, the functions of the NSMF (Network Slice Management Function) and the NSSMF (Network Slice Sub-network Management Function) described in the specification "TS28 533" of 3GPP (Registered Trademark) (Third Generation Partnership Project). The NSMF is a function for generating and managing network slices and provides management services for NSIs. The NSSMF is a function for generating and managing network slice sub-networks that constitute a part of a network slice and provides management services for NSSIs.

[0087] Here, the slice manager unit 92 may output a configuration management instruction related to the instantiation of network slices to the configuration management unit 76. Then, the configuration management unit 76 may execute configuration management such as setting according to the configuration management instruction.

[0088] In addition, the slice manager unit 92 may present two IP addresses to the SDN controller 74 and output an instruction to create a communication path between these two IP addresses.

[0089] In this embodiment, for example, the configuration management unit 76 executes configuration management such as setting of an element group such as an NF according to a configuration management instruction received from the life cycle management unit 94 or the slice manager unit 92.

[0090] In this embodiment, for example, the SDN controller 74 creates a communication path between two IP addresses associated with a creation instruction for a communication path received from the lifecycle management unit 94 or the slice manager unit 92. The SDN controller 74 may create a communication path between two IP addresses using a known path calculation method such as Flex Algo, for example.

[0091] Here, for example, the SDN controller 74 may construct NSIs and NSSIs for aggregation routers, servers, etc. existing between communication paths using segment routing technology (for example, SRv6 (Segment Routing IPv6)). Also, the SDN controller 74 issues commands to set a common VLAN (Virtual Local Area Network) for a plurality of NFs to be configured, and commands to assign bandwidth and priority indicated by the setting information to the VLAN, thereby generating NSIs and NSSIs across the plurality of NFs to be configured.

[0092] Note that the SDN controller 74 may execute operations such as changing the maximum value of the available bandwidth for communication between two IP addresses without constructing a network slice.

[0093] The platform system 30 according to this embodiment may include a plurality of SDN controllers 74. And the plurality of SDN controllers 74 may each execute processes such as creating a communication path for a group of network devices such as an AG associated with the SDN controller 74.

[0094] In this embodiment, for example, the monitoring function unit 72 monitors the element group included in the communication system 1 according to a given management policy. Here, the monitoring function unit 72 may monitor the element group according to a monitoring policy specified by a purchaser when purchasing a network service, for example.

[0095] In this embodiment, the monitoring function unit 72 performs monitoring at various levels, such as the slice level, NS level, NF level, CNFC level, and the hardware level of servers and the like.

[0096] For example, the monitoring function unit 72 may set a module that outputs metric data to hardware such as a server or software elements included in the communication system 1 so that monitoring at the above-mentioned various levels can be performed. Here, for example, the NF may output metric data indicating metrics measurable (specifiable) in the NF to the monitoring function unit 72. Also, the server may output metric data indicating metrics related to hardware measurable (specifiable) in the server to the monitoring function unit 72.

[0097] Further, for example, the monitoring function unit 72 may deploy a sidecar container on the server to aggregate metric data indicating metrics output from a plurality of containers in units of CNFC (microservices). This sidecar container may include an agent called an exporter. The monitoring function unit 72 may repeatedly execute, at a given monitoring interval, a process of obtaining metric data aggregated in units of microservices from the sidecar container by using the mechanism of a monitoring tool such as Prometheus that can monitor container management tools such as Kubernetes.

[0098] For example, the monitoring function unit 72 may monitor performance indicator values for the performance indicators described in "TS 28.552, Management and orchestration; 5G performance measurements" or "TS 28.554, Management and orchestration; 5G end to end Key Performance Indicators (KPI)". And the monitoring function unit 72 may obtain metric data indicating the monitored performance indicator values.

[0099] And in this embodiment, the monitoring function unit 72 generates performance index value data indicating the performance index values of the elements included in the communication system 1 in a given aggregation unit by executing a process (enrichment) of aggregating metric data in the given aggregation unit.

[0100] For example, for one gNB, by aggregating metric data indicating the metrics of the elements (for example, network nodes such as DU42 and CU44) under the gNB, performance index value data of the gNB is generated. In this way, performance index value data indicating the communication performance in the area covered by the gNB is generated. Here, for example, in each gNB, performance index value data indicating a plurality of types of communication performance such as traffic volume (throughput) and latency may be generated. Note that the communication performance indicated by the performance index value data is not limited to traffic volume and latency.

[0101] Then, the monitoring function unit 72 outputs the performance index value data generated by the above-described enrichment to the data bus unit 68.

[0102] In this embodiment, the data bus unit 68 receives, for example, the performance index value data output from the monitoring function unit 72. Then, based on the one or more received performance index value data, the data bus unit 68 generates a performance index value file including the one or more performance index value data. Then, the data bus unit 68 outputs the generated performance index value file to the big data platform unit 66.

[0103] In addition, elements such as network slices, NS, NF, CNFC, etc. included in the communication system 1 and hardware such as servers notify the monitoring function unit 72 of various alerts (for example, alerts triggered by the occurrence of a failure).

[0104] Then, when the monitoring function unit 72 receives the above-mentioned alert notification, for example, it outputs alert message data indicating the notification to the data bus unit 68. Then, the data bus unit 68 generates an alert file by collecting alert message data indicating one or more notifications into one file, and outputs the alert file to the big data platform unit 66.

[0105] In this embodiment, the big data platform unit 66 accumulates, for example, the performance index value file and the alert file output from the data bus unit 68.

[0106] In this embodiment, for example, a plurality of pre-trained machine learning models are stored in the AI unit 70 in advance. The AI unit 70 executes estimation processing such as future prediction processing of the usage status and service quality of the communication system 1 using various machine learning models stored in the AI unit 70. The AI unit 70 may generate estimation result data indicating the result of the estimation processing.

[0107] The AI unit 70 may execute the estimation processing based on the file stored in the big data platform unit 66 and the above-mentioned machine learning model. This estimation processing is suitable for making long-term trend predictions at low frequencies.

[0108] Also, the AI unit 70 can acquire the performance index value data stored in the data bus unit 68. The AI unit 70 may execute the estimation processing based on the performance index value data stored in the data bus unit 68 and the above-mentioned machine learning model. This estimation processing is suitable for making short-term predictions at high frequencies.

[0109] In this embodiment, for example, the Performance Management Unit 88 calculates a performance index value (e.g., KPI) based on a plurality of metric data, based on the metrics indicated by these metric data. The Performance Management Unit 88 may calculate a performance index value (e.g., a performance index value related to an end-to-end network slice), which is a comprehensive evaluation of multiple types of metrics that cannot be calculated from a single metric data. The Performance Management Unit 88 may generate comprehensive performance index value data indicating the performance index value that is a comprehensive evaluation.

[0110] Note that the Performance Management Unit 88 may obtain the above-mentioned performance index value file from the Big Data Platform Unit 66. Also, the Performance Management Unit 88 may obtain estimation result data from the AI Unit 70. Then, based on at least one of the performance index value file or the estimation result data, a performance index value such as KPI may be calculated. Note that the Performance Management Unit 88 may directly obtain metric data from the Monitoring Function Unit 72. Then, based on the metric data, a performance index value such as KPI may be calculated.

[0111] In this embodiment, for example, the Fault Management Unit 86 detects the occurrence of a fault in the communication system 1 based on at least any one of the above-mentioned metric data, the above-mentioned alert notification, the above-mentioned estimation result data, and the above-mentioned comprehensive performance index value data. The Fault Management Unit 86 may detect the occurrence of a fault that cannot be detected from a single metric data or a single alert notification, for example, based on a predetermined logic. The Fault Management Unit 86 may generate detection fault data indicating the detected fault.

[0112] Note that the Fault Management Unit 86 may directly obtain metric data and alert notifications from the Monitoring Function Unit 72. Also, the Fault Management Unit 86 may obtain a performance index value file and an alert file from the Big Data Platform Unit 66. Also, the Fault Management Unit 86 may obtain alert message data from the Data Bus Unit 68.

[0113] In this embodiment, for example, the policy manager unit 90 executes a predetermined determination process based on at least any one of the above-described metric data, the above-described performance index value data, the above-described alert message data, the above-described performance index value file, the above-described alert file, the above-described estimation result data, the above-described comprehensive performance index value data, and the above-described detected failure data.

[0114] Then, the policy manager unit 90 may execute an action according to the result of the determination process. For example, the policy manager unit 90 may output an instruction to construct a network slice to the slice manager unit 92. Also, the policy manager unit 90 may output an instruction for scaling or replacing elements to the lifecycle management unit 94 according to the result of the determination process.

[0115] The policy manager unit 90 according to this embodiment is capable of acquiring the performance index value data stored in the data bus unit 68. Then, the policy manager unit 90 may execute a predetermined determination process based on the performance index value data acquired from the data bus unit 68. Also, the policy manager unit 90 may execute a predetermined determination process based on the alert message data stored in the data bus unit 68.

[0116] In this embodiment, for example, the ticket management unit 84 generates a ticket indicating the content to be notified to the administrator of the communication system 1. The ticket management unit 84 may generate a ticket indicating the content of the generated failure data. Also, the ticket management unit 84 may generate a ticket indicating the values of the performance index value data and the metric data. Also, the ticket management unit 84 may generate a ticket indicating the determination result by the policy manager unit 90.

[0117] Then, the ticket management unit 84 notifies the administrator of the communication system 1 of the generated ticket. For example, the ticket management unit 84 may send an email with the generated ticket attached to the email address of the administrator of the communication system 1.

[0118] Hereinafter, the generation of the performance index value file, the determination process based on the performance index value data stored in the data bus unit 68, and the estimation process based on the performance index value data stored in the data bus unit 68 will be further described.

[0119] FIG. 7 is a diagram schematically showing an example of the data bus unit 68 according to the present embodiment. As shown in FIG. 7, the data bus unit 68 according to the present embodiment includes, for example, a plurality of queues 100 that hold performance index value data in a first-in first-out list structure.

[0120] And each queue 100 belongs to either the first queue group 102a or the second queue group 102b.

[0121] Also, in the present embodiment, for example, in the monitoring function unit 72, a plurality of aggregation processes 104 are operating. Each aggregation process 104 has an element to be aggregated in the aggregation process 104 set in advance. For example, each aggregation process 104 has a gNB to be aggregated in the aggregation process 104 set in advance. And each aggregation process 104 acquires metric data from an NF (for example, RU40, DU42, and CU-UP44b) under the gNB that is the aggregation target in the aggregation process 104. And the aggregation process 104 executes an enrichment process of generating performance index value data indicating the communication performance of the gNB based on the acquired metric data.

[0122] Also, in the present embodiment, for example, the aggregation process 104 and the queue 100 are associated in advance. For convenience, in FIG. 7, it is shown that the aggregation process 104 and the queue 100 are associated in a one-to-one relationship, but the aggregation process 104 and the queue 100 may be associated in a many-to-many relationship.

[0123] Hereinafter, the aggregation process 104 associated with the queue 100 included in the first queue group 102a shall be referred to as the first group aggregation process 104a. Also, the aggregation process 104 associated with the queue 100 included in the second queue group 102b shall be referred to as the second group aggregation process 104b.

[0124] Then, each first group aggregation process 104a aggregates metric data from the previous aggregation to the current time associated with the first group aggregation process 104a at a predetermined time interval (for example, every minute), thereby generating performance metric value data.

[0125] The first group aggregation process 104a acquires metric data from one or more NFs associated with the first group aggregation process 104a, for example, at one-minute intervals. Then, the first group aggregation process 104a aggregates the metric data for the same aggregation period to generate performance metric value data for the aggregation period.

[0126] Then, each time the first group aggregation process 104a generates performance metric value data, it enqueues the performance metric value data into one or more queues 100 associated with the first group aggregation process 104a.

[0127] Then, each second group aggregation process 104b aggregates metric data from the previous aggregation to the current time associated with the second group aggregation process 104b at a predetermined time interval (for example, every 15 minutes), thereby generating performance metric value data.

[0128] The second group aggregation process 104b acquires metric data from one or more NFs associated with the second group aggregation process 104b, for example, at 15-minute intervals. Then, the second group aggregation process 104b aggregates the metric data for the same aggregation period to generate performance metric value data for the aggregation period.

[0129] Then, each time the second group aggregation process 104b generates performance metric value data, the performance metric value data is queued in one or more queues 100 associated with the second group aggregation process 104b.

[0130] In this embodiment, the maximum number of performance metric value data that can be stored in the queues 100 included in the first queue group 102a is predetermined. Here, for example, it is assumed that a maximum of 240 pieces of performance metric value data can be stored in the queue 100. That is, the maximum number is "240".

[0131] Also, in this embodiment, the maximum number of performance metric value data that can be stored in the queues 100 included in the second queue group 102b is predetermined. Here, for example, it is assumed that a maximum of 4 pieces of performance metric value data can be stored in the queue 100. That is, the maximum number is "4".

[0132] And, in this embodiment, for example, in the policy manager unit 90, a plurality of determination processes 106 (see FIGS. 8 and 9) are operating. Some of these determination processes 106 execute determination processing based on the performance metric value data stored in the data bus unit 68, and the rest execute determination processing based on the files stored in the big data platform unit 66.

[0133] Among the determination processes 106 according to this embodiment, there is one that acquires performance metric value data indicating the actual value of the performance metrics related to the communication system 1. For example, in response to performance metric value data being queued in the queue 100 included in the first queue group 102a, there is a determination process 106 that acquires the performance metric value data.

[0134] Note that, in this embodiment, for the queues 100 included in the first queue group 102a, any performance metric value data included in the queue 100 can be accessed (acquired) without dequeuing.

[0135] Then, based on the performance metric value data to be acquired, the determination process 106 determines the state of the communication system 1. Here, for example, the state of an element associated with the determination process 106 included in the communication system 1 may be determined. For example, the state of an element that is the aggregation target in the first aggregation process 104a that generated the performance metric value data acquired by the determination process 106 may be determined. Hereinafter, such a determination process 106 will be referred to as an achievement determination process 106a.

[0136] In this embodiment, for example, the achievement determination process 106a and the queue 100 are associated in advance. For the sake of convenience, FIGS. 8 and 9 show that the achievement determination process 106a and the queue 100 are associated in a one-to-one relationship, but the achievement determination process 106a and the queue 100 may be associated in a many-to-many relationship.

[0137] Here, for example, in response to the performance metric value data being enqueued in the queue 100 included in the first queue group 102a, the data bus unit 68 may output a notification indicating that the performance metric value data has been enqueued to one or more achievement determination processes 106a associated with the queue 100.

[0138] Then, in response to receiving the notification, the achievement determination process 106a that has received the notification may acquire the latest performance metric value data stored in the queue 100.

[0139] In addition, in the determination process 106 according to the present embodiment, there is one that acquires estimation result data indicating the estimation result by the estimation process 108 (see FIG. 9) associated with the determination process 106. Then, based on the acquired estimation result data, the determination process 106 determines the state of the communication system 1. Here, for example, the state of an element associated with the determination process 106 included in the communication system 1 may be determined. For example, the state of an element that is the aggregation target in the first aggregation process 104a that generates the performance index value data acquired by the estimation process 108 may be determined. Hereinafter, such a determination process 106 will be referred to as a prediction determination process 106b.

[0140] In addition, in the present embodiment, for example, in the AI unit 70, a plurality of estimation processes 108 (see FIG. 9) are operating. Some of these estimation processes 108 execute an estimation process based on the performance index value data stored in the data bus unit 68, and the rest execute an estimation process based on the files stored in the big data platform unit 66.

[0141] In addition, in the present embodiment, for example, the estimation process 108 and the queue 100 are associated in advance. For the sake of convenience, in FIG. 8, it is shown that the estimation process 108 and the queue 100 are associated in a one-to-one relationship, but the estimation process 108 and the queue 100 may be associated in a many-to-many relationship.

[0142] Then, in the present embodiment, for example, each estimation process 108 acquires the performance index value data stored in the queue 100 included in the first queue group 102a corresponding to the estimation process 108. Then, based on the performance index value data, the estimation process executes an estimation process predetermined in the estimation process 108.

[0143] Here, for example, in response to the performance index value data being queued in the queue 100 included in the first queue group 102a, the estimation process 108 acquires the performance index value data of the most recent predetermined number or the most recent predetermined period, including at least the latest performance index value data among the performance index value data stored in the queue 100.

[0144] Here, for example, in response to the performance index value data being queued in the queue 100 included in the first queue group 102a, the data bus unit 68 may output a notification indicating that the performance index value data has been queued to one or a plurality of estimation processes 108 associated with the queue 100.

[0145] Then, in response to receiving the notification, the estimation process 108 that has received the notification may acquire the performance index value data of the most recent predetermined number or the most recent predetermined period, including at least the latest performance index value data among the performance index value data stored in the queue 100.

[0146] Here, for example, the estimation process 108 shown in FIG. 9 acquires 60 estimated index value data including the latest performance index value data. These performance index value data correspond to the performance index value data of the most recent 60 minutes including the latest performance index value data. Then, the estimation process 108 executes an estimation process based on the performance index value data.

[0147] For example, it is assumed that the first group aggregation process 104a associated with a specific gNB generates performance index value data related to the gNB by aggregating metric data related to elements included in the gNB (for example, elements under the gNB). Then, it is assumed that the estimation process 108 that acquires the performance index value data generated by the first group aggregation process 104a acquires 60 performance index value data including the latest performance index value data stored in the queue 100 in response to the performance index value data being queued in the queue 100.

[0148] In this case, the estimation process 108 uses the learned machine learning model pre-stored in the AI unit 70 to predict the level of the network load of the gNB from the current time to 20 minutes ahead based on these 60 performance metric value data. Here, for example, as the level of the network load of the gNB, predictions such as traffic volume (throughput) and latency may be executed.

[0149] This machine learning model may be, for example, an existing prediction model. Also, for example, this machine learning model may be a learned machine learning model in which supervised learning using a plurality of training data elements has been executed in advance. And each of these plurality of training data elements may include, for example, learning input data indicating the traffic volume of the gNB for 60 minutes up to a given time point that are different from each other, and teacher data indicating the level of the network load (for example, traffic volume and latency) of the gNB from the given time point to 20 minutes ahead of the given time point.

[0150] Note that the estimation process 108 does not necessarily need to acquire a part of the performance metric value data stored in the queue 100 as described above, and may acquire all the performance metric value data stored in the queue 100.

[0151] Then, the estimation process 108 outputs estimation result data indicating the execution result (estimation result) of the estimation process to the prediction determination process 106b associated with the estimation process 108. Then, the prediction determination process 106b acquires the estimation result data. And the prediction determination process 106b determines the state of the communication system 1 based on the acquired estimation result data.

[0152] As described above, the aggregation process 104, the performance determination process 106a, the prediction determination process 106b, and the estimation process 108 are associated with the queue 100 according to the present embodiment.

[0153] Further, in this embodiment, for example, the data bus unit 68 generates a performance index value file including at least a part of the performance index value data stored in the queue 100 at a frequency lower than the frequency at which the AI unit 70 acquires the performance index value data.

[0154] For example, the data bus unit 68 may generate a performance index value file including the performance index value data stored in the queue 100 after the timing when the performance index value file was last generated at a predetermined time interval.

[0155] Here, the time interval may or may not match the time (60 minutes in the above example) corresponding to the maximum number of performance index value data that can be stored in the queue 100 included in the first queue group 102a.

[0156] Further, for example, when all the performance index value data included in the generated performance index value file has been dequeued, the data bus unit 68 may generate a file including all the performance index value data stored in the queue 100. That is, when all the performance index value data stored in the queue 100 has been replaced, a file including all the performance index value data stored in the queue 100 may be generated.

[0157] Also, in this embodiment, when 60 pieces of performance index value data are stored in the queue 100 included in the first queue group 102a and new performance index value data is enqueued, the oldest performance index value data stored in the queue 100 is dequeued. That is, the oldest performance index value data stored in the queue 100 is deleted from the queue 100.

[0158] And in this embodiment, when four performance index value data are stored in the queue 100 included in the second queue group 102b, the data bus unit 68 generates a performance index value file that combines these four performance index value data into one file. Then, the data bus unit 68 outputs the generated performance index value file to the big data platform unit 66.

[0159] And the data bus unit 68 dequeues all the performance index value data stored in the queue 100. That is, all the performance index value data stored in the queue 100 are deleted from the queue 100.

[0160] In this way, the processes executed in response to the generation of the performance index value file are different between the queue 100 included in the first queue group 102a and the queue 100 included in the second queue group 102b. In the queue 100 included in the second queue group 102b, all the performance index value data stored in the queue 100 are deleted from the queue 100 in response to the generation of the performance index value file. On the other hand, in the queue 100 included in the first queue group 102a, dequeuing in response to the generation of the performance index value file is not executed.

[0161] In this embodiment, for example, when purchasing a network service, a purchaser of the network service can select an option related to monitoring settings. In the following description, it is assumed that the purchaser of the network service can select any one of a low-level option, a middle-level option, or a high-level option.

[0162] Here, for example, when the low-level option is selected, when the network service is constructed, not only the elements included in the network service but also the queue 100 associated with the element and the aggregation process 104 associated with the element are generated as shown in FIG. 7. In this case, the performance index value file related to the elements included in the network service is accumulated in the big data platform unit 66.

[0163] Also, for example, when a medium-level option is selected, when constructing a network service, not only the elements included in the network service but also, similar to the low-level option, the queue 100 associated with the element and the aggregation process 104 associated with the element are generated. Further, as shown in FIG. 8, the performance determination process 106a associated with the queue 100 is also generated.

[0164] At this time, the policy manager unit 90 may refer to the inventory data to check the attributes of the elements associated with the generated performance determination process 106a. Then, the policy manager unit 90 may generate a performance determination process 106a in which a workflow according to the confirmed attributes is set. And the performance determination process 106a may execute a determination process by executing the workflow set in the performance determination process 106a.

[0165] For example, the performance determination process 106a may determine the necessity of scale-out based on the acquired performance metric value data.

[0166] And the platform system 30 may execute scale-out of the elements included in the communication system 1, for example, in this embodiment, in response to a determination that scale-out is necessary. For example, the policy manager unit 90, the lifecycle management unit 94, the container management unit 78, and the configuration management unit 76 may cooperate with each other to execute scale-out. For example, in response to a determination that scale-out is necessary based on the performance metric value data related to a specific gNB, scale-out of the DU 42 or CU-UP 44b included in the gNB may be executed.

[0167] For example, the performance evaluation process 106a may determine whether the acquired performance metric value data meets a predetermined first scale-out condition. Here, it may be determined whether the performance metric value indicated by the performance metric value data exceeds a threshold value th1. This performance metric value may be a value indicating the level of network load, such as the traffic volume (throughput), latency, etc. Then, in response to being determined to meet the first scale-out condition (for example, being determined that the performance metric value exceeds the threshold value th1), scale-out of the elements included in the communication system 1 may be executed.

[0168] Also, for example, when a high-level option is selected, when constructing a network service, not only the elements included in the network service, but also the queue 100 associated with the element, the aggregation process 104 associated with the element, and the performance evaluation process 106a associated with the queue 100 are generated, similar to the low-level option and the middle-level option.

[0169] Furthermore, as shown in FIG. 9, the AI unit 70 generates an estimation process 108 associated with the performance evaluation process 106a, and the policy manager unit 90 generates a prediction determination process 106b associated with the performance evaluation process 106a. Here, for example, the estimation process 108 and the prediction determination process 106b may be activated. Also, at this time, instantiation of the learned machine learning model may be executed together. Then, the estimation process 108 may execute an estimation using the machine learning model instantiated in this way.

[0170] And the prediction determination process 106b may execute a predetermined determination process based on the estimation result data output by the estimation process 108 associated with the prediction determination process 106b. For example, the prediction determination process 106b may determine the necessity of scale-out based on the prediction result of the network load.

[0171] In this embodiment, for example, as shown in FIG. 9, in response to performance index value data being enqueued in the queue 100 included in the first queue group 102a, the performance determination process 106a acquires the enqueued performance index value data, and the estimation process 108 may acquire the performance index value data of the most recent predetermined number or the most recent predetermined period including at least the enqueued performance index value data among the performance index value data stored in the queue 100. In this way, in response to performance index value data being enqueued in the queue 100, the enqueued performance index value data may be acquired by both the performance determination process 106a and the estimation process 108.

[0172] Then, the performance determination process 106a may determine whether scale - out is necessary based on the performance index value data to be acquired.

[0173] Also, the estimation process 108 may generate estimation result data indicating a prediction result of network load based on the performance index value data to be acquired. Then, the estimation process 108 may output the generated estimation result data to the prediction determination process 106b. Then, the prediction determination process 106b may acquire the estimation result data.

[0174] Then, the prediction determination process 106b may determine whether scale - out is necessary based on the acquired estimation result data.

[0175] Note that it is not necessary for the AI unit 70 to generate the estimation process 108 and for the policy manager unit 90 to generate the prediction determination process 106b. For example, the performance determination process 106a may generate the estimation process 108 and the policy manager unit 90.

[0176] Then, in this embodiment, for example, in response to it being determined that scale - out is necessary, the platform system 30 executes scale - out of the elements included in the communication system 1.

[0177] For example, the prediction determination process 106b may determine whether the predicted value of the network load indicated by the estimation result data satisfies a predetermined second scale-out condition. For example, it may be determined whether the predicted value exceeds a threshold th2. Here, for example, it may be determined whether any of a plurality of predicted values from the current time to 20 minutes ahead of the current time exceeds the threshold th2. This predicted value may be a value indicating the level of network load, such as traffic volume (throughput), latency, etc. Then, in response to being determined to satisfy the second scale-out condition, scale-out of the elements included in the communication system 1 may be executed. Note that the second scale-out condition may be the same as or different from the above-described first scale-out condition.

[0178] Also, in the present embodiment, when a purchaser of network services selects a medium-level option, the elements included in the communication system 1 may be made specifiable. And a performance determination process 106a for the specified elements may be generated.

[0179] Also, in the present embodiment, when a purchaser of network services selects a high-level option, the elements included in the communication system 1 may be made specifiable. And a performance determination process 106a, an estimation process 108, and a prediction determination process 106b for the specified elements may be generated.

[0180] Also, when a high-level option is selected, the performance determination process 106a may not be generated. Also, options related to monitoring settings may be made changeable according to the purchaser's request.

[0181] Also, in the present embodiment, for example, the AI unit 70 determines at least one (for example, at least one suitable for the communication system 1) from among a plurality of machine learning models used for a given prediction purpose related to the communication system 1.

[0182] Hereinafter, the determination process of the machine learning model will be further described.

[0183] In the following description, it is assumed that the network service purchased by the purchaser has been operating for a certain period of time, and the performance index value file for the elements included in the network service has been pre-accumulated in the big data platform unit 66.

[0184] Also, in the communication system 1 according to the present embodiment, in the AI unit 70, for each of a plurality of given prediction purposes, a plurality of machine learning models used for the prediction purpose are stored, and these machine learning models are in an instantiable state.

[0185] And it is assumed that the inventory database 82 or the AI unit 70 stores the model management data illustrated in FIG. 10, which is data for managing these machine learning models.

[0186] FIG. 10 shows the model management data associated with one prediction purpose. In the present embodiment, for example, in the inventory database 82 or the AI unit 70, the model management data associated with each of the plurality of prediction purposes is stored.

[0187] As shown in FIG. 10, the model management data includes target management data and AI management data.

[0188] The target management data included in the model management data is data associated with the above-described prediction purpose. The target management data includes, for example, a target ID and target data, as shown in FIG. 10. The target ID included in the target management data is, for example, an identifier of the prediction purpose associated with the target management data. The target data included in the target management data is, for example, data indicating the prediction purpose associated with the target management data. In the target management data shown in FIG. 10, the prediction purpose indicated by the target data is expressed as "a1".

[0189] The machine learning model according to this embodiment may output predicted values of at least one type of performance index value. And the type of performance index value to be predicted may be indicated in the target data. For example, the type of performance index value that is the predicted value output by the machine learning model may be indicated in the target data. Specifically, for example, the value of the target data may be "throughput", "latency", "number of resists", "number of connection completions", "number of active users", etc.

[0190] Further, in the target data, prediction purposes (for example, the type of element and the type of performance index value predicted for the type of element) related to specific types of elements included in the communication system 1, such as "UPF throughput", may be indicated.

[0191] Further, in the target data, the type of value calculated based on a plurality of types of performance index values may be indicated. For example, an arithmetic expression for calculating a comprehensive performance evaluation value based on throughput and latency may be set in the value of the target data.

[0192] The AI management data included in the model management data is data for managing the machine learning model used for the prediction purpose associated with the model management data. The model management data includes a plurality of AI data respectively associated with different machine learning models. And the AI data includes an AIID and one or more input performance index value data.

[0193] For example, when three machine learning models with a prediction purpose of "a1" are prepared, as shown in FIG. 10, the model management data includes three AI data. Note that the number of machine learning models used for one prediction purpose is not limited to three.

[0194] The AIID included in the AI data is an identifier of a machine learning model used for a prediction purpose associated with the model management data. In the example of FIG. 10, it is shown that the AIIDs of three machine learning models with the prediction purpose of "a1" are "001", "002", and "003", respectively.

[0195] The input performance index value data included in the AI data is data indicating the types of performance index values input to the machine learning model associated with the AI data. In this embodiment, for example, the same number of input performance index value data as the number of performance index values input to the machine learning model is included in the AI data associated with the machine learning model.

[0196] In the example of FIG. 10, it is shown that the type of performance index value input to the machine learning model with the AIID of "001" is "b11". Also, it is shown that the types of performance index values input to the machine learning model with the AIID of "002" are "b21" and "b22". Also, it is shown that the types of performance index values input to the machine learning model with the AIID of "003" are "b31", "b32", and "b33".

[0197] Thus, the number of types of performance index values input to the machine learning model may differ depending on the machine learning model. Also, in the example of FIG. 10, the number of types of performance index values input to the machine learning model is from 1 to 3, but the number of types of performance index values input to the machine learning model may be 4 or more.

[0198] Also, the types of performance index values input to a certain machine learning model may be included in the types of performance index values input to another machine learning model. Also, a part of the types of performance index values input to a certain machine learning model and a part of the types of performance index values input to another machine learning model may overlap. For example, "b11" and "b21" may be the same type of performance index value.

[0199] In addition, in the present embodiment, the type of performance index value indicated by the input performance index value data and the type of performance index value associated with the prediction target indicated by the target data may be the same or different.

[0200] For example, "a1" and "b11" may be the same type of performance index value. For example, when predicting the throughput after a certain point in time based on the output when the actual value of the throughput at a certain point in time is input to the machine learning model, it corresponds to the case where the type of performance index value indicated by the input performance index value data and the type of performance index value indicated by the target data are the same.

[0201] Specific examples of the type of performance index value to be input include "throughput", "latency", "number of resistors", "number of connection completions", "number of active users", and the like.

[0202] Here, the input performance index value data may indicate the type of element and the type of performance index value for the type of element. For example, when the performance index value "throughput" for the element "UPF" is input to the machine learning model, the AI data associated with the machine learning model may include the input performance index value data with the value "UPF throughput".

[0203] In the present embodiment, for example, as shown in FIG. 11, the AI unit 70 instantiates the unlearned machine learning model 110 and generates a learning process 112 and a test process 114 associated with the machine learning model 110.

[0204] Here, for example, it is assumed that three unlearned machine learning models 110 with AIIDs of "001", "002", and "003" are instantiated. Hereinafter, the machine learning models 110 with AIIDs of "001", "002", and "003" will be respectively expressed as machine learning model 110a, machine learning model 110b, and machine learning model 110c.

[0205] Then, it is assumed that a learning process 112a associated with the machine learning model 110a and a test process 114a associated with the machine learning model 110a are generated. Also, it is assumed that a learning process 112b associated with the machine learning model 110b and a test process 114b associated with the machine learning model 110b are generated. Also, it is assumed that a learning process 112c associated with the machine learning model 110c and a test process 114c associated with the machine learning model 110c are generated.

[0206] In the present embodiment, as described above, the performance index value file related to the elements included in the network service purchased by the purchaser of the network service is stored in the big data platform unit 66.

[0207] Then, in the present embodiment, for example, the AI unit 70 acquires data indicating the time series of the actual values of a plurality of types of performance index values related to the communication system 1.

[0208] A part of the data thus acquired corresponds to test data indicating the time series of the actual values of a plurality of types of performance index values related to the communication system 1. And the rest corresponds to training data indicating the time series of the actual values of a plurality of types of performance index values related to the communication system 1.

[0209] In the following description, it is assumed that the plurality of types at least include "a1", "b11", "b21", "b22", "b31", "b32", "b33" shown in FIG. 10.

[0210] Here, the training data is different from the test data. For example, data indicating the performance index values up to a certain point in time may be used as the training data, and data indicating the performance index values after that point in time may be used as the test data. Or, data indicating the performance index values up to a certain point in time may be used as the test data, and data indicating the performance index values after that point in time may be used as the training data.

[0211] Here, for example, the AI unit 70 acquires at least one performance index value file related to elements included in the network service purchased by the above-mentioned purchaser and stored in the big data platform unit 66.

[0212] Then, the learning process 112 generates a training data element set shown in FIG. 12 based on training data that is part of the data included in the acquired performance index value file. As shown in FIG. 12, the training data element set includes a plurality of training data elements, and each training data element includes learning input data and teacher data.

[0213] Here, for example, the learning process 112a may generate a training data element including learning input data including performance index value data of which the type of performance index value is "b11" and teacher data including performance index value data of which the type of performance index value is "a1", both included in the performance index value file.

[0214] Then, the learning process 112b may generate a training data element including learning input data including performance index value data of which the type of performance index value is "b21" and performance index value data of which the type of performance index value is "b22", and teacher data including performance index value data of which the type of performance index value is "a1", all included in the performance index value file.

[0215] Then, the learning process 112c may generate a training data element including learning input data including performance index value data of which the type of performance index value is "b31", performance index value data of which the type of performance index value is "b32", and performance index value data of which the type of performance index value is "b33", and teacher data including performance index value data of which the type of performance index value is "a1", all included in the performance index value file.

[0216] FIG. 13 is a diagram schematically showing an example of a training data element generated by the learning process 112c. Here, for example, performance index value data D1 indicating a performance index value of type "b31", performance index value data D2 indicating a performance index value of type "b32", and performance index value data D3 indicating a performance index value of type "b33" for a period of length T1 (e.g., 60 minutes) up to a certain reference time point are generated. And teacher data D4 including performance index value data indicating a performance index value of type "a1" for a period of length T2 (e.g., a period from the reference time point to 20 minutes ahead of that time point) from the reference time point is generated. And a training data element including the learning input data and the teacher data thus generated is generated.

[0217] And a set of training data elements including a plurality of training data elements generated as described above for various reference time points is generated by the learning process 112c.

[0218] Similarly, sets of training data elements are generated by the learning processes 112a and 112b.

[0219] Here, when the machine learning model 110 accepts the input of performance index value data at a plurality of time points, the learning input data includes performance index value data at a plurality of time points as shown in FIG. 13. On the other hand, when the machine learning model 110 accepts the input of performance index value data at one time point, the learning input data includes performance index value data at one time point.

[0220] Also, when the machine learning model 110 outputs predicted values at a plurality of time points, the teacher data includes performance index value data at a plurality of time points as shown in FIG. 13. On the other hand, when the machine learning model 110 outputs a predicted value at one time point, the teacher data includes performance index value data at one time point.

[0221] Then, the learning process 112 uses the set of training data elements generated as described above to perform learning of the machine learning model 110 associated with the learning process 112, thereby generating a learned machine learning model 110.

[0222] Here, for example, as shown in FIG. 13, the learning process 112c may calculate a value of a given evaluation function (error function) based on the output D5 when the learning input data included in the training data element is input to the machine learning model 110c and the teacher data D4 included in the training data element. Then, the learning process 112c may update the parameters of the machine learning model 110c based on the calculated value of the evaluation function. Then, by updating the parameters of the machine learning model 110c based on each of the plurality of training data elements included in the set of training data elements generated by the learning process 112c, learning of the machine learning model 110c may be performed, and as a result, a learned machine learning model 110c may be generated.

[0223] Similarly, by performing learning on the machine learning model 110a using the set of training data elements generated by the learning process 112a, a learned machine learning model 110a may be generated. Also, by performing learning on the machine learning model 110b using the set of training data elements generated by the learning process 112b, a learned machine learning model 110b may be generated.

[0224] Here, as described above, the machine learning model may output a predicted value calculated based on a plurality of types of performance index values.

[0225] In this case, the teacher data may include performance index value data indicating the plurality of types of performance index values. Then, a comprehensive performance evaluation value may be calculated according to a given calculation formula based on the plurality of types of performance index values. Then, a value of a given evaluation function (error function) may be calculated based on the calculated comprehensive performance evaluation value and the predicted value of the comprehensive performance evaluation value output from the machine learning model.

[0226] Alternatively, training data elements including teacher data in which a comprehensive performance evaluation value calculated according to a given calculation formula based on a plurality of types of performance index values is set may be generated. Then, based on the comprehensive performance evaluation value indicated by the teacher data and the predicted value of the comprehensive performance evaluation value output from the machine learning model, the value of a given evaluation function (error function) may be calculated.

[0227] Also, as described above, the machine learning model may output a predicted value of the same type of performance index value as the input performance index value.

[0228] In this case, training data elements including learning input data indicating a certain type of performance index value for a period of length T1 up to a reference time point and teacher data indicating the same type of performance index value for a period of length T2 from the reference time point may be generated. Then, based on the performance evaluation value indicated by the teacher data and the predicted value output from the machine learning model, the value of a given evaluation function (error function) may be calculated.

[0229] Also, the test process 114 generates a set of test data elements shown in FIG. 14 based on the test data included in the performance index value file obtained as described above. As shown in FIG. 14, the set of test data elements includes a plurality of test data elements, and each test data element includes test input data and comparison target data.

[0230] Here, for example, the test process 114a may generate a test data element including test input data including performance index value data of which the type of performance index value is "b11" included in the performance index value file and comparison target data including performance index value data of which the type of performance index value is "a1".

[0231] Then, the test process 114b may generate a test data element including test input data containing performance index value data with the type of performance index value being "b21" and performance index value data with the type of performance index value being "b22" included in the performance index value file, and comparison target data containing performance index value data with the type of performance index value being "a1".

[0232] Then, the test process 114c may generate a test data element including test input data containing performance index value data with the type of performance index value being "b31", performance index value data with the type of performance index value being "b32", and performance index value data with the type of performance index value being "b33" included in the performance index value file, and comparison target data containing performance index value data with the type of performance index value being "a1".

[0233] FIG. 15 is a diagram schematically showing an example of a test data element generated by the test process 114c. Here, for example, test input data including performance index value data D6 indicating a performance index value of type "b31", performance index value data D7 indicating a performance index value of type "b32", and performance index value data D8 indicating a performance index value of type "b33" for a period of length T1 (e.g., 60 minutes) up to a certain reference time point is generated. And comparison target data D9 including performance index value data indicating a performance index value of type "a1" for a period of length T2 (e.g., a period from the reference time point to 20 minutes ahead of that time point) from the reference time point is generated. And a test data element including the thus generated test input data and comparison target data is generated.

[0234] Then, a set of test data elements including a plurality of test data elements generated as described above for various reference time points will be generated by the test process 114c.

[0235] Similarly, a set of test data elements is generated by the test process 114a and the test process 114b.

[0236] In this way, test data elements in a format similar to the format of the training data will be generated.

[0237] In this embodiment, for example, the types of performance index values indicated by the performance index value data included in the learning input data corresponding to the machine learning model are the same as the types of performance index values indicated by the performance index value data included in the test input data corresponding to the machine learning model. Also, the types of performance index values indicated by the performance index value data included in the teacher data corresponding to the machine learning model are the same as the types of performance index values indicated by the performance index value data included in the comparison target data corresponding to the machine learning model.

[0238] Also, in this embodiment, for example, the number of performance index value data included in the learning input data corresponding to the machine learning model is the same as the number of performance index value data included in the test input data corresponding to the machine learning model. Also, the number of performance index value data included in the teacher data corresponding to the machine learning model is the same as the number of performance index value data included in the comparison target data corresponding to the machine learning model.

[0239] Note that in this embodiment, as described above, the training data and the test data are different data, and the training data is not diverted to the test data.

[0240] Then, in this embodiment, for example, the AI unit 70 inputs input data corresponding to the machine learning model 110 to each of the plurality of learned machine learning models 110 used for a given prediction purpose related to the communication system 1. Here, the input data is a part of the test data and is data indicating the actual value of the performance index value at at least one time point of at least one type. Also, the input data input to each of the plurality of learned machine learning models 110 is different from each other. Then, in this embodiment, for example, the AI unit 70 obtains, as the output of the machine learning model 110, a predicted value at a prediction time point after any of the above at least one time point.

[0241] Here, as described above, the type of the actual value indicated by the input data and the type of the predicted value output from the machine learning model 110 may be the same or different.

[0242] For example, the test process 114 inputs test input data indicating a performance index value at at least one point in time included in the test data element into the trained machine learning model 110. Then, the test process 114 obtains the output when the test input data is input to the machine learning model 110. This output indicates a predicted value at a prediction time point later than any of the above-mentioned at least one point in time. For example, as shown in FIG. 15, the test process 114c obtains an output D10 when the test data element is input to the machine learning model 110c. The predicted value indicated by this output D10 is a predicted value of a performance index value of type "a1".

[0243] Then, in this embodiment, for each of the plurality of trained machine learning models, the AI unit 70 evaluates the accuracy of the prediction related to the prediction purpose by the machine learning model based on the obtained predicted value and a part of the actual value indicating the at least one type of prediction time point corresponding to the predicted value in the test data.

[0244] For example, the test process 114 evaluates the accuracy of the prediction related to the above-mentioned prediction purpose by the machine learning model 110 based on the comparison target data included in the test data element and the output when the test input data included in the test data element is input to the machine learning model 110.

[0245] Here, for example, the test process 114 may calculate the value of a given evaluation function (error function) based on the comparison target data included in the test data element and the output when the test input data included in the test data element is input to the machine learning model 110. Then, the test process 114 may calculate the representative value (sum, average, etc.) of the evaluation function calculated for a plurality of test data elements as the evaluation value of the accuracy of the prediction related to the above-described prediction purpose by the machine learning model.

[0246] For example, the test process 114c calculates the value of a given evaluation function based on the comparison target data D9 and the output D10 included in the test data element. Then, the test process 114c evaluates the accuracy of the prediction related to the prediction purpose "a1" by the machine learning model 110c based on the values of the evaluation function calculated for each of the plurality of test data elements included in the test data element set.

[0247] Then, based on the above-described evaluation results of the accuracy of the prediction for each of the plurality of machine learning models 110, the AI unit 70 determines at least one of the plurality of learned machine learning models 110. Here, for example, a machine learning model suitable for the communication system 1 is determined.

[0248] Here, for example, the AI unit 70 may determine the machine learning model 110 with the smallest representative value of the evaluation function as the machine learning model suitable for the network service. Alternatively, for example, the AI unit 70 may determine one or more machine learning models 110 with a representative value of the evaluation function smaller than a predetermined value as the machine learning model suitable for the network service.

[0249] Here, as described above, the machine learning model may output a predicted value calculated based on a plurality of types of performance index values.

[0250] In this case, the comparison target data may include performance index value data indicating the plurality of types of performance index values. Then, based on the plurality of types of performance index values, a comprehensive performance evaluation value may be calculated according to a given calculation formula. And based on the calculated comprehensive performance evaluation value and the predicted value of the comprehensive performance evaluation value output from the machine learning model, the value of a given evaluation function (error function) may be calculated.

[0251] Alternatively, a test data element including comparison target data in which a comprehensive performance evaluation value calculated according to a given calculation formula based on a plurality of types of performance index values is set may be generated. And based on the comprehensive performance evaluation value indicated by the comparison target data and the predicted value of the comprehensive performance evaluation value output from the machine learning model, the value of a given evaluation function (error function) may be calculated.

[0252] Also, as described above, the machine learning model may output a predicted value of the same type of performance index value as the input performance index value.

[0253] In this case, a test data element including test input data indicating a certain type of performance index value for a period of length T1 up to a reference time point and comparison target data indicating the certain type of performance index value for a period of length T2 from the reference time point may be generated. And based on the performance evaluation value indicated by the comparison target data and the predicted value output from the machine learning model, the value of a given evaluation function (error function) may be calculated.

[0254] Then, the AI unit 70 may add the types of performance metric values that need to be added for using the machine learning model determined in this way to the monitoring targets by the performance determination process 106a. For example, a performance determination process 106a associated with the types of performance metric values that need to be added for using the machine learning model determined in this way may be generated. And the determination process (in other words, the process of monitoring at least one type of performance metric value related to the communication system 1) may be executed by the generated performance determination process 106a.

[0255] And an estimation process 108 and a prediction determination process 106b associated with the performance determination process 106a may be generated. Here, for example, the estimation process 108 and the prediction determination process 106b may be activated. And the estimation process 108 may predict the performance metric values of the communication system 1 using the learned machine learning model determined in this way.

[0256] Alternatively, the machine learning model determined in this way may be recommended to users such as purchasers. For example, the recommendation screen shown in FIG. 16 may be displayed on the terminal used by the purchaser of the network service. And in response to the purchase button 120 arranged on the recommendation screen being clicked, the performance determination process 106a, the estimation process 108, and the prediction determination process 106b may be generated. And the estimation process 108 may predict the performance metric values of the communication system 1 using the learned machine learning model determined in this way.

[0257] Also, in the present embodiment, for each of the plurality of prediction purposes, at least one (for example, at least one suitable for the communication system 1) may be determined from among the plurality of machine learning models used for the prediction purpose.

[0258] In the above description, the processes described above are executed based on the performance index value files stored in the big data platform unit 66. However, the processes described above may also be executed based on the performance index value files stored in the queue 100.

[0259] Also, in the present embodiment, even if the types of performance index values input to the machine learning model are the same, if the length of time covered by the performance index values input to the machine learning model (e.g., T1 described above) or the length of time covered by the predicted values output from the machine learning model (e.g., T2 described above) is different, these machine learning models may be treated as different machine learning models.

[0260] Here, an example of the processing flow regarding the determination of a machine learning model suitable for a network service purchased by a specific purchaser, which is a part of the communication system 1 and is performed by the platform system 30 according to the present embodiment, will be described with reference to the flowchart illustrated in FIG. 17.

[0261] Note that in this processing example, it is assumed that the prediction purpose of the machine learning model to be determined is given. And it is assumed that there are a plurality of machine learning models used for the prediction purpose.

[0262] First, the AI unit 70 acquires at least one performance index value file related to elements included in the network service purchased by the purchaser, which is stored in the big data platform unit 66 (S101). In the following processing, a part of the data included in the performance index value file acquired by the processing shown in S101 is used as training data, and the rest is used as test data.

[0263] Then, the AI unit 70 generates a plurality of training data element sets associated with each of the plurality of machine learning models used for the prediction purpose based on the training data acquired by the processing shown in S101 (S102).

[0264] Then, the AI unit 70 generates a plurality of trained machine learning models by performing learning for each of the plurality of machine learning models using the set of training data elements generated by the process shown in S102 (S103).

[0265] Then, the AI unit 70 generates a plurality of sets of test data elements associated with each of the plurality of machine learning models based on the test data acquired by the process shown in S101 (S104).

[0266] Then, for each of the plurality of trained machine learning models generated by the process shown in S103, the AI unit 70 calculates an evaluation value indicating the accuracy of the prediction related to the prediction purpose for the machine learning model using the set of test data elements associated with the machine learning model (S105).

[0267] Then, based on the evaluation values calculated for the plurality of machine learning models in the process shown in S105, the AI unit 70 determines at least one of the plurality of machine learning models as a machine learning model suitable for the network service (S106), and the process shown in this processing example ends.

[0268] For the same prediction purpose related to the communication system 1, a plurality of machine learning models with different types of input data may be prepared.

[0269] According to the present embodiment, as described above, based on the actual performance values of the performance index values related to the elements included in the communication system 1 (for example, the elements included in the network service), the prediction accuracy for each of the plurality of machine learning models is evaluated. Then, based on the evaluation result of the accuracy, at least one of the plurality of machine learning models is determined.

[0270] In this way, according to the present embodiment, it becomes possible to accurately determine a machine learning model suitable for the communication system from among the plurality of machine learning models used for a given prediction purpose related to the communication system.

[0271] Further, in the present embodiment, the machine learning model may be determined based on the types of performance index values that need to be added to the monitoring target in order to use the machine learning model, as described below.

[0272] In the following description, it is assumed that a purchaser of network services has selected medium-level options when purchasing network services. And for some elements included in the network service, it is assumed that the determination process by the performance determination process 106a is being executed.

[0273] And it is assumed that the performance determination target data indicating the types of elements monitored by the performance determination process 106a and the types of performance index values monitored for the elements are stored in the inventory database 82.

[0274] In this case, for each of the plurality of machine learning models used for a given prediction purpose related to the communication system 1, the AI unit 70 may specify the additional performance index value types that are the types of performance index values that need to be added to the monitoring target in order to use the machine learning model.

[0275] For example, for each of the plurality of machine learning models, among the types of performance index values that are inputs to the machine learning model, the types of performance index values not included in the monitoring target may be specified as the additional performance index value types.

[0276] For example, for each of the plurality of AI data included in the model management data including the target management data indicating a given prediction purpose, it may be determined whether the types of performance index values indicated by the input performance index value data included in the AI data are shown in the above-described performance determination target data.

[0277] Among the types of performance index values indicated by the input performance index value data, the types of performance index values not shown in the above-mentioned performance determination target data may be determined as additional performance index value types for the machine learning model associated with the AI data.

[0278] Then, the AI unit 70 may determine at least one of the plurality of machine learning models based on the additional performance index value types specified for each machine learning model.

[0279] Here, the AI unit 70 may determine at least one of the plurality of machine learning models based on the number of additional performance index value types. For example, the machine learning model associated with the AI data with the smallest number of specified additional performance index value types may be determined. Alternatively, the machine learning model associated with the AI data with the number of specified additional performance index value types less than a predetermined number may be determined.

[0280] Further, the AI unit 70 may determine at least one of the plurality of machine learning models based on the ratio of the number of additional performance index value types to the number of types of performance index values input to the machine learning model. For example, the machine learning model associated with the AI data with the smallest ratio of the number of specified additional performance index value types to the total number of input performance index value data may be determined. Alternatively, the machine learning model associated with the AI data with the ratio of the number of specified additional performance index value types less than a predetermined ratio to the total number of input performance index value data may be determined.

[0281] Also, in the present embodiment, the AI unit 70 may determine the machine learning model based on the evaluation result of the prediction accuracy and the above-mentioned additional performance index value types.

[0282] For example, a plurality of machine learning models may be determined based on the evaluation result of the prediction accuracy. Then, based on the additional performance index value types, at least one machine learning model may be determined by narrowing down the plurality of machine learning models thus determined.

[0283] Also, for example, a plurality of machine learning models may be determined based on the types of additional performance index values. Then, based on the evaluation result of the prediction accuracy, at least one machine learning model may be determined by narrowing down the plurality of machine learning models thus determined.

[0284] And, as described above, the AI unit 70 may add the types of performance index values that need to be added for using the machine learning model thus determined to the monitoring targets by the actual result determination process 106a. An actual result determination process 106a associated with the types of performance index values that need to be added for using the trained machine learning model thus determined may be generated.

[0285] And, an estimation process 108 and a prediction determination process 106b associated with the actual result determination process 106a may be generated. Here, for example, the estimation process 108 and the prediction determination process 106b may be activated. And the estimation process 108 may predict the performance index values of the communication system 1 using the trained machine learning model thus determined.

[0286] Even if the prediction purposes are the same, various patterns can be assumed for the types of actual result values of the performance index values input to the machine learning model, which correspond to the explanatory variables for prediction.

[0287] On the other hand, since the types of performance index values for which the monitoring of the actual result values is being performed vary depending on the situation, even for machine learning models used for the same prediction purpose, some may be suitable for the communication system 1 while others may not be.

[0288] In the examples described above, based on the types of additional performance index values, a machine learning model suitable for the communication system 1 is determined. For example, a machine learning model with a small burden of additional monitoring is determined. In this way, according to the examples described above, it becomes possible to more accurately determine a machine learning model suitable for the communication system 1 that is used for predicting the performance index values of the communication system 1.

[0289] Also, in the present embodiment, the AI unit 70 may determine a machine learning model used for prediction in each of a plurality of time zones for each of the time zones.

[0290] For example, based on the performance index value data during the daytime, the prediction accuracy for each of a plurality of machine learning models may be evaluated. Then, based on the evaluation result of the accuracy, a machine learning model used for prediction during the daytime may be determined. Also, based on the performance index value data at night, the prediction accuracy for each of a plurality of machine learning models may be evaluated. Then, based on the evaluation result of the accuracy, a machine learning model used for prediction at night may be determined.

[0291] Also, for example, based on the performance index value data on weekdays, the prediction accuracy for each of a plurality of machine learning models may be evaluated. Then, based on the evaluation result of the accuracy, a machine learning model used for prediction on weekdays may be determined. Also, based on the performance index value data on holidays, the prediction accuracy for each of a plurality of machine learning models may be evaluated. Then, based on the evaluation result of the accuracy, a machine learning model used for prediction on holidays may be determined.

[0292] In this way, for each of a plurality of time zones, it becomes possible to accurately determine a machine learning model suitable for the time zone that is used for predicting the performance index values of the communication system 1.

[0293] Note that the present invention is not limited to the above-described embodiments.

[0294] For example, in this embodiment, instead of scaling out elements of RAN 32 such as gNB, scaling out elements of the core network system 34 may be performed. For example, scaling out of AMF 46, SMF 48, or UPF 50 may be performed. Also, in this case, performance metric value data related to elements of the core network system 34 may be used for determining whether to perform scaling out. Alternatively, for this determination, performance metric value data related to elements of RAN 32 and elements of the core network system 34 may be used.

[0295] Similarly, transport scaling out may be performed.

[0296] Also, in this embodiment, a purchaser of network services may be able to refer to the content of a performance metric value file for elements included in the network service stored in the big data platform unit 66, for example, via a dashboard screen.

[0297] Also, the determination process of the machine learning model described above and processes related to the determination process may be executed by functional modules other than the AI unit 70.

[0298] Also, the functional units according to this embodiment are not limited to those shown in FIG. 3.

[0299] Also, the functional units according to this embodiment do not have to be NFs in 5G. For example, the functional units according to this embodiment may be network nodes in 4G such as eNodeB, vDU, vCU, P-GW (Packet Data Network Gateway), S-GW (Serving Gateway), MME (Mobility Management Entity), HSS (Home Subscriber Server), etc.

[0300] In addition, the functional unit according to the present embodiment may be implemented using a hypervisor type or host type virtualization technology instead of a container type virtualization technology. Further, the functional unit according to the present embodiment does not necessarily have to be implemented by software, and may be implemented by hardware such as an electronic circuit. Further, the functional unit according to the present embodiment may be implemented by a combination of an electronic circuit and software.

[0301] The technology described in the present disclosure can also be expressed as follows. [1] Test data acquisition means for acquiring test data showing a time series of actual values of a plurality of types of performance index values related to a communication system, For each of a plurality of trained machine learning models used for a given prediction purpose related to the communication system, where the input data showing the actual values at at least one time point of at least one of the types, which is a part of the test data, are different from each other, input the input data corresponding to the machine learning model, and obtain, as the output of the machine learning model, a predicted value at a prediction time point after any of the time points. For each of the plurality of trained machine learning models, based on the obtained predicted value and a part of the test data showing the actual value at the prediction time point of at least one of the types corresponding to the predicted value, prediction accuracy evaluation means for evaluating the accuracy of the prediction related to the prediction purpose by the machine learning model. Model determination means for determining at least one of the plurality of trained machine learning models based on the evaluation result of the accuracy. A model determination system characterized by including the above. [2] The machine learning model outputs a predicted value of at least one type of the performance index value, The type of the actual value indicated by the input data is different from the type of the predicted value. The model determination system according to [1], characterized by the above. [3] The machine learning model outputs predicted values of at least one type of the performance index values, and the type of the actual performance values indicated by the input data is the same as the type of the predicted values. The model determination system according to [1], characterized in that. [4] Learning means for generating the plurality of trained machine learning models by performing learning using data indicating actual performance values of the plurality of types of performance index values related to the communication system, which are data different from the test data. The model determination system according to any one of [1] to [3], characterized in that. [5] Monitoring means for monitoring at least one type of performance index value related to the communication system, Additional performance index value type specifying means for specifying, for each of the plurality of trained machine learning models, the type of additional performance index values that need to be added to the monitoring target for using the machine learning model. The model determination means determines the machine learning model based on the evaluation result of the accuracy and the type of the additional performance index values. The model determination system according to any one of [1] to [4], characterized in that. [6] Monitoring target addition means for adding, to the monitoring target by the monitoring means, the performance index values of the type of the additional performance index values that need to be added for using the determined machine learning model. The model determination system according to [5], characterized in that. [7] Monitoring means for monitoring at least one type of performance index value related to the communication system, Monitoring target addition means for adding, to the monitoring target by the monitoring means, the type of performance index values that need to be added for using the determined machine learning model. The model determination system according to any one of [1] to [4], characterized in that. [8] Predicting means for predicting a performance index value of the communication system using the machine learning model to be determined, The model determination system according to any one of [1] to [7], characterized in that. [9] For each of a plurality of time zones, the model determination means determines the machine learning model used for prediction in that time zone. The model determination system according to any one of [1] to [8], characterized in that.

[10] Obtaining test data showing a time series of actual values of a plurality of types of performance index values related to a communication system; For each of a plurality of trained machine learning models used for a given prediction purpose related to the communication system, where the input data showing the actual values at at least one time point of at least one of the types, which is part of the test data, are different from each other, inputting the input data corresponding to the machine learning model, and obtaining, as the output of the machine learning model, a predicted value at a prediction time point after any of the time points; For each of the plurality of trained machine learning models, based on the obtained predicted value and a part of the test data showing the actual value at the prediction time point of at least one of the types corresponding to the predicted value, evaluating the accuracy of the prediction related to the prediction purpose by the machine learning model; Determining at least one of the plurality of trained machine learning models based on the evaluation result of the accuracy; A model determination method, characterized by including.

Claims

1. A data acquisition process for acquiring performance data indicating time series of actual values of a plurality of types of performance index values related to a communication system, for each of a plurality of trained machine learning models in which at least one of the corresponding types of performance index values is different, inputting input data that is part of the acquired performance data and indicates actual values at at least one time point up to a first time point of the performance index value corresponding to the machine learning model, and acquiring, as an output of the machine learning model, a predicted value at a second time point after the first time point; a predicted value acquisition process; a prediction accuracy evaluation process for evaluating the prediction accuracy for each of the plurality of trained machine learning models based on the predicted value at the second time point of the plurality of trained machine learning models and the actual value at the second time point of the acquired performance data; a model determination process for determining, based on the evaluation result of the accuracy, to use at least one of the plurality of trained machine learning models for prediction at a third time point after the second time point. A model determination system.

2. The machine learning model outputs predicted values of at least one type of the performance index value, wherein the type of the actual value at at least one time point up to the first time point indicated by the input data input to the machine learning model is different from the type of the predicted value output by the machine learning model. The model determination system according to Claim 1.

3. The machine learning model outputs predicted values of at least one type of the performance index value, wherein the type of the actual value at at least one time point up to the first time point indicated by the input data input to the machine learning model is the same as the type of the predicted value output by the machine learning model. The model determination system according to Claim 1.

4. A learning process for generating the plurality of trained machine learning models by performing learning using data indicating actual values of the plurality of types of performance index values related to the communication system, which is data different from the performance data. The model determination system according to Claim 1.

5. A monitoring process for monitoring at least one type of performance index value related to the communication system, For each of the plurality of learned machine learning models, an additional performance metric type identification process for identifying an additional performance metric type, which is the type of performance metric value that needs to be added to the object of the monitoring in order to use the machine learning model, is further executed. In the model determination process, based on the evaluation result of the accuracy and the additional performance metric type, a machine learning model used for prediction at the third time point is determined. The model determination system according to claim 1.

6. A monitoring target addition process for adding a performance metric value of the additional performance metric type that needs to be added in order to use the machine learning model used for prediction at the third time point to the monitoring target in the monitoring process is further executed. The model determination system according to claim 5.

7. A monitoring process for monitoring at least one type of performance metric value related to the communication system, A monitoring target addition process for adding a type of performance metric value that needs to be added in order to use the machine learning model used for prediction at the third time point to the monitoring target in the monitoring process is further executed. The model determination system according to claim 1.

8. A prediction process for predicting the performance metric value of the communication system using the machine learning model used for prediction at the third time point is further executed. The model determination system according to claim 1.

9. In the model determination process, for each of a plurality of time periods, the machine learning model used for prediction in that time period is determined. The model determination system according to claim 1.

10. Obtaining performance data indicating a time series of actual values of a plurality of types of performance metric values related to the communication system, For each of a plurality of learned machine learning models in which at least one of the corresponding types of performance metric values is different, inputting input data indicating actual values at at least one time point up to the first time point of the performance metric value corresponding to the machine learning model, which is a part of the obtained performance data, and obtaining a predicted value at a second time point after the first time point as the output of the machine learning model. Evaluating the accuracy of prediction for each of the plurality of learned machine learning models based on the predicted value at the second time point of the plurality of learned machine learning models and the actual value at the second time point of the obtained performance data. Based on the evaluation result of the accuracy, determining to use at least one of the plurality of learned machine learning models for prediction at a third time point after the second time point; A model determination method executed by one or more computers, including the above.

Images