Methods, systems, and computer-readable media for autonomous network test case generation

Abstract

A method for autonomously generating a network function test case includes detecting a fault condition in a network function of a core network of a telecommunications network. The method includes, in response to detecting the fault condition, autonomously generating a network function test case based on the fault condition. The network function test case includes one or more network state parameters detected at the time of detecting the fault condition. The method includes providing the network function test case to a network test system configured to execute the network function test case by repeating the one or more network state parameters and determining whether the network function repeats the fault condition.

Description

[0001] Priority Declaration

[0002] This application claims priority to U.S. Patent Application Serial No. 17 / 207,393, filed March 19, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This article describes topics related to testing telecommunications networks. More specifically, it describes methods, systems, and computer-readable media for generating test cases for autonomous networks. Background Technology

[0004] The 3rd Generation Partnership Project (3GPP) is a collaborative project among telecommunications standards associations. 3GPP defines the specifications for mobile phone systems in telecommunications networks, including 3G, 4G, and Long Term Evolution (LTE) networks.

[0005] The next generation of networks under 3GPP is 5G. The goals of the 5G specification are high data rates, reduced latency, energy efficiency, lower costs, higher system capacity, and an increased number of connected devices.

[0006] In multi-vendor environments, various 3GPP-defined 5G network functions are deployed. During production deployments, network functions may experience failures / errors under different traffic scenarios. Telecom vendors are adopting continuous integration / continuous delivery (CI / CD) processes. This introduces automated testing processes before and after software delivery.

[0007] Given these and other difficulties, there is a need for methods, systems, and computer-readable media for generating autonomous network test cases. Summary of the Invention

[0008] A method for autonomously generating network function test cases includes detecting fault conditions in the network functions of a core network of a telecommunications network. The method includes autonomously generating network function test cases based on the detected fault conditions. The network function test cases include one or more network state parameters detected when the fault conditions are detected. The method includes providing the network function test cases to a network testing system configured to execute the network function test cases by repeating the one or more network state parameters and determining whether the network function repeats the fault conditions.

[0009] According to another aspect of the subject matter described in this article, detecting fault conditions includes analyzing one or more network performance metrics or one or more network performance alerts, or both.

[0010] According to another aspect of the subject matter described in this article, detecting fault conditions includes parsing one or more network function application logs or one or more network function traces, or both.

[0011] According to another aspect of the subject described in this article, executing network function test cases involves repeatedly executing network function test cases for multiple different network traffic conditions.

[0012] According to another aspect of the subject matter described in this article, detecting fault conditions includes providing network function monitoring data to a machine learning classifier trained based on training monitoring data of network functions.

[0013] According to another aspect of the subject matter described herein, providing network function test cases to a network testing system includes associating network function test cases with one or more other network function test cases for network functions from a reference library of test cases for network functions and the core network of the telecommunications network.

[0014] According to another aspect of the subject matter described herein, providing network function test cases to a network testing system includes identifying a reference library of test cases for network function test cases that do not yet exist in the core network of the telecommunications network.

[0015] According to another aspect of the subject matter described herein, a system for autonomously generating network function test cases includes at least one processor and memory. The system also includes an autonomous test case generator implemented by the at least one processor, configured to detect fault conditions in the network functions of the core network of a telecommunications network, and autonomously generate network function test cases based on the detected fault conditions. The network function test cases include one or more network state parameters detected when the fault conditions are detected. The autonomous test case generator is configured to provide the network function test cases to a network testing system configured to execute the network function test cases by repeating one or more network state parameters and determining whether the network function repeats the fault conditions.

[0016] According to another aspect of the subject matter described in this article, detecting fault conditions includes analyzing one or more network performance metrics or one or more network performance alerts, or both.

[0017] According to another aspect of the subject matter described in this article, detecting fault conditions includes parsing one or more network function application logs or one or more network function traces, or both.

[0018] According to another aspect of the subject described in this article, executing network function test cases involves repeatedly executing network function test cases for multiple different network traffic conditions.

[0019] According to another aspect of the subject matter described in this article, detecting fault conditions includes providing network function monitoring data to a machine learning classifier trained based on training monitoring data of network functions.

[0020] According to another aspect of the subject matter described herein, providing network function test cases to a network testing system includes associating network function test cases with one or more other network function test cases for network functions from a reference library of test cases for network functions and the core network of the telecommunications network.

[0021] According to another aspect of the subject matter described herein, providing network function test cases to a network testing system includes identifying a reference library of test cases for network function test cases that do not yet exist in the core network of the telecommunications network.

[0022] According to another aspect of the subject matter described herein, a non-transitory computer-readable medium is provided having executable instructions stored thereon, which, when executed by a computer's processor, control the computer to perform steps. These steps include detecting fault conditions in the network functions of a core network of a telecommunications network, and autonomously generating network function test cases based on the detected fault conditions. The network function test cases include one or more network state parameters detected when the fault conditions are detected. These steps include providing the network function test cases to a network testing system configured to execute the network function test cases by repeating one or more network state parameters and determining whether the network function repeats the fault conditions.

[0023] The subjects described herein can be implemented using software in conjunction with hardware and / or firmware. For example, the subjects described herein can be implemented as software executed by a processor. In one example implementation, the subjects described herein can be implemented using a computer-readable medium having computer-executable instructions stored thereon, which control computer execution steps when executed by a computer's processor.

[0024] Example computer-readable media suitable for implementing the subject matter described herein include non-transitory devices, such as disk storage devices, chip memory devices, programmable logic devices, and application-specific integrated circuits (ASICs). Furthermore, computer-readable media implementing the subject matter described herein may reside on a single device or computing platform, or may be distributed across multiple devices or computing platforms. Attached Figure Description

[0025] The subject matter described herein will now be explained with reference to the accompanying drawings, in which:

[0026] Figure 1 This is a block diagram illustrating the network architecture of a 5G system.

[0027] Figure 2This is a block diagram of a sample network environment for the autonomous test case generator;

[0028] Figure 3 This is a block diagram illustrating an example structure of the autonomous test case generator; and

[0029] Figure 4 This is a flowchart of an example method for autonomously generating network function test cases. Detailed Implementation

[0030] The topics described herein relate to methods, systems, and computer-readable media for preventing subscriber identifiers from being leaked from telecommunications networks.

[0031] In 5G telecommunications networks, network nodes that provide services are called producer network functions (NFs). Network nodes that consume services are called consumer NFs. A network function can be both a producer NF and a consumer NF, depending on whether it is consuming or providing services. An NF instance is an instance of a producer NF that provides services. A given producer NF can include multiple NF instances.

[0032] In multi-vendor environments, various 3GPP-defined 5G network functions are deployed. During production deployments, network functions may experience failures / errors under different traffic scenarios. Telecom vendors are adopting continuous integration / continuous delivery (CI / CD) processes. This introduces automated testing processes before and after software delivery.

[0033] Conventional systems lack mechanisms for automatically generating test scenarios upon detecting faults. This specification describes methods and systems for detecting fault conditions by analyzing application logs, alert data, metric data, and trace data, and automatically preparing test cases. Test cases can serve as a test case for the system (e.g., such as...). The input to the Communications 5G Automated Test Suite (a behavioral data-driven testing framework) is provided. The methods and systems described in this specification can be used to prepare reports with relevant details of the core network for fault scenarios.

[0034] Figure 1 This is a block diagram illustrating the network architecture of a 5G system. Figure 1 The architecture includes NRF 100 and SCP101, which can reside in the same Home Public Land Mobile Network (HPLMN). NRF 100 can maintain profiles of available producer NF service instances and their supported services, and allow consumer NFs or SCPs to subscribe and be notified of new / updated producer NF service instance registrations.

[0035] SCP 101 also supports service discovery and producer NF instance selection. SCP 101 can perform load balancing for connections between consumer and producer NFs. Additionally, using the methods described herein, SCP 101 can perform selection and routing based on preferred NF locations.

[0036] NRF 100 is a repository of service profiles for NF or producer NF instances. To communicate with a producer NF instance, a consumer NF or SCP must obtain the NF or service profile or the producer NF instance from NRF 100. The NF or service profile is a JavaScript Object Notation (JSON) data structure defined in 3GPP Technical Specification (TS) 29.510.

[0037] exist Figure 1 In this context, any node (except NRF 100) can be either a consumer NF or a producer NF, depending on whether it is requesting or providing services. In the example shown, the node includes a Policy Control Function (PCF) 102 that performs policy-related operations in the network, a User Data Management (UDM) function 104 that manages user data, and an Application Function (AF) 106 that provides application services.

[0038] Figure 1 The node shown also includes a Session Management Function (SMF) 108 that manages the session between Access and Mobility Management Function (AMF) 110 and PCF 102. AMF 110 performs mobility management operations similar to those performed by a Mobility Management Entity (MME) in a 4G network. Authentication Server Function (AUSF) 112 performs authentication services for User Equipment (UE) seeking network access, such as User Equipment (UE) 114.

[0039] The Network Slice Selection Function (NSSF) 116 provides network slicing services for devices seeking access to specific network capabilities and characteristics associated with a network slice. The Network Exposure Function (NEF) 118 provides an application programming interface (API) for application functions seeking information about Internet of Things (IoT) devices and other UEs attached to the network. NEF 118 performs functions similar to the Service Capability Exposure Function (SCEF) in 4G networks.

[0040] Radio Access Network (RAN) 120 connects User Equipment (UE) 114 to the network via a radio link. This can be achieved using a g-Node B (gNB). Figure 1(Not shown in the image) or other wireless access points to access the radio access network 120. The user plane function (UPF) 122 can support various proxy functions for user plane services. An example of such proxy function is the Multipath Transmission Control Protocol (MPTCP) proxy function.

[0041] UPF 122 also supports performance measurement functions, which UE 114 can use to obtain network performance measurements. Figure 1 The diagram also illustrates data network (DN) 124, through which the UE accesses data network services, such as Internet services.

[0042] SEPP 126 filters incoming traffic from another PLMN and performs topology hiding for traffic leaving the home PLMN. SEPP 126 can communicate with the SEPP in the external PLMN that manages the security of the external PLMN. Therefore, traffic between NFs in different PLMNs can traverse two SEPP functions, one for the home PLMN and the other for the external PLMN.

[0043] Figure 1 An autonomous test case generator 150 is shown. The autonomous test case generator 150 is part of or communicates with the core network of a telecommunications network. The autonomous test case generator 150 is implemented by at least one processor and is configured to detect fault conditions in the network functions of the core network of the telecommunications network, and in response to detecting fault conditions, autonomously generate network function test cases based on the fault conditions.

[0044] Network function test cases include one or more network state parameters detected when detecting fault conditions. An autonomous test case generator 150 is configured to provide network function test cases to a network testing system configured to execute the test cases by repeating one or more network state parameters and determining whether the network function repeats the fault condition.

[0045] Compared to some conventional systems, the core network of a telecommunications network can achieve one or more of the following advantages using the autonomous test case generator 150:

[0046] Currently, 5G deployments can include various 5G network functions deployed using various deployment models that serve different PLMNs and network slices.

[0047] Network operators and software vendors face the challenge of maintaining network functionality with zero or minimal downtime.

[0048] • Automated testing systems are likely a key priority for network operators using and adopting certain network functions. The Autonomous Test Case Generator 150 enables network operators and software vendors to automatically and continuously enhance the test case suites within automated testing systems.

[0049] • When integrated into the entire core network system, the autonomous test case generator 150 can be useful.

[0050] The Autonomous Test Case Generator 150 improves Key Performance Indicators (KPIs) and Service Level Agreements (SLAs) by detecting network functionality issues and reporting them to teams such as product and operations.

[0051] • The Autonomous Test Case Generator 150 can be used to automatically create error reports with details related to specific faults.

[0052] The Autonomous Test Case Generator 150 can be used to reduce the overhead of manually handling the detection and creation of 5G fault conditions.

[0053] • The Autonomous Test Case Generator 150 can be used to create robust core network functions.

[0054] • The autonomous test case generator 150 can be used to periodically verify the integrity of network functions using automatically generated test cases at specified times (e.g., off-peak hours) to declare network functions operating according to specifications.

[0055] Figure 2 This is a block diagram of an example network environment 200 used by the autonomous test case generator 150. The autonomous test case generator 150 is configured to detect fault conditions of a first network function 202 by receiving monitoring data. Network function 202 may be a consumer of data, for example, from a second network function 204. The autonomous test case generator 150 can perform monitoring of both network function 202 and network function 204.

[0056] Monitoring data for network function 202 may include, for example, alarms and metrics 206 or logs and traces 208, or both. In some examples, detecting a fault condition includes analyzing one or more network performance metrics or one or more network performance alarms, or both. In some examples, detecting a fault condition includes parsing one or more network function application logs or one or more network function traces, or both.

[0057] Generally, the autonomous test case generator 150 can use any appropriate technique to detect fault conditions. For example, detecting fault conditions may include providing network function monitoring data to a machine learning classifier trained based on training monitoring data of network functions.

[0058] In response to the detection of a fault condition, the autonomous test case generator 150 is configured to autonomously generate network function test cases 210 for network function 202 based on the fault condition. Test case 210 specifies one or more network state parameters detected when the fault condition is detected.

[0059] Test case 210 is provided to an automated network testing system 212 configured to execute test case 210. Manual testing is typically resource-intensive to run and maintain, can be very time-consuming, lacks adequate coverage, and is prone to errors due to its repetitive nature. This has led to the introduction and appeal of automating these tests. Automated testing is used to improve the speed of execution of verification, inspection, or any other repeatable tasks throughout the software development, integration, and deployment lifecycle.

[0060] Automated network testing system 212 can be a behavioral data-driven testing framework, such as Communications 5G Automated Test Suite. The Automated Test Suite (ATS) allows network operators to execute software test cases using automated testing tools and then compare the actual results with expected or predicted results. No user intervention is required in this process. ATS can be used as a software implementation on the system under test to check whether the system functions as expected and provides end-to-end and regression testing for 4G and 5G scenarios, including interoperability test cases and network function (NF) simulations.

[0061] As network traffic evolves, test cases and reports can be updated. Using a testing system that regularly reviews and adjusts test cases or develops new ones is useful for maintaining network operations. In today's virtualized and cloud-native environments, 4G / 5G applications are no longer deployed on proprietary hardware; the underlying environment may change and is often beyond the control of network operators. Powerful regression testing that is tailored to network operator needs and provides meaningful reports and data is useful. The ability to quickly deploy these new test cases is particularly useful for adding interoperability and policy rule test cases, which can be done quickly, run daily, and cover the subscriber / subscription lifecycle.

[0062] For example, an automated network testing system 212 can execute test case 210 by repeating network state parameters and determining whether network function 202 repeats fault conditions. In some examples, executing test case 210 includes repeatedly executing test case 210 for different network traffic conditions.

[0063] In some examples, the automated network testing system 212 is configured to associate test case 210 with one or more other network function test cases for network function 202 from a reference library of test cases for the core network of the telecommunications network. The automated network testing system 212 and / or the autonomous test case generator 150 may be configured to determine, before adding test case 210, that test case 210 does not already exist in the reference library of test cases for the core network of the telecommunications network.

[0064] The automated network testing system 212 can use any suitable deployment model, such as:

[0065] • In-Cluster Deployment

[0066] • Out-of-Cluster Deployment

[0067] Based on the in-cluster deployment model, the automated network testing system 212 can coexist within the same cluster where NF is deployed. This deployment model is useful for in-cluster testing.

[0068] According to the off-cluster deployment model, network operators can deploy the automated testing system 212 in a separate cluster, different from the cluster in which NF is deployed.

[0069] This deployment model is useful for performing "outside-the-cluster" tests because it:

[0070] More aligned with production use cases

[0071] • It is rare for all NFs to coexist in the same cluster.

[0072] Test case 210 can also be provided to the computer systems of the network operations team 214, the network function product team 216, and the error database 218 that stores software problems related to network functions.

[0073] Figure 3 This is a block diagram illustrating an example structure of the autonomous test case generator 150.

[0074] The autonomous test case generator 150 is configured to receive monitoring data 302. The monitoring data 302 may include, for example, network function logs, traces, metrics, and alerts from different network functions.

[0075] The autonomous test case generator 150 is implemented on at least one processor and memory 304. The autonomous test case generator 150 may include an error detector and a data parser 306 implemented on at least one processor and memory 304. The autonomous test case generator 150 may include a test case creator and an event renderer 308 implemented on at least one processor and memory 304.

[0076] During operation, the error detector and data parser 306 may perform one or more of the following operations:

[0077] The error detector and data parser 306 can continuously receive and analyze monitoring data 302, such as metrics, alarms, traces, and application logs of one or more network functions.

[0078] The error detector and data parser 306 ensure that error logs, traces, metrics, and alert information are collected in the event of any failure.

[0079] • In some examples, the error detector and data parser 306 can use machine learning algorithms to apply analytics-driven rules. For example, Classification and Regression Tree (CART) can be applied to the collected monitoring data to generate one or more error trigger points, such as input service operations, the data involved, Uniform Resource Identifiers (URIs), and the cause of the failure.

[0080] The error detector and data parser 306 may include a machine learning classifier trained on training data specific to the type of network function being monitored. The network function developer can provide appropriate training data that specifies, for example, the expected operation of the network function under various network operating states.

[0081] During operation, the test case creator and event renderer 308 can perform one or more of the following operations:

[0082] • The test case creator and event renderer 308 can ensure that detected faults are checked against a reference test library 310 that stores existing test cases. If a newly detected test case is not part of an existing test suite, the test case creator and event renderer 308 can generate the test case, for example, by using one or more network parameters stored when a fault condition is detected.

[0083] • The test case creator and event presenter 308 can arrange test cases based on 5G NF type, 5G service operation, error category, or using any appropriate type of category.

[0084] The test case creator and event presenter 308 can create test cases based on detected fault conditions, and if the test cases do not exist in the reference test library 310, they are sent to the automated network testing system 212. The automated network testing system 212 can then store the test cases in the test case library 312.

[0085] • The test case creator and event presenter 308 can transmit details of detected fault conditions to the computer systems of the network operations and product development teams 314.

[0086] To illustrate the operation of the autonomous test case generator 150, consider the following example of a 5G autonomous test case generator that can handle detected fault conditions of type Hypertext Transfer Protocol (HTTP) code 500.

[0087] Assume that the error detector and data parser 306 detects alarms triggered by faults by analyzing application logs and trace data received in monitoring data 302. The error detector and data parser 306 analyze the detected failure conditions and determine that the metric data includes an HTTP code 500 transmission failure response.

[0088] Error detector and data parser 306 can perform the following operations:

[0089] • Rules for finding HTTP status code 500

[0090] • Use the action items defined for the rule

[0091] • Apply machine learning algorithms, such as CART

[0092] • Collect 5G input data based on the classification and decision results of applied machine learning algorithms, for example:

[0093] Enter 5G service operation

[0094] HTTP message URL

[0095] HTTP input body

[0096] Error message

[0097] Error metrics and alert details

[0098] The test case creator and event renderer 308 can generate 5G test cases, for example, by storing data such as:

[0099] Scenario: 5G NF PLMN-ID update

[0100] Expected result: 200-OK-Success

[0101] URI:http: / / <http-api-root> / NFServiceOperation / 5GNFInstanceId

[0102] Use Case:

[0103] Given an initialization test suite

[0104] • Initialize an NF connection using NFID1ocnf-microserviceName.NFName NFPort

[0105] Then use NF NFID1

[0106] Then check and set the NF-Namespace.

[0107] Then send the custom header NFID1Content-Type=application / Jason-patch+json.

[0108] Then execute a partial NFProfile update in PlmnIdUpdateInput.json NFID1

[0109] Then verify the HTTP response code 200.

[0110] Then set the user variable nfInstaceId from the HTTP response.

[0111] Then obtain the NFInstance with NFID1.

[0112] The test case creator and event renderer 308 can then transfer the test cases to the automated network testing system 212.

[0113] The test case creator and event presenter 308 can then transmit a message containing event details to the computer systems of the network operations and product development teams 314. For example, the message might specify:

[0114] Event Time: <>

[0115] Event details:

[0116] • Error metric: TxErrorDetected

[500]

[0117] • Error alert: AlertCriticalRate

[500]

[0118] URI:

[0119] http: / / <http-api-root> / NFServiceOperation / 5GNFInstanceId

[0120] JSON body: content

[0121] Figure 4 This is a flowchart of Example Method 400 for autonomously generating network function test cases.

[0122] Method 400 includes detecting fault conditions (402) in network functions of the core network of the telecommunications network. In some examples, detecting fault conditions includes analyzing one or more network performance metrics or one or more network performance alerts, or both. In some examples, detecting fault conditions includes parsing one or more network function application logs or one or more network function traces, or both.

[0123] Generally, method 400 may include using any appropriate technique to detect fault conditions. For example, detecting fault conditions may include providing network function monitoring data to a machine learning classifier trained based on training monitoring data of network functions.

[0124] Method 400 includes autonomously generating network function test cases (404) based on the detected fault condition. The network function test cases include one or more network state parameters detected when the fault condition is detected. Network state parameters can be any suitable type of data characterizing the operation of the network or network function, or both, at the time the fault condition is detected or prior to its detection. For example, network state parameters can specify network load, the type of messages sent, network function log data, etc.

[0125] Generally, generating network functional test cases involves storing network state parameters, allowing the network test system to repeat conditions that lead to failures. In some cases, network functional test cases may include operational parameters such as network load. In others, network functional test cases may specify certain actions that occur before a failure condition is detected.

[0126] For example, network function test cases can specify a sequence of messages to be sent to the network function before a fault condition is detected. In these examples, the automated testing system can repeat the message sequence after a network function update to determine if the fault condition is repeated.

[0127] Method 400 includes providing network function test cases to a network testing system configured to execute the network function test cases by repeating one or more network state parameters and determining whether the network function repeats a failure condition (406). In some examples, executing network function test cases includes repeatedly executing the network function test cases against multiple different network traffic conditions.

[0128] Providing network function test cases to a network testing system may include associating network function test cases with one or more other network function test cases from a reference library of test cases for the network function and the core network of the telecommunications network. Providing network function test cases to the network testing system includes determining that the network function test cases do not yet exist in the reference library of test cases for the core network of the telecommunications network.

[0129] The scope of this disclosure includes any feature or combination of features disclosed in this specification (express or implicit), or any generalization of the disclosed features, whether or not such features or generalizations alleviate any or all of the problems described in this specification. Therefore, new claims may be made against any such combination of features during the examination of this application (or an application claiming priority to this application).

[0130] In particular, with reference to the appended claims, the features of the dependent claims may be combined with the features of the independent claims, and the features of the individual independent claims may be combined in any suitable manner, not just in the specific combinations listed in the appended claims.

Claims

1. A method for autonomously generating network functional test cases, the method comprising: Detecting faults in the network functions of the core network of a telecommunications network, wherein the network functions are 5G network functions; In response to the detection of a fault condition, network function test cases are generated autonomously based on the fault condition. The network function test cases include one or more network state parameters detected when the fault condition is detected, and the network function test cases specify a sequence of network messages to be sent to the network function before the fault condition is detected. as well as Network function test cases are provided to a network testing system, which is configured to execute network function test cases to repeatedly send network message sequences to the network function and determine whether the network function repeats fault conditions.

2. The method of claim 1, wherein detecting fault conditions includes analyzing one or more network performance metrics or one or more network performance alerts or both.

3. The method of claim 1 or 2, wherein detecting fault conditions includes parsing one or more network function application logs or one or more network function traces or both.

4. The method as described in any of the preceding claims, wherein executing network function test cases includes repeatedly executing network function test cases for multiple different network traffic conditions.

5. The method of any of the preceding claims, wherein detecting a fault condition comprises providing network function monitoring data to a machine learning classifier trained based on the network function training monitoring data.

6. The method of any of the preceding claims, wherein providing network function test cases to the network testing system includes associating the network function test cases with one or more other network function test cases for network functions in a reference library of test cases for network functions and the core network of the telecommunications network.

7. The method of any of the preceding claims, wherein providing network function test cases to the network testing system includes determining that the network function test cases do not yet exist in a reference library of test cases for the core network of the telecommunications network.

8. A system for autonomously generating network functional test cases, the system comprising: Network testing system; At least one processor and memory; as well as An autonomous test case generator, implemented by the at least one processor and configured as follows: Detecting faults in the network functions of the core network of a telecommunications network, wherein the network functions are 5G network functions; In response to the detection of a fault condition, network function test cases are generated autonomously based on the fault condition. The network function test cases include one or more network state parameters detected when the fault condition is detected, and the network function test cases specify a sequence of network messages to be sent to the network function before the fault condition is detected. as well as Network function test cases are provided to the network testing system, which is configured to execute the network function test cases to repeatedly send network message sequences to the network function and determine whether the network function repeats the fault condition.

9. The system of claim 8, wherein detecting fault conditions includes analyzing one or more network performance metrics or one or more network performance alerts or both.

10. The system of claim 8 or 9, wherein detecting fault conditions includes parsing one or more network function application logs or one or more network function traces or both.

11. The system of any one of claims 8 to 10, wherein executing network function test cases includes repeatedly executing network function test cases for multiple different network traffic conditions.

12. The system of any one of claims 8 to 11, wherein detecting a fault condition includes providing network function monitoring data to a machine learning classifier trained based on the network function training monitoring data.

13. The system of any one of claims 8 to 12, wherein providing network function test cases to the network testing system includes associating the network function test cases with one or more other network function test cases for network functions in a reference library of test cases for network functions and the core network of the telecommunications network.

14. The system of any one of claims 8 to 13, wherein providing network function test cases to the network testing system includes determining that the network function test cases do not yet exist in a reference library of test cases for the core network of the telecommunications network.

15. A non-transitory computer-readable medium having executable instructions stored thereon, the executable instructions, when executed by a computer's processor, controlling the computer to perform steps including: Detecting faults in the network functions of the core network of a telecommunications network, wherein the network functions are 5G network functions; In response to the detection of a fault condition, network function test cases are autonomously generated based on the fault condition. These test cases include one or more network state parameters detected when the fault condition is detected, and specify a sequence of network messages to be sent to the network function before the fault condition is detected. Network function test cases are provided to a network testing system, which is configured to execute the network function test cases to repeatedly send network message sequences to the network function and determine whether the network function repeats fault conditions.

16. The non-transitory computer-readable medium of claim 15, wherein detecting fault conditions includes analyzing one or more network performance metrics or one or more network performance alerts or both.

17. The non-transitory computer-readable medium of claim 15 or 16, wherein detecting a fault condition includes parsing one or more network function application logs or one or more network function traces or both.

18. The non-transitory computer-readable medium of any one of claims 15 to 17, wherein performing network function test cases includes repeatedly performing network function test cases for multiple different network traffic conditions.

19. The non-transitory computer-readable medium of any one of claims 15 to 18, wherein detecting a fault condition includes providing network function monitoring data to a machine learning classifier trained based on the network function training monitoring data.

Images