Inference context history in communication network

Abstract

Some embodiments relate to network data analysis in a communication network. In some embodiments, an inference output generated by the AI model is accessed. The inference output is generated in a network context of the communication network. Data indicative of a network context is collected. The inference output is stored as an inference context history along with data indicative of the network context. The inference context history is used for network data analysis of the communication network.

Description

Technical Field

[0001] The present disclosure relates generally to the field of communications. In particular, some embodiments relate to network data analysis in a communication network. Background Art

[0002] Artificial intelligence and machine learning technologies are increasingly being used in mobile communications technology, and in particular for fifth generation (5G) and subsequent network technologies. For example, the network data analysis function (NWDAF) in the 5G core (5GC) generates analysis by processing management and network data. The NWDAF may use artificial intelligence (AI) and / or machine learning (ML) technologies. Summary of the invention

[0003] This summary is provided to introduce a series of concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] According to some aspects, the subject matter of the independent claims is provided. Some further aspects are defined in the dependent claims.

[0005] According to a first aspect, a device includes: at least one processor; and at least one memory including computer program code, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, enable the device to perform operations, the operations comprising: for one or more AI models: accessing an inference output generated by the AI ​​model, the inference output being generated in a network context of a communication network; collecting data indicating the network context; and storing the inference output together with the data indicating the network context as an inference context history; and using the inference context history for network data analysis of the communication network.

[0006] According to an example embodiment of the first aspect, using reasoning context history for network data analysis includes: selecting an AI model for performing network data analysis from one or more AI models, the AI ​​model being selected based on the reasoning context history; and running the selected AI model to generate network data analysis.

[0007] According to an example embodiment of the first aspect, storing the inference output together with the data indicating the network context comprises: associating the network context with a label; and storing the inference output together with the label.

[0008] According to an example embodiment of the first aspect, storing the inference output together with data indicating the network context includes: associating the data indicating the network context with a label; and wherein selecting an AI model for performing network data analysis includes: selecting the AI ​​model based on the inference context history associated with the label.

[0009] According to an example embodiment of the first aspect, associating a network context with a tag includes: performing a similarity check between data representing the network context and data representing a historical network context stored in a context tag table; assigning a tag to the network context based on a result of the similarity check; and updating the context tag table by storing the data indicating the network context together with the tag in the context tag table.

[0010] According to an example embodiment of the first aspect, using reasoning context history for network data analysis includes: monitoring the performance of one or more AI models among the AI ​​models to be used for network data analysis, the performance of the AI ​​models being monitored by analyzing reasoning context history associated with the AI ​​models.

[0011] According to an example embodiment of the first aspect, the inference output is a prediction for a parameter, and wherein using the inference context history for network data analysis includes: comparing the prediction for the parameter with a measured value of the parameter; and triggering retraining of the AI ​​model based on determining that the prediction for the parameter does not match the measured value of the parameter.

[0012] According to an example embodiment of the first aspect, the data indicating the network context includes one or more of the following items: slice load level, network function load, network performance, or user equipment related analysis; general network status; or information about the type of sub-area of ​​the area of ​​interest.

[0013] According to a second aspect, a method includes: for one or more AI models: accessing an inference output generated by the AI ​​model, the inference output being generated in a network context of a communication network; collecting data indicating the network context; and storing the inference output together with the data indicating the network context as an inference context history; and using the inference context history for network data analysis of the communication network.

[0014] According to a third aspect, a device includes: a component for accessing an inference output generated by an AI model, the inference output being generated in a network context of a communication network; a component for collecting data indicating the network context; a component for storing the inference output together with the data indicating the network context as an inference context history; and a component for using the inference context history for network data analysis of the communication network.

[0015] According to a fourth aspect, a computer program includes instructions for causing a device to perform at least the following items: for one or more AI models: accessing an inference output generated by the AI ​​model, the inference output being generated in a network context of a communication network; collecting data indicating the network context; storing the inference output together with the data indicating the network context as an inference context history; and using the inference context history for network data analysis of the communication network. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are included to provide a further understanding of example embodiments and constitute a part of this specification, illustrate example embodiments and together with the description help explain the principles of example embodiments. In the drawings:

[0017] Figure 1 An example of a communication network is illustrated.

[0018] Figure 2 is a signaling diagram illustrating an example method for configuring tracking of inference context history.

[0019] Figure 3 is a signaling diagram illustrating an example method for building and storing network context information.

[0020] Figure 4 is a signaling diagram illustrating an example method for inference context history tracking and storage.

[0021] Figure 5A , Figure 5B ,as well as Figure 5C is an illustration of an example method for using inference context history for AI / ML model selection.

[0022] Figure 6 An example embodiment of the subject matter described herein is illustrated, which illustrates a flowchart representation of a method.

[0023] In the drawings, the same reference numerals are used to denote the same components. DETAILED DESCRIPTION

[0024] Reference will now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings. The detailed description provided below in conjunction with the drawings is intended as a description of this example and is not intended to represent the only form in which the present disclosure may be constructed or used. The description sets forth the functions of the example and the sequence of steps for constructing and operating the example. However, the same or equivalent functions and sequences may be implemented by different example embodiments.

[0025] Applications and / or functions using AI models, particularly ML models, may be applied in different network contexts, which may produce different outputs when reasoning. In order to manage the reasoning performance of AI models, it may be necessary to track and store the reasoning outputs of AI models and associate these reasoning outputs with the network contexts in which they were generated.

[0026] Some embodiments provide a mechanism for associating the inference history of one or more AI models with the network context in which those inferences were made. Some embodiments provide a mechanism for tracking the history of decisions made by different analytics or their AI models. Some embodiments provide a mechanism for coordinating those decisions with the network context in which they were made. Some embodiments allow analytics consumers to characterize the quality of the analytics produced based on the historical decisions made by the AI ​​models.

[0027] Some embodiments use inference history for analyzing producer selection (e.g., based on actual mobile network contexts or use cases), performing AI model performance monitoring and management (e.g., retraining of AI models that do not work properly in a specific use case / network context), and / or performing understanding of AI model behavior.

[0028] Some embodiments allow network operators and / or management to collect reasoning history and associate it with network context. Some embodiments enhance the NWDAF functionality so that it can automatically build context information on which the AI ​​model runs, and store the context information together with the reasoning output (e.g., decision, prediction, recommendation, analysis, statistics) generated by the AI ​​model.

[0029] Figure 1 An example system in which various embodiments of the present disclosure may be implemented is illustrated. The system may include a communication network 1000, such as a fifth generation (5G) new radio (NR) network. In at least some embodiments, the communication network 1000 may include: one or more massive machine-to-machine (M2M) networks, (multiple) massive machine type communications (mMTC) networks, (multiple) Internet of Things (IoT) networks, (multiple) Industrial Internet of Things (IIoT) networks, (multiple) enhanced mobile broadband (eMBB) networks, (multiple) ultra-reliable low latency communications (URLLC) networks, etc. In other words, the communication network 1000 may be configured to serve different service types and / or use cases, and may be logically viewed as including one or more networks.

[0030] The communication network 1000 may be configured, for example, according to a 5G digital cellular communication network defined by the Third Generation Partnership Project (3GPP). In one example, the communication network 1000 may operate according to 3GPP 5G NR (New Radio). However, it should be understood that the example embodiments presented herein are not limited to the example network and may be applied to any current or future wireless communication network or combination thereof, such as other types of cellular networks, short-range wireless networks, broadcast or multicast networks, etc.

[0031] The communication network 1000 may include a network data analysis function (NWDAF) 100. The NWDAF 100 may collect network data, perform network analysis, and provide network analysis to an NWDAF service consumer 120. A single instance or multiple instances of the NWDAF 100 may be deployed in a public land mobile network (PLMN). If multiple NWDAF instances are deployed, the architecture supports deploying the NWDAF as a central network function (NF), a collection of distributed NFs, or a combination of the two. The NWDAF 100 may interact with different entities for different purposes.

[0032] The NWDAF service consumer 120 may be a network function (NF) 121 (e.g., a policy control function (PCF), a network slice selection function (NSSF), a mobility function (AMF)), an application function (AF) 122, a process or function for provisioning and managing a network or an element within a network (e.g., operations, administration, and maintenance (OAM) 123). The NWDAF service consumer 120 may configure the NWDAF 100 to track reasoning decisions and associate the reasoning decisions with the network context in which the reasoning decisions were made.

[0033] The NWDAF 100 may interact with an analytical data repository function (ADRF) 110 for information storage and retrieval. The ADRF 110 may store and retrieve collected data and analysis. The ADRF 110 may allow other NFs 121 to store and retrieve data.

[0034] The NWDAF 100 may include a model training logic function (MTLF) 103 for training one or more AI models. The one or more AI models may include one or more ML models.

[0035] The NWDAF 100 may include an analysis logic function (AnLF) 101. The AnLF 101 is a logic function that provides analysis services to one or more NWDAF service consumers 120. The AnLF 101 may perform reasoning and derive reasoning outputs such as analysis, statistics, decisions, and / or predictions. The AnLF 101 may use an AI model trained by the MTLF 103.

[0036] The NWDAF 100 may include a NWDAF decision logging logic function (DLLF) 102. The DLLF 102 may be configured by a mobile network operator and / or a management plane.

[0037] DLLF 102 may collect, store, and retrieve the reasoning history of one or more AI models, and in particular the reasoning history of one or more AI models used by AnLF 101 to provide analysis services. The reasoning history of an AI model may include data related to one or more reasonings performed by the AI ​​model, and in particular data related to one or more reasoning outputs generated by the AI ​​model. The reasoning output may be a decision, prediction, recommendation, statistics, and / or analysis.

[0038] DLLF 102 may store the AI ​​model reasoning history in ADRF 110. DLLF 102 may periodically collect the AI ​​model reasoning history within a specified time period, or throughout the life cycle of the AI ​​model, or throughout the life cycle of an application or function using the AI ​​model.

[0039] DLLF 102 may collect, store, and retrieve data related to inferences performed by an AI model that are related to the network context in which the AI ​​model performs the inferences. The network context of the inferences is the operational context of the communication network in which the inferences are performed.

[0040] DLLF 102 can collect, mark, store, and retrieve network context information. Network context information includes data indicating the network context of the communication network. Network context information may include actual network conditions, such as slice load levels, NF loads, network performance, or UE-related analysis. Network context information may include analysis of actual network conditions captured by NWDAF AnLF101. Network context information may include network status information, such as faults, management data from OAM (such as key performance indicators (KPIs), drive test minimization (MDT), TraceJob information, base station and / or cell power outages), logs, and / or historical analysis. By running regional monitoring analysis, the context information may also include statistical information about the data distribution of the area of ​​interest (AOI).

[0041] DLLF 102 may associate observed network contexts with one or more network context tags. This association may be used to associate reasoning history with network context information. To perform this association, DLLF 102 may perform a similarity check so that similar observed network contexts receive the same network context tag.

[0042] DLLF 102 or ADRF 110 may maintain and report a mapping between observed network contexts and corresponding network context tags. The mapping may be stored as one or more context tag tables 111. Context tags may be identified by context IDs. Context tag tables 111 may associate network context information with one or more context IDs identifying one or more context tags.

[0043] DLLF 102 can generate, store, and retrieve reasoning context history. Reasoning context history can be a match between AI model reasoning history and context information. DLLF 102 can track different reasoning outputs generated by a given AI / ML in different network contexts. DLLF 102 can generate and store matches between reasoning outputs and network contexts. This match enables the decision made by the AI ​​model to be associated with a specific network context. For example, this information can be used to select the best analysis producer, or manage AI models based on their performance in actual mobile network contexts and / or use cases.

[0044] The NWDAF 100 may generate analysis in a manner that utilizes the inference context history. The NWDAF 100 may use the inference context history to select an AI model based on the actual network context. The inference context history may be used to identify changes in the actual network context and select the AI ​​model that performs best in the actual network context.

[0045] AnLF 101 may send network context information representing the actual network context to DLLF 102 and, in response, request a context ID that matches the network context information. AnLF 101 may then use the context ID to select one or more AI models for performing analysis services in the actual network context.

[0046] AnLF 101 may send network context information representing the actual network context to DLLF 102, and in response, request a combination of context ID and reasoning history that matches the network context information. AnLF 101 may then use the combination of context ID and reasoning history to select one or more AI models for a specific network context.

[0047] AnLF 101 may send network context information representing the actual network context to DLLF 102 and, in response, request selection of one or more AI models for a particular network context.

[0048] MTLF 103 may use the reasoning context history to detect outdated AI models. For example, MTLF 103 may detect whether an AI model is outdated by comparing the predictions made by the AI ​​model with actual measurements. If MTLF 103 determines that the AI ​​model is outdated, MTLF 103 may trigger retraining of the AI ​​model (e.g., global retraining or retraining for a specific network context).

[0049] NWDAF 100 may use reasoning context history to monitor AI model performance. NWDAF 100 may use reasoning context history to evaluate the performance of AI models in different network contexts. Specifically, NWDAF 100 may use reasoning context history to measure the generalization ability of AI models. NWDAF 100 may use reasoning context history to understand AI model behavior. Specifically, NWDAF 100 may use reasoning context history to explain reasoning outputs (e.g., analysis, statistics, predictions, recommendations, and / or decisions) generated by AI models.

[0050] The data format of the inference context history may include one or more of the following parameters:

[0051]

[0052]

[0053] Figure 2 is a signaling diagram illustrating an example method for configuring tracking of inference context history. Specifically, embodiments may be used to configure tracking of inference context history at DLLF 102. Figure 2 One example is presented of operations at the NWDAF 100. Some of the operations described may not be present in all example embodiments, and example embodiments may also include additional features / operations described elsewhere in this specification.

[0054] At operation 201, the NWDAF service consumer 120 (e.g., NF 121, AF 121, or OAM 123) sends a request to AnLF 101 to track the inference context history of the analytical service. The request may be sent when the service consumer 120 subscribes to or requests the analytical service provided by the NWDAF 100. For example, this may be performed by setting an attribute (e.g., inference_history_track attribute) to "true" in a subscription request (e.g., Nnwdaf_AnalyticsSubscription_Subscribe). AnLF 101 may send a confirmation to the NWDAF service consumer 120.

[0055] At operation 202, AnLF 101 instructs DLLF 102 to track the inference context history of the requested analysis service. For example, this may be performed by setting a property (eg, inference_history_track property) to "true" in a subscription request.

[0056] At operation 203, the DLLF 102 accesses the context tag table 111. The context tag table 111 stores the association between the network context information and the context ID. If the context tag table 111 is stored at the ADRF 110, the DLLF 102 retrieves the context tag table 111 from the ADRF 110. This can be done by sending a data management storage request (e.g., Nnadrf_DataManagement_Storage_Request) to the ADRF 110.

[0057] At operation 204, ADRF 110 forwards context tag table 111 to DLLF 102. DLLF 102 may send context tag table 111 in a data management storage response (eg, Nnadrf_DataManagement_Storage_Response). DLLF 102 may use the context tag table to collect, tag, store, and / or retrieve network context information.

[0058] Figure 3 is a signaling diagram illustrating an example method for constructing and storing network context information. Specifically, Figure 3 The method may be used to generate the context tag table 111 . Figure 3 One example is presented of operations at the NWDAF 100. Some of the operations described may not be present in all example embodiments, and example embodiments may also include additional features / operations described elsewhere in this specification.

[0059] At operation 301, the NWDAF service consumer 120 (e.g., NF 121, AF 121, or OAM 123) sends a request to the MWDAF AnLF 101 to track the inference context history of the analysis service. The request may be sent when the service consumer 120 subscribes to or requests the analysis service provided by the NWDAF 100. For example, this may be performed by setting an attribute (e.g., inference_history_track attribute) to "true" in a subscription request (e.g., Nnwdaf_AnalyticsSubscription_Subscribe). The requested analysis may be identified by an analysis ID. The AnLF 101 may send a confirmation to the NWDAF service consumer 120.

[0060] At operation 302 , the AnLF 101 accesses and / or retrieves information about one or more AI models that can be used to provide the requested analysis. The requested analysis can be identified by an analysis ID. Information about the AI ​​models can be accessed from the MTLF 103 .

[0061] At operation 303, the AnLF 101 collects network context information. The network context information may be use case specific. For example, depending on the requested analysis, the network context information may include different information. The network context information may be generated by running network data analysis, as, for example, slice load level, NF load, network performance, or UE related analysis. The network context information may also be generated by collecting network status information as faults, relevant management data from OAM as KPIs, MDT, TraceJobs for power outages or base stations and / or cells, logs and / or historical analysis. The context information may also include statistical information about the data distribution of the AOI by running regional monitoring analysis.

[0062] At operation 304, AnLF 101 forwards the network context information to DLLF 102. For example, this may be done using the Ndllf_context_information service.

[0063] At operation 305, the DLLF 102 performs a similarity check between the context information received from the AnLF 101 and the context information collected in the past. If a context tag table exists, the DLLF 102 may perform a similarity check between the context information received from the AnLF 101 and the context information stored in the context tag table. The similarity check may be performed using any suitable similarity metric and / or algorithm. For example, the similarity check may be performed using a nearest neighbor method to identify similarities between two or more objects based on an algorithmic distance function.

[0064] At operation 306, the DLLF 102 generates or updates a context tag table using the context information received at operation 304. This can be done by assigning a context ID to the context information received at operation 304 based on the result of the similarity check. In particular, this can be done by mapping similar network context information to the same network context ID. For example, this can be done using a clustering algorithm.

[0065] In the case where the context tag table is stored at the DLLF 102 , at operation 307 a , the DLLF 102 stores the generated or updated context tag table.

[0066] In the case where the context tag table is stored at the ADRF 110, at operation 307b, the DLLF 102 forwards the context tag table to the ADRF 110 and instructs the ADRF 110 to store the context tag table. At operation 308b, the ADRF 110 stores the context tag table. At operation 309b, the ADRF 110 notifies the DLLF 102 that the context tag table has been successfully stored.

[0067] At operation 310 , the DLLF 102 notifies the AnLF 101 that the generated or updated context tag table has been successfully updated and stored.

[0068] Figure 4 is a signaling diagram illustrating an example method for inference context history tracking and storage. Figure 4 One example of operations at the NWDAF 100 is presented. Some of the described operations may not be present in all example embodiments, and example embodiments may also include additional features / operations described elsewhere in this specification.

[0069] At operation 401, the NWDAF service consumer 120 (e.g., NF 121, AF 121, or OAM 123) sends a request to the MWDAF AnLF 101 to track the inference context history of the analysis service. The request may be sent when the service consumer 120 subscribes to or requests the analysis service provided by the NWDAF 100. For example, this may be performed by setting an attribute (e.g., inference_history_track attribute) to "true" in a subscription request (e.g., Nnwdaf_AnalyticsSubscription_Subscribe). The requested analysis may be identified by an analysis ID. The AnLF 101 may send a confirmation to the NWDAF service consumer 120.

[0070] At operation 402 , AnLF 101 accesses and / or retrieves information about one or more AI models that can be used to provide the requested analysis. The requested analysis can be identified by an analysis ID. Information about the AI ​​models can be accessed from MTLF 103 .

[0071] At operation 403 , the AnLF 101 collects network context information. This may be performed similarly to operation 303 .

[0072] At operation 404, AnLF 101 runs one or more AI models to produce the requested analysis. The AI ​​models perform one or more inferences and generate one or more inference outputs.

[0073] At operation 405, AnLF 101 sends the output of the reasoning and network context information representing the network context in which the reasoning is performed to DLLF 102. The output of the reasoning may include one or more of decisions, predictions, recommendations, analysis and / or statistics generated by the AI ​​model. In addition to the output of the reasoning and the network context information, AnLF 101 may also send the model ID of the AI ​​model used to perform the reasoning, and the analysis ID of the analysis service that has adopted the AI ​​model. Tracking of reasoning can be performed periodically, within a specific time window, or throughout the AI ​​model life cycle. The forwarding of the results of the reasoning can be performed by using the Ndllf_DataManagement_Store service. AnLF 101 may further instruct DLLF 102 to store the results of the reasoning.

[0074] At operation 406, the DLLF 102 performs a similarity check between the context information received from the AnLF 101 and the context information collected in the past. If there is a context tag table, the DLLF 102 may perform a similarity check between the context information received from the AnLF 101 and the context information stored in the context tag table. The similarity check may be performed similarly to operation 305.

[0075] At operation 407, DLLF 102 generates or updates a context tag table using the context information received at operation 405. This can be done by assigning a context ID to the received context information based on the result of the similarity check. In particular, this can be done by mapping similar network context information to the same network context ID. For example, this can be done using an appropriate clustering algorithm. This can be done similarly to operation 306.

[0076] If the inference context history is stored at the DLLF 102, then at operation 408a, the DLLF 102 generates or updates the inference context history using the inference output received at operation 405 and the context ID determined for the network context at operation 407. The inference context history associates the inference output with the network context in which the inference was performed. At operation 409a, the DLLF 102 may notify the AnLF 101 that the inference context history has been successfully generated / updated.

[0077] If the reasoning context history is stored at the ADRF 110, then at operation 408b, the DLLF 102 sends the reasoning output and the network context ID to the ADRF 110. This can be included in the storage request. The DLLF 102 can also send the model ID, the analysis ID. At operation 409b, the ADRF 110 generates or updates the reasoning context history using the reasoning output received at operation 405 and the context ID determined for the network context at operation 407. This can be performed similarly to operation 408a. At operation 410b, the ADRF 110 notifies the DLLF 102 that the reasoning context history has been successfully generated / updated. At operation 411b, the DLLF 102 notifies the AnLF 101 that the reasoning context history has been generated / updated.

[0078] At operation 412, the AnLF 102 provides the requested analytics report to the NWDAF service consumer 120. This may be done using Nnwdaf_AnalyticsSubscription_Notify.

[0079] Figure 5A is a signaling diagram illustrating an example method for using inference context history for AI / ML model selection. Figure 5A One example is presented of operations at the NWDAF 100. Some of the operations described may not be present in all example embodiments, and example embodiments may also include additional features / operations described elsewhere in this specification.

[0080] At operation 501, the NWDAF service consumer 120 (e.g., NF 121, AF 121, or OAM 123) sends a request to the MWDAF AnLF 101 to track the inference context history of the analysis service. The request may be sent when the service consumer 120 subscribes to or requests the analysis service provided by the NWDAF 100. For example, this may be performed by setting an attribute (e.g., inference_history_track attribute) to "true" in a subscription request (e.g., Nnwdaf_AnalyticsSubscription_Subscribe). The requested analysis may be identified by an analysis ID. The AnLF 101 may send a confirmation to the NWDAF service consumer 120.

[0081] At operation 502, the AnLF 101 collects network context information. This may be performed similarly to operation 303. The network context corresponding to the network context information collected at operation 502 is referred to as an actual network context.

[0082] At operation 503a, the AnLF 101 sends a request to the DLLF 102 to request the DLLF 102 to provide a context ID for the network context information collected at operation 502. The request includes the network context information collected at operation 502. The request may be sent using the Ndllf_ContextMatching_Request service.

[0083] At operation 504a, DLLF 102 performs a similarity check between the network context information collected at operation 502 and the network context information collected in the past. If a context tag table exists, DLLF 102 may perform a similarity check between the network context information collected at operation 502 and the context information stored in the context tag table. The similarity check may be performed similarly to operation 305.

[0084] At operation 505a, the DLLF 102 updates the context tag table using the context information received at operation 405. This may be done by assigning a context ID to the network context information collected at operation 502 based on the result of the similarity check. This may be done similarly to operation 306.

[0085] At operation 506a, the DLLF 102 sends the context ID assigned to the context information received at operation 405 to the AnLF 101. This may be done using the Ndllf_ContextMatching_Request_response service.

[0086] At operation 507a, the AnLF 101 accesses and / or retrieves information about one or more candidate AI models. The one or more candidate AI models may be identified based on a context ID assigned to the context information received at operation 405. Specifically, the AnLF 101 may access and / or retrieve information about one or more candidate AI models previously used in a network context associated with the context ID assigned to the context information received at operation 405. The information about the one or more AI models may be accessed from the MTLF 103.

[0087] At operation 508a, AnLF 101 sends a request for reasoning context history associated with one or more candidate AI models identified at operation 507a to ADRF 110. The request may specify a model ID for one or more candidate AI models identified at operation 507a. AnLF 101 may request only the reasoning context history associated with the context ID received at operation 506a. The request may specify the context ID received at operation 506a together with the model ID of the candidate AI model. In addition, AnLF 101 may request reasoning context history only for a specified time interval, one or more specified AOIs, and / or other specified parameters. If not specified, all available reasoning context histories for the specified AI model may be returned.

[0088] At operation 509a, the ADRF 110 returns the inference context history requested at operation 508a. This may be done using the Ndllf_InferenceHistory_Request_response service.

[0089] At operation 510a, AnLF 101 selects at least one AI model from one or more candidate AI models based on the reasoning context history received at operation 509a. In particular, AnLF 101 can select an AI model by identifying the best performing AI model in the network context associated with the same context ID in the reasoning context history. The performance of the AI ​​model can be evaluated using performance indicators such as model reasoning accuracy. If AnLF 101 detects a change in the current network context, AnLF 101 can decide to select an AI model based on the updated current network context.

[0090] At operation 511 , the AnLF 101 runs the selected at least one AI model to generate the requested analysis.

[0091] At operation 512, the AnLF 102 provides the requested analytics report to the NWDAF service consumer 120. This may be done using Nnwdaf_AnalyticsSubscription_Notify. The AnLF may monitor the network context and request a new AI model when a change is detected. The same monitoring process may also be performed by the DLLF.

[0092] Figure 5B is a signaling diagram illustrating an example method for using inference context history for AI / ML model selection. Figure 5BOne example is presented of operations at the NWDAF 100. Some of the operations described may not be present in all example embodiments, and example embodiments may also include additional features / operations described elsewhere in this specification.

[0093] At operation 501 , the NWDAF service consumer 120 (eg, the NF 121 , the AF 121 , or the OAM 123 ) sends a request for tracing the inference context history of the analysis service to the MWDAFAnLF 101 .

[0094] At operation 502, the AnLF 101 collects network context information. This may be performed similarly to operation 303. The network context corresponding to the network context information collected at operation 502 is referred to as an actual network context.

[0095] At operation 503 b , the AnLF 101 accesses and / or retrieves information about one or more candidate AI models. The information about one or more AI models may be accessed from the MTLF 103 .

[0096] At operation 504b, AnLF 101 sends a request to ADRF 110 for reasoning context histories associated with one or more candidate AI models identified at operation 503b. The request may specify a model ID for each candidate AI model identified in operation 503b. AnLF 101 may request only reasoning context histories associated with the actual context network. The request may specify the context information collected at operation 502. In addition, AnLF 101 may request reasoning context histories only for specified time intervals, one or more specified AOIs, and / or other specified parameters. If not specified, all available reasoning context histories for the specified AI model may be returned.

[0097] At operation 505b, DLLF 102 performs a similarity check between the network context information collected at operation 502 and the network context information collected in the past. If a context tag table exists, DLLF 102 may perform a similarity check between the network context information collected at operation 502 and the context information stored in the context tag table. The similarity check may be performed similarly to operation 305.

[0098] At operation 506b, the DLLF 102 updates the context tag table using the context information collected at operation 502. This may be done by assigning a context ID to the network context information collected at operation 502 based on the result of the similarity check. This may be done similarly to operation 306.

[0099] At operation 507b, the ADRF 110 returns the inference context history requested at operation 504b. This may be done using the Ndllf_InferenceHistory_Request_response service.

[0100] At operation 508b, AnLF 101 selects at least one AI model from one or more candidate AI models based on the inference context history received at operation 507b. In particular, AnLF 101 may select an AI model by identifying the best performing AI model in the network context associated with the same context ID in the inference context history. If AnLF 101 detects a change in the current network context, AnLF 101 may decide to select an AI model based on the updated current network context.

[0101] At operation 511 , the AnLF 101 runs the selected at least one AI model to generate the requested analysis.

[0102] At operation 512, the AnLF 102 provides the requested analytics report to the NWDAF service consumer 120. This may be done using Nnwdaf_AnalyticsSubscription_Notify. The AnLF may monitor the network context and request a new AI model when a change is detected. The same monitoring process may also be performed by the DLLF.

[0103] Figure 5C is a signaling diagram illustrating an example method for using inference context history for AI / ML model selection. Figure 5C One example of operations at the NWDAF 100 is presented. Some of the operations described may not be present in all example embodiments, and example embodiments may also include additional features / operations described elsewhere in this specification.

[0104] At operation 501, the NWDAF service consumer 120 (e.g., NF 121, AF 121, or OAM 123) sends a request to the MWDAF AnLF 101 to track the inference context history of the analysis service. The request may be sent when the service consumer 120 subscribes to or requests the analysis service provided by the NWDAF 100. For example, this may be performed by setting an attribute (e.g., inference_history_track attribute) to "true" in a subscription request (e.g., Nnwdaf_AnalyticsSubscription_Subscribe). The requested analysis may be identified by an analysis ID. The AnLF 101 may send a confirmation to the NWDAF service consumer 120.

[0105] At operation 502, the AnLF 101 collects network context information. This may be performed similarly to operation 303. The network context corresponding to the network context information collected at operation 502 is referred to as an actual network context.

[0106] At operation 503c, AnLF 101 sends a request to DLLF 102 to request DLLF 102 to provide an AI model for providing the requested analysis in the actual network context. The request includes the network context information collected at operation 502. The request may also include an analysis ID. The request may be sent using the Ndllf_MLmodelRetrieval_Request service.

[0107] At operation 504c, DLLF 102 performs a similarity check between the network context information collected at operation 502 and the network context information collected in the past. If a context tag table exists, DLLF 102 may perform a similarity check between the network context information collected at operation 502 and the context information stored in the context tag table. The similarity check may be performed similarly to operation 305.

[0108] At operation 505c, the DLLF 102 updates the context tag table using the context information collected at operation 502. This may be done by assigning a context ID to the network context information collected at operation 502 based on the result of the similarity check. This may be done similarly to operation 306.

[0109] At operation 506c, the DLLF 102 accesses and / or retrieves information about one or more candidate AI models. The one or more candidate AI models may be identified based on a context ID assigned to the context information received at operation 405. Specifically, the AnLF 101 may access and / or retrieve information about one or more candidate AI models previously used in a network context associated with the context ID assigned to the context information received at operation 405. The information about the one or more AI models may be accessed from the MTLF 103.

[0110] At operation 507c, the DLLF 102 selects at least one AI model from the one or more candidate AI models. In particular, the AnLF 101 may select an AI model by identifying the best performing AI model in the network context associated with the same context ID in the inference context history. If the AnLF 101 detects a change in the current network context, the AnLF 101 may decide to select an AI model based on the updated current network context.

[0111] At operation 508c, the DLLF 102 provides the at least one AI model selected at operation 507c to the AnLF 101. This may be done using the Ndllf_MLmodelRetrieval_Request_response service.

[0112] At operation 511 , the AnLF 101 runs the selected AI model to generate the requested analysis.

[0113] At operation 512, the AnLF 102 provides the requested analytics report to the NWDAF service consumer 120. This may be done using Nnwdaf_AnalyticsSubscription_Notify. The AnLF may monitor the network context and request a new AI model when a change is detected. The same monitoring process may also be performed by the DLLF.

[0114] Figure 6 An example embodiment of the subject matter described herein is illustrated, which illustrates a flowchart representation of a method.

[0115] At operation 610 , the method may include accessing an inference output generated by the AI ​​model, the inference output generated in a network context of a communication network.

[0116] At operation 620, the method may include collecting data indicative of a network context.

[0117] At operation 630 , the method may include storing the inference output together with data indicating the network context as an inference context history.

[0118] At operation 640, the method may include using the inference context history for network data analysis of the communication network. Using the inference context history for communication network data analysis may include selecting one or more AI models to perform network data analysis, monitoring the performance of one or more AI models to be used for network data analysis, and / or triggering retraining of the AI ​​models to be used for network data analysis.

[0119] As described in the appended claims and throughout the specification, further features of these methods result directly from the functionality and parameters of the NWDAF 100 and are therefore not repeated here. As described in connection with the various exemplary embodiments, different variants of these methods may also be applied.

[0120] An apparatus may include means for performing any aspect of the method(s) described herein. According to an example embodiment, the means include at least one processor and a memory including program code, the at least one processor and the program code being configured to, when executed by the at least one processor, cause the execution of any aspect of the method.

[0121] The functions described herein may be formed at least in part by one or more computer program product components (such as software components). According to an example embodiment, the apparatus includes a processor configured by a program code that, when executed, performs an example embodiment of the described operations and functions. Alternatively or in addition, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, exemplary hardware logic component types that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and graphics processing units (GPUs).

[0122] Any range or device value given herein may be expanded or altered without losing the effect sought. In addition, any embodiment may be combined with another embodiment unless expressly prohibited.

[0123] Although the subject matter has been described in language specific to structural features and / or acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Instead, the specific features and acts described above are disclosed as examples of implementing the claims, and other equivalent features and acts are intended to be within the scope of the claims.

[0124] It should be understood that the benefits and advantages described above may relate to one embodiment or to several embodiments. The embodiments are not limited to embodiments that solve any or all of the problems described, nor are they limited to embodiments that have any or all of the benefits and advantages described. It should also be understood that reference to "an" item may refer to one or more of these items.

[0125] The steps or operations of the methods described herein may be performed in any suitable order, or simultaneously where appropriate. In addition, individual blocks may be deleted from any method without departing from the scope of the subject matter described herein. Aspects of any of the above embodiments may be combined with aspects of any of the above embodiments to form additional embodiments without losing the effects sought.

[0126] The term "comprising" is used herein to mean including the identified methods, blocks, or elements, but such blocks or elements do not include an exclusive list and the method or apparatus may contain additional blocks or elements.

[0127] As used in this application, the term "circuitry" may refer to one or more or all of the following: (a) a pure hardware circuit implementation (such as an implementation in analog and / or digital circuitry only), and (b) a combination of hardware circuitry and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuitry with software / firmware, and (ii) any portion of hardware processor(s) (including digital signal processor(s)), software, and memory(s) with software that work in conjunction to enable a device (such as a mobile phone or server) to perform various functions), and (c) hardware circuitry and / or processor(s), such as microprocessor(s) or portions of microprocessor(s) that require software (e.g., firmware) to operate, but may not be present when software is not required for operation. This definition of circuitry applies to all uses of the term in this application, including in any claims.

[0128] As another example, as used in this application, the term circuitry also covers an implementation of only a hardware circuit or processor (or multiple processors) or portions of a hardware circuit or processor and their accompanying software and / or firmware. For example, the term circuitry also covers a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device, if applicable to a particular claim element.

[0129] It should be understood that the above description is given as an example only, and various modifications may be made by those skilled in the art. The above description, examples and data provide a complete description of the structure and use of the exemplary embodiments. Although various embodiments have been described above with a certain degree of clarity or with reference to one or more individual embodiments, those skilled in the art may make many modifications to the disclosed embodiments without departing from the scope of this specification.

Claims

1. A device comprising: at least one processor; as well as at least one memory storing instructions, wherein when the instructions are executed by the at least one processor, the apparatus performs at least operations, the operations comprising: For one or more artificial intelligence (AI) models: accessing an inference output generated by the AI ​​model, the inference output being generated in a network context of a communication network; collecting data indicative of the network context; and storing the inference output together with the data indicative of the network context as an inference context history; and The inference context history is used for network data analysis of the communication network.

2. The apparatus of claim 1, wherein using the inference context history for network data analysis comprises: Selecting an AI model from among the one or more AI models for performing the network data analysis, the AI ​​model being selected based on the reasoning context history; as well as The selected AI model is run to generate the network data analysis.

3. The apparatus of any one of claims 1 to 2, wherein storing the inference output together with the data indicative of the network context comprises: associating the network context with a tag; as well as The inference output is stored together with the label.

4. The apparatus of claim 2, wherein storing the inference output with the data indicative of the network context comprises: associating data indicative of the network context with the tag; and Wherein selecting the AI ​​model for performing the network data analysis includes: selecting the AI ​​model based on the reasoning context history associated with the tag.

5. The apparatus of claim 3 or 4, wherein associating the network context with the tag comprises: performing a similarity check between the data representing the network context and data representing historical network contexts stored in a context tag table; assigning a label to the network context based on a result of the similarity check; as well as The context tag table is updated by storing the data indicative of the network context together with the tag in the context tag table.

6. The apparatus according to claim 1 to 5, wherein using the inference context history for network data analysis comprises: The performance of one or more of the AI ​​models to be used for network data analysis is monitored, the performance of the AI ​​models being monitored by analyzing the reasoning context history associated with the AI ​​models.

7. The apparatus of claims 1 to 6, wherein the inference output is a prediction for a parameter, and wherein using the inference context history for network data analysis comprises: comparing the prediction for the parameter to a measured value of the parameter; as well as Based on determining that the prediction for the parameter does not match the measured value of the parameter, retraining of the AI ​​model is triggered.

8. The apparatus of any preceding claim, wherein the data indicative of the network context comprises one or more of: Slice load level, network function load, network performance, or user device related analysis, General network status, or Information about the type of sub-regions of the region of interest.

9. A method comprising: For one or more AI models: accessing an inference output generated by the AI ​​model, the inference output being generated in a network context of a communication network; collecting data indicative of the network context; as well as storing the inference output together with the data indicative of the network context as an inference context history; as well as The inference context history is used for network data analysis of the communication network.

10. An apparatus comprising: means for accessing an inference output generated by the AI ​​model, the inference output being generated in a network context of a communication network; means for collecting data indicative of said network context; means for storing said inference output together with said data indicative of said network context as an inference context history; as well as Means for using the inference context history for network data analysis of the communication network.

11. A computer program comprising instructions for causing an apparatus to perform at least the following: For one or more AI models: accessing an inference output generated by the AI ​​model, the inference output being generated in a network context of a communication network; collecting data indicative of the network context; as well as storing the inference output together with the data indicative of the network context as an inference context history; as well as The inference context history is used for network data analysis of the communication network.

Images