Apparatus and communication method for ai / ML operation

Abstract

A communication method for artificial intelligence (AI) / machine learning (ML) operations includes determining one or more ML model trustworthiness services, where the one or more ML model trustworthiness services include one or more ML model provisioning services and / or one or more ML trustworthiness services. The one or more ML model provisioning services include analysis information of the requested ML model to be used, and the analysis information includes a list of one or more analysis identifiers (IDs) and network function (NF) consumer information.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of communication systems, and more particularly, to an apparatus and a communication method for artificial intelligence (AI) / machine learning (ML) operation, such as AI / ML trustworthiness service for definition of services and their parameters. BACKGROUND

[0002] Currently, the standardization activities of researching artificial intelligence (AI) / machine learning (ML) functions are being carried out in the 3rd generation partnership project (3GPP) work. However, in the current technology, the aspect of AI / ML trustworthiness service is not yet clear in the 5G core network (5G core network, 5GC).

[0003] Therefore, there is a need for an apparatus and a communication method for AI / ML operation, such as AI / ML trustworthiness service, which can solve these problems and other problems. SUMMARY

[0004] The present disclosure aims to propose an apparatus and a communication method for artificial intelligence (AI) / machine learning (ML) operation service for definition of services and their parameters, which can solve these problems and other problems in the prior art.

[0005] In a first aspect of the present disclosure, a communication method for AI / ML operation includes determining one or more ML model trustworthiness services, wherein the one or more ML model trustworthiness services include one or more ML model provisioning services and / or one or more ML trustworthiness services.

[0006] In a second aspect of the present disclosure, a communication device includes a determiner for discovering trustworthiness capabilities for analytics and / or models.

[0007] In a third aspect of the present disclosure, an ML training (MLT) management service (MnS) producer includes a memory, a transceiver, and a processor coupled to the memory and the transceiver. The MLT MnS producer is configured to perform the above-mentioned method.

[0008] In a fourth aspect of the present disclosure, an MLT MnS consumer includes a memory, a transceiver, and a processor coupled to the memory and the transceiver. The MLT MnS consumer is configured to perform the above method.

[0009] In a fifth aspect of the present disclosure, a network device includes a memory, a transceiver, and a processor coupled to the memory and the transceiver. The network device is configured to perform the above method.

[0010] In a sixth aspect of the present disclosure, a non-transitory machine-readable storage medium has stored thereon instructions that, when executed by a computer, cause the computer to perform the above method.

[0011] In a seventh aspect of the present disclosure, a chip includes a processor configured to invoke and run a computer program stored in a memory, so that a device installed with the chip performs the above method.

[0012] In an eighth aspect of the present disclosure, a computer-readable storage medium stores a computer program that causes a computer to perform the above method.

[0013] In a ninth aspect of the present disclosure, a computer program product includes a computer program that causes a computer to perform the above method.

[0014] In a tenth aspect of the present disclosure, a computer program causes a computer to perform the above method. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present disclosure or related technologies, the following drawings used in the embodiments will be briefly introduced. Obviously, the drawings are only some of the embodiments of the present disclosure, and one of ordinary skill in the art can obtain other drawings from these drawings without paying any price.

[0016] Figure 1 is a block diagram of a non-roaming 5G system architecture for implementing some embodiments presented herein.

[0017] Figure 2 is a block diagram of data collection architecture from arbitrary 5GC network functions (NFs) for implementing some embodiments presented herein.

[0018] Figure 3 is a block diagram of data collection architecture using data collection coordination for implementing some embodiments presented herein.

[0019] Figure 4 is a block diagram of network data analytics exposure architecture for implementing some embodiments presented herein.

[0020] Figure 5is a block diagram of an open architecture for network data analytics using data collection coordination to implement some embodiments presented herein.

[0021] Figure 6 is a block diagram of a trained ML model provisioning architecture to implement some embodiments presented herein.

[0022] Figure 7 is a block diagram of a network device according to embodiments of the present disclosure.

[0023] Figure 8 is a flowchart illustrating a communication method for artificial intelligence (AI) / machine learning (ML) operation according to embodiments of the present disclosure.

[0024] Figure 9 is a block diagram of a communication device according to embodiments of the present disclosure.

[0025] Figure 10 is a block diagram of a machine learning training (MLT) device according to embodiments of the present disclosure.

[0026] Figure 11 is a block diagram of an example of a computing device according to embodiments of the present disclosure.

[0027] Figure 12 is a block diagram of a communication system according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0028] The technical problems, structural features, implementation purposes and effects of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Specifically, the terms in the embodiments of the present disclosure are only for describing certain embodiments, and are not intended to limit the present disclosure.

[0029] Figure 1 A non-roaming 5G system architecture for implementing some embodiments presented herein is shown. In the 5G non-roaming system architecture, network functions communicate with each other through service-based interfaces in the core network (CN). A user equipment (UE) can communicate with the core network to establish control signaling and enable the UE to use services from the CN. Examples of control signaling functions are registration, connection and mobility management, authentication and authorization, session management, etc. After control signaling has been established, the UE can then utilize user plane functions to send and receive data to and from data networks (DNs) such as the Internet.

[0030] The 5G system architecture includes the following network functions (NFs): authentication server function (AUSF), access and mobility management function (AMF), DN such as an operator service, an Internet access, or a third-party service, unstructured data storage function (UDSF), network exposure function (NEF), network repository function (NRF), network slice admission control function (NSACF), network slice-specific and SNPN authentication and authorization function (NSSAAF), network slice selection function (NSSF), policy control function (PCF), session management function (SMF), unified data management (UDM), unified data repository (UDR), user plane function (UPF), UE radio capability management function (UCMF), application function (AF), (radio) access network ((R)AN), 5G-equipment identity register (5G-EIR), network data analytics function (NWDAF), charging function (CHF), time sensitive networking AF (TSCAF), location management function (LMF), provisioning function (Provisioning F), and provisioning data management (Provisioning DM).TSN AF), a timesensitive communication and time synchronization function (TSCTSF), a data collection coordination function (DCCF), an analytics data repository function (ADRF), a messaging framework adaptor function (MFAF), and a non-seamless WLAN offload function (NSWOF).

[0031] The following description highlights Figure 1 Some capabilities of network functions (NFs) related to control signaling in the 5G core network (5GC).

[0032] Access and mobility function (AMF): The UE sends N1 messages to the AMF through the RAN node to perform control plane signaling such as registration, connection management, mobility management, access authentication and authorization, etc.

[0033] Session management function (SMF): The SMF is responsible for session management related to establishing a PDU session to enable the UE to send data to a data network (DN) such as the Internet or to an application server and other session management related functions.

[0034] Policy control function (PCF): The PCF provides a policy framework to govern network behavior, access subscription information, make policy decisions, etc.

[0035] Authentication server function (AUSF): The AUSF supports UE authentication for 3GPP access and untrusted non-3GPP access.

[0036] Unified data management / repository (UDM / UDR): The UDM / UDR supports 3GPP AKA authentication credential generation, user identification handling, subscription management and storage, etc.

[0037] Network slice selection function (NSSF): The NSSF is related to aspects of network slice management such as selecting network slice instances for UEs, managing NSSAI, etc.

[0038] Network repository function (NRF): The NRF supports service discovery functions in the 5G network.

[0039] Network Exposure Function (NEF): The NEF supports exposure of capabilities and events in the core network to third parties, application functions (AFs), edge computing, etc.

[0040] To enable both control plane and user plane communications, the RAN node provides access for communication from the UE to the core network. The UE establishes a PDU session with the CN to send data traffic over the (R)AN and UPF nodes of the 5G system (5GS) on the user plane. The uplink traffic is sent by the UE, and the downlink traffic is received by the UE using the established PDU session. The data traffic flows between the UE and the DN through the intermediate nodes (R)AN and UPF.

[0041] In some embodiments, the storage of trusted services can be defined in a non-roaming 5G system architecture as shown in FIG. 1, if there is a trustworthiness requirement in a non-roaming 5G system architecture as shown in FIG. 2, the way to access the storage for this operation can be selected for the appropriate network function (NF) that supports trustworthiness. In addition, Figure 1 Figure 1 Figure 1 A 5G system architecture is shown for implementing some embodiments regarding the provision of trusted services, the location of the service in the mobile network, the communication of the service with other services, and the corresponding trustworthiness parameters that are supported as part of the service.

[0042] As the use of artificial intelligence (AI) / machine learning (ML) in mobile networks increases, the need to ensure that AI / ML-supported services are trustworthy also increases. The following terms in the proposal for rules for AI can include: Trusted Machine Learning: This can propose a set of seven key requirements that a machine learning system must meet to be considered trustworthy. The details of each requirement are as follows: Human agency and oversight: AI systems can empower humans to make informed decisions and promote their fundamental rights. At the same time, there needs to be a mechanism for appropriate oversight, which can be achieved through human-in-the-loop, human-on-the-loop, and human-in-command approaches.

[0043] Technical robustness and security: AI systems need to be resilient and secure. AI systems need to be secure, ensuring that there are fallback options in the event of a failure, and have accuracy, reliability, and reproducibility. This can be the only way to ensure that unintended harm can be minimized and prevented.

[0044] ​​Privacy and data management: In addition to ensuring full respect for privacy and data protection, appropriate data management mechanisms are ensured, taking into account the quality and integrity of data, ensuring legal access to data.

[0045] Transparency: Data, systems, and AI business models are transparent. Traceability mechanisms help achieve this. In addition, the AI system and its decisions can be explained in a way that fits the relevant stakeholders. Humans need to be aware that they are interacting with an AI system and can understand the capabilities and limitations of that system.

[0046] Diversity, non-discrimination, and fairness: Unfair biases are avoided, as they can have multiple negative impacts, from marginalization of vulnerable groups to exacerbation of discriminatory biases. To promote diversity, AI systems should be accessible to all, with or without any disability, and be involved by relevant stakeholders throughout the life cycle of the AI system.

[0047] Accountability: Mechanisms can be established to ensure responsibility and accountability for AI systems and their outcomes. Auditability enables the evaluation of algorithms, data, and design processes, in which it plays a key role, especially in critical applications. In addition, adequate, easily accessible remedies can also be ensured.

[0048] Social and environmental well-being: AI systems can benefit all humans, including future generations. Therefore, it can be ensured that AI systems are sustainable and environmentally friendly. In addition, AI systems can take into account the environment, including other living beings, and carefully assess their impact on society and society.

[0049] As 3GPP is working on the standardization of AI / ML functionality in mobile networks, it is necessary to extend this standardization and define trustworthy aspects related to AI / ML functionality in mobile networks, including corresponding parameters, storage of parameters, network functions and network services, parameter handling. Some embodiments of the present disclosure are about defining storage of trustworthy services in the network, if there is a trustworthiness requirement in the network, for appropriate network functions that support trustworthiness, the way to access this storage can be selected for this operation. Some embodiments of the present disclosure analyze the 5G architecture and requirements and propose how to embed trustworthiness aspects into the 5G architecture and requirements.

[0050] Figure 2 A data collection architecture from any 5GC network function (NF) for implementing some embodiments presented herein is shown. As Figure 2As shown, the 5G system architecture allows the NWDAF to collect data from any 5GC NF. The NWDAF belongs to the same public land mobile network (PLMN) as the 5GC NF that provides the data. The Nnf interface is defined for the NWDAF to request subscription for data delivery for a specific context, cancel subscription for data delivery, and request specific reporting of data for a specific context. The 5G system architecture allows the NWDAF to retrieve management data from the OAM by invoking network management function (OAM) services. The 5G system architecture allows the NWDAF to collect data from any 5GC NF or OAM using a data collection coordination function (DCCF) through the associated Ndccf service. The 5G system architecture allows the NWDAF and DCCF to collect data from the NWDAF through the associated Nnwdaf_DataManagement service. The 5G system architecture allows the MFAF to extract data from the NWDAF through the associated Nnwdaf_DataManagement service.

[0051] Figure 3 A data collection architecture using data collection coordination for implementing some of the embodiments presented herein is shown. As Figure 3 shown, the Ndccf interface is defined for the NWDAF to support subscription request for data delivery from the DCCF, cancel subscription for data delivery, and request specific reporting of data. The DCCF requests data from the data source using the Nnf service if the data has not already been collected. The DCCF can collect the data and deliver it to the NWDAF, or the DCCF can rely on a messaging framework to collect the data from the NFs and deliver it to the NWDAF.

[0052] Figure 4 A network data analytics exposure architecture for implementing some of the embodiments presented herein is shown. As Figure 4As shown, the 5G system architecture allows any 5GC NF to request network analytics information from a NWDAF that contains an analytics logical function (AnLF). The NWDAF belongs to the same PLMN as the 5GC NF that consumes the analytics information. The Nnwdaf interface is defined for 5GC NFs to request subscription to network analytics delivery for a specific context, cancel subscription to network analytics delivery, and request specific reporting of network analytics for a specific context. The 5G system architecture also allows other consumers such as OAM and charging enablement function (CEF) to request network analytics information from the NWDAF. The 5G system architecture allows any NF to obtain analytics from the NWDAF using the DCCF function through the associated Ndccf service. The 5G system architecture allows the NWDAF and DCCF to request historical analytics from the NWDAF through the associated Nnwdaf_DataManagement service. The 5G system architecture allows the MFAF to extract historical analytics from the NWDAF through the associated Nnwdaf_DataManagement service.

[0053] Figure 5 An open architecture for network data analytics using data collection coordination is shown for implementing some of the embodiments presented herein. As shown in Figure 5 The Ndccf interface is defined for any NF to support subscription request to network analytics, cancel subscription to network analytics, and request specific reporting of network analytics. If analytics have not already been collected, the DCCF requests analytics from the NWDAF using the Nnwdaf service. The DCCF can collect and deliver the analytics to the NF, or the DCCF can rely on a messaging framework to collect and deliver the analytics to the NF.

[0054] Figure 6 A trained ML model provisioning architecture is shown for implementing some of the embodiments presented herein. As shown in Figure 6 The 5G system architecture allows a NWDAF that contains an analytics logical function (AnLF) to use a trained ML model provisioning service from another NWDAF that contains a model training logical function (MTLF). The Nnwdaf interface is used by the NWDAF that contains the AnLF to request and subscribe to the trained ML model provisioning service. The NWDAF that contains the AnLF can be the only consumer of the trained ML model provisioning service.

[0055] The NWDAF can contain the following logical functions: Analytics Logic Function (AnLF): A logical function in the NWDAF that performs reasoning, derives analytics information (i.e., derives statistics and / or predictions based on analytics consumer requests), and exposes analytics services, i.e., Nnwdaf_AnalyticsSubscription or Nnwdaf_Analyticslnfo.

[0056] Model Training Logic Function (MTLF): A logical function in the NWDAF that trains machine learning (ML) models and exposes new training services (e.g., provides trained ML models).

[0057] The NWDAF can contain either the MTLF or the AnLF, or both. Analytics information is either statistical information of past events or predictive information. There can be different NWDAF instances in the 5GC, which can be specialized according to the type of analytics. The capabilities of a NWDAF instance are described in the NWDAF profile stored in the NRF. To guarantee the accuracy of the analytics output for an analytics identifier (ID), the NWDAF detects and can remove input data from abnormal UEs based on UE abnormal behavior analytics, including abnormal UE list and observation time window, from itself or other NWDAF, and then can generate a new ML model and / or analytics output for the analytics ID without input data related to the abnormal UE list within the observation time window, and then send / update the ML model information and / or analytics output to the subscribed NWDAF service consumers.

[0058] To support NF discovery and selection of NWDAF instances containing MTLF, AnLF, or both, and capable of providing the required services (e.g., analytics exposure or ML model provisioning) for the required type of analytics, each NWDAF instance, in addition to other NRF registration elements of the NF profile, should provide a list of supported analytics IDs (possibly per supported service) when registering to the NRF. NFs that need to discover NWDAF instances that support some specific services for a specific type of analytics can query the NRF for NWDAFs that support the required services and the required analytics IDs. Consumers, i.e., 5GC NFs and OAM, decide how to use the data analytics provided by the NWDAF. Interactions between 5GC NFs and the NWDAF occur within a PLMN. The NWDAF is not aware of the NF application logic. The NWDAF can use subscription data, but only for statistical purposes. The NWDAF architecture allows for arranging multiple NWDAF instances in a hierarchy / tree with a flexible number of layers / branches. The number and organization of the hierarchy layers and the capabilities of each NWDAF instance remain a deployment choice.

[0059] In a layered deployment, when there is no DCCF, MFAF in the network, the NWDAF can provide data collection exposure capability for generating analytics based on data collected by other NWDAFs. To make the NWDAF discoverable in some network deployments, the NWDAF can be configured to register in the UDM (Nudm_UECM_Registration service operation) for the UEs it serves and the related analytics ID (e.g., for UE mobility analytics). The registration in the UDM can happen when the NWDAF starts to serve a UE or collect data for a UE. The NWDAF can be de-registered in the UDM when it deletes the analytics context for the UE of the related analytics ID.

[0060] Some solutions of the present disclosure propose to include a trusted service in the network. In some embodiments, a communication method for AI / ML operations includes discovering trustworthiness capabilities for analytics and / or models. Specifically, in some examples, a NWDAF supports AnLF capabilities, and / or a NWDAF supports MTLF capabilities. Some embodiments are about defining a trusted service, its location in the mobile network, its communication with other services, and the corresponding trustworthiness parameters that will be supported as part of this service. Some embodiments of the present disclosure define how to discover the trustworthiness capabilities of the corresponding analytics and the corresponding models in the following solutions: 1. A NWDAF supporting AnLF whose analytics parameters include trustworthiness.

[0061] 2. A NWDAF supporting MTLF whose analytics parameters correspond to the trained ML models. Optionally, the existing model provisioning service is re-used and extended to include trustworthiness parameters. Optionally, a new service for trustworthiness is introduced.

[0062] 3. In addition, for the case where no NWDAF is deployed in the network, or no NWDAF supports MTLF, or there is a NWDAF supporting MTLF but not supporting the related model ID, and the models can be used directly by different network functions, the models can be provisioned directly into the NRF and discovered by the corresponding network functions.

[0063] NWDAF discovery and selection The NWDAF service consumer uses the NWDAF discovery principles to select a NWDAF that supports the requested analytics information, the required analytics capabilities, and / or the requested ML model information. In some embodiments, the NWDAF can be enabled to support a MTLF with trustworthiness capabilities for ML models. In some embodiments, the NWDAF can be enabled to support an AnLF with trustworthiness capabilities for analytics. Different deployments can require different discovery and selection parameters. Different ways of performing the discovery and selection mechanism depend on different types of analytics / data (NF-related analytics / data and UE-related analytics / data). NF-related refers to analytics / data that does not require a SUPI nor a group of SUPIs (e.g., NF load analytics). UE-related refers to analytics / data that requires a SUPI or a group of SUPIs (e.g., UE mobility analytics).

[0064] To use the NRF discovery to include the AnLF: If the analytics is NF-related and the NWDAF service consumer (other than the NWDAF) cannot provide a region of interest for the requested data analytics, the NWDAF service consumer can select a NWDAF with a large service area from the candidate NWDAFs from the discovery response. Or, in case the consumer receives a NWDAF with aggregation capabilities, the consumer preferably selects a NWDAF with a large service area, with aggregation capabilities.

[0065] If the selected NWDAF cannot provide the requested data analytics, for example, because the NF to be contacted is outside the service area of the NWDAF, the selected NWDAF can reject the analytics request / subscription, or the selected NWDAF can query the NRF for another target NWDAF through the service area of the NF to be contacted. If the analytics is UE-related and the NWDAF service consumer (other than the NWDAF) cannot provide a region of interest for the requested data analytics, the NWDAF service consumer can select a NWDAF with a large service area from the candidate NWDAFs from the discovery response. Or, in case the consumer receives a NWDAF with aggregation capabilities, the consumer preferably selects a NWDAF with a large service area, with aggregation capabilities.

[0066] If the selected NWDAF is not able to provide analytics for the requested UE (e.g. the NWDAF serves a different service area), the selected NWDAF can reject the analytics request / subscription or the selected NWDAF can determine the AMF serving the UE, request the UE location information from the AMF and query the NRF for another target NWDAF serving the area where the UE is located. If the analytics is related to the UE and if the NWDAF instance indicates the weight of the TAI in its profile, the NWDAF service consumer can use the weight of the TAI to decide which NWDAF to select. If the NWDAF service consumer needs to discover a NWDAF containing AnLF with accuracy check capability, the consumer can query the NRF with the accuracy check capability provided in the discovery request. If the NWDAF service consumer needs to discover a NWDAF capable of collecting data from specific data sources identified by NF set ID or NF type of the specific data source, the consumer can query the NRF with the NF set ID or NF type of the specific data source provided in the discovery request.

[0067] To discover NWDAFs that have been registered in the UDM for a given UE: NWDAF service consumer or other NWDAF interested in UE related data or analytics, if supported, can query the UDM to discover the NWDAF instances that have been serving the given UE. If the NWDAF service consumer needs to discover a NWDAF with data collection exposure capability, the NWDAF service consumer can discover through the NRF the NWDAFs providing the Nnwdaf_DataManagement service and their associated data source NF type or their associated data source NF set ID.

[0068] To discover NWDAFs containing MTLF through the NRF: When trained ML models are available for one or more analytics IDs, the NWDAF containing MTLF shall include the ML model provisioning service (i.e., Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo) as one of the services supported during registration in the NRF. The NWDAF containing MTLF can provide to the NRF a list of analytics IDs corresponding to the trained ML models, and if possible, the ML model filter information for the trained ML model for each analytics ID. In this version of the specification, only the S-NSSAI and area of interest from the ML model filter information for the trained ML model for each analytics ID can be registered to the NRF during registration of the NWDAF containing MTLF. For each analytics ID, the NWDAF containing MTLF can also include a ML model interoperability indicator when registering to the NRF if the NWDAF containing MTLF supports ML model interoperability.

[0069] The ML model interoperability indicator includes a list of NWDAF providers (vendors) that are allowed to retrieve ML models from this NWDAF containing MTLF. The ML model interoperability indicator also indicates that the NWDAF containing MTLF supports interoperable ML models requested by the NWDAFs of the vendors in the list.

[0070] The S-NSSAI and area of interest from the ML model filter information are located within the S-NSSAI and NWDAF service area information indicated in the NF profile of the NWDAF containing MTLF, respectively.

[0071] In the process of discovery of the NWDAF containing MTLF, the consumer (i.e., NWDAF containing AnLF) can include in the request the target NF type (i.e., NWDAF), analytics ID, S-NSSAI, area of interest of the required trained ML model, ML model interoperability indicator, and NF consumer information. The NRF returns to the NF consumer one or more candidate instances of the NWDAF containing MTLF, each candidate instance of the NWDAF containing MTLF including the analytics ID, and if possible, the ML model filter information for the available trained ML model.

[0072] If the NWDAF service consumer needs to discover the NWDAF containing MTLF with accuracy check capability, the consumer can query the NRF while providing the accuracy check capability in the discovery request.

[0073] To discover the NWDAF containing MTLF with federated learning (FL) capability through the NRF: The NWDAF containing MTLF supporting FL as a server shall also include FL capability type (i.e. FL server), time interval of supporting FL as FL capability information during registration in NRF. The NWDAF containing MTLF supporting FL as a client shall also include FL capability type (i.e. FL client), time interval of supporting FL as FL capability information during registration in NRF and the NWDAF can also include NF type where data can be collected as input for local model training. The NWDAF containing MTLF can indicate in FL capability for a specific analytics ID that it supports both FL server and FL client.

[0074] In the process of discovery of NWDAF containing MTLF as FL server, the consumer (e.g. NWDAF containing MTLF) includes in the request FL capability type as FL server, time period of interest, and if available, ML model filter information for the ML model trained for each analytics ID. The NRF returns to the consumer one or more candidate instances of NWDAF containing MTLF as FL server. In the process of discovery of NWDAF containing MTLF as FL client, the consumer (e.g. FL server) includes in the request FL capability type as FL client, time period of interest, ML model filter information for the ML model trained for each analytics ID, list of NF types. The NRF returns to the consumer one or more candidate instances of NWDAF containing MTLF as FL client. The service consumer discovering NWDAF containing MTLF with FL capability is limited to NWDAF containing MTLF.

[0075] The PCF can be aware through signaling which NWDAFs are being used by the AMF, SMF, and UPF for a specific UE. This enables the PCF to select the same NWDAF instance that has been used for a specific UE. In roaming architecture, the NWDAF with roaming exchange capability to request analytics or input data (RE-NWDAF) is discovered through NRF. The consumer in the same PLMN as the RE-NWDAF discovers the RE-NWDAF by querying the NWDAF indicated in its NRF profile with roaming exchange capability. The consumer in the visited PLMN (i.e. RE-NWDAF) discovers the RE-NWDAF by querying the NWDAF in the target PLMN that supports the specific service defined for roaming. The RE-NWDAF (if not using delegated discovery) uses procedures to discover the RE-NWDAF in a different PLMN (i.e. HPLMN or VPLMN) where the detailed parameters are determined based on the analytics request or subscription from the consumer 5GC NF, operator policy, user consent and / or local configuration.

[0076] Examples: To define the trustworthiness service in 5GC, the content of ML model trustworthiness is described as follows: Content of ML model trustworthiness Content of ML model provisioning The consumer of the ML model provisioning service (i.e. NWDAF containing AnLF) can provide the input parameters listed as follows: - Information of the analysis for which the requested ML model is to be used, including: - List of analysis IDs: Identifying the analysis for which the ML model is used.

[0077] - NF consumer information: Identifying the provider of the NWDAF containing the AnLF.

[0078] NOTE 1: NF consumer information such as provider ID.

[0079] - [Optional] Use case context: Indicating the context in which the analysis is used to select the most relevant ML model. The use case context is a string.

[0080] NOTE 2: When several ML models are available for the requested analysis ID, the NWDAF containing the AnLF can use the parameter "use case context" to select the most relevant ML model.

[0081] - [Optional] ML model interoperability information. This is provider specific information conveying e.g. the requested model file format, model execution environment, required level of explainability, etc. The encoding, format, and values of the ML model interoperability information are not specified as it is provider specific information and agreed between providers if necessary for sharing purposes.

[0082] - [Optional] ML model filter information: Enabling selection of the ML model requested for the analysis, e.g. S-NSSAI, area of interest. The parameter types in the ML model filter information are the same as in the analysis filter information defined in the procedure. The ML model filter information can also include trustworthiness requirements parameters: data type; version; sampling frequency; sampling weight; model weight in case of linking / merging with another model; labeled / unlabeled data; risk level (e.g. not acceptable, high, limited); fairness; robustness; privacy; security; reliability; and / or traceability.

[0083] - [Optional] Target of the ML model reporting: Indicating the object for which the ML model is requested, e.g. specific UE, group of UEs, or any UE (i.e. all UEs).

[0084] - [Optional] Representative ratio of the request: When the ML model reports a target is a UE group, the minimum percentage of UEs in the group for which data is a non-empty set and can be used in the model training.

[0085] - ML model report information with the following parameters: - (only for Nnwdaf_MLModelProvision_Subscribe) ML model report information parameters, in line with the event report information parameters.

[0086] - [Optional] ML model target periodicity: indicates the time interval [start, end] for which the ML model is requested for analytics. The time interval is expressed in terms of actual start time and actual end time (e.g., by UTC time).

[0087] - [Optional] Inference input data information: contains information on various settings expected to be used by the AnLF during inference, e.g.: - "input data" expected to be used, where each data is optionally accompanied by a metric indicating the granularity for which the data will be used (i.e., sampling rate, maximum number of input values, and / or maximum time interval between samples of that input data).

[0088] NOTE 3: This can be a subset of possible input data specified for a certain analytics type.

[0089] - data sources expected to be used, which are a list of NF instance (or NF set) identifiers.

[0090] - notification target address (+ notification correlation ID) as defined in clause 4.15.1 of TS 23.502 [3], allowing to associate a notification received from the NWDAF containing the MTLF with this subscription.

[0091] - [Optional] indication of support for multiple ML models.

[0092] - [Optional] accuracy level of interest.

[0093] - [Optional] time when the model is needed: indicates the latest time at which the consumer expects to receive the ML model.

[0094] - [Optional] number of ML models, indicating the maximum number of multiple ML models that the ML model provider, e.g., the NWDAF (MTLF), can provide to the consumer of the ML model.

[0095] NOTE 4: Multiple ML model filter information consists of the indication of support for multiple ML models, the accuracy level of interest, the number of ML models.

[0096] - [Optional] ML model monitoring information: This is the information provided to the NWDAF containing the MTLF, which can include ML model metrics (i.e. ML model accuracy), ML model monitoring reporting mode (accuracy reporting interval or pre-determined status (ML model accuracy threshold)). Depending on the reporting mode, the NWDAF containing the MTLF reports the model accuracy to the NWDAF containing the AnLF periodically or when the ML model accuracy exceeds the ML model accuracy threshold, i.e. the accuracy is higher or lower than the ML model accuracy threshold.

[0097] - [Optional] ML model accuracy monitoring information with the following parameters: - [Optional] Analytics accuracy threshold: indicates the accuracy threshold of the ML model requested by the consumer. This threshold can also be used as an indication for the MTLF to be triggered to perform the accuracy monitoring operation for the ML model provisioned to the AnLF.

[0098] - [Optional] DataSetTag and ADRF ID if available: indicates the inference data (including input data, predictions, and true value data at the time the predictions are pointed to) stored in the ADRF, which can be used by the MTLF for re-training or re-provisioning the ML model.

[0099] The NWDAF containing the MTLF provides the consumer of the ML model provisioning service operation with the output information listed as follows: - (Only for Nnwdaf_MLModelProvision_Notify) notification correlation information.

[0100] - A set of pairs of unique ML model identifier and ML model information for each analytics ID requested by the service consumer.

[0101] ML model information, which includes: - ML model file address (e.g. URL or FQDN); or - [Optional] ML model degradation indicator: indicates whether the provided ML model is degraded.

[0102] - [Optional] validity period: indicates the time period for which the provided ML model information is applicable.

[0103] - [Optional] spatial validity: indicates the area for which the provided ML model information is applicable.

[0104] - [Optional] ML model representative ratio: indicates the percentage of UEs in the group for which the data in the group was used in the training of the ML model when the ML model reporting targets a group of UEs.

[0105] - [Optional] Training input data information: Information containing various settings that the MTLF has used during training, e.g.: - "Input data" that has been used, where each data is optionally accompanied by metrics indicating the characteristics of the data and the granularity at which the data has been used (i.e. sampling rate, maximum number of input values and / or maximum time interval between samples of that input data, data range including maximum and minimum values, mean and standard deviation, and data distribution when applicable) and the time at which the data was obtained, i.e. timestamp and duration.

[0106] - Data source related to the "input data" used for the ML model training.

[0107] - ADRF (set) ID.

[0108] When the ADRF (set) ID is provisioned, the storage transaction ID can also be provisioned.

[0109] NOTE 5: This can be a subset of possible input data specified for a certain analytics type.

[0110] - Data source that has been used as a list of NF instance (or NF set) identifiers.

[0111] NOTE 6: Spatial validity and validity period are determined by the MTLF internal logic and are a subset of the AoI if provided in the ML model filter information and the ML model target period, respectively.

[0112] NOTE 7: When different models are available for an analytics ID, the data source information enables ML model selection or the data source information enables the consumer to avoid selecting ML models that have used data from a specific data source or data characterized by specific data characteristics at a specific time.

[0113] - [Optional] ML model accuracy information: Indicates the accuracy of the ML model when the requested analytics accuracy threshold is requested, which includes: - Accuracy information of the ML model.

[0114] - [Optional] ML model metrics (i.e. ML model accuracy).

[0115] - [Optional] ML model trustworthiness: information containing various trustworthiness parameters resulting from the respective model training of the MTLF, e.g.: data type; version; sampling frequency; sampling weight; model weight in case of linking / merging with another model; labeled / unlabeled data; explainability level; risk level (e.g. unacceptable, high, limited); fairness; robustness; privacy; security; reliability; traceability; ML decision confidence score (a numerical value representing the trustworthiness / quality of a given decision generated by the AI / ML inference function); and / or value quality score of data, which is a numerical value representing the trustworthiness / quality of a given observation and measurement type.

[0116] Furthermore, instead of using an ML model provisioning service, a newly defined ML trustworthiness service can be used as one of the supported services. The ML trustworthiness service is provided during the registration of the NWDAF containing the MTLF with trustworthiness capabilities in the NRF when trained ML models are available for one or more analytics IDs. The ML trustworthiness service can contain information of various trustworthiness parameters resulting from the respective model training of the MTLF, e.g.: priority of fallback mechanisms between trusted AI solutions, untrusted AI solutions, and non-AI solutions for ensuring security; and / or (a list of) analytics IDs corresponding to trained ML models and ML model filter information for the trained ML models for each analytics ID.

[0117] The ML model filter information can comprise the following trustworthiness related parameters: data type; version; sampling frequency; sampling weight; model weight in case of linking / merging with another model; labeled / unlabeled data; explainability level; risk level (e.g. unacceptable, high, limited); fairness; robustness; privacy; security; reliability; traceability; ML decision confidence score (a numerical value representing the trustworthiness / quality of a given decision generated by the AI / ML inference function); and / or value quality score of data, which is a numerical value representing the trustworthiness / quality of a given observation and measurement type. The same list of parameters (defined above) can be used in the consumer-to-producer request for service and in the response.

[0118] Example: For the management analytics, there is a machine learning training (ML training, MLT) function.

[0119] An ML training function acting as an ML training MnS producer role can consume various data for the purpose of ML training. The ML training capability is provided by the ML training MnS producer to authorized consumers in the context of the SBMA. The internal business logic of the ML training utilizes current and historical relevant data, including those listed below, to monitor the network and / or services related to the ML model, prepare data, trigger and conduct training: performance measurement (PM) and key performance indicator (KPI); trace / MDT / RLF / RCEF data; QoE and service experience data; analytics data provided by the NWDAF; alarm information and notifications; CM information and notifications; MDA reports from the MDA MnS producer; management data from non-3GPP systems; and / or other data that can be used for training.

[0120] In an operational environment prior to deploying an ML entity for performing inference, there is a need to train the ML model associated with the ML entity (e.g., by an ML training function, which can be a separate entity or external to the AI / ML inference function). ML entity training refers to the training of the ML model associated with the ML entity. The ML entity is trained by an ML training (MLT) MnS producer, which can be triggered by a request from one or more MLT MnS consumers or initiated by the MLT MnS producer (e.g., based on model evaluation).

[0121] In some examples, the following table shows potential extension requirements to support trustworthiness.

[0122]

[0123] Figure 7An example of a network device 300 according to embodiments of the present disclosure is shown. The network device 300 is used to implement some embodiments herein. Some embodiments of the present disclosure can be implemented into the network device 300 using any suitably configured hardware and / or software. The network device 300 can include a memory 301, a transceiver 302, and a processor 303 coupled to the memory 301 and the transceiver 302. The processor 303 can be used to implement the proposed functions, procedures, and / or methods described in the present specification. Layers of radio interface protocols can be implemented in the processor 303. The memory 301 is operatively coupled with the processor 303 and stores a variety of information to operate the processor 303. The transceiver 302 is operatively coupled with the processor 303 and transmits and / or receives radio signals. The processor 303 can include application-specific integrated circuit (ASIC), other chip sets, logic circuit, and / or a data processing device. The memory 301 can include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media and / or other storage devices. The transceiver 302 can include a baseband circuit that processes a radio frequency signal. When the present embodiment is implemented with software, the techniques described herein can be implemented with a module (e.g., a procedure, a function, etc.) that performs the functions described herein. These modules can be stored on the memory 301 and executed by the processor 303. The memory 301 can be implemented within the processor 303 or implemented outside the processor 303 in which case it can be communicatively coupled to the processor 303 via various means as is known in the art.

[0124] In some embodiments, the memory 301 stores executable instructions that, when executed by the processor, cause the processor 303 to implement operations including determining one or more ML model trustworthiness services, wherein the one or more ML model trustworthiness services include one or more ML model provisioning services and / or one or more ML trustworthiness services.

[0125] Figure 8 A communication method for AI / ML operations according to embodiments of the present disclosure is shown. Figure 8is an example of a communication method 400 for AI / ML operations according to embodiments of the present disclosure. The communication method 400 for AI / ML operations is used to implement some embodiments of the present disclosure. Some embodiments of the present disclosure can be implemented into the communication method 400 for AI / ML operations using any suitably configured hardware and / or software. In some embodiments, the communication method 400 for AI / ML operations includes operations 402 of determining one or more ML model trustworthiness services, wherein the one or more ML model trustworthiness services include one or more ML model provisioning services and / or one or more ML trustworthiness services.

[0126] Figure 9 A communication device according to embodiments of the present disclosure is shown. Figure 9 In some embodiments, the communication device 500 includes a determiner 501 for determining one or more ML model trustworthiness services, wherein the one or more ML model trustworthiness services include one or more ML model provisioning services and / or one or more ML trustworthiness services.

[0127] Figure 10 An MLT device according to embodiments of the present disclosure is shown. The MLT device can include an MLT management service (MnS) producer and / or an MLT MnS consumer. Figure 10 In some embodiments, the MLT MnS producer is for determining one or more ML model trustworthiness services, wherein the one or more ML model trustworthiness services include one or more ML model provisioning services and / or one or more ML trustworthiness services. In some examples, the MLT MnS producer is for receiving an ML training request from the MLT MnS consumer, sending a response to the MLT MnS consumer indicating whether the ML training request is accepted, and / or sending a training result to the MLT MnS consumer.

[0128] Figure 10 In some embodiments, the MLT MnS consumer is for determining one or more ML model trustworthiness services, wherein the one or more ML model trustworthiness services include one or more ML model provisioning services and / or one or more ML trustworthiness services. In some embodiments, the MLT MnS consumer is for sending an ML training request to the MLT MnS producer, receiving a response from the MLT MnS producer indicating whether the ML training request is accepted, and / or receiving a training result from the MLT MnS producer.

[0129] Further, Figure 10It is shown that in some embodiments, the ML training capability is provided by the MLT MnS producer to one or more consumers. The ML training can be triggered by a request from one or more MLT MnS consumers. The consumer, for example, can be a network function, a management function, an operator, or another functional differentiation that triggers the ML training, the MLT MnS consumer requests the MLT MnS producer to train the ML model. In the ML training request, the consumer shall specify the inference type indicating the function or purpose of the ML entity, for example, CoverageProblemAnalysis. The MLT MnS producer can perform the training according to the specified inference type. The consumer can provide a data source containing the training data that is considered as input candidates for the training. In order to obtain effective training results, the consumer can also specify its requirements on the model performance (e.g., accuracy, etc.) in the training request.

[0130] Figure 10 It is shown that in some embodiments, the MLT MnS producer provides a response to the consumer indicating whether the request is accepted. If the request is accepted, the MLT MnS producer decides when to initiate the ML training considering the request from the consumer. Once the training is decided, the producer performs the following operations: selects the training data considering the candidate training data provided by the consumer. Since the training data directly affects the performance of the algorithm and the trained ML entity, the MLT MnS producer can examine the training data provided by the consumer and decide not to select these training data, select some of these training data, or select all of these training data. In addition, the MLT MnS producer can select some other available training data; trains the ML entity using the selected training data; and provides the training result (including the location of the trained ML model or entity, etc.) to the MLT MnS consumer.

[0131] Producer-initiated ML training The MLT MnS producer can initiate the ML training, for example, based on the feedback or new training data received from the consumer according to the performance evaluation of the ML model, or when new training data describing new network status / events becomes available that is not from the consumer.

[0132] When the MLT MnS producer decides to initiate the ML training, the producer performs the following operations: selects the training data; trains the ML entity using the selected training data; and provides the training result (including the location of the trained ML entity, etc.) to the MLT MnS consumer that has subscribed to receive the ML training result.

[0133] ML model and ML entity selection For a given machine learning based use case, different entities applying the respective ML model or AI / ML inference function can have different inference requirements and capabilities. For example, one consumer has specific responsibilities and wants to have AI / ML inference functions supported by ML models or entities trained for an urban downtown area where mobile users move at a speed not exceeding 30 km / hr. On the other hand, another consumer can support a rural environment, whereby he wants to have ML models and AI / ML inference functions suitable for this type of environment. Different consumers need to know the available versions of ML entities and variants of trained ML models or entities and need to select the appropriate version for their respective conditions.

[0134] Furthermore, there is no guarantee that the available ML models / entities have been trained according to the characteristics expected by the consumer. It follows that the consumer needs to know the conditions under which the ML model or ML entity is trained and then enables the consumer to select the model that best suits their conditions and needs.

[0135] Trained models can differ in complexity and performance. For example, a generic comprehensive and complex model can have been trained in a cloud-like environment, but when such a model cannot be used in a gNB, instead, a less complex model trained as a derivative of this generic model can be a better candidate. Furthermore, multiple less complex models can be trained at different levels of complexity and performance, which would then allow to deliver different relevant models to different network functions depending on the operating conditions and performance requirements. Network functions need to know the available alternative models and request and replace them interactively when needed and according to observed inference related constraints and performance requirements.

[0136] Managing ML training procedures This machine learning capability relates to the way for managing and controlling the ML model / entity training procedures. In order to achieve the desired results for any machine learning related use case, it is necessary to train the ML models applied for such analysis and decision making with appropriate data. The training can be done in the managed function or in the managing function. In both cases, the network (or its OAM system) needs to have not only the required training capabilities but also the way to manage the training of the ML models / entities. Consumers need to be able to interact with the training procedure, for example, to pause or restart the procedure; and also to manage and control the requests related to any such training procedure.

[0137] Handling errors in data and ML decisions ML models / entities are trained on high-quality data, i.e., data that is correctly collected and reflects the true network state to represent the intended context for which the ML entity is to operate. High-quality data is free from errors, e.g.: Inaccurate measurements of additive noise (e.g., RSRP, SINR, or QoE estimates).

[0138] Missing values or missing complete records, e.g., due to communication link failure.

[0139] Records with significant delay in communication (in case of online measurements).

[0140] If there were no errors, the ML entity could rely on a small amount of accurate inputs without leveraging the redundancy present in the training data. However, during inference, the ML entity is likely to encounter these inconsistencies. When this happens, the ML entity shows higher errors in the inference output even if redundant and uncorrupted data is available from other sources.

[0141] Therefore, the system needs to account for errors and inconsistencies in the input data, and the consumers should handle the decisions made based on such erroneous and inconsistent data. The system can: 1) enable the functionality to train the ML entity in a way that it is prepared to handle errors in the training data, i.e., to identify errors in the data during training; 2) enable the MLT MnS consumer to be aware of the possibility of the ML entity using erroneous input data.

[0142] Figures 1 to 10 It is shown that in some embodiments, the one or more ML model provisioning services include analytics information for the requested ML model to be used, and the analytics information includes a list of one or more analytics IDs and NF consumer information. In some embodiments, the NF consumer information includes ML model filter information configured to enable selection of the ML model to be requested for analytics. In some embodiments, the ML model filter information includes at least one of the following trustworthiness related parameters: data type; version; sampling frequency; sampling weight; model weight in case of linking / merging with another model; labeled / unlabeled data; risk level; fairness; robustness; privacy; security; safety; and / or reliability. In some embodiments, the ML model filter information includes ML model trustworthiness containing information of one or more trustworthiness parameters resulting from model training by the MTLF.

[0143] In some embodiments, the one or more trustworthiness parameters include: data type; version; sampling frequency; sampling weight; model weight in case of linking / merging with another model; labeled / unlabeled data; explainability level; risk level; fairness; robustness; privacy; security; safety; reliability; traceability; ML decision confidence score; and / or value quality score of data. In some embodiments, the one or more ML trustworthiness services include information of one or more trustworthiness parameters resulting from model training by the MTLF. In some embodiments, the one or more trustworthiness parameters of the one or more ML trustworthiness services include: priority of fallback mechanisms between trusted AI solutions, untrusted AI solutions, and non-AI solutions for ensuring security; and / or one or more analytics IDs corresponding to trained ML models and ML model filter information for trained ML models of each analytics ID.

[0144] In some embodiments, the method further includes discovering, using the one or more ML model trustworthiness services, trustworthiness capabilities for analytics and / or models. In some embodiments, discovering the trustworthiness capabilities for analytics and / or models is performed by the NRF. In some embodiments, discovering the trustworthiness capabilities for analytics and / or models includes: the NWDAF being enabled to support the MTLF with trustworthiness capabilities for ML models; and / or the NWDAF being enabled to support the AnLF with trustworthiness capabilities for analytics; and / or the NRF including trustworthiness capability provisioning for each ML model. In some embodiments, the NRF includes trustworthiness capability provisioning for each ML model in case that the NWDAF is not deployed in the network; in case that the NWDAF does not support the MTLF; or in case that the NWDAF supports the MTLF but does not support related model IDs and / or models are used by different network functions. In some embodiments, the NWDAF containing the MTLF with trustworthiness capabilities provides one or more ML model provisioning services and / or one or more ML trustworthiness services during registration in the NRF when trained ML models are available for one or more analytics IDs.

[0145] The business benefits of some embodiments are as follows. 1. Addressing the problems in the prior art and other problems. 2. Handling trustworthiness of analytics and ML models in 5GC and extending the handling of trustworthiness in the management domain (MDA). 3. One of the key mechanisms required is to define trustworthiness capabilities as part of the ML services used by the network, and this is what the present invention implements, both as a request from the consumer of the service and in the response from the producer of the service. Thus, some embodiments of the present invention enable trustworthiness services in the 5GC and in the management domain. Some embodiments of the present disclosure can be used in many applications. Some embodiments of the present disclosure are used by chip set vendors, communication system development vendors, automobile manufacturers including cars, trains, trucks, buses, bicycles, motorcycles, helmets, etc., drones (unmanned aerial vehicles), smartphone manufacturers, communication devices for public safety, AR / VR / MR device manufacturers for e.g. gaming, conferences / seminars, educational purposes. Some embodiments of the present disclosure are a combination of “technologies / processes” that can be adopted in video standards to create end products. Some embodiments of the present disclosure propose technical mechanisms. At least one proposed solution, method, system, and apparatus of some embodiments of the present disclosure can be used for current and / or new / future standards related to communication systems such as UEs, base stations, network equipment, and / or communication systems. Compliant products follow at least one proposed solution, method, system, and apparatus of some embodiments of the present disclosure. The proposed solution, method, system, and apparatus are widely used in UEs, base stations, network equipment, and / or communication systems. With implementation of at least one proposed solution, method, system, and apparatus of some embodiments of the present disclosure, at least one modification / improvement to the method and apparatus for charging reporting for AI / ML operations is considered for standardization.

[0146] Furthermore, once AI / ML starts to be adopted in 5G / 6G networks, the demand for supporting AI / ML trustworthiness can increase. Reasons can include: local regulations allow the use of AI / ML services that comply with certain levels of e.g. fairness in mobile networks; and / or service providers request network equipment providers to support certain levels of AI / ML trustworthiness, e.g. robustness and explainability. Thus, some embodiments of the present application can serve as a basis for 3GPP standardization (starting from Release 19) to enable a standardized approach for such AI / ML trustworthiness and support such AI / ML trustworthiness within message structures and by standardized network functions. It is not an “end product”, but part of the network implementation that creates 5G network products.

[0147] Figure 11is an example of a computing device 1100 according to embodiments of the present disclosure. Any suitable computing device can be used to perform the operations described herein. For example, Figure 11 The apparatus and / or methods illustrated in FIG. 11 can be implemented using any suitable hardware and / or software configured as desired. For example, Figures 1 to 10 An example of a computing device 1100 that can implement the apparatus and / or methods illustrated in FIG. 11 is shown. In some embodiments, the computing device 1100 can include a processor 1112 that is communicatively coupled to a memory 1114 and that executes computer executable program code and / or accesses information stored in the memory 1114. The processor 1112 can include a microprocessor, an application specific integrated circuit (“ASIC”), a state machine, or other processing means. The processor 1112 can include any of a plurality of processing devices, including one processing device. Such processors can include computer readable medium storing or in communication with storing instructions that, when executed by the processor 1112, cause the processor to perform operations described herein.

[0148] The memory 1114 can include any suitable non-transitory computer readable medium. A computer readable medium can include any electronic, optical, magnetic, or other storage device capable of providing a processor with computer readable instructions or other program code. Non-limiting examples of computer readable medium include magnetic disks, memory chips, ROM, RAM, ASICs, configured processors, optical storage devices, magnetic tape or other magnetic storage devices, or any other medium that a computer processor can read from or write to. Instructions can include processor specific instructions generated by a compiler and / or an interpreter from code written in any suitable computer programming language including, for example, C, C++, C#, visual basic, java, python, perl, javascript, and actionscript.

[0149] The computing device 1100 can also include a bus 1116. The bus 1116 can communicatively couple one or more components of the computing device 1100. The computing device 1100 can also include a number of external or internal devices such as input or output devices. For example, the computing device 1100 is shown with an input / output (input / output, “I / O”) interface 1118 that can receive input from or provide output to one or more input devices 1120 or one or more output devices 1122. The one or more input devices 1120 and the one or more output devices 1122 can be communicatively coupled to the I / O interface 1118. The communicative coupling can be achieved by any suitable means, such as by a connection through a printed circuit board, by a connection through a cable, by a communication transmitted wirelessly, etc. Non-limiting examples of input devices 1120 include a touchscreen (e.g., one or more cameras to image a touch area or pressure sensors to detect changes in pressure caused by a touch), a mouse, a keyboard, or any other device that can be used to generate input events in response to physical actions of a user of the computing device. Non-limiting examples of output devices 1122 include a liquid crystal display (LCD) screen, an external monitor, a speaker, or any other device that can be used to display or otherwise present output generated by the computing device.

[0150] The computing device 1100 can execute program code that configures the processor 1112 to perform one or more of the operations described above with respect to some of the embodiments shown in FIG. 11. Figures 1 to 10 The program code can reside in the memory 1114 or any suitable computer-readable medium, and can be executed by the processor 1112 or any other suitable processor.

[0151] The computing device 1100 can also include at least one network interface device 1124. The network interface device 1124 can include any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks 1128. Non-limiting examples of network interface devices 1124 include Ethernet adapters, modems, etc. The computing device 1100 can transmit messages as electrical or optical signals through the network interface device 1124.

[0152] Figure 12 is a block diagram of an example of a communication system 1200 in accordance with embodiments of the present disclosure. The embodiments described herein can be implemented into the communication system 1200 using any suitable configuration of hardware and / or software. Figure 12A communication system 1200 is shown, including radio frequency (RF) circuitry 1210, baseband circuitry 1220, application circuitry 1230, memory / storage 1240, a display 1250, a camera 1260, a sensor 1270, and an input / output (I / O) interface 1280 coupled with each other as shown.

[0153] The application circuitry 1230 can include circuitry such as, but not limited to, one or more single-core or multi-core processors. The processors can include any combination of general-purpose processors and dedicated processors (such as graphics processors, application processors, etc.). The processors can be coupled with memory / storage and used to execute instructions stored in the memory / storage to enable various applications and / or operating systems to run on the system. The communication system 1200 can execute program code that configures the application circuitry 1230 to perform one or more of the operations described above with respect to the Figures 1 to 9 program code can reside in the application circuitry 1230 or any suitable computer-readable medium and can be executed by the application circuitry 1230 or any other suitable processor.

[0154] The baseband circuitry 1220 can include circuitry such as, but not limited to, one or more single-core or multi-core processors. The processors can include baseband processors. The baseband circuitry can handle various radio control functions. These can include, but are not limited to, signal modulation, encoding, decoding, radio frequency shifting, etc. In some embodiments, the baseband circuitry can provide for communication compatible with one or more wireless technologies. For example, in some embodiments, the baseband circuitry can support communication compatible with 5G, 4G (e.g., Long Term Evolution (LTE)), 3G (e.g., Universal Terrestrial Radio Access Network (UTRAN), etc.), WiMAX, etc. In some embodiments, the baseband circuitry can include one or more audio digital signal processor(s) (DSP) 1221. The baseband circuitry can be configured to process video calls, for example.

[0155] In various embodiments, baseband circuit 1220 may include circuitry that operates using signals strictly considered not to be in the baseband frequency range. For example, in some embodiments, the baseband circuitry may include circuitry that operates using signals having an intermediate frequency (IF) between the baseband frequency and the radio frequency (RF). RF circuitry 1210 enables communication with a wireless network using modulated electromagnetic radiation over a non-solid-state medium. In various embodiments, RF circuitry may include switches, filters, amplifiers, etc., to facilitate communication with a wireless network. In various embodiments, RF circuitry 1210 may include circuitry that operates using signals strictly considered not to be in the radio frequency range. For example, in some embodiments, RF circuitry may include circuitry that operates using signals having an intermediate frequency (IF) between the baseband frequency and the radio frequency.

[0156] In various embodiments, the above regarding Figures 1 to 11 The transmitter circuitry, control circuitry, or receiver circuitry discussed in the apparatus and / or methods illustrated may be wholly or partially embodied in one or more of the RF circuitry, baseband circuitry, and / or application circuitry. As used herein, “circuit” may refer to, be part of, or may include: an ASIC, electronic circuitry, processor (shared, dedicated, or grouped) and / or memory (shared, dedicated, or grouped), combinational logic circuitry, and / or other suitable hardware components that provide the described functionality, executing one or more software or firmware programs. In some embodiments, the electronic device circuitry may be implemented in one or more software or firmware modules, or the functionality associated with the circuitry may be implemented by one or more software or firmware modules. In some embodiments, some or all of the components of the baseband circuitry, application circuitry, and / or memory / storage device may be implemented together on a system on chip (SOC). The memory / storage device 1240 may be used to load and store, for example, data and / or instructions for the system. The memory / storage device for one embodiment may include any combination of suitable volatile memory (such as dynamic random access memory, DRAM) and / or non-volatile memory (such as flash memory).

[0157] In various embodiments, the I / O interface 1280 can include one or more user interfaces designed to enable user interaction with the system and / or peripheral component interfaces designed to enable peripheral component interaction with the system. User interfaces can include, but are not limited to, physical keyboards or keypads, touch pads, speakers, microphones, etc. Peripheral component interfaces can include, but are not limited to, non-volatile memory ports, universal serial bus (USB) ports, audio jacks, and power interfaces. In various embodiments, the sensors 1270 can include one or more sensing devices to determine environmental conditions and / or location information related to the system. In some embodiments, the sensors can include, but are not limited to, gyroscopes, accelerometers, proximity sensors, ambient light sensors, and positioning units. The positioning units can also be part of, or interact with, the baseband circuitry and / or RF circuitry to communicate with components of a positioning network (e.g., global positioning system (GPS) satellites).

[0158] In various embodiments, the display 1250 can include a display such as a liquid crystal display and a touch screen display. In various embodiments, the communication system 1200 can be a mobile computing device such as, but not limited to, a laptop computing device, a tablet computing device, a netbook, an ultrabook, a smartphone, AR / VR glasses, etc. In various embodiments, the system can have more or less components, and / or different architectures. Where appropriate, methods described herein can be implemented as a computer program. The computer program can be stored on a storage medium, such as a non-transitory storage medium.

[0159] Those of ordinary skill in the art will understand that each unit, algorithm, and step described and disclosed in the embodiments of the present disclosure is implemented using electronic hardware or a combination of software and electronic hardware for a computer. Whether the function is implemented in hardware or software depends on the conditions of the application and the design requirements of the technical solution. Those of ordinary skill in the art can use different ways to implement the function of each specific application, and such implementation shall not exceed the scope of the present disclosure. Those of ordinary skill in the art can understand that, since the working processes of the systems, devices and units in the above-described embodiments are basically the same, the working processes of the systems, devices and units in the above-described embodiments can be referred to. For the sake of easy description and simplicity, these working processes will not be described in detail.

[0160] It can be understood that the disclosed system, device and method in the embodiments of the present disclosure can be implemented in other ways. The above-mentioned embodiments are merely exemplary. The division of units is merely based on logical functions and there are other divisions in implementation. A plurality of units or components can be combined or integrated in another system. It is also possible to omit or skip some features. On the other hand, the mutual coupling, direct coupling or communication coupling shown or discussed operates indirectly or communicatively through some ports, devices or units in the form of electrical, mechanical or other kinds.

[0161] The units as separate components for explanation are physically separate or not physically separate. The units for display are physical units or not physical units, i.e. located in one place or distributed on multiple network units. Some or all of the units are used according to the purpose of the embodiments. In addition, each functional unit in each embodiment can be integrated in one processing unit which is physically independent, or integrated in one processing unit with two or more units.

[0162] If the software functional unit is realized, used and sold as a product, it can be stored in a readable storage medium in a computer. Based on this understanding, the technical solutions proposed by the present disclosure can be basically or partially realized in the form of a software product. Alternatively, part of the technical solutions beneficial to the conventional technology can be realized in the form of a software product. The software product in the computer is stored in a storage medium, which includes a plurality of commands for a computing device (such as a personal computer, a server or a network device) to run all or some of the steps disclosed by the embodiments of the present disclosure. The storage medium includes a USB disk, a mobile hard disk, a ROM, a RAM, a floppy disk or other kinds of media capable of storing program codes.

[0163] Although the present disclosure has been described in conjunction with the embodiments considered to be the most practical and preferred, it should be understood that the present disclosure is not limited to the disclosed embodiments, but is intended to cover various arrangements made without departing from the scope of the appended claims in the broadest interpretation.

Claims

1. A communication method for artificial intelligence (AI) / machine learning (ML) operations, comprising: Identify one or more ML model credibility services, wherein the one or more ML model credibility services include one or more ML model provisioning services and / or one or more ML credibility services.

2. The method according to claim 1, wherein, The one or more ML model provisioning services include analytics information for the requested ML model to be used, and the analytics information includes a list of one or more analytics identifiers (IDs) and network function (NF) consumer information.

3. The method according to claim 2, wherein, The NF consumer information includes ML model filter information configured to enable the selection of the ML model to be requested for analysis.

4. The method according to claim 3, wherein, The ML model filter information includes at least one of the following confidence-related parameters: data type; version; sampling frequency; sampling weight; Model weights in the case of linking / merging with another model; labeled / unlabeled data; risk level; fairness; robustness; privacy; security; safety; and / or reliability.

5. The method according to claim 3, wherein, The ML model filter information includes ML model credibility, which contains information about one or more credibility parameters generated by the Model Training Logic Function (MTLF) during model training.

6. The method according to claim 5, wherein, The one or more confidence parameters include: data type; version; sampling frequency; sampling weight; model weight in the case of linking / merging with another model; labeled / unlabeled data; interpretability level; risk level; fairness; robustness; privacy; security; safety; reliability; traceability; ML decision confidence score; and / or data value quality score.

7. The method according to any one of claims 1 to 6, wherein, The one or more ML credibility services include information on one or more credibility parameters generated by model training using MTLF.

8. The method according to claim 7, wherein, The one or more trust parameters of the one or more ML trust services include: the priority of trusted AI solutions, untrusted AI solutions, and fallback mechanisms between non-AI solutions to ensure security; and / or one or more analysis IDs corresponding to the trained ML model and the ML model filter information for the trained ML model for each analysis ID.

9. The method according to any one of claims 1 to 8, further comprising: Use one or more of the ML model credibility services to discover credibility capabilities for analytics and / or models.

10. The method according to claim 9, wherein, The ability to determine the credibility of an analysis and / or model is achieved through the Network Repository Function (NRF).

11. The method according to claim 9 or 10, wherein, The ability to determine the credibility of an analysis and / or model includes: Network Data Analysis Functionality (NWDAF) is enabled to support Model Training Logic Functionality (MTLF) with the aforementioned credibility capability for ML models; and / or The NWDAF is enabled to support the Analysis Logic Function (AnLF) with the credibility capability for the analysis; and / or NRF includes the credibility capability supply for each ML model.

12. The method according to claim 11, wherein, In the absence of NWDAF deployed in the network; in the absence of NWDAF supporting the MTLF; or in the presence of NWDAF supporting the MTLF but not supporting the associated model ID and / or the model being used by different network functions, the NRF includes the provision of credibility capabilities for each ML model.

13. The method according to claim 11, wherein, When a trained ML model is available for use with one or more analytics IDs, the NWDAF containing the MTLF with the said credibility capability provides the one or more ML model provisioning services and / or the one or more ML credibility services during its registration with the NRF.

14. A communication device, comprising: A determiner for determining one or more ML model credibility services, wherein the one or more ML model credibility services include one or more ML model provisioning services and / or one or more ML credibility services.

15. The communication device according to claim 14, wherein, The one or more ML model provisioning services include analytics information for the requested ML model to be used, and the analytics information includes a list of one or more analytics identifiers (IDs) and network function (NF) consumer information.

16. The communication device according to claim 15, wherein, The NF consumer information includes ML model filter information configured to enable the selection of the ML model to be requested for analysis.

17. The communication device according to claim 16, wherein, The ML model filter information includes at least one of the following confidence-related parameters: data type; version; sampling frequency; sampling weight; Model weights in the case of linking / merging with another model; labeled / unlabeled data; risk level; fairness; robustness; privacy; security; safety; and / or reliability.

18. The communication device according to claim 16, wherein, The ML model filter information includes ML model credibility, which contains information about one or more credibility parameters generated by the Model Training Logic Function (MTLF) during model training.

19. The communication device according to claim 18, wherein, The one or more confidence parameters include: data type; version; sampling frequency; sampling weight; model weight in the case of linking / merging with another model; labeled / unlabeled data; interpretability level; risk level; fairness; robustness; privacy; security; safety; reliability; traceability; ML decision confidence score; and / or data value quality score.

20. The communication device according to any one of claims 14 to 19, wherein, The one or more ML credibility services include information on one or more credibility parameters generated by model training using MTLF.

21. The communication device according to claim 20, wherein, The one or more trust parameters of the one or more ML trust services include: the priority of trusted AI solutions, untrusted AI solutions, and fallback mechanisms between non-AI solutions to ensure security; and / or one or more analysis IDs corresponding to the trained ML model and the ML model filter information for the trained ML model for each analysis ID.

22. The communication device according to any one of claims 14 to 21, further comprising: Use one or more of the ML model credibility services to discover credibility capabilities for analytics and / or models.

23. The communication device according to claim 22, wherein, The ability to determine the credibility of an analysis and / or model is achieved through the Network Repository Function (NRF).

24. The communication device according to claim 22 or 23, wherein, The ability to determine the credibility of an analysis and / or model includes: Network Data Analysis Functionality (NWDAF) is enabled to support Model Training Logic Functionality (MTLF) with the aforementioned credibility capability for ML models; and / or The NWDAF is enabled to support the Analysis Logic Function (AnLF) with the credibility capability for the analysis; and / or NRF includes the credibility capability supply for each ML model.

25. The communication device according to claim 24, wherein, In the absence of NWDAF deployed in the network; in the absence of NWDAF supporting the MTLF; or in the presence of NWDAF supporting the MTLF but not supporting the associated model ID and / or the model being used by different network functions, the NRF includes the provision of credibility capabilities for each ML model.

26. The communication device according to claim 24, wherein, When a trained ML model is available for use with one or more analytics IDs, the NWDAF containing the MTLF with the said credibility capability provides the one or more ML model provisioning services and / or the one or more ML credibility services during its registration with the NRF.

27. A producer of ML training (MLT) management service (MnS), comprising: Memory; transceiver; as well as A processor coupled to the memory and the transceiver; The MLT MnS producer is used to perform the method according to any one of claims 1 to 13.

28. The MLT MnS producer according to claim 24, wherein, The transceiver is used to receive ML training requests from the MLT MnS consumer, send a response to the MLT MnS consumer indicating whether the ML training request is accepted, and / or send training results to the MLT MnS consumer.

29. An MLT MnS consumer, comprising: Memory; transceiver; as well as A processor coupled to the memory and the transceiver; The MLT MnS consumer is used to perform the method according to any one of claims 1 to 13.

30. The MLT MnS consumer according to claim 29, wherein, The transceiver is used to send ML training requests to the MLT MnS producer, receive responses from the MLT MnS producer indicating whether the ML training requests are accepted, and / or receive training results from the MLT MnS producer.

31. A network device, comprising: Memory; transceiver; as well as A processor coupled to the memory and the transceiver; The network device is used to perform the method according to any one of claims 1 to 13.

32. A non-transitory machine-readable storage medium having instructions stored thereon that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 13.

33. A chip, comprising: A processor for calling and running a computer program stored in memory to cause a device on which the chip is mounted to perform the method according to any one of claims 1 to 13.

34. A computer-readable storage medium storing a computer program, wherein, The computer program causes the computer to perform the method according to any one of claims 1 to 13.

35. A computer program product comprising a computer program, wherein, The computer program causes the computer to perform the method according to any one of claims 1 to 13.

36. A computer program, wherein, The computer program causes the computer to perform the method according to any one of claims 1 to 13.

Images