TECHNOLOGIES FOR MACHINE LEARNING ENTITY TRAINING AND TESTING FOR FIFTH GENERATION SYSTEMS

Abstract

Embodiments herein relate to the training and / or testing of a machine learning (ML) model. In particular, the training and / or testing of the ML model can be based on communication between a management services (MnS) producer and an MnS consumer. The communication can be based on or related to one or more managed object instances (MOIs). Further embodiments may be described and / or claimed.

Description

CROSS-REFERENCE TO RELATED REGISTRATION

[0001] The present application claims priority over U.S. Preliminary Patent Application No. 63 / 501,063, filed on May 9, 2023, and U.S. Preliminary Patent Application No. 63 / 501,081, filed on May 9, 2023. BACKGROUND

[0002] Artificial intelligence / machine learning (AI / ML) can be used in fifth-generation (5G) systems (5GS), including 5G Core (5GC), next-generation (NG) radio access networks (RANs), and management systems (e.g., MDA). ML training aspects were specified in 3GPP TS 28.105 v17.3.0 (2023-03-30) (“[TS28105]”), which supports consumer-requested training and producer-initiated training. However, it is unclear whether and how initial ML training, ML retraining, and the joint training of a group of ML entities are supported. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The embodiments are easily understood with reference to the following detailed description in conjunction with the accompanying drawings. For the sake of clarity, identical reference numerals denote identical structural elements. The embodiments are illustrated in the figures of the accompanying drawings as examples and are not exhaustive. Fig. Figure 1 illustrates an example of ML training requested by a Management Training (MLT) Administration Service (MnS) consumer, according to different embodiments. Fig. Figure 2 illustrates an example regarding the propagation of erroneous information in relation to ML in a wireless network according to different embodiments. Fig. Figure 3 illustrates an exemplary NRM fragment that can be used for ML training, according to different embodiments. Fig.Figure 4 illustrates an alternative exemplary NRM fragment that can be used for ML training, according to different embodiments. Fig. Figure 5 illustrates an exemplary inheritance hierarchy that can be used for ML training-related NRMs, according to various embodiments. Fig. Figure 6 illustrates an exemplary network resource model (NRM) fragment that can be used for ML testing, according to various embodiments. Fig. Figure 7 illustrates an exemplary inheritance hierarchy that can be used for ML test-related NRMs, according to various embodiments. Fig. Figure 8 illustrates a network according to different embodiments. Fig. Figure 9 schematically illustrates a wireless network 900 according to various embodiments. Fig.Figure 10 is a block diagram illustrating components according to some exemplary embodiments that are capable of reading instructions from a machine-readable or computer-readable medium (e.g., a non-volatile machine-readable storage medium) and performing one or more of any of the methodologies discussed herein. Fig. Figure 11 illustrates a network according to different embodiments. Fig. Figure 12 illustrates a simplified block diagram of artificial (AI)-assisted communication between a UE and a RAN according to different embodiments. Fig. Section 13 presents an exemplary procedure for implementing the various embodiments discussed herein. Fig. Section 14 presents another exemplary procedure for implementing the various embodiments discussed herein. DETAILED DESCRIPTION 1. EDGE ENABLER INSTANTIZATION ASPECTS

[0004] This disclosure relates, among other things, to AI / ML management capabilities and services for 5GS, where AI / ML is used, including management and orchestration (e.g., Management Data Analysis (MDA); see, for example, Third Generation Partnership Project (3GPP) Technical Specification (TS) 28.104 v17.3.0 (2023-03-30) (“[TS28104]”)) and 5G networks (e.g., NWDAF; see, for example, [TS23288]). This disclosure also describes various aspects of the functionality and service framework for AI / ML management.

[0005] The legacy ML entity is defined as a data type for an attribute that may not be visible or accessible to the consumer as the managed object instance (MOI). This disclosure provides improvements to ML training (e.g., as defined in [TS28105]) to support initial ML retraining, ML retraining, and co-training of a group of ML entities.

[0006] In addition, during the ML entity training phase, the machine learning entity may need to be tested after training and validation to evaluate its performance when performing inference using the test data. If the test result meets expectations, the ML entity can proceed to the next phase (such as emulation, loading, inference, etc.); otherwise, it may need to be retrained.

[0007] Testing can be performed on, at, or through the ML test function, which may be the same as the ML training function or a separate function. Testing of the ML entity was investigated in 3GPP TR 28.908, and the normative solutions are to be specified in [TS28105]. The solutions for testing ML entities investigated in 3GPP TR 28.908 do not contain sufficient detail for testing ML models and / or to be the standard solutions.

[0008] The following discussion provides several examples of Information Object Class (IOC) names and attribute names; however, the IOCs and attributes may have alternative names to those provided below. 1.1. ML-TRAINING APPLICATION CASES AND REQUIREMENTS 1.1.1. DESCRIPTION

[0009] In an operational environment, before the ML entity is used to perform inference, the ML model associated with the ML entity must be trained (e.g., by an ML training function, which may be a separate entity or an external entity to the AI / ML inference function). The ML training can be the initial training of an ML entity or the retraining of an existing ML entity. In this disclosure, the term "ML entity training" can refer to ML model training associated with an ML entity.

[0010] The ML entity is trained by the ML Training (MLT) MnS producer, and the training can be triggered by one or more requests from one or more MLT MnS consumers or initiated by the MLT MnS producer (e.g., as a result of a model capability assessment). 1.1.2. USE CASE 1.1.2.1. CONSUMER-REQUIRED ML TRAINING

[0011] The machine learning (ML) training capabilities are provided by an MLT-MnS generator to one or more consumers. See, for example, Fig. 1. As in Fig.As shown in Figure 1, machine learning (ML) training can be triggered by a request from one or more MLT-MnS consumers. The consumer could be, for example, a network function, an administrative function, an operator, or another functional differentiation to trigger ML training, where the MLT-MnS consumer requests the MLT-MnS producer to train the ML model. In the ML training request, the consumer should specify the inference type, which indicates the function or purpose of the ML entity (e.g., CoverageProblemAnalysis). The MLT-MnS producer can then perform the training according to the designated inference type. The consumer can provide the one or more data sources containing the training data, which are considered input candidates for training. To obtain valid training results, consumers can also specify their model performance requirements (e.g., accuracy, etc.).) designate in the training requirement.

[0012] The MLT-MnS producer provides the consumer with a response indicating whether the request was accepted. If the request is accepted, the MLT-MnS producer decides when to start ML training, taking into account the request(s) from the consumer(s). Once the training is decided, the producer can perform one or more of the following: - Selecting training data, taking into account the consumer-provided candidate training data. Since the training data directly influences the algorithm and the performance of the trained ML entity, the MLT-MnS creator can examine the consumer-provided training data and decide to select none, some, or all of it. Additionally, the MLT-MnS creator can select some other training data that is available. - Training the ML entity using the selected training data; and - Providing the training results (including the identifier of the initially trained ML entity or the version number of the newly trained ML entity, the training performance, etc.) to the one or more MLT-MnS consumers. 1.1.2.2. MANUFACTURER-INITIATED ML TRAINING

[0013] Alternatively, in some embodiments, ML training can be initiated by the MLT-MnS generator, for example as a result of a performance evaluation of the ML model based on feedback or new training data received from the consumer, or when new non-consumer training data describing the new network state / network events becomes available.

[0014] When the MLT-MnS generator decides to start ML training, the generator can perform one or more of the following: - Selecting the training data; - Training the ML entity using the selected training data; and - Providing the training results (including the identifier of the initially trained ML entity or the version number of the newly trained ML entity, the training performance, etc.) to the one or more MLT-MnS consumers who are subscribed to receive the ML training results. 1.1.2.3. ML MODEL AND ML ENTITY SELECTION

[0015] For a given machine learning-based use case, different entities applying a particular ML model or AI / ML inference function may have different inference requirements and capabilities. For example, a consumer with specific responsibilities might want an AI / ML inference function powered by an ML model or entity trained for a central urban business district where mobile users travel at speeds not exceeding 30 km / h. Conversely, another consumer for the same use case might be supporting a rural environment and therefore want an ML model and AI / ML inference function suited to that type of environment. The different consumers need to be aware of the available versions of ML entities with their variations of trained ML models or entities, and to ensure that the appropriate ML model or entity is used.select the one that is suitable for their respective conditions.

[0016] Furthermore, there may be no guarantee that the available machine learning models / entities have been trained according to the characteristics consumers expect. Therefore, consumers need to know the conditions for which the machine learning models or entities were trained in order to select the models that best fit their specific conditions and needs.

[0017] The trained models can differ in complexity and performance. For example, a generic, comprehensive, and complex model might have been trained in a cloud-like environment, but such a model cannot be used in gNB, and instead, a less complex model trained as a derivative of this generic model might be a better candidate. Furthermore, several less complex models with varying levels of complexity and performance could be trained, which would then allow different relevant models to be delivered depending on operating conditions and performance requirements for different network functions.The network functions must be aware of the available alternative models and request and replace them interactively when needed, depending on the observed inference-related limitations and performance requirements. 1.1.2.4. MANAGING ML TRAINING PROCESSES

[0018] This machine learning capability refers to means for managing and controlling ML model / entity training processes.

[0019] To achieve the desired results in any machine learning-relevant use case, the ML model used for such analysis and decision-making must be trained with suitable data. This training can be performed in a managed function or in a management function.

[0020] In both cases, the network (or its operations, administration, and maintenance (OAM) system) may not only need to possess the necessary training capabilities but also the resources to manage the training of the machine learning models / entities. Consumers must be able to interact with the training process, for example, to pause or restart it, and must also manage and control the requirements related to such a training process. 1.1.2.5. HANDLING ERRORS IN DATA AND ML DECISIONS

[0021] Traditionally, the ML models / entities (e.g., ML-Entity1 and ML-Entity2) are in Fig. 2) trained on good-quality data, i.e., data that was collected correctly and reflects the real network state to represent the expected context in which the ML entity is intended to operate. Good-quality data is error-free, such as: - Inaccurate measurements with additional noise (such as reference signal reception power (RSRP), signal-to-interference-plus-noise ratio (SINR), or quality-of-experience (QoE) estimates). - Missing values ​​or entire missing records, e.g. due to communication link errors. - Recordings that are communicated with a significant delay (in the case of online measurements).

[0022] Without errors, a machine learning entity can depend on just a few precise inputs and may not need to exploit the redundancy present in the training data. However, during inference, the machine learning entity will very likely encounter these inconsistencies. When this happens, the machine learning entity may exhibit a high error rate in its inference outputs, even if redundant and uncorrupted data from other sources is available.

[0023] Therefore, the system may need to account for errors and inconsistencies in the input data, and consumers should be able to handle decisions made based on such erroneous and inconsistent data. The system should (1) provide functions to perform training in a way that prepares the ML entities to deal with the errors in the training data, i.e., to identify the errors in the data during training; and / or (2) allow MLT-MnS consumers to be aware of the possibility of erroneous input data being used by the ML entity. 1.1.1.1. JOINT ML ENTITY TRAINING

[0024] An AI / ML inference function can use one or more ML entities to perform the inference(s). When multiple ML entities are used, these entities can work together in a coordinated manner, such as in a sequence or even a more complex structure. In this case, any change in the performance of one ML entity can affect another and consequently impact the overall performance of the entire AI / ML inference function.

[0025] There are different ways in which a group of machine learning (ML) entities can be coordinated. One example is when the output of one ML entity can be used as input to another, forming a sequence of interconnected ML entities. Another example is when multiple ML entities provide output in parallel (either the same output type, where outputs can be merged (e.g., using weights), or their outputs are needed in parallel as input to another ML entity). The group of ML entities must be used in a coordinated manner to support an AI / ML inference function. In various implementations, the way the ML entities are grouped and the relationships between them within the group are named can be vendor-specific.

[0026] Therefore, it may be desirable for these coordinated ML entities to be trained or retrained together, so that the group of these ML entities can complete a more complex task together with better performance.

[0027] The joint ML entity training can be initiated by the MnS producer or the MnS consumer, with the grouping of ML entities being shared between the MnS producer and the MnS consumer. 1.1.1.2. CONSUMER-REQUESTED ML ENTITY TESTS

[0028] After the ML entity training is complete, and if the trained ML entity's performance meets the expectations for both the training and validation data, the ML entity is made available to the consumer(s) via the ML training report. Before the ML entity is applied to the target AI / ML inference function or emulation environment, the ML test MnS generator (which may be the same as or different from the ML training MnS generator) must allow the consumer to evaluate the ML entity's performance through the ML testing process using the test data provided by the consumer. The test data can have the same pattern as the input portion of the training data. The consumer (e.g., an operator) can also stop, pause, or resume the ML entity testing process as needed.

[0029] Once testing is complete, the MnS producer can report the performance of the ML entity on the test data to the MnS consumer.

[0030] If the test performance is not acceptable or does not meet the predefined requirements, the consumer can request the ML training generator to retrain the ML entity with specific training data and / or performance requirements. 1.1.1.3. CONTROL OF PRODUCER-INITIATED ML ENTITY TESTS

[0031] The ML entity tests can also be initiated by the MnS creator after the ML entity has been trained and validated. A consumer (e.g., an operator) may need to control (e.g., stop, pause, resume, etc.) and manage the test process initiated by the MnS creator. For example, the operator may want to define the guidelines (e.g., allowed time window, maximum number of tests, etc.) for testing a given ML entity that can be executed. 1.1.1.4. REQUIREMENTS FOR ML ENTITY TESTS

[0032] Table 1 provides at least some exemplary requirements that may relate to ML entity testing, as follows: Table 1 Requirement designation Description Related Use Cases REQ-ML_TEST-1 The ML test MnS generator should have the capability to allow an authorized consumer to request the testing of a specified ML entity using continuous data streams provided by the consumer. Consumer-requested ML entity tests (see, for example, section 1.1.1.2 above) REQ-ML_TEST-2 The ML test MnS generator should have the ability to initiate an ML entity test and allow the MnS consumer to define the policy for the tests. Producer-initiated ML entity tests (see, for example, section 1.1.1.3 above) REQ-ML_TEST-3 The ML test MnS generator should have the capability to perform the ML entity tests and allow the authorized consumer to query and be informed about the status of the ML entity test process. Consumer-requested ML entity tests (see, for example, section 1.1.1.2 above) and producer-initiated ML entity tests (see, for example, section 1.1.1.3 above) REQ-ML_TEST-4 The ML test MnS generator should have the capability to allow the authorized MnS consumer to stop, suspend, and resume the ML entity test process. Consumer-requested ML entity tests (see, for example, section 1.1.1.2 above) and producer-initiated ML entity tests (see, for example, section 1.1.1.3 above) REQ-ML_TEST-5 The ML test MnS generator should have the ability to report the performance of the ML entity when it performs inference on the test data. Consumer-requested ML entity tests (see, for example, section 1.1.1.2 above) and producer-initiated ML entity tests (see, for example, section 1.1.1.3 above) 1.2. SOLUTIONS 1.2.1. RELATIONSHIPS (TRAINING)

[0033] This clause concerns the set of classes (e.g., IOCs) that encapsulate the information relevant for machine learning model training. Details regarding Unified Modeling Language (UML) semantics can be found, for example, in 3GPP TS 32.156 v18.1.0 (2023-03-30) (“[TS32156]”). Examples of IOCs are provided, for example, in Fig. 3, which represents an exemplary NRM fragment in relation to ML training. 1.2.2. HEREDITY (TRAINING)

[0034] This clause refers to an IOC inheritance hierarchy for ML training-related NRMs. An example of such a hierarchy is, for example, in Fig. 4 to be seen. 1.2.3. RELATIONSHIPS (TESTING)

[0035] This clause concerns the set of classes (e.g., IOCs) that encapsulate the information relevant for ML model testing. Examples of IOCs can be found, for example, in the Fig. 5 and Fig.Figure 6 shows exemplary NRM fragments in relation to ML tests. 1.2.4. HEREDITY (TESTING)

[0036] This clause refers to an IOC inheritance hierarchy for ML test-related NRMs. An example of such a hierarchy is, for example, in Fig. 7 can be seen. 1.2.5. MLTRAININGFUNCTION

[0037] The IOC-ML TrainingFunction represents the entity that performs the machine learning training. The entity represented by the ML TrainingFunction MOI supports the training of one or more machine learning entities and the joint training of one or more groups of machine learning entities. Example attributes of the ML TrainingFunction MOI can be as shown in Table 2. Table 2 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable mLEntityList (ML entity list) M T F F F 1.2.6. MLTRAININGREQUEST (ML-TRAININGSANFORDERUNG)

[0038] The IOC-MLTrainingRequest represents the machine learning model training request generated by the machine learning training (MnS) consumer. The MLTrainingRequest MOI is contained within an MLTrainingFunction MOI. Each MLTrainingRequest is associated with at least one machine learning entity.

[0039] The ML TrainingRequest can have a source to identify its origin, which can be used to prioritize training resources for different sources. These sources can be, for example, network functions, operator roles, or other functional differentiations.

[0040] Each MLTrainingRequest can specify the expectedRunTimeContext, which describes the specific conditions for which the ML entity should be trained.

[0041] If the request is accepted, the ML training MnS generator decides when to start the ML training. Once the MnS generator decides to start the training based on the request, it instantiates one or more MLTrainingProcess MOI(s) (ML training processes) responsible for performing one or more of the following: - collects (more) data for training if the training data is unavailable or the data is available but insufficient for training; - prepares and selects the necessary training data, taking into account the candidate training data provided by the consumer's request. The ML training MnS generator can examine the candidate training data provided by the consumer and select none, some, or all of it for training. Additionally, the ML training MnS generator can select some other training data that is available to meet the consumer's requirements for ML entity training; and / or - trains the ML entity using the selected and prepared training data.

[0042] The ML TrainingRequest can have a requestStatus field to represent the status of the specific ML TrainingRequest: - The attribute values ​​are “NOT_STARTED”, “TRAINING _IN_PROGRESS”, “SUSPENDED”, “FINISHED”, and “CANCEL”. - When the value changes to “TRAINING_IN_PROGRESS”, the ML Training MnS creator instantiates one or more MLTrainingProcess MOI(s) representing the training process(s) being performed according to the request and notifies the MLT MnS consumer(s) who has subscribed to the notification.

[0043] When the entire training process associated with this requirement is completed, the value changes to "FINISHED".

[0044] Attributes of ML TrainingRequest can be as shown in Table 3. Table 3 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable inferenceType CM T F F T mLEntityld (ML entity ID) M T T F T candidateTrainingDataSource (Candidate Training Data Source) O T T F T trainingDataQualityScore (Training Data Quality Score) O T T F T trainingRequestSource(TrainingRequestEvaluation) M T T F T requestStatus (Request Status) M T F F T expectedRuntimeContext O T T F T performance requirements M T T F T cancelRequest (cancellation request) O T T F T suspendRequest O T T F T Attribute relating to the role mLEntityToTrainRef(Reference of the ML entity to be trained) CM T F F T mlEntityGroupToTrainRef (Reference of the ML entity group to be trained) M T T F T 1.2.7. MLTRAININGREPORT (ML-TRAININGSBERICHT)

[0045] The IOC-ML TrainingReport represents the machine learning (ML) model training report provided by the training MnS generator. The MLTrainingReport MOI is contained within an ML TrainingFunction MOI. Attributes of the ML TrainingReport can be as shown in Table 4, and constraints on the attributes from Table 4 can be as shown in Table 5. Table 4 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable mLEntityId (ML entity ID) M T F F T areConsumerTrainingDataUsed (are consumer training data used) M T F F T Used Consumer Training Data CM T F F T confidence indication O T F F T Model Performance Training M T F F T Are New Training Data Used? M T F F T Attribute relating to the role trainingRequest Ref (Training Request Reference) CM T F F T trainingProcessRef (Training Process Reference) M T F F T lastTrainingRef (Reference of the last training session) CM T F F T mLEnityGeneratedRef(ML-Entity-Generated Reference) M T F F T mLEnityGroupGeneratedRef (ML entity group n-generated reference) M T F F T Table 5 Designation definition usedConsumerTrainingDataSupport Qualifier (Support Qualifier for usedConsumerTrainingData) Condition: The value of the attribute areConsumerTrainingDataUsed is ALL or PARTILY. trainingRequestRef SupportQualifier (Support Qualifier for Training Request Reference) Condition: The ML TrainingReport-MOI represents the report for the ML model training that was requested by the MnS consumer (via MLTrainingRequest-MOI). lastTrainingRef SupportQualifier Condition: The MLTrainingReport-MOI represents the report for the ML model training that was not initial training (i.e., the model was previously trained). 1.2.8. MLTRAININGPROCESS

[0046] The IOC-MLTrainingProcess represents the machine learning (ML) training process. An ML TrainingProcess MOI can be instantiated for each ML TrainingRequest MOI or a set of ML TrainingRequest MOIs. For each ML entity being trained, an ML TrainingProcess is instantiated; that is, an ML TrainingProcess is associated with exactly one ML entity. The ML TrainingProcess can be associated with one or more ML TrainingRequest MOIs.

[0047] The ML TrainingProcess does not need to correspond to a specific MLTrainingRequest; that is, an ML TrainingRequest does not have to be associated with a specific ML TrainingProcess. The ML TrainingProcess can be managed separately from the ML TrainingRequest MOI. For example, the ML TrainingRequest MOI might come from consumers that are network functions, while the operator wants to manage the ML TrainingProcess, which is instantiated according to the requirements. Thus, the ML TrainingProcess can be associated with one or more ML TrainingRequest MOIs.

[0048] Each MLTrainingProcess instance must be managed differently from its associated ML entity, even though the MLTrainingProcess can only be associated with one ML entity. For example, the MLTrainingProcess can be triggered to begin with a specific version of the ML entity, and multiple MLTrainingProcess instances can be triggered for different versions of the ML entity. In both cases, the MLTrainingProcess instances are still associated with the same ML entity but are managed separately from the ML entity.

[0049] Each MLTrainingProcess has a priority that can be used to prioritize the execution of different MLTrainingProcess instances. By default, the priority of the ML TrainingProcess can be directly related to the priority of the MLTrainingRequest for which the MLTrainingProcess is instantiated.

[0050] Each ML TrainingProcess can have one or more termination conditions that are used to define the points at which the ML TrainingProcess can end.

[0051] The “progressStatus” attribute represents the status of the ML model training and contains information that the ML training MnS consumer can use to monitor progress and results. The data type of this attribute is “ProcessMonitor” (see 3GPP TS 28.622

[12] ). The following specializations are provided for this data type for the ML training process: - The "Status" attribute values ​​are "RUNNING", "CANCELLING", "SUSPENDED", "FINISHED", and "CANCELLED". The other values ​​are not used. - The "Timer" attribute is not used. - If the "Status" is "RUNNING", the "progressStateInfo" attribute should indicate one of the following states: "COLLECTING_DATA", "PREPARING_TRAINING_DATA", "TRAINING". No specifications are provided for the attribute "resultStateInfo". However, vendor-specific information may be provided.

[0052] When the training is completed with a status of "FINISHED", the MLT-MnS generator provides the training report to the MLT-MnS consumer by generating an MLTrainingReport MOI.

[0053] Attributes of the MLTrainingProcess can be as shown in Table 6, and constraints relating to the MLTrainingProcess can be as shown in Table 7. Table 6 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable mL TrainingProcessId (ML training process ID) M T T F T priority M T T F T Termination Conditions M T T F T progress Status M T F F T cancel process O T T F T suspend process O T T F T Attribute that relates to the role trainingRequestRef (Training Request Reference) CM T F F T mlEntityBeingTrainedRef (Reference of the trained ML entity) CM T F F T trainingReportRef (Training Report Reference) M T F F T Table 7 Designation definition trainingRequestRefSupport Qualifier (Support Qualifier for Training Request Reference) Condition: The ML TrainingReport-MOI represents the report for the ML model training that was requested by the training MnS consumer (via MLTrainingRequest-MOI). 1.2.9. MLENTITY (ML-ENTITÄT)

[0054] This IOC represents the ML entity. The algorithm of the ML model or the ML entity is not to be standardized. The ML entity can contain three types of contexts: TrainingContext, which is the context in which the ML entity was trained; ExpectedRunTimeContext, which is the context in which the ML entity is expected to be used; and / or RunTimeContext, which is the context in which the ML entity is used. Attributes of the ML entity can be as shown in Table 8. Table 8 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable mLEntityId(ML-Entity-ID) M T F F T inferenceType M T F F T mLEntityVersion (ML entity version) M T F F T expectedRunTimeContext (expected runtime context) O T T F T trainingContext(training context) CM T F F T runTimeContext (Runtime context) O T F F T

[0055] In some examples, the `trainingContext` is a support qualifier. In other examples, the `trainingContext` represents the status and conditions related to the training and should be added when the training is complete. 1.2.10. MLENTITYGROUP (ML-ENTITÄTSGRUPPE)

[0056] This IOC represents the group of ML entities that can be trained together and perform inference in a coordinated manner. How the ML entities coordinate within the group is described by the relationships between them. The grouping of the ML entities and their relationships are determined by the MnS producer and cannot be changed by the MnS consumer. This IOC is generated by the MnS producer and cannot be modified by the MnS consumer. 1.2.11. MLENTITIESRELATIONPERLEVEL < <datatype>> (ML entity relationships per level < <datentyp>>)

[0057] This data type represents the relationships between the ML entities for each level of an ML entity group. The relationship between the ML entities within each level can be sequential or parallel. The output of one level of the ML entity group is used as input for the next (higher) level. The level is specified by a number, starting with one, with a larger number indicating a higher level. Attributes relating to MLEntitiesRelationPerLevel can be as shown in Table 9. Table 9 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable levelNumber (Level Number) M T F F F mLEntitiesrelation (ML entity relationship) M T F F F Attribute relating to the role mLEntityRefList(Reference list of ML entities) M T F F T 1.2.12. ATTRIBUTE PROPERTIES

[0058] Properties relating to one or more of the attributes listed or described above (e.g., training-related attributes and / or any other attribute described herein) may be as shown in Table 10. Table 10 Attribute label Documentation and permissible values Characteristics mLEntityId(ML-Entity-ID) It identifies the ML entity. It is unique in every MnS generator. AllowedValues: n / a Type: StringMultiplicity: 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: True candidateTrainingDataSource (Candidate Training Data Source) It provides the address(es) of the candidate training data source provided by the MnS consumer. The detailed training data format is vendor-specific. AllowedValues: n / a Type: String multiplicity: *isOrdered: False isUnique: True defaultValue: None isNullable: True inferenceType (Inference type) It specifies the type of inference supported by the ML model. allowedValues: the values ​​of the MDA type (see 3GPP TS 28.104 [2]), analysis ID(s) of the NWDAF (see 3GPP TS23.288 [3]), types of inference for RAN intelligence and Type: String multiplicity: 1isOrdered: kAisUnique: kAdefaultValue: None vendor-specific extensions. isNullable:True areConsumerTrainingDataUsed (are consumer training data used) It indicates whether the consumer-provided training data was used for ML model training. AllowedValues: ALL, PARTIALLY, NONE. Type: Enummultiplicity: 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: True usedConsumerTrainingData(usedConsumerTrainingData) It provides the address(es) where lists of consumer-provided training data used for ML model training are located. allowedValues: n / a Type: String multiplicity: *isOrdered: False isUnique: True defaultValue: None isNullable: True trainingRequestRef (Training Request Reference) These are the DN(s) of the associated MLTrainingRequest MOI(s). allowedValues: DN. Type: DN (see TS32.156

[13] )multiplicity: *isOrdered: FalseisUnique: True defaultValue(Default value): NoneNullable(isNullable): True trainingProcessRef (Training Process Reference) These are the DN(s) of the associated MLTrainingProcess MOI(s) that generated the MLTrainingReporter. allowedValues ​​(permitted values): DN. Type: DN (see TS32.156

[13] ) multiplicity: 1 isOrdered: k isUnique: k DefaultValue: None isNullable: True trainingReportRef (Training Report Reference) It is the DN of the MLTrainingReport-MOI, which represents the reports of the ML training. allowedValues: DN. Type: DN (see TS32.156

[13] ) multiplicity: 1 isOrdered: k isUnique: k DefaultValue: None isNullable: True lastTrainingRef (Reference of the last training session) It is the DN of the MLTrainingReport-MOI, which represents the reports for the last training of the ML model. Type: DN (see 3GPPTS 32.156

[13] ) multiplicity: 1 Attribute label Documentation and permissible values Characteristics allowedValues ​​(permittedValues): DN. isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: True confidenceIndication(confidence indication) It indicates the confidence (in percent) that the ML model would perform inference on data with the same distribution as training data. allowedValues: { 0..100}. Type: Integer multiplicity: 1isOrdered: kAisUnique: kDefaultValue: NoneisNullable: False trainingRequestSource(TrainingRequestEvaluation) It describes the entity that requested the MLTrainingRequest MOI to be instantiated. Type: Integer multiplicity: 1isOrdered: kAisUnique: kDefaultValue: NoneisNullable: False Request status It describes the status of a specific ML- Type: Enummultiplicity ) Training requirement.allowedValues: NOT_STARTED, TRAINING_IN_PROGRESS, CANCELLING, SUSPENDED, FINISHED and CANCELLED. (Multiplicity): 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: False mL TrainingProcessId (ML training process ID) It identifies the training process. It is unique in every instantiated process in the MnS generator. allowedValues: n / a Type: StringMultiplicity: 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: True priority It specifies the priority of the training process. The priority can be used by the ML training to schedule the training processes. A lower value indicates a higher priority. AllowedValues: { 0..65535}. Type: DN (see TS32.156

[13] )multiplicity: 1isOrdered: kAisUnique: kAdefaultValue: 0isNullable: Incorrect Termination Conditions It specifies the conditions that the ML training MnS generator must consider to terminate a specific training process. allowedValues: MODELUPDATED_IN_INFERENCE_FUNCTION (Model updated in inference function), INFERENCEFUNCTION_TERMINATED (Inference function terminated), INFERENCEFUNCTION_UPGRADED (Inference function upgraded), INFERENCE_CONTEXT_CHANGED (Inference context changed). Type: StringMultiplicity: 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: False progressStatus It indicates the status of the ML training process. allowedValues ​​(allowed values): n / a Type: ProcessMonitor (see TS 28.622

[12] ) multiplicity: 1 isOrdered: n / a isUnique: n / a DefaultValue: None isNullable: False mLEntityVer It gives the version number. Type: String sion (ML entity version) the ML entity an.allowedValues ​​(allowable values): n / a multiplicity: 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: False performance requirements It indicates the expected performance of a trained ML entity when run on the training data. allowedValues: n / a Type: ModelPerformance Multiplicity: * isOrdered: k A isUnique: k DefaultValue: None IsNullable: True modelperformanceTraining(model performance training) It indicates the performance rating of the ML entity when performed on the training data. allowedValues: n / a Type: Model Performance; Multiplicity: * is Ordered: n / a; is Unique: n / a defaultValue(Default value): NoneNullable(isNullable): False mL TrainingProcess.progressStatus.progressStateInfo(ML TrainingProcess.ProgressStatus.ProgressStateInfo) It provides the following specialization for the "progressStateInfo" attribute of the "ProcessMonitor" data type for the "ML TrainingProcess". When ML training is in progress and "mLTrainingProcess.progressStatus.status" is equal to "RUNNING", it provides more detailed progress information. Valid values ​​for "mLTrainingProcess.progressStatus.status" = "RUNNING": - COLLECTING_DATA (Collecting data) - PREPARING_TRAINING_DATA (Preparing training data) - TRAINING (Training) The valid values ​​for "mLTrainingProcess.progressStatus.status" = "CANCELLED" are Type: String multiplicity: 0..1isOrdered: noAisUnique: noAdefaultValue: NoneisNullable: False Attribute label Documentation and permissible values Characteristics Vendor-specific. inferenceOutputName (Inference output name) It specifies the name of an inference output of an ML entity. allowedValues: the name of the MDA output IEs (see 3GPP TS 28.104 [2]), the name of the analysis output IEs of NWDAF (see TS 23.288 [3]), the name(s) of the RAN intelligence inference output IEs and the vendor-specific extensions. Type: StringMultiplicity: 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: False performanceMetric It specifies the performance metric used to evaluate the performance of an ML entity, e.g., "accuracy," "precision," "F1 score," etc. AllowedValues: n / a Type: StringMultiplicity: 1isOrdered: kAisUnique: TrueDefaultValue: NoneIsNullable: False performanceScore (performance rating) It indicates the performance rating (in units of percent) of an ML entity when inference is performed on a specific data set (Grade)(Note). Type: Real multiplicity: 1isOrdered: kAisUnique: kAdefaultValue each Performance metrics can vary depending on the type of model used for different types of machine learning (ML) models. For example, the metric for numerical prediction might be accuracy; for classification, the metric might be a combination of precision and recall, such as the "F1 score". Allowed values: { 0..100}. (Default value): NoneNullable (isNullable): False cancelRequest (cancellation request) It indicates whether the ML training MnS consumer cancels the ML training request. Setting this attribute to "TRUE" cancels the ML training request. Cancellation is possible if the requestStatus is "NOT_STARTED", "TRAINING_IN_PROGRESS", or "SUSPENDED". Setting the attribute to "FALSE" has no observable result. The default value is "FALSE". AllowedValues: TRUE, FALSE. Type: Boolean multiplicity: 0..1 isOrdered: kAisUnique: kDefaultValue: FALSE isNullable: FALSE suspendRequest It indicates whether the ML training MnS consumer suspends the ML training request. Setting this attribute to "TRUE" suspends the ML training request. Suspension is possible if the requestStatus is not "FINISHED". Setting the attribute to "FALSE" has no observable result. The default value is "FALSE". AllowedValues: TRUE, FALSE. Type: Boolean multiplicity: 0..1 isOrdered: kAisUnique: kDefaultValue: FALSE isNullable: FALSE cancelProcess (cancellation process) It indicates whether the ML training MnS consumer aborts the ML training process. Setting this attribute to "TRUE" cancels the ML training request. Abort is possible if the "mLTrainingProcess.progressStatus.status" is not in the "FINISHED" state. Setting the attribute to "FALSE" has no observable result. The default value is set to "FALSE". AllowedValues: TRUE, FALSE. Type: Boolean multiplicity: 0..1 isOrdered: kAisUnique: kDefaultValue: FALSE isNullable: FALSE suspend process It indicates whether the ML training MnS consumer suspends the ML training process. Setting this attribute to "TRUE" suspends the ML training request. Suspension is possible if the "mLTrainingProcess.progressStatus.status" is not in the state "FINISHED", "CANCELLING", or "CANCELLED". Setting the attribute to "FALSE" has no observable result. The default value is set to "FALSE". AllowedValues: TRUE, FALSE. Type: Boolean multiplicity: 0..1 isOrdered: kAisUnique: kDefaultValue: FALSE isNullable: FALSE inference EntityRef (Reference of Inferentiality) It describes the target entities that will use the ML entity for inference. Type: DN (see 3GPPTS 32.156

[13] ) multiplicity: *isOrdered: False isUnique: True defaultValue: None isNullable: True dataProviderRef (reference of the It describes the entities that have provided data or Type: DN (see 3GPPTS 32.156

[13] ) data provider) should provide the resources that the ML entity needs, for example, for training or inference. multiplicity: *isOrdered: FalseisUnique: TruedefaultValue: NoneisNullable: True areNewTrainingDataUsed (are new training data used) It indicates whether the other new training data was used for ML model training. allowedValues: TRUE, FALSE. Type: Boolean multiplicity: 1isOrdered: kAisUnique: kDefaultValue: NoneisNullable: False trainingDataQualityScore (Training Data Quality Score) It provides a numerical value representing the reliability / quality of a given observation and measurement type. The lowest value indicates the lowest level of data reliability, meaning the data is unusable. Type: Real multiplicity: 0..1 isOrdered: no AisUnique: no Default value: None isNullable: False Values): { 0..100}. decisionConfidenceScore (decision confidence score) It is the numerical value that represents the reliability / quality of a given decision generated by the AI / ML inference function. The lowest value indicates the lowest level of decision reliability, meaning the data is unusable. Allowed values: { 0..100}. Type: Real multiplicity: 0..1 isOrdered: no AisUnique: no Default value: None isNullable: False expectedRuntimeContext This describes the context in which an ML entity is expected to be applied, and / or the RunTimeContext, which is the context in which the ML model or ML entity is applied. `allowedValues`: n / a Type: MLContext (ML context) multiplicity: 0..1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: False trainingContext(training context) This specifies the context in which the ML entity was trained. allowedValues: n / a Type: MLContext (ML context) multiplicity: 1isOrdered: kAisUnique (isUnique): kAdefaultValue(Default value): NoneIsNullable (isNullable): False runTimeContext (Runtime context) This specifies the context in which the ML model or ML entity is applied. `allowedValues`: n / a Type: MLContext (ML context) multiplicity: 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: False mLEntityToTrainRef (reference of the ML entity to be trained) It identifies the DN of the ML entity that is requested for training. allowedValues: DN Type: DN (see 3GPPTS 32.156

[13] ) multiplicity: 1 isOrdered: False isUnique: True defaultValue: None isNullable: True mLEnityGeneratedRef (ML entity-generated It identifies the DN of the ML entity that is generated by the ML training. Type: DN (see 3GPPTS 32.156

[13] )multiplicity Reference) allowedValues ​​(permitted values): DN (Multiplicity): 1isOrdered: FalseisUnique: TruedefaultValue: NoneisNullable: True mLEntityToTrainRef (reference of the ML entity to be trained) It identifies the DN of the ML entity that is requested for training. allowedValues: DN Type: DN (see 3GPPTS 32.156

[13] ) multiplicity: 1 isOrdered: False isUnique: True defaultValue: None isNullable: True mLEnityGeneratedRef (ML entity-generated reference) It identifies the DN of the ML entity generated by ML training. allowedValues: DN Type: DN (see 3GPPTS 32.156

[13] ) multiplicity: 1 isOrdered: False isUnique: True defaultValue: None isNullable: True mlEntityGroupToTrainRef (Reference of the ML entity group to be trained) It identifies the DN of the MlEntityGroup that is requested for training. allowedValues: DN Type: DN (see 3GPPTS 32.156

[13] ) multiplicity: 1 isOrdered: False isUnique: True defaultValue: None isNullable: True mlEntityBeingTrainedRef(Reference of the trained ML entity) It identifies the DN of the ML entity being trained through an ML training process. allowedValues: DN Type: DN (see 3GPPTS 32.156

[13] ) multiplicity: 1 isOrdered: False isUnique: True defaultValue: None isNullable: True mLEnityGroupGeneratedRef(ML-Entity Group Generated Reference) It identifies the DN of the MlEntityGroup, which is generated by ML training. allowedValues: DN Type: DN (see 3GPPTS 32.156

[13] ) multiplicity: 1 isOrdered: False isUnique (isUnique): TrueDefaultValue(DefaultValue): NoneIsNullable (isNullable): True mlEntityGroupProfile (ML Entity Group Profile 1) It identifies the ML entity group that was initially trained or retrained. allowedValues: n / a Type: ML EntitiesRelationPerLevel (ML entity relationship per level) multiplicity: *isOrdered: False isUnique: True defaultValue: None isNullable: False levelNumber (Level Number) It specifies the level number of the ML entities within an ML entity group. allowedValues: allowedValues: { 1..100}. Type: Integer multiplicity: 1isOrdered: FalseisUnique: No Default value: NoneisNullable: False MlEntitiesRelationPerLevel (ML There is a relationship between the ML entities on a Type: Enummultiplicity (Entity relationship per level) Level of an ML entity group. The relationship could be "in sequence" or "in parallel". If the relationship is "insequence", then the sequence is represented by the order of the ML entities provided in the LEntityRefList attribute. AllowedValues: IN_SEQUENCE, IN_PARALLEL (Multiplicity): 1isOrdered: FalseisUnique: noAdefaultValue: NoneisNullable: False memberMLEntityRefList (Reference list of member ML entities) It identifies the list of member ML entities within a level of an ML entity group. allowedValues: DN list Type: DN (see 3GPPTS 32.156

[13] ) multiplicity: *isOrdered: TrueisUnique: TruedefaultValue: NoneisNullable: True NOTE: If the performanceScore is intended to indicate the performance rating for ML training, the dataset is the training dataset. 1.2.13. MLTESTINGFUNCTION (ML-TESTFUNCTION)

[0059] The ML entity tests can be performed within the ML training function or in a separate function. If the ML entity tests are performed in a function separate from the ML training function, the IOC-MLTestingFunction (ML test function) is instantiated and represents the logical function that performs the ML entity tests. The entity represented by the MLTestingFunction MOI supports the testing of one or more ML entities. Attributes relating to the MLTestingFunction can be as shown in Table 11. Table 11 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable mLEntityList (ML entity list) M T F F F 1.2.14. MLTESTINGREQUEST (ML-TESTANFORDERUNG)

[0060] The IOC-MLTestingRequest represents the ML entity test request generated by the ML test MnS consumer.

[0061] The MLTestingRequest MOI is contained within an MLTestingFunction MOI or ML TrainingFunction MOI, which represents the logical function that performs the ML entity tests. Each MLTestingRequest is associated with at least one ML entity.

[0062] The MLTestingRequest can include a source to identify its origin, which can be used to prioritize test resources for different sources. These sources could be, for example, network functions, operator roles, or other functional differentiations.

[0063] If the request is accepted, the ML test MnS generator decides when to start the ML tests. Once the MnS generator decides to start testing based on the request, the ML training MnS generator instantiates one or more MLTestingProcess MOI(s) (ML test processes) responsible for performing the following: - collects (more) data for testing if the test data is unavailable or the data is available but insufficient for training; - prepares and selects the necessary test data, taking into account the test data provided by the consumer's request. The ML training MnS generator can examine the consumer's provided test data and select none, some, or all of it for testing. Additionally, the ML training MnS generator can select other available test data to meet the consumer's ML entity training requirements. - tests the ML entity by performing an inference using the selected test data, - reports the performance of the ML entity when it performs on the selected test data.

[0064] The MLTestingRequest can have a requestStatus field to represent the status of the request: - The attribute values ​​are “NOT_STARTED”, “TRAINING _IN_PROGRESS”, “SUSPENDED”, “FINISHED”, and “CANCEL”. - When the value changes to “TESTING_IN_PROGRESS” (tests are being performed), the ML test MnS creator instantiates one or more MLTestingProcess MOI(s) representing the test process(s) being performed according to the request and notifies the MnS consumer(s) who has subscribed to the notification.

[0065] When the entire testing process associated with this requirement is complete, the value changes to "FINISHED".

[0066] Attributes relating to MLTesting Request can be as shown in Table 12. Table 12 Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable Attribute label O T T F T testingRequestSource (Test Request Source) M T T F T requestStatus (Request Status) M T F F T cancelRequest (cancellation request) O T T F T suspendRequest O T T F T Attribute relating to the role mLEntityToTestRef(Reference of the ML entity to be tested) CM T F F T 1.2.15. MLTESTINGREPORT (ML-TESTBERICHT)

[0067] The IOC-MLTestingReport represents the ML entity test report provided by the ML test MnS producer (e.g., to the MLT MnS consumer). The MLTestingReport MOI is contained within an MLTestingFunction MOI or MLTrainingFunction MOI, which represents the logical function that performs the ML entity tests. Attributes relating to MLTestingReport can be as shown in Table 13. Example attribute constraints that can relate to MLTestingReport are shown in Table 14. Table 13 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable areConsumerTestingDataUsed (are consumer testing data used) M T F F T usedConsumerTestingData (usedConsumerTestingData) CM T F F T modelPerformanceTesting(model performance testing) M T F F T mlTestingResult (ML test result) M T F F T Attribute relating to the role testingRequestRef (Test Request Reference) CM T F F T testingProcessRef (Test process reference) M T F F T Table 14 Designation definition usedConsumerTestingDataSupport Qualifier Condition: The value of the attribute areConsumerTestingDataUsed is ALL or PARTILY. testingRequestRef SupportQualifier (Support Qualifier for Test Request Reference) Condition: The MLTestingReport-MOI represents the report for the ML model testing requested by the MnS consumer (via MLTestingRequest-MOI). lastTestingRef SupportQualifier Condition: The MLTestingReport-MOI represents the report for the ML testing that was not initial testing (i.e., the model was previously tested). 1.2.16. MLTESTINGPROCESS

[0068] The IOC-MLTestingProcess represents the ML testing process.

[0069] One or more MLTestingProcess MOIs can be instantiated for each MLTestingRequest MOI or a set of MLTestingRequest MOIs.

[0070] For each ML entity MOI being tested, an MLTestingProcess is instantiated; that is, an MLTestingProcess MOI is associated with exactly one ML entity MOI. The MLTrainingProcess can be associated with one or more MLTrainingRequest MOIs.

[0071] The MLTestingProcess does not have to correspond to a specific MLTestingRequest, i.e., an MLTestingRequest does not have to be associated with a specific MLTestingProcess.

[0072] Each MLTestingProcess has a priority that can be used to prioritize the execution of different MLTestingProcess instances. By default, the priority of the MLTestingProcess can be in a 1:1 relationship.

[0073] The "progressStatus" attribute represents the status of the ML entity test and contains information that the ML training MnS consumer can use to monitor progress and results. The data type of this attribute is "ProcessMonitor" (see, for example, [TS28622]). The following specializations are provided for this data type for ML entity tests: - The "Status" attribute values ​​are "RUNNING", "CANCELLING", "SUSPENDED", "FINISHED", and "CANCELLED". The other values ​​are not used. - The "Timer" attribute is not used. - If the "Status" is "RUNNING", the "progressStateInfo" attribute should indicate one of the following states: "COLLECTING_DATA" (data is being collected), "PREPARING_TESTING_DATA" (test data is being prepared), "TESTING" (test is being performed). No specifications are provided for the attribute "resultStateInfo". However, vendor-specific information may be provided.

[0074] When testing is completed with a status of "FINISHED", the ML-Test-MnS generator provides the test report to the MnS consumer by generating an MLTestingReport-MOI.

[0075] Examples of attributes that may be related to the MLT testing process are shown in Table 15. Table 15 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable priority M T T F T progressStatus M T F F T cancel process O T T F T suspend process O T T F T Attribute that relates to the role testingRequestRef (Test Request Reference) CM T F F T testingReportRef (Test report reference) M T F F T mlEntityBeingTestedRef (Reference of the tested ML entity) CM T F F T 1.2.17. ML TESTING POLICY (ML TESTING GUIDELINE)

[0076] This IOC represents the policy for MnS producer-initiated ML entity testing. An ML test policy can be applicable to one or more ML entities. Attributes relating to the ML test policy can be as shown in Table 16. Table 16 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable mLEntityTestingpolicy (ML-Entitytestellini e) M T F F T Attribute relating to the role 1.2.18. TESTING POLICY < <datatype>> (Test policy < <datentyp>>)

[0077] This data type specifies the ML testing policy for one or more ML entities. Attributes related to TestingPolicy can be as shown in Table 17. Table 17 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable mLEntityIdList (ML Entity ID List) M T T F T testingTimeWindows (Test time window) M T T F T maxNumberTesting (maximum number of tests) M T T F T 1.2.19. TIMEWINDOW < <datatype>> (Time window < <datentyp>>)

[0078] This < <datatype>> represents a time duration. Attributes relating to TimeWindow can be as shown in Table 18. Table 18 Attribute label Support Qualifier isReadable isWritable isInvariant(isInvariant) isNotifyable startTime (Start time) M T T F T stopTime (Stop time) M T T F T 1.2.20. ATTRIBUTE PROPERTIES

[0079] Exemplary properties of one or more of the attributes herein (e.g., one or more of the test-related attributes and / or any other attribute herein) or relating thereto may be as shown in Table 19. Table 19 Attribute label Documentation and permissible values Characteristics priority It specifies the priority of the process. The priority can be used to schedule the corresponding processes. A lower value indicates a higher priority. AllowedValues: { 0..65535}. Type: Distinguished Name (DN) (see e.g. [TS32156]) multiplicity: 1 isOrdered: n / a isUnique: n / a Default Value: 0 isNullable: False cancelRequest (cancellation request) It indicates whether the MnS consumer cancels the request. Setting this attribute to "TRUE" cancels the request. Cancellation is possible if the requestStatus is "NOT_STARTED", "TRAINING_IN_PROGRESS", or "SUSPENDED". Setting the attribute to "FALSE" has no observable result. The default value is "FALSE". AllowedValues: TRUE, FALSE. Type: Boolean multiplicity: 0..1 isOrdered: no AisUnique: no Default value: FALSE isNullable: FALSE suspendRequest It indicates whether the MnS consumer suspends the request. Setting this attribute to "TRUE" suspends the request. Suspension is possible if the requestStatus is not "FINISHED". Setting the attribute to "FALSE" has no observable result. The default value is set to "FALSE". allowedValues ​​(allowed values) Type: Boolean multiplicity: 0..1 isOrdered: no AisUnique: no Default value: FALSE isNullable: FALSE Values): TRUE, FALSE. cancel process It indicates whether the MnS consumer terminates the process. Setting this attribute to "TRUE" cancels the ML training request. Termination is possible if the "mLTrainingProcess.progressStatus.status" is not in the "FINISHED" state. Setting the attribute to "FALSE" has no observable result. The default value is set to "FALSE". AllowedValues: TRUE, FALSE. Type: Boolean multiplicity: 0..1 isOrdered: no AisUnique: no Default value: FALSE isNullable: FALSE suspend process It indicates whether the MnS consumer suspends the process. Setting this attribute to "TRUE" suspends the process. Suspension is possible if the process status is not "FINISHED", "CANCELLING", or "CANCELLED". Setting the attribute to "FALSE" has no observable result. The default value is "FALSE". Allowed values: TRUE, FALSE. Type: Boolean multiplicity: 0..1 isOrdered: no AisUnique: no Default value: FALSE isNullable: FALSE mLEntityIdList (ML Entity ID List) It identifies a list of ML entities. allowedValues: n / a Type: String multiplicity: *isOrdered: kAisUnique:TrueDefaultValue:NoneIsNullable:False candidateTestingDataSource(candidate testing data source) It provides the address(es) of the candidate test data source provided by the MnS consumer. The detailed test data format is vendor-specific. AllowedValues: n / a Type: String multiplicity: *isOrdered:FalseUnique:TruedefaultValue:NoneisNullable:True testingRequestSource (Test Request Source) It describes the entity that requested the ML TestingRequest MOI to be instantiated. Type: String multiplicity: 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: False MLTestingRequest.requestStatus(ML-Testanforderung.Anf forderungsstatus) It describes the status of a specific ML test request. allowedValues: NOT_STARTED, TESTING_IN_PROGRESS, CANCELLING, SUSPENDED, FINISHED and CANCELLED. Type: Enum multiplicity: 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: False mLEntityToTestRef (reference of the ML entity to be tested) It identifies the DN of the ML entity that is requested for testing. Type: DN (see e.g. [TS32156]) multiplicity: 1 allowedValues ​​(permitted values): DN isOrdered: False; Unique: True; Default Value: None; Nullable: True areConsumerTestingDataUsed (are consumer testing data used) It indicates whether the consumer-provided test data was used for the ML entity tests. AllowedValues: ALL, PARTIALLY, NONE. Type: Enum multiplicity: 1isOrdered: kAisUnique: kAdefaultValue: NoneisNullable: True usedConsumerTestingData (usedConsumerTestingData) It provides the address(es) where lists of consumer-provided test data used for ML entity testing are located. allowedValues: n / a Type: String multiplicity: *isOrdered:FalseUnique:TruedefaultValue:NoneisNullable:True modelPerformanceTesting(model performance testing) It indicates the performance rating of the ML entity when performed on the test data. allowedValues: n / a Type: Model Performance Multiplicity: *isOrdered: k AisUnique: k Default Value: None IsNullable: False mlTestingResult (ML test result) It provides the address where the test result (including the inference result for each test data sample) will be delivered. The detailed test result format is vendor-specific. Allowed values: n / a Type: String multiplicity: 1isOrdered:FalseUnique:TruedefaultValue:NoneisNullable:True testingRequestRef (Test Request Reference) It identifies the DN of the MLTestingRequest-MOI.allowedValues ​​(allowed values): DN Type: DN (see e.g. [TS32156]) multiplicity: 1 isOrdered: False isUnique: True defaultValue: None isNullable: True testingProcessRef (Test process reference) It identifies the DN of the ML TestingProcess-MOI.allowedValues ​​(allowed values): DN Type: DN (see e.g. [TS32156]) multiplicity: 1 isOrdered: False isUnique: True defaultValue: None isNullable: True ML TestingProcess.progressStatus(ML-Testprozess.Fortschrittsstatus) It indicates the status of the ML test process. allowedValues ​​(permitted values): n / a Type: ProcessMonitor (see e.g. [TS28622]) multiplicity: 1 isOrdered: n / a isUnique: n / a DefaultValue: None isNullable: False mlEntityBeingTestedRef(Reference of the tested ML entity) It identifies the DN of the ML entity MOI being tested. allowedValues: DN Type: DN (see e.g. [TS32156]) multiplicity: 1 isOrdered: False isUnique: True defaultValue: None isNullable: True testingReportRef (Test report reference) It identifies the DN of the ML entity being tested by an ML testing process. `allowedValues`: DN Type: DN (see e.g. [TS32156]) multiplicity: 1 isOrdered: False isUnique: True defaultValue: None isNullable: True mLEntityTestingpolicy (ML Entity Testing Policy) It specifies the ML entity testing policy. allowedValues: n / a Type: TestingPolicy Multiplicity: *isOrdered: k AisUnique: k DefaultValue: None IsNullable: False testingTimeWindows (Test time window) It specifies the time windows in which ML entity tests are allowed. allowedValues: n / a Type: TimeWindow; multiplicity: *isOrdered: k; aisUnique: k; defaultValue: None; isNullable: False start time It indicates the start time. Type: DateTime (see e.g. [TS32156]) multiplicity: 1 isOrdered: n / a isUnique: n / a DefaultValue: None isNullable: True stopTime (Stop time) It displays the stop time. Type: DateTime (see e.g. [TS32156]) multiplicity: 1 isOrdered: n / a Attribute label Documentation and permissible values Characteristics isUnique: kAdefaultValue: NoneisNullable: True NOTE: If the performanceScore is intended to indicate the performance rating for ML training, the dataset is the training dataset. If the performanceScore is intended to indicate the performance rating for ML tests, the dataset is the test dataset. 2. ASPECTS OF THE CELLULAR NETWORK

[0080] Fig. Figures 8-12 illustrate various systems, devices, and components that can implement aspects of disclosed embodiments.

[0081] Fig. Figure 8 illustrates a Network 800 according to various embodiments. The Network 800 can operate in a manner consistent with 3GPP technical specifications for LTE or 5G / NR systems. However, the exemplary embodiments are not limited in this respect, and the described embodiments may apply to other networks that benefit from the principles described herein, such as future 3GPP systems or the like.

[0082] The Network 800 can include a UE 802, which can contain any mobile or non-mobile computing device designed to communicate with a RAN 804 via an over-the-air connection. The UE 802 can be communicatively coupled to the RAN 804 through a Uu interface.The UE 802 can be, among other things, a smartphone, a tablet computer, a wearable computer device, a desktop computer, a laptop computer, in-vehicle infotainment, an in-vehicle entertainment device, a combination instrument, a head-up display device, an on-board diagnostic device, a mobile dashboard equipment, a mobile data terminal, an electronic engine management system, an electronic / engine control unit, an electronic / engine control module, an embedded system, a sensor, a microcontroller, a control module, an engine management system, a networked device, a machine-type communication device, an M2M or D2D device, an IoT device, etc.

[0083] In some embodiments, the Network 800 can include multiple UEs directly coupled to each other via a sidelink interface. The UEs can be M2M / D2D devices communicating using physical sidelink channels, such as PSBCH, PSDCH, PSSCH, PSCCH, PSFCH, etc.

[0084] In some embodiments, the UE 802 can additionally communicate with an AP 806 via an over-the-air connection. The AP 806 can manage a WLAN connection, which can be used to offload some or all of the network traffic from the RAN 804. The connection between the UE 802 and the AP 806 can be consistent with any IEEE 802.11 protocol, with the AP 806 potentially being a Wireless Fidelity (WiFi®) router. In some embodiments, the UE 802, the RAN 804, and the AP 806 can utilize cellular WLAN aggregation (for example, LWA / LWIP). Cellular WLAN aggregation can involve the UE 802 being configured by the RAN 804 to utilize both cellular radio resources and WLAN resources.

[0085] The RAN 804 can include one or more access nodes, for example, the AN 808. The AN 808 can terminate air interface protocols for the UE 802 by providing access stratum protocols, including RRC, PDCP, RLC, MAC, and L1 protocols. In this way, the AN 808 can enable data / voice connectivity between the CN 820 and the UE 802. In some embodiments, the AN 808 can be implemented in a discrete device or as one or more software entities running on server computers, for example, as part of a virtual network that can be called a CRAN or virtual baseband unit pool. The AN 808 can be referred to as BS, gNB, RAN node, eNB, ng-eNB, NodeB, RSU, TRxP, TRP, etc.The AN 808 can be a macrocell base station or a low-power base station for deploying femtocells, picocells, or other similar cells with smaller coverage areas, smaller user capacity, or higher bandwidth compared to macrocells.

[0086] In embodiments where the RAN 804 includes multiple ANs, they can be interconnected via an X2 interface (if the RAN 804 is an LTE RAN) or an Xn interface (if the RAN 804 is a 5G RAN). The X2 / Xn interfaces, which in some embodiments may be separated into control / user-level interfaces, allow the ANs to communicate information related to handovers, data / context transfers, mobility, load management, interference coordination, and so on.

[0087] The ANs of the RAN 804 can each manage one or more cells, cell groups, component carriers, etc., to provide the UE 802 with an air interface for network access. The UE 802 can be connected to multiple cells simultaneously, provided by the same or different ANs of the RAN 804. For example, the UE 802 and the RAN 804 can use carrier aggregation to allow the UE 802 to connect to a variety of component carriers, each corresponding to a Pcell or Scell. In dual connectivity scenarios, a first AN can be a master node providing an MCG, and a second AN can be a secondary node providing an SCG. The first / second AN can be any combination of eNB, gNB, ng-eNB, etc.

[0088] The RAN 804 can provide the air interface over either a licensed or an unlicensed spectrum. To operate in the unlicensed spectrum, nodes can use LAA, eLAA, and / or feLAA mechanisms based on CA technology with PCells / Scells. Before accessing the unlicensed spectrum, nodes can perform medium / carrier capture operations based, for example, on a Listen-Before-Talk (LBT) protocol.

[0089] In V2X scenarios, the UE 802 or the AN 808 can be, or function as, an RSU, which can refer to any transportation infrastructure entity used for V2X communications. An RSU can be implemented in or by a suitable AN or a stationary (or relatively stationary) UE. An RSU can be implemented in or by: a UE can be referred to as a "UE-type RSU"; an eNB can be referred to as an "eNB-type RSU"; a gNB can be referred to as a "gNB-type RSU"; and so on. In one example, an RSU is a computing device coupled to a high-frequency circuitry located at the roadside, providing connectivity support for passing vehicle UEs.The RSU can also include an internal data storage circuitry to store intersection map geometry, traffic statistics, media, and applications / software for capturing and controlling ongoing vehicle and pedestrian traffic. The RSU can provide very low-latency communications required for high-speed events such as collision avoidance, traffic alerts, and the like. Additionally or alternatively, the RSU can provide other cellular / WLAN communication services. The RSU components can be housed in a weatherproof enclosure suitable for outdoor installation and can include network interface control to provide a wired connection (e.g., Ethernet) to a traffic signal controller or backhaul network.

[0090] In some embodiments, the RAN 804 can be an LTE-RAN 810 with eNBs, for example, the eNB 812. The LTE-RAN 810 can provide an LTE air interface with the following characteristics: 15 kHz SCS; CP-OFDM waveform for DL ​​and SC-FDMA waveform for UL; turbo codes for data and TBCC for control, etc. The LTE air interface can rely on CSI-RS for CSI acquisition and beam management; PDSCH / PDCCH-DMRS for PDSCH / PDCCH demodulation; and CRS for cell search and initial acquisition, channel quality measurements, and channel estimation for coherent demodulation / detection at the UE. The LTE air interface can operate on sub-6 GHz bands.

[0091] In some embodiments, the RAN 804 can be an NG-RAN 814 with gNBs, for example the gNB 816, or ng-eNBs, for example the ng-eNB 818. The gNB 816 can connect to 5G-enabled UEs using a 5G NR interface. The gNB 816 can connect to a 5G core via an NG interface, which may include an N2 or N3 interface. The ng-eNB 818 can also connect to the 5G core via an NG interface, but can also connect to a UE via an LTE air interface. The gNB 816 and the ng-eNB 818 can connect to each other via an Xn interface.

[0092] In some embodiments, the NG interface can be divided into two parts: an NG user level (NG-U) interface, which carries traffic data between the nodes of the NG-RAN 814 and a UPF 848 (e.g., N3 interface), and an NG control level (NG-C) interface, which is a signaling interface between the nodes of the NG-RAN 814 and an AMF 844 (e.g., N2 interface).

[0093] The NG-RAN 814 can provide a 5G-NR air interface with the following characteristics: variable SCS; CP-OFDM for DL, CP-OFDM and DFT-s-OFDM for UL; polar, repeat, simplex, and Reed-Muller codes for control, and LDPC for data. The 5G-NR air interface can rely on CSI-RS and PDSCH / PDCCH-DMRS, similar to the LTE air interface. The 5G-NR air interface may not use CRS but can use PBCH-DMRS for PBCH demodulation; PTRS for phase tracking for PDSCH; and a tracking reference signal for timing. The 5G-NR air interface can operate on FR1 bands, which include sub-6 GHz bands, or FR2 bands, which include bands from 24.25 GHz to 52.6 GHz. The 5G-NR air interface can include an SSB, which is an area of ​​a downlink resource grid that includes PSS / SSS / PBCH.

[0094] In some implementations, the 5G NR air interface can utilize BWPs for various purposes. For example, BWP can be used for dynamic SCS adjustment. For instance, the UE 802 can be configured with multiple BWPs, each with a different SCS. When a BWP change is specified to the UE 802, the transmission's SCS is also modified. Another use case for BWP is related to power saving. Specifically, multiple BWPs for the UE 802 can be configured with varying amounts of frequency resources (e.g., PRBs) to support data transmission under different traffic load scenarios. A BWP containing fewer PRBs can be used for data transmission under low traffic load conditions, while allowing power savings on the UE 802 and, in some cases, on the gNB 816.A BWP that contains a larger number of PRBs can be used for scenarios with higher traffic loads.

[0095] The RAN 804 is communicatively coupled to the CN 820, which contains network elements to provide various functions for supporting data and telecommunications services for customers / subscribers (for example, users of the UE 802). The components of the CN 820 can be implemented in a single physical node or in separate physical nodes. In some embodiments, NFV can be used to virtualize any or all of the functions provided by the network elements of the CN 820 onto physical computing / storage resources in servers, switches, etc. A logical instantiation of the CN 820 can be referred to as a network slice, and a logical instantiation of a portion of the CN 820 can be referred to as a network subslice.

[0096] In some embodiments, the CN 820 can be an LTE-CN 822, which can also be referred to as an EPC. The LTE-CN 822 can include the MME 824, the SGW 826, the SGSN 828, the HSS 830, the PGW 832, and the PCRF 834, which are coupled to each other via interfaces (or "reference points"), as shown. The functions of the elements of the LTE-CN 822 can be briefly introduced below.

[0097] The MME 824 can implement mobility management functions to track the current location of the UE 802 to enable paging, carrier activation / deactivation, handovers, gateway selection, authentication, etc.

[0098] The SGW 826 can establish an S1 interface to the RAN and route data packets between the RAN and the LTE-CN 822. The SGW 826 can serve as a local mobility anchor point for inter-RAN node handover and can also provide an anchor for inter-3GPP mobility. Other responsibilities may include lawful interception, charge collection, and some policy enforcement.

[0099] The SGSN 828 can track the location of the UE 802 and perform security functions and access control. Additionally, the SGSN 828 can perform inter-EPC node signaling for mobility between different RAT networks; PDN and S-GW selection as specified by the MME 824; MME selection for handovers, etc. The S3 reference point between the MME 824 and the SGSN 828 can enable user and carrier information exchange for inter-3GPP access network mobility in inactive / active states.

[0100] The HSS 830 can include a network user database containing subscription-related information to support the handling of communication sessions by network entities. The HSS 830 can provide support for routing / roaming, authentication, authorization, name / address resolution, location dependencies, and more. An S6a reference point between the HSS 830 and the MME 824 can enable the transfer of subscription and authentication data to authenticate / authorize user access to the LTE-CN 820.

[0101] The PGW 832 can terminate an SGi interface to a data network (DN) 836, which may include an application / content server 838. The PGW 832 can route data packets between the LTE CN 822 and the data network 836. The PGW 832 can be coupled to the SGW 826 via an S5 reference point to enable user-level tunneling and tunnel management. The PGW 832 can also include a node for policy enforcement and charge calculation data collection (for example, PCEF). Additionally, the SGi reference point between the PGW 832 and the data network 836 can be an external public PDN, a private PDN, or an internal PDN, for example, for providing IMS services. The PGW 832 can be coupled to a PCRF 834 via a Gx reference point.

[0102] The PCRF 834 is the policy and charge calculation control element of the LTE-CN 822. The PCRF 834 can be communicatively coupled with the App / Content Server 838 to determine appropriate QoS and charge calculation parameters for service flows. The PCRF 834 can provide associated rules to a PCEF (via a Gx reference point) with appropriate TFT and QCI.

[0103] In some embodiments, the CN 820 can be a 5GC 840. The 5GC 840 can be an AUSF 842, AMF 844, SMF 846, UPF 848, NSSF 850, NEF 852, NRF 854, PCF 856, UDM 858, and AF 860, which, as shown, are coupled to each other via interfaces (or "reference points"). The functions of the elements of the 5GC 840 can be briefly introduced below.

[0104] The AUSF 842 can store data for UE 802 authentication and handle authentication-related functionality. The AUSF 842 can provide a common authentication framework for different access types. In addition to communicating with other 5GC 840 components via reference points, as shown, the AUSF 842 can have a service-based NausF interface.

[0105] The AMF 844 can allow other functions of the 5GC 840 to communicate with the UE 802 and the RAN 804 and to subscribe to notifications about mobility events related to the UE 802. The AMF 844 can also handle registration management (for example, registering the UE 802), connection management, reachability management, mobility management, lawful interception of AMF-related events, and access authentication and authorization. The AMF 844 can provide transport for SM messages between the UE 802 and the SMF 846 and act as a transparent proxy for routing SM messages. The AMF 844 can also provide transport for SMS messages between the UE 802 and an SMSF. The AMF 844 can interact with the AUSF 842 and the UE 802 to perform various security anchor and context management functions.Furthermore, the AMF 844 can be an endpoint of a RAN-CP interface, which may include or be an N2 reference point between the RAN 804 and the AMF 844; and the AMF 844 can be a termination point of NAS(N1) signaling and perform NAS encryption and integrity protection. The AMF 844 can also support NAS signaling with the UE 802 via an N3-IWF interface.

[0106] The SMF 846 can be responsible for SM (for example, session setup, tunnel management between the UPF 848 and an AN 808); UE IP address assignment and management (including optional authorization); selection and control of a UP function; configuration of traffic control at the UPF 848 to route traffic to a suitable destination; completion of interfaces to policy control functions; control of part of policy enforcement, charge calculation, and QoS; lawful interception (for SM events and interface to the LI system); completion of SM portions of NAS messages; downlink data notification; initiation of AN-specific SM information sent via the AMF 844 over N2 to the AN 808; and determining an SSC mode of a session.SM can refer to the management of a PDU session, and a PDU session or "session" can refer to a PDU connectivity service that provides or enables the exchange of PDUs between the UE 802 and the 836 data network.

[0107] The UPF 848 can act as an anchor point for intra-RAT and inter-RAT mobility, an external PDU session point for interconnecting with the 836 data network, and a branch point to support a multi-homed PDU session. The UPF 848 can also perform packet routing and forwarding, packet inspection, user-level policy rule enforcement, lawful packet interception (UP collection), traffic usage reporting, user-level QoS handling (e.g., packet filtering, gating, UL / DL rate enforcement), uplink traffic verification (e.g., SDF-to-QoS flow mapping), uplink and downlink transport-level packet marking, and downlink packet buffering and data notification triggering. The UPF 848 can include an uplink classifier to assist in routing traffic flows to a data network.

[0108] The NSSF 850 ​​can select a set of network slice instances to serve the UE 802. The NSSF 850 ​​can also determine permissible NSSAIs and the mapping to the subscribed S-NSSAIs, if necessary. The NSSF 850 ​​can also determine an AMF set to use to serve the UE 802, or a list of candidate AMFs based on a suitable configuration and possibly by querying the NRF 854. The selection of a network slice instance set for the UE 802 can be triggered by the AMF 844, with which the UE 802 is registered, by interacting with the NSSF 850, which may result in a change to the AMF. The NSSF 850 ​​can interact with the AMF 844 via an N22 reference point. and can communicate with another NSSF in a visited network via an N31 reference point (not shown). Additionally, the NSSF 850 ​​can have a service-based NSSF interface.

[0109] The NEF 852 can securely discover services and capabilities provided by 3GPP third-party network functions, internal discovery / re-discovery, AFs (e.g., AF 860), edge computing or fog computing systems, and so on. In such configurations, the NEF 852 can authenticate, authorize, or throttle the AFs. The NEF 852 can also translate information exchanged with the AF 860 and information exchanged with internal network functions. For example, the NEF 852 can translate between an AF service identifier and internal 5GC information. The NEF 852 can also receive information from other network functions based on discovered capabilities of other network functions. This information can be stored on the NEF 852 as structured data or on a data storage network function using standardized interfaces.The stored information can then be re-examined by other NFs and AFs using the NEF 852, or used for other purposes, such as analytics. Additionally, the NEF 852 can feature a service-based Nnef interface.

[0110] The NRF 854 can support service discovery functions, receive NF discovery requests from NF instances, and provide information about discovered NF instances to the NF instances. The NRF 854 also maintains information about available NF instances and their supported services. As used herein, the terms "instantiate," "instantiation," and the like can refer to the creation of an instance, and an "instance" can refer to a specific occurrence of an object, such as that which might occur during the execution of program code. Additionally, the NRF 854 can have a service-based Nnrf interface.

[0111] The PCF 856 can provide policy rules to control plane functions for enforcement and can also support a unified policy framework to govern network behavior. The PCF 856 can also implement a front end to access subscription information relevant to policy decisions in a UDR of the UDM 858. In addition to communicating with functions via reference points, as shown, the PCF 856 features a service-based Npcf interface.

[0112] The UDM 858 can handle subscription-related information to support the management of communication sessions by network entities and can store UE 802 subscription data. For example, subscription data can be communicated between the UDM 858 and the AMF 844 via an N8 reference point. The UDM 858 can comprise two parts: an application front end and a UDR. The UDR can store subscription and policy data for the UDM 858 and the PCF 856, and / or structured data for discovery and application data (including application detection PFDs and application request information for multiple UE 802s) for the NEF 852. The UDR 221 can feature the service-based Nudr interface to allow the UDM 858, PCF 856, and NEF 852 to access a specific set of stored data, as well as to read and update notifications of relevant data changes in the UDR (e.g.,The UDM can add, modify, delete, and subscribe users. It can include a UDM frontend, which handles credential processing, site management, subscription management, and so on. Multiple different frontends can serve the same user in different transactions. The UDM frontend accesses subscription information stored in the UDR and performs authentication credential processing, user identification handling, access authorization, registration / mobility management, and subscription management. In addition to communicating with other network interfaces (NFs) via reference points, as shown, the UDM 858 can have the service-based Nudm interface.

[0113] The AF 860 can provide application influence on traffic routing, provide access to the NEF, and interact with the policy framework for policy control.

[0114] In some embodiments, the 5GC 840 can enable edge computing by selecting operator / third-party services that are geographically close to a point where the UE 802 connects to the network. This can reduce latency and network load. To provide edge computing implementations, the 5GC 840 can select a UPF 848 near the UE 802 and perform traffic routing from the UPF 848 to the Data Network 836 via the N6 interface. This can be based on UE subscription data, UE location, and information provided by the AF 860. In this way, the AF 860 can influence UPF (re)selection and traffic routing. Based on the operator's deployment, if the AF 860 is considered a trusted entity, the network operator can allow the AF 860 to interact directly with relevant network functions. Additionally, the AF 860 can feature a service-based Naf interface.

[0115] The data network 836 can represent various network operator services, internet access, or third-party services, which may be provided by one or more servers, including, for example, the application / content server 838.

[0116] Fig. Figure 9 schematically illustrates a wireless network 900 according to various embodiments. The wireless network 900 can include a UE 902 in wireless communication with an AN 904. The UE 902 and the AN 904 can be similar to and essentially interchangeable with the similarly named components described elsewhere herein.

[0117] The UE 902 can be communicatively coupled to the AN 904 via a 906 connection. The 906 connection is illustrated as an air interface to enable communicative coupling and can be compatible with cellular communication protocols, such as an LTE protocol or a 5G NR protocol operating at millimeter wave or sub-6 GHz frequencies.

[0118] The UE 902 can include a host platform 908 coupled to a modem platform 910. The host platform 908 can include an application processing circuit arrangement 912, which can be coupled to a protocol processing circuit arrangement 914 of the modem platform 910. The application processing circuit arrangement 912 can run various applications for the UE 902 that produce / receive application data. The application processing circuit arrangement 912 can further implement one or more layer operations to transmit / receive application data to / from a data network. These layer operations can include transport (for example, UDP) operations and internet (for example, IP) operations.

[0119] The Protocol Processing Circuit Assembly 914 can implement one or more layer operations to enable the transmission or reception of data over the 906 link. The layer operations implemented by the Protocol Processing Circuit Assembly 914 can include, for example, MAC, RLC, PDCP, RRC, and NAS operations.

[0120] The 910 modem platform may further include a 916 digital baseband circuit arrangement, which may implement one or more layer operations that are “under” layer operations performed by the 914 protocol processing circuit arrangement in a network protocol stack. These operations may include, for example, PHY operations, including one or more of HARQ-ACK functions, scrambling / descrambling, encoding / decoding, layer mapping / demapping, modulation symbol mapping, receive symbol / bit metric determination, multi-antenna port precoding / decoding, which may include one or more of spacetime, space frequency, or space coding, reference signal generation / detection, preamble sequence generation and / or decoding, synchronization sequence generation / detection, control channel signal blind decoding, and other related functions.

[0121] The modem platform 910 can further include a transmit circuit assembly 918, a receive circuit assembly 920, an RF circuit assembly 922, and an RF front end (RFFE) 924, which may include or be connected to one or more antenna panels 926. In short, the transmit circuit assembly 918 can include a digital-to-analog converter, a mixer, intermediate frequency (IF) components, etc.; the receive circuit assembly 920 can include an analog-to-digital converter, a mixer, IF components, etc.; the RF circuit assembly 922 can include a low-noise amplifier, a power amplifier, power tracking components, etc.; the RFFE 924 can include filters (for example, acoustic surface / volume wave filters), switches, antenna tuners, beamforming components (for example, phase array antenna components), etc.The selection and arrangement of the components of the transmit circuit arrangement 918, the receive circuit arrangement 920, the RF circuit arrangement 922, the RFFE 924, and the antenna panels 926 (generally referred to as "transmit / receive components") may be specific to details of a particular implementation, such as whether the communication is TDM or FDM, takes place in mmWave or sub-6 GHz frequencies, etc. In some embodiments, the transmit / receive components may be arranged in multiple parallel transmit / receive chains, may be located in the same or different chips / modules, etc.

[0122] In some embodiments, the protocol processing circuit arrangement 914 may include one or more instances of a control circuit arrangement (not shown) for providing control functions for the transmit / receive components.

[0123] UE reception can be established through and via the antenna panels 926, the RFFE 924, the RF circuit arrangement 922, the receive circuit arrangement 920, the digital baseband circuit arrangement 916, and the protocol processing circuit arrangement 914. In some embodiments, the antenna panels 926 can receive a transmission from the AN 904 by receiving beamforming signals received by multiple antennas / antenna elements of the one or more antenna panels 926.

[0124] A UE transmission can be established through and via the protocol processing circuit arrangement 914, the digital baseband circuit arrangement 916, the transmit circuit arrangement 918, the RF circuit arrangement 922, the RFFE 924, and the antenna panels 926. In some embodiments, the transmitting components of the UE 904 can apply a spatial filter to the data to be transmitted in order to form a transmit beam that is emitted by the antenna elements of the antenna panels 926.

[0125] Similar to the UE 902, the AN 904 can include a host platform 928 coupled to a modem platform 930. The host platform 928 can include an application processing circuit arrangement 932 coupled to a protocol processing circuit arrangement 934 of the modem platform 930. The modem platform can further include a digital baseband circuit arrangement 936, a transmit circuit arrangement 938, a receive circuit arrangement 940, an RF circuit arrangement 942, an RFFE circuit arrangement 944, and antenna panels 946. The components of the AN 904 can be similar to, and essentially interchangeable with, the similarly named components of the UE 902.In addition to performing data transmission / data reception as described above, the components of the AN 908 can perform various logical functions, including, for example, RNC functions such as radio carrier management, dynamic uplink and downlink radio resource management, and data packet scheduling.

[0126] Fig. Figure 10 is a block diagram illustrating components according to some exemplary embodiments that are capable of reading instructions from a machine-readable or computer-readable medium (e.g., a non-volatile machine-readable storage medium) and executing one or more of any of the methodologies discussed herein. In particular, Figure 10 shows Fig. 10 a diagrammatic representation of hardware resources 1000 including one or more processors (or processor cores) 1010, one or more memory / storage devices 1020 and one or more communication resources 1030, each of which may be communicatively coupled via a bus 1040 or other interface circuit arrangement. For embodiments in which node virtualization (e.g. NFV) is used, a hypervisor 1002 may be executed to provide an execution environment for one or more network slices / subslices to utilize the hardware resources 1000.

[0127] The 1010 processors can, for example, include a 1012 processor and a 1014 processor. The 1010 processors can be, for example, a central processing unit (CPU), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a graphics processing unit (GPU), a DSP such as a baseband processor, an ASIC, an FPGA, a radio frequency integrated circuit (RFIC), another processor (including those discussed herein), or any suitable combination thereof.

[0128] The memory / storage devices 1020 can include main memory, disk storage, or any suitable combination thereof. The memory / storage devices 1020 can include, among other things, any type of volatile, non-volatile, or semi-volatile memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state storage, etc.

[0129] The communication resources 1030 can include interlink or network interface controllers, components, or other suitable devices for communicating with one or more peripheral devices 1004, one or more databases 1006, or other network elements over a network 1008. For example, the communication resources 1030 can include wired communication components (e.g., for coupling via USB, Ethernet, etc.), cellular communication components, NFC components, Bluetooth® (e.g., Bluetooth® Low Energy) components, WiFi® components, and other communication components.

[0130] The instructions 1050 may comprise software, a program, an application, an applet, an app, or other executable code to cause at least one of the processors 1010 to perform one or more of the methodologies discussed herein. The instructions 1050 may reside wholly or partially within at least one of the processors 1010 (for example, within the processor's cache memory), the memory / storage devices 1020, or any suitable combination thereof. Furthermore, any part of the instructions 1050 may be transferred from any combination of the peripheral devices 1004 or the databases 1006 to the hardware resources 1000. Accordingly, the memory of the processors 1010, the memory / storage devices 1020, the peripheral devices 1004, and the databases 1006 are examples of computer-readable and machine-readable media.

[0131] Fig. Figure 11 illustrates a Network 1100 according to various embodiments. The Network 1100 can operate in a manner consistent with 3GPP technical specifications or technical reports for 6G systems. In some embodiments, the Network 1100 can operate concurrently with the Network 800. For example, in some embodiments, the Network 1100 can share one or more frequency or bandwidth resources with the Network 800. As a specific example, a UE (e.g., UE 1102) can be configured to operate in both the Network 1100 and the Network 800. Such a configuration can be based on a UE that includes a circuit arrangement designed to communicate with frequency and bandwidth resources of both the Network 800 and the Network 1100. In general, multiple elements of the Network 1100 can share one or more characteristics with elements of the Network 800.For the sake of brevity and clarity, such elements may not be repeated in the description of Network 1100.

[0132] The Network 1100 can include a UE 1102, which can be any mobile or non-mobile computing device designed to communicate with a RAN 1108 via an over-the-air connection. The UE 1102 can be similar to the UE 802, for example. The UE 1102 can be, among other things, a smartphone, tablet computer, wearable computing device, desktop computer, laptop computer, in-vehicle infotainment system, in-vehicle entertainment device, instrument cluster, head-up display device, on-board diagnostic device, mobile dashboard equipment, mobile data terminal, electronic engine management system, electronic / engine control unit, electronic / engine control module, embedded system, sensor, microcontroller, control module, engine management system, networked device, machine-type communication device, M2M or D2D device, IoT device, etc.

[0133] Although in Fig. Not specifically shown in Figure 11, the 1100 network in some embodiments can include a variety of UEs directly coupled to each other via a sidelink interface. The UEs can be M2M / D2D devices that communicate using physical sidelink channels, such as, but not limited to, PSBCH, PSDCH, PSSCH, PSCCH, PSFCH, etc. Likewise, although this is shown in Figure 11, the network can also include a variety of UEs that are directly coupled to each other via a sidelink interface. Fig. 11 not specifically shown, the UE 1102 may be communicatively coupled with an AP, such as the AP 806, as with reference to Fig. 8 described. Although this in Fig. Not specifically shown in Figure 11, RAN 1108 may additionally include one or more ANs, such as AN 808, in some embodiments, as shown with reference to Fig. 8 described. The RAN 1108 and / or the AN of the RAN 1108 may be referred to as a base station (BS), a RAN node, or by any other term or name.

[0134] The UE 1102 and the RAN 1108 can be configured to communicate over an air interface that may be referred to as a sixth-generation (6G) air interface. The 6G air interface may include one or more features, such as communication in a terahertz (THz) or sub-THz bandwidth, or shared communication and acquisition. As used herein, the term "shared communication and acquisition" may refer to a system that enables wireless communication as well as radar-based acquisition over various types of multiplexes. As used herein, THz or sub-THz bandwidths may refer to communication in the frequency ranges of 80 GHz and above. Such frequency ranges may additionally or alternatively be referred to as "millimeter wave" or "mmWave" frequency ranges.

[0135] The RAN 1108 enables communication between the UE 1102 and a 6G core network (CN) 1110. Specifically, the RAN 1108 allows the transmission and reception of data between the UE 1102 and the 6G-CN 1110. The 6G-CN 1110 can include various functions, such as NSSF 850, NEF 852, NRF 854, PCF 856, UDM 858, AF 860, SMF 846, and AUSF 842. The 6G-CN 1110 can additionally include UPF 848 and DN 836, as shown in [reference to relevant documentation]. Fig. 11 shown.

[0136] Additionally, the RAN 1108 can include various supplementary functions that are in addition to or alternative to the functions of a legacy cellular network, such as a 4G or 5G network. Two such functions can include a Compute Control Function (Comp-CF) 1124 and a Compute Service Function (Comp-SF) 1136. The Comp-CF 1124 and the Comp-SF 1136 can be parts or functions of the Compute Service Layer. The Comp-CF 1124 can be a control layer function that provides functionalities such as managing the Comp-SF 1136, creating and managing compute task contexts (e.g., create, read, modify, delete), interacting with the underlying compute infrastructure for compute resource management, and so on. The Comp-SF 1136 can be a user layer function that acts as the gateway to interface compute service users (such as the UE 1102) and compute nodes behind a Comp-SF instance.Some functionalities of the Comp-SF 1136 can include: parsing compute service data received from users to calculate tasks executable by compute nodes; maintaining a service mesh entry gateway or service API gateway; enforcing service and charge policies; performance monitoring and telemetry collection, etc. In some implementations, a single instance of the Comp-SF 1136 can serve as the user-level gateway for a cluster of compute nodes. A single instance of the Comp-CF 1124 can control one or more instances of the Comp-SF 1136.

[0137] Two other such functions can include a Communication Control Function (Comm-CF) 1128 and a Communication Service Function (Comm-SF) 1138, which can be parts of the Communication Service Layer. The Comm-CF 1128 can be the control layer function for managing the Comm-SF 1138, creating / configuring / releasing communication sessions, and managing the communication session context. The Comm-SF 1138 can be a user layer function for data transport. The Comm-CF 1128 and the Comm-SF 1138 can be considered upgrades of the SMF 846 and the UPF 848, respectively, with respect to a 5G system in Fig. As described in section 8, the upgrades provided by the Comm-CF 1128 and the Comm-SF 1138 can enable service-conscious transport. For legacy data transport (e.g., 4G or 5G), the SMF 846 and the UPF 848 can still be used.

[0138] Two other such functions can include a Data Control Function (Data-CF) 1122 and a Data Service Function (Data-SF) 1132, which can be parts of the data service layer. The Data-CF 1122 can be a control layer function and provides functionalities such as managing the Data-SF 1132, creating / configuring / releasing data services, managing the context of data services, etc. The Data-SF 1132 can be a user layer function and act as the gateway between data service users (such as the UE 1102 and the various functions of the 6G-CN 1110) and data service endpoints behind the gateway. Specific functionalities can include: parsing data service user data and forwarding it to appropriate data service endpoints, generating charge data, and reporting the data service status.

[0139] Another such function is the Service Orchestration and Chaining Function (SOCF) 1120, which can discover, orchestrate, and chain communication / computing / data services provided by functions on the network. Upon receiving service requests from users, the SOCF 1120 can interact with one or more of the Comp-CF 1124, Comm-CF 1128, and Data-CF 1122 to identify instances of the Comp-SF 1136, Comm-SF 1138, and Data-SF 1132, configure service resources, and create the service chain. This chain could contain multiple instances of the Comp-SF 1136, Comm-SF 1138, and Data-SF 1132 and their associated compute endpoints. Workload processing and data movement can then be performed within the created service chain. SOCF 1120 can also be responsible for maintaining, updating, and releasing a created service chain.

[0140] Another such function is the Service Registry Function (SRF) 1114, which can act as a registry for system services provided at the user level, such as services provided by service endpoints behind gateways of Comp-SF 1136 and Data-SF 1132, and services provided by UE 1102. SRF 1114 can be considered a counterpart to NRF 854, which can act as the registry for network functions.

[0141] Other such functions may include an Evolved Service Communications Proxy (eSCP) and a Service Infrastructure Control Function (SICF) 1126, which can provide a service communications infrastructure for control-plane and user-plane services. The eSCP can be related to the 5G Service Communications Proxy (SCP), adding user-plane service communications proxy capabilities. The eSCP is therefore expressed in two parts: eCSP-C 1112 and eSCP-U 1134 for control-plane and user-plane service communications proxy, respectively. The SICF 1126 can control and configure eCSP instances with respect to service traffic routing policies, access rules, load balancing configurations, performance monitoring, and so on.

[0142] Another such function is AMF 1144. AMF 1144 can be similar to 844, but with additional functionality. In particular, AMF 1144 can involve a potential functional repartitioning, such as moving the message forwarding functionality from AMF 1144 to RAN 1108.

[0143] Another such function is the Service Orchestration Disclosure Function (SOEF) 1118. The SOEF can be designed to disclose service orchestration and chaining services to external users, such as applications.

[0144] The UE 1102 can include an additional function called Compute Client Service Function (Comp-CSF) 1104. The Comp-CSF 1104 can have both control-plane and user-plane functionalities and can interact with corresponding network-side functions, such as SOCF 1120, Comp-CF 1124, Comp-SF 1136, Data-CF 1122, and / or Data-SF 1132 for service discovery, request / response, compute workload exchange, etc. The Comp-CSF 1104 can also work with network-side functions to decide whether a compute task should be executed on the UE 1102, the RAN 1108, and / or an element of the 6G-CN 1110.

[0145] The UE 1102 and / or the Comp-CSF 1104 can include a Service Mesh Proxy 1106. The Service Mesh Proxy 1106 can act as a proxy for service-to-service communication at the user level. Capabilities of the Service Mesh Proxy 1106 can include addressing, security, load balancing, etc.

[0146] Fig. Figure 12 illustrates a simplified block diagram of artificial intelligence (AI)-assisted communication between a UE 1205 and a RAN 1210 according to various embodiments. In particular, as described in more detail below, AI / machine learning (ML) models can be used or leveraged to enable over-the-air communication between the UE 1205 and the RAN 1210.

[0147] The UE 1205 and / or the RAN 1210 can operate in a manner consistent with 3GPP technical specifications or technical reports for 6G systems. In some embodiments, the wireless cellular communication between the UE 1205 and the RAN 1210 can be part of, or operate concurrently with, the 1100, 800, and / or other networks described herein.

[0148] The UE 1205 may be similar to the UE 1102, the UE 802, and / or any other UE described herein, and may share one or more features with it. The UE 1205 may be, among other things, a smartphone, a tablet computer, a wearable computing device, a desktop computer, a laptop computer, in-vehicle infotainment, an in-vehicle entertainment device, a combination instrument, a head-up display device, an on-board diagnostic device, a mobile dashboard equipment, a mobile data terminal, an electronic engine management system, an electronic / engine control unit, an electronic / engine control module, an embedded system, a sensor, a microcontroller, a control module, an engine management system, a networked device, a machine-type communication device, an M2M or D2D device, an IoT device, etc.The RAN 1210 may be similar to the RAN 814, the RAN 1108 and / or any other RAN described herein and may share one or more features with it.

[0149] As in Fig. As can be seen in section 12, the AI-related elements of UE 1205 may be similar to the AI-related elements of RAN 1210. For the sake of discussion, a description of the various elements is provided from the perspective of UE 1205; however, it is understood that such a discussion or description applies to similarly named / numbered elements of RAN 1210 unless explicitly stated otherwise.

[0150] As previously noted, the UE 1205 may include various elements or functions related to AI / ML. Such elements may be implemented as hardware, software, firmware, and / or any combination thereof. In embodiments, one or more of the elements may be implemented as a separate element within the same hardware (e.g., chip or multiprocessor chip), software (e.g., a computational program), or firmware.

[0151] One such element can be a data repository 1215. The data repository 1215 can be responsible for data collection and storage. In particular, the data repository 1215 can collect and store RAN configuration parameters, measurement data, key performance indicators (KPIs), model capability metrics, etc., for model training, updates, and inference. More generally, collected data is stored in the repository. Stored data can be discovered and extracted from the data repository 1215 by other elements. As can be seen, for example, the inference data selection / filter element 1250 can retrieve data from the data repository 1215. In various embodiments, the UE 1205 can be configured to discover and request data from the data repository 1210 in the RAN, and vice versa.More generally, the data repository 1215 of UE 1205 can be communicatively linked to the data repository 1215 of RAN 1210, so that the respective data repositories of UE and RAN can share collected data with each other.

[0152] Another such element could be a training data selection / filter function block 1220. The training data selection / filter function block 1220 can be designed to generate training, validation, and test datasets for model training. Training data can be extracted from the data repository 1215. Data can be selected / filtered based on the specific AI / ML model to be trained. Data can optionally be transformed / enhanced / preprocessed (e.g., normalized) before being loaded into datasets. The training data selection / filter function block 1220 can label data in supervised learning datasets. The generated datasets can then be fed into the model training of the model training function block 1225.

[0153] As noted above, another such element can be the model training function block 1225. This function block can be responsible for training and updating (retraining) the ML models. The selected model can be trained using the input datasets (including training, validation, and testing) from the training data selection / filter function block. The model training function block 1225 can produce trained and tested ML models that are ready for use. The generated trained and tested models can be stored in a model repository 1235.

[0154] Model repository 1235 can be responsible for storing and disclosing AI / ML models (both trained and untrained). One or more trained / updated models can be stored in model repository 1235. Models and model parameters can be discovered and requested by other functional blocks (e.g., the training data selection / filter functional block 1220 and / or the model training functional block 1225). In some embodiments, UE 1205 can discover and request AI / ML models from model repository 1235 of RAN 1210. Likewise, RAN 1210 can discover and / or request AI / ML models from model repository 1235 of UE 1205. In some embodiments, the RAN 1210 can configure models and / or model parameters in the UE 1205's model repository 1235.

[0155] Another such element can be a Model Management Function Block 1240. The Model Management Function Block 1240 can be responsible for managing the AI / ML model produced by the Model Training Function Block 1225. Such management functions can include deploying a trained model, monitoring model performance, and so on. In model deployment, the Model Management Function Block 1240 can allocate and schedule hardware and / or software resources for inference based on received, trained, and tested models. As used herein, "inference" refers to the process of using one or more trained AI / ML models to generate data analyses, actions, policies, and so on, based on input inference data.During performance monitoring, the Model Management Function Block 1240 can decide, based on wireless performance KPIs and model capability metrics, to terminate the running model, start model retraining, select a different model, etc. In embodiments, the Model Management Function Block 1240 of the RAN 1210 may be able to configure model management policies in the UE 1205, as shown.

[0156] Another such element can be an inference data selection / filter function block 1250. The inference data selection / filter function block 1250 can be responsible for generating datasets for model inference at the inference function block 1245, as described below. In particular, inference data can be extracted from the data repository 1215. The inference data selection / filter function block 1250 can select and / or filter the data based on the AI / ML model used. Data can be transformed / augmented / preprocessed using the same transformation / extension / preprocessing as in the training data selection / filtering, as described in relation to function block 1220. The generated inference dataset can be fed into the inference function block 1245.

[0157] Another such element can be the inference function block 1245. The inference function block 1245 can be responsible for performing the inference, as described above. Specifically, the inference function block 1245 can consume the inference data set provided by the inference data selection / filter function block 1250 and produce one or more results. Such results can be or include data analyses, actions, policies, etc. The one or more results can be provided to the performance measurement function block 1230.

[0158] The performance measurement function block 1230 can be configured to measure model performance metrics (e.g., accuracy, model bias, runtime latency, etc.) of deployed and executed models based on one or more inference results for monitoring purposes. Model performance data can be stored in the data repository 1215. 3. EXAMPLE PROCEDURES

[0159] In some embodiments, the electronic device(s), network(s), system(s), chip(s) or component(s), or parts or implementations thereof, may be the Fig. 8-12 or any other figure herein be designed to carry out one or more processes, techniques, or procedures as described herein, or parts thereof. Such a process is in Fig. Figure 13 illustrates a process to be performed by a machine learning (ML) test management services (MnS) generator, which may involve one or more elements of an ML test MnS generator and / or one or more electronic devices that include and / or implement an ML test MnS generator. The process may involve, at 1305, identifying a request to test a machine learning (ML) model based on a first managed object instance (MOI) received by an ML test MnS consumer; at 1310, executing the request to test the ML model by the MnS generator based on that request; and at 1315, delivering an ML test report related to the testing of the ML model to the MnS consumer via a second MOI by the MnS generator.

[0160] Another such process is in Fig. 14 shown. The process of Fig. 14 may involve or relate to a process to be performed by a machine learning (ML) test management services (MnS) consumer, one or more elements of an ML test MnS consumer, and / or one or more electronic devices that include and / or implement an ML test MnS consumer. The process may involve, at 1405, transmitting a request to test a machine learning (ML) model to an ML test MnS provider via a first managed object instance (MOI); and at 1410, identifying, based on a second MOI received from the MnS producer, an ML test report relating to the testing of the ML model by the MnS provider, the testing of the ML model by the MnS provider occurring in response to the receipt of the first MOI.

[0161] For one or more embodiments, at least one of the components shown in one or more of the preceding figures can be configured to perform one or more operations, one or more techniques, one or more processes, and / or one or more methods, as set out in the example section below. For example, the baseband circuit, as described above in conjunction with one or more of the preceding figures, can be configured to operate according to one or more of the examples below. For another example, a circuit arrangement associated with a UE, base station, network element, etc., as described above in conjunction with one or more of the preceding figures, can be configured to operate according to one or more of the examples set out in the example section below. 4. EXEMPLARY IMPLEMENTATIONS

[0162] Additional examples of the methods, devices, systems, and networks described herein include the following non-restrictive implementations. Each of the following non-restrictive examples can stand alone or can be combined in any permutation or combination with any one or more of the other examples provided below or in the entire present disclosure.

[0163] Example 1 includes a procedure for operating an Administrative Service (MnS) producer designed to support ML testing, wherein the procedure includes: initiating machine learning (ML) tests on behalf of an MnS consumer; generating one or more ML test processes to test one or more ML entities; and sending an ML test report to the MnS consumer.

[0164] Example 2 includes the procedure of Example 1 and / or one or more other examples herein, wherein the procedure includes: receiving an ML test request from the MnS consumer.

[0165] Example 3 includes the procedure of Example 2 and / or one or more other examples herein, wherein the procedure includes: responding to the MnS consumer to indicate whether the request is accepted.

[0166] Example 4 includes the procedure of Examples 1-3 and / or one or more other examples herein, wherein the ML testing requirement is for testing a single ML entity or a group of ML entities.

[0167] Example 5 includes the procedure of Example 4 and / or one or more other examples herein, wherein the ML test requirement includes one or more of the following: candidate test data; test requirement source; and / or a DN of the ML entity to be tested.

[0168] Example 6 includes the procedure of Examples 1-5 and / or one or more other examples herein, wherein the procedure includes: receiving an ML test guideline to test the ML entities; and testing the ML entities according to the ML test guideline.

[0169] Example 7 includes the procedure of Example 6 and / or one or more of the other examples herein, wherein the ML test policy includes at least one of the following: a set of ML entities affected by the ML test policy; a time window allowed for testing the one or more ML entities; and / or a maximum number of allowed test iterations, epochs, and / or processes.

[0170] Example 8 includes the procedure of Examples 1-7 and / or one or more other examples herein, wherein the ML testing process includes at least one of the following: priority information; progress status; and / or a set of attributes to abort or suspend the process.

[0171] Example 9 includes the procedure of Examples 1-8 and / or one or more other examples herein, wherein the ML test report includes at least one of the following: an indication of whether consumer-provided test data was used; which consumer-provided test data was used; information regarding the performance of the ML entity; the ML entity test result; the associated ML test requirement; and / or the associated ML test process.

[0172] Example 10 includes the procedure of Examples 1-9 and / or one or more other examples herein, wherein the ML test requirement, ML test process, ML test report and ML test policy are represented by respective managed object instances (MOIs) of corresponding information object classes (IOCs).

[0173] Example 11 includes the procedure of Example 10 and / or one or more of the other examples herein, wherein one or more of the respective MOIs representing the ML test requirement, the ML test process, the ML test report and / or the ML test policy are contained by an MOI of an ML test function or an MOI of an ML training function.

[0174] Example 12 includes the procedure of Examples 1-11 and / or one or more other examples herein, wherein the MnS generator is located in an ML training function.

[0175] Example 13 includes the procedure of Examples 1-12 and / or one or more other examples herein, wherein the MnS generator is located in an ML test function that is separate from the ML training function.

[0176] Example B1 includes a procedure for operating an Administrative Service (MnS) producer designed to support ML training, wherein the procedure includes: initiating machine learning (ML) training on behalf of an MnS consumer; producing one or more ML training processes to train one or more ML entities; and sending an ML training report to the consumer.

[0177] Example B2 includes the procedure of Example B1 and / or one or more other examples herein, wherein the procedure includes: Receiving an ML training request from an MnS consumer.

[0178] Example B3 includes the procedure of Example B2 and / or one or more other examples herein, wherein the procedure includes: responding to the MnS consumer to indicate whether the request is accepted.

[0179] Example B4 includes the procedure of Example B1 and / or one or more other examples herein, wherein the procedure includes: initiating ML training without a request from the MnS consumer.

[0180] Example B5 includes the procedure of Examples B1-B4 and / or one or more other examples herein, wherein the ML training requirement is for training an initial ML entity, retraining an ML entity, or jointly training a group of ML entities.

[0181] Example B6 includes the procedure of Example B5 and / or one or more other examples herein, wherein the ML training requirement includes the inference type of the ML entity for the initial training.

[0182] Example B7 includes the procedure of Examples B5-B6 and / or one or more other examples herein, wherein the ML training request includes an identifier (DN) of an ML entity for retraining.

[0183] Example B8 includes the procedure of Examples B5-B7 and / or one or more other examples herein, wherein the ML training requirement includes an identifier (DN) of an ML entity group for joint training.

[0184] Example B9 includes the procedure of Examples B1-B8 and / or one or more other examples herein, where the ML training process specifies the trained ML entity.

[0185] Example B10 includes the procedure of Examples B1-B9 and / or one or more other examples herein, wherein the ML training report specifies the identifier (DN) of the ML entity generated by the training.

[0186] Example B11 includes the procedure of Examples B1-B10 and / or one or more other examples herein, wherein the ML training report specifies the identifier (DN) of the ML entity group generated by the training.

[0187] Example B12 includes the procedure of Examples B6-B11 and / or one or more other examples herein, where the ML entity group is represented by a Managed Object Instance (MOI).

[0188] Example B13 includes the procedure of Examples B6-B10 and / or one or more other examples herein, where the ML entity is represented by a Managed Object Instance (MOI).

[0189] Example B14 includes the procedure of Examples B12-B13 and / or one or more other examples herein, wherein the ML entity group is associated with one or more ML entities.

[0190] Example B15 includes the procedure of Example B14 and / or one or more other examples herein, wherein the MOI representing the ML entity group contains one or more levels.

[0191] Example B16 includes the procedure of Example B15 and / or one or more other examples herein, wherein each level of the ML entity group is associated with one or more ML entities.

[0192] Example B17 includes the procedure of Example B16 and / or one or more other examples herein, providing the relationship between the ML entities for each level of the ML entity group.

[0193] Example B18 includes the procedure of Example B17 and / or one or more other examples herein, wherein the relationship between the ML entities is 'sequential' or 'parallel' for each level of the ML entity group.

[0194] Example B19 includes the procedure of Examples B14-B18 and / or one or more other examples herein, wherein the MOI representing the ML entity is inherited from an upper IOC.

[0195] Example B20 includes the procedure of Examples B14-B19 and / or one or more other examples herein, wherein the MOI representing the ML entity group is inherited from an upper IOC.

[0196] Example B21 includes the procedure of Examples B15-B20 and / or one or more other examples herein, wherein the MOI representing the ML entity is inherited from the upper IOC.

[0197] Example B22 includes the procedure of Examples B14-B21 and / or one or more other examples herein, wherein the MOI representing the ML entity is contained within an MOI of the ML TrainingFunction-IOC.

[0198] Example B23 includes the procedure of Examples B15-B22 and / or one or more other examples herein, wherein the MOI representing the ML entity group is contained within an MOI of the ML TrainingFunction-IOC.

[0199] Example C1 includes a method to be performed by a machine learning (ML) test management services (MnS) generator, one or more elements of an ML test MnS generator, and / or one or more electronic devices that include and / or implement an ML test MnS generator, wherein the method comprises: identifying, based on a first managed object instance (MOI) received by an ML test MnS consumer, a request to test a machine learning (ML) model; performing, by the MnS generator based on the request, the testing of the ML model; and providing, by the MnS generator to the MnS consumer via a second MOI, an ML test report relating to the testing of the ML model.

[0200] Example C2 includes the procedure of Example C1 and / or any other example herein, wherein the second MOI is based on an ML TestingReport information object class (IOC).

[0201] Example C3 includes the procedure of Example C1-C2 and / or any other example herein, where the first MOI is based on an ML TestingRequestIOC.

[0202] Example C4 includes the procedure of one of the examples C1-C3 and / or another example herein, wherein the second MOI has an attribute that indicates the performance of the ML model during the test.

[0203] Example C5 includes the procedure of Example C4 and / or any other example herein, wherein the attribute indicating the performance of the ML model being tested is a ModelPerformanceTesting attribute.

[0204] Example C6 includes the procedure of one of the examples C1-C5 and / or another example herein, wherein the second MOI has an attribute that indicates a result of testing the ML model.

[0205] Example C7 includes the procedure of Example C6 and / or any other example herein, wherein the attribute indicating the result of testing the ML model is an mlTestingResult attribute.

[0206] Example C8 includes the procedure of one of the examples C1-C7 and / or another example herein, wherein the first MOI has an attribute that specifies a status of the requirement to test the ML model.

[0207] Example C9 includes the procedure of Example C8 and / or any other example herein, where the attribute indicating the status of the request to test the ML model is a requestStatus attribute.

[0208] Example C10 includes the procedure of one of the examples C1-C9 and / or another example herein, wherein the first MOI has an attribute relating to the termination of the request to test the ML model.

[0209] Example C11 includes the procedure of Example C10 and / or any other example herein, where the attribute relating to canceling the request to test the ML model is a cancelRequest attribute.

[0210] Example C12 includes the procedure of one of the examples C1-C11 and / or another example herein, wherein the first MOI has an attribute relating to the suspension of the requirement to test the ML model.

[0211] Example C13 includes the procedure of Example C12 and / or any other example herein, where the attribute relating to the suspension of the request for testing the ML model is a suspendRequest attribute.

[0212] Example D1 includes a procedure to be performed by a machine learning (ML) test management services (MnS) consumer, one or more elements of an ML test MnS consumer, and / or one or more electronic devices that include and / or implement an ML test MnS consumer, wherein the procedure comprises: transmitting, to an ML test MnS provider via a first managed object instance (MOI), a request to test a machine learning (ML) model; and identifying, based on a second MOI received from the MnS producer, an ML test report relating to the testing of the ML model by the MnS provider, wherein the testing of the ML model by the MnS provider occurs in response to the receipt of the first MOI.

[0213] Example D2 includes the procedure of Example D1 and / or any other example herein, wherein the second MOI is based on an ML TestingReport information object class (IOC).

[0214] Example D3 includes the procedure of Example D1-D2 and / or any other example herein, where the first MOI is based on an ML TestingRequestIOC.

[0215] Example D4 includes the procedure of one of the examples D1-D3 and / or another example herein, wherein the second MOI has an attribute that indicates the performance of the ML model during the test.

[0216] Example D5 includes the procedure of Example D4 and / or any other example herein, wherein the attribute indicating the performance of the ML model being tested is a ModelPerformanceTesting attribute.

[0217] Example D6 includes the procedure of one of the examples D1-D5 and / or another example herein, wherein the second MOI has an attribute that specifies a result of testing the ML model.

[0218] Example D7 includes the procedure of Example D6 and / or any other example herein, wherein the attribute indicating the result of testing the ML model is an mlTestingResult attribute.

[0219] Example D8 includes the procedure of one of the examples D1-D7 and / or another example herein, wherein the first MOI has an attribute that specifies a status of the requirement to test the ML model.

[0220] Example D9 includes the procedure of Example D8 and / or any other example herein, where the attribute indicating the status of the request to test the ML model is a requestStatus attribute.

[0221] Example D10 includes the procedure of one of the examples D1-D9 and / or another example herein, wherein the first MOI has an attribute relating to the termination of the request to test the ML model.

[0222] Example D11 includes the procedure of Example D10 and / or any other example herein, where the attribute relating to canceling the request to test the ML model is a cancelRequest attribute.

[0223] Example D12 includes the procedure of one of the examples D1-D11 and / or another example herein, wherein the first MOI has an attribute relating to the suspension of the requirement to test the ML model.

[0224] Example D13 includes the procedure of Example D12 and / or any other example herein, where the attribute relating to the suspension of the request for testing the ML model is a suspendRequest attribute.

[0225] Example Z01 includes one or more computer-readable media containing instructions, wherein execution of the instructions by a processor circuit arrangement is intended to cause the processor circuit arrangement to perform the procedure according to one of Examples 1-D13.

[0226] Example Z02 includes a computer program that contains the instructions from Example Z01.

[0227] Example Z03 includes an application programming interface that defines functions, procedures, variables, data structures and / or protocols for the computer program according to Example Z02.

[0228] Example Z04 includes an API or specification that defines functions, procedures, variables, data structures, protocols and the like that define or include the use of any of Examples 1-D13 or parts thereof, or are otherwise related to any of Examples 1-D13 or parts thereof.

[0229] Example Z05 includes a device comprising a circuit arrangement loaded with the instructions from Example Z01.

[0230] Example Z06 includes a device comprising a circuit arrangement that is operable to execute the instructions of Example Z01.

[0231] Example Z07 includes an integrated circuit comprising the processor circuit arrangement of Example Z01 and / or the one or more computer-readable media of Example Z01.

[0232] Example Z08 includes a computing system comprising one or more computer-readable media and the processor circuit arrangement according to Example Z01.

[0233] Example Z09 includes a facility that contains means for carrying out the instructions according to Example Z01.

[0234] Example Z10 includes a signal that is generated as a result of executing the instructions according to Example Z01.

[0235] Example Z11 includes a data unit that is created as a result of executing the instructions according to Example Z01.

[0236] Example Z12 includes the data unit of Example Z10 and / or one or more other examples herein, where the data unit is a datagram, a network packet, a data frame, a data segment, a protocol data unit (PDU), a service data unit (SDU), a message, or a database object.

[0237] Example Z13 contains a signal that is encoded with the data unit of examples Z11 and / or Z12.

[0238] Example Z14 includes an electromagnetic signal carrying the instructions according to Example Z01.

[0239] Example Z15 includes a facility comprising means for carrying out the procedure according to any one of Examples 1-D13 and / or one or more other examples herein.

[0240] Example Z16 includes an edge compute node that runs a service as part of one or more edge applications instantiated on a virtualization infrastructure, the service being related to one of Examples 1-D13, parts thereof and / or one or more other examples herein.

[0241] Embodiments may be referred to herein individually and / or collectively for the sake of simplicity only, and without any intention of voluntarily limiting the scope of protection of this application to any single aspect or concept, where in fact more than one is disclosed. Although specific aspects have been illustrated and described herein, it is understood that any arrangement calculated to achieve the same purpose may replace the specific aspects shown. This disclosure is intended to cover any and all adaptations or variations of various aspects. Combinations of the above aspects and other aspects not specifically described herein will be apparent to those skilled in the art upon review of the above description. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 501,063

[0001] US 63 / 501,081

[0001] < / datatype> < / datentyp> < / datatype> < / datentyp> < / datatype> < / datentyp> < / datatype>

Claims

[1] Electronic device comprising: one or more processors designed to implement a machine learning (ML) test management services (MnS) generator; and one or more computer-readable media containing instructions that, when executed, are intended to cause the ML-Test-MnS generator to: Identify, based on an initial managed object instance (MOI) received by an ML test MnS consumer, a request to test a machine learning model (ML model); Performing, by the MnS generator based on the requirement, the testing of the ML model; and Providing, by the MnS producer, to the MnS consumer via a second MOI, an ML test report relating to the testing of the ML model. [2] Electronic device according to claim 1, wherein the second MOI is based on an MLTestingReport information object class (IOC). [3] Electronic device according to claim 1, wherein the first MOI is based on an ML TestingRequestIOC. [4] Electronic device according to one of claims 1-3, wherein the second MOI has an attribute that indicates the performance of the ML model during the test. [5] Electronic device according to claim 4, wherein the attribute indicating the performance of the tested ML model is a modelPerformanceTesting attribute. [6] Electronic device according to one of claims 1-3, wherein the second MOI has an attribute that indicates a result of testing the ML model. [7] Electronic device according to claim 6, wherein the attribute indicating the result of testing the ML model is an mlTestingResult attribute. [8] Electronic device according to one of claims 1-3, wherein the first MOI has an attribute that indicates a status of the request to test the ML model. [9] Electronic device according to claim 8, wherein the attribute indicating the status of the request to test the ML model is a requestStatus attribute. [10] Electronic device according to one of claims 1-3, wherein the first MOI has an attribute relating to the termination of the request to test the ML model. [11] Electronic device according to claim 10, wherein the attribute relating to the cancellation of the request to test the ML model is a cancelRequest attribute. [12] Electronic device according to one of claims 1-3, wherein the first MOI has an attribute relating to the suspension of the requirement to test the ML model. [13] Electronic device according to claim 12, wherein the attribute relating to the suspension of the request to test the ML model is a suspendRequest attribute. [14] Electronic device comprising: one or more processors designed to implement a machine learning (ML) test management service (MnS) consumer; and one or more computer-readable media containing instructions which, when executed, are intended to cause the MnS consumer to: Transmitted to an ML test MnS provider via an initial managed object instance (MOI), a request to test an ML model, where the initial MOI is based on an MLTestingRequest information object class (IOC); and Identify, based on a second MOI received from the MnS producer, an ML test report relating to the testing of the ML model by the MnS producer, wherein the second MOI is based on an ML TestingReport-IOC and wherein the testing of the ML model by the MnS provider takes place in response to the receipt of the first MOI. [15] Electronic device according to claim 14, wherein the second MOI has a modelPerformanceTesting attribute. [16] Electronic device according to claim 14, wherein the second MOI has a mlTestingResult attribute. [17] Electronic device according to one of claims 14-16, wherein the first MOI has a requestStatus attribute. [18] Electronic device according to one of claims 14-16, wherein the first MOI has a cancelRequest attribute. [19] Electronic device according to one of claims 14-16, wherein the first MOI has a suspendRequest attribute.

Images