Technologies for machine learning entity training and testing for fifth generation systems
The enhancement of ML training and testing frameworks for fifth-generation systems supports initial, re-training, and joint training of ML entities, addressing the lack of comprehensive standards in current frameworks, and ensuring the reliability and performance of AI/ML inference functions in 5G networks through structured training and testing processes.
Patent Information
- Application Number
- PCT/US2024/028271
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2024-05-08
- Publication Date
- 2024-11-14
AI Technical Summary
Current machine learning (ML) entity training and testing frameworks for fifth-generation systems lack comprehensive support for initial training, re-training, and joint training of ML entities, as well as detailed standard solutions for ML entity testing, which are essential for ensuring the performance and reliability of AI/ML inference functions in 5G networks.
The proposed solution enhances ML training capabilities to support initial, re-training, and joint training of ML entities, incorporating a structured framework for ML entity testing that includes defined Information Object Classes (IOCs) and processes for training and testing management, enabling better data selection, training data preparation, and performance evaluation.
This enhancement ensures that ML entities are thoroughly trained and tested, improving the accuracy and reliability of AI/ML inference functions in 5G networks by providing standardized and comprehensive training and testing processes, thereby enhancing the overall performance and adaptability of ML models.
Smart Images

Figure US2024028271_14112024_PF_FP_ABST
Abstract
Description
[0001] TECHNOLOGIES FOR MACHINE LEARNING ENTITY TRAINING AND TESTING FOR FIFTH GENERATION SYSTEMS
[0002] CROSS REFERENCE TO RELATED APPLICATION
[0003] The present application claims priority to U.S. Provisional Patent Application No. 63 / 501,063, which was filed May 9, 2023; and to U.S. Provisional Patent Application No. 63 / 501,081, which was filed May 9, 2023.
[0004] BACKGROUND
[0005] Artificial intelligence / machine learning (AI / ML) may be used in fifth generation (5G) systems (5GS), including the 5G core (5GC), next generation (NG)-radio access networks (RANs), and management system (e.g., MDA). ML training aspects have been specified in 3GPP TS 28.105 V17.3.0 (2023-03-30) (“[TS28105]”), which supports consumer-requested training and producer-initiated training. However, it is unclear whether and how the ML initial training, ML re-training and joint training of a group of ML entities are supported.
[0006] BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings.
[0008] Figure 1 illustrates an example of ML training requested by a management training (MLT) management services (MnS) consumer, in accordance with various embodiments.
[0009] Figure 2 illustrates an example related to propagation of erroneous information related to ML in a wireless network, in accordance with various embodiments.
[0010] Figure 3 illustrates an example NRM fragment that may be used for ML training, in accordance with various embodiments.
[0011] Figure 4 illustrates an alternative example NRM fragment that may be used for ML training, in accordance with various embodiments.
[0012] Figure 5 illustrates an example inheritance hierarchy that may be used for ML training related NRMs, in accordance with various embodiments.
[0013] Figure 6 illustrates an example network resource model (NRM) fragment that may be used for ML testing, in accordance with various embodiments.
[0014] Figure 7 illustrates an example inheritance hierarchy that may be used for ML testing related NRMs, in accordance with various embodiments.
[0015] Figure 8 illustrates a network in accordance with various embodiments. Figure 9 schematically illustrates a wireless network 900 in accordance with various embodiments.
[0016] Figure 10 is a block diagram illustrating components, according to some example embodiments, able to read instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and perform any one or more of the methodologies discussed herein.
[0017] Figure 11 illustrates a network in accordance with various embodiments.
[0018] Figure 12 illustrates a simplified block diagram of artificial (Al)-assisted communication between a UE and a RAN , in accordance with various embodiments
[0019] Figure 13 depicts an example procedure for practicing the various embodiments discussed herein.
[0020] Figure 14 depicts another example procedure for practicing the various embodiments discussed herein.
[0021] DETAILED DESCRIPTION
[0022] 1. EDGE ENABLER INSTANTIATION ASPECTS
[0023] The present disclosure is related to, inter alia, AI / ML management capabilities and services for 5GS where AI / ML is used, including management and orchestration (e.g., management data analytics (MDA); see e.g., third generation partnership project (3GPP) technical specification (TS) 28.104 vl7.3.0 (2023-03-30) (“[TS28104]”)) and 5G networks (e.g., NWDAF; see e.g., [TS23288]). The present disclosure also describes various aspects of the functionality and service framework for AI / ML management.
[0024] The legacy ML entity is defined as a datatype for an attribute, which cannot be visible and accessible to the consumer as the Managed Object Instance (MOI). The present disclosure provides enhancements to the ML training (e.g., defined in [TS28105]) to support ML initial retraining, ML re-training, and joint training of a group of ML entities.
[0025] Additionally, during the ML entity training phase, after the training and validation, the ML entity may need to be tested to evaluate the performance of the ML entity when it conducts inference using the testing data. If the testing result meets the expectation, the ML entity may proceed with the next phase (such as emulation, loading, inference, etc.), otherwise the ML entity may need to be re-trained.
[0026] The testing may be performed on, at, or by the ML testing function which may be the same as ML training function or a separate function. The ML entity testing has been studied in 3GPP TR 28.908, and the normative solutions are to be specified in [TS28105]. The solutions for ML entity testing studied in 3GPP TR 28.908 do not contain enough details for testing ML models and / or to be the standard solutions The following discussion provides various examples of Information Object Class (IOC) names and attribute names, however, the IOCS and attributes can have alternative names to those provided infra.
[0027] 1.1. ML TRAINING USE CASES AND REQUIREMENTS
[0028] 1.1.1. DESCRIPTION
[0029] In an operational environment, before the ML entity is deployed to conduct inference, the ML model associated with the ML entity needs to be trained (e.g., by ML training function which may be a separate or an external entity to the AI / ML inference function). The ML training can be initial training of an ML entity or re-training of an existing ML entity. In the present disclosure, the term “ML entity training” may refer to ML model training associated with an ML entity.
[0030] The ML entity is trained by the ML training (MLT) MnS producer, and the training can be triggered by request(s) from one or more MLT MnS consumer(s), or initiated by the MLT MnS producer (e.g., as a result of model performance evaluation).
[0031] 1.1.2. USE CASES
[0032] I.1.2.1. ML TRAINING REQUESTED BY CONSUMER
[0033] The ML training capabilities are provided by an MLT MnS producer to one or more consumer(s). See e.g., Figure 1. As shown in Figure 1, the ML training may be triggered by the request(s) from one or more MLT MnS consumer(s). The consumer may be for example a network function, a management function, an operator, or another functional differentiation To trigger an ML training, the MLT MnS consumer requests the MLT MnS producer to train the ML model. In the ML training request, the consumer should specify the inference type which indicates the function or purpose of the ML entity (e.g., CoverageProblemAnalysis). The MLT MnS producer can perform the training according to the designated inference type. The consumer may provide the data source(s) that contain(s) the training data which are considered as inputs candidates for training. To obtain the valid training outcomes, consumers may also designate their requirements for model performance (e.g., accuracy, etc) in the training request.
[0034] The MLT MnS producer provides a response to the consumer indicating whether the request was accepted. If the request is accepted, the MLT MnS producer decides when to start the ML training with consideration of the request(s) from the consumer(s). Once the training is decided, the producer may perform one or more of the following:
[0035] - select the training data, with consideration of the consumer-provided candidate training data. Because the training data directly influences the algorithm and performance of the trained ML Entity, the MLT MnS producer may examine the consumer's provided training data and decide to select none, some or all of them. In addition, the MLT MnS producer may select some other training data that are available;
[0036] - train the ML entity using the selected training data; and
[0037] - provide the training results (including the identifier of the initially trained ML entity or the version number of the re-trained ML entity, training performance, etc.) to the MLT MnS consumer(s).
[0038] 1.1.2.2. ML TRAINING INITI TED BY PRODUCER
[0039] Alternatively, in some embodiments, the ML training may be initiated by the MLT MnS producer, for instance as a result of performance evaluation of the ML model, based on feedback or new training data received from the consumer, or when new training data which are not from the consumer describing the new network status / events become available.
[0040] When the MLT MnS producer decides to start the ML training, the producer may perform one or more of the following:
[0041] - select the training data;
[0042] - train the ML entity using the selected training data; and
[0043] - provide the training results (including the identifier of the initially trained ML entity or the version number of the re-trained ML entity, training performance, etc.) to the MLT MnS consumer(s) who have subscribed to receive the ML training results.
[0044] 1.1.2.3. ML MODEL AND ML ENTITY SELECTION
[0045] For a given machine learning-based use case, different entities that apply the respective ML model or AI / ML inference function may have different inference requirements and capabilities. For example, one consumer with specific responsibility and wish to have an AI / ML inference function supported by an ML model or entity trained for city central business district where mobile users move at speeds not exceeding 30 km / hr. On the other hand, another consumer, for the same use case may support a rural environment and as such wishes to have an ML model and AI / ML inference function fitting that type of environment. The different consumers need to know the available versions of ML entities, with the variants of trained ML models or entities and to select the appropriate one for their respective conditions.
[0046] Besides, there may be no guarantee that the available ML models / entities have been trained according to the characteristics that the consumers expect. As such the consumers need to know the conditions for which the ML models or ML entities have been trained to then enable them to select the models that are best fit to their conditions and needs.
[0047] The models that have been trained may differ in terms of complexity and performance. For example, a generic comprehensive and complex model may have been trained in a cloudlike environment but when such a model cannot be used in the gNB and instead, a less complex model, trained as a derivative of this generic model, could be a better candidate. Moreover, multiple less complex models could be trained with different level of complexity and performance which would then allow different relevant models to be delivered to different network functions depending on operating conditions and performance requirements. The network functions need to know the alternative models available and interactively request and replace them when needed and depending on the observed inference-related constraints and performance requirements.
[0048] 1.1.2.4. MANAGING ML TRAINING PROCESSES
[0049] This machine learning capability relates to means for managing and controlling ML model / entity training processes.
[0050] To achieve the desired outcomes of any machine learning relevant use-case, the ML model applied for such analytics and decision making, needs to be trained with the appropriate data. The training may be undertaken in managed function or in a management function.
[0051] In either case, the network (or the operations, administration, and maintenance (OAM) system thereof) may not only need to have the required training capabilities, but may also need to also have the means to manage the training of the ML models / entities. The consumers need to be able to interact with the training process, e.g., to suspend or restart the process; and also need to manage and control the requests related to any such training process.
[0052] 1.1.2.5. HANDLING ERRORS IN DATA AND ML DECISIONS
[0053] Traditionally, the ML models / entities (e.g., ML entityl and ML entity2 in Figure 2) are trained on good quality data, i.e. data that were collected correctly and reflected the real network status to represent the expected context in which the ML entity is meant to operate. Good quality data is void of errors, such as:
[0054] - Imprecise measurements, with added noise (such as reference signal received power (RSRP), signal to interference plus noise ratio (SINR), or quality of experience (QoE) estimations).
[0055] - Missing values or entire records, e.g., because of communication link failures.
[0056] - Records which are communicated with a significant delay (in case of online measurements).
[0057] Without errors, an ML entity can depend on a few precise inputs, and may not need to exploit the redundancy present in the training data. However, during inference, the ML entity is very likely to come across these inconsistencies. When this happens, the ML entity may show high error in the inference outputs, even if redundant and uncorrupted data are available from other sources.
[0058] As such the system may need to account for errors and inconsistencies in the input data and the consumers should deal with decisions that are made based on such erroneous and inconsistent data. The system should (1) enable functions to undertake the training in a way that prepares the ML entities to deal with the errors in the training data, i.e. to identify the errors in the data during training; and / or (2) enable the MLT MnS consumers to be aware of the possibility of erroneous input data that are used by the ML entity.
[0059] 1.1.1.1. ML ENTITY JOINT TRAINING
[0060] An AI / ML inference function may use one or more ML entities to perform the inference(s). When multiple ML entities are employed, these ML entities may operate together in a coordinated way, such as in a sequence, or even a more complicated structure. In this case, any change in the performance of one ML entity may impact another, and consequently impact the overall performance of the whole AI / ML inference function.
[0061] There are different ways in which the group of ML entities may coordinate. An example is the case where the output of one ML entity can be used as input to another ML entity forming a sequence of interlinked ML entities. Another example is the case where multiple ML entities provide the output in parallel (either the same output type where outputs may be merged (e.g., using weights), or their outputs are needed in parallel as input to another ML entity. The group of ML entities needs to be employed in a coordinated way to support an AI / ML inference function. In various embodiments, how to group the ML entities and designate the relations between the ML entities within the group may be vendor- specific.
[0062] Therefore, it may be desirable that these coordinated ML entities can be trained or retrained jointly, so that the group of these ML entities can complete a more complex task jointly with better performance.
[0063] The ML entity joint training may be initiated by the MnS producer or the MnS consumer, with the grouping of the ML entities shared by the MnS producer with the MnS consumer.
[0064] 1.1.1.2. CONSUMER-REQUESTED ML ENTITY TESTING
[0065] After completing the ML entity training, and when the performance of the trained ML entity meets the expectations on both training and validation data, the ML entity is made available to the consumer(s) via the ML training report. Before applying the ML entity to the target AI / ML inference function or emulation environment, the ML testing MnS producer (which may the same as or different from the ML trainining MnS producer) may need to allow the consumer to evaluate the performance of the ML entity via the ML testing process using the consumer’s provided testing data. The testing data may have the same pattern as the input part of the training data. The consumer (e.g., an operator) may also stop, suspend, or resume the ML entity testing process when necessary.
[0066] After the testing is done, the MnS producer may report the performance of the ML entity on the testing data to the MnS consumer. If the testing performance is not acceptable or does not meet the pre-defined requirements, the consumer may request the ML training producer to re-train the ML entity with specific training data and / or performance requirements.
[0067] I.I.I.3. CONTROL OF PRODUCER-INITIATED ML ENTITY TESTING The ML entity testing may be also initiated by the MnS producer, after the ML entity is trained and validated. A consumer (e.g., an operator) may need to control (e.g., stop, suspend, resume, and / or the like) and manage the testing process that is initiated by the MnS producer.
[0068] For example, the operator may wish to define the policies (e.g, allowed time window, maximum number of testing, and / or the like) for the testing of a given ML entity may be executed. REQUIREMENTS FOR ML ENTITY TESTING
[0069] Table 1 provides at least some example requirements that may be related to ML entity testing as follows:
[0070] Table 1
[0071]
[0072] 1.2. SOLUTIONS
[0073] 1.2.1. RELATIONSHIPS (TRAINING)
[0074] This clause relates to the set of classes (e.g., IOCS) that encapsulates the information relevant to ML model training. Details related to unified modelling language (UML) semantics may be seen at, for example, 3GPP TS 32.156 vl8.1.0 (2023-03-30) (“[TS32156]”)). Examples of the IOCs may be seen, for example, in Figure 3 which depicts an example NRM fragment related to ML training.
[0075] 1.2.2. NHERITANCE (TRAINING) This clause relates an IOC inheritance hierarchy for ML training related NRMs. An example of such a hierarchy may be seen, for example, in Figure 4.
[0076] 1.2.3. RELATIONSHIPS (TESTING)
[0077] This clause relates to the set of classes (e.g., IOCs) that encapsulates the information relevant to ML model testing. Examples of the IOCs may be seen, for example, in Figures 5 and 6 which depict example NRM fragments related to ML testing.
[0078] 1.2.4. INHERITANCE (TESTING) This clause relates an IOC inheritance hierarchy for ML testing related NRMs. An example of such a hierarchy may be seen, for example, in Figure 7.
[0079] 1.2.5. MLTRAININGFUNCTION
[0080] The IOC MLTrainingFunction represents the entity that undertakes ML training. The entity represented by MLTrainingFunction MOI supports training of one or more MLEntity(s) and joint training of one or more groups of ML entities. Example attributes of the MLTrainingFunction MOI may be as shown in Table 2
[0081] Table 2
[0082] 1.2.6. MLTRAININGREQUEST
[0083] The IOC MLTrainingRequest represents the ML model training request that is created by the ML training MnS consumer. The MLTrainingRequest MOI is contained under one MLTrainingFunction MOI. Each MLTrainingRequest is associated to at least one MLEntity.
[0084] The MLTrainingRequest may have a source to identify where it is coming from, and which may be used to prioritize the training resources for different sources. The sources may be for example the network functions, operator roles, or other functional differentiations.
[0085] Each MLTrainingRequest may indicate the expectedRunThneContext that describes the specific conditions for which the MLEntity should be trained for.
[0086] In case the request is accepted, the ML training MnS producer decides when to start the ML training. Once the MnS producer decides to start the training based on the request, the ML training MnS producer instantiates one or more MLTrainingProcess MOI(s) that are responsible to perform one or more of the following:
[0087] - collects (more) data for training, if the training data are not available or the data are available but not sufficient for the training;
[0088] - prepares and selects the required training data, with consideration of the consumer’ s request provided candidate training data if any. The ML training MnS producer may examine the consumer's provided candidate training data and select none, some or all of them for training. In addition, the ML training MnS producer may select some other training data that are available in order to meet the consumer’ s requirements for the MLentity training; and / or
[0089] - trains the MLEntity using the selected and prepared training data.
[0090] The MLTrainingRequest may have a requeststatus field to represent the status of the specific MLTrainingRequest:
[0091] - The attribute values are "NOT_STARTED", "TRAINING_IN_PROGRESS", "SUSPENDED", "FINISHED", and "CANCELLED".
[0092] - When value turns to "TRAINING_IN_PROGRESS", the ML training MnS producer instantiates one or more MLTrainingProcess MOI(s) representing the training process(es) being performed per the request and notifies the MLT MnS consumer(s) who subscribed to the notification.
[0093] When all of the training process associated to this request are completed, the value turns to "FINISHED".
[0094] Attributes of MLTrainingRequest may be as shown in Table 3
[0095] Table 3
[0096] 1.2.7. MLTRAININGREPORT
[0097] The IOC MLTrainingReport represents the ML model training report that is provided by the training MnS producer. The MLTrainingReport MOI is contained under one MLTrainingFunction MOI. Attributes of MLTraining Report may be as shown in Table 4, and constraints on the attributes of Table 4 may be as shown in Table 5.
[0098] Table 4
[0099] Table 5 1.2.8. MLTRAININGPROCESS
[0100] The IOC MLTrainingProcess represents the ML training process. One MLTrainingProcess MOI may be instantiated for each MLTrainingRequest MOI or a set of MLTrainingRequest MOIs. For each MLEntity under training, a MLTrainingProcess is instantiated, i.e. an MLTrainingProcess is associated with exactly one MLEntity. The MLTrainingProcess may be associated with one or more MLTrainingRequest MOI.
[0101] The MLTrainingProcess does not have to correspond to a specific MLTrainingRequest, i.e. a MLTrainingRequest does not have to be associated to a specific MLTrainingProcess. The MLTrainingProcess may be managed separately from the MLTrainingRequest MOIs, e.g. the MLTrainingRequest MOI may come from consumers which are network functions while the operator may wish to manage the MLTrainingProcess that is instantiated following the requests. Thus, the MLTrainingProcess may be associated to either one or more MLTrainingRequest MOI.
[0102] Each MLTrainingProcess instance needs to be managed differently from the related MLEntity, although the MLTrainingProcess may be associated to only one MLEntity. Lor example, the MLTrainingProcess may be triggered to start with a specific version of the MLEntity and multiple MLTrainingProcess instances may be triggered for different versions of the MLEntity. In either case the MLTrainingProcesse instances are still associated with the same MLEntity but are managed separately from the MLEntity.
[0103] Each MLTrainingProcess has a priority that may be used to prioritize the execution of different MLTrainingProcesse instances. By default, the priority of the MLTrainingProcess may be related in a 1:1 manner with the priority of the MLTrainingRequest for which the MLTrainingProcess is instantiated.
[0104] Each MLTrainingProcess may have one or more termination conditions used to define the points at which the MLTrainingProcess may terminate.
[0105] The "progressStatus" attribute represents the status of the ML model training and includes information the ML training MnS consumer can use to monitor the progress and results. The data type of this attribute is "ProcessMonitor" (see 3GPP TS 28.622
[0012] ). The following specializations are provided for this data type for the ML training process:
[0106] - The "status" attribute values are "RUNNING", "CANCELLING", "SUSPENDED", "FINISHED", and "CANCELLED". The other values are not used.
[0107] - The "timer" attribute is not used.
[0108] - When the "status" is equal to "RUNNING" the "progressStatelnfo" attribute shall indicate one of the following states: "COLLECTING_DATA",
[0109] "PREPARING_TRAINING_DATA", "TRAINING".
[0110] - No specifications are provided for the "results tatelnfo" attribute. Vendor specific information may be provided though.
[0111] When the training is completed with "status" equal to "FINISHED", the MLT MnS producer provides the training report, by creating an MLTrainingReport MOI, to the MLT MnS consumer. Attributes of the MLTrainingProcess may be as shown in Table 6, and constraints related to the MLTraining Process may be as shown in Table 7.
[0112] Table 6 Table 7 1.2.9. MLENTITY
[0113] This IOC represents the ML entity. The algorithm of ML model or ML entity is not to be standardized. The MLEntity may contain 3 types of contexts - TrainingContext which is the context under which the MLEntity has been trained, the ExpectedRunTimeContext which is the context where an MLEntity is expected to be applied or / and the RunTimeContext which is the context where the ML entity is being applied. Attributes of MLEntity may be as shown in Table 8.
[0114] Table 8
[0115] In some examples, the trainingContext is a support qualifier. In some examples, the trainingContext represents the status and conditions related to training and should be added when training is completed
[0116] 1.2.10. MLENTITYGROUP
[0117] This IOC represents the group of ML entities, which can be trained jointly and perform inference in a coordinated way. How the ML entities coordinate within the group is described by the relations between the ML entities. The grouping of the ML entities and their relations are decided by MnS producer and cannot be changed by the MnS consumer. This IOC is created by the MnS producer and cannot be modified by the MnS consumer.
[0118] 1.2.11. MLENTITIESRELATIONPERLEVEL «dataType»
[0119] This data type represents the relations between the ML entities for each level of an ML entity group. The relation of the ML entities within each level could be in sequence or in parallel. The output of one level of the ML entity group is used as input for the next (higher) level. The level is indicated by a number, starting from one, and a bigger number indicates a higher level. Attributes related to MLEntitiesRelationPerLevel may be as shown in Table 9. Table 9
[0120] 1.2.12. ATTRIBUTE PROPERTIES
[0121] Properties related to one or more of the previously-listed or described attributes (e.g., training-related attributes and / or some other attribute described herein) may be as shown in Table 10.
[0122] Table 10
[0123] 1.2.13. MLTESTINGFUNCTION
[0124] The ML entity testing may be conducted in the ML training function, or in a separate function. In case the ML entity testing is conducted in a function separate from the ML training function, the IOC MLTestingFunction is instantiated and represents the logical function that undertakes ML entity testing. The entity represented by MLTestingFunction MOI supports testing of one or more MLEntity(s). Attributes related to MLTestingFunction may be as shown in Table 11.
[0125] Table 11
[0126] 1.2.14. MLTESTINGREQUEST
[0127] The IOC MLTestingRequest represents the ML entity testing request that is created by the ML testing MnS consumer.
[0128] The MLTestingRequest MOI is contained under one MLTestingFunction MOI or MLTrainingFunction MOI which represents the logical function that conducts the ML entity testing. Each MLTestingRequest is associated to at least one MLEntity.
[0129] The MLTestingRequest may have a source to identify where it is coming from, and which may be used to prioritize the testing resources for different sources. The sources may be for example the network functions, operator roles, or other functional differentiations.
[0130] In case the request is accepted, the ML testing MnS producer decides when to start the ML testing. Once the MnS producer decides to start the testing based on the request, the ML training MnS producer instantiates one or more MLTestingProcess MOI(s) that are responsible to perform the followings:
[0131] - collects (more) data for testing, if the testing data are not available or the data are available but not sufficient for the training;
[0132] - prepares and selects the required testing data, with consideration of the consumer’s request provided testing data if any. The ML training MnS producer may examine the consumer's provided testing data and select none, some or all of them for testing. In addition, the ML training MnS producer may select some other testing data that are available in order to meet the consumer’s requirements for the MLentity training;
[0133] - tests the MLEntity by performing inference using the selected testing data,
[0134] - reports the performance of the MLEntity when it performs on the selected testing data.
[0135] The MLTestingRequest may have a requeststatus field to represent the status of the request:
[0136] - The attribute values are "NOT_STARTED", "TRAINING_IN_PROGRESS", "SUSPENDED", "FINISHED", and "CANCELLED".
[0137] - When value turns to "TESTING_IN_PROGRESS", the ML testing MnS producer instantiates one or more MLTestingProcess MOI(s) representing the testing process(es) being performed per the request and notifies the MnS consumer(s) who subscribed to the notification.
[0138] When all of the testing processes associated to this request are completed, the value turns to "FINISHED".
[0139] Attributes related to MLTesting Request may be as shown in Table 12.
[0140] Table 12
[0141] 1.2.15. MLTESTINGREPORT
[0142] The IOC MLTestingReport represents the ML entity testing report that is provided by the ML testing MnS producer (e.g., to the MLT MnS consumer). The MLTestingReport MOI is contained under one MLTestingFunction MOI or MLTrainingFunction MOI which represents the logical function that conducts the ML entity testing. Attributes related to MLTestingReport may be as shown in Table 13. Example attribute constraints that may be related to MLTestingReport are shown in Table 14. Table 13
[0143] Table 14
[0144] 1.2.16. MLTESTINGPROCESS
[0145] The IOC MLTestingProcess represents the ML testing process.
[0146] One or more MLTestingProcess MOI may be instantiated for each MLTestingRequest MOI or a set of MLTestingRequest MOIs.
[0147] For each MLEntity MOI under testing, an MLTestingProcess is instantiated, i.e. an MLTestingProcess MOI is associated with exactly one MLEntity MOI. The MLTrainingProcess may be associated with one or more MLTrainingRequest MOI.
[0148] The MLTestingProcess does not have to correspond to a specific MLTestingRequest, i.e., a MLTestingRequest does not have to be associated to a specific MLTestingProcess.
[0149] Each MLTestingProcess has a priority that may be used to prioritize the execution of different MLTestingProcess instances. By default, the priority of the MLTestingProcess may be related in a 1:1
[0150] The "progressStatus" attribute represents the status of the ML entity testing and includes information the ML training MnS consumer can use to monitor the progress and results. The data type of this attribute is "ProcessMonitor" (see e.g., [TS28622]). The following specializations are provided for this data type for the ML entity testing process:
[0151] - The "status" attribute values are "RUNNING", "CANCELLING", "SUSPENDED", "FINISHED", and "CANCELLED". The other values are not used.
[0152] - The "timer" attribute is not used.
[0153] - When the "status" is equal to "RUNNING" the "progressStatelnfo" attribute shall indicate one of the following states: "COLLECTING_DATA", "PREP ARING_TESTING_D ATA", "TESTING".
[0154] - No specifications are provided for the "results tatelnfo" attribute. Vendor specific information may be provided though. When the testing is completed with "status" equal to "FINISHED", the ML testing MnS producer provides the testing report, by creating an MLTestingReport MOI, to the MnS consumer.
[0155] Example attributes that may be related to MLT Testing Process are shown in Table 15.
[0156] Table 15
[0157] 1.2.17. MLTESTINGPOLICY
[0158] This IOC represents the policy for MnS producer-initiated ML entity testing. One ML testing policy may be applicable to one or more ML entities. Attributes related to ML Testing Policy may be as shown in Table 16.
[0159] Table 16
[0160] 1.2.18. TESTINGPOLICY «dataType»
[0161] This data type specifies the ML testing policy for one or more ML entities. Attributes related to TestingPolicy may be as shown in Table 17.
[0162] Table 17
[0163] 1.2.19. TIMEWINDOW «dataType»
[0164] This «dataType» represents a time duration. Attributes related to TimeWindow may be as shown in Table 18.
[0165] Table 18
[0166] 1.2.20. ATTRIBUTE PROPERTIES Example properties of, or related to, one or more of the attributes herein (e.g., one or more of the testing related attributes and / or some other attribute herein), may be as shown in Table 19. Table 19
[0167] 2. CELLULAR NETWORK ASPECTS
[0168] Figures 8-12 illustrate various systems, devices, and components that may implement aspects of disclosed embodiments. Figure 8 illustrates a network 800 in accordance with various embodiments. The network 800 may operate in a manner consistent with 3GPP technical specifications for LTE or 5G / NR systems. However, the example embodiments are not limited in this regard and the described embodiments may apply to other networks that benefit from the principles described herein, such as future 3GPP systems, or the like. The network 800 may include a UE 802, which may include any mobile or non-mobile computing device designed to communicate with a RAN 804 via an over-the-air connection. The UE 802 may be communicatively coupled with the RAN 804 by a Uu interface. The UE 802 may be, but is not limited to, a smartphone, tablet computer, wearable computer device, desktop computer, laptop computer, in-vehicle infotainment, in-car entertainment device, instrument cluster, head-up display device, onboard diagnostic device, dashtop mobile equipment, mobile data terminal, electronic engine management system, electronic / engine control unit, electronic / engine control module, embedded system, sensor, microcontroller, control module, engine management system, networked appliance, machine-type communication device, M2M or D2D device, loT device, etc.
[0169] In some embodiments, the network 800 may include a plurality of UEs coupled directly with one another via a sidelink interface. The UEs may be M2M / D2D devices that communicate using physical sidelink channels such as, but not limited to, PSBCH, PSDCH, PSSCH, PSCCH, PSFCH, etc.
[0170] In some embodiments, the UE 802 may additionally communicate with an AP 806 via an over-the-air connection. The AP 806 may manage a WLAN connection, which may serve to offload some / all network traffic from the RAN 804. The connection between the UE 802 and the AP 806 may be consistent with any IEEE 802.11 protocol, wherein the AP 806 could be a wireless fidelity (Wi-Fi®) router. In some embodiments, the UE 802, RAN 804, and AP 806 may utilize cellular- WLAN aggregation (for example, LWA / LWIP). Cellular- WLAN aggregation may involve the UE 802 being configured by the RAN 804 to utilize both cellular radio resources and WLAN resources.
[0171] The RAN 804 may include one or more access nodes, for example, AN 808. AN 808 may terminate air-interface protocols for the UE 802 by providing access stratum protocols including RRC, PDCP, RLC, MAC, and LI protocols. In this manner, the AN 808 may enable data / voice connectivity between CN 820 and the UE 802. In some embodiments, the AN 808 may be implemented in a discrete device or as one or more software entities running on server computers as part of, for example, a virtual network, which may be referred to as a CRAN or virtual baseband unit pool. The AN 808 be referred to as a BS, gNB, RAN node, eNB, ng-eNB, NodeB, RSU, TRxP, TRP, etc. The AN 808 may be a macrocell base station or a low power base station for providing femtocells, picocells or other like cells having smaller coverage areas, smaller user capacity, or higher bandwidth compared to macrocells.
[0172] In embodiments in which the RAN 804 includes a plurality of ANs, they may be coupled with one another via an X2 interface (if the RAN 804 is an LTE RAN) or an Xn interface (if the RAN 804 is a 5G RAN). The X2 / Xn interfaces, which may be separated into control / user plane interfaces in some embodiments, may allow the ANs to communicate information related to handovers, data / context transfers, mobility, load management, interference coordination, etc.
[0173] The ANs of the RAN 804 may each manage one or more cells, cell groups, component carriers, etc. to provide the UE 802 with an air interface for network access. The UE 802 may be simultaneously connected with a plurality of cells provided by the same or different ANs of the RAN 804. For example, the UE 802 and RAN 804 may use carrier aggregation to allow the UE 802 to connect with a plurality of component carriers, each corresponding to a Pcell or Scell. In dual connectivity scenarios, a first AN may be a master node that provides an MCG and a second AN may be secondary node that provides an SCG. The first / second ANs may be any combination of eNB, gNB, ng-eNB, etc.
[0174] The RAN 804 may provide the air interface over a licensed spectrum or an unlicensed spectrum. To operate in the unlicensed spectrum, the nodes may use LAA, eLAA, and / or feLAA mechanisms based on CA technology with PCells / Scells. Prior to accessing the unlicensed spectrum, the nodes may perform medium / carrier- sensing operations based on, for example, a listen-before-talk (LBT) protocol.
[0175] In V2X scenarios the UE 802 or AN 808 may be or act as a RSU, which may refer to any transportation infrastructure entity used for V2X communications. An RSU may be implemented in or by a suitable AN or a stationary (or relatively stationary) UE. An RSU implemented in or by: a UE may be referred to as a “UE-type RSU”; an eNB may be referred to as an “eNB-type RSU”; a gNB may be referred to as a “gNB-type RSU”; and the like. In one example, an RSU is a computing device coupled with radio frequency circuitry located on a roadside that provides connectivity support to passing vehicle UEs. The RSU may also include internal data storage circuitry to store intersection map geometry, traffic statistics, media, as well as applications / software to sense and control ongoing vehicular and pedestrian traffic. The RSU may provide very low latency communications required for high speed events, such as crash avoidance, traffic warnings, and the like. Additionally or alternatively, the RSU may provide other cellular / WLAN communications services. The components of the RSU may be packaged in a weatherproof enclosure suitable for outdoor installation, and may include a network interface controller to provide a wired connection (e.g., Ethernet) to a traffic signal controller or a backhaul network.
[0176] In some embodiments, the RAN 804 may be an LTE RAN 810 with eNBs, for example, eNB 812. The LTE RAN 810 may provide an LTE air interface with the following characteristics: SCS of 15 kHz; CP-OFDM waveform for DL and SC-FDMA waveform for UL; turbo codes for data and TBCC for control; etc. The LTE air interface may rely on CSI-RS for CSI acquisition and beam management; PDSCH / PDCCH DMRS for PDSCH / PDCCH demodulation; and CRS for cell search and initial acquisition, channel quality measurements, and channel estimation for coherent demodulation / detection at the UE. The LTE air interface may operating on sub-6 GHz bands.
[0177] In some embodiments, the RAN 804 may be an NG-RAN 814 with gNBs, for example, gNB 816, or ng-eNBs, for example, ng-eNB 818. The gNB 816 may connect with 5G-enabled UEs using a 5G NR interface. The gNB 816 may connect with a 5G core through an NG interface, which may include an N2 interface or an N3 interface. The ng-eNB 818 may also connect with the 5G core through an NG interface, but may connect with a UE via an LTE air interface. The gNB 816 and the ng-eNB 818 may connect with each other over an Xn interface.
[0178] In some embodiments, the NG interface may be split into two parts, an NG user plane (NG-U) interface, which carries traffic data between the nodes of the NG-RAN 814 and a UPF 848 (e.g., N3 interface), and an NG control plane (NG-C) interface, which is a signaling interface between the nodes of the NG-RAN814 and an AMF 844 (e.g., N2 interface).
[0179] The NG-RAN 814 may provide a 5G-NR air interface with the following characteristics: variable SCS; CP-OFDM for DL, CP-OFDM and DFT-s-OFDM for UL; polar, repetition, simplex, and Reed-Muller codes for control and LDPC for data. The 5G-NR air interface may rely on CSI-RS, PDSCH / PDCCH DMRS similar to the LTE air interface. The 5G-NR air interface may not use a CRS, but may use PBCH DMRS for PBCH demodulation; PTRS for phase tracking for PDSCH; and tracking reference signal for time tracking. The 5G-NR air interface may operating on FR1 bands that include sub-6 GHz bands or FR2 bands that include bands from 24.25 GHz to 52.6 GHz. The 5G-NR air interface may include an SSB that is an area of a downlink resource grid that includes PSS / SSS / PBCH.
[0180] In some embodiments, the 5G-NR air interface may utilize BWPs for various purposes. For example, BWP can be used for dynamic adaptation of the SCS. For example, the UE 802 can be configured with multiple BWPs where each BWP configuration has a different SCS. When a BWP change is indicated to the UE 802, the SCS of the transmission is changed as well. Another use case example of BWP is related to power saving. In particular, multiple BWPs can be configured for the UE 802 with different amount of frequency resources (for example, PRBs) to support data transmission under different traffic loading scenarios. A BWP containing a smaller number of PRBs can be used for data transmission with small traffic load while allowing power saving at the UE 802 and in some cases at the gNB 816. A BWP containing a larger number of PRBs can be used for scenarios with higher traffic load.
[0181] The RAN 804 is communicatively coupled to CN 820 that includes network elements to provide various functions to support data and telecommunications services to customers / subscribers (for example, users of UE 802). The components of the CN 820 may be implemented in one physical node or separate physical nodes. In some embodiments, NFV may be utilized to virtualize any or all of the functions provided by the network elements of the CN 820 onto physical compute / storage resources in servers, switches, etc. A logical instantiation of the CN 820 may be referred to as a network slice, and a logical instantiation of a portion of the CN 820 may be referred to as a network sub-slice. In some embodiments, the CN 820 may be an LTE CN 822, which may also be referred to as an EPC. The LTE CN 822 may include MME 824, SGW 826, SGSN 828, HSS 830, PGW 832, and PCRF 834 coupled with one another over interfaces (or “reference points”) as shown. Functions of the elements of the LTE CN 822 may be briefly introduced as follows.
[0182] The MME 824 may implement mobility management functions to track a current location of the UE 802 to facilitate paging, bearer activation / deactivation, handovers, gateway selection, authentication, etc.
[0183] The SGW 826 may terminate an SI interface toward the RAN and route data packets between the RAN and the LTE CN 822. The SGW 826 may be a local mobility anchor point for inter-RAN node handovers and also may provide an anchor for inter-3GPP mobility. Other responsibilities may include lawful intercept, charging, and some policy enforcement.
[0184] The SGSN 828 may track a location of the UE 802 and perform security functions and access control. In addition, the SGSN 828 may perform inter-EPC node signaling for mobility between different RAT networks; PDN and S-GW selection as specified by MME 824; MME selection for handovers; etc. The S3 reference point between the MME 824 and the SGSN 828 may enable user and bearer information exchange for inter-3GPP access network mobility in idle / active states.
[0185] The HSS 830 may include a database for network users, including subscription-related information to support the network entities’ handling of communication sessions. The HSS 830 can provide support for routing / roaming, authentication, authorization, naming / addressing resolution, location dependencies, etc. An S6a reference point between the HSS 830 and the MME 824 may enable transfer of subscription and authentication data for authenticating / authorizing user access to the LTE CN 820.
[0186] The PGW 832 may terminate an SGi interface toward a data network (DN) 836 that may include an application / content server 838. The PGW 832 may route data packets between the LTE CN 822 and the data network 836. The PGW 832 may be coupled with the SGW 826 by an S5 reference point to facilitate user plane tunneling and tunnel management. The PGW 832 may further include a node for policy enforcement and charging data collection (for example, PCEF). Additionally, the SGi reference point between the PGW 832 and the data network 8 36 may be an operator external public, a private PDN, or an intra-operator packet data network, for example, for provision of IMS services. The PGW 832 may be coupled with a PCRF 834 via a Gx reference point.
[0187] The PCRF 834 is the policy and charging control element of the LTE CN 822. The PCRF 834 may be communicatively coupled to the app / content server 838 to determine appropriate QoS and charging parameters for service flows. The PCRF 832 may provision associated rules into a PCEF (via Gx reference point) with appropriate TFT and QCI.
[0188] In some embodiments, the CN 820 may be a 5GC 840. The 5GC 840 may include an AUSF 842, AMF 844, SMF 846, UPF 848, NSSF 850, NEF 852, NRF 854, PCF 856, UDM 858, and AF 860 coupled with one another over interfaces (or “reference points”) as shown. Functions of the elements of the 5GC 840 may be briefly introduced as follows.
[0189] The AUSF 842 may store data for authentication of UE 802 and handle authentication- related functionality. The AUSF 842 may facilitate a common authentication framework for various access types. In addition to communicating with other elements of the 5GC 840 over reference points as shown, the AUSF 842 may exhibit an Nausf service-based interface.
[0190] The AMF 844 may allow other functions of the 5GC 840 to communicate with the UE 802 and the RAN 804 and to subscribe to notifications about mobility events with respect to the UE 802. The AMF 844 may be responsible for registration management (for example, for registering UE 802), connection management, reachability management, mobility management, lawful interception of AMF-related events, and access authentication and authorization. The AMF 844 may provide transport for SM messages between the UE 802 and the SMF 846, and act as a transparent proxy for routing SM messages. AMF 844 may also provide transport for SMS messages between UE 802 and an SMSF. AMF 844 may interact with the AUSF 842 and the UE 802 to perform various security anchor and context management functions. Furthermore, AMF 844 may be a termination point of a RAN CP interface, which may include or be an N2 reference point between the RAN 804 and the AMF 844; and the AMF 844 may be a termination point of NAS (Nl) signaling, and perform NAS ciphering and integrity protection. AMF 844 may also support NAS signaling with the UE 802 over an N3 IWF interface.
[0191] The SMF 846 may be responsible for SM (for example, session establishment, tunnel management between UPF 848 and AN 808); UE IP address allocation and management (including optional authorization); selection and control of UP function; configuring traffic steering at UPF 848 to route traffic to proper destination; termination of interfaces toward policy control functions; controlling part of policy enforcement, charging, and QoS; lawful intercept (for SM events and interface to LI system); termination of SM parts of NAS messages; downlink data notification; initiating AN specific SM information, sent via AMF 844 over N2 to AN 808; and determining SSC mode of a session. SM may refer to management of a PDU session, and a PDU session or “session” may refer to a PDU connectivity service that provides or enables the exchange of PDUs between the UE 802 and the data network 836.
[0192] The UPF 848 may act as an anchor point for intra- RAT and inter- RAT mobility, an external PDU session point of interconnect to data network 836, and a branching point to support multi-homed PDU session. The UPF 848 may also perform packet routing and forwarding, perform packet inspection, enforce the user plane part of policy rules, lawfully intercept packets (UP collection), perform traffic usage reporting, perform QoS handling for a user plane (e.g., packet filtering, gating, UL / DL rate enforcement), perform uplink traffic verification (e.g., SDF-to-QoS flow mapping), transport level packet marking in the uplink and downlink, and perform downlink packet buffering and downlink data notification triggering. UPF 848 may include an uplink classifier to support routing traffic flows to a data network.
[0193] The NSSF 850 may select a set of network slice instances serving the UE 802. The NSSF 850 may also determine allowed NSSAI and the mapping to the subscribed S-NSSAIs, if needed. The NSSF 850 may also determine the AMF set to be used to serve the UE 802, or a list of candidate AMFs based on a suitable configuration and possibly by querying the NRF 854. The selection of a set of network slice instances for the UE 802 may be triggered by the AMF 844 with which the UE 802 is registered by interacting with the NSSF 850, which may lead to a change of AMF. The NSSF 850 may interact with the AMF 844 via an N22 reference point; and may communicate with another NSSF in a visited network via an N31 reference point (not shown). Additionally, the NSSF 850 may exhibit an Nnssf service-based interface.
[0194] The NEF 852 may securely expose services and capabilities provided by 3GPP network functions for third party, internal exposure / re-exposure, AFs (e.g., AF 860), edge computing or fog computing systems, etc. In such embodiments, the NEF 852 may authenticate, authorize, or throttle the AFs. NEF 852 may also translate information exchanged with the AF 860 and information exchanged with internal network functions. For example, the NEF 852 may translate between an AF-Service-Identifier and an internal 5GC information. NEF 852 may also receive information from other NFs based on exposed capabilities of other NFs. This information may be stored at the NEF 852 as structured data, or at a data storage NF using standardized interfaces. The stored information can then be re-exposed by the NEF 852 to other NFs and AFs, or used for other purposes such as analytics. Additionally, the NEF 852 may exhibit an Nnef service-based interface.
[0195] The NRF 854 may support service discovery functions, receive NF discovery requests from NF instances, and provide the information of the discovered NF instances to the NF instances. NRF 854 also maintains information of available NF instances and their supported services. As used herein, the terms “instantiate,” “instantiation,” and the like may refer to the creation of an instance, and an “instance” may refer to a concrete occurrence of an object, which may occur, for example, during execution of program code. Additionally, the NRF 854 may exhibit the Nnrf service-based interface. The PCF 856 may provide policy rules to control plane functions to enforce them, and may also support unified policy framework to govern network behavior. The PCF 856 may also implement a front end to access subscription information relevant for policy decisions in a UDR of the UDM 858. In addition to communicating with functions over reference points as shown, the PCF 856 exhibit an Npcf service-based interface.
[0196] The UDM 858 may handle subscription-related information to support the network entities’ handling of communication sessions, and may store subscription data of UE 802. For example, subscription data may be communicated via an N8 reference point between the UDM 858 and the AMF 844. The UDM 858 may include two parts, an application front end and a UDR. The UDR may store subscription data and policy data for the UDM 858 and the PCF 856, and / or structured data for exposure and application data (including PFDs for application detection, application request information for multiple UEs 802) for the NEF 852. The Nudr service-based interface may be exhibited by the UDR 221 to allow the UDM 858, PCF 856, and NEF 852 to access a particular set of the stored data, as well as to read, update (e.g., add, modify), delete, and subscribe to notification of relevant data changes in the UDR. The UDM may include a UDM-FE, which is in charge of processing credentials, location management, subscription management and so on. Several different front ends may serve the same user in different transactions. The UDM-FE accesses subscription information stored in the UDR and performs authentication credential processing, user identification handling, access authorization, registration / mobility management, and subscription management. In addition to communicating with other NFs over reference points as shown, the UDM 858 may exhibit the Nudm servicebased interface.
[0197] The AF 860 may provide application influence on traffic routing, provide access to NEF, and interact with the policy framework for policy control.
[0198] In some embodiments, the 5GC 840 may enable edge computing by selecting operator / 3rd party services to be geographically close to a point that the UE 802 is attached to the network. This may reduce latency and load on the network. To provide edge-computing implementations, the 5GC 840 may select a UPF 848 close to the UE 802 and execute traffic steering from the UPF 848 to data network 836 via the N6 interface. This may be based on the UE subscription data, UE location, and information provided by the AF 860. In this way, the AF 860 may influence UPF (re)selection and traffic routing. Based on operator deployment, when AF 860 is considered to be a trusted entity, the network operator may permit AF 860 to interact directly with relevant NFs. Additionally, the AF 860 may exhibit an Naf service-based interface. The data network 836 may represent various network operator services, Internet access, or third party services that may be provided by one or more servers including, for example, application / content server 838.
[0199] Figure 9 schematically illustrates a wireless network 900 in accordance with various embodiments. The wireless network 900 may include a UE 902 in wireless communication with an AN 904. The UE 902 and AN 904 may be similar to, and substantially interchangeable with, like-named components described elsewhere herein.
[0200] The UE 902 may be communicatively coupled with the AN 904 via connection 906. The connection 906 is illustrated as an air interface to enable communicative coupling, and can be consistent with cellular communications protocols such as an LTE protocol or a 5G NR protocol operating at mmWave or sub-6GHz frequencies.
[0201] The UE 902 may include a host platform 908 coupled with a modem platform 910. The host platform 908 may include application processing circuitry 912, which may be coupled with protocol processing circuitry 914 of the modem platform 910. The application processing circuitry 912 may run various applications for the UE 902 that source / sink application data. The application processing circuitry 912 may further implement one or more layer operations to transmit / receive application data to / from a data network. These layer operations may include transport (for example UDP) and Internet (for example, IP) operations
[0202] The protocol processing circuitry 914 may implement one or more of layer operations to facilitate transmission or reception of data over the connection 906. The layer operations implemented by the protocol processing circuitry 914 may include, for example, MAC, RLC, PDCP, RRC and NAS operations.
[0203] The modem platform 910 may further include digital baseband circuitry 916 that may implement one or more layer operations that are “below” layer operations performed by the protocol processing circuitry 914 in a network protocol stack. These operations may include, for example, PHY operations including one or more of HARQ-ACK functions, scrambling / descrambling, encoding / decoding, layer mapping / de-mapping, modulation symbol mapping, received symbol / bit metric determination, multi-antenna port precoding / decoding, which may include one or more of space-time, space-frequency or spatial coding, reference signal generation / detection, preamble sequence generation and / or decoding, synchronization sequence generation / detection, control channel signal blind decoding, and other related functions.
[0204] The modem platform 910 may further include transmit circuitry 918, receive circuitry 920, RF circuitry 922, and RF front end (RFFE) 924, which may include or connect to one or more antenna panels 926. Briefly, the transmit circuitry 918 may include a digital-to-analog converter, mixer, intermediate frequency (IF) components, etc.; the receive circuitry 920 may include an analog-to-digital converter, mixer, IF components, etc.; the RF circuitry 922 may include a low-noise amplifier, a power amplifier, power tracking components, etc.; RFFE 924 may include filters (for example, surface / bulk acoustic wave filters), switches, antenna tuners, beamforming components (for example, phase-array antenna components), etc. The selection and arrangement of the components of the transmit circuitry 918, receive circuitry 920, RF circuitry 922, RFFE 924, and antenna panels 926 (referred generically as “transmit / receive components”) may be specific to details of a specific implementation such as, for example, whether communication is TDM or FDM, in mmWave or sub-6 gHz frequencies, etc. In some embodiments, the transmit / receive components may be arranged in multiple parallel transmit / receive chains, may be disposed in the same or different chips / modules, etc.
[0205] In some embodiments, the protocol processing circuitry 914 may include one or more instances of control circuitry (not shown) to provide control functions for the transmit / receive components.
[0206] A UE reception may be established by and via the antenna panels 926, RFFE 924, RF circuitry 922, receive circuitry 920, digital baseband circuitry 916, and protocol processing circuitry 914. In some embodiments, the antenna panels 926 may receive a transmission from the AN 904 by receive-beamforming signals received by a plurality of antennas / antenna elements of the one or more antenna panels 926.
[0207] A UE transmission may be established by and via the protocol processing circuitry 914, digital baseband circuitry 916, transmit circuitry 918, RF circuitry 922, RFFE 924, and antenna panels 926. In some embodiments, the transmit components of the UE 904 may apply a spatial filter to the data to be transmitted to form a transmit beam emitted by the antenna elements of the antenna panels 926.
[0208] Similar to the UE 902, the AN 904 may include a host platform 928 coupled with a modem platform 930. The host platform 928 may include application processing circuitry 932 coupled with protocol processing circuitry 934 of the modem platform 930. The modem platform may further include digital baseband circuitry 936, transmit circuitry 938, receive circuitry 940, RF circuitry 942, RFFE circuitry 944, and antenna panels 946. The components of the AN 904 may be similar to and substantially interchangeable with like-named components of the UE 902. In addition to performing data transmission / reception as described above, the components of the AN 908 may perform various logical functions that include, for example, RNC functions such as radio bearer management, uplink and downlink dynamic radio resource management, and data packet scheduling. Figure 10 is a block diagram illustrating components, according to some example embodiments, able to read instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and perform any one or more of the methodologies discussed herein. Specifically, Figure 10 shows a diagrammatic representation of hardware resources 1000 including one or more processors (or processor cores) 1010, one or more memory / storage devices 1020, and one or more communication resources 1030, each of which may be communicatively coupled via a bus 1040 or other interface circuitry. For embodiments where node virtualization (e.g., NFV) is utilized, a hypervisor 1002 may be executed to provide an execution environment for one or more network slices / sub- slices to utilize the hardware resources 1000.
[0209] The processors 1010 may include, for example, a processor 1012 and a processor 1014. The processors 1010 may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a DSP such as a baseband processor, an ASIC, an FPGA, a radio-frequency integrated circuit (RFIC), another processor (including those discussed herein), or any suitable combination thereof.
[0210] The memory / storage devices 1020 may include main memory, disk storage, or any suitable combination thereof. The memory / storage devices 1020 may include, but are not limited to, any type of volatile, non-volatile, or semi- volatile memory such as dynamic random access memory (DRAM), static random access memory (SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), Flash memory, solid-state storage, etc.
[0211] The communication resources 1030 may include interconnection or network interface controllers, components, or other suitable devices to communicate with one or more peripheral devices 1004 or one or more databases 1006 or other network elements via a network 1008. For example, the communication resources 1030 may include wired communication components (e.g., for coupling via USB, Ethernet, etc.), cellular communication components, NFC components, Bluetooth® (or Bluetooth® Low Energy) components, Wi-Fi® components, and other communication components.
[0212] Instructions 1050 may comprise software, a program, an application, an applet, an app, or other executable code for causing at least any of the processors 1010 to perform any one or more of the methodologies discussed herein. The instructions 1050 may reside, completely or partially, within at least one of the processors 1010 (e.g., within the processor’s cache memory), the memory / storage devices 1020, or any suitable combination thereof. Furthermore, any portion of the instructions 1050 may be transferred to the hardware resources 1000 from any combination of the peripheral devices 1004 or the databases 1006. Accordingly, the memory of processors 1010, the memory / storage devices 1020, the peripheral devices 1004, and the databases 1006 are examples of computer-readable and machine-readable media.
[0213] Figure 11 illustrates a network 1100 in accordance with various embodiments. The network 1100 may operate in a matter consistent with 3GPP technical specifications or technical reports for 6G systems. In some embodiments, the network 1100 may operate concurrently with network 800. For example, in some embodiments, the network 1100 may share one or more frequency or bandwidth resources with network 800. As one specific example, a UE (e.g., UE 1102) may be configured to operate in both network 1100 and network 800. Such configuration may be based on a UE including circuitry configured for communication with frequency and bandwidth resources of both networks 800 and 1100. In general, several elements of network 1100 may share one or more characteristics with elements of network 800. For the sake of brevity and clarity, such elements may not be repeated in the description of network 1100.
[0214] The network 1100 may include a UE 1102, which may include any mobile or non-mobile computing device designed to communicate with a RAN 1108 via an over-the-air connection. The UE 1102 may be similar to, for example, UE 802. The UE 1102 may be, but is not limited to, a smartphone, tablet computer, wearable computer device, desktop computer, laptop computer, in-vehicle infotainment, in-car entertainment device, instrument cluster, head-up display device, onboard diagnostic device, dashtop mobile equipment, mobile data terminal, electronic engine management system, electronic / engine control unit, electronic / engine control module, embedded system, sensor, microcontroller, control module, engine management system, networked appliance, machine-type communication device, M2M or D2D device, loT device, etc.
[0215] Although not specifically shown in Figure 11, in some embodiments the network 1100 may include a plurality of UEs coupled directly with one another via a sidelink interface. The UEs may be M2M / D2D devices that communicate using physical sidelink channels such as, but not limited to, PSBCH, PSDCH, PSSCH, PSCCH, PSFCH, etc. Similarly, although not specifically shown in Figure 11, the UE 1102 may be communicatively coupled with an AP such as AP 806 as described with respect to Figure 8. Additionally, although not specifically shown in Figure 11, in some embodiments the RAN 1108 may include one or more ANss such as AN 808 as described with respect to Figure 8. The RAN 1108 and / or the AN of the RAN 1108 may be referred to as a base station (BS), a RAN node, or using some other term or name.
[0216] The UE 1102 and the RAN 1108 may be configured to communicate via an air interface that may be referred to as a sixth generation (6G) air interface. The 6G air interface may include one or more features such as communication in a terahertz (THz) or sub-THz bandwidth, or joint communication and sensing. As used herein, the term “joint communication and sensing” may refer to a system that allows for wireless communication as well as radar-based sensing via various types of multiplexing. As used herein, THz or sub-THz bandwidths may refer to communication in the 80 GHz and above frequency ranges. Such frequency ranges may additionally or alternatively be referred to as “millimeter wave” or “mmWave” frequency ranges.
[0217] The RAN 1108 may allow for communication between the UE 1102 and a 6G core network (CN) 1110. Specifically, the RAN 1108 may facilitate the transmission and reception of data between the UE 1102 and the 6G CN 1110. The 6G CN 1110 may include various functions such as NSSF 850, NEF 852, NRF 854, PCF 856, UDM 858, AF 860, SMF 846, and AUSF 842. The 6G CN 1110 may additional include UPF 848 and DN 836 as shown in Figure 11.
[0218] Additionally, the RAN 1108 may include various additional functions that are in addition to, or alternative to, functions of a legacy cellular network such as a 4G or 5G network. Two such functions may include a Compute Control Function (Comp CF) 1124 and a Compute Service Function (Comp SF) 1136. The Comp CF 1124 and the Comp SF 1136 may be parts or functions of the Computing Service Plane. Comp CF 1124 may be a control plane function that provides functionalities such as management of the Comp SF 1136, computing task context generation and management (e.g., create, read, modify, delete), interaction with the underlying computing infrastructure for computing resource management, etc.. Comp SF 1136 may be a user plane function that serves as the gateway to interface computing service users (such as UE 1102) and computing nodes behind a Comp SF instance. Some functionalities of the Comp SF 1136 may include: parse computing service data received from users to compute tasks executable by computing nodes; hold service mesh ingress gateway or service API gateway; service and charging policies enforcement; performance monitoring and telemetry collection, etc. In some embodiments, a Comp SF 1136 instance may serve as the user plane gateway for a cluster of computing nodes. A Comp CF 1124 instance may control one or more Comp SF 1136 instances.
[0219] Two other such functions may include a Communication Control Function (Comm CF) 1128 and a Communication Service Function (Comm SF) 1138, which may be parts of the Communication Service Plane. The Comm CF 1128 may be the control plane function for managing the Comm SF 1138, communication sessions creation / configuration / releasing, and managing communication session context. The Comm SF 1138 may be a user plane function for data transport. Comm CF 1128 and Comm SF 1138 may be considered as upgrades of SMF 846 and UPF 848, which were described with respect to a 5G system in Figure 8. The upgrades provided by the Comm CF 1128 and the Comm SF 1138 may enable service-aware transport. For legacy (e.g., 4G or 5G) data transport, SMF 846 and UPF 848 may still be used.
[0220] Two other such functions may include a Data Control Function (Data CF) 1122 and Data Service Function (Data SF) 1132 may be parts of the Data Service Plane. Data CF 1122 may be a control plane function and provides functionalities such as Data SF 1132 management, Data service creation / configuration / releasing, Data service context management, etc. Data SF 1132 may be a user plane function and serve as the gateway between data service users (such as UE 1102 and the various functions of the 6G CN 1110) and data service endpoints behind the gateway. Specific functionalities may include include: parse data service user data and forward to corresponding data service endpoints, generate charging data, report data service status.
[0221] Another such function may be the Service Orchestration and Chaining Function (SOCF) 1120, which may discover, orchestrate and chain up communication / computing / data services provided by functions in the network. Upon receiving service requests from users, SOCF 1120 may interact with one or more of Comp CF 1124, Comm CF 1128, and Data CF 1122 to identify Comp SF 1136, Comm SF 1138, and Data SF 1132 instances, configure service resources, and generate the service chain, which could contain multiple Comp SF 1136, Comm SF 1138, and Data SF 1132 instances and their associated computing endpoints. Workload processing and data movement may then be conducted within the generated service chain. The SOCF 1120 may also responsible for maintaining, updating, and releasing a created service chain.
[0222] Another such function may be the service registration function (SRF) 1114, which may act as a registry for system services provided in the user plane such as services provided by service endpoints behind Comp SF 1136 and Data SF 1132 gateways and services provided by the UE 1102. The SRF 1114 may be considered a counterpart of NRF 854, which may act as the registry for network functions.
[0223] Other such functions may include an evolved service communication proxy (eSCP) and service infrastructure control function (SICF) 1126, which may provide service communication infrastructure for control plane services and user plane services. The eSCP may be related to the service communication proxy (SCP) of 5G with user plane service communication proxy capabilities being added. The eSCP is therefore expressed in two parts: eCSP-C 1112 and eSCP- U 1134, for control plane service communication proxy and user plane service communication proxy, respectively. The SICF 1126 may control and configure eCSP instances in terms of service traffic routing policies, access rules, load balancing configurations, performance monitoring, etc.
[0224] Another such function is the AMF 1144. The AMF 1144 may be similar to 844, but with additional functionality. Specifically, the AMF 1144 may include potential functional repartition, such as move the message forwarding functionality from the AMF 1144 to the RAN 1108.
[0225] Another such function is the service orchestration exposure function (SOEF) 1118. The SOEF may be configured to expose service orchestration and chaining services to external users such as applications.
[0226] The UE 1102 may include an additional function that is referred to as a computing client service function (comp CSF) 1104. The comp CSF 1104 may have both the control plane functionalities and user plane functionalities, and may interact with corresponding network side functions such as SOCF 1120, Comp CF 1124, Comp SF 1136, Data CF 1122, and / or Data SF 1132 for service discovery, request / response, compute task workload exchange, etc. The Comp CSF 1104 may also work with network side functions to decide on whether a computing task should be run on the UE 1102, the RAN 1108, and / or an element of the 6G CN 1110.
[0227] The UE 1102 and / or the Comp CSF 1104 may include a service mesh proxy 1106. The service mesh proxy 1106 may act as a proxy for service-to-service communication in the user plane. Capabilities of the service mesh proxy 1106 may include one or more of addressing, security, load balancing, etc.
[0228] Figure 12 illustrates a simplified block diagram of artificial (Al)-assisted communication between a UE 1205 and a RAN 1210, in accordance with various embodiments. More specifically, as described in further detail below, Al / machine learning (ML) models may be used or leveraged to facilitate over-the-air communication between UE 1205 and RAN 1210.
[0229] One or both of the UE 1205 and the RAN 1210 may operate in a matter consistent with 3GPP technical specifications or technical reports for 6G systems. In some embodiments, the wireless cellular communication between the UE 1205 and the RAN 1210 may be part of, or operate concurrently with, networks 1100, 800, and / or some other network described herein.
[0230] The UE 1205 may be similar to, and share one or more features with, UE 1102, UE 802, and / or some other UE described herein. The UE 1205 may be, but is not limited to, a smartphone, tablet computer, wearable computer device, desktop computer, laptop computer, in- vehicle infotainment, in-car entertainment device, instrument cluster, head-up display device, onboard diagnostic device, dashtop mobile equipment, mobile data terminal, electronic engine management system, electronic / engine control unit, electronic / engine control module, embedded system, sensor, microcontroller, control module, engine management system, networked appliance, machine-type communication device, M2M or D2D device, loT device, etc. The RAN 1210 may be similar to, and share one or more features with, RAN 814, RAN 1108, and / or some other RAN described herein. As may be seen in Figure 12, the Al-related elements of UE 1205 may be similar to the Al-related elements of RAN 1210. For the sake of discussion herein, description of the various elements will be provided from the point of view of the UE 1205, however it will be understood that such discussion or description will apply to equally named / numbered elements of RAN 1210, unless explicitly stated otherwise.
[0231] As previously noted, the UE 1205 may include various elements or functions that are related to AI / ML. Such elements may be implemented as hardware, software, firmware, and / or some combination thereof. In embodiments, one or more of the elements may be implemented as part of the same hardware (e.g., chip or multi-processor chip), software (e.g., a computing program), or firmware as another element.
[0232] One such element may be a data repository 1215. The data repository 1215 may be responsible for data collection and storage. Specifically, the data repository 1215 may collect and store RAN configuration parameters, measurement data, performance key performance indicators (KPIs), model performance metrics, etc., for model training, update, and inference. More generally, collected data is stored into the repository. Stored data can be discovered and extracted by other elements from the data repository 1215. For example, as may be seen, the inference data selection / filter element 1250 may retrieve data from the data repository 1215. In various embodiments, the UE 1205 may be configured to discover and request data from the data repository 1210 in the RAN, and vice versa. More generally, the data repository 1215 of the UE 1205 may be communicatively coupled with the data repository 1215 of the RAN 1210 such that the respective data repositories of the UE and the RAN may share collected data with one another.
[0233] Another such element may be a training data selection / filtering functional block 1220. The training data selection / filter functional block 1220 may be configured to generate training, validation, and testing datasets for model training. Training data may be extracted from the data repository 1215. Data may be selected / filtered based on the specific AI / ML model to be trained. Data may optionally be transformed / augmented / pre-processed (e.g., normalized) before being loaded into datasets. The training data selection / filter functional block 1220 may label data in datasets for supervised learning. The produced datasets may then be fed into model training the model training functional block 1225.
[0234] As noted above, another such element may be the model training functional block 1225. This functional block may be responsible for training and updating(re-training) AI / ML models. The selected model may be trained using the fed-in datasets (including training, validation, testing) from the training data selection / filtering functional block. The model training functional block 1225 may produce trained and tested AI / ML models which are ready for deployment. The produced trained and tested models can be stored in a model repository 1235.
[0235] The model repository 1235 may be responsible for AI / ML models’ (both trained and untrained) storage and exposure. Trained / updated model(s) may be stored into the model repository 1235. Model and model parameters may be discovered and requested by other functional blocks (e.g., the training data selection / filter functional block 1220 and / or the model training functional block 1225). In some embodiments, the UE 1205 may discover and request AI / ML models from the model repository 1235 of the RAN 1210. Similarly, the RAN 1210 may be able to discover and / or request AI / ML models from the model repository 1235 of the UE 1205. In some embodiments, the RAN 1210 may configure models and / or model parameters in the model repository 1235 of the UE 1205.
[0236] Another such element may be a model management functional block 1240. The model management functional block 1240 may be responsible for management of the AI / ML model produced by the model training functional block 1225. Such management functions may include deployment of a trained model, monitoring model performance, etc. In model deployment, the model management functional block 1240 may allocate and schedule hardware and / or software resources for inference, based on received trained and tested models. As used herein, “inference” refers to the process of using trained AI / ML model(s) to generate data analytics, actions, policies, etc. based on input inference data. In performance monitoring, based on wireless performance KPIs and model performance metrics, the model management functional block 1240 may decide to terminate the running model, start model re-training, select another model, etc. In embodiments, the model management functional block 1240 of the RAN 1210 may be able to configure model management policies in the UE 1205 as shown.
[0237] Another such element may be an inference data selection / filtering functional block 1250. The inference data selection / filter functional block 1250 may be responsible for generating datasets for model inference at the inference functional block 1245, as described below. Specifically, inference data may be extracted from the data repository 1215. The inference data selection / filter functional block 1250 may select and / or filter the data based on the deployed AI / ML model. Data may be transformed / augmented / pre-processed following the same transformation / augmentation / pre-processing as those in training data selection / filtering as described with respect to functional block 1220. The produced inference dataset may be fed into the inference functional block 1245.
[0238] Another such element may be the inference functional block 1245. The inference functional block 1245 may be responsible for executing inference as described above. Specifically, the inference functional block 1245 may consume the inference dataset provided by the inference data selection / filtering functional block 1250, and generate one or more outcomes. Such outcomes may be or include data analytics, actions, policies, etc. The outcome(s) may be provided to the performance measurement functional block 1230.
[0239] The performance measurement functional block 1230 may be configured to measure model performance metrics (e.g., accuracy, model bias, run-time latency, etc.) of deployed and executing models based on the inference outcome(s) for monitoring purpose. Model performance data may be stored in the data repository 1215.
[0240] 3. EXAMPLE PROCEDURES
[0241] In some embodiments, the electronic device(s), network(s), system(s), chip(s) or component(s), or portions or implementations thereof, of Figures 8-12, or some other figure herein, may be configured to perform one or more processes, techniques, or methods as described herein, or portions thereof. One such process is depicted in Figure 13, which may include or relate to method to be performed by a machine learning (ML) testing management services (MnS) producer, one or more elements of a ML testing MnS producer, and / or one or more electronic devices that include and / or implement a ML testing MnS producer. The process may include identifying, at 1305 based on a first managed object instance (MOI) received from a ML testing MnS consumer, a request for testing of a machine learning (ML) model; performing, at 1310 by the MnS producer based on the request, the testing of the ML model; and providing, at 1315 by the MnS producer to the MnS consumer via a second MOI, a ML testing report that is related to the testing of the ML model.
[0242] Another such process is depicted in Figure 14. The process of Figure 14 may include or relate to a method to be performed by a machine learning (ML) testing management services (MnS) consumer, one or more elements of a ML testing MnS consumer, and / or one or more electronic devices that include and / or implement a ML testing MnS consumer. The process may include transmitting, at 1405 to a ML testing MnS provider via a first managed object instance (MOI), a request for testing of a machine learning (ML) model; and identifying, at 1410 based on a second MOI received from the MnS producer, a ML testing report that is related to the testing of the ML model by the MnS provider, wherein the testing of the ML model by the MnS provider is responsive to receipt of the first MOI.
[0243] For one or more embodiments, at least one of the components set forth in one or more of the preceding figures may be configured to perform one or more operations, techniques, processes, and / or methods as set forth in the example section below. For example, the baseband circuitry as described above in connection with one or more of the preceding figures may be configured to operate in accordance with one or more of the examples set forth below. For another example, circuitry associated with a UE, base station, network element, etc. as described above in connection with one or more of the preceding figures may be configured to operate in accordance with one or more of the examples set forth below in the example section.
[0244] 4. EXAMPLE IMPLEMENTATIONS
[0245] Additional examples of the presently described methods, devices, systems, and networks discussed herein include the following, non-limiting implementations. Each of the following nonlimiting examples may stand on its own or may be combined in any permutation or combination with any one or more of the other examples provided below or throughout the present disclosure.
[0246] Example 1 includes a method of operating a Management Service (MnS) producer configured to support ML testing, the method comprising: initiating machine learning (ML) testing on behalf of an MnS consumer; creating one or more ML testing processes for testing one or more ML entities; and sending an ML testing report to the MnS consumer.
[0247] Example 2 includes the method of example 1 and / or some other example(s) herein, wherein the method includes: receiving an ML testing request from the MnS consumer.
[0248] Example 3 includes the method of example 2 and / or some other example(s) herein, wherein the method includes: responding to the MnS consumer to indicate whether the request is accepted.
[0249] Example 4 includes the method of examples 1-3 and / or some other example(s) herein, wherein the ML testing request is for the testing of a single ML entity or a group of ML entities.
[0250] Example 5 includes the method of example 4 and / or some other example(s) herein, wherein the ML testing request includes one or more of: candidate testing data; testing request source; and / or a DN of the ML entity to be tested.
[0251] Example 6 includes the method of examples 1-5 and / or some other example(s) herein, wherein the method includes: receiving an ML testing policy for testing the ML entities; and testing the ML entities according to the ML testing policy.
[0252] Example 7 includes the method of example 6 and / or some other example(s) herein, wherein the ML testing policy includes at least one of: a set of ML entities affected by the ML testing policy; time window allowed for testing the one or more ML entities; and / or maximum number of allowed testing iterations, epochs, and / or processes.
[0253] Example 8 includes the method of examples 1-7 and / or some other example(s) herein, wherein the ML testing process includes at least one of: priority information; progress status; and / or a set of attributes for canceling or suspending the process.
[0254] Example 9 includes the method of examples 1-8 and / or some other example(s) herein, wherein the ML testing report includes at least one of the following information: indication of whether the consumer provided testing data have been used; what consumer provided testing data has been used; information related to performance of the ML entity; ML entity testing result; the associated ML testing request; and / or the associated ML testing process.
[0255] Example 10 includes the method of examples 1-9 and / or some other example(s) herein, wherein the ML testing request, the ML testing process, the ML testing report, and the ML testing policy are represented by respective Managed Object Instances (MOIs) of corresponding Information Object Classes (IOCS).
[0256] Example 11 includes the method of example 10 and / or some other example(s) herein, wherein one or more of the respective MOIs representing the ML testing request, the ML testing process, the ML testing report, and / or the ML testing policy are contained by an MOI of an ML testing function of MOI of an ML training function.
[0257] Example 12 includes the method of examples 1-11 and / or some other example(s) herein, wherein the MnS producer is located in an ML training function.
[0258] Example 13 includes the method of examples 1-12 and / or some other example(s) herein, wherein the MnS producer is located in an ML testing function that is separate from the ML training function.
[0259] Example Bl includes a method of operating a Management Service (MnS) producer configured to support ML training, the method comprising: initiating machine learning (ML) training on behalf of an MnS consumer; creating one or more ML training processes for training one or more ML entities; and sending an ML training report to the consumer.
[0260] Example B2 includes the method of example Bl and / or some other example(s) herein, wherein the method includes: receiving an ML training request from an MnS consumer.
[0261] Example B3 includes the method of example B2 and / or some other example(s) herein, wherein the method includes: responding to the MnS consumer to indicate whether the request is accepted.
[0262] Example B4 includes the method of example Bl and / or some other example(s) herein, wherein the method includes: initiating the ML training without the MnS consumer’s request
[0263] Example B5 includes the method of examples B 1-B4 and / or some other example(s) herein, wherein the ML training request is for the training of an initial ML entity, or re-retraining of an ML entity, or joint training of a group of ML entities.
[0264] Example B6 includes the method of example B5 and / or some other example(s) herein, wherein the ML training request includes the inference type of the ML entity for the initial training.
[0265] Example B7 includes the method of examples B5-B6 and / or some other example(s) herein, wherein the ML training request includes an identifier (DN) of an ML entity for the re-training. Example B8 includes the method of examples B5-B7 and / or some other example(s) herein, wherein the ML training request includes an identifier (DN) of an ML entity group for the joint training.
[0266] Example B9 includes the method of examples B 1-B8 and / or some other example(s) herein, wherein the ML training process indicates the ML entity being trained.
[0267] Example BIO includes the method of examples B1-B9 and / or some other example(s) herein, wherein the ML training report indicates the identifier (DN) of the ML entity generated by the training.
[0268] Example Bl l includes the method of examples B1-B10 and / or some other example(s) herein, wherein the ML training report indicates the identifier (DN) of the ML entity group generated by the training.
[0269] Example B12 includes the method of examples B6-B11 and / or some other example(s) herein, wherein ML entity group is represented by an Managed Object Instance (MOI).
[0270] Example B13 includes the method of examples B6-B10 and / or some other example(s) herein, wherein ML entity is represented by an Managed Object Instance (MOI).
[0271] Example B14 includes the method of examples B12-B13 and / or some other example(s) herein, wherein the ML entity group is associated with one or more ML entities.
[0272] Example B15 includes the method of example B14 and / or some other example(s) herein, wherein the MOI representing the ML entity group contains one or more levels.
[0273] Example B16 includes the method of example B15 and / or some other example(s) herein, wherein each level of the ML entity group is associated with one or more ML entities.
[0274] Example B17 includes the method of example B16 and / or some other example(s) herein, wherein the relation among the ML entities for each level of the ML entity group is provided.
[0275] Example B18 includes the method of example B17 and / or some other example(s) herein, wherein relation among the ML entities for each level of the ML entity group is “in sequence” or “in parallel”.
[0276] Example B19 includes the method of examples B14-B18 and / or some other example(s) herein, wherein the MOI representing the ML entity is inherited from a top IOC.
[0277] Example B20 includes the method of examples B14-B19 and / or some other example(s) herein, wherein the MOI representing the ML entity group is inherited from a top IOC.
[0278] Example B21 includes the method of examples B15-B20 and / or some other example(s) herein, wherein the MOI representing the ML entity is inherited from Top IOC.
[0279] Example B22 includes the method of examples B14-B21 and / or some other example(s) herein, wherein the MOI representing the ML entity is contained by an MOI of MLTrainingEunction IOC. Example B23 includes the method of examples B15-B22 and / or some other example(s) herein, wherein the MOI representing the ML entity group is contained by an MOI of MLTrainingFunction IOC.
[0280] Example Cl includes a method to be performed by a machine learning (ML) testing management services (MnS) producer, one or more elements of a ML testing MnS producer, and / or one or more electronic devices that include and / or implement a ML testing MnS producer, wherein the method comprises: identifying, based on a first managed object instance (MOI) received from a ML testing MnS consumer, a request for testing of a machine learning (ML) model; performing, by the MnS producer based on the request, the testing of the ML model; and providing, by the MnS producer to the MnS consumer via a second MOI, a ML testing report that is related to the testing of the ML model.
[0281] Example C2 includes the method of example Cl, and / or some other example herein, wherein the second MOI is based on a MLTestingReport information object class (IOC).
[0282] Example C3 includes the method of any of examples C1-C2, and / or some other example herein, wherein the first MOI is based on a MLTestingRequestlOC.
[0283] Example C4 includes the method of any of examples C1-C3, and / or some other example herein, wherein the second MOI has an attribute that indicates performance of the ML model during testing.
[0284] Example C5 includes the method of example C4, and / or some other example herein, wherein the attribute that indicates performance of the tested ML model is a modelPerformanceTesting attribute.
[0285] Example C6 includes the method of any of examples C1-C5, and / or some other example herein, wherein the second MOI has an attribute that indicates a result of testing the ML model.
[0286] Example C7 includes the method of example C6, and / or some other example herein, wherein the attribute that indicates the result of testing the ML model is a mlTestingResult attribute.
[0287] Example C8 includes the method of any of examples C1-C7, and / or some other example herein, wherein the first MOI has an attribute that indicates a status of the request for testing of the ML model.
[0288] Example C9 includes the method of example C8, and / or some other example herein, wherein the attribute that indicates the status of the request for testing of the ML model is a requeststatus attribute.
[0289] Example CIO includes the method of any of examples C1-C9, and / or some other example herein, wherein the first MOI has an attribute related to cancellation of the request for testing of the ML model. Example Cl l includes the method of example CIO, and / or some other example herein, wherein the attribute related to cancellation of the request for testing of the ML model is a cancelRequest attribute.
[0290] Example C12 includes the method of any of examples Cl-Cl 1, and / or some other example herein, wherein the first MOI has an attribute related to suspension of the request for testing of the ML model.
[0291] Example C13 includes the method of example Cl 2, and / or some other example herein, wherein the attribute related to suspension of the request for testing of the ML model is a suspendRequest attribute.
[0292] Example DI includes a method to be performed by a machine learning (ML) testing management services (MnS) consumer, one or more elements of a ML testing MnS consumer, and / or one or more electronic devices that include and / or implement a ML testing MnS consumer, wherein the method comprises: transmitting, to a ML testing MnS provider via a first managed object instance (MOI), a request for testing of a machine learning (ML) model; and identifying, based on a second MOI received from the MnS producer, a ML testing report that is related to the testing of the ML model by the MnS provider, wherein the testing of the ML model by the MnS provider is responsive to receipt of the first MOI.
[0293] Example D2 includes the method of example DI, and / or some other example herein, wherein the second MOI is based on a MLTestingReport information object class (IOC).
[0294] Example D3 includes the method of any of examples D1-D2, and / or some other example herein, wherein the first MOI is based on a MLTestingRequestlOC.
[0295] Example D4 includes the method of any of examples D1-D3, and / or some other example herein, wherein the second MOI has an attribute that indicates performance of the ML model during testing.
[0296] Example D5 includes the method of example D4, and / or some other example herein, wherein the attribute that indicates performance of the tested ML model is a modelPerformanceTesting attribute.
[0297] Example D6 includes the method of any of examples D1-D5, and / or some other example herein, wherein the second MOI has an attribute that indicates a result of testing the ML model.
[0298] Example D7 includes the method of example D6, and / or some other example herein, wherein the attribute that indicates the result of testing the ML model is a mlTestingResult attribute.
[0299] Example D8 includes the method of any of examples D1-D7, and / or some other example herein, wherein the first MOI has an attribute that indicates a status of the request for testing of the ML model. Example D9 includes the method of example D8, and / or some other example herein, wherein the attribute that indicates the status of the request for testing of the ML model is a requeststatus attribute.
[0300] Example DIO includes the method of any of examples D1-D9, and / or some other example herein, wherein the first MOI has an attribute related to cancellation of the request for testing of the ML model.
[0301] Example Dl l includes the method of example DIO, and / or some other example herein, wherein the attribute related to cancellation of the request for testing of the ML model is a cancelRequest attribute.
[0302] Example D 12 includes the method of any of examples DI -DI 1, and / or some other example herein, wherein the first MOI has an attribute related to suspension of the request for testing of the ML model.
[0303] Example D13 includes the method of example D12, and / or some other example herein, wherein the attribute related to suspension of the request for testing of the ML model is a suspendRequest attribute.
[0304] Example Z01 includes one or more computer readable media comprising instructions, wherein execution of the instructions by processor circuitry is to cause the processor circuitry to perform the method of any one of examples 1-D13.
[0305] Example Z02 includes a computer program comprising the instructions of example Z01.
[0306] Example Z03 includes an Application Programming Interface defining functions, methods, variables, data structures, and / or protocols for the computer program of example Z02.
[0307] Example Z04 includes an API or specification defining functions, methods, variables, data structures, protocols, and the like, defining or involving use of any of examples 1 -DI 3 or portions thereof, or otherwise related to any of examples 1 -DI 3 or portions thereof.
[0308] Example Z05 includes an apparatus comprising circuitry loaded with the instructions of example Z01.
[0309] Example Z06 includes an apparatus comprising circuitry operable to run the instructions of example Z01.
[0310] Example Z07 includes an integrated circuit comprising one or more of the processor circuitry of example Z01 and the one or more computer readable media of example Z01.
[0311] Example Z08 includes a computing system comprising the one or more computer readable media and the processor circuitry of example Z01.
[0312] Example Z09 includes an apparatus comprising means for executing the instructions of example Z01.
[0313] Example Z10 includes a signal generated as a result of executing the instructions of example Z01.
[0314] Example Zl l includes a data unit generated as a result of executing the instructions of example Z01.
[0315] Example Z12 includes the data unit of example Z10 and / or some other example(s) herein, wherein the data unit is a datagram, network packet, data frame, data segment, a Protocol Data Unit (PDU), a Service Data Unit (SDU), a message, or a database object.
[0316] Example Z13 includes a signal encoded with the data unit of examples Z11 and / or Z12.
[0317] Example Z14 includes an electromagnetic signal carrying the instructions of example Z01.
[0318] Example Z15 includes an apparatus comprising means for performing the method of any one of examples 1-D13 and / or some other example(s) herein.
[0319] Example Z16 includes an edge compute node executing a service as part of one or more edge applications instantiated on virtualization infrastructure, the service being related to any of examples 1-D13, portions thereof, and / or some other example(s) herein.
[0320] Embodiments may be referred to herein, individually and / or collectively, merely for convenience and without intending to voluntarily limit the scope of this application to any single aspect or concept if more than one is in fact disclosed. Thus, although specific aspects have been illustrated and described herein, it should be appreciated that any arrangement calculated to achieve the same purpose may be substituted for the specific aspects shown. This disclosure is intended to cover any and all adaptations or variations of various aspects. Combinations of the above aspects and other aspects not specifically described herein will be apparent to those of skill in the art upon reviewing the above description.
Claims
Claims1. An electronic device comprising: one or more processors configured to implement a machine learning (ML) testing management services (MnS) producer; and one or more computer-readable media comprising instructions that, upon execution of the instructions, are to cause the ML testing MnS producer to: identify, based on a first managed object instance (MOI) received from a ML testing MnS consumer, a request for testing of a machine learning (ML) model; perform, by the MnS producer based on the request, the testing of the ML model; and provide, by the MnS producer to the MnS consumer via a second MOI, a ML testing report that is related to the testing of the ML model.
2. The electronic device of claim 1, wherein the second MOI is based on a MLTestingReport information object class (IOC).
3. The electronic device of claim 1, wherein the first MOI is based on a MLTestingRequestlOC.
4. The electronic device of any of claims 1-3, wherein the second MOI has an attribute that indicates performance of the ML model during testing.
5. The electronic device of claim 4, wherein the attribute that indicates performance of the tested ML model is a modelPerformanceTesting attribute.
6. The electronic device of any of claims 1-3, wherein the second MOI has an attribute that indicates a result of testing the ML model.
7. The electronic device of claim 6, wherein the attribute that indicates the result of testing the ML model is a mlTestingResult attribute.
8. The electronic device of any of claims 1-3, wherein the first MOI has an attribute that indicates a status of the request for testing of the ML model.
9. The electronic device of claim 8, wherein the attribute that indicates the status of the request for testing of the ML model is a requeststatus attribute.
10. The electronic device of any of claims 1-3, wherein the first MOI has an attribute related to cancellation of the request for testing of the ML model.
11. The electronic device of claim 10, wherein the attribute related to cancellation of the request for testing of the ML model is a cancelRequest attribute.
12. The electronic device of any of claims 1-3, wherein the first MOI has an attribute related to suspension of the request for testing of the ML model.
13. The electronic device of claim 12, wherein the attribute related to suspension of the request for testing of the ML model is a suspendRequest attribute.
14. An electronic device comprising: one or more processors configured to implement a machine learning (ML) testing management services (MnS) consumer; and one or more computer readable media comprising instructions that upon execution of the instructions, are to cause the MnS consumer to: transmit, to a ML testing MnS provider via a first managed object instance (MOI), a request for testing of a ML model, wherein the first MOI is based on a MLTestingRequest information object class (IOC); and identify, based on a second MOI received from the MnS producer, a ML testing report that is related to the testing of the ML model by the MnS producer, wherein the second MOI is based on a MLTestingReport IOC, and wherein the testing of the ML model by the MnS provider is responsive to receipt of the first MOI.
15. The electronic device of claim 14, wherein the second MOI has a modelPerformanceTesting attribute.
16. The electronic device of claim 14, wherein the second MOI has a mlTestingResult attribute.
17. The electronic device of any of claims 14-16, wherein the first MOI has a requeststatus attribute.
18. The electronic device of any of claims 14-16, wherein the first MOI has a cancelRequest attribute.
19. The electronic device of any of claims 14-16, wherein the first MOI has a suspendRequest attribute.
Citation Information
Patent Citations
Deep learning guide device and method
CN111931944A
Apparatus for use in wireless communication system
CN115884234A
System and method for providing robust artificial intelligence inference in edge computing devices
US20230108920A1
A method and system for testing machine learning models
WO2022047470A1
