Systems and methods for artificial intelligence / machine learning model group validation

Abstract

Systems and methods for artificial intelligence (AI) / machine learning (ML) model group validation are discussed herein. A user equipment (UE) receives, from an AI / ML model group management server, an indication that a first AI / ML model is approved for use, and performs a corresponding inference using a group of AI / ML models that includes the first AI / ML model. An AI / ML mode group management server receives, from an AI / ML model network function, a request to approve a first AI / ML model for use, performs first group testing of a group of AI / ML models comprising the first AI / ML model and one or more additional AI / ML models, and sends a corresponding response. An AI / ML model network function sends, to an AI / ML model group management server, a request to approve an AI / ML model for use, the request comprising the AI / ML model, and receives a response to the request that indicates that the AI / ML model is approved for use.

Description

SYSTEMS AND METHODS FOR ARTIFICIAL INTELLIGENCE / MACHINELEARNING MODEL GROUP VALIDATIONTECHNICAL FIELD

[0001] This application relates generally to wireless communication systems, including wireless communication systems implementing artificial intelligence (AI) / machine learning (ML) model functionality.BACKGROUND

[0002] Wireless mobile communication technology uses various standards and protocols to transmit data between a base station and a wireless communication device. Wireless communication system standards and protocols can include, for example, 3rd Generation Partnership Project (3GPP) Long Term Evolution (LTE) (e.g., 4G), 3GPP New Radio (NR) (e.g., 5G), and Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard for Wireless Local Area Networks (WLAN) (commonly known to industry groups as Wi-Fi®).

[0003] As contemplated by the 3GPP, different wireless communication systems’ standards and protocols can use various radio access networks (RANs) for communicating between a base station of the RAN (which may also sometimes be referred to generally as a RAN node, a network node, or simply a node) and a wireless communication device known as a user equipment (UE). 3GPP RANs can include, for example, Global System for Mobile communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE) RAN (GERAN), Universal Terrestrial Radio Access Network (UTRAN), Evolved Universal Terrestrial Radio Access Network (E-UTRAN), and / or Next-Generation Radio Access Network (NG-RAN).

[0004] Each RAN may use one or more radio access technologies (RATs) to perform communication between the base station and the UE. For example, the GERAN implements GSM and / or EDGE RAT, the UTRAN implements Universal Mobile Telecommunication System (UMTS) RAT or other 3GPP RAT, the E-UTRAN implements LTE RAT (sometimes simply referred to as LTE), and NG-RAN implements NR RAT (sometimes referred to herein as 5G RAT, 5G NR RAT, or simply NR). In certain deployments, the E-UTRAN may also implement NR RAT. In certain deployments, NG-RAN may also implement LTE RAT.1P68765WO1 4897-2098-5188\1

[0005] A base station used by a RAN may correspond to that RAN. One example of an E-UTRAN base station is an Evolved Universal Terrestrial Radio Access Network (E- UTRAN) Node B (also commonly denoted as evolved Node B, enhanced Node B, eNodeB, or eNB). One example of an NG-RAN base station is a next generation Node B (also sometimes referred to as a g Node B or gNB).

[0006] A RAN provides its communication services with external entities through its connection to a core network (CN). For example, E-UTRAN may utilize an Evolved Packet Core (EPC) while NG-RAN may utilize a 5G Core Network (5GC).BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0007] To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced.

[0008] FIG. 1 illustrates an AI / ML operational workflow.

[0009] FIG. 2 illustrates an AI / ML functional framework.

[0010] FIG. 3 illustrates a UE that includes an AI / ML model group, according to embodiments discussed herein.

[0011] FIG. 4 A, FIG. 4B, and FIG. 4C illustrate various examples of how a group of AI / ML models may work in a coordinated manner to generate an inference, according to embodiments discussed herein.

[0012] FIG. 5 illustrates an AI / ML operational workflow.

[0013] FIG. 6 illustrates a diagram showing communications between a network and a server that may occur with respect to an AI / ML model that is intended for use at a UE as part of an AI / ML model group.

[0014] FIG. 7 illustrates a diagram showing communications between a network, a server, and a UE that may occur corresponding to group testing for an AI / ML model for approval for use within an AI / ML model group at the UE, according to embodiments discussed herein.

[0015] FIG. 8A, FIG. 8B, and FIG. 8C illustrate different ways that a group testing mechanism could impact an interaction between a UE and a network.2P68765WO1 4897-2098-5188\1

[0016] FIG. 9 illustrates a diagram showing communications between a network and a server that may occur with respect to updating an AI / ML model that is used at a UE as part of an AI / ML model group.

[0017] FIG. 10 illustrates a flow diagram for the performance of AI / ML model group monitoring, according to embodiments discussed herein.

[0018] FIG. 11 illustrates a flow diagram for communications between a server, a UE, and a network for AI / ML model group monitoring, according to embodiments discussed herein.

[0019] FIG. 12 illustrates a flow diagram for communications between a server, a UE, and a network for AI / ML model group monitoring, according to embodiments discussed herein.

[0020] FIG. 13 illustrates an example of a model identification mechanism that may be used to identify an AI / ML model, according to embodiments discussed herein.

[0021] FIG. 14 illustrates a diagram showing communications between a network, a UE, and a server that may occur corresponding to UE-centric group testing for an AI / ML model for approval for use within an AI / ML model group at the UE, according to embodiments discussed herein.

[0022] FIG. 15 illustrates a diagram showing communications between a network, a UE, and a server that may occur corresponding to UE-centric group monitoring for an AI / ML model within an AI / ML model group at the UE, according to embodiments discussed herein.

[0023] FIG. 16 illustrates a diagram showing cases for impacts of various types of communications between a server, a network, and a UE as may (or may not) be reflected within a specification for a wireless communication system, according to embodiments herein.

[0024] FIG. 17 illustrates a method of a UE operating with a network of a wireless communication system, according to embodiments discussed herein.

[0025] FIG. 18 illustrates a method of an AI / ML model group management server, according to embodiments discussed herein.

[0026] FIG. 19 illustrates a method of an AI / ML model network function of a network of a wireless communication system, according to embodiments discussed herein.3P68765WO1 4897-2098-5188\1

[0027] FIG. 20 illustrates a method of a UE operating with a network of a wireless communication system, according to embodiments discussed herein.

[0028] FIG. 21 illustrates an example architecture of a wireless communication system, according to embodiments disclosed herein.

[0029] FIG. 22 illustrates a system for performing signaling between a wireless device and a network device, according to embodiments disclosed herein.DETAILED DESCRIPTION

[0030] Various embodiments are described with regard to a UE. However, reference to a UE is merely provided for illustrative purposes. The example embodiments may be utilized with any electronic component that may establish a connection to a network and is configured with the hardware, software, and / or firmware to exchange information and data with the network. Therefore, the UE as described herein is used to represent any appropriate electronic component.

[0031] FIG. 1 illustrates an AI / ML operational workflow 100. The AI / ML operational workflow 100 may represent one embodiment for a “lifecycle” of an AI / ML model.

[0032] The AI / ML operational workflow 100 illustrates a training phase 102 that includes ML training 104 and ML testing 106, an emulation phase 108 that includes ML emulation 110, a deployment phase 112 that includes ML entity loading 114, and an inference phase 116 that includes an AI / ML inference 118.

[0033] The ML training 104 of the training phase 102 may include functionality for training (including initial training and any re-training) of an AL / ML model or a group of AL / ML models. It may also incorporate validation of an AI / ML model to evaluate its performance on training data and / or validation data.

[0034] The ML testing 106 of the training phase 102 may include functionality for testing the validated AI / ML model using testing data in order to evaluate its performance.

[0035] The ML emulation 110 of the emulation phase 108 may include functionality for generating one or more inferences using the AI / ML model within an emulation environment for purposes of evaluating the performance of the AI / ML model prior to applying it at the target network or system.4P68765WO1 4897-2098-5188\1

[0036] The ML entity loading 114 of the deployment phase 112 may include functionality for making a trained AI / ML model available for use at / within a target AI / ML inference function.

[0037] The AI / ML inference 118 of the inference phase 116 may represent functionality for real-world use of inferencing with the AI / ML model as part of the aforementioned AI / ML inference function.

[0038] It is contemplated that an AI / ML model corresponding to the AI / ML operational workflow 100 may travel between the various states within the AI / ML operational workflow 100 as indicated by the illustrated arrows during its lifecycle.

[0039] Note that additional details for the AI / ML operational workflow 100 may be as provided in a specification for a wireless communication system see, e.g., 3 GPP Technical Specification (TS) 28.105 “Artificial Intelligence / Machine Learning (AI / ML) management,” v. 18.4.0 (June 2024) (hereinafter “TS 28.105”)). However, it may be that specification for wireless communication systems do not provide a unified AI / ML lifecycle management (LCM) and workflow.

[0040] FIG. 2 illustrates an AI / ML functional framework 200. The AI / ML functional framework 200 may apply in cases of, for example, AI / ML model use over / with NR air interfaces.

[0041] As illustrated, the AI / ML functional framework 200 uses a data collection function 202, a model training function 204, a management function 206, an inference function 208, and a model storage function 210. The data collection function 202 may send training data 212 to the model training function 204, may send monitoring data 214 to the management function 206, and / or may send inference data 216 to the inference function 208.

[0042] The model training function 204 may send a trained / updated model 224 to the model storage function 210.

[0043] The management function 206 may send a performance feedback / retraining request 218 to the model training function 204, may send a model transfer / delivery request 226 to the model storage function 210, and / or may send a management instruction 222 to the inference function 208.

[0044] The inference function 208 may send an inference output 220 to the management function 206.5P68765WO1 4897-2098-5188\1

[0045] The model storage function 210 may perform model transfer / delivery 228 to the inference function 208.

[0046] Note that additional details for an AI / ML functional framework 200 may be as provided in a specification for a wireless communication system see, e.g., 3GPP TS 38.843 “Study on Artificial Intelligence (AI) / Machine Learning (ML) for NR air interface,” v. 18.0.0 (December 2023)).

[0047] Various aspects of AI / ML model joint training are now discussed.

[0048] It may be considered that each of one or more AI / ML models supports / generates a certain type of inference. An AI / ML inference function may accordingly use one or more of the AI / ML models to perform one or more inference(s). When multiple AI / ML models are employed, these AI / ML models may operate together in a coordinated way (e.g., such as in a sequence, or even in a more complicated structure / more tightly coordinated manner). In such a case, any change in the performance of one AI / ML model may impact the operation of / result of another, and consequently impact the overall performance of the overall AI / ML inference function.

[0049] There are different ways in which the group of AI / ML models may coordinate. An example is the case where the output of one AI / ML model can be used as input to another AI / ML model, thereby forming a sequence of interlinked AI / ML models.

[0050] Another example is a case where multiple AI / ML models provide an output in parallel. For such examples, it may be that the AI / ML models all output the same output type, where the outputs may then be merged (e.g., using weights). Alternatively, it may be that the outputs of the AI / ML models are used in parallel as inputs to another AI / ML model.

[0051] In whatever circumstance, it will be understood that the group of AI / ML models is employed in a coordinated way in order to support the AI / ML inference function.

[0052] Accordingly, it is desirable that the AI / ML models can be trained or re-trained in a joint manner, so that the group of these AI / ML models can complete a more complex task jointly as described, while achieving a target level of performance.

[0053] The AI / ML model joint training may be initiated by an ML training management service (MnS) producer or an ML training MnS consumer, with the grouping of the AI / ML models shared by the ML training MnS producer with the ML training MnS consumer.6P68765WO1 4897-2098-5188\1

[0054] It will be understood that the control / manipulation of an AI / ML model as described herein may in some contexts be understood to occur through an “ML entity,” which is a manageable artifact of an AI / ML model. An ML entity may contain metadata related to the model. Such metadata may include, for example, an applicable runtime context for the AI / ML model.

[0055] See also 3GPP TS 28.105, section 6.2a.1.2.6.

[0056] Various aspects of AI / ML model joint testing are now discussed.

[0057] As contemplated by discussion herein, a group of AI / ML models may work in a coordinated manner to effectuate results for complex use cases. In such cases, an AI / ML model may represent just one of multiple steps / one portion of an inference process effectuated through an AI / ML inference function (e.g., a case where the inference outputs of the one AI / ML model are used as the inputs to the next AI / ML model that is also used for the inference processes of the AI / ML inference function).

[0058] Such a group of AI / ML models may be generated by an ML training function. The group, including all contained AI / ML models, needs to be tested. After the ML testing of the group, an MnS producer provides the testing results to a consumer. Note that the use case under discussion here may be understood to relate to AI / ML model testing during a training phase and not necessarily to testing cases in the context of AI / ML models that have been deployed.

[0059] See also 3GPP TS 28.105, section 6.2a.3.2.3.

[0060] Various possibilities for site / location of an AI / ML model as used for inferencing are possible. A first such possibility is an AI / ML model where the inference is performed entirely at the network. Such AI / ML models may be referred to herein as “network- sided AI / ML models.”

[0061] Another such possibility corresponds to a use of paired AI / ML model(s) over which a joint inference is performed across the UE and the network (e.g., a first part of the inference is performed by the UE and then a remaining part of the inference is performed by a base station at the network, or vice versa). Such AI / ML models may be referred to herein as “two-sided (AI / ML) models.”

[0062] Another such possibility is an AI / ML model where the inference is performed entirely at the UE. Such AI / ML models may be referred to herein as “UE-sided AI / ML models.”7P68765WO1 4897-2098-5188\1

[0063] Various embodiments herein relate to the use of groups of AI / ML models by a UE. In such contexts, it may be understood that an AI / ML model that is “at” a UE relates to either a UE-sided AI / ML model or a UE part of a two-sided AI / ML model, and is focused on the corresponding inferencing behavior that is being performed at the UE. Herein, such groups of AI / ML models may sometimes also / alternatively be referred to herein as “AI / ML model groups.”

[0064] FIG. 3 illustrates a UE 300 that includes an AI / ML model group 302, according to embodiments discussed herein. As illustrated, the AI / ML model group 302 includes an AI / ML model for channel state information (CSI) 304 that is a UE 300 part of a two- sided AI / ML model. Further, the AI / ML model group 302 includes an AI / ML model for beam management 306 that is a UE-sided AI / ML model. Finally, the AI / ML model group 302 includes an AI / ML model for positioning 308 that is a UE-sided AI / ML model.

[0065] Note that it is contemplated that any type of AI / ML model (UE-sided, two- sided, network- sided) could be trained at one or more training entities (e.g., at a UE, at a server (such as an AI / ML model group management server), and / or at a network (e.g., by an AI / ML model network function)) that is other than the site of the AI / ML model as understood for purposes of the operational use for the AI / ML model once deployed for inferencing purposes.

[0066] It is noted that, in some cases, joint training of a group of AI / ML models (as described elsewhere herein) is not possible or feasible. For example, it may be that, from among the group of AI / ML models that are intended to work together to develop a joint inference, individual ones depend on vendor-specific software and / or hardware deployment choices that are not shared / communicated between vendors and / or are not otherwise ascertainable ahead of time. The individual AI / ML model(s) of the group according to such circumstances may accordingly be “black boxes” that are provided to the UE as pre-trained at another entity of the wireless communication system (e.g., that is controlled by the corresponding vendor).

[0067] As a UE may not be in control of (the entirety of) the training for each of the group of AI / ML models it is using in such cases (and thus the existence of a full joint training at / by the UE may not be assumed), it is accordingly beneficial to develop mechanisms for group AI / ML model testing and / or monitoring that account for this circumstance. Such testing / monitoring permits the UE to react in cases where the group8P68765WO1 4897-2098-5188\1of AI / ML models will not perform / is not performing as desired. Within such contexts, the interaction between different entities (e.g., network entities, UE entities, and any corresponding vendor entities for the same) may be defined to enable such testing and performance monitoring for such a group of AI / ML models (both before and after deployment).

[0068] Accordingly, embodiments herein relate to mechanisms for validation and / or monitoring of an overall performance of a group of UE AI / ML models (that includes one or more UE-sided AI / ML models and / or UE parts of one or more two-sided AI / ML model) corresponding to cases where the AI / ML models are “owned” (e.g., trained by / at) by different entities.

[0069] FIG. 4 A, FIG. 4B, and FIG. 4C illustrate various examples of how a group of AI / ML models may work in a coordinated manner to generate an inference, according to embodiments discussed herein.

[0070] FIG. 4A illustrates a first type of coordination where an output 406 of a first AI / ML model 402 of the group is also an input of a second AI / ML model 404 of the group.

[0071] FIG. 4B illustrates a second type of coordination where an output 408 of a first AI / ML model 410 controls the selection of one of multiple next AI / ML models 420 (selection of one of the second AI / ML model 412, the third AI / ML model 414, the fourth AI / ML model 416, and / or the fifth AI / ML model 418) for use as a second stage of generating an inference. In such cases, group performance testing may be used to determine whether this arrangement as trained will select a correct / expected one of the multiple next AI / ML models 420.

[0072] FIG. 4C illustrates a third type of coordination where one or more group inferences are generated by multiple AI / ML models 422. The multiple AI / ML models 422 may include the first AI / ML model 424, the second AI / ML model 426, the third AI / ML model 428, and the fourth AI / ML model 430, as illustrated. In the example of FIG. 4C, the multiple AI / ML models 422 jointly perform a first group inference 432 that is subject to a power requirement threshold, as illustrated. Further, the multiple AI / ML models 422 jointly perform a second group inference 434 that is subject to a latency requirement threshold, as illustrated. In such cases, group performance testing may be used to ensure that applicable thresholds (e.g., for the overall latency or power, as illustrated) for certain scenarios are met.9P68765WO1 4897-2098-5188\1

[0073] FIG. 5 illustrates an AI / ML operational workflow 500. The AI / ML operational workflow 500 may represent one embodiment for a “lifecycle” of an AI / ML model that is used within a group of AI / ML models for joint inferencing.

[0074] The AI / ML operational workflow 500 may be understood as a modified version of the AI / ML operational workflow 100 of FIG. 1, with analogous elements retaining the reference numerals used from FIG. 1.

[0075] In addition to the element discussed in relation to FIG. 1, the AI / ML operational workflow 500 further includes a group testing phase 502 implementing ML group testing 504. The ML group testing 504 of the group testing phase 502 may include functionality for testing the AI / ML model as part of an applicable AI / ML model group (an AI / ML model group in which the AI / ML model participates to generate a joint inference).

[0076] Further, in FIG. 5, the inference phase 116 further implements ML group monitoring 506. The ML group monitoring 506 of the inference phase 116 may include functionality for monitoring the operation of the AI / ML model group in which the AI / ML model participates, such that the UE can react appropriately to any inaccurate, unexpected, undefined, and / or out-of-bounds output from that AI / ML model group.

[0077] As in FIG. 1, it is contemplated that an AI / ML model corresponding to the AI / ML operational workflow 500 of FIG. 5 may travel between the various states within the AI / ML operational workflow 500 as indicated by the illustrated arrows during its lifecycle.

[0078] Embodiments herein relate to the testing and / or monitoring of the performance of group of AI / ML models sourced from different entities. Various embodiments herein discuss aspects of discovery / determination of dependencies between different AI / ML models.

[0079] Various embodiments herein discuss aspects of interactions between a network of a wireless communication system, a UE of the wireless communication system, and a third party server capable of communicating with one or both of these such that group testing is enabled for cases where joint training did not occur / was not feasible / possible. In some embodiments, results of the group testing are used to accept or reject a new / updated AI / ML model and / or to trigger an update of an AI / ML model.10P68765WO1 4897-2098-5188\1Embodiments for Group Testing for AI / ML Model Approval

[0080] FIG. 6 illustrates a diagram 600 showing communications between a network 602 and a server 604 that may occur with respect to an AI / ML model that is intended for use at a UE 606 as part of an AI / ML model group 608. The operative entity at the network 602 may be an AI / ML model network function. The server 604 may be an AI / ML model group management server.

[0081] The AI / ML model under discussion could be any AI / ML model type as represented in the AI / ML model group 608. As shown, this means that the AI / ML model could have undergone any one or more of network training 610, UE training 612, and / or server training 614 as part of its training phase. In other words, the AI / ML model under discussion could have been generated by the UE 606 or provided to the UE 606 by the network 602, the server 604 or some other device that is other than the UE 606.

[0082] The network 602 may send the server 604 an AI / ML model approval request 616 that requests that an AI / ML model be added to the UE 606. The AI / ML model approval request 616 may include the AI / ML model itself. In some cases, the AI / ML model may be derived at / by the network 602 using some form of network testing during a testing and training phase 624 at the network, as illustrated. In other cases, the AI / ML model is initially sourced by the network 602 from some other entity and simply passed on to the server 604 using the AI / ML model approval request 616.

[0083] Prior to adding the AI / ML model to the AI / ML model group 608 at the UE 606, the AI / ML model should first pass group testing 618 as part of the AI / ML model group 608 (e.g., the testing occurs with the AI / ML model provisionally participating in the AI / ML model group 608). The group testing 618 may be performed at the server 604, as shown.

[0084] Various mechanisms of the group testing 618 are contemplated. For example, the group testing 618 may test for a UE AI / ML aggregated power used by the AI / ML model group 608. The UE AI / ML aggregated power may represent a power use at the UE 606 attributable to the operation of the AI / ML model group 608 as a whole. This test may be used to ensure that the overall power consumption of the AI / ML model group 608 at the UE 606 will not exceed a corresponding threshold.

[0085] In another example, the group testing 618 may test for UE AI / ML aggregated computation resources used by the AI / ML model group 608. The UE AI / ML aggregated computation resources may represent the computation resources used by the AI / ML11P68765WO1 4897-2098-5188\1model group 608 as a whole. This test may be used to ensure that the overall computation resources for the AI / ML model group 608 at the UE 606 will not exceed a corresponding threshold.

[0086] In another example, the group testing 618 may test group performance of the AI / ML model group 608 to ensure that it is / remains within acceptable bounds once the AI / ML model under discussion is used as part of the AI / ML model group 608.

[0087] In another example, the group testing 618 may test for a feature AI / ML aggregated power used by the AI / ML model group 608. The feature AI / ML aggregated power may represent a power use at the UE 606 attributable to one feature of the operation of the AI / ML model group 608 (e.g., a beam management feature, a CSI feature, a mobility feature, etc.). This test may be used to ensure that the power consumption of the AI / ML model group 608 at the UE 606 with respect to that feature will not exceed a corresponding threshold.

[0088] In another example, the group testing 618 may test for feature AI / ML aggregated computation resources used by the AI / ML model group 608. The feature AI / ML aggregated computation resources may represent the computation resources used at the UE attributable to one feature of the operation of the AI / ML model group 608 (e.g., a beam management feature, a CSI feature, a mobility feature, etc.). This test may be used to ensure that the computation resources for the AI / ML model group 608 at the UE 606 with respect to that feature will not exceed a corresponding threshold.

[0089] In another example, the group testing 618 may test the performance of a feature of the AI / ML model group 608 (e.g., a beam management feature, a CSI feature, a mobility feature, etc.) to ensure that it is / remains within acceptable bounds once the AI / ML model under discussion is used as part of the AI / ML model group 608.

[0090] Once the group testing 618 is complete, the server 604 responds to the network 602 with one of an approval indication 620 or a rejection indication 622. In the case that the AI / ML model passes the group testing 618, the approval indication 620 is used. The approval indication 620 indicates to the network 602 that the AI / ML model is approved for use.

[0091] In the case that the AI / ML model does not pass the group testing 618, the rejection indication 622 is used. The rejection indication 622 indicates to the network 602 that the AI / ML model is not approved for use. As illustrated, the rejection indication 622 may include a cause of the rejection.12P68765WO1 4897-2098-5188\1

[0092] FIG. 7 illustrates a diagram 700 showing communications between a network 702, a server 704, and a UE 706 that may occur corresponding to group testing for an AI / ML model for approval for use within an AI / ML model group at the UE 706, according to embodiments discussed herein. The operative entity at the network 702 may be an AI / ML model network function. The server 704 may be an AI / ML model group management server.

[0093] Preliminarily, the network 702 performs ML training 708 and / or ML testing 710 for an individual AI / ML model (or, in other cases, a joint training and testing for multiple AI / ML models).

[0094] The network 702 then sends the server 704 an AI / ML model approval request 712. This AI / ML model approval request 712 acts as a request for approval from the server 704 for the network 702 to distribute the AI / ML model to one or more UEs (e.g., the UE 706). The network 702 includes one or AI / ML models to be tested in the AI / ML model approval request 712.

[0095] Further, in some cases, the network 702 may include a device list in the AI / ML model approval request 712, indicating that the requested approval is to be for a particular subset of devices in the applicable wireless communication system (e.g., certain device types, devices of certain vendors, etc.).

[0096] The server 704 then runs a pair of regression tests corresponding to the functionality for which the AI / ML model will operate (e.g., beam management, CSI, mobility, etc., as the case may be). In the first regression test 714, the server 704 performs a regression over a representative set of scenarios without including / using the AI / ML model in question. The result of this regression is used as a baseline for different key performance indicator(s) (KPI(s)) (performance, power use, etc.) of the corresponding functionality.

[0097] In the second regression test 716, the server 704 runs another regression corresponding to the functionality using the AI / ML model group (including the AI / ML model under test).

[0098] The overall result of the group testing performed by the server 704 is determined by performing a comparison 718 of the individual results of the first regression test 714 and the second regression test 716. As part of this comparison, applicable KPI(s) for the functionality in question under each regression test are13P68765WO1 4897-2098-5188\1compared. Based on this comparison, the server 704 determines whether or not the AI / ML model passes the group testing procedure.

[0099] The qualifications for a “pass” may be set by the server 704. In some embodiments, group testing passes if the regression testing that includes the AI / ML model achieves the same or better KPIs than the regression testing that does not use the AI / ML. In other embodiments, the server 704 may define another rule. For example, if the new AI / ML model targets a significant enhancement in performance, the server 704 may instead accept some level of the degradation in power-related KPIs.

[0100] The server 704 then sends the group testing result to the network 702 in the AI / ML model approval response 720. The embodiment of FIG. 7 illustrates a case where the AI / ML model approval response 720 indicates that the AI / ML model is approved for use (e.g., a case where the AI / ML model passed the testing performed by the server 704).

[0101] In some cases, an AI / ML model approval response 720 might indicate that the AI / ML model is approved for use for all requested devices indicated in the AI / ML model approval request 712. In some cases, the AI / ML model approval response 720 could indicate that the AI / ML model is approved for specific types of device types (in which case, this information should be included in the response, as shown).

[0102] In other (non-illustrated) cases, the AI / ML model approval response 720 might instead indicate that the use of the AI / ML model is rejected (e.g., in a case where the AI / ML model did not pass the testing performed by the server 704).

[0103] Note that an approval (e.g., a certificate) as indicated by the server 704 may specify one or more conditions upon the use of the AI / ML model. Examples of such conditions include, but are not limited to, a time limit condition, a spatial limit condition, a UE power level condition, a geographic location condition, a UE type condition, etc.

[0104] In the event that (as illustrated) the AI / ML model approval request 712 is indeed approved, the server 704 may provide the UE 706 with an allowed model(s) indication 722 that includes a listing of one or more AI / ML models that are approved for use (and which includes the AI / ML model in question).

[0105] Then, in the event that the network 702 sends a new AI / ML model to the UE 706 for use (e.g., using the illustrated AI / ML model update message 724), the UE 706 may first check if the AI / ML model is approved (or not) using the information from the allowed model(s) indication 722.14P68765WO1 4897-2098-5188\1

[0106] Note that in some embodiments, the server 704 is configured to perform its operations as here explained with respect to / for some particular category or type of UEs (e.g., UEs of a particular vendor).

[0107] FIG. 8A, FIG. 8B, and FIG. 8C illustrate different ways that a group testing mechanism could impact an interaction between a UE 804 and a network 806.

[0108] FIG. 8A illustrates a flow diagram 802 for communications between a UE 804 and a network 806 in a UE registration context. The operative entity at the network 806 may be an AI / ML model network function.

[0109] The UE 804 sends the network 806 a registration request message 808. As part of the registration request message 808, the UE 804 may inform the network 806 about the allowed AI / ML models (as received from a server (e.g., an AI / ML model group management server), as previously discussed).

[0110] The flow diagram 802 illustrates that the registration procedure then continues, with the network 806 responding to the registration request message 808 with a registration acceptance message 810, and with the UE 804 responding to the registration acceptance message 810 with a registration complete message 812.[OHl] The network 806 may then send an AI / ML model update request message 814 to the UE 804 that requests the UE 804 to include / use the AI / ML model. This request for the inclusion / use of the AI / ML model may be used in cases where the AI / ML model was previously indicated as an allowed AI / ML model as part of the registration request message 808.

[0112] Assuming that the requested AI / ML model is still allowed, the UE 804 may reply to the AI / ML model update request message 814 by sending the network 806 an AI / ML model update accept message 816 and begins using the AI / ML model.

[0113] If, on the other hand, the UE 804 determines that the requested AI / ML model is not included in the allowed listing of AI / ML models, then the UE 804 replies to the AI / ML model update request message 814 with an AI / ML model update reject message 818. In some such cases, the AI / ML model update reject message 818 may include a cause (e.g., not allowed) for the rejection.

[0114] Note that this final check by the UE 804 after receiving the AI / ML model update request message 814 may be useful for cases where an AI / ML model was allowed at the time the registration request message 808 was sent by the UE 804 but is no longer15P68765WO1 4897-2098-5188\1allowed by the time of the AI / ML model update request message 814 (e.g., due to nonillustrated communications of the UE 804 with a server).

[0115] FIG. 8B illustrates a flow diagram 820 for communications between a UE 822 and a network 824 in an identification procedure context. The operative entity at the network 824 may be an AI / ML model network function.

[0116] Preliminarily, the network 824 send the UE 822 an identity request message 826. The identity request message 826 includes a request for an international mobile station equipment identity software version (IMEISV) indication corresponding to the UE 822, as illustrated.

[0117] The flow diagram 802 illustrates that the identification procedure then continues, with the UE 822 responding to the identity request message 826 with an identity response message 828 that includes the IMEISV.

[0118] Using the IMEISV received in the identity response message 828, the network 824 identifies a type of the UE 822. The network 824 uses this information to then identify one or more AI / ML models that are allowed for this type of device (e.g., based on an allowed list as previously received from a server (e.g., an AI / ML model group management server), as previously discussed). The network 824 may then send an AI / ML model update request message 830 to the UE 822 that requests the UE 822 to include / use an AI / ML model so identified based on the IMEISV of the UE 822.

[0119] Assuming that the requested AI / ML model is still allowed, the UE 822 may reply to the AI / ML model update request message 830 by sending the network 824 an AI / ML model update accept message 832 and begins the use of the AI / ML model.

[0120] If, on the other hand, the UE 822 determines that the requested AI / ML model is not included in an allowed listing of AI / ML models, then the UE 822 replies to the AI / ML model update request message 830 with an AI / ML model update reject message 834. In some such cases, AI / ML model update reject message 834 may include a cause (e.g., not allowed) for the rejection.

[0121] Note that this final check by the UE 822 after receiving the AI / ML model update request message 830 may be useful for cases where information at the network 824 about allowed AI / ML models for an IMEISV is out of date (e.g., due to non-illustrated communications of the UE 804 with a server having updated IMEISV correspondence information that has not yet been provided to the network 824).16P68765WO1 4897-2098-5188\1

[0122] FIG. 8C illustrates a flow diagram 836 for communications between a UE 838 and a server 840 in a certificate request context. The operative entity of the server 840 may be an AI / ML model group management server.

[0123] FIG. 8C illustrates that the UE server 840 sends the UE 838 an AI / ML model update message 842.

[0124] The AI / ML model update message 842 may be sent as part of a certificate request received at the server 840 from a network (e.g., an AI / ML model network function at the network). The network may request the server 840 to provide a listing of approved AI / ML models to correspondingly approved devices. Accordingly, the AI / ML model update message 842 is sent to the UE 838 in order to establish / update the UE 838 with up-to-date information about AI / ML models that are approved for use at the UE 838.Embodiments for AI / ML Model Group Monitoring

[0125] It is contemplated that, after adding an AI / ML model to the UE for use in the AI / ML model group, the UE and / or the server may continue to monitor the performance of the AI / ML model / AI / ML model group to ensure that the system meets desired group KPIs (e.g., including UE and / or feature aggregated power KPI(s), UE and / or feature aggregated computation KPI(s), and / or group performance KPI(s), etc.).

[0126] FIG. 9 illustrates a diagram 900 showing communications between a network 602 and a server 604 that may occur with respect to updating an AI / ML model that is used at a UE 606 as part of an AI / ML model group 608. The operative entity at the network 602 may be an AI / ML model network function. The server 604 may be an AI / ML model group management server.

[0127] Note that mechanism illustrated in FIG. 9 uses many analogous elements to that of FIG. 6. In such cases, the reference numerals from FIG. 6 are re-used and should be understood analogously in the context of a procedure for updating an existing AI / ML model at the UE 606.

[0128] As illustrated, upon a determination that there has been a degradation in group performance and / or change in the desired / applicable KPIs (e.g., as when new use case becomes applicable), the server 604 may send the network 602 a model update request 902 that triggers the network 602 to update an applicable AI / ML model. As illustrated,17P68765WO1 4897-2098-5188\1the model update request 902 may include new requirement(s) that the AI / ML model should satisfy corresponding to the anticipated update of the AI / ML model.

[0129] The AI / ML model may be updated through the use of a testing and training phase 624 at the network 602 (which may be carried out in view of any new requirement(s) provided in the model update request 902), and then the AI / ML model may be approved by the server 604 using the rest of the illustrated procedure from the AI / ML model approval request 616 onwards, in analogous manner to that which was described in the discussion of FIG. 6 herein.

[0130] FIG. 10 illustrates a flow diagram 1000 for the performance of AI / ML model group monitoring, according to embodiments discussed herein. The flow diagram 1000 may describe behaviors carried out by a server (e.g., an AI / ML model group management server) as discussed herein.

[0131] In addition to any monitoring of an individual AI / ML model, the server may perform AI / ML model group monitoring that tests for any undesirable degradation in functional or non-functional requirements of the AI / ML model group. Accordingly, the flow diagram 1000 illustrates that the server starts 1002 the AI / ML model group monitoring.

[0132] As illustrated, the AI / ML group monitoring may be triggered 1004 on a periodic basis. Alternatively and / or additionally, the AI / ML group monitoring may be triggered 1004 by some monitored-for event. Such an event may be, for example, a reported delay in a handover scenario that impacts UE performance, a detection of an increase in power consumption at the UE that is abnormal / greater than usual, etc.

[0133] As part of the AI / ML model group monitoring, the server performs 1006 a regression test that includes the models in the AI / ML model group. The result of the regression test may be compared with a reference result (e.g., another regression test that does not include / use the AI / ML model) in order to determine an overall result of the testing process generally.

[0134] If the server determines 1008 that the overall result of the testing is a pass, then no more action is needed and the server ends 1010 (the present instance of) the flow diagram 1000 (e.g., until another trigger 1004 occurs).

[0135] If the server determines 1008 that the overall result of the testing is a failure, then the server performs 1012 additional tests to determine a root cause of the failure and any corresponding corrective actions (e.g., corresponding to one or more identified18P68765WO1 4897-2098-5188\1AI / ML model(s) of the AI / ML model group). Such corrective actions may include, for example: a reversion from a use of a current version of an AI / ML model to a use of a prior version of that AI / ML model, a reversion from a use of an AI / ML model to classical code (e.g., to non-AI / ML-based code), and / or a trigger / instruction to re- train / update an AI / ML model.

[0136] The server then informs 1014 the network (e.g., an AI / ML model network function) and the UE about the degradation and any corresponding corrective actions.

[0137] FIG. 11 illustrates a flow diagram 1100 for communications between a server 1102, a UE 1104, and a network 1106 for AI / ML model group monitoring, according to embodiments discussed herein. The operative entity at the network 1106 may be an AI / ML model network function. The server 1102 may be an AI / ML model group management server.

[0138] As illustrated, once the server 1102 identifies an AI / ML model of a group of AI / ML models that is a root cause of undesirable characteristics of the AI / ML model group, the server 1102 sends the UE 1104 an AL / ML model update indication 1108. The AL / ML model update indication 1108 may include an action list that instructs the UE 1104 in an action that is to be carried out with respect to the identified AI / ML model.

[0139] The UE 1104 accordingly performs 1110 the requested corrective action. This corrective action may be, for example, a reversion from a use of a current version of the AI / ML model to a use of a prior version of the AI / ML model, and / or an instruction to remove the AI / ML model from a listing of approved AI / ML models.

[0140] Then, as illustrated, the UE 1104 may send an AI / ML model update indication 1112 to the network 1106. The AI / ML model update indication 1112 may include a new listing of approved AI / ML models that has been updated per the carrying out of the instructed corrective action (e.g., an updated listing of allowed models that accounts for any removal of the AI / ML model).

[0141] FIG. 12 illustrates a flow diagram 1200 for communications between a server 1202, a UE 1204, and a network 1206 for AI / ML model group monitoring, according to embodiments discussed herein. The operative entity at the network 1206 may be an AI / ML model network function. The server 1202 may be an AI / ML model group management server.

[0142] As illustrated, once the server 1202 identifies an AI / ML model of a group of AI / ML models that is a root cause of undesirable characteristics of the AI / ML model19P68765WO1 4897-2098-5188\1group, the server 1202 sends the network 1206 an AI / ML model update indication 1208. The AI / ML model update indication 1208 may include an action list that instructs the network 1206 in an action that is to be carried out with respect to the identified AI / ML model.

[0143] The network 1206 accordingly performs 1210 the requested corrective action. This corrective action may be, for example, the performance of additional training for the AI / ML model and the use of a corresponding new approval request or simply a sending of a new (e.g., replacement) AI / ML model to the UE.

[0144] Then, as illustrated, the network 1206 may send an AI / ML model update request message 1212 to the UE 1204. The AI / ML model update request message 1212 may include an updated / new (as the case may be) AI / ML model to the UE 1204 that is on the (updated) listing of approved AI / ML models.

[0145] If the new / updated AI / ML model is accepted at the UE 1204, the UE 1204 sends the network 1206 an AI / ML model update accept message 1214.

[0146] If the new / updated AI / ML model is not accepted at the UE 1204 (e.g., due to an intervening update of a listing of approved AI / ML models from the server 1202 to the UE 1204 after the time of the AI / ML model update indication 1208), the UE 1204 instead sends the network 1206 an AI / ML model update reject message 1216. As illustrated, this AI / ML model update reject message 1216 may include an indication of the cause (e.g., not allowed) for the rejection of the AI / ML model to the network 1206.

[0147] When a new / updated AI / ML model is provided / used with respect to one feature, it may be that, based on AI / ML model group testing, it is desirable to also change an AI / ML model for a different feature. For example, when an AI / ML model used for beam prediction is changed, it may be beneficial to also change an AI / ML model used for mobility prediction as well in order to ensure optimum or acceptable performance across the AI / ML model group generally.

[0148] The server or network may provide / define groupings of AI / ML models for different features that should be used simultaneously. To enable this desired flexibility under the AI / ML model group testing mechanism, it may be that dependencies between AI / ML models are described in a table object, as in the below table:20P68765WO1 4897-2098-5188\1Table 1 : Example Model Dependencies

[0149] Table 1 illustrates an example of dependencies that may exist between AI / ML models. For example, assume that “Model B” in Table 1 is a mobility-related AI / ML model while “Model D” is a positioning-related model. Table 1 illustrates that Model B depends on both itself and on Model D.

[0150] FIG. 13 illustrates an example 1300 of a model identification mechanism that may be used to identify an AI / ML model, according to embodiments discussed herein.

[0151] Model identification may facilitate the communication of model dependencies. For example, it may be understood that model identification relates to a process / method of identifying an AI / ML model in a way that uses a common understanding between the network, the server, and / or the UE.

[0152] The example 1300 includes various parts represented by one or more bits, as illustrated. A first of these parts may be a model type part 1302 that identifies whether the AI / ML model is a UE-sided AI / ML model, a network-sided AI / ML model, or a two- sided AI / ML model.21P68765WO1 4897-2098-5188\1

[0153] A second of these parts is a model group part 1304 that indicates a feature for which the AI / ML model may be used (e.g., CSI, beam management, mobility, etc.).

[0154] A third of these parts is a model sub-group part 1306. The model sub-group part 1306 may identify a sub-category within the feature of the model group part 1304 for which the AI / ML model may be used. For example, if the model group part 1304 indicates that the AI / ML model is for a mobility feature, the model sub-group part 1306 may indicate that the AI / ML model is (more particularly) for use in handover cases that fall under the overall mobility feature.

[0155] A fourth of these parts includes an AI / ML model identifier (ID) part 1308 that provides an ID for the AI / ML model.

[0156] A fifth of these parts includes an AI / ML model version part 1310 that relates that the AI / ML model is a particular version of the AI / ML model identified by the model ID part 1308.

[0157] Corresponding to various embodiments, a network (e.g., an AI / ML model network function) may provide a UE with different AI / ML models to be used in different locations. For example, the network may provide / be able to provide a UE with a first AI / ML model for handover that is to be used in the United States and a second (substantively different) AI / ML model for handover that is to be used in Germany. Accordingly, a server (e.g., an AI / ML model group management server) may perform different tests to test the performance of an AI / ML model for these different use cases (e.g., to test the UE performance in different countries). Further, the server may provide different certificates to the network and / or the UE based on the different use cases.

[0158] Accordingly, when a UE moves from one location to another, before switching to a new AI / ML model, the UE may first check that the new AI / ML model has passed a corresponding AI / ML model group testing check.

[0159] It may be in some situations that one or more KPIs could be ignored. For example, power-consumption-related KPIs could be ignored if the AI / ML model is to be for rare scenarios and / or for short periods.

[0160] In some cases, a dependency between AI / ML models may be represented by a scalar 0 < k < 1 where X relates some amount of dependency on a variable basis. For example, there may be no dependence when = 0, some amount of dependency when = 0.5, and full dependency when = 1. Such dependencies may be used to prioritize AI / ML model group testing performance among the various AI / ML models.22P68765WO1 4897-2098-5188\1Additional Embodiments for UE-Centric AI / ML Model Group Testing

[0161] FIG. 14 illustrates a diagram 1400 showing communications between a network 1402, a UE 1404, and a server 1406 that may occur corresponding to UE-centric group testing for an AI / ML model for approval for use within an AI / ML model group at the UE 1404, according to embodiments discussed herein. The operative entity at the network 1402 may be an AI / ML model network function. The server 1406 may be an AI / ML model group management server.

[0162] Preliminarily, the network 1402 performs ML training 1408 and / or ML testing 1410 for an individual AI / ML model (or, in other cases, a joint training and testing for multiple AI / ML models).

[0163] The network 1402 then sends the UE 1404 an AI / ML model update message 1412 that communicates the new AI / ML model to the UE 1404. The UE 1404 may mark the received AI / ML model as “under-test” until it verifies that its behavior while using the AI / ML model fulfills the relevant KPI(s) (e.g., group KPI(s), including UE and / or feature aggregate power, UE and / or feature aggregate computation, group performance, etc.). To perform the testing, the UE 1404 may depend on prior results and / or in-device regression test(s). For example, the UE 1404 may run an old version of the AI / ML model (or classical code) to determine a baseline as compared to a result generated using the new version of the AI / ML model.

[0164] As illustrated, the UE 1404 runs a pair of regression tests corresponding to the functionality for which the AI / ML model will operate (e.g., beam management, CSI, mobility, etc., as the case may be). In the first regression test 1414, the UE 1404 performs a regression over a representative set of scenarios without including / using the AI / ML model in question. The result of this regression is used as a baseline for different KPI(s) (performance, power use, etc.) of the corresponding functionality.

[0165] In the second regression test 1416, the UE 1404 runs another regression corresponding to the functionality using the AI / ML model group (including the AI / ML model under test).

[0166] The overall result of the group testing performed by the UE 1404 is determined by performing a comparison 1418 of the individual results of the first regression test 1414 and the second regression test 1416. As part of this comparison, applicable KPI(s) for the functionality in question under each regression test are compared. Based on this23P68765WO1 4897-2098-5188\1comparison, the UE 1404 determines whether or not the AI / ML model passed the group testing procedure.

[0167] Then, the UE 1404 may accordingly inform the network 1402 whether the new AI / ML model is accepted (or not). If the new AI / ML model is accepted by the UE 1404, the UE 1404 considers it as “verified model.” The UE 1404 may correspondingly provide the network 1402 with the AI / ML model update accept message 1420 indicating that the new AI / ML model is accepted for use.

[0168] On the other hand, if the new AI / ML model is rejected by the UE 1404, the UE may correspondingly provide the network 1402 with the AI / ML model update reject message 1422. As illustrated, this AI / ML model update reject message 1422 may indicate a cause for the rejection of the AI / ML model to the network 1402.

[0169] As also illustrated, the UE 1404 may send the server 1406 a group testing result 1424 for the testing of the AI / ML model. As part of the group testing result 1424, the UE 1404 may request the server 1406 to store the latest stable / verified AI / ML model(s). For example, the group testing result 1424 may include a request for the server 1406 to store a new AI / ML model in the event that the UE 1404 approves the new AI / ML model in the manner just discussed. The server 1406 may be configured to further share the group testing result 1424 with other devices within the wireless communication system (e.g., other UEs of the wireless communication system that are of the same type as the UE 1404).Additional Embodiments for UE-Centric AI / ML Model Group Monitoring

[0170] FIG. 15 illustrates a diagram 1500 showing communications between a network 1502, a UE 1504, and a server 1506 that may occur corresponding to UE-centric group monitoring for an AI / ML model within an AI / ML model group at the UE 1504, according to embodiments discussed herein. The operative entity at the network 1502 may be an AI / ML model network function. The server 1506 may be an AI / ML model group management server.

[0171] As illustrated, the UE 1504 may perform monitoring, during which it detects / predicts degradation 1508 corresponding to the use of an AI / ML model group. In response, the UE 1504 determines 1510 (identifies) an AI / ML model of the AI / ML model group that is a root cause of the degradation by running additional tests / procedures on the AI / ML models of the group.24P68765WO1 4897-2098-5188\1

[0172] After identifying the AI / ML model that is degraded, the UE 1504 sends the network 1502 an AI / ML model update indication 1512 comprising a degradation report that identifies that the AI / ML model is no longer valid.

[0173] The reception of the AI / ML model update indication 1512 may trigger retraining of the AI / ML model at the network 1502. Accordingly, the network 1502 performs ML training 1514 and / or ML testing 1516 for the AI / ML model (or, in other cases, a joint training and testing for multiple AI / ML models including the AI / ML model).

[0174] The network 1502 then sends the UE 1504 an AI / ML model update request 1518 that includes the new / updated AI / ML model.

[0175] The UE 1504 may then perform group testing for the new / updated AI / ML model (e.g., as was described in relation to FIG. 14). Based on the result of this group testing, the UE 1504 may accordingly inform the network 1502 whether the new AI / ML model is accepted (or not). If the new AI / ML model is accepted by the UE 1504, the UE 1504 considers it as “verified model.” The UE 1504 may correspondingly provide the network 1502 with the AI / ML model update accept message 1520 indicating that the new AI / ML model is accepted for use.

[0176] On the other hand, if the new AI / ML model is rejected by the UE 1504, the UE may correspondingly provide the network 1502 with the AI / ML model update reject message 1522. As illustrated, this AI / ML model update reject message 1522 may indicate a cause for the rejection of the AI / ML model to the network 1502.

[0177] As also illustrated, the UE 1504 may send the server 1506 a group testing result 1524 for the testing of the AI / ML model. As part of the group testing result 1524, the UE 1504 may request the server 1506 to store the latest stable / verified AI / ML model(s). For example, the group testing result 1524 may include a request for the server 1506 to store the new AI / ML model in the event that the UE 1504 approves the new AI / ML model in the manner just discussed. The server 1506 may be configured to further share the group testing result 1524 with other devices within the wireless communication system (e.g., other UEs of the wireless communication system that are of the same type as the UE 1504).

[0178] Note that in some cases, to mitigate degradation, a UE 1504 may request and / or download any latest stable versions of the AI / ML model(s) for the AI / ML model group from the server 1506 and / or the network 1502.25P68765WO1 4897-2098-5188\1Embodiments for Specification Impacts

[0179] FIG. 16 illustrates a diagram 1600 showing cases for impacts of various types of communications between a server 1602, a network 1604, and a UE 1606 as may (or may not) be reflected within a specification for a wireless communication system, according to embodiments herein. The operative entity at the network 1604 may be an AI / ML model network function. The server 1602 may be an AI / ML model group management server. The specification for the wireless communication system may be, for example, a 3 GPP-controlled specification.

[0180] As has been described herein, the performance of AI / ML model group testing and / or group monitoring uses various communication interfaces. Broadly speaking, categories for these interfaces include server-network interface(s) 1608, network-UE interface(s) 1610, and network-UE interface(s) 1612.

[0181] In some cases, it is expected that network-UE interface(s) 1610 are to be defined / controlled per the specification for the wireless communication system.

[0182] Further, it is expected that the specification for the wireless communication system may or may not define the server-network interface(s) 1608. Note that it may be that the server 1602 and the network 1604 may be implemented by different parties (e.g., the server 1602 may be implemented by a UE vendor while the network 1604 is implemented by a network operator). Accordingly, one or more of various corresponding interfaces (NWDAF, NEF and AF interfaces) could be implicated. There may thus be benefits to formally coordinating the uses of these interfaces as between the (potentially different) controlling entities for the server 1602 and the network 1604 through the specification for the wireless communication system.

[0183] Finally, it is contemplated that, in some cases, the network-UE interface(s) 1612 between the UE 1606 and the server 1602 are outside a scope of a specification for the wireless communication system.

[0184] It is further noted that a specification for the wireless communication system may define one or more regression test cases for group testing corresponding to the issuance of an approval for the AI / ML model (the issuance of a new certificate for an AI / ML model).26P68765WO1 4897-2098-5188\1Additional Discussion of Various Embodiments

[0185] Embodiments disclosed herein provide frameworks for group AI / ML testing that allows a wireless communication system to test and / or monitor impacts of individual AI / ML model(s) / AI / ML model group(s). The operative entity at the network for these embodiments may be an AI / ML model network function. A server for these embodiments may be an AI / ML model group management server.

[0186] These frameworks include interaction(s) between a network and a server (e.g., that is operated by a UE vendor) to obtain an AI / ML group test approval / certificate.

[0187] These frameworks include interaction(s) between a network and a server (e.g., that is operated by a UE vendor) for responding to any degradation (e.g., as reported from an AI / ML group monitoring process).

[0188] These frameworks include interaction(s) between a network and a UE to update an AI / ML model. Such interactions contemplate both acceptance and rejection cases (e.g., rejection cases where an AI / ML model is not included in a verified models list).

[0189] These frameworks include interaction(s) between a UE and a network for reporting a degradation on the overall system as part of an AI / ML model group monitoring process.

[0190] FIG. 17 illustrates a method 1700 of a UE operating with a network of a wireless communication system, according to embodiments discussed herein. The method 1700 includes receiving 1702, from an AI / ML model group management server, an indication that a first AI / ML model is approved for use. The method 1700 further includes performing 1704, based on the indication that the first AI / ML model is approved for use, an inference using a group of AI / ML models that includes the first AI / ML model.

[0191] In some embodiments, the method 1700 further includes receiving, from an AI / ML model network function of the network, the first AI / ML model. In some such embodiments, the method 1700 further includes identifying, to the AI / ML model network function, that the first AI / ML model is useable at the UE as part of a registration procedure between the network and the UE, and the first AI / ML model is received from the AI / ML model network function after the registration procedure. In some such embodiments, the method 1700 further includes identifying, to the AI / ML model network function, that the first AI / ML model is useable at the UE as part of an27P68765WO1 4897-2098-5188\1identification procedure between the network and the UE, wherein the first AI / ML model is received from the AI / ML model network function after the identification procedure.

[0192] In some embodiments, the method 1700 further includes receiving, from the AI / ML model group management server, a listing of approved AI / ML models. In some such embodiments, the method 1700 further includes receiving, from an AI / ML model network function of the network, a second AI / ML model; determining that the second AI / ML model is not included in the listing of approved AI / ML models; and rejecting the second AI / ML model based on determining that the second AI / ML model is not included in the listing of approved AI / ML models.

[0193] In some embodiments, the method 1700 further includes sending, to an AI / ML model network function of the network, a listing of approved AI / ML models.

[0194] In some embodiments, the method 1700 further includes receiving, from the AI / ML model group management server, the first AI / ML model.

[0195] In some embodiments, the method 1700 further includes receiving, from the AI / ML model group management server, an instruction to carry out a corrective action with respect to the first AI / ML model; and performing, in response to the instruction, the corrective action. In some such embodiments, the corrective action comprises reverting from using a current version of the first AI / ML model to using a prior version of the first AI / ML model. In some such embodiments, the corrective action comprises reverting from using the first AI / ML model to using a corresponding non-AI / ML-model-based functionality. In some such embodiments, the corrective action comprises removing the first AI / ML model from a listing of approved AI / ML models, and the method 1700 further includes sending, to an AI / ML model network function of the network, the listing of approved AI / ML models after removing the first AI / ML model from the listing of approved AI / ML models.

[0196] In some embodiments, the method 1700 further includes identifying that a condition for using the first AI / ML model is not met; and ending use by the UE of the first AI / ML model in response to identifying that the condition for using the first AI / ML model is not met. In some such embodiments, the condition is received from the AI / ML model group management server. In some such embodiments, the condition is received from an AI / ML model network function. In some such embodiments, the condition comprises a time limit condition. In some such embodiments, the condition comprises a spatial limit condition. In some such embodiments, the condition comprises a power28P68765WO1 4897-2098-5188\1level condition. In some such embodiments, the condition comprises a geographic location condition.

[0197] FIG. 18 illustrates a method 1800 of an AI / ML model group management server, according to embodiments discussed herein. The method 1800 includes receiving 1802, from an AI / ML model network function of a network of a wireless communication system, a request to approve a first AI / ML model for use, the request comprising the first AI / ML model. The method 1800 further includes performing 1804 first group testing of a group of AI / ML models comprising the first AI / ML model and one or more additional AI / ML models. The method 1800 further includes sending 1806, to the AI / ML model network function, based on a first result of the first group testing, a response to the request that indicates that the first AI / ML model is approved for use.

[0198] In some embodiments of the method 1800, performing the first group testing of the group of AI / ML models comprises: performing a first regression test that does not use the first AI / ML model; performing a second regression test that uses the group of AI / ML models comprising the first AI / ML model and the one or more additional AI / ML models; and generating the first result of the first group testing based on a comparison of the first regression test to the second regression test.

[0199] In some embodiments of the method 1800, the request indicates a UE at which the first AI / ML model is intended to be used. In some such embodiments, the response indicates that the approval of the first AI / ML model is with respect to the UE.

[0200] In some embodiments of the method 1800, the response further includes a condition for using the first AI / ML model. In some such embodiments, the condition comprises a time limit condition. In some such embodiments, the condition comprises a spatial limit condition. In some such embodiments, the condition comprises a power level condition. In some such embodiments, the condition comprises a UE type condition. In some such embodiments, the condition comprises a geographic location condition.

[0201] In some embodiments, the method 1800 further includes sending, to a UE of the wireless communication system, an indication that the first AI / ML model is approved for use.

[0202] In some embodiments, the method 1800 further includes sending, to a UE of the wireless communication system, the first AI / ML model.29P68765WO1 4897-2098-5188\1

[0203] In some embodiments, the method 1800 further includes identifying a group monitoring trigger corresponding to the first AI / ML model; performing, in response to the group monitoring trigger, second group testing of the group of AI / ML models; identifying, based on a second result of the second group testing, that the first AI / ML model has degraded; determining a corrective action for the first AI / ML model in response to identifying that the first AI / ML model has degraded; and instructing one of the AI / ML model network function and a UE of the wireless communication system that is using the first AI / ML model to carry out the corrective action. In some such embodiments, the AI / ML model group management server instructs the AI / ML model network function to carry out the corrective action, and wherein the corrective action comprises performing additional training and requesting a new approval for the first AI / ML model. In some such embodiments, the AI / ML model group management server instructs the AI / ML model network function to carry out the corrective action, and wherein the corrective action comprises sending the UE a replacement AI / ML model for the first AI / ML model. In some such embodiments, the AI / ML model group management server instructs the UE to carry out the corrective action, and wherein the corrective action comprises reverting from using a current version of the first AI / ML model to using a prior version of the first AI / ML model. In some such embodiments, the AI / ML model group management server instructs the UE to carry out the corrective action, and wherein the corrective action comprises removing the first AI / ML model from a listing of approved AI / ML models.

[0204] FIG. 19 illustrates a method 1900 of an AI / ML model network function of a network of a wireless communication system, according to embodiments discussed herein. The method 1900 includes sending 1902, to an AI / ML model group management server, a request to approve an AI / ML model for use, the request comprising the AI / ML model. The method 1900 further includes receiving 1904, from the AI / ML model group management server, a response to the request that indicates that the AI / ML model is approved for use.

[0205] In some embodiments of the method 1900, the request indicates a UE of the wireless communication system at which the AI / ML model is intended to be used. In some such embodiments, the response indicates that the approval of the AI / ML model is with respect to the UE.30P68765WO1 4897-2098-5188\1

[0206] In some embodiments of the method 1900, the response further comprises a condition for using the AI / ML model within the wireless communication system. In some such embodiments, the condition comprises a time limit condition. In some such embodiments, the condition comprises a spatial limit condition. In some such embodiments, the condition comprises a power level condition. In some such embodiments, the condition comprises a UE type condition. In some such embodiments, the condition comprises a geographic location condition.

[0207] In some embodiments, the method 1900 further includes sending, to a UE of the wireless communication system, the AI / ML model. In some such embodiments, the method 1900 further includes receiving, from the UE, a listing of approved AI / ML models. In some such embodiments, the method 1900 further includes identifying the AI / ML model from a listing of approved AI / ML models received from the UE as part of a registration procedure between the network and the UE, and the AI / ML model is sent to the UE after the registration procedure. In some embodiments, the method 1900 further includes identifying the AI / ML model from a listing of approved AI / ML models received from the UE as part of an identification procedure between the network and the UE, and the AI / ML model is sent to the UE after the identification procedure.

[0208] In some embodiments, the method 1900 further includes training the AI / ML model.

[0209] In some embodiments, the method 1900 further includes testing the AI / ML model.

[0210] In some embodiments, the method 1900 further includes receiving, from the AI / ML model group management server, an instruction to carry out a corrective action with respect to the AI / ML model; and performing, in response to the instruction, the corrective action. In some such embodiments, the corrective action comprises sending a UE of the wireless communication system a replacement AI / ML model for the AI / ML model. In some such embodiments, the corrective action comprises performing additional training and requesting a new approval for the AI / ML model, and the method 1900 includes sending the AI / ML model to a UE of the wireless communication system after performing the additional training and receiving the new approval.

[0211] FIG. 20 illustrates a method 2000 of a UE operating with a network of a wireless communication system, according to embodiments discussed herein. The method 2000 includes receiving 2002, from an AI / ML model network function of a31P68765WO1 4897-2098-5188\1network of a wireless communication system, a request to approve a first AI / ML model for use, the request comprising the first AI / ML model. The method 2000 further includes performing 2004 first group testing of a group of AI / ML models comprising the first AI / ML model and one or more additional AI / ML models. The method 2000 further includes sending 2006, to the AI / ML model network function, based on a first result of the first group testing, a response to the request that indicates that the first AI / ML model is approved for use.

[0212] In some embodiments, the method 2000 further includes sending the first AI / ML model to an AI / ML model group management server.

[0213] In some embodiments of the method 2000, performing the first group testing of the group of AI / ML models comprises: performing a first regression test that does not use the first AI / ML model; performing a second regression test that uses the group of AI / ML models comprising the first AI / ML model and the one or more additional AI / ML models; and generating the first result of the first group testing based on a comparison of the first regression test to the second regression test.

[0214] In some embodiments, the method 2000 further includes performing second group testing of the group of AI / ML models; identifying, based on a second result of the second group testing, that the first AI / ML model has degraded; and sending, to the AI / ML model network function, an indication that the first AI / ML model is invalid.

[0215] FIG. 21 illustrates an example architecture of a wireless communication system 2100, according to embodiments disclosed herein. The following description is provided for an example wireless communication system 2100 that operates in conjunction with the LTE system standards and / or 5G or NR system standards as provided by 3 GPP technical specifications.

[0216] As shown by FIG. 21, the wireless communication system 2100 includes UE 2102 and UE 2104 (although any number of UEs may be used). In this example, the UE 2102 and the UE 2104 are illustrated as smartphones (e.g., handheld touchscreen mobile computing devices connectable to one or more cellular networks), but may also comprise any mobile or non-mobile computing device configured for wireless communication.

[0217] The UE 2102 and UE 2104 may be configured to communicatively couple with a RAN 2106. In embodiments, the RAN 2106 may be NG-RAN, E-UTRAN, etc. The UE 2102 and UE 2104 utilize connections (or channels) (shown as connection 2108 and connection 2110, respectively) with the RAN 2106, each of which comprises a physical32P68765WO1 4897-2098-5188\1communications interface. The RAN 2106 can include one or more base stations (such as base station 2112 and base station 2114) that enable the connection 2108 and connection 2110.

[0218] In this example, the connection 2108 and connection 2110 are air interfaces to enable such communicative coupling, and may be consistent with RAT(s) used by the RAN 2106, such as, for example, an LTE and / or NR.

[0219] In some embodiments, the UE 2102 and UE 2104 may also directly exchange communication data via a sidelink interface 2116. The UE 2104 is shown to be configured to access an access point (shown as AP 2118) via connection 2120. By way of example, the connection 2120 can comprise a local wireless connection, such as a connection consistent with any IEEE 802.11 protocol, wherein the AP 2118 may comprise a Wi-Fi® router. In this example, the AP 2118 may be connected to another network (for example, the Internet) without going through a CN 2124.

[0220] In embodiments, the UE 2102 and UE 2104 can be configured to communicate using orthogonal frequency division multiplexing (OFDM) communication signals with each other or with the base station 2112 and / or the base station 2114 over a multicarrier communication channel in accordance with various communication techniques, such as, but not limited to, an orthogonal frequency division multiple access (OFDMA) communication technique (e.g., for downlink communications) or a single carrier frequency division multiple access (SC-FDMA) communication technique (e.g., for uplink and ProSe or sidelink communications), although the scope of the embodiments is not limited in this respect. The OFDM signals can comprise a plurality of orthogonal subcarriers.

[0221] In some embodiments, all or parts of the base station 2112 or base station 2114 may be implemented as one or more software entities running on server computers as part of a virtual network. In addition, or in other embodiments, the base station 2112 or base station 2114 may be configured to communicate with one another via interface 2122. In embodiments where the wireless communication system 2100 is an LTE system (e.g., when the CN 2124 is an EPC), the interface 2122 may be an X2 interface. The X2 interface may be defined between two or more base stations (e.g., two or more eNBs and the like) that connect to an EPC, and / or between two eNBs connecting to the EPC. In embodiments where the wireless communication system 2100 is an NR system (e.g., when CN 2124 is a 5GC), the interface 2122 may be an Xn interface. The Xn interface is33P68765WO1 4897-2098-5188\1defined between two or more base stations (e.g., two or more gNBs and the like) that connect to 5GC, between a base station 2112 (e.g., a gNB) connecting to 5GC and an eNB, and / or between two eNBs connecting to 5GC (e.g., CN 2124).

[0222] The RAN 2106 is shown to be communicatively coupled to the CN 2124. The CN 2124 may comprise one or more network elements 2126, which are configured to offer various data and telecommunications services to customers / subscribers (e.g., users of UE 2102 and UE 2104) who are connected to the CN 2124 via the RAN 2106. The components of the CN 2124 may be implemented in one physical device or separate physical devices including components to read and execute instructions from a machine- readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium).

[0223] In embodiments, the CN 2124 may be an EPC, and the RAN 2106 may be connected with the CN 2124 via an SI interface 2128. In embodiments, the SI interface 2128 may be split into two parts, an SI user plane (Sl-U) interface, which carries traffic data between the base station 2112 or base station 2114 and a serving gateway (S-GW), and the SI -MME interface, which is a signaling interface between the base station 2112 or base station 2114 and mobility management entities (MMEs).

[0224] In embodiments, the CN 2124 may be a 5GC, and the RAN 2106 may be connected with the CN 2124 via an NG interface 2128. In embodiments, the NG interface 2128 may be split into two parts, an NG user plane (NG-U) interface, which carries traffic data between the base station 2112 or base station 2114 and a user plane function (UPF), and the SI control plane (NG-C) interface, which is a signaling interface between the base station 2112 or base station 2114 and access and mobility management functions (AMFs).

[0225] Generally, an application server 2130 may be an element offering applications that use internet protocol (IP) bearer resources with the CN 2124 (e.g., packet switched data services). The application server 2130 can also be configured to support one or more communication services (e.g., VoIP sessions, group communication sessions, etc.) for the UE 2102 and UE 2104 via the CN 2124. The application server 2130 may communicate with the CN 2124 through an IP communications interface 2132.

[0226] An AI / ML model group management server as discussed herein may be one example of an application server 2130. In other words, as discussed herein, an application server 2130 may be configured to carry out the behaviors discussed in34P68765WO1 4897-2098-5188\1relation to FIG. 18. For example, an application server 2130 may receive, from an AI / ML model network function of a network of a wireless communication system, a request to approve a first AI / ML model for use, the request comprising the first AI / ML model, perform first group testing of a group of AI / ML models comprising the first AI / ML model and one or more additional AI / ML models, and send, to the AI / ML model network function, based on a first result of the first group testing, a response to the request that indicates that the first AI / ML model is approved for use, as discussed herein.

[0227] FIG. 22 illustrates a system 2200 for performing signaling 2234 between a wireless device 2202 and a network device 2218, according to embodiments disclosed herein. The system 2200 may be a portion of a wireless communications system as herein described. The wireless device 2202 may be, for example, a UE of a wireless communication system. The network device 2218 may be, for example, a base station (e.g., an eNB or a gNB) of a wireless communication system.

[0228] The wireless device 2202 may include one or more processor(s) 2204. The processor(s) 2204 may execute instructions such that various operations of the wireless device 2202 are performed, as described herein. The processor(s) 2204 may include one or more baseband processors implemented using, for example, a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein.

[0229] The wireless device 2202 may include a memory 2206. The memory 2206 may be a non-transitory computer-readable storage medium that stores instructions 2208 (which may include, for example, the instructions being executed by the processor(s) 2204). The instructions 2208 may also be referred to as program code or a computer program. The memory 2206 may also store data used by, and results computed by, the processor(s) 2204.

[0230] The wireless device 2202 may include one or more transceiver(s) 2210 that may include radio frequency (RF) transmitter circuitry and / or receiver circuitry that use the antenna(s) 2212 of the wireless device 2202 to facilitate signaling (e.g., the signaling 2234) to and / or from the wireless device 2202 with other devices (e.g., the network device 2218) according to corresponding RATs.35P68765WO1 4897-2098-5188\1

[0231] The wireless device 2202 may include one or more antenna(s) 2212 (e.g., one, two, four, or more). For embodiments with multiple antenna(s) 2212, the wireless device 2202 may leverage the spatial diversity of such multiple antenna(s) 2212 to send and / or receive multiple different data streams on the same time and frequency resources. This behavior may be referred to as, for example, multiple input multiple output (MIMO) behavior (referring to the multiple antennas used at each of a transmitting device and a receiving device that enable this aspect). MIMO transmissions by the wireless device 2202 may be accomplished according to precoding (or digital beamforming) that is applied at the wireless device 2202 that multiplexes the data streams across the antenna(s) 2212 according to known or assumed channel characteristics such that each data stream is received with an appropriate signal strength relative to other streams and at a desired location in the spatial domain (e.g., the location of a receiver associated with that data stream). Certain embodiments may use single user MIMO (SU-MIMO) methods (where the data streams are all directed to a single receiver) and / or multi user MIMO (MU-MIMO) methods (where individual data streams may be directed to individual (different) receivers in different locations in the spatial domain).

[0232] In certain embodiments having multiple antennas, the wireless device 2202 may implement analog beamforming techniques, whereby phases of the signals sent by the antenna(s) 2212 are relatively adjusted such that the (joint) transmission of the antenna(s) 2212 can be directed (this is sometimes referred to as beam steering).

[0233] The wireless device 2202 may include one or more interface(s) 2214. The interface(s) 2214 may be used to provide input to or output from the wireless device 2202. For example, a wireless device 2202 that is a UE may include interface(s) 2214 such as microphones, speakers, a touchscreen, buttons, and the like in order to allow for input and / or output to the UE by a user of the UE. Other interfaces of such a UE may be made up of transmitters, receivers, and other circuitry (e.g., other than the transceiver(s) 2210 / antenna(s) 2212 already described) that allow for communication between the UE and other devices and may operate according to known protocols (e.g., Wi-Fi®, Bluetooth®, and the like).

[0234] The wireless device 2202 may include an AI / ML model group management module 2216. The AI / ML model group management module 2216 may be implemented via hardware, software, or combinations thereof. For example, the AI / ML model group management module 2216 may be implemented as a processor, circuit, and / or36P68765WO1 4897-2098-5188\1instructions 2208 stored in the memory 2206 and executed by the processor(s) 2204. In some examples, the AI / ML model group management module 2216 may be integrated within the processor(s) 2204 and / or the transceiver(s) 2210. For example, the AI / ML model group management module 2216 may be implemented by a combination of software components (e.g., executed by a DSP or a general processor) and hardware components (e.g., logic gates and circuitry) within the processor(s) 2204 or the transceiver(s) 2210.

[0235] The AI / ML model group management module 2216 may be used for various aspects of the present disclosure, for example, aspects of FIG. 17 and / or FIG. 20. The AI / ML model group management module 2216 may configure the wireless device 2202 to, for example, receive, from an AI / ML model group management server, an indication that a first AI / ML model is approved for use and to perform, based on the indication that the first AI / ML model is approved for use, an inference using a group of AI / ML models that includes the first AI / ML model, as discussed herein. The AI / ML model group management module 2216 may configure the wireless device 2202 to, for example, receive, from an AI / ML model network function of the network, a request to approve a first AI / ML model for use, the request comprising the first AI / ML model; perform first group testing of a group of AI / ML models comprising the first AI / ML model and one or more additional AI / ML models; and send, to the AI / ML model network function, based on a first result of the first group testing, a response to the request that indicates that the first AI / ML model is approved for use.

[0236] The network device 2218 may include one or more processor(s) 2220. The processor(s) 2220 may execute instructions such that various operations of the network device 2218 are performed, as described herein. The processor(s) 2220 may include one or more baseband processors implemented using, for example, a CPU, a DSP, an ASIC, a controller, an FPGA device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein.

[0237] The network device 2218 may include a memory 2222. The memory 2222 may be a non-transitory computer-readable storage medium that stores instructions 2224 (which may include, for example, the instructions being executed by the processor(s) 2220). The instructions 2224 may also be referred to as program code or a computer program. The memory 2222 may also store data used by, and results computed by, the processor(s) 2220.37P68765WO1 4897-2098-5188\1

[0238] The network device 2218 may include one or more transceiver(s) 2226 that may include RF transmitter circuitry and / or receiver circuitry that use the antenna(s) 2228 of the network device 2218 to facilitate signaling (e.g., the signaling 2234) to and / or from the network device 2218 with other devices (e.g., the wireless device 2202) according to corresponding RATs.

[0239] The network device 2218 may include one or more antenna(s) 2228 (e.g., one, two, four, or more). In embodiments having multiple antenna(s) 2228, the network device 2218 may perform MIMO, digital beamforming, analog beamforming, beam steering, etc., as has been described.

[0240] The network device 2218 may include one or more interface(s) 2230. The interface(s) 2230 may be used to provide input to or output from the network device 2218. For example, a network device 2218 that is a base station may include interface(s) 2230 made up of transmitters, receivers, and other circuitry (e.g., other than the transceiver(s) 2226 / antenna(s) 2228 already described) that enables the base station to communicate with other equipment in a core network, and / or that enables the base station to communicate with external networks, computers, databases, and the like for purposes of operations, administration, and maintenance of the base station or other equipment operably connected thereto.

[0241] The network device 2218 may include an AI / ML model group management module 2232. The AI / ML model group management module 2232 may be implemented via hardware, software, or combinations thereof. For example, the AI / ML model group management module 2232 may be implemented as a processor, circuit, and / or instructions 2224 stored in the memory 2222 and executed by the processor(s) 2220. In some examples, the AI / ML model group management module 2232 may be integrated within the processor(s) 2220 and / or the transceiver(s) 2226. For example, the AI / ML model group management module 2232 may be implemented by a combination of software components (e.g., executed by a DSP or a general processor) and hardware components (e.g., logic gates and circuitry) within the processor(s) 2220 or the transceiver(s) 2226.

[0242] The AI / ML model group management module 2232 may be used for various aspects of the present disclosure, for example, aspects of FIG. 19. The AI / ML model group management module 2232 may configure the network device 2218 to, for example, send, to an AI / ML model group management server, a request to approve an38P68765WO1 4897-2098-5188\1AI / ML model for use, the request comprising the AI / ML model and receive, from the AI / ML model group management server, a response to the request that indicates that the AI / ML model is approved for use, as discussed herein.

[0243] Embodiments contemplated herein include an apparatus comprising means to perform one or more elements of any of the method 1700 and the method 2000. This apparatus may be, for example, an apparatus of a UE (such as a wireless device 2202 that is a UE, as described herein).

[0244] Embodiments contemplated herein include one or more non-transitory computer-readable media comprising instructions to cause an electronic device, upon execution of the instructions by one or more processors of the electronic device, to perform one or more elements of any of the method 1700 and the method 2000. This non-transitory computer-readable media may be, for example, a memory of a UE (such as a memory 2206 of a wireless device 2202 that is a UE, as described herein).

[0245] Embodiments contemplated herein include an apparatus comprising logic, modules, or circuitry to perform one or more elements of any of the method 1700 and the method 2000. This apparatus may be, for example, an apparatus of a UE (such as a wireless device 2202 that is a UE, as described herein).

[0246] Embodiments contemplated herein include an apparatus comprising: one or more processors and one or more computer-readable media comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform one or more elements of any of the method 1700 and the method 2000. This apparatus may be, for example, an apparatus of a UE (such as a wireless device 2202 that is a UE, as described herein).

[0247] Embodiments contemplated herein include a signal as described in or related to one or more elements of any of the method 1700 and the method 2000.

[0248] Embodiments contemplated herein include a computer program or computer program product comprising instructions, wherein execution of the program by a processor is to cause the processor to carry out one or more elements of any of the method 1700 and the method 2000. The processor may be a processor of a UE (such as a processor(s) 2204 of a wireless device 2202 that is a UE, as described herein). These instructions may be, for example, located in the processor and / or on a memory of the UE (such as a memory 2206 of a wireless device 2202 that is a UE, as described herein).39P68765WO1 4897-2098-5188\1

[0249] Embodiments contemplated herein include an apparatus comprising means to perform one or more elements of the method 1800. This apparatus may be, for example, an apparatus of a base station (such as a network device 2218 that is a base station, as described herein). This apparatus may be, for example, an apparatus of a CN (such as a network device 2218 that is located in the CN, as described herein).

[0250] Embodiments contemplated herein include one or more non-transitory computer-readable media comprising instructions to cause an electronic device, upon execution of the instructions by one or more processors of the electronic device, to perform one or more elements of the method 1800. This non-transitory computer- readable media may be, for example, a memory of a base station (such as a memory 2222 of a network device 2218 that is a base station, as described herein). This non-transitory computer-readable media may be, for example, a memory of a CN (such as a memory 2222 of a network device 2218 that is located in the CN as described herein).

[0251] Embodiments contemplated herein include an apparatus comprising logic, modules, or circuitry to perform one or more elements of the method 1800. This apparatus may be, for example, an apparatus of a base station (such as a network device 2218 that is a base station, as described herein). This apparatus may be, for example, an apparatus of a CN (such as a network device 2218 that is located in the CN, as described herein).

[0252] Embodiments contemplated herein include an apparatus comprising: one or more processors and one or more computer-readable media comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform one or more elements of the method 1800. This apparatus may be, for example, an apparatus of a base station (such as a network device 2218 that is a base station, as described herein). This apparatus may be, for example, an apparatus of a CN (such as a network device 2218 that is located in a CN, as described herein).

[0253] Embodiments contemplated herein include a signal as described in or related to one or more elements of the method 1800.

[0254] Embodiments contemplated herein include a computer program or computer program product comprising instructions, wherein execution of the program by a processing element is to cause the processing element to carry out one or more elements of the method 1800. The processor may be a processor of a base station (such as a processor(s) 2220 of a network device 2218 that is a base station, as described herein).40P68765WO1 4897-2098-5188\1These instructions may be, for example, located in the processor and / or on a memory of the base station (such as a memory 2222 of a network device 2218 that is a base station, as described herein). The processor may be a processor of a CN (such as a processor(s) 2220 of a network device 2218 that is located in a CN, as described herein). These instructions may be, for example, located in the processor and / or on a memory of the CN (such as a memory 2222 of a network device 2218 that is located in the CN, as described herein).

[0255] For one or more embodiments, at least one of the components set forth in one or more of the preceding figures may be configured to perform one or more operations, techniques, processes, and / or methods as set forth herein. For example, a baseband processor as described herein in connection with one or more of the preceding figures may be configured to operate in accordance with one or more of the examples set forth herein. For another example, circuitry associated with a UE, base station, network element, etc. as described above in connection with one or more of the preceding figures may be configured to operate in accordance with one or more of the examples set forth herein.

[0256] Any of the above described embodiments may be combined with any other embodiment (or combination of embodiments), unless explicitly stated otherwise. The foregoing description of one or more implementations provides illustration and description, but is not intended to be exhaustive or to limit the scope of embodiments to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of various embodiments.

[0257] Embodiments and implementations of the systems and methods described herein may include various operations, which may be embodied in machine-executable instructions to be executed by a computer system. A computer system may include one or more general-purpose or special-purpose computers (or other electronic devices). The computer system may include hardware components that include specific logic for performing the operations or may include a combination of hardware, software, and / or firmware.

[0258] It should be recognized that the systems described herein include descriptions of specific embodiments. These embodiments can be combined into single systems, partially combined into other systems, split into multiple systems or divided or combined in other ways. In addition, it is contemplated that parameters, attributes, aspects, etc. of41P68765WO1 4897-2098-5188\1one embodiment can be used in another embodiment. The parameters, attributes, aspects, etc. are merely described in one or more embodiments for clarity, and it is recognized that the parameters, attributes, aspects, etc. can be combined with or substituted for parameters, attributes, aspects, etc. of another embodiment unless specifically disclaimed herein.

[0259] It is well understood that the use of personally identifiable information should follow privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy of users. In particular, personally identifiable information data should be managed and handled so as to minimize risks of unintentional or unauthorized access or use, and the nature of authorized use should be clearly indicated to users.

[0260] Although the foregoing has been described in some detail for purposes of clarity, it will be apparent that certain changes and modifications may be made without departing from the principles thereof. It should be noted that there are many alternative ways of implementing both the processes and apparatuses described herein. Accordingly, the present embodiments are to be considered illustrative and not restrictive, and the description is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.42P68765WO1 4897-2098-5188\1

Claims

CLAIMS1. A method of a user equipment (UE) operating with a network of a wireless communication system, comprising: receiving, from an artificial intelligence (AI) / machine learning (ML) model group management server, an indication that a first AI / ML model is approved for use; and performing, based on the indication that the first AI / ML model is approved for use, an inference using a group of AI / ML models that includes the first AI / ML model.

2. The method of claim 1, further comprising receiving, from an AI / ML model network function of the network, the first AI / ML model.

3. The method of claim 2, further comprising identifying, to the AI / ML model network function, that the first AI / ML model is useable at the UE as part of a registration procedure between the network and the UE, wherein the first AI / ML model is received from the AI / ML model network function after the registration procedure.

4. The method of claim 2, further comprising identifying, to the AI / ML model network function, that the first AI / ML model is useable at the UE as part of an identification procedure between the network and the UE, wherein the first AI / ML model is received from the AI / ML model network function after the identification procedure.

5. The method of claim 1, further comprising receiving, from the AI / ML model group management server, a listing of approved AI / ML models.

6. The method of claim 5, further comprising: receiving, from an AI / ML model network function of the network, a second AI / ML model; determining that the second AI / ML model is not included in the listing of approved AI / ML models; and rejecting the second AI / ML model based on determining that the second AI / ML model is not included in the listing of approved AI / ML models.

7. The method of claim 1, further comprising sending, to an AI / ML model network function of the network, a listing of approved AI / ML models.43P68765WO1 4897-2098-5188\18. The method of claim 1, further comprising receiving, from the AI / ML model group management server, the first AI / ML model.

9. The method of claim 1, further comprising: receiving, from the AI / ML model group management server, an instruction to carry out a corrective action with respect to the first AI / ML model; and performing, in response to the instruction, the corrective action.

10. The method of claim 9, wherein the corrective action comprises reverting from using a current version of the first AI / ML model to using a prior version of the first AI / ML model.

11. The method of claim 9, wherein the corrective action comprises reverting from using the first AI / ML model to using a corresponding non-AI / ML-model-based functionality.

12. The method of claim 9, wherein the corrective action comprises removing the first AI / ML model from a listing of approved AI / ML models, and further comprising sending, to an AI / ML model network function of the network, the listing of approved AI / ML models after removing the first AI / ML model from the listing of approved AI / ML models.

13. The method of claim 1, further comprising: identifying that a condition for using the first AI / ML model is not met; and ending use by the UE of the first AI / ML model in response to identifying that the condition for using the first AI / ML model is not met.

14. The method of claim 13, wherein the condition is received from the AI / ML model group management server.

15. The method of claim 13, wherein the condition is received from an AI / ML model network function.

16. The method of claim 13, wherein the condition comprises a time limit condition.

17. The method of claim 13, wherein the condition comprises a spatial limit condition.

18. The method of claim 13, wherein the condition comprises a power level condition.44P68765WO1 4897-2098-5188\119. The method of claim 13, wherein the condition comprises a geographic location condition.

20. A method of an artificial intelligence (AI) / machine learning (ML) model group management server, comprising: receiving, from an AI / ML model network function of a network of a wireless communication system, a request to approve a first AI / ML model for use, the request comprising the first AI / ML model; performing first group testing of a group of AI / ML models comprising the first AI / ML model and one or more additional AI / ML models; and sending, to the AI / ML model network function, based on a first result of the first group testing, a response to the request that indicates that the first AI / ML model is approved for use.

21. The method of claim 20, wherein performing the first group testing of the group of AI / ML models comprises: performing a first regression test that does not use the first AI / ML model; performing a second regression test that uses the group of AI / ML models comprising the first AI / ML model and the one or more additional AI / ML models; and generating the first result of the first group testing based on a comparison of the first regression test to the second regression test.

22. The method of claim 20, wherein the request indicates a user equipment (UE) at which the first AI / ML model is intended to be used.

23. The method of claim 22, wherein the response indicates that the approval of the first AI / ML model is with respect to the UE.

24. The method of claim 20, wherein the response further comprises a condition for using the first AI / ML model.

25. The method of claim 24, wherein the condition comprises a time limit condition.

26. The method of claim 24, wherein the condition comprises a spatial limit condition.

27. The method of claim 24, wherein the condition comprises a power level condition.45P68765WO1 4897-2098-5188\128. The method of claim 24, wherein the condition comprises a UE type condition.

29. The method of claim 24, wherein the condition comprises a geographic location condition.

30. The method of claim 20, further comprising sending, to a user equipment (UE) of the wireless communication system, an indication that the first AI / ML model is approved for use.

31. The method of claim 20, further comprising sending, to a user equipment (UE) of the wireless communication system, the first AI / ML model.

32. The method of claim 20, further comprising: identifying a group monitoring trigger corresponding to the first AI / ML model; performing, in response to the group monitoring trigger, second group testing of the group of AI / ML models; identifying, based on a second result of the second group testing, that the first AI / ML model has degraded; determining a corrective action for the first AI / ML model in response to identifying that the first AI / ML model has degraded; and instructing one of the AI / ML model network function and a user equipment (UE) of the wireless communication system that is using the first AI / ML model to carry out the corrective action.

33. The method of claim 32, wherein the AI / ML model group management server instructs the AI / ML model network function to carry out the corrective action, and wherein the corrective action comprises performing additional training and requesting a new approval for the first AI / ML model.

34. The method of claim 32, wherein the AI / ML model group management server instructs the AI / ML model network function to carry out the corrective action, and wherein the corrective action comprises sending the UE a replacement AI / ML model for the first AI / ML model.

35. The method of claim 32, wherein the AI / ML model group management server instructs the UE to carry out the corrective action, and wherein the corrective action46P68765WO1 4897-2098-5188\1comprises reverting from using a current version of the first AI / ML model to using a prior version of the first AI / ML model.

36. The method of claim 32, wherein the AI / ML model group management server instructs the UE to carry out the corrective action, and wherein the corrective action comprises removing the first AI / ML model from a listing of approved AI / ML models.

37. The method of claim 20, further comprising: identifying a that the first AI / ML model has a dependency on a second AI / ML model; including the second AI / ML model in the group of AI / ML models during the first group testing; wherein the response to the request further indicates that the second AI / ML model is to be used with the first AI / ML model.

38. A method of an artificial intelligence (AI) / machine learning (ML) model network function of a network of a wireless communication system, comprising: sending, to an AI / ML model group management server, a request to approve an AI / ML model for use, the request comprising the AI / ML model; and receiving, from the AI / ML model group management server, a response to the request that indicates that the AI / ML model is approved for use.

39. The method of claim 38, wherein the request indicates a user equipment (UE) of the wireless communication system at which the AI / ML model is intended to be used.

40. The method of claim 39, wherein the response indicates that the approval of the AI / ML model is with respect to the UE.

41. The method of claim 38, wherein the response further comprises a condition for using the AI / ML model within the wireless communication system.

42. The method of claim 41, wherein the condition comprises a time limit condition.

43. The method of claim 41, wherein the condition comprises a spatial limit condition.

44. The method of claim 41, wherein the condition comprises a power level condition.

45. The method of claim 41, wherein the condition comprises a UE type condition.47P68765WO1 4897-2098-5188\146. The method of claim 41, wherein the condition comprises a geographic location condition.

47. The method of claim 38, further comprising sending, to a user equipment (UE) of the wireless communication system, the AI / ML model.

48. The method of claim 47, further comprising receiving, from the UE, a listing of approved AI / ML models.

49. The method of claim 47, further comprising identifying the AI / ML model from a listing of approved AI / ML models received from the UE as part of a registration procedure between the network and the UE, wherein the AI / ML model is sent to the UE after the registration procedure.

50. The method of claim 47, further comprising identifying the AI / ML model from a listing of approved AI / ML models received from the UE as part of an identification procedure between the network and the UE, wherein the AI / ML model is sent to the UE after the identification procedure.

51. The method of claim 38, further comprising training the AI / ML model.

52. The method of claim 38, further comprising testing the AI / ML model.

53. The method of claim 38, further comprising: receiving, from the AI / ML model group management server, an instruction to carry out a corrective action with respect to the AI / ML model; and performing, in response to the instruction, the corrective action.

54. The method of claim 53, wherein the corrective action comprises sending a user equipment (UE) of the wireless communication system a replacement AI / ML model for the AI / ML model.

55. The method of claim 53, wherein the corrective action comprises performing additional training and requesting a new approval for the AI / ML model, and further comprising sending the AI / ML model to a user equipment (UE) of the wireless communication system after performing the additional training and receiving the new approval.48P68765WO1 4897-2098-5188\156. A method of user equipment (UE) operating with a network of a wireless communication system, comprising: receiving, from an artificial intelligence (AI) / machine learning (ML) model network function of the network, a request to approve a first AI / ML model for use, the request comprising the first AI / ML model; performing first group testing of a group of AI / ML models comprising the first AI / ML model and one or more additional AI / ML models; and sending, to the AI / ML model network function, based on a first result of the first group testing, a response to the request that indicates that the first AI / ML model is approved for use.

57. The method of claim 56, further comprising sending the first AI / ML model to an AI / ML model group management server.

58. The method of claim 56, wherein performing the first group testing of the group of AI / ML models comprises: performing a first regression test that does not use the first AI / ML model; performing a second regression test that uses the group of AI / ML models comprising the first AI / ML model and the one or more additional AI / ML models; and generating the first result of the first group testing based on a comparison of the first regression test to the second regression test.

59. The method of claim 56, further comprising: performing second group testing of the group of AI / ML models; identifying, based on a second result of the second group testing, that the first AI / ML model has degraded; and sending, to the AI / ML model network function, an indication that the first AI / ML model is invalid.

60. An apparatus comprising means to perform the method of any of claim 1 to claim 59.

61. A computer-readable media comprising instructions to cause an electronic device, upon execution of the instructions by one or more processors of the electronic device, to perform the method of any of claim 1 to claim 59.49P68765WO1 4897-2098-5188\162. An apparatus comprising logic, modules, or circuitry to perform the method of any of claim 1 to claim 59.

63. A baseband processor for a user equipment (UE) that is configured to cause the UE to perform one or more elements of any one of claim 1 to claim 19 and claim 56 to claim 59.

64. A baseband processor for a base station that is configured to cause the base station to perform one or more elements of any one of claim 38 to claim 55.50P68765WO1 4897-2098-5188\1

Images