Apparatus, method and computer program

Abstract

An apparatus is provided including means for performing receiving, from an entity on which a machine learning model is configured, performance information and model delta information related to the machine learning model, determining, based at least on the received performance information and model delta information, an updated model delta threshold, and sharing, with the entity, the updated model delta threshold, wherein the model delta threshold is suitable for determining, by the entity on which the machine learning model is configured, whether to share further information related to the machine learning model, and providing, to the entity on which the machine learning model is configured, an indication of the updated model delta threshold.

Description

Technical Field

[0001] This application relates to a method, apparatus, system, and computer program, and specifically, but not exclusively, to user equipment (UE) machine learning (ML) updates based on network-side model incremental threshold configuration. Background Technology

[0002] A communication system can be viewed as a facility that enables communication sessions between two or more communication devices or provides these devices with access to a network. Mobile or wireless communication networks are an example of communication networks. Services can be provided to the communication devices by an application server.

[0003] Such communication networks operate according to standards provided by, for example, 3GPP (3rd Generation Partnership Project) or ETSI (European Telecommunications Standards Institute). Examples of these standards are the so-called 4G (fourth generation) and 5G (fifth generation) standards provided by 3GPP. Summary of the Invention

[0004] In a first aspect, an apparatus is provided, comprising components for performing: receiving performance information and model increment information related to the machine learning model from an entity on which a machine learning model is configured; determining an updated model increment threshold based at least on the received performance information and model increment information; and sharing the updated model increment threshold with the entity, wherein the model increment threshold is adapted to: determine by the entity on which the machine learning model is configured whether to share additional information related to the machine learning model; and provide an indication of the updated model increment threshold to the entity on which the machine learning model is configured.

[0005] If the performance information indicates a value higher than the first threshold, the updated model increment threshold can be equal to the model increment value indicated by the model increment information; and if the performance information indicates a value lower than the first threshold, the updated model increment threshold can be equal to the minimum of the sum of the model increment threshold and the offset and the model increment value indicated by the model increment information.

[0006] The device may include components for performing the following: receiving additional information related to the machine learning model.

[0007] Additional information related to the machine learning model may include at least one of the following: model increment information related to the updated machine learning model, performance information related to the updated machine learning model, or the updated machine learning model.

[0008] The apparatus may include components for performing the following: determining to update the machine learning model based on additional information, and at least providing instructions to an entity on which the machine learning model is configured to update the machine learning model.

[0009] Additional information may include the updated machine learning model and a comparison component for performing the following: determining to provide the updated machine learning model to at least one other entity on which the machine learning model is configured.

[0010] The apparatus may include components for receiving performance information and model increment information related to the machine learning model from a plurality of entities on which a machine learning model is configured, and performing: determining a confidence value for the correspondence between the received model increment information and the performance.

[0011] The apparatus may include components for providing an entity on which a machine learning model is configured to configure incremental parameters of the machine learning model.

[0012] Entities on which machine learning models are configured can include user devices or network functions.

[0013] In a second aspect, an apparatus is provided, comprising components for: determining to update a machine learning model configured at the apparatus; updating the machine learning model; determining model increment information based on the machine learning model and the updated machine learning model; determining to provide information related to the machine learning model to a network entity based on a comparison between the model increment information and a model increment threshold; and providing information related to the machine learning model to the network entity.

[0014] The apparatus may include components for: further determining performance information based on the machine learning model and the updated machine learning model, providing model increment information and performance information to the network entity, and receiving, in response, an updated model increment threshold from the network entity.

[0015] Information related to a machine learning model may include at least one of the following: model increment information, performance information related to the machine learning model, or an updated machine learning model.

[0016] Model incremental information can be determined at least based on the difference between the weights of the machine learning model and the weights of the updated machine learning model.

[0017] Model increment information can indicate at least one of the global machine learning increment value and the vector machine learning increment value.

[0018] In a third aspect, a method is provided, comprising performing: receiving performance information and model increment information related to the machine learning model from an entity on which a machine learning model is configured; determining an updated model increment threshold based at least on the received performance information and model increment information; and sharing the updated model increment threshold with the entity, wherein the model increment threshold is suitable for: determining by the entity on which the machine learning model is configured whether to share additional information related to the machine learning model; and providing an indication of the updated model increment threshold to the entity on which the machine learning model is configured.

[0019] If the performance information indicates a value higher than the first threshold, the updated model increment threshold can be equal to the model increment value indicated by the model increment information; and if the performance information indicates a value lower than the first threshold, the updated model increment threshold can be equal to the minimum of the sum of the model increment threshold and the offset and the model increment value indicated by the model increment information.

[0020] The method may include performing: receiving additional information related to the machine learning model.

[0021] Additional information related to the machine learning model may include at least one of the following: model increment information related to the updated machine learning model, performance information related to the updated machine learning model, or the updated machine learning model.

[0022] The method may include performing: determining, based on additional information, to update the machine learning model, and at least providing instructions to the entity on which the machine learning model is configured to update the machine learning model.

[0023] Additional information may include the updated machine learning model and a comparison component for performing the following: determining to provide the updated machine learning model to at least one other entity on which the machine learning model is configured.

[0024] The method may include receiving performance information and model increment information related to the machine learning model from multiple entities on which machine learning models are configured, and performing: determining a confidence value for the correspondence between the received model increment information and the performance.

[0025] The method may include providing an entity on which a machine learning model is configured to configure incremental parameters of the machine learning model.

[0026] Entities on which machine learning models are configured can include user devices or network functions.

[0027] In a fourth aspect, a method is provided, comprising determining to update a machine learning model configured at a device, updating the machine learning model, determining model increment information based on the machine learning model and the updated machine learning model, determining to provide information related to the machine learning model to a network entity based on a comparison between the model increment information and a model increment threshold, and providing information related to the machine learning model to the network entity.

[0028] The method may include further determining performance information based on the machine learning model and the updated machine learning model, providing model increment information and performance information to the network entity, and receiving an updated model increment threshold from the network entity in response.

[0029] Information related to a machine learning model may include at least one of the following: model increment information, performance information related to the machine learning model, or an updated machine learning model.

[0030] Model incremental information can be determined at least based on the difference between the weights of the machine learning model and the weights of the updated machine learning model.

[0031] Model increment information can indicate at least one of the global machine learning increment value and the vector machine learning increment value.

[0032] In a fifth aspect, an apparatus is provided comprising at least one processor and at least one memory storing instructions which, when executed by the processor, cause the apparatus to perform at least the method according to a third or fourth aspect.

[0033] In a sixth aspect, a computer-readable medium including instructions that, when executed by a device, cause the device to perform at least the method according to the third or fourth aspect.

[0034] In a seventh aspect, a non-transitory computer-readable medium is provided, including program instructions for causing a device to execute at least the method according to the third or fourth aspect.

[0035] Many different embodiments have been described above. It should be understood that other embodiments can be provided by any combination of two or more of the above embodiments. Attached Figure Description

[0036] Embodiments will now be described by way of example only with reference to the accompanying drawings, in which: Figure 1 A schematic diagram of an example 5GS communication system is shown; Figure 2 A schematic diagram of an example mobile communication device is shown; Figure 3A schematic diagram of an example control device is shown; Figure 4 An example call flow for model download and LCM activities to enable ML features is shown; Figure 5 A flowchart of a method according to an example embodiment is shown; Figure 6 A flowchart of a method according to an example embodiment is shown; Figure 7 The call flow according to the example embodiment is shown; Figure 8 The flowchart shows the selection of data thresholds for the adapted model and the use of model increments; Figure 9 A flowchart illustrating the selection of thresholds for the adapted model data is shown. Detailed Implementation

[0037] Before explaining the examples in detail, refer to Figure 1 , Figure 2 and Figure 3 A brief explanation of some general principles of wireless communication systems and mobile communication devices is provided to help understand the underlying technology of the described examples.

[0038] Examples of suitable communication systems are the 5G or NR concepts. The network architecture in NR can resemble the network architecture of Advanced Long Term Evolution (LTE). Base stations in NR systems can be referred to as next-generation NodeBs (gNBs). Changes to the network architecture can depend on the need to support various radio technologies and more granular Quality of Service (QoS) support, as well as some on-demand requirements for, for example, QoS levels to support the user's Quality of Experience (QoE). Furthermore, network-aware services and applications, and service and application-aware networks, may bring about changes to the architecture. These involve Information Center Network (ICN) and User-Centric Content Delivery Network (UC-CDN) approaches. NR can use multiple-input multiple-output (MIMO) antennas, far more base stations or nodes than LTE (the so-called small cell concept), including macro sites cooperating with smaller stations, and may also employ various radio technologies to achieve better coverage and enhanced data rates.

[0039] Future networks can leverage Network Functions Virtualization (NFV), a network architecture concept that proposes virtualizing network node functions as "building blocks" or entities that can be operatively connected or linked together to provide services. Virtualized network functions (VNFs) can include one or more virtual machines running computer program code using standard or general-purpose type servers instead of custom hardware. Deployment can be based on cloud-native network functions (CNFs), where network functions comprise one or more container pods. Cloud computing or data storage can also be utilized. In radio communications, this might mean that node operations will be performed, at least partially, within a server, host, or node operatively coupled to a remote radio head. Node operations can also be distributed across multiple servers, nodes, or hosts. It should also be understood that the division of labor between core network operations and base station operations may differ from that in LTE, or may even be absent.

[0040] Figure 1 A schematic diagram of a 5G system (5GS) 100 is shown. The 5GS may include a user equipment (UE) 102 (which may also be referred to as a communication device or terminal), a 5G radio access network (5GRAN) 104, a 5G core network (5GCN) 106, one or more internal or external application functions (AF) 108, and one or more data networks (DN) 110.

[0041] The example 5G core network (CN) includes functional entities. 5GCN 106 may include one or more Access and Mobility Management Functions (AMF) 112, one or more Session Management Functions (SMF) 114, Authentication Server Function (AUSF) 116, Unified Data Management (UDM) 118, one or more User Plane Functions (UPF) 120, Unified Data Repository (UDR) 122, and / or Network Exposure Function (NEF) 124. The UPF is controlled by the SMF (Session Management Function) that receives policies from the PCF (Policy Control Function).

[0042] The CN can connect to the UE via a radio access network (RAN) or via a fixed access function (N3IWF). The 5G RAN may include one or more gNodeB (gNB) distributed unit (DU) functions connected to one or more gNodeB (gNB) centralized unit (CU) functions. The RAN may include one or more access nodes.

[0043] The User Plane Function (UPF), known as the PDU Session Anchor (PSA), is responsible for forwarding frames back and forth between the DN and the tunnel established through 5G to (multiple) UEs that exchange services with the DN.

[0044] Now refer to Figure 2A more detailed description of possible mobile communication devices, Figure 2 A schematic partial cross-sectional view of a communication device 200 is shown. Such a communication device is generally referred to as a user equipment (UE) or terminal. Suitable mobile communication devices can be provided by any device capable of transmitting and receiving radio signals. Non-limiting examples include mobile stations (MS) or mobile devices (such as mobile phones or so-called smartphones), computers equipped with wireless interface cards or other wireless interface facilities (e.g., USB dongles), personal digital assistants (PDAs) or tablet computers equipped with wireless communication capabilities, VoIP phones, portable computers, desktop computers, image capture terminal devices (such as digital cameras), gaming terminal devices, music storage and playback devices, in-vehicle wireless terminal devices, wireless endpoints, mobile stations, laptop embedded devices (LEEs), laptop mounted devices (LMEs), smart devices, wireless customer premises equipment (CPEs), or any combination thereof. Mobile communication devices can provide, for example, data communication for carrying communications such as voice, email, text messages, multimedia, etc. Therefore, a variety of services can be provided to the user via the user's communication device. Non-limiting examples of these services include two-way or multiplexed calling, data communications or multimedia services, or simply access to data communications network systems such as the Internet. Broadcast or multicast data may also be provided to users. Non-limiting examples of content include downloads, television and radio programs, videos, advertisements, various alarms, and other information.

[0045] Mobile devices typically include at least one data processing entity 201, at least one memory 202, and other possible components 203 for software and hardware assistance in performing tasks designed to be performed, including controlling access to an access system and communication with the access system and other communication devices. The data processing, storage, and other related components may be located on appropriate circuit boards and / or chipsets. This feature is indicated by reference numeral 204. Users can control the operation of the mobile device using a suitable user interface (e.g., keypad 205, voice commands, touchscreen or touchpad, combinations thereof). A display 208, speaker, and microphone may also be provided. Furthermore, mobile communication devices may include appropriate connectors (wired or wireless) for connecting to other devices and / or connecting external accessories (e.g., hands-free devices) to the mobile communication device.

[0046] Mobile device 200 can receive signals via air or radio interface 207 through appropriate means for receiving, and can transmit signals via appropriate means for transmitting radio signals. Figure 2In this diagram, the transceiver device is schematically designated by box 206. The transceiver device 206 may be provided, for example, by means of radio components and associated antenna arrangements. The antenna arrangements may be located inside or outside the mobile device.

[0047] Figure 3 An example of a control device 300 for a communication system is shown, which is coupled to and / or used to control a station accessing the system, such as a RAN node (e.g., a base station, eNB, or gNB), a relay node, or a core network node (such as an MME or Serving Gateway (S-GW) or Packet Data Network Gateway (P-GW)), or a core network function (such as an AMF / SMF), or a server or host. The method can be implemented in a single control device or across more than one control device. The control device can be integrated with or external to a node or module of the core network or RAN. In some embodiments, the base station includes a separate control device unit or module. In other embodiments, the control device can be another network element, such as a radio network controller or a spectrum controller. In some embodiments, each base station can have such a control device as well as control devices provided in the radio network controller. The control device 300 can be arranged to provide control over communications within the service area of ​​the system. The control device 300 includes at least one memory 301, at least one data processing unit 302, 303, and an input / output interface 304. The control device can be coupled to a receiver and transmitter of the base station via the interface. The receiver and / or transmitter can be implemented as a radio front end or a remote radio head.

[0048] Machine learning (ML) is a branch of artificial intelligence (AI) that uses algorithms that learn from data and make predictions from inputs without explicit programming.

[0049] Machine learning models can include neural networks. Neural networks consist of nodes, or "artificial neurons," connected by edges. The output of each neuron is calculated by a nonlinear function, called an activation function, which is fed to other connected neurons. Neurons and edges typically have weights that adjust as learning progresses. Neurons can be aggregated into layers. Different layers can perform different transformations on their inputs.

[0050] AI / ML-based solutions are anticipated to be applicable to several use cases in radio access networks. Initial use cases have been identified in the 3GPP standards for RAN1 and RAN3, including, for example, energy saving, CSI compression, CSI feedback enhancement, positioning accuracy enhancement, and beam management.

[0051] Enabling AI / ML solutions in entities within a communication network (e.g., gNB, UE, CN, OAM, LMF) may require transferring (or delivering) AI / ML models from one entity to another destination entity for inference / prediction using the model in the destination entity. The signaling and protocols used for transferring ML models are being discussed.

[0052] Some possible solutions may include gNB transmitting (multiple) AI / ML models to UE via CP (e.g., RRC signaling) or UP data, CN (other than LMF) transmitting (multiple) AI / ML models to UE via CP (e.g., NAS signaling) or UP data, LMF transmitting (multiple) AI / ML models to UE via CP (e.g., LPP signaling) or UP data, or server (e.g., OAM, OTT) transmitting (multiple) AI / ML models to UE (e.g., transparent to 3GPP).

[0053] The advantages and disadvantages of CP and UP-based solutions for model delivery need to be studied, but there is no consensus on incremental model configuration.

[0054] Figure 4 The call flow for model download and lifecycle management (LCM) activities for enabling ML features is shown. In step 1, the model is stored at the network. In step 2, the UE requests model A. In step 3, the network provides model A to the UE during download. In step 4, LCM operations related to model A are performed (e.g., inference, monitoring, retraining, etc.). Once the model is downloaded to the UE, the UE can receive additional information for other LCM operations for any ML-enabled features. This additional information may include, for example, configurations for data collection, model training, model usage, additional hardware / software (deployment) configurations for running the model, configurations for generating measurement reports, or configurations for feedback to the NW, etc.

[0055] After the model is deployed in the UE device, it remains to be clarified whether the model being used (or “active”) in the UE does not require any updates at all, or whether the model can be changed (model switching) or updated (model retraining) over time.

[0056] To ensure expected ML performance under all conditions and environmental changes, it may be necessary to update the ML model once it is deployed to accommodate these changes in environment / context. For example, the data used to train the UEML model may become outdated due to changes in the radio environment, and the initial active model may need to be updated / tuned.

[0057] After training / retraining / tuning / applying transfer learning, regardless of the specific platform (e.g., TensorFlow, PyTorch, MXNet), at least some of the following information about the model can be stored: model weights (for inference), model state, training configuration (e.g., optimizer, loss, metric, learnable parameters), checkpoints (for resuming training / retraining), or model architecture.

[0058] This information can be used to configure the ML model.

[0059] Typically, model updates refer to the re-estimation of parameters of a model or parts thereof, achieved through retraining or refinement. In some cases, model updates are performed by an entity different from the one implementing inference. Therefore, the updated model may need to be transmitted to the UE. Conversely, the model can be updated at the UE and may need to be transmitted to the NW (or the NW may need to be informed of the updated content). However, if the difference between the existing model and the updated model is not significant, transmission may be unnecessary and a waste of radio resources. Even when the model is not transmitted, it is valuable for the network to evaluate how significant the model update is relative to the achieved performance as part of the ML monitoring process.

[0060] According to 3GPP specifications, UE model updates may affect and / or change supported ML-enabled functions / features; that is, the network may take specific actions before, during, or after a UE ML model update. Considering UE mobility, for the same functions / ML-enabled features, UEs may need to adjust the same model in different environments (e.g., cell changes). Therefore, for certain use cases and scenarios where supported ML-enabled functions / features are affected in terms of required network configuration, reporting configuration, signaling, etc., it is important to define a network-controlled UE ML update mechanism and how to implement that mechanism.

[0061] There is a problem: how to monitor the ML model running on the UE side without explicit model transmission when the UE model is updated.

[0062] Figure 5 A flowchart of a method according to an example embodiment is shown. The method may be performed at / on / by a device, such as a network entity. An entity on which a machine learning model is configured may include a user equipment or a network function or node, such as, but not limited to, a gNB, CN, OAM, or LMF.

[0063] In 501, the method includes receiving performance information and model increment information related to the machine learning model from an entity on which a machine learning model is configured.

[0064] In 502, the method includes determining an updated model increment threshold based at least on received performance information and model increment information, and sharing the updated model increment threshold with an entity, wherein the model increment threshold is suitable for: determining by the entity on which a machine learning model is configured whether to share additional information related to the machine learning model. The method includes updating the model increment threshold based at least on received performance information and model increment information.

[0065] In 503, the method includes providing an indication of an updated model increment threshold to an entity on which a machine learning model is configured. Providing an indication of an updated model threshold to an entity can be considered as sharing the updated model threshold with the entity.

[0066] Figure 6 A flowchart of a method according to an example embodiment is shown. The method can be performed at / on / by a device, for example, as described in reference... Figure 2 The described UE or network function or node, such as, but not limited to, gNB, CN, OAM, or LMF.

[0067] In 601, the method includes determining an update to the machine learning model configured at the device.

[0068] In 602, the method includes updating the machine learning model.

[0069] In 603, the method includes determining model increment information based on the machine learning model and the updated machine learning model.

[0070] In 604, the method includes determining the information to be provided to network entities that is relevant to the machine learning model based on a comparison between model increment information and model increment threshold.

[0071] In 605, the method includes providing network entities with information related to machine learning models.

[0072] For reference Figure 5 and Figure 6 The described method provides a framework for monitoring ML models based on model increment information and its correspondence with the performance achievable by a given ML function. Consider two ML models; the model increment refers to the difference between the old and new models.

[0073] As mentioned above Figure 5 and Figure 6The described method can provide an estimate of the model increment (i.e., difference or gap) between models (e.g., between the initial model and the updated model). The UE can assess how significant this model increment is from a UE-side performance perspective. This model increment (which is an example of model increment information) can then be used on the network side to track the performance of the considered ML function and the extent to which it has been modified / updated, and to determine monitoring actions accordingly. Another benefit of such methods is facilitating recovery and fallback to older model versions based on model increment information.

[0074] Depending on the use case and ML model architecture under consideration, model increment information can translate into different impacts on model performance. For example, for one function, a 10% model increment value may involve models that are very close in terms of performance and accuracy, while for another function, a 10% model increment value may be higher, and the performance of the two ML models under consideration may be quite different.

[0075] Performance information can indicate achievable performance and may include accuracy information, such as an indication of accuracy or other performance-related information. Performance information may be associated with one or more performance criteria.

[0076] The method may involve estimating a model increment threshold that is customized for a given ML function and conforms to the supported model increment configuration.

[0077] The method can provide a framework (including procedures and corresponding signaling) for performing ML model monitoring based on model increment information and, in some embodiments, based on corresponding achievable performance. The method provides an estimate of a model increment threshold tailored to a given ML function and conforming to the supported model increment configuration: the model increment should translate this difference between two models through a Global ML Model Increment (GMLD) value or a Vector ML Model Increment (VMLD).

[0078] Model incremental information can be determined at least based on the difference between the weights of the machine learning model and the weights of the updated machine learning model.

[0079] Model increment information can indicate at least one of the global machine learning increment value and the vector machine learning increment value.

[0080] The GMLD value can be defined as a unique value, for example, the empirical CDF at the 95th percentile at the selected threshold, or it can be the mean squared error (MSE) between the total weights of the two models.

[0081] VMLD can be defined as a vector that provides a more detailed view of the model's incremental information. In practice, VMLD can transform differences (as calculated in the case of GMLD), but only per layer. In this case, layer indices may need to be indicated by values ​​(0, 1, 2) or names (Dense_1, Conv_1).

[0082] On the network side, there is an adaptive selection of the configured ML model size threshold. The ML model size threshold is customized for ML functions and conforms to the supported model increment configuration (supported by the UE and can be GMLD or VMLD). It transforms the correspondence between model increment information and achievable model performance.

[0083] On the network side, the aggregation of GMLD / VLMD values ​​reported by UEs from multiple selected UEs can be used to obtain the confidence level of the correspondence between model / functional performance and GMLD / VLMD information.

[0084] On the network side, model and / or functional performance monitoring can be provided using model increment information with corresponding achievable performance and relying on UE reports of model performance changes (e.g., accuracy). One advantage is that model updates on the UE side can be characterized and quantified taking into account the achieved functional performance and without requiring complete model information (via model delivery).

[0085] Signaling enhancements can be provided to collect the information needed for adaptive selection of model increment thresholds and to enable model monitoring using model increment information.

[0086] Figure 7 An example signaling diagram according to an example embodiment is shown. Figure 7 Different exchanges between the UE and the network are described to enable model-incremental monitoring on the network side with UE assistance. The UE is an example of an entity on which a machine learning model is configured. Other entities on which ML models can be configured include network entities such as gNB, CN, OAM, or LMF.

[0087] In step 1, the UE initiates the use of model increment by requesting its configuration from the network via a Machine Learning Incremental Configuration Message (MLDC).

[0088] In step 2, the network responds by indicating the selected MLDC policy. The MLDC policy is an example of a configuration used to configure the incremental parameters of a machine learning model. Step 2 is an example of providing the entity on which the machine learning model is configured with a configuration for configuring the incremental parameters of the machine learning model.

[0089] MLDC strategies can be provided via the control plane through mechanisms such as RRC, MAC CE, and DCI. The MLDC strategy can specify the configuration for the optimization phase, which aims to estimate the model increment threshold, and the configuration for the monitoring phase, which aims to monitor model performance using model increment information and the selected model increment threshold. Machine learning model increment parameters can include the type of model increment parameter (e.g., GMLD or VMLD).

[0090] In step 3, on the UE side, the model is updated. The model can be triggered, for example, by model performance degradation or the detection of environmental changes. This is an example of determining which machine learning model is configured at the device and updating the machine learning model.

[0091] Subsequently, in steps 4 through 9, the UE estimates the model increment (which can be a GMLD value or a VMLD value) based on the configuration set by the network. This is an example of determining model increment information based on a machine learning model and an updated machine learning model. The model increment, or model increment value δ, is an example of model increment information.

[0092] Steps 10 to 12 can define a first stage or optimization stage, in which the network selects a model increment threshold for the ML function under consideration.

[0093] In step 10, the UE sends model accuracy information (after the model update) and the corresponding model increment. The model accuracy information is an example of performance information based on the updated machine learning model, and the corresponding model increment is an example of model increment information based on the machine learning model and the updated machine learning model. Step 10 is an example of providing model increment information and performance information to network entities.

[0094] In step 11, the network selects a threshold, and in step 12, it indicates the threshold to the UE. Steps 11 and 12 are examples of determining an updated model increment threshold based at least on the received performance information and model increment information, sharing the updated model increment threshold with the entity, and providing an indication of the updated model increment threshold to the entity on which a machine learning model is configured.

[0095] Steps 13 through 15 can define a second phase or a monitoring phase. In step 13, the UE calculates the model increment each time the model is updated and compares it with a threshold indicated by the network (in step 12). This is an example of a comparison between the model increment information and the model increment threshold.

[0096] If the model increment is higher than the model increment threshold, the UE sends model information (in step 14). Model information is an example of information related to the machine learning model. Step 14 determines examples of machine learning model-related information to be provided to the network entity based on a comparison between the model increment information and the model increment threshold. Machine learning model-related information may include at least one of the following: model increment information, performance information related to the machine learning model, or an updated machine learning model.

[0097] In step 15, the network performs monitoring based on information related to the ML model received from at least one UE (at least one UE conforming to the model increment threshold condition), which is model increment information in this example. This has the advantages of filtering the ML model-related information transmitted to the network into relevant information (in terms of performance) and allowing the network to make more effective monitoring decisions thereafter.

[0098] In an example embodiment, in order to collect more samples with a mapping or correspondence between performance and model increment (e.g., between the accuracy gap acc_gap and the model increment value δ), when possible (e.g., under low traffic and sufficient battery level), the network may request one or more UEs to perform model updates (e.g., retraining / refinement) for the same ML-enabled features / features.

[0099] The first stage, or optimization stage, can be referred to as the adaptive selection of the model increment threshold.

[0100] Model increment information can indicate the increment (or gap) between two ML models. The two ML models can be two neural networks with similar architectures but different weight values. The second model can be an updated version of a model that has already been updated (refined / retrained) on the UE side. Model increment information can include the model increment value δ (e.g., GMLD or VLMD value).

[0101] Figure 8 An example process is described, highlighting the use of model increment values ​​and the use of thresholds that are updated dynamically for a given ML function (e.g., for use cases such as ML-based localization).

[0102] exist Figure 7 In step 1, the ML model is updated (i.e., retrained or fine-tuned) when predefined criteria (e.g., environmental changes are detected). This is an example of determining the update configured on the machine learning model at the device and updating the machine learning model; In step 2, the incremental value δ of the MLDC is calculated based on the data received from the network.

[0103] Given a pre-selected model increment threshold The estimated δ is then compared to this. If the estimated δ is large, the new model is considered different from the previous model, and its relevant information (e.g., model increment and corresponding performance gap) should subsequently be shared with the network (as shown in step 6). This step determines examples of information related to the machine learning model to be provided to network entities based on a comparison between the model increment information and the model increment threshold. Otherwise, the model information is not shared with the network.

[0104] Simultaneously, in step 3, the accuracy of the updated model is estimated, which provides a result of the model accuracy / performance gap. Indication of model performance and model increment value The mapping between them, where: ,in This refers to the accuracy of the updated model, and This refers to the accuracy of the initial model. Performance information related to the updated machine learning model can include... or .

[0105] In step 4, the threshold Updated. This step is performed by the network, taking into account previously reported accuracy gaps. With model increment value This is a mapping. This is an example of determining the update model increment threshold based on the received performance information and model increment information.

[0106] In the example embodiment, at the start of the optimization phase, the network can select a lower [specified value]. This leads to frequent sharing of model information (e.g., model increments, performance, or even the complete model to be delivered) each time the model is updated. However, by gathering more knowledge about the relationship between model increments and achievable model accuracy, the network can adjust the model increment threshold based on the desired accuracy requirements of the ML function. Make more appropriate adjustments. In addition, information collected from the UE regarding the evolution of that particular model (with model increments) can be used for model monitoring.

[0107] Note that performance and accuracy indicate how well the model performs. However, when the model is updated, this does not indicate how different the updated model is from the previous model, or therefore how much ML model data needs to be transferred to other entities due to the update. Therefore, the model increment threshold is used as a criterion to indicate whether it is necessary to share model information with other entities (e.g., transfer the model to the gNB).

[0108] Figure 9 A flowchart is shown, detailing example steps for incremental threshold updates of the model considering different scenarios. It corresponds to... Figure 8 An example of step 4 (model incremental threshold selection) is provided. However, other implementations of step 4 can be used for the proposed framework.

[0109] if Above the threshold and ,but Updated to .

[0110] if Below the threshold and Then, use Formula 2 as described in the following example embodiment to update. .

[0111] In an embodiment, if the performance information indicates a value higher than a first threshold, the updated model increment threshold is equal to the model increment value indicated by the model increment information; and if the performance information indicates a value lower than the first threshold, the updated model increment threshold is equal to the minimum of the sum of the model increment threshold and the offset and the model increment value indicated by the model increment information.

[0112] The following are the model increment thresholds. Selected example.

[0113] Initially, the model increment threshold was chosen to be a low initial value: .

[0114] Here, the model increment between the initial model and the updated model is calculated as a percentage difference between the weight values ​​as follows: ,in It is a vector of updated model weights, and It is the vector of initial model weights [Formula 1] The initial model increment threshold was chosen to be low to ensure that model information is shared each time the model is updated, since no knowledge has been collected at this stage.

[0115] The first UE1 updates its model, calculating the model increment according to network instructions. (Use Formula 1), and calculate the relevant performance as a percentage. ,in and These refer to the model accuracy of the updated model and the initial model, respectively. In this example, UE1 indicates a performance gap of 3% (which is relatively low). This shared information is used, and pre-selected fixed values ​​are assumed. The network updates the threshold to: [Formula 2] In practice, the new model did not bring significant performance improvements in this situation, therefore the model increment threshold could be increased to [a higher threshold]. However, we can achieve this through settings. The value is fixed (e.g., 3%), making it more conservative. Therefore, in this example, the model increment threshold becomes... .

[0116] The second UE2 also updates the ML model and estimates the model increment threshold. (Using Formula 1) and the corresponding model accuracy gap Since the performance difference is judged to be significant in this case (e.g., above the preselection limit of 15%), we will... and Compare them.

[0117] because This means that in this case, model information will be shared, and then It remains at 8%.

[0118] The third UE3 also updates the ML model and estimates the model increment threshold. (Using Formula 1) and the corresponding model accuracy gap Since the performance difference is deemed insignificant in this case (because it is below the 15% preselection limit), the model increment threshold should be updated according to Equation 1, which yields... .

[0119] Note that in this example, we consider updating the model incremental threshold based on each UE report. However, this process can be made more robust to account for situations where multiple UEs report multiple model incremental values ​​with varying performance differences. In this case, we recommend that the network not immediately update the model incremental threshold, but instead collect and aggregate information from multiple UEs before updating the threshold (e.g., for UEs with the same...). However, for the two UEs with performance gaps of 5% and 18% respectively, no update will be performed until more reports are received from other UEs corresponding to that specific 20% model increment.

[0120] The selection of the model increment threshold is related to the correspondence between indicating model updates and achieved performance. The main advantage is that it provides insight into model evolution / updates without requiring the entire model to be transferred, which helps the network make better decisions regarding model / function switching. The network can receive model information from the UE whenever relevant, specifically when the model increment exceeds the selected model threshold. In this way, the network can understand different significant model updates (in terms of achievable performance gains). Subsequently, the network can take relevant actions during this monitoring process. For example, model updates with high performance gains and high model increment values ​​reported to the network from multiple UEs can trigger a network request for model change / update for all UEs.

[0121] Additional information related to the machine learning model may include at least one of the following: model increment information related to the updated machine learning model, performance information related to the updated machine learning model, or the updated machine learning model.

[0122] In an example embodiment, when additional information includes an updated machine learning model, the method may include determining to provide the updated machine learning model to at least one additional entity on which the machine learning model is configured.

[0123] In the example embodiment, as referred to Figure 5 The described method may include receiving performance information and model increment information related to the machine learning model from multiple entities on which machine learning models are configured, and performing: determining a confidence value for the correspondence between the received model increment information and the performance.

[0124] For example, as part of the monitoring process, the network can solidify its perception of the correlation between model increments and the corresponding performance gaps / gains (in the case of one or more UEs reporting erroneous information). In practice, confidence levels can be correlated with information relevant to model increments by examining the variance between the values ​​reported by different UEs (considering the same ML model / feature). In this way, monitoring decisions can also take into account the potential erroneous reports from some UEs.

[0125] In one embodiment, the network may determine to retrain the machine learning model based on additional information and provide instructions to at least the entity on which the machine learning model is configured to retrain the machine learning model.

[0126] Table 1 shows how to select actions at the network based on the model increment threshold provided to the UE.

[0127]

[0128] If model performance degrades significantly, changing the entire architecture of the ML model is one solution. However, if the ML model performs worse than expected, NW can first update the model (recalculating the weights but maintaining the same structure). This is less costly and less time-consuming than changing the entire ML model structure (which requires searching for a new structure and recalculating the relevant parameters from scratch). This is an example of determining to retrain the ML model based on additional information.

[0129] While maintaining the same model structure, different model update techniques exist. Depending on the extent and implementation of these updates (retraining time, input date used, etc.), the updates will produce different performance results. Quantization and pruning can further influence the final model performance.

[0130] For model performance monitoring discussed in RAN1, alignment with two different datasets used for accuracy is crucial. For example, the UE obtains model A from the NW. The UE can retrain model A to model AA using dataset 1. Now, in performance monitoring, the UE will send ||A-AA|| to the NW. Given only a single value or performance metric, the NW will subsequently be unaware of what changes have occurred within the UE. However, if the model increment is sent by the UE, the NW knows which parameters caused the changes, and that the changes do not solely stem from different dataset distributions. The above provides a framework not only for indicating how the model changes under a given environment, but also for the NW to learn how the model changes from additional model increment information that considers the model's performance impact on the considered use case (in other words, ML functionality).

[0131] The apparatus may include components for: receiving performance information and model increment information related to the machine learning model from an entity on which a machine learning model is configured; determining an updated model increment threshold based at least on the received performance information and model increment information; and sharing the updated model increment threshold with the entity, wherein the model increment threshold is suitable for: determining by the entity on which the machine learning model is configured whether to share additional information related to the machine learning model; and providing an indication of the updated model increment threshold to the entity on which the machine learning model is configured.

[0132] The device may include a network entity, be a network entity or be included in a network entity, or may include a chipset for performing at least some actions of the network entity or for the network entity.

[0133] The apparatus may include components for: determining an updated machine learning model configured at the apparatus, updating the machine learning model, and determining model increment information based on the machine learning model and the updated machine learning model. Based on the comparison between the incremental information of the model and the incremental threshold of the model, it is determined whether to provide information related to the machine learning model to the network entity or to provide information related to the machine learning model to the network entity.

[0134] The apparatus may include, as referenced Figure 2 The described UE or network function or node, such as but not limited to gNB, CN, OAM or LMF, is a user equipment, network function or node or is included in a user equipment, network function or node, or may include a chipset for performing / for at least some actions of a user equipment, network function or node.

[0135] In the embodiments, means including network functions refers to a device / apparatus configured to perform or perform at least a portion of the functions of network functions, or means a device / apparatus including a circuit system (e.g., a chipset) configured to perform or perform at least a portion of the functions of network functions.

[0136] It should be understood that the device may include or be coupled to other units or modules, such as a radio section or radio head, for transmission and / or reception. Although the device has been described as a single entity, different modules and memories may be implemented in one or more physical or logical entities.

[0137] Note that while some embodiments have been described with respect to 5G networks, similar principles can be applied to other networks and communication systems, such as 6G networks or 5G-Advanced networks. Therefore, although some example architectures of wireless networks, technologies, and standards have been described above by way of example, these embodiments can be applied to any other suitable form of communication system besides those shown and described herein.

[0138] It should also be noted that although exemplary embodiments have been described above, several changes and modifications can be made to the disclosed solutions without departing from the scope of the invention.

[0139] As used herein, “at least one of the following: ” and “at least one of ” and similar wording (where the list of two or more elements is connected by “and” or “or”) means at least one of the elements, or at least any two or more of the elements, or all the elements.

[0140] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of this disclosure can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, but this disclosure is not limited thereto. Although various aspects of this disclosure may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, as non-limiting examples, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0141] As used in this application, the term "circuit" may refer to one or more or all of the following: (a) Hardware circuit implementation only (such as implementation in analog and / or digital circuits only) and (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of analog and / or digital (multiple) hardware circuits and software / firmware, and (ii) Any part of a (multiple) hardware processor having software (including (multiple) digital signal processors), software, and (multiple) memories that work together to enable a device (such as a mobile phone or a server) to perform various functions, and (multiple) hardware circuits and / or (multiple) processors (such as (multiple) microprocessors or a portion thereof) that require software (e.g., firmware) to operate, but may be absent when the software is not required to operate.

[0142] This definition of "circuit" applies to all uses of the term in this application (including in any claim). As another example, as used in this application, the term "circuit" also covers implementations of hardware circuitry or processors (or processors in general) and their accompanying software and / or firmware. The term "circuit" also covers, for example and if applicable to a particular claim element, baseband integrated circuits or processor integrated circuits for mobile devices or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

[0143] Embodiments of this disclosure may be implemented by computer software executable by a data processor of a mobile device (e.g., in a processor entity), or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products, including software routines, applets, and / or macros) may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof.

[0144] Furthermore, it should be noted in this regard that any block in the logical flow shown in the accompanying drawings may represent a program step, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. Software may be stored on physical media such as memory chips or blocks of memory implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs. Physical media are non-transitory media. As used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, not tactile), not a limitation on the persistence of data storage (e.g., RAM and ROM).

[0145] The memory can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. As a non-limiting example, the data processor can be of any type suitable for the local technical environment and can include one or more of general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), FPGAs, gate-level circuits, and processors based on multi-core processor architectures.

[0146] The embodiments of this disclosure can be practiced in various components such as integrated circuit modules. The design of integrated circuits is primarily a highly automated process. Complex and powerful software tools can be used to transform logic-level designs into semiconductor circuit designs ready for etching and formation on semiconductor substrates.

[0147] The scope of protection sought by the various embodiments of this disclosure is set forth in the independent claims. Embodiments and features described in this specification that do not fall within the scope of the independent claims, if any, are to be interpreted as examples useful for understanding the various embodiments of this disclosure.

[0148] The foregoing description has provided a complete and informative description of exemplary embodiments of the present disclosure by way of non-limiting example. However, various modifications and adjustments may become apparent to those skilled in the art when read in conjunction with the accompanying drawings and appended claims, given the foregoing description. Nevertheless, all such and similar modifications to the teachings of this disclosure will still fall within the scope of the invention as defined in the appended claims. Indeed, there are other embodiments that include combinations of one or more embodiments with any other embodiments discussed above.

Claims

1. An apparatus comprising components for performing the following: Receive performance information and model increment information related to the machine learning model from an entity on which a machine learning model is configured; Based at least on the received performance information and the model increment information, an updated model increment threshold is determined, and the updated model increment threshold is shared with the entity, wherein the model increment threshold is suitable for: determining by the entity on which the machine learning model is configured whether to share additional information related to the machine learning model; and Provide the entity on which the machine learning model is configured with an indication of the updated model increment threshold.

2. The apparatus of claim 1, wherein if the performance information indicates a value higher than a first threshold, the updated model increment threshold is equal to the model increment value indicated by the model increment information, and if the performance information indicates a value lower than the first threshold, the updated model increment threshold is equal to the minimum of the sum of the model increment threshold and the offset and the model increment value indicated by the model increment information.

3. The apparatus according to claim 1 or 2, comprising components for performing the following: Receive additional information related to the machine learning model.

4. The apparatus of claim 3, wherein the additional information associated with the machine learning model includes at least one of the following: model increment information associated with the updated machine learning model, performance information associated with the updated machine learning model, or the updated machine learning model.

5. The apparatus according to claim 3 or 4, comprising components for performing: Based on the additional information, determine to update the machine learning model; and At least the entity on which the machine learning model is configured is provided with instructions to update the machine learning model.

6. The apparatus of any one of claims 4 or 5, wherein the additional information comprises the updated machine learning model and a comparison component for performing the following: determining to provide the updated machine learning model to at least one additional entity on which the machine learning model is configured.

7. The apparatus according to any one of claims 1 to 6, comprising: receiving performance information and model increment information related to the machine learning model from a plurality of entities on which the machine learning model is configured, and performing: determining a confidence value for the correspondence between the received model increment information and the performance.

8. The apparatus according to any one of claims 1 to 7, comprising components for: Provide the entity on which the machine learning model is configured with configuration for configuring incremental parameters of the machine learning model.

9. The apparatus according to any one of claims 1 to 8, wherein the entity on which the machine learning model is configured includes a user equipment or network function.

10. An apparatus comprising components for: Determine the update to the machine learning model configured at the device; Update the machine learning model; Based on the machine learning model and the updated machine learning model, determine the model increment information; Based on the comparison between the model increment information and the model increment threshold, it is determined to provide information related to the machine learning model to the network entity; as well as Provide the network entity with the information related to the machine learning model.

11. The apparatus of claim 10, comprising components for: Based on the machine learning model and the updated machine learning model, performance information is further determined; Provide the model increment information and the performance information to the network entities; as well as In response, an updated model increment threshold is received from the network entity.

12. The apparatus of claim 10 or claim 11, wherein the information associated with the machine learning model includes at least one of the following: model increment information, performance information associated with the machine learning model, or the updated machine learning model.

13. The apparatus of any one of claims 10 to 12, wherein the model increment information is determined at least based on the difference between the weights of the machine learning model and the weights of the updated machine learning model.

14. The apparatus of claim 13, wherein the model increment information indicates at least one of a global machine learning increment value and a vector machine learning increment value.

15. A method comprising: Receive performance information and model increment information related to the machine learning model from an entity on which a machine learning model is configured; Based at least on the received performance information and the model increment information, an updated model increment threshold is determined and shared with the entity, wherein the updated model increment threshold is adapted to be determined by the entity on which the machine learning model is configured whether to share additional information related to the machine learning model; as well as Provide the entity on which the machine learning model is configured with an indication of the updated model increment threshold.

16. A method comprising: Determine the update to the machine learning model configured at the device; Update the machine learning model; Based on the machine learning model and the updated machine learning model, determine the model increment information; Based on the comparison between the model increment information and the model increment threshold, it is determined to provide information related to the machine learning model to the network entity; as well as Provide the network entity with the information related to the machine learning model.

17. An apparatus comprising at least one processor and at least one memory, the at least one memory storing instructions that, when executed by the processor, cause the apparatus to perform at least the following: Receive performance information and model increment information related to the machine learning model from an entity on which a machine learning model is configured; Based at least on the received performance information and the model increment information, an updated model increment threshold is determined, and the updated model increment threshold is shared with the entity, wherein the model increment threshold is suitable for: determining by the entity on which the machine learning model is configured whether to share additional information related to the machine learning model; and Provide the entity on which the machine learning model is configured with an indication of the updated model increment threshold.

18. An apparatus comprising at least one processor and at least one memory, the at least one memory storing instructions that, when executed by the processor, cause the apparatus to perform at least the following: Determine the update to the machine learning model configured at the device; Update the machine learning model; Based on the machine learning model and the updated machine learning model, determine the model increment information; Based on the comparison between the model increment information and the model increment threshold, it is determined to provide information related to the machine learning model to the network entity; as well as Provide the network entity with the information related to the machine learning model.

19. A computer-readable medium comprising instructions that, when executed by a device, cause the device to perform at least the following operations: Receive performance information and model increment information related to the machine learning model from an entity on which a machine learning model is configured; Based at least on the received performance information and the model increment information, an updated model increment threshold is determined, and the updated model increment threshold is shared with the entity, wherein the model increment threshold is suitable for: determining by the entity on which the machine learning model is configured whether to share additional information related to the machine learning model; and Provide the entity on which the machine learning model is configured with an indication of the updated model increment threshold.

20. A computer-readable medium comprising instructions that, when executed by a device, cause the device to perform at least the following operations: Determine the update to the machine learning model configured at the device; Update the machine learning model; Based on the machine learning model and the updated machine learning model, determine the model increment information; Based on the comparison between the model increment information and the model increment threshold, it is determined to provide information related to the machine learning model to the network entity; as well as Provide the network entity with the information related to the machine learning model.

Images