Method, apparatus and computer program

Abstract

In one embodiment, an apparatus is provided that includes sending a request for model training to a management service producer, where the request includes an indication of one or more critical data slices and / or data contexts to be used in the model training; and receiving a performance report from the management service producer based on the request for model training, the performance report including one or more performance metrics and one or more corresponding performance scores of the model over the one or more critical data slices and / or data contexts.

Description

Technical Field

[0001] The present application relates to a method, apparatus, system and computer program, and particularly to, but not limited to, performing model training on key data slices and / or data contexts. Background Art

[0002] A communication system may be viewed as a facility that enables a communication session between two or more entities (such as user terminals, base stations and / or other nodes) by providing a carrier wave between the various entities involved in the communication path. A communication system may be provided, for example, by means of a communication network and one or more compatible communication devices. A communication session may include, for example, data communications for carrying communications such as voice, video, electronic mail (email), text messaging, multimedia and / or content data. Non-limiting examples of the services provided include two-way or multi-way calls, data communications or multimedia services, and access to a data network system (such as the Internet).

[0003] In a wireless communication system, at least a portion of a communication session between at least two stations takes place over a wireless link. Examples of wireless systems include public land mobile networks (PLMNs), satellite-based communication systems, and different wireless local area networks, such as wireless local area networks (WLANs). Some wireless systems may be divided into cells, and are therefore often referred to as cellular systems.

[0004] A user may access a communication system by means of an appropriate communication device or terminal. The user's communication device may be referred to as a user equipment (UE) or user device. The communication device is equipped with appropriate signal receiving and transmitting means to enable communication, for example, to access a communication network or to communicate directly with other users. The communication device may access a carrier provided by a site (e.g., a base station of a cell) and transmit and / or receive communications on the carrier.

[0005] Communication systems and associated equipment typically operate according to a given standard or specification, which specifies what the various entities associated with the system are allowed to do and how it is achieved. Communication protocols and / or parameters for the connection are also typically defined. An example of a communication system is UTRAN (3G radio). Other examples of communication systems are the Long Term Evolution (LTE) of the Universal Mobile Telecommunications System (UMTS) radio access technology and the so-called 5G or New Radio (NR) networks. NR is being standardized by the Third Generation Partnership Project (3GPP). Summary of the invention

[0006] According to one aspect, a device is provided, comprising means for: sending a request for model training to a management service producer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; and receiving a performance report from the management service producer based on the request for model training, wherein the performance report includes one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0007] The device can be used to: obtain information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training from the management service producer; wherein the one or more key data slices and / or data contexts to be used in the model training can be selected based on the obtained information.

[0008] The acquisition may include: sending a request to the management service producer for information indicating the one or more data slices and / or data contexts that have been previously used and / or can be used for the model training; and receiving from the management service producer information indicating the one or more data slices and / or data contexts that have been previously used and / or can be used for the model training.

[0009] The request for model training may also include information indicating one or more key data slices and / or data context performance requirements, wherein the performance report may be based on the information indicating the one or more key data slices and / or data context performance requirements.

[0010] The request for model training may also include information indicating one or more overall model performance requirements.

[0011] The apparatus may be configured to: send a request for the performance report to the management service producer; and receive the performance report in response.

[0012] The performance report may also include information indicative of the overall performance of the model.

[0013] According to one aspect, a device is provided, comprising means for: receiving a request for model training from a management service consumer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; performing model training on the one or more key data slices and / or data contexts; and sending a performance report to the management service consumer based on the model training, the performance report including one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0014] Performing the model training may include at least one of the following: using one or more residual attention modules to model the residual between the key data slice and / or data context level and the overall model prediction; resampling or assigning higher weights to key data slice or data context specific training samples; performing multi-task learning, in which key data slice or data context specific tasks are learned using hard parameters or soft parameter sharing; and mapping the key data slice and / or data context to train each smaller machine learning model together with the model.

[0015] The device can be used to: acquire model test data in response to performing the model training; filter the model test data to generate filtered model test data, wherein the filtered model test data only includes data from the one or more key data slices and / or data contexts; and generate a performance report based on the filtered model test data.

[0016] The apparatus may be configured to provide the management service consumer with information indicating one or more data slices and / or data contexts that have been previously used and / or may be used for training the model.

[0017] The device can be used to: receive a request for information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training from the management service consumer; and send information indicating the one or more data slices and / or data contexts that have been previously used and / or can be used for the model training to the management service consumer.

[0018] The apparatus may be configured to: receive a request for the performance report from the management service consumer; and send the performance report in response.

[0019] The model may be an artificial intelligence and / or machine learning model, wherein the management service producer may be an artificial intelligence and / or machine learning training function.

[0020] The request for model training may also include information indicating one or more key data slices and / or data context performance requirements, wherein the performance report may be based on the information indicating the one or more key data slices and / or data context performance requirements.

[0021] The request for model training may also include information indicating one or more overall model performance requirements.

[0022] The performance report may also include information indicative of the overall performance of the model.

[0023] According to one aspect, a device is provided, comprising at least one processor and at least one memory comprising computer program code, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the device to at least: send a request for model training to a management service producer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; and based on the request for model training, receive a performance report from the management service producer, wherein the performance report includes one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0024] The at least one memory and the at least one processor can be configured to enable the device to: obtain, from the management service producer, information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training; wherein the one or more key data slices and / or data contexts to be used in the model training can be selected based on the obtained information.

[0025] The at least one memory and the at least one processor can be configured to enable the device to obtain the information by sending a request to the management service producer for information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training; and to receive information from the management service producer indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training.

[0026] The request for model training may also include information indicating one or more key data slices and / or data context performance requirements, wherein the performance report may be based on the information indicating the one or more key data slices and / or data context performance requirements.

[0027] The request for model training may also include information indicating one or more overall model performance requirements.

[0028] The at least one memory and the at least one processor may be configured to cause the apparatus to: send a request for the performance report to the management service producer; and receive the performance report in response.

[0029] The performance report may also include information indicative of the overall performance of the model.

[0030] According to one aspect, a device is provided, comprising at least one processor and at least one memory comprising computer program code, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, enable the device to at least: receive a request for model training from a management service consumer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; perform model training on the one or more key data slices and / or data contexts; and based on the model training, send a performance report to the management service consumer, wherein the performance report includes one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0031] The at least one memory and the at least one processor can be configured to cause the device to perform the model training by at least one of the following: using one or more residual attention modules to model the residual between the key data slice and / or data context level and the overall model prediction; resampling or assigning higher weights to key data slice or data context specific training samples; performing multi-task learning, wherein key data slice or data context specific tasks are learned using hard parameters or soft parameter sharing; and mapping the key data slice and / or data context to train each smaller machine learning model together with the model.

[0032] The at least one memory and the at least one processor can be configured to enable the device to: acquire model test data in response to performing the model training; filter the model test data to generate filtered model test data, wherein the filtered model test data only includes data from the one or more key data slices and / or data contexts; and generate a performance report based on the filtered model test data.

[0033] The at least one memory and the at least one processor may be configured to cause the apparatus to: provide information to the management service consumer indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training.

[0034] The at least one memory and the at least one processor can be configured to enable the device to: receive a request from the management service consumer for information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training; and send information to the management service consumer indicating the one or more data slices and / or data contexts that have been previously used and / or can be used for the model training.

[0035] The at least one memory and the at least one processor may be configured to cause the apparatus to: receive a request for the performance report from the management service consumer; and send the performance report in response.

[0036] The model may be an artificial intelligence and / or machine learning model, wherein the management service producer may be an artificial intelligence and / or machine learning training function.

[0037] The request for model training may also include information indicating one or more key data slices and / or data context performance requirements, wherein the performance report may be based on the information indicating the one or more key data slices and / or data context performance requirements.

[0038] The request for model training may also include information indicating one or more overall model performance requirements.

[0039] The performance report may also include information indicative of the overall performance of the model.

[0040] According to one aspect, a method is provided, comprising: sending a request for model training to a management service producer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; and based on the request for model training, receiving a performance report from the management service producer, wherein the performance report includes one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0041] The method may include: obtaining, from the management service producer, information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training; wherein, the one or more key data slices and / or data contexts to be used in the model training may be selected based on the obtained information.

[0042] The acquisition may include: sending a request to the management service producer for information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for model training; and receiving information from the management service producer indicating one or more data slices and / or data contexts that have been previously used and / or can be used for model training.

[0043] The request for model training may also include information indicating one or more key data slices and / or data context performance requirements, wherein the performance report may be based on the information indicating the one or more key data slices and / or data context performance requirements.

[0044] The request for model training may also include information indicating one or more overall model performance requirements.

[0045] The method may include sending a request for the performance report to the managed service producer; and receiving the performance report in response.

[0046] The performance report may also include information indicative of the overall performance of the model.

[0047] According to one aspect, a method is provided, comprising: receiving a request for model training from a management service consumer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; performing model training on the one or more key data slices and / or data contexts; and sending a performance report to the management service consumer based on the model training, wherein the performance report includes one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0048] Performing the model training may include at least one of the following: using one or more residual attention modules to model the residual between the key data slice and / or data context level and the overall model prediction; resampling or assigning higher weights to key data slice or data context specific training samples; performing multi-task learning, in which key data slice or data context specific tasks are learned with hard parameters or soft parameter sharing; and mapping the key data slice and / or data context to train various smaller machine learning models together with the model.

[0049] The method may include: in response to performing the model training, acquiring model test data; filtering the model test data to generate filtered model test data, wherein the filtered model test data only includes data from the one or more key data slices and / or data contexts; and generating a performance report based on the filtered model test data.

[0050] The method may include providing information to the management service consumer indicating one or more data slices and / or data contexts that have been previously used and / or may be used for the model training.

[0051] The method may include: receiving a request for information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training from the management service consumer; and sending information indicating the one or more data slices and / or data contexts that have been previously used and / or can be used for the model training to the management service consumer.

[0052] The method may include receiving a request for the performance report from the management service consumer; and sending the performance report in response.

[0053] The model may be an artificial intelligence and / or machine learning model, wherein the management service producer may be an artificial intelligence and / or machine learning training function.

[0054] The request for model training may also include information indicating one or more key data slices and / or data context performance requirements, wherein the performance report may be based on the information indicating the one or more key data slices and / or data context performance requirements.

[0055] The request for model training may also include information indicating one or more overall model performance requirements.

[0056] The performance report may also include information indicative of the overall performance of the model.

[0057] According to one aspect, a computer-readable medium comprising program instructions is provided for causing an apparatus to perform at least the following: sending a request for model training to a management service producer, wherein the request comprises an indication of one or more key data slices and / or data contexts to be used in the model training; and receiving a performance report from the management service producer based on the request for model training, wherein the performance report comprises one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0058] The program instructions may cause the device to execute: obtaining, from the management service producer, information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for model training; wherein, the one or more key data slices and / or data contexts to be used in the model training may be selected based on the obtained information.

[0059] The acquisition may include: sending a request to the management service producer for information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for model training; and receiving from the management service producer information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for model training.

[0060] The request for model training may also include information indicating one or more key data slices and / or data context performance requirements, wherein the performance report may be based on the information indicating the one or more key data slices and / or data context performance requirements.

[0061] The request for model training may also include information indicating one or more overall model performance requirements.

[0062] The program instructions may cause the apparatus to: send a request for the performance report to the managed service producer; and receive the performance report in response.

[0063] The performance report may also include information indicative of the overall performance of the model.

[0064] According to one aspect, a computer-readable medium comprising program instructions is provided for causing an apparatus to perform at least the following: receiving a request for model training from a management service consumer, wherein the request comprises an indication of one or more key data slices and / or data contexts to be used in the model training; performing model training on the one or more key data slices and / or data contexts; and sending a performance report to the management service consumer based on the model training, the performance report comprising one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0065] Performing the model training may include at least one of the following: using one or more residual attention modules to model the residual between the key data slice and / or data context level and the overall model prediction; resampling or assigning higher weights to key data slice or data context specific training samples; performing multi-task learning, in which key data slice or data context specific tasks are learned with hard parameters or soft parameter sharing; and mapping the key data slice and / or data context to train various smaller machine learning models together with the model.

[0066] The program instructions may cause the device to execute: in response to performing the model training, obtaining model test data; filtering the model test data to generate filtered model test data, wherein the filtered model test data only includes data from the one or more key data slices and / or data contexts; and generating a performance report based on the filtered model test data.

[0067] The program instructions may cause the apparatus to perform: providing information to the management service consumer indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training.

[0068] The program instructions may cause the device to perform: receiving a request for information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for the model training from the management service consumer; and sending information indicating the one or more data slices and / or data contexts that have been previously used and / or can be used for the model training to the management service consumer.

[0069] The program instructions may cause the apparatus to: receive a request for the performance report from the management service consumer; and send the performance report in response.

[0070] The model may be an artificial intelligence and / or machine learning model, wherein the management service producer may be an artificial intelligence and / or machine learning training function.

[0071] The request for model training may also include information indicating one or more key data slices and / or data context performance requirements, wherein the performance report may be based on the information indicating the one or more key data slices and / or data context performance requirements.

[0072] The request for model training may also include information indicating one or more overall model performance requirements.

[0073] The performance report may also include information indicative of the overall performance of the model.

[0074] According to one aspect, there is provided a non-transitory computer-readable medium including program instructions for causing an apparatus to execute a method according to at least any one of the preceding aspects.

[0075] In the above, many different embodiments have been described. It should be understood that further embodiments can be provided by combining any two or more of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0077] Figure 1 shows a representation of a network system according to some example embodiments;

[0078] Figure 2 shows a representation of a control device according to some example embodiments;

[0079] Figure 3 shows representations of apparatus according to some example embodiments;

[0080] Figure 4 Methods according to some examples are shown;

[0081] Figure 5 shows a signaling exchange according to some examples; and

[0082] Figure 6 Representations of methods for implementing a residual attention module for learning based on key data slices or data contexts according to some examples are shown. DETAILED DESCRIPTION

[0083] In some of the following embodiments, certain embodiments are explained with reference to a mobile communication device capable of communicating via a wireless cellular system and a mobile communication system serving such a mobile communication device. Figure 1 , 2 and 3 briefly explain certain general principles of wireless communication systems, their access systems and mobile communication devices to help understanding the technology on which the described examples are based.

[0084] Figure 1 A schematic diagram of a 5G system (5GS) is shown. The 5GS may include a terminal or user equipment (UE), a 5G radio access network (5GRAN) or a next generation radio access network (NG-RAN), a 5G core network (5GC), one or more application functions (AFs), and one or more data networks (DNs).

[0085] 5G-RAN may include one or more gNodeBs (GNBs), or one or more gNodeB (GNB) distributed unit functions connected to one or more gNodeB (GNB) centralized unit functions. 5GC may include the following entities: Network Slice Selection Function (NSSF); Network Exposure Function; Network Repository Function (NRF); Policy Control Function (PCF); Unified Data Management (UDM); Application Function (AF); Authentication Server Function (AUSF); Access and Mobility Management Function (AMF); and Session Management Function (SMF).

[0086] Figure 2 Shows the control Figure 1An example of a control device 200 for the functions of a 5GRAN or 5GC shown. The control device may include at least one random access memory (RAM) 211a, at least one read-only memory (ROM) 211b, at least one processor 212, 213, and an input / output interface 214. At least one processor 212, 213 may be coupled to the RAM 211a and the ROM 211b. At least one processor 212, 213 may be configured to execute appropriate software code 215. The software code 215 may, for example, allow one or more steps to be executed to perform one or more aspects of the present invention. The software code 215 may be stored in the ROM 211b. The control device 200 may be interconnected with another control device 200 that controls another function of the 5GRAN or 5GC. In some embodiments, each function of the 5GRAN or 5GC includes a control device 200. In an alternative embodiment, two or more functions of the 5GRAN or 5GC may share a control device.

[0087] Figure 3 An example of a terminal 300 is shown, such as Figure 1 The terminal shown. The terminal 300 can be provided by any device capable of transmitting and receiving radio signals. Non-limiting examples include user equipment, a mobile station (MS) or a mobile device (such as a mobile phone or a so-called "smartphone"), a computer equipped with a wireless interface card or other wireless interface facilities (e.g., a USB dongle), a personal data assistant (PDA) or a tablet equipped with wireless communication capabilities, a machine type communication (MTC) device, an Internet of Things (IoT) type communication device, or any combination of these or the like. The terminal 300 can provide, for example, data communication for bearer communication. The communication can be one or more of voice, electronic mail (email), text message, multimedia, data, machine data, etc.

[0088] The terminal 300 may receive signals over the air or radio interface 307 via suitable means for receiving, and may transmit signals via suitable means for transmitting radio signals. Figure 3 In the mobile device, the transceiver arrangement is schematically represented by block 306. The transceiver arrangement 306 may be provided, for example, by means of a radio component and an associated antenna arrangement. The antenna arrangement may be arranged internally or externally to the mobile device.

[0089] The terminal 300 may be equipped with at least one processor 301, at least one memory ROM 302a, at least one RAM 302b and other possible components 303 for performing its designed tasks in software and hardware assistance, including controlling access to and communication with access systems and other communication devices. The at least one processor 301 is coupled to the RAM 302b and the ROM 302a. The at least one processor 301 may be configured to execute appropriate software code 308. The software code 308 may, for example, allow one or more aspects of the present invention to be performed. The software code 308 may be stored in the ROM 302a.

[0090] The processor, memory and other related control means may be provided on an appropriate circuit board and / or in a chipset. This feature is indicated by reference numeral 304. The device may optionally have a user interface, such as a keyboard 305, a touch screen or touch pad, a combination thereof, etc. Optionally, one or more of a display, a speaker and a microphone are provided, depending on the type of device.

[0091] 3GPP TS28.105 specifies artificial intelligence (AI) / machine learning (ML) management functions and services for 5GS. Some example network deployments can use AI / ML functions or models to perform decisions to enhance network performance.

[0092] 3GPP TS28.105 provides a method in which a management service (MnS) consumer can request training or retraining of an AI / ML model from an MnS producer. In some examples, the request can also provide a preferred level of model performance. As used herein, an AI / ML model can be defined as a mathematical algorithm that can be "trained" by data and human expert input as examples to replicate the decisions that an expert would make when provided with the same information.

[0093] In response, the MnS producer (such as an AI / ML training function) can collect the necessary data and train the AI / ML model corresponding to the request. The MnS producer can then send a performance evaluation report related to the AI / ML model (re)training to the MnS consumer.

[0094] AI / ML models trained by MnS producers can achieve high “overall” performance on metrics such as accuracy because the AI / ML models can be optimized for “overall” quality.

[0095] However, AI / ML models may severely underperform on critical data slices and / or data contexts (such as data samples related to a specific UE or cell or tracking area or network slice or time period) - for example by producing unacceptable failure rates.

[0096] Therefore, the challenge for MnS producers is to maintain the overall quality of AI / ML models while improving key data slices or data context specific performance metrics that are important to MnS consumers.

[0097] For example, an MnS consumer may determine that the predicted quality of service (QoS) AI / ML model performs poorly for UEs within the coverage of cell 1 only during nighttime. In this example, the data context may include a cell identifier indicating cell 1 and a time period indicating nighttime. For an MnS consumer, it may be beneficial to have a mechanism to indicate critical data slices or data contexts, i.e., classifying UEs belonging to cell 1, and classifying daytime vs. nighttime based on timestamps, which MnS producers should pay more attention to during AI / ML model (re)training.

[0098] As another example, the MnS consumer may determine that the predictive handover AI / ML model performs poorly only for high-speed UEs, even if the total number of handover failures is less than a certain threshold across all UEs. For the MnS consumer, it would be beneficial to have a mechanism to indicate critical data slices or data contexts, i.e., separate high-speed UEs from other UEs, to which the MnS producer should pay more attention during AI / ML model (re)training.

[0099] In the current 3GPP specifications, there is no means for MnS consumers to provide / specify key data slices or data contexts in AI / ML model (re)training requests, including the preferred level of key data slice or data context specific model performance. Therefore, there is no means for MnS producers to perform data slice or data context based learning (to successfully model the importance of key data slices or specific data contexts) to improve the key data slice or data context level performance of AI / ML models without affecting the overall AI / ML model performance. In addition, there is no means for MnS producers to provide key data slice or data context specific AI / ML model performance evaluation reports to MnS consumers.

[0100] Some embodiments may address one or more of these issues.

[0101] refer to Figure 4 , which illustrates methods according to some examples.

[0102] At 400 , the method includes sending a request for model training to a management service producer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in model training.

[0103] At 402, the method includes receiving, from a management service producer and based on a request for model training, a performance report including one or more performance metrics and one or more corresponding performance scores for the model on one or more key data slices and / or data contexts.

[0104] At 404 , the method includes receiving a request for model training from a management service consumer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in model training.

[0105] At 406 , the method includes performing model training on one or more key data slices and / or data contexts.

[0106] At 408, the method includes sending a performance report to a management service consumer based on the model training, which includes one or more performance metrics and one or more corresponding performance scores of the model on one or more key data slices and / or data contexts.

[0107] Some embodiments may enable MnS consumers to discover data slices and / or data contexts that have been used by MnS producers during AI / ML model (re)training or will be used by MnS producers for AI / ML model (re)training. Some embodiments may enable MnS consumers to provide / specify one or more key data slices or data contexts, including the preferred level of key data slice or data context specific model performance that MnS producers should consider / satisfy during AI / ML model (re)training.

[0108] Some embodiments may enable MnS producers to provide MnS consumers with one or more data slices and / or data contexts that producers use to train existing models. Some embodiments may enable MnS producers to perform key data slice or data context based learning to successfully model key data slice or data context importance and improve key data slice or data context level performance without affecting overall AI / ML model performance. Some embodiments may enable MnS producers to evaluate and report to MnS consumers a list of one or more key data slice or data context specific model performance metrics.

[0109] refer to Figure 5 , which illustrates a method according to some embodiments, where a MnS consumer 500 communicates with a MnS producer 502. In some examples, the MnS producer may be an AI / ML training function.

[0110] At 504, the MnS consumer determines one or more key data slices and / or data contexts.

[0111] This determination can be based on granular use case specific requirements or granular performance degradation analysis. In some examples, the MnS consumer can use information received from the MnS producer, such as information indicating that the MnS producer supports one or more data slices and / or data contexts for AI / ML training, or information indicating one or more data slices and / or data contexts that the MnS consumer has previously used for AI / ML training.

[0112] For example, in the predictive QoS use case, the MnS consumer may determine that the AI / ML model performs poorly only for UEs within the coverage of Cell ID 1 at night. Based on this determination, the MnS consumer may identify that the key data slices or data contexts that require more attention from the MnS consumer during (re)training are the data samples from UEs within the coverage of Cell ID 1 between the time period of 19:00–21:00 hours.

[0113] In another example, in a predictive handoff use case, the MnS consumer may determine that the AI / ML model performs poorly only for high-speed UEs. Based on this determination, the MnS consumer may identify that the key data slices or data contexts that require more attention from the MnS consumer during (re)training are data samples from UEs moving at speeds exceeding 50 km / hr.

[0114] At 506 , the MnS consumer discovers data slices or data contexts that have been used by the MnS producer during AI / ML model (re)training, or that the MnS producer is going to use for AI / ML model (re)training.

[0115] For example, Figure 5 As shown, the MnS consumer may send a discovery request message to the MnS producer (e.g., an AIMLDataInformationDiscoveryRequest operation). In response, the MnS producer may send a corresponding report to the MnS consumer (e.g., an AIMLDataInformationDiscoveryReport operation).

[0116] The MnS producer may determine one or more data slices or data contexts to include in the report based on the training data used for AI / ML model (re)training.

[0117] The training data may be selected from candidate training data provided by MnS consumers and / or from training data collected via other data sources.

[0118] As an example, in the predictive QoS AI / ML use case, the data slice or data context may include but is not limited to time period (timestamp feature), tracking area (tracking area identifier feature), UE (UE identifier feature), cell (cell identifier feature).

[0119] In a further example, in a predicted handover use case, the data slice or data context may include but is not limited to UE (UE identifier feature), cell (cell identifier feature), tracking area (tracking area identifier feature), UE speed (speed feature), time period (timestamp feature), etc.

[0120] In some examples, data slices or data contexts can be categorized into the following domains:

[0121] ●Time domain, such as time period (timestamp feature)

[0122] ● Geographical domain, such as tracking area (tracking area identification feature), paging area (paging area identification feature), coverage area

[0123] ● Target object domain, i.e. UE (UE identifier feature), cell (cell identifier feature)

[0124] ●Mobility domain, i.e. UE speed (speed characteristics)

[0125] ● A combination of the above, such as a UE within a specific tracking area.

[0126] It should be understood that further categories and examples of data slices or data context information may be used in other examples.

[0127] At 508, the MnS consumer sends a request for AI / ML model (re)training to the MnS producer. The request indicates one or more key data slices or data contexts that the MnS producer should focus on during the AI / ML model (re)training.

[0128] The request may also include key data slice or data context specific AI / ML model performance requirements that the MnS producer should meet during AI / ML model (re)training.

[0129] The request may also include the overall AI / ML model performance requirements that the MnS producer should meet during AI / ML model (re)training. As used herein, overall AI / ML model performance may include the performance of the model on data slices and / or data contexts, including but not limited to key data slices and / or data contexts. For example, a key data context may indicate that model training is to focus on a specific time of day (e.g., 11:00-13:00), while the overall performance may be based on the performance of the model throughout the day (i.e., 00:00-23:59).

[0130] In some examples, the request may also include one or more key data slice or data context specific AI / ML model performance measurements or metrics for use by the MnS producer during (re)training.

[0131] In some examples, the key data slice or data context specific AI / ML model performance measurements or metrics may be different for different key data slices or data contexts. For example, if the cost of a false positive is high for a particular key data slice or data context, the requirement may include a preferred level of precision, while if the cost of a false negative is high for a particular key data slice or data context, the requirement may include a preferred level of recall.

[0132] At 510 , the MnS producer performs key data slice or data context based learning to successfully model the importance of the key data slice or data context.

[0133] Learning can improve the performance of AI / ML models at the level of critical data slices or data contexts without affecting the overall AI / ML model performance.

[0134] In some examples, the MnS producer may utilize one or more of the following when performing step 510:

[0135] ○ Using a residual attention module to explicitly model the residual between a key data slice or level of data context and the overall prediction (described in more detail below);

[0136] ○ Resample or assign higher weights to key data slices or data context specific training samples so that the training model tries to minimize the error on the categories with larger weights;

[0137] ○ Use multi-task learning, where key data slices or data context specific tasks are learned with hard or soft parameter sharing;

[0138] ○ Optionally, data slices or data contexts can be mapped to train a single smaller ML model on the MnS producer side and augment the original trained ML model along the way.

[0139] As described above, in some examples, the MnS producer may implement a residual attention module when performing step 510 .

[0140] For example, Chen, Vincent, Sen Wu, Alexander J. Ratner, Jen Weng, and Christopher Ré "Slice-based learning: A programming model for residual learning in critical data slices. Advances in neural information systems 32 (2019) Slice-based learning: A programming model for residual learning in critical data slices." Advances in neural information processing systems 32 (2019) proposed a slice-based learning model. Features such as slice functions (SFs) can specify critical data subsets for which models (such as AI / ML models) should provide additional capacity.

[0141] According to Chen et al., any model can leverage SF to learn slice-expert representations, which can then be combined with an attention mechanism for slice-aware predictions. The approach maintains parameter-efficient representations while improving by 19.0F1 over the baseline on slices and 4.6F1 overall on datasets covering language understanding (e.g., SuperGLUE), computer vision, and production-scale industrial systems.

[0142] refer to Figure 6 , which shows a representation of a method for implementing a residual attention module for learning based on key data slices or data context according to Chen et al.

[0143] The developer writes SF(λi=1, ..., k) 600 on an input such as an image 601. The developer can specify any backbone architecture 602 as a feature extractor. The features are shared parameters of k slice residual attention modules (SRAM) 604.

[0144] Each SRAM learns a slice indication head 604a, as supervised by λi, which outputs a prediction qi indicating which slice the example belongs to. The SRAM also learns a slice expert representation 604b, trained only on examples belonging to that slice using a shared slice prediction head 606, which produces a prediction pi on the original task mode and is supervised by the masked ground truth label of the corresponding slice.

[0145] The attention mechanism 608, a re-weights these representations ri into a combined slice-aware representation 610. The final prediction head 612 makes model predictions based on the slice-aware representation.

[0146] For example, a developer may determine that an autonomous car cannot detect cyclists at night with the desired accuracy. The developer may write a first SF that detects bicycles by calling a pre-trained object detector for bicycles, and a second SF that classifies night vs. day based on average pixel intensity. The first SRAM may be generated using the first SF and the backbone architecture, and the second SRAM may be generated using the second SF and the backbone architecture. Both SRAMs may also use a shared slice prediction head to make slice-aware predictions that are reweighted and used to generate the prediction head, such as there is no cyclist in the input image. This can be repeated for many more input images to help train the model for nighttime cyclist detection.

[0147] While one specific example method of a learning mechanism has been provided, it should be understood that in other examples, any other suitable method for performing the learning at step 510 may be implemented.

[0148] At 512 , the MnS producer evaluates the performance of the model from the 510 training.

[0149] In some examples, the MnS producer may filter the test data set to include only data from key slices or contexts indicated by the MnS consumer. That is, the MnS producer may generate filtered model test data that includes only data from one or more key data slices and / or data contexts. The MnS producer may use the filtered model test data set to evaluate the performance of the training model.

[0150] As mentioned above, the performance metric may be different for different key data slices or data contexts. If the MnS consumer does not request a specific metric, the MnS producer may determine the performance metric directly by the MnS producer.

[0151] For example, if the cost of a false positive is high for a particular key data slice or data context, then the MnS producer may determine that a precision metric may be reported, whereas if the cost of a false negative is high for a particular key data slice or data context, then the MnS producer may determine that a recall metric may be reported.

[0152] At 514, the MnS consumer may request one or more key data slices and / or data context specific AI / ML model performance reports from the MnS producer. The request may include relevant performance metrics and corresponding performance scores that the MnS producer wants to use in the report. In addition, the MnS consumer may also request an overall AI / ML model performance report.

[0153] At 516, the MnS producer sends a report to the MnS consumer.

[0154] The report may include one or more key data slices or data context specific AI / ML model performance reports. The performance report may include relevant performance metrics and corresponding performance scores requested by the MnS consumer, and optionally, an overall AI / ML model performance report. The report sent at step 516 may be sent in response to the request sent by the MnS consumer at step 514.

[0155] Therefore, in Figure 5 In an example method, the MnS consumer may determine one or more key data slices and / or data contexts and request the MnS producer to perform AI / ML (re)training to focus on the one or more determined key data slices and / or data contexts. The MnS producer may accordingly perform AI / ML (re)training and provide an AI / ML model performance report, including information indicating one or more performance metrics of the AI / ML model for the one or more determined key data slices and / or data contexts.

[0156] Thus, the AI / ML model performance report can provide an indication to the MnS consumer of how well the AI / ML model is operating on key data slices and / or data contexts. The MnS consumer can use this indication to further optimize network performance - for example, the MnS consumer can continue to receive analytics from the same model, request to retrain the model, request to terminate the use of the model, or request to use another model, depending on the reported data slice specific model performance from the MnS producer.

[0157] In some examples, a device is provided, comprising a device for sending a request for model training to a management service producer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; and based on the request for model training, receiving a performance report from the management service producer, the performance report including one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0158] In some examples, the device may include at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, together with the at least one processor, cause the device to at least: send a request for model training to a management service producer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; and based on the request for model training, receive a performance report from the management service producer, which includes one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0159] In some examples, a device is provided, comprising means for: receiving a request for model training from a management service consumer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; performing model training on the one or more key data slices and / or data contexts; and based on the model training, sending a performance report to the management service consumer, the performance report including one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0160] In some examples, the device may include at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, together with the at least one processor, cause the device to at least: receive a request for model training from a management service consumer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; perform model training on the one or more key data slices and / or data contexts; and based on the model training, send a performance report to the management service consumer, the performance report including one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

[0161] It should be understood that these devices may include or be coupled to other units or modules for transmission and / or reception, such as radio components or radio heads. Although these device arrangements have been described as one entity, different modules and memories may be implemented in one or more physical or logical entities.

[0162] It is worth noting that although some embodiments have been described with respect to 5G networks, similar principles can be applied to other networks and communication systems. Therefore, although certain embodiments are described above by way of example with reference to certain example architectures of wireless networks, technologies and standards, the embodiments can be applied to any other suitable form of communication system in addition to the communication system shown and described herein.

[0163] It is also noted herein that while the above describes example embodiments, there are numerous variations and modifications which may be made to the disclosed solution without departing from the scope of the present invention.

[0164] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Certain aspects of the present application may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present application is not limited thereto. Although various aspects of the present application may be illustrated and described by block diagrams, flow charts, or using some other graphical representations, it is fully understood that the boxes, devices, systems, techniques, or methods described herein may be implemented as non-limiting examples in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof.

[0165] As used in this application, the term "circuitry" may refer to one or more or all of the following:

[0166] (a) hardware-only circuit implementations (such as implementations in analog and / or digital circuits only); and

[0167] (b) a combination of hardware circuitry and software such as (where applicable):

[0168] (i) a combination of analog and / or digital hardware circuitry and software / firmware, and

[0169] (ii) any portion of a hardware processor and software (including a digital signal processor), software and memory that work together to enable a device such as a mobile phone or server to perform various functions;

[0170] (c) A hardware circuit and / or processor, such as a microprocessor or portion of a microprocessor, that requires software (e.g., firmware) to operate, but the software may not be present when not required to operate.

[0171] This definition of circuitry applies to one or all uses of the term in this application, including in any claims. As a further example, as used in this application, the term "circuitry" also covers an implementation of only a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. For example, if applicable to a particular claim, the term "circuitry" also covers a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0172] Embodiments of the present application may be implemented (such as in a processor entity) by computer software executable by a data processor of a mobile device, or by hardware, or by a combination of software and hardware. Computer software or programs, also referred to as program products, include software routines, applets and / or macros, which may be stored in a data storage medium readable by any device, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components that are configured to perform an embodiment when the program is running. One or more computer executable components may be at least one software code or a portion thereof.

[0173] Also in this regard, it should be noted that any block in the logic flow of the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs. Physical media are non-transitory media.

[0174] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based storage devices, magnetic storage devices and systems, optical storage devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an FPGA, a gate-level circuit based on a multi-core processor architecture, and a processor, as non-limiting examples.

[0175] Embodiments of the present application may be practiced in various components such as integrated circuit modules. The design of integrated circuits is a highly automated process. Complex and powerful software tools are available to convert logic-level designs into semiconductor circuit designs that can be etched and formed on semiconductor substrates.

[0176] The scope of protection sought by various embodiments of the present application is set forth by the independent claims. Embodiments and features described in this specification that do not fall within the scope of the independent claims (if any) should be interpreted as examples of various embodiments that help understand the content of the present application.

[0177] The foregoing description has provided by way of non-limiting examples a complete and informative description of exemplary embodiments of the present application. However, various modifications and adaptations will be apparent to those skilled in the relevant art when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications to the teachings of the present application will still fall within the scope of the present invention as defined by the appended claims. In fact, there are further embodiments that include a combination of one or more embodiments with any other embodiments previously discussed.

Claims

1. An apparatus comprising means for: Send a request for model training to the management service producer, where: The request includes an indication of one or more key data slices and / or data contexts to be used in training the model; as well as Based on the request for model training, a performance report is received from the management service producer, the performance report comprising one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

2. The device according to claim 1, wherein: The device is used for: obtaining, from the management service producer, information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for training the model; Wherein, the one or more key data slices and / or data contexts to be used in the model training are selected based on the acquired information.

3. The device of claim 2, wherein: The acquisition includes: sending a request to the management service producer for information indicating the one or more data slices and / or data contexts that have been previously used and / or can be used for training the model; and Information indicating the one or more data slices and / or data contexts that have been previously used and / or can be used for training the model is received from the management service producer.

4. A device as claimed in any one of the preceding claims, wherein: The request for model training also includes information indicating one or more key data slices and / or data context performance requirements, Wherein, the performance report is based on information indicating the performance requirements of the one or more key data slices and / or data context.

5. A device as claimed in any one of the preceding claims, wherein: The request for model training also includes information indicating one or more overall model performance requirements.

6. A device as claimed in any one of the preceding claims, wherein: The device is used for: sending a request for the performance report to the managed service producer; and receiving the performance report in response.

7. Apparatus as claimed in any preceding claim, wherein: The performance report also includes information indicating the overall performance of the model.

8. An apparatus comprising means for: Receive a request for model training from a management service consumer, where: The request includes an indication of one or more key data slices and / or data contexts to be used in training the model; Performing model training on the one or more key data slices and / or data contexts; as well as Based on the model training, a performance report is sent to the management service consumer, wherein the performance report includes one or more performance metrics of the model on the one or more key data slices and / or data contexts and one or more corresponding performance scores.

9. The device of claim 8, wherein: Executing the model training includes at least one of the following: Modeling the residual between key data slices and / or data context levels and the overall model prediction using one or more residual attention modules; Resample or assign higher weights to key data slices or data context specific training samples; Perform multi-task learning, where key data slices or data context specific tasks are learned with hard or soft parameter sharing; as well as The key data slices and / or data context are mapped to train various smaller machine learning models together with the model.

10. The device according to claim 8 or 9, wherein: The device is used for: In response to performing the model training, acquiring model testing data; filtering the model test data to generate filtered model test data, wherein the filtered model test data includes only data from the one or more key data slices and / or data contexts; and A performance report is generated based on the filtered model test data.

11. The apparatus according to any one of claims 8 to 10, wherein: The device is used for: Information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for training the model is provided to the management service consumer.

12. The device of claim 11, wherein: The device is used for: receiving, from the management service consumer, a request for information indicating one or more data slices and / or data contexts that have been previously used and / or can be used for training the model; as well as Information indicating the one or more data slices and / or data contexts that have been previously used and / or can be used for the model training is sent to the management service consumer.

13. The apparatus of any one of claims 8 to 12, wherein: The device is used for: receiving a request for the performance report from the management service consumer; and sending the performance report in response.

14. Apparatus as claimed in any preceding claim, wherein: The model is an artificial intelligence and / or machine learning model, and wherein the management service producer is an artificial intelligence and / or machine learning training function.

15. Apparatus as claimed in any preceding claim, wherein: The request for model training also includes information indicating one or more key data slices and / or data context performance requirements, Wherein, the performance report is based on information indicating the performance requirements of the one or more key data slices and / or data context.

16. Apparatus as claimed in any preceding claim, wherein: The request for model training also includes information indicating one or more overall model performance requirements.

17. A device as claimed in any preceding claim, wherein: The performance report also includes information indicating the overall performance of the model.

18. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured, with the at least one processor, to cause the apparatus to at least: Send a request for model training to the management service producer, where: The request includes an indication of one or more key data slices and / or data contexts to be used in training the model; as well as Based on the request for model training, a performance report is received from the management service producer, the performance report comprising one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

19. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured, with the at least one processor, to cause the apparatus to at least: Receive a request for model training from a management service consumer, where: The request includes an indication of one or more key data slices and / or data contexts to be used in training the model; Performing model training on the one or more key data slices and / or data contexts; as well as Based on the model training, a performance report is sent to the management service consumer, wherein the performance report includes one or more performance metrics of the model on the one or more key data slices and / or data contexts and one or more corresponding performance scores.

20. A method comprising: sending a request for model training to a management service producer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; and Based on the request for model training, a performance report is received from the management service producer, the performance report comprising one or more performance metrics and one or more corresponding performance scores of the model on the one or more key data slices and / or data contexts.

21. A method comprising: receiving a request for model training from a management service consumer, wherein the request includes an indication of one or more key data slices and / or data contexts to be used in the model training; Performing model training on the one or more key data slices and / or data contexts; and Based on the model training, a performance report is sent to the management service consumer, wherein the performance report includes one or more performance metrics of the model on the one or more key data slices and / or data contexts and one or more corresponding performance scores.

Images