Method, apparatus and computer program

Abstract

An apparatus is disclosed. The apparatus comprises means for receiving a request for data analysis; means for, in response to receiving the request, performing a determination of whether a single machine learning model or a plurality of machine learning models are required to meet the request; a component for generating a machine learning construction based on the determination, the machine learning construction defining an arrangement of one or more machine learning models; and means for causing a machine learning process to be initiated according to the configuration.

Description

Technical Field

[0001] The present application relates to methods, apparatuses, and computer programs, particularly but not exclusively to machine learning within a communication system. Background Art

[0002] A communication system can be regarded as a facility that enables a communication session between two or more entities (such as user terminals, base stations, and / or other nodes) by providing a carrier between various entities involved in a communication path. A communication system can be provided, for example, via a communication network and one or more compatible communication devices. A communication session can include, for example, data communication for carrying communications, such as voice, video, email, text messages, multimedia, and / or content data, etc. Non-limiting examples of the services provided include two-way or multi-way calls, data communication, or multimedia services, as well as access to a data network system (such as the Internet).

[0003] In a wireless communication system, at least a part of a communication session between at least two stations occurs via a wireless link. Examples of wireless systems include public land mobile networks (PLMNs), satellite communication systems, and different wireless local networks, such as wireless local area networks (WLANs). Some wireless systems can be divided into cells and are therefore commonly referred to as cellular systems.

[0004] A user can access a communication system via a suitable communication device or terminal. The user's communication device can be referred to as a user equipment (UE) or user device. The communication device is provided with suitable signal receiving and transmitting means for enabling communication, for example, enabling access to a communication network or direct communication with other users. The communication device can access a carrier provided by a station (such as a base station of a cell) and send and / or receive communications on the carrier.

[0005] Communication systems and associated devices typically operate according to a given standard or specification that defines what the various entities associated with the system are allowed to do and how they should be implemented. The communication protocols and / or parameters that should be used for connections are also typically defined. An example of a communication system is UTRAN (3G radio). Other examples of communication systems include the long-term evolution (LTE) of the universal mobile telecommunications system (UMTS) radio access technology and the so-called 5G or new radio (NR) networks. NR is being standardized by the 3rd Generation Partnership Project (3GPP). Summary of the Invention

[0006] According to a first aspect, an apparatus is disclosed, comprising: components for receiving a request for data analysis; components for performing a determination, in response to receiving the request, as to whether a single machine learning model or multiple machine learning models are needed to satisfy the request; components for generating a machine learning construct based on the determination, the machine learning construct defining an arrangement of one or more machine learning models; and components for causing a machine learning process to be initiated according to the construct.

[0007] According to some examples, causing the machine learning process to be initiated includes: causing training of one or more machine learning models.

[0008] According to some examples, causing training includes: sending a message to a training function to perform joint training of one or more machine learning models.

[0009] According to some examples, the message to the training function includes: an indication of data to be used by the training function when performing the training.

[0010] According to some examples, the indication of data to be used includes one or more of the following: information on data to be extracted; transformation information; information on how the transformed data should be loaded to a consumer; information on the size of the data payload to be used; context information.

[0011] According to some examples, the apparatus includes: components for receiving a response from the training function, the response indicating whether the training function is able to perform some or all of the requested training.

[0012] According to some examples, the apparatus includes: components for modifying the machine learning construct when the training function is able to perform some, but not all, of the requested training.

[0013] According to some examples, causing the machine learning process to be initiated includes: causing an inference process to be performed at one or more machine learning models.

[0014] According to some examples, the apparatus includes: components for, when it is determined that multiple machine learning models are needed, sending the machine learning construct or a subset of the construct to another apparatus such that the other apparatus can assist in satisfying the request.

[0015] According to some examples, the apparatus includes: components for receiving training results from one or more machine learning models.

[0016] According to some examples, the apparatus includes: components for receiving performance-related information from one or more machine learning models.

[0017] According to some examples, the machine learning construct defines one or more of: a machine learning pipeline; one or more machine learning functions.

[0018] According to some examples, the machine learning construct includes an arrangement of machine learning models arranged in one or more parallel and one or more hierarchical levels.

[0019] According to some examples, the machine learning construct includes an arrangement of machine learning models arranged in one or more parallel and one or more hierarchical levels, and the construct provides one or more of the following: information about data to be extracted; transformation information; information on how the transformed data should be loaded into a consumer; information on the size of the data payload to be used; context information for one or more parallel and one or more hierarchical levels.

[0020] According to some examples, the output from a machine learning model in a first hierarchical level forms an input to a machine learning model in a second hierarchical level.

[0021] According to some examples, the apparatus includes one or more of the following: a management data analysis function; an artificial intelligence / machine learning training generator.

[0022] According to some examples, the component includes at least one processor; and at least one memory including computer program code, the at least one memory and the computer program code being configured to cause the apparatus to perform in conjunction with the at least one processor.

[0023] According to a second aspect, there is provided an apparatus including: at least one processor; and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus to at least perform: receiving a request for data analysis; in response to receiving the request, performing a determination as to whether a single machine learning model or multiple machine learning models are needed to satisfy the request; generating, based on the determination, a machine learning construct that defines an arrangement of one or more machine learning models; and causing a machine learning process to be initiated according to the construct.

[0024] According to a third aspect, there is provided a method including: receiving a request for data analysis; in response to receiving the request, performing a determination as to whether a single machine learning model or multiple machine learning models are needed to satisfy the request; generating, based on the determination, a machine learning construct that defines an arrangement of one or more machine learning models; and causing a machine learning process to be initiated according to the construct.

[0025] According to some examples, causing a machine learning process to be initiated includes causing training of one or more machine learning models.

[0026] According to some examples, causing training includes sending a message to a training function to perform joint training of one or more machine learning models.

[0027] According to some examples, a message to a training function includes: an indication of data to be used by the training function when performing training.

[0028] According to some examples, the indication of data to be used includes one or more of the following: information about data to be extracted; transformation information; information on how the transformed data should be loaded to a consumer; information on the size of the data payload to be used; context information.

[0029] According to some examples, the method includes: receiving a response from a training function, the response indicating whether the training function is able to perform some or all of the requested training.

[0030] According to some examples, the training function may perform some, but not all, of the requested training to modify a machine learning construct.

[0031] According to some examples, initiating a machine learning process includes: causing an inference process to be executed at one or more machine learning models.

[0032] According to some examples, when it is determined that multiple machine learning models are needed, sending components of a machine learning construct or a subset of the construct to another device so that the other device can assist in fulfilling the request.

[0033] According to some examples, the method includes: receiving training results from one or more machine learning models.

[0034] According to some examples, the method includes: receiving performance-related information from one or more machine learning models.

[0035] According to some examples, the machine learning construct defines one or more of: a machine learning pipeline; one or more machine learning functions.

[0036] According to some examples, the machine learning construct includes: an arrangement of machine learning models arranged in one or more parallel and one or more hierarchical levels.

[0037] According to some examples, the machine learning construct includes: an arrangement of machine learning models arranged in one or more parallel and one or more hierarchical levels, and the construct provides one or more of the following: information about data to be extracted; transformation information; information on how the transformed data should be loaded to a consumer; information on the size of the data payload to be used; context information for one or more parallel and one or more hierarchical levels.

[0038] According to some examples, the output from a machine learning model in a first level forms the input to a machine learning model in a second level.

[0039] According to some examples, the method is performed by one or more of the following: managing data analysis functions; an artificial intelligence / machine learning training generator.

[0040] According to a fourth aspect, there is provided a computer program comprising instructions for causing an apparatus to perform at least the following: receiving a data analysis request; in response to receiving the request, performing a determination as to whether a single machine learning model or multiple machine learning models are needed to satisfy the request; generating a machine learning configuration based on the determination, the machine learning configuration defining an arrangement of one or more machine learning models; and causing a machine learning process to be initiated according to the configuration.

[0041] According to a fifth aspect, there is provided a computer program comprising instructions stored thereon for performing at least the following: receiving a request for data analysis; in response to receiving the request, performing a determination as to whether a single machine learning model or multiple machine learning models are needed to satisfy the request; generating a machine learning configuration based on the determination, the machine learning configuration defining an arrangement of one or more machine learning models; and causing a machine learning process to be initiated according to the configuration.

[0042] According to a sixth aspect, there is provided a non-transitory computer-readable medium comprising program instructions for causing an apparatus to perform at least the following: receiving a request for data analysis; in response to receiving the request, performing a determination as to whether a single machine learning model or multiple machine learning models are needed to satisfy the request; generating a machine learning configuration based on the determination, the machine learning configuration defining an arrangement of one or more machine learning models; and causing a machine learning process to be initiated according to the configuration.

[0043] According to a seventh aspect, there is provided a non-transitory computer-readable medium comprising program instructions stored thereon for performing at least the following: receiving a request for data analysis; in response to receiving the request, performing a determination as to whether a single machine learning model or multiple machine learning models are needed to satisfy the request; generating a machine learning configuration based on the determination, the machine learning configuration defining an arrangement of one or more machine learning models; and causing a machine learning process to be initiated according to the configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0045] Figure 1 A representation of a system incorporating AI / ML elements according to some example embodiments is schematically shown;

[0046] Figure 2 A representation of an ML configuration is schematically shown;

[0047] Figure 3is a signaling diagram according to an exemplary embodiment;

[0048] Figure 4 is a signaling diagram according to an exemplary embodiment;

[0049] Figure 5 is a signaling diagram according to an exemplary embodiment;

[0050] Figure 6a and 6b is a signaling diagram according to an exemplary embodiment;

[0051] Figure 7 is a signaling diagram according to an exemplary embodiment;

[0052] Figure 8 is a flowchart according to an exemplary embodiment;

[0053] Figure 9 schematically shows a control device according to an exemplary embodiment;

[0054] Figure 10 shows a schematic diagram of a non - volatile memory medium storing instructions which, when executed by a processor, allow the processor to perform one or more steps of the method of some embodiments. Detailed Description

[0055] Artificial intelligence (AI) and machine learning (ML) technologies are being increasingly adopted in 5G systems (5GS) and are regarded as enablers for 6G mobile networks. The NWDAF (Network Data Analytics Function) in 5G Core (5GC) and the MDAF (Management Data Analytics Function) in OAM (Operations, Administration, and Maintenance) bring intelligence and generate analytics by processing management and network data, and may adopt AI and ML technologies.

[0056] Analysis consumers can take actions enforced in the mobile network based on the analysis / prediction / suggestions generated by MDAF / NWDAF. Examples of decisions may include handover of UEs, traffic control, power - on / off of base stations, etc.

[0057] As will be explained in more detail below, according to an example, AI / ML entities can collaborate in different ways. One example is that the output of one AI / ML entity can be used as the input to another AI / ML entity, thus forming a series of interconnected AI / ML entities. Another example is the case where multiple AI / ML entities provide their outputs in parallel (the same output type, where the outputs can be merged (e.g., using weights), or their outputs are required in parallel as inputs in an automated process or as inputs to another AI / ML entity. When building a single supporting ML solution for a given use case, such a modular approach helps to reuse AI / ML entities in different use cases. In addition, it can facilitate the replacement, change, and improvement of individual AI / ML entities within a complex use case.

[0058] Based on the complexity of the use case, a single AI / ML entity or multiple AI / ML entities may be required. For simple use cases, applying only a single AI / ML entity is sufficient. For complex use cases or vendor-specific extensions, multiple AI / ML entities may need to be employed in a potentially collaborative manner, such as sequentially, in parallel, or in any structure that incorporates AI / ML entities. Given the complexity of the required mapping between the use case and the potentially multiple AI / ML entities, services should be supported to facilitate such mapping, e.g., determining whether a particular use case can be implemented by a single AI / ML entity or a group of AI / ML entities, configuring individual AI / ML entities based on their interdependencies within the group, enabling joint training of interdependent AI / ML entities, etc.

[0059] In 3GPP SA5, TS28.105 defines the process for training AI / ML entities (entities with AI / ML capabilities). For the purpose of explanation, Figure 6.6.2.1-1 of TS28.105 is reproduced in Figure 1 Before deploying the AI / ML entity 106 to perform inference (i.e., using the trained model 108 to make predictions), the training of the AI / ML entity 106 needs to be performed. For example, the training can be performed by a separate or external entity related to the inference function, such as the AI / ML Training (AIMLT) MnS generator 102. The training can be triggered by a request from one or more AIMLT MnS consumers 104. Alternatively, the training can be initiated by the AIMLT MnS generator 102 (e.g., as a result of model evaluation).

[0060] This disclosure recognizes that the standardization group and related specifications do not cover the situation where multiple ML models need to be employed to provide the requested analytical predictions or recommendations.

[0061] The analysis, prediction, or advice required by an AIMLT consumer 104 for a particular problem or requirement can be provided by employing a single or multiple ML models, ML pipelines, or ML-enabled functions. Current standardization specifications only address the case where there is a one-to-one mapping between a consumer's (e.g., consumer 104) request and an ML model 108. This approach may be insufficient because it implies that for each possible use case or network problem that a consumer may want to solve using an ML-based solution, a dedicated ML model, ML pipeline, or ML-enabled function would need to be developed. This also limits the reusability of ML models, ML pipelines, or ML-enabled functions that are already available. Therefore, the present disclosure identifies that enabling the combination and reuse of ML models, ML pipelines, or ML-enabled functions would be useful to address a wide variety of network use cases and problems (some even on-demand) without imposing a significant burden on development costs and standardization efforts. Additionally, the present disclosure identifies that an enabler may be needed to select and combine ML models, ML pipelines, or ML-enabled functions from different vendors and ensure their interoperability.

[0062] In the 3GPP SA5 context, TS28.104 defines that the MDA (Management Data Analysis) process can utilize AI / ML technologies. The MDA function can optionally be deployed as one or more AI / ML-enabled functions, where the relevant models are used for inference according to the corresponding MDA capabilities. The inferenceType (see TS28.105) parameter indicates the type of inference supported by the ML model. The value of this parameter may correspond to the MDA type / capability for RAN intelligence (defined in TS28.104) or the analysis ID (defined in TS23.288) inference type, as well as vendor-specific extensions. In cases where there is no one-to-one mapping between the requested analysis (MDA capabilities) and a single ML model, i.e., where multiple models rather than a single model need to be employed to provide the requested analysis, the training of such a model combination needs to be ensured. In such cases, it has not been considered how the MDA function should serve such requests. For example, it has not been considered how MDA should ensure the training of the entire ML combination (multiple models), and the coordination / synchronization between the model output and input, or how the provision of the sequence or parallel / concurrency of ML models should be carried out.

[0063] In the 3GPP SA2 context, an NWDAF service consumer may request one or more analytics IDs, which may mean that instead of a single model being used to process the request, a combination of ML models may be used, where the output of one model is used as input to another model or to ML models that provide output concurrently. This disclosure identifies that in such a case, it is not clear how the NWDAF should serve such a request. For example, it is not clear how the consumer NWDAF should ensure the coordination and / or synchronization between the model output and input, how the sequencing or parallel / concurrency provision of ML models should be performed, and how the interaction between the AnLF and the MTLF should be performed.

[0064] Examples in this disclosure allow a network entity to specify and convey a combination of given sets: one or more ML models; one or more ML pipelines; one or more (possibly different) ML-enabled functions from one or more vendors. The network entity can then train and execute the ML constructs when needed to provide ML-based services. In some examples, the training can be limited to part or all of the ML constructs. Example data attributes used to specify the ML constructs are defined in Table 1 below.

[0065]

[0066]

[0067] Table 1

[0068] In Table 1, the support qualifier "M" means mandatory. The support qualifier CM means "conditionally mandatory". This means that if the condition under which the attribute should exist is met, the attribute should be considered mandatory. For example, in this disclosure, if the construct structure includes an embedded construct under hierarchy / parallel, the ID of such a construct should be provided as mandatory.

[0069] In the examples, the processes added and / or improved by this disclosure may be useful in the context of 3GPP SA5 on MDA or 3GPP SA2 on NWDAF, but are not limited thereto.

[0070] Figure 2An ML system at 200 is shown schematically. The system 200 in this example includes ML constructs 202, 204, 206. A "construct" can be considered to include information about one or more parameters of an ML system (or a part of an ML system). Additionally or alternatively, a "construct" can be considered to include information about the arrangement of ML entities. For example, the "arrangement" may relate to the way the output from one ML entity is used by another ML entity, how entities should be trained, etc. In some examples, the arrangement of an ML system can be defined by a construct configuration file. Thus, in some examples, a construct configuration file can be considered to define one or more parameters and / or entities of an ML system. Figure 2 The principle of a construct built based on "hierarchical" (shown schematically at 208) and "parallel" (shown schematically at 210, 212, and 214) is shown schematically. For example, each level can include functions. Functions can include (by non-limiting example) one or more of the following: mobility prediction; QoS prediction; traffic control (TS); QoE optimization. Parallel can include ML models, ML pipelines, or ML-enabled functions, which themselves can include one or more of the above functions. Thus, each construct can include one or more levels and one or more parallel, each level and / or parallel including a combination of ML functions. For example, looking at construct 206, at level 1, this provides mobility prediction, QoS prediction, and TS, which occur in parallel at level 1 and are fed into QoE optimization at level 2. Thus, overall, construct 206 is designed to provide an ML tool for optimizing QoE. In Figure 2 the example, the output of construct 206 is at level 2, the output of construct 204 is at level 3, and the output of construct 202 is at level 4. In Figure 2 the example, the functions of a series of levels occur in sequence, and parallel functions can occur simultaneously. It will be understood that Figure 2 the architecture of the construct in is by example. The concept of a construct is not limited to a specific number of levels and parallel.

[0071] In the 3GPP SA5 context, based on the current specification TS28.105, the AIMLT MnS consumer 104 (in this example, which can be regarded as a management data analytics (MDA) vendor or service) requests the AIMLT MnS generator 102 to train the AI / ML model 108 or AI / ML-enabled function. In the AI / ML training request, the consumer 104 can specify the inference type. In some examples, the inference type refers to the type of information that needs to be inferred. For example, if information related to network coverage needs to be obtained, the inference type can be set to "CoverageProblemAnalysis", etc. Then, the AIMLT MnS generator 102 can perform the training according to the specified inference type. However, if the specified inference type does not map to a single AI / ML model available at the generator, but multiple interrelated AI / ML models need to be employed, the AIMLT MnS generator 102 may not be able to perform the required training independently. This disclosure introduces an enabler to allow the AIMLT MnS generator 102 (AI / ML training management service, see TS28.105) to handle the situation where multiple interrelated A / ML models need to be trained to address a request from the consumer 104, for example, a request for training a set of ML models that are executed sequentially and / or in parallel / concurrency. In an example, this may include an extension of the AIMLTrainingRequestIOC specified in TS28.105 by including information about the ML construction profile. In an example, the ML construction profile provides information about the ML models that need to be involved, and their interrelationships (such as parallel, sequential execution, etc.). In an example, IOCAIMLTrainingRequest represents an ML model training request created by an AI / ML training MnS consumer.

[0072] In an example, if such a request is considered acceptable by the generator 102, that is, if the generator can fully or partially train the ML construction that meets the interrelationship requirements, the AI / ML training MnS generator 102 decides when and how to start the AI / ML training. If multiple ML models need to be jointly trained (related to each other), the MnS generator 102 can start the training based on the information obtained in the ML construction profile. In an example, the AI / ML training MnS generator 102 instantiates one or more functions or procedures (e.g., (multiple) AI / MLTrainingProcess management object instances ((multiple) MOI)), which are responsible for performing the following:

[0073] - Collect data for training. For example, this can consider the interrelationships between ML models in an ML construct. For example, if the output of a first model is used as input to a second model, the data used for training the second model needs to be collected accordingly.

[0074] - Prepare training data for each ML model in the construct based on the information in the ML construct profile. For example, based on the ETLPipe information included in the ML construct, specific extraction, transformation, and loading of data that is input to a specific ML model, ML pipeline, or ML-enabled function within a given tier and parallelism need to be performed.

[0075] - Train AI / ML entities based on the TrainAnalyticsPipe information from the ML construct. For example, only the TrainAnalyticsPipe for specific ML models, ML pipelines, or ML-enabled functions in parallel at a given tier for which it is set to "yes" need to be trained.

[0076] In the 3GPP SA2 context, it can be considered that the present disclosure introduces an enabler that allows the NWDAF to handle requests for statistical data or predictions by adopting an ML construct, such as a set of ML models executed sequentially and / or in parallel / concurrency. This can include introducing:

[0077] 1. The "ML construct capability" registered by the NWDAF in its NF profile within the NRF. This can be used as an indication that the NWDAF is capable of handling an ML construct by performing:

[0078] a. Determining whether an ML construct is needed for implementing the analysis of a specific use case and request

[0079] b. Determining the profile of the ML construct (see Table 1), i.e.,

[0080] i. Determining which ML models are necessary as building blocks of the ML construct ii. Determining the order in which the ML models that are building blocks of the ML construct are executed, e.g., sequentially and / or in parallel

[0081] c. Based on the defined profile of the ML construct request or performing the training of the ML construct, either jointly training all ML models from the construct or only training a selected set of ML models from the construct

[0082] 2. An extension of the process for providing ML models by introducing information related to the ML model construct, such as information about the order and / or parallelism / concurrency of ML models in the construct, or the performance requirements of the ML construct.

[0083] As described above, the ML construct can be considered to include one or more parameters. Examples of constructs are provided below:

[0084] ("ConstructID": constructID,

[0085] "TrainConstruct": "yes" / "no",

[0086] "Levels": 1..n (

[0087] "Paralells": 1..m (

[0088] <“AnalyticsID or embedded ConstructID”: analyticsID / ConstructID>,

[0089] <“ETLPipe”: Metadata or URLofETLPipeDefinition>,

[0090] <“AnalyticsPipe”: <<Metadata or URLofAnalyticsPipe>>,

[0091] <“TrainAnalyticsPipe”: “yes” / ”no”>,

[0092] “Output”: <Metadata or URLofOutputDefinition> ) ) )

[0096] The "<>" in the above means optional. In some examples, the levels are built from level 1 as the first level and from level n as the output level of the construct. In some examples, the parallel analysis entities in each level are identified by their analysis IDs. In some examples, these parallel entities are built side by side from parallel 1 to parallel m in the level. In some examples, the scheduling between levels is performed in the following form: when all the output data on the current level are ready, start executing the next level, from level i to i + 1. Or, when all the output data of level n are ready, output them as the output of the construct.

[0097] The ML constructs can be defined in computer program code and are readable such that one or more entities or functions in the ML system can read the constructs and understand the one or more functions they need to provide (and / or the manner in which the function should be provided). As a non-limiting example, the constructs can include one or more of the following fields: the ID of the construct; whether the construct requires training; the number of levels; the number of parallels; the type of output data to be provided.

[0098] In some examples, the constructs are provided as attributes. In some examples, the constructs are provided as attributes that can be transmitted between entities. For example, the constructs can be provided as attributes in messages between entities, such as: NWDAF; NWDAF and its consumers: AI / ML training MnS generator and its consumers.

[0099] In the context of 3GPP SA5, the constructs can be part of or an extension of AIMLTrainingRequest.

[0100] In some examples, the training request (e.g., IOC AIMLTrainingRequest) represents a request for training of an ML model created by an MnS consumer. For example, the request can be used by MDAS to support the analysis of certain requests, such as one or more of the following: coverage optimization; quality of experience (QoE) optimization. In cases where the requested analysis can only be provided by adopting ML constructs, a corresponding request for (partial or full) training of such ML constructs may need to be issued by the consumer. Table 2 below shows the extensions proposed in AIMLTrainingRequest, where the new extensions are shown in italics.

[0101]

[0102] Table 2

[0103] In the above table, O = Optional, M = Mandatory. T = True, F = False, CM = Conditionally Mandatory

[0104] Now consider two example options and discuss them in turn below.

[0105] Option 1

[0106] Option 1 is about Figure 3 discussed, Figure 3Shows the communication between a first network entity (e.g., MDA MnS consumer 320), a second network entity (e.g., MDA function 304), and a third network entity (e.g., AIMLT MnS generator 302). The MDA MnS consumer 320 can also be referred to as a consumer network function (NF) and can be in the form of, for example, an OAM function. In some examples, the MDA function 304 can be in the form of an AIMLT MnS consumer 304. In some examples, the (multiple) AIMLT generators 302 can be in the form of or include an AIML training function. It should be understood that there can be multiple "consumers" within the system that can consume (e.g., obtain information) from different services. For Figure 3 example, the MDA MnS consumer 320 ("first consumer") can be a consumer of an analysis service. For example, it analyzes coverage and capacity issues, predicts future network states, and gives some suggestions on the best (re)configuration. Thus, in some examples, the MDA MnS consumer 320 can be regarded as a network operator interested in the optimal configuration of network objects. The MDA function 304 (e.g., AIMLT MnS consumer) can consume an ML training service that performs ML model training. Thus, the MDA function 304 ("second consumer") can be regarded as an MDA service / function that, in order to perform the required analysis, needs to perform training according to an ML model and thus consumes the ML training service from the AIML training generator 302.

[0107] In Option 1, the AIMLT MnS consumer 304 determines the ML construction profile and provides this information to the (multiple) AIMLT MnS generators 302 for training the ML model accordingly.

[0108] At S301, the MDA MnS consumer 320 issues a request for a specific analysis, such as QoE, coverage optimization, etc. In this request, the MDA MnS consumer 320 can specify the MDA type / MDA capabilities as defined in TS28.104. In some examples, the MDA MnS consumer 320 can also specify other consumer-exported values and issue this request to the AIMLT MnS consumer 304.

[0109] At S302, the MDA function 304 determines that it may need to employ one or more ML models to provide the required analysis. Before the corresponding models can provide the required analysis, they may need to be trained. Thus, the MDA function 304 can take the form of or adopt an AIMLT MnS (AI / ML Training Management Service) consumer, as it needs to consume the available training services. The AIMLT MnS consumer 304 is configured to identify which ML models need to be employed and export an ML construction profile. For example, an ML construction profile that describes the models required for the QoE analysis shown in Figure 2 and their interrelationships can take the following form: "ConstructID":QoEconstruct_ID1,

[0110] "TrainConstruct":"yes",

[0111] "Level": 1 (

[0112] "Paralell": 1 (

[0113] <“AnalyticsID or embedded ConstructID”:MobilityPrediction_ID1>,

[0114] <“ETLPipe”:URLofETLMobilityPrediction_ID1>,

[0115] <“AnalyticsPipe”:<<URLofMobilityPrediction_ID1>>,

[0116] <“TrainAnalyticsPipe”:“yes””>,

[0117] “Output”:<URLofOutputMobilityPrediction_ID1> ) )

[0120] “Levels”: 2 (

[0121] “Paralells”: 1 (

[0122] <“AnalyticsID or embedded ConstructID”:QoSPrediction_ID1>,

[0123] <“ETLPipe”:URL of ETL QoSPrediction_ID1>,

[0124] <“AnalyticsPipe”:<<URL of QoSPrediction_ID1>>,

[0125] <“TrainAnalyticsPipe”:“yes”>,

[0126] “Output”:<URL of Output QoSPrediction_ID1> ) )

[0129] “Levels”: 3 (

[0130] “Paralells”: 1 (

[0131] <“AnalyticsID or embedded ConstructID”:TSI_D1>,

[0132] <“ETLPipe”:URL of ETL TSI_D1>,

[0133] <“AnalyticsPipe”:<<URL of TSI_D1>>,

[0134] <“TrainAnalyticsPipe”:“yes”>,

[0135] “Output”:<Metadata or URL of Output TSI_D1> ) )

[0138] “Levels”: 4 (

[0139] “Paralells”: 1 (

[0140] <“AnalyticsID or embedded ConstructID”:QoE_ID1>,

[0141] <“ETLPipe”:URL of ETL QoE_ID1>,

[0142] <“AnalyticsPipe”:<<URL of QoE_ID1>>,

[0143] <“TrainAnalyticsPipe”:“yes”>,

[0144] “Output”:<URLofOutput QoE_ID1> ) )

[0147] In the example, the ML construct provides information about the interrelationships between the ML models in the construct, as well as information about how the output of one model is used as the input of another model. For example, since Figure 2 the ML construct 1 (QoEconstruct_ID1) in

[0148] At S303, the MDA function 304 (e.g., in the form of an AIMLT MnS consumer) issues a training request to the (multiple) AIMLT MnS generators 302. In some examples, the MDA function 304 does this by including the derived ML construct profile determined at S303. It should be noted that since the ML construct profile includes multiple ML models, these models may need to be trained by different AIMLT MnS generators. In some examples, the training request can be considered for the joint training of the models. Thus, for multiple ML models, they can be grouped and then trained together. For example, when trained together, the models can use the same or similar training data, and / or the output of one model can be provided as input data for another model. In the presence of an overall ML model, joint training can also be used, which is constructed as an aggregation of multiple models trained together. The multiple models trained together can be located in parallel and / or sequentially. For example, there can be two ML models that take different inputs and perform prediction or classification tasks (each model having its own loss function). The outputs of these two models can be used as input to a third model, which performs the next or final prediction. These three models can be considered as an overall model together, and they are all trained together during the same training phase. This means that in some examples, the models are not necessarily all trained independently. In some examples, the training request can be considered to include an indication of the data to be used by the training function when performing the training. In some examples, the indication of the data to be used includes one or more of the following: information about the data to be extracted; transformation information; the size of the data payload to be used; a description of the process; context information. For example, "transformation information" can refer to how the data is processed. For example, after the raw data has been extracted from different locations, it may need to be preprocessed. Such preprocessing can include any one or more of the following: filtering or cleaning; removing duplicates; removing invalid samples; formatting the data. For example, "description of the process" can include information on how the transformed data should be loaded into the consumer, such as starting by loading all available data and then only loading incremental data changes (e.g., periodically). For example, "context information" can include information about the state and conditions of the network in which the construct will be run. This can be, for example, information about network objects (such as gNBs) that will be affected by the ML operation. The ML operation can include, for example, a change in configuration parameters, or information about the time, geographical location, or load conditions in which the ML operation will be valid, etc. This can be provided, for example, as expectedRuntimeContext. It should be noted that the context information can refer to one or both of the following: the context in which the ML model is trained; the context in which the ML model will be deployed (e.g., expected to be run during inference).

[0149] At S304, each of the (multiple) AIMLT MnS generators 302 may be able to train the entire ML construct, or only a part of the ML construct (ML sub-profile). Thus, the (multiple) AIMLT MnS generators 302 may give a response providing information about the ability of the generator 302 to train the entire ML construct or a part thereof (sub-construct). In the case where the entire ML construct can be trained by a single generator, such a response may include or only include the ML construct ID (e.g., QoEconstruct_ID1 from the above example). In the case where the generator 302 can only train a sub-construct, at this step, the generator 302 provides information about the sub-construct it can train. The sub-construct information may include information about the levels and / or parallelism that the generator 302 can train (e.g., QoEconstruct_ID1 Level-1, Parallel-1). If the entire ML construct can be trained by a single AIMLT MnS generator, the generator 302 will correspondingly perform the training and provide the result in S306. In this case, S305 will be omitted.

[0150] As described above, S305 is related to the case where no single generator is able to train the complete ML construct as defined in the ML construct profile. In such a case, the AIMLT MnS consumer 304 receives feedback from all the AIMLT MnS generators about their ability to train parts of the ML construct (see S304), and may adjust the training request based on the available training capabilities from the generators. In some examples, this may include splitting the ML construct profile into sub-construct profiles that can be trained at a single generator while preserving the required interrelationships between the ML models.

[0151] At S306, the (multiple) AIMLT MnS generators 302 provide the training result. In some examples, the training result includes the location of the trained ML model. If the training has only been performed on a part of the ML construct (sub-construct), the result may also include additional ML construct-related information, such as the identifier of the sub-construct and performance-related information during training associated with that identifier. For example, "performance-related information" may include performance metrics about the accuracy of the model on its predictions.

[0152] Option 2

[0153] Option 2 is further discussed with respect to Figure 4 and is Figure 4Shows the communication between a first network entity (e.g., MDA MnS consumer 420), a second network entity (e.g., MDA function 404), and a third network entity and a fourth network entity (e.g., first AIMLT MnS generator 402 and second AIMLT MnS generator 403). The MDA MnS consumer 420 can also be referred to as a consumer network function (NF) and can be in the form of, for example, an OAM function. In some examples, the MDA function 404 can be in the form of an AIMLT MnS consumer. In some examples, the (multiple) AIMLT generators 402 and 403 can be in the form of an AIML training function or include an AIML training function. In Option 2, the AIMLT MnS generator 402 determines the ML construction profile.

[0154] At S401, the MDA MnS consumer 420 issues a specific analysis request, such as QoE, coverage optimization, etc., to the MDA function 404. In this request, the MDA MnS consumer 420 can specify the MDA type / MDA capabilities as defined in TS28.104, but it can also specify other consumer-derived values.

[0155] Similar to Option 1, at S402, the MDA function 404 may need to employ one or more ML models to provide the required analysis. Before being able to provide the required analysis, the corresponding model needs to be trained. Thus, the MDA function 404 can become or include an AIMLT MnS (AI / ML training management service) consumer as it needs to consume the available training services. Then, the MDA function 404 (e.g., AIMLT MnS consumer) issues a training request to the AIMLT generator.

[0156] At S403, the first AIMLT MnS generator 402 determines whether a single or multiple ML models are needed to address the request. For example, if the AIMLT MnS generator can train a single ML model that matches the required analysis, then no ML construction is needed and the single model will be trained and provided to the consumer. Otherwise, the AIMLT MnS generator 402 can reject the training request or infer that multiple ML models available at the single or multiple AIMLT MnS generators should be used. In the case of multiple ML models, the first AIMLT MnS generator 402 derives the corresponding ML construction profile.

[0157] At S404, if all ML models in the construct can be trained by a single generator (e.g., generator 1402), the training is performed by a single AIMLT MnS generator (e.g., generator 402), and the result is provided to the consumer 404. In some examples, the result is provided as part of the ML construct configuration file, i.e., within the "output" information unit. The output information unit may represent the definition / metadata or URL of the output data for accessing the ML construct instance.

[0158] At S405, if all ML models are not available on a single generator, but only a subset of the required ML models can be trained, i.e., a sub-construct, generator 1402 can create a new ML construct configuration file from the unavailable ML models and issue a training request to other AIMLT MnS generators (e.g., generator 2403) to train the remaining part of the initially requested ML construct. That is, generator 1402 can be considered to act as a consumer of the AIMLT MnS service. For example, generator 1402 may only be able to train the following part of QoEconstruct_ID1:

[0159] “Level”: 1 (

[0160] “Paralell”: 1 (

[0161] <“AnalyticsID or embedded ConstructID”:MobilityPrediction_ID1>,

[0162] <“ETLPipe”:URLofETLMobilityPrediction_ID1>,

[0163] <“AnalyticsPipe”:<<URLofMobilityPrediction_ID1>>,

[0164] <“TrainAnalyticsPipe”:“yes””>,

[0165] “Output”:<URLofOutputMobilityPrediction_ID1> ) )

[0168] “Levels”: 2 (

[0169] “Paralells”: 1 (

[0170] <“AnalyticsID or embedded ConstructID”:QoSPrediction_ID1>,

[0171] <“ETLPipe”:URLofETL QoSPrediction_ID1>,

[0172] <“AnalyticsPipe”:<<URLof QoSPrediction_ID1>>,

[0173] <“TrainAnalyticsPipe”:“yes”>,

[0174] “Output”:<URLofOutput QoSPrediction_ID1> ) )

[0177] Therefore, generator 1402 can define a new ML construct profile (sub-construct) from the remaining levels that are not available at generator 1402. Then, generator 1402 can issue a training request for such a sub-construct, such as:

[0178] “ConstructID”:QoEsub_construct_ID1,

[0179] “TrainConstruct”:“yes”,

[0180] “Levels”: 1 (

[0181] “Paralells”: 1 (

[0182] <“AnalyticsID or embedded ConstructID”:TSI_D1>,

[0183] <“ETLPipe”:URLofETL TSI_D1>,

[0184] <“AnalyticsPipe”:<<URLof TSI_D1>>,

[0185] <“TrainAnalyticsPipe”:“yes”>,

[0186] “Output”:<Metadata or URLofOutput TSI_D1> ) )

[0189] “Levels”: 2 (

[0190] “Paralells”: 1 (

[0191] <“AnalyticsID or embedded ConstructID”:QoE_ID1>,

[0192] <“ETLPipe”:URLofETL QoE_ID1>,

[0193] <“AnalyticsPipe”:<<URLof QoE_ID1>>,

[0194] <“TrainAnalyticsPipe”:“yes”>,

[0195] “Output”:<URLofOutput QoE_ID1> ) )

[0198] S406 to S408 are the same as S304 to S306 in Option 1, where the (multiple) AIMLT MnS generators (e.g., generator 2 403) provide feedback on the capabilities of the sub-constructs of the training request. After receiving all relevant feedback from the generators, generator 1 402 can adjust its request accordingly.

[0199] At S409, generator 1 402 can combine the inputs received from different generators (e.g., generator 403) regarding the training results.

[0200] Then, generator 1 410 can provide the combined information to the AIMLT MnS consumer 404, as shown in S410.

[0201] It should be noted that for both Option 1 and Option 2, if needed, information about the ML constructs can be additionally provided to the consumer network functions (e.g., OAM, AF, etc.).

[0202] In the context of 3GPP SA2, the present disclosure contemplates the following options:

[0203] Option 1: AnLF with ML construct capabilities (ML construct training at a single MTLF)

[0204] Option 1a: AnLF with ML construct capabilities (ML construct training at multiple MTLFs)

[0205] Option 2: MLTLF with ML construction capabilities (ML construction training at a single MTLF)

[0206] Option 2a: MLTLF with ML construction capabilities (ML construction training at multiple MTLFs)

[0207] Options 1, 1a, and 2 are accordingly shown in Figures 5 to 7 the message sequence diagram of and are described in more detail below. Option 2a is not shown separately as the process described for Option 2 (single MTLF) applies.

[0208] Option 1

[0209] See Figure 5 , which shows a signaling diagram of the communication between the consumer network function 550, the NWDAF containing AnLF 552, and the NWDAF containing MTLF 554.

[0210] At S501, the consumer NF 550 (e.g., OAM or AF) requests certain analytics. This can be done by providing an analytics ID (e.g., "QoE optimization") and a set of additional parameters (e.g., S-NSSAI, NF instance ID, area of interest to which the analytics should be applied, preferred accuracy level of the analytics, etc.).

[0211] At S502, the NWDAF 552 identifies the need for ML construction. For example, this need can be identified because it is determined that there is no MTLF that can train a single model or has a single model trained corresponding to the requested analytics ID. The NWDAF 552 defines a configuration file for the ML construction required to solve the requested analytics ID.

[0212] At S503, the ML model configuration process is performed by sending a configuration subscription message of the NWDAF containing AnLF 552 to the NWDAF containing MTLF 554. In some examples, this process is extended with additional information contained in the ML construction configuration file, e.g., which models need to be executed in sequence and which models are parallel. Additionally, additional parameters can be provided. Optionally, a preferred level of certain performance metrics of the ML construction (e.g., accuracy, F1 score, AUC, R2, etc.) can be indicated.

[0213] At S504, the NWDAF including MTLF 554 performs training of the model upon request, as described in the ML construction configuration file. Such training can be performed on all ML models from the construction or only on a selected set of ML models. After receiving the ML construction configuration file, the NWDAF including MTLF 554 constructs the ML construction architecture and trains the ML construction, "freezing" the ML models (if any) that must be kept unchanged.

[0214] At S505, the NWDAF including MTLF 554 evaluates the performance of the ML construction during training.

[0215] At S506, the NWDAF including MTLF 554 sends a notification message to the NWDAF including AnLF 552. In some examples, this message includes the Nnwdaf_MLModelProvision_Notify message, and the Nnwdaf_MLModelProvision_Notify message includes the ML construction ID of the successfully trained ML construction configuration file (which meets the preferred performance metric level from the request) and information about the ML construction performance metric.

[0216] Option 1a

[0217] Reference Figure 6a and 6b ( Figure 6b is Figure 6a (continued from), a signaling diagram showing the communication between the consumer NF 650, the NWDAF including AnLF 652, the NWDAF including MTLF1 654, the NWDAF including MTLF2 656, the NWDAF including MTLF3 658, and the Data Collection Coordination Function (DCCF) 660 is shown in the figure.

[0218] At S601, the consumer NF 650 (e.g., OAM or AF) requests certain analysis in a request message from the NWDAF including AnLF 652. For example, this is done by providing an analysis ID (e.g., "QoE optimization") and a set of additional parameters (e.g., S-NSSAI, NF instance ID, the area of interest to which the analysis should be applied, the preferred accuracy level of the analysis, etc.).

[0219] At S602, the NWDAF with AnLF 652 identifies the need for ML construction (i.e., there is no MTLF that can train a single model or has trained a single model corresponding to the requested analysis ID), and defines the analysis ID as part of the ML construction and information about the order / parallelism of the models from the construction (e.g., which ML models should be parallel and which models should be sequential). Additionally, the NWDAF with AnLF 652 may need to split the derived configuration file into sub-constructions (i.e., as constructions within a construction), which should be trained at different MTLFs based on the availability of ML models at different MTLFs. Optionally, different ML construction configuration file options can be derived by the NWDAF using AnLF 652 and requests for training can be made.

[0220] At S603, S604, S605, the NWDAF containing AnLF 652 contacts the responsible NWDAF containing the MTLF (654, 656, 658), providing information about the relevant ML (sub)-construction to be trained. The MTLFs 654, 656, 658 may need to be coordinated to train the ML models as defined in the ML construction profile. It should be noted that according to TS23.288, ML model configuration / sharing between multiple MTLFs is not supported in Release 17. This means that the NWDAF with AnLF 652 needs to perform the coordination of model provision between multiple MTLFs. There are two options to perform this coordination:

[0221] Option A: NWDAF (AnLF) coordinates using the DCCF (Data Collection and Coordination Function) mechanism (S7 to S9) 。

[0222] At S606, the NWDAF (AnLF) 652 requests data coordination by invoking the Ndccf_DataManagement_Subscribe service operation (data specification - defines the data to be collected as input to other MTLFs, e.g., analysis ID, filter information, etc., time window - start / stop for data collection, formatting instruction - determines when the notification is sent to the MTLF, e.g., event-based notification, processing instruction - parameter name, parameter value, and attributes to be reported to the MTLF, ADRF - whether the output of the MTLF should be stored in the ADRF), as described in clause 8.2.2 of TS23.288. In the example, the NWDAF 652 can specify the involved MTLFs (e.g., 654, 656, 658) as the notification endpoints for receiving the data. In the example, the MTLF is notified when the input (output of another model or gradient) of the MTLF is available so that it can perform the forward / backward phase.

[0223] At S607, S608, and S609, the DCCF 660 sends data to all MTLF notification endpoints (e.g., 654, 656, 658) as requested in S606.

[0224] Option B: NWDAF (AnLF) directly transmits coordination instructions to the involved MTLF

[0225] At S610, the NWDAF with AnLF 652 can notify the NWDAF with MTLF 654, 656, 658 of the order of training of different profile parts, i.e., the order of training for constructing the ML model. For this purpose, the NWDAF 652 can use the ML model target period as defined in TS23.288. This parameter indicates the time interval [start, end] of the ML model for analysis that is requested. The NWDAF with AnLF 652 can use this parameter to indicate to each MTLF 654, 656, 658 for each ML model from the associated sub-construction when the training should start, when the training should be completed, etc. If different ML models (as a single entity) need to be trained together, the AnLF 652 sets the environment in such a way that each MTLF 654, 656, 658 knows where to write its output (during the forward phase), which will be used as input by other MTLFs, and where to wait for the corresponding gradients from the MTLF that uses the previous output as input (which may be needed to perform the backward phase). Thus, in some examples, the AnLF 652 can notify the MTLF 654, 656, 658 of the URL or entity, e.g., ADRF, where the required data is written / expected. Once new data is available, the notification can be sent (e.g., via the DCCFDataManagement_Notify service operation) to the corresponding MTLF, and the corresponding MTLF can perform the corresponding forward / backward steps in this way. In cases where different ML models can be trained independently, the AnLF can wait for the ML model to be trained and then run it to provide the required input to the next MTLF, e.g., by leveraging the ADRF and / or DCCF.

[0226] Then, for Option A and Option B, the method continues at S611.

[0227] At S611, S612, and S613, training of the requested models is performed as described in the ML (sub)-construct configuration file. Optionally, MTLF 654, 656, 658 may notify AnLF 652 to perform inference on one or more trained models or models that do not require training and store the inference output into the ADRF (Analytical Data Repository Function) such that the data can be collected and used as input for training of additional models, e.g., by using the DCCF service or directly providing the input to the MTLF that requires it. The performance during training may also be evaluated. As described above, in cases where different ML models need to be trained together, MTLF 654, 656, 658 follow the information received by AnLF 652 and wait for the required data (output of the ML models in the forward step, and corresponding gradients during the backward phase) to train the managed ML models.

[0228] At S614, S615, and S616, MTLF 654, 656, and 658 provide AnLF 652 with information about the ML models used for training the associated ML sub-construct configuration file, including information about the performance of the ML models.

[0229] At S617, during inference, AnLF 652 applies the sequence / parallelism of the ML models identified from the ML construct configuration file. Optionally, if different options for the ML construct configuration file have been exported in S602, the NWDAF may evaluate which ML construct to apply for inference based on the performance metrics reported in S614, S615, S616.

[0230] Option 2 MTLF with ML construction capabilities (ML construction training at a single MTLF)

[0231] is shown in Figure 7 .

[0232] Reference Figure 7 , shows a signaling diagram of the communication between the consumer network function 750, the NWDAF including AnLF 752, and the NWDAF including MTLF 754.

[0233] At S701, the consumer NF 750 (e.g., OAM or AF) requests certain analysis by providing the analysis ID (e.g., "QoE optimization") and a set of additional parameters (e.g., S-NSSAI, NF instance ID, area of interest to which the analysis should apply, preferred accuracy level of the analysis, etc.) to the NWDAF AnLF 752.

[0234] At S702, the NWDAF AnLF 752 determines that there is no single matching model based on the analysis ID information that can be trained by the MTLF. This means that an ML construction may need to be employed (if available), and the MTLF with the corresponding capabilities needs to be involved.

[0235] At S703, the NWDAF AnLF 752 requests a model provision from the MTLF 754 with ML construction capabilities according to a request from the consumer.

[0236] At S704, the NWDAF MTLF 754 identifies whether the ML construction is feasible based on the MTLF capabilities (e.g., a list of analysis IDs that can be trained by this MTLF). In some examples, the NWDAF containing the MTLF 754 can provide the NRF with the analysis ID(s) corresponding to the trained ML model(s), and may provide the ML model filter information (if available) for the trained ML model(s) for each (multiple) analysis ID. Based on the available list of analysis IDs and corresponding models, the MTLF 754 can derive an ML construction profile. Optionally, the MTLF 754 can derive different options for the ML construction profile related to the analysis ID as requested by the consumer. This means identifying different options for implementing the requested analysis ID in terms of different combinations of ML models that are building blocks of the ML construction along with information about the order and / or parallel execution of such models.

[0237] At S705, based on the derived ML construction profile, the MTLF 754 can perform training. Such training can be performed on all ML models from the construction or only on a selected set of ML models. In cases where there are different levels in the ML construction, the output of one level can be used as the training input for another level. The process continues until all levels in the ML construction are trained.

[0238] At S706, the MLTF 754 can perform an evaluation of these different combinations for the ML construction in terms of its performance and select the best performing combination that meets the required levels of certain performance metrics.

[0239] At S707, the MTLF 754 provides the NWDAF AnLF with the ML construction profile of the ML construction that meets the requirements of the request from the consumer, such as the file addresses of the models that are used as building blocks of the ML construction. If available, additional information can be provided, such as information about the ML construction performance metrics.

[0240] In this disclosure, the training of a model is referred to. The training of the model includes providing training data to the model so that the model can be trained. For example, if the model is to be used to predict QoS, the training data provided to the model may include information on past QoS performance data. If the model is to be used to predict UE mobility, past data on UE mobility will be provided to train the model. If the model is to be used to predict quality of experience (QoE), past data on QoE will be provided to train the model. If the model is to be used to predict traffic steering (TS), past data on TS will be provided to train the model. Of course, it can be understood that these are non-limiting examples, and in practice, the type of training data provided to a particular model to be trained will depend on the context in which the model will be used for subsequent inference deployment (prediction). The "context" may include the type of problem to be used and / or the use case.

[0241] As described in the above example, the training of the model can be initiated by a training request (e.g., see S303 in Figure 3 and S402 in Figure 4 etc.). In some examples, the training request is sent as an IOC AIMLTrainingRequest. In the example, AIMLTrainingRequest represents an AI / ML model training request created by an AI / ML training MnS consumer. To support the joint training of a group of AI / ML entities, this IOC can capture information about the group of AI / ML entities and the relationships between the AI / ML entities, such as the AIMLEntityGroupProfile data type. In some examples, such a data type may contain one or more of the following attributes:

[0242] · AIMLEntityGroup ID - A unique identity value that identifies an AIMLEntity group instance.

[0243] · JointTrainingIndicator - Which indicates whether the AIMLEntity group instance needs to be trained (perform joint training of the AIMLEntities in the group)

[0244] · Levels - An integer range (1, n), which indicates that the AIMLEntity group has n levels, where the integer within the range indicates a specific level, starting from 1. Each level will include AIMLEntities. The output data of one level is used as the input data for the next level.

[0245] ·Parallels - An integer range (1, m) that indicates a series of parallel AIML Entities within a given level, starting from 1. The output data of the parallel AIML Entities within a given level is used as the input data for the next level.

[0246] ·expectedRunTimeContext - This may include information related to specific extraction, transformation, and data loading as input to a specific AIML Entity within a given level and parallel.

[0247] ·AI / MLEntitiyID - Within a given level and parallel

[0248] In some examples, if multiple ML models need to be co - trained (related to each other), the MnS generator can start training based on the information obtained from the AIML Entity Group Profile. In an example, the AI / ML training MnS generator instantiates multiple AI / ML Training Process MOIs that are responsible for performing one or more of the following:

[0249] ·Collect data for training, taking into account the inter - relationships between AIML Entities in the AIML Entity Group Profile. For example, if the output of the first AIML Entity is used as the input to the second AIML Entity, the data for training the second model may need to be collected accordingly.

[0250] ·Prepare training data for each AIML Entity in the group based on the information in the AIML Entity Group Profile. That is, based on the expectedRunTimeContext included in the AIML Entity Group Profile, specific extraction, transformation, and data loading as input to a specific AIML Entity within a given level and parallel may need to be performed.

[0251] ·Train AI / ML entities based on the JointTrainingIndicator

[0252] From the above, it will be understood that the present disclosure may help solve complex use cases where it may be necessary to apply multiple collaborative AI / ML entities. To implement the example in practice, some system requirements may be needed. These system requirements may include one or more of the following:

[0253] · The 3GPP management system shall have the ability to enable authorized consumers to request existing AI / ML entities available within a vendor for an AI / ML MnS generator to perform AI / ML inference.

[0254] · The AI / ML MnS generator shall have the ability to report to an authorized consumer the AI / ML Entity as a decision described by a triple <(multiple) object, parameter, metric>, where the entries indicate accordingly: the object or object type for which the AI / ML Entity can perform optimization or control; the configuration parameters regarding the object or object type that the ML app optimizes or controls to achieve the desired result; and the network metric that the ML app optimizes through its actions.

[0255] · The AI / ML MnS generator shall have the ability to report to an authorized consumer the AI / ML Entity as an analysis described as a tuple <(multiple) object, characteristic>, where the entries indicate accordingly: the object or object type for which the ML app can perform analysis; and the network characteristic (related to the object or object type) for which the ML app generates the analysis.

[0256] · The 3GPP management system shall have the ability to enable authorized consumers to request a set of AI / ML entities that work together to solve a specific use case for training.

[0257] · The 3GPP management system shall have the ability to enable authorized consumers to manage and configure multiple training processes, such as starting, pausing, or restarting training; or adjusting training conditions and / or characteristics based on a group of AI / ML entities.

[0258] · The AIMLT MnS generator shall have the ability to provide consumers with the training results of a group of AI / ML entities (including the locations of the trained AI / ML entities in the group).

[0259] Figure 8 is a flowchart of a method according to an example. Figure 8 The flowchart is from the perspective of a device. For example, the device may include a Management Data Analytics (MDA) function or an Artificial Intelligence / Machine Learning Training (AIMLT) generator.

[0260] As shown at S801, the method includes: receiving a request for data analysis.

[0261] At S802, in response to receiving the request, the method includes: determining whether a single machine learning model or multiple machine learning models are needed to satisfy the request.

[0262] At S803, the method includes: generating a machine learning architecture based on a determination, the machine learning architecture defining an arrangement of one or more machine learning models.

[0263] At S804, the method includes: causing a machine learning process to be initiated according to the architecture.

[0264] Figure 9 An example of a control device 900 for controlling the functions of the present disclosure is shown. The control device may include at least one random access memory (RAM) 911a, at least one read only memory (ROM) 911b, at least one processor 912, 913, and an input / output interface 914. The at least one processor 912, 913 may be coupled to the RAM 911a and the ROM 911b. The at least one processor 912, 913 may be configured to execute appropriate software code 915. The software code 915 may, for example, allow the device to execute one or more steps to perform one or more aspects of the present aspect. The software code 915 may be stored in the ROM 911b. The control device 900 may be interconnected with another control device 900 that controls another function of the radio access network. In some embodiments, each function of the radio access network includes a control device 900.

[0265] Figure 10 A schematic diagram of non-volatile memory media 1000a (such as a computer optical disc (CD) or digital versatile disc (DVD)) and 1000b (such as a universal serial bus (USB) memory stick) is shown, storing instructions and / or parameters 1002, the instructions and / or parameters 1002 which, when executed by a processor, allow the processor to execute Figure 8 one or more of the steps of the method.

[0266] The processor, storage, and other related control devices may be provided on a suitable circuit board and / or chipset. This feature is represented by reference 304. The device may optionally have a user interface, such as a keyboard 305, a touch screen or touchpad, a combination thereof, etc. Optionally, one or more of a display, a speaker, and a microphone may be provided depending on the type of the device.

[0267] It should be understood that these devices may include or be coupled to other units or modules etc. used in or for transmission and / or reception, such as radio components or radio heads. Although these devices have been described as one entity, different modules and memories may be implemented in one or more physical or logical entities.

[0268] It should be noted that although some embodiments have been described with respect to 5G networks, similar principles can be applied to other networks and communication systems. Thus, although certain embodiments have been described above by way of example with reference to certain exemplary architectures for wireless networks, technologies, and standards, the embodiments can be applied to any other suitable form of communication system other than the communication systems shown and described herein.

[0269] It should also be noted herein that although example embodiments have been described above, various changes and modifications can be made to the disclosed solutions without departing from the scope of the present invention.

[0270] As used herein, “at least one of the following: <list of two or more elements>” and “at least one of <list of two or more elements>” and similar phrases, where the list of two or more elements is joined by “and” or “or”, means at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0271] In general, the various embodiments can be implemented in hardware or a special circuit system, software, logic, or any combination thereof. Certain aspects of the present disclosure can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, but the present disclosure is not limited thereto. Although the various aspects of the present disclosure can be illustrated and described in terms of block diagrams, flowcharts, or other graphical representations, it is fully understood, by way of non-limiting example, that the blocks, devices, systems, technologies, or methods described herein can be implemented in hardware, software, firmware, special circuits or logic, general hardware or a controller or other computing device, or some combination thereof.

[0272] As used in this application, the term “circuitry” may refer to one or more or all of the following:

[0273] (a) only hardware circuit implementations (such as, only implementations in analog and / or digital circuitry) and

[0274] (b) combinations of hardware circuits and software, such as, as applicable:

[0275] (i) combinations of (one or more) analog and / or digital hardware circuits and software / firmware and

[0276] (ii) any portions of (one or more) hardware processors with software (including (one or more) digital signal processors), software, and (one or more) memories, which work together to cause a device (such as a mobile phone or a server) to perform various functions and

[0277] (c) One or more hardware circuits and / or one or more processors (such as one or more microprocessors or a part of one or more microprocessors) that require software (such as firmware) to operate, but the software may not exist when software operation is not required.

[0278] This definition of circuitry applies to all uses of the term in this application, including in any claims. As a further example, as used in this application, the term "circuitry" also encompasses implementations of only hardware circuits or processors (or multiple processors) or a part of a hardware circuit or processor and their (or their) accompanying software and / or firmware. The term "circuitry" also encompasses, for example, if applicable to a particular claim element, a baseband integrated circuit or a processor integrated circuit for a mobile device or a similar integrated circuit in a server, a cellular network device, or other computing or network devices.

[0279] Embodiments of the present disclosure can be implemented by computer software executable by a data processor of a mobile device (such as in a processor entity), or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) include software routines, applets, and / or macros, can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product can include one or more computer-executable components, and one or more computer-executable components are configured to perform embodiments when the program is run. One or more computer-executable components can be at least one software code or a part thereof.

[0280] In addition, in this regard, it should be noted that any block of the logical flow in the figures can represent a program step, or interconnected logical circuits, blocks, and functions, or a combination of program steps and logical circuits, blocks, and functions. Software can be stored on physical media such as memory chips or memory blocks implemented within a processor, magnetic media (such as hard disks or floppy disks), and optical media (such as, for example, DVDs and their data variant CDs). The physical media are non-transitory media.

[0281] As used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, rather than a signal), rather than a limitation on the persistence of data storage (e.g., RAM versus ROM).

[0282] The memory can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor can be of any type suitable for the local technical environment and, by way of non-limiting example, can include one or more of the following: general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), gate-level circuits, and processors based on multi-core processor architectures.

[0283] Embodiments of the present disclosure can be practiced in various components, such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Sophisticated and powerful software tools can be used to transform a logic-level design into a semiconductor circuit design that is ready to be etched and formed on a semiconductor substrate.

[0284] The scope of protection sought by the various embodiments of the present disclosure is defined by the independent claims. Embodiments and features described in this specification that do not fall within the scope of the independent claims, if any, should be construed as examples that help in understanding the various embodiments of the present disclosure.

[0285] The foregoing description has provided a complete and informative description of the exemplary embodiments of the present disclosure by way of non-limiting examples. However, various modifications and adaptations may become apparent to those skilled in the relevant art in view of the foregoing description when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of the present disclosure will still fall within the scope of the invention as defined in the appended claims. In fact, there are additional embodiments that include combinations of one or more embodiments with any other previously discussed embodiments.

Claims

1. An apparatus, comprising: means for receiving a request for data analysis; means for determining, in response to receiving the request, whether a single machine learning model or multiple machine learning models are needed to satisfy the request; means for generating a machine learning configuration based on the determination, the machine learning configuration defining an arrangement of one or more machine learning models; and means for causing a machine learning process to be initiated according to the configuration.

2. The apparatus according to claim 1, wherein causing the machine learning process to be initiated comprises: causing training of the one or more machine learning models.

3. The apparatus according to claim 2, wherein causing the training comprises: sending a message to a training function to perform joint training of the one or more machine learning models.

4. The apparatus according to claim 3, wherein the message to the training function comprises: an indication of data to be used by the training function when performing the training.

5. The apparatus according to claim 4, wherein the indication of data to be used comprises one or more of the following: information on data to be extracted; transformation information; information on how the transformed data should be loaded into a consumer; information on the size of the data payload to be used; context information.

6. The apparatus according to any one of claims 3 to 5, comprising: means for receiving a response from the training function, the response indicating whether the training function is able to perform some or all of the requested training.

7. The apparatus according to claim 6, comprising: means for modifying the machine learning configuration when the training function is able to perform some, but not all, of the requested training.

8. The apparatus according to any one of claims 1 to 7, wherein causing the machine learning process to be initiated comprises: causing an inference process to be performed at the one or more machine learning models.

9. The apparatus according to any one of claims 1 to 8, comprising: means for: when it is determined that multiple machine learning models are needed, sending the machine learning configuration or a subset of the configuration to another apparatus such that the other apparatus can assist in satisfying the request.

10. The apparatus according to any of the preceding claims, comprising: means for receiving training results from the one or more machine learning models.

11. The apparatus according to any of the preceding claims, comprising: means for receiving performance-related information from the one or more machine learning models.

12. The apparatus according to any of the preceding claims, wherein the machine learning configuration defines one or more of: a machine learning pipeline; one or more machine learning functions.

13. The apparatus according to any of the preceding claims, wherein the machine learning configuration comprises: an arrangement of machine learning models arranged in one or more parallel and one or more hierarchical arrangements.

14. The apparatus according to claim 5, wherein the machine learning configuration comprises: An arrangement of machine learning models arranged in one or more parallel and one or more hierarchical levels, and the construction provides one or more of the following: information about the data to be extracted; Transformation information; Information on how the transformed data should be loaded into the consumer; Information on the size of the data payload to be used; Context information for the one or more parallel and one or more hierarchical levels.

15. The apparatus according to claim 13 or claim 14, wherein the output from the machine learning model in the first level forms the input to the machine learning model in the second level.

16. The apparatus according to any one of claims 1 to 15, wherein the apparatus comprises one or more of the following: a management data analysis function; an artificial intelligence / machine learning training generator.

17. A method, comprising: receiving a request for data analysis; in response to receiving the request, performing a determination as to whether a single machine learning model or multiple machine learning models are needed to satisfy the request; generating a machine learning construction based on the determination, the machine learning construction defining an arrangement of one or more machine learning models; and initiating a machine learning process according to the construction.

18. The method according to claim 17, wherein the initiating of the machine learning process comprises: causing training of the one or more machine learning models.

19. The method according to claim 18, wherein the causing of the training comprises: sending a message to a training function to perform joint training of the one or more machine learning models.

20. The method according to claim 19, wherein the message to the training function comprises: an indication of the data to be used by the training function when performing the training.

21. The method according to claim 20, wherein the indication of the data to be used comprises one or more of the following: information about the data to be extracted; transformation information; information on how the transformed data should be loaded into the consumer; information on the size of the data payload to be used; context information.

22. The method according to any one of claims 19 to 21, comprising: receiving a response from the training function, the response indicating whether the training function can perform some or all of the requested training.

23. The method according to claim 22, comprising: modifying the machine learning construction when the training function can perform some, but not all, of the requested training.

24. The method according to any one of claims 17 to 23, wherein the initiating of the machine learning process comprises: causing an inference process to be performed at the one or more machine learning models.

25. The method according to any one of claims 17 to 24, comprising: when it is determined that multiple machine learning models are needed, sending the machine learning construction or a subset of the construction to another device so that the other device can assist in satisfying the request.

26. The method according to any one of claims 17 to 25, comprising: receiving training results from the one or more machine learning models.

27. The method according to any one of claims 17 to 25, comprising: receiving performance-related information from the one or more machine learning models.

28. The method according to any one of claims 17 to 27, wherein the machine learning construct defines one or more: machine learning pipelines; one or more machine learning functions.

29. The method according to any one of claims 17 to 28, wherein the machine learning construct comprises: an arrangement of machine learning models arranged in one or more parallel and one or more hierarchical levels.

30. The method according to claim 21, wherein the machine learning construct comprises: an arrangement of machine learning models arranged in one or more parallel and one or more hierarchical levels, and the construct provides one or more of the following: information on data to be extracted; transformation information; information on how the transformed data should be loaded into a consumer; information on the size of the data payload to be used; context information for the one or more parallel and one or more hierarchical levels.

31. The method according to claim 29 or claim 30, wherein the output from a machine learning model in a first level forms an input to a machine learning model in a second level.

32. The method according to any one of claims 17 to 31, performed by one or more of the following: a management data analysis function; an artificial intelligence / machine learning training generator.

33. A computer program comprising instructions for causing a device to perform at least the following: receiving a request for data analysis; in response to receiving the request, performing a determination as to whether a single machine learning model or multiple machine learning models are needed to satisfy the request; generating, based on the determination, a machine learning construct that defines an arrangement of one or more machine learning models; and initiating a machine learning process according to the construct.

Images