Collaborative distributed learning for a telecommunications core network

Abstract

A computer-implemented method is provided performed by a computing device for collaborative distributed training of a ML model in a telecommunications core network The method includes associating (1308) different features with a data sample in an aligned order for respective participants in the telecommunication core network where the participants respectively include a respective collaborative learning active participant model. The method further includes performing (1314) a training process including at least one of (i) accessing a respective output of the respective collaborative learning active participant models trained with the different features and concatenating the respective outputs into a concatenated input to the ML model, and (ii) provisioning a collaborative learning passive participant model to a ML model consumer for inference using the concatenated input provided by the respective participants.

Description

COLLABORATIVE DISTRIBUTED LEARNING FOR A TELECOMMUNICATIONS CORE NETWORKTECHNICAL FIELD

[0001] The present disclosure is related to wireless communication systems and more particularly to collaborative distributed learning of a machine learning (ML) model in a telecommunications network.BACKGROUND

[0002] A fifth generation (5G) core (5GC) includes operational elements referred to as network functions (NFs). Each NF may have a specific role and responsibility in the network such as policy control function (PCF), session management function (SMF), network data analytics function (NWDAF), etc. Consequently, each NF may monitor different aspects of the network which, as a consequence, creates a different (or heterogeneous) feature space.

[0003] NWDAF in 5GC is responsible for generating analytics reports upon received requests from other NFs or operations, administration and maintenance (CAM), which can either be statistics from the past or predictions for the future for different types of events referred to as analytics ids. To be able to generate prediction, machine learning (ML) models for analytics ids can be trained and used for inference by NWDAF. Thus, NWDAF leverages two internal logical functions (LFs) often referred to as a model training logical function (MTLF) and an analytics logical function (AnLF).

[0004] Inside NWDAF, MTLF has a role to train ML model(s) for each analytics id and AnLF is a consumer of the trained ML models to perform inference and generate predictions. A horizontal federated learning (HFL) scheme is standardized (see e.g., Third Generation Partnership Project (3GPP) TS 23.288 v18.3.0, "3rdGeneration Partnership Project; Technical Specification Group Service and System Aspects; Architecture enhancements for 5G System (5GS) to support network data analytics services”, 2023-09), where several instances of NWDAF, which can be referred to as clients or participants, work together to train an ML model. The process can be coordinated by another instance of NWDAF, which can be referred to as a server or a computing device, which can collect partially trained ML models from the participants and merge them together to form a single ML model which can be provisioned to an ML model consumer (e.g., to NWDAF (AnLF) for inference. In HFL, because all partially trained ML models are trained using the same set of features, HFL is applicable to one domain (that is inside the 5GC domain).SUMMARY

[0005] In some approaches in a collaborative distributed learning context in a telecommunications core network, such as federated learning, ML models are trained locally (e.g., by NWDAF clients) using the same features, which means that the locally trained ML models are exchanged / exposed among the clients. In some other approaches, such as vertical federated learning (VFL), ML models are trained using different features associated to each client. Use of multiple, different features, however, introduces a challenge in feature selection since different participants may have a different volume of data for each feature for the same samples. Another challenge may be that because of the nature of collaborative learning such as VFL, features which are extracted from samples aredistributed among the participants that are participating in the training process. Thus, the order of the features and samples needs to be preserved and aligned. An approach for doing so, however, is lacking.

[0006] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. In some embodiments, a computer-implemented method performed by a computing device for collaborative distributed training of a ML model in a telecommunications core network is provided. The method includes associating different features with a data sample in an aligned order for respective participants in the telecommunications core network where the participants respectively comprise a respective collaborative learning active participant model. The method further includes performing a training process comprising at least one of (I) accessing a respective output of the respective collaborative learning active participant models trained with the different features and concatenating the respective outputs into a concatenated input to the ML model, and (ii) provisioning a collaborative learning passive participant model to a ML model consumer for inference using the concatenated input provided by the respective participants.

[0007] According to other embodiments, a computing device or non-transitory readable medium is provided to perform the above method.

[0008] Certain embodiments may provide one or more of the following technical advantages. In some embodiments, feature and sample alignment is provided, which may provide robustness and efficiency to collaborative distributed learning and maintain correctness and integrity of the trained ML model. Moreover, the feature and sample alignment may provide flexibility in the training process of ML models that are used by NWDAF, for example, for inference and generating analytics reports. Using VFL, for example, ML models can be trained locally within different domains or among different training clients within same domain, without exposing training data that can compromise privacy and / or policies among domains / entities in the network. Further, some embodiments include cost estimation of feature and sample distribution to respective participants in the collaborative distributed learning (e.g., VFL process) which may reduce costs and improve efficiency.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments of inventive concepts. In the drawings:

[0010] Figure 1 is a schematic drawing illustrating an example of a split of an ML model for VFL in accordance with some embodiments;

[0011] Figure 2 is a schematic diagram illustrating an example of horizontal federated learning in a network data analytics function (NWDAF) in accordance with some embodiments;

[0012] Figure 3 is a schematic diagram illustrating an example of vertical horizontal federated learning in a NWDAF in accordance with some embodiments;

[0013] Figure 4 is a schematic diagram illustrating an example architecture for a feature manager in accordance with some embodiments;

[0014] Figure 5 is a block diagram of a NWDAF in accordance with some embodiments;

[0015] Figure 6 is a sequence diagram illustrating an example registration and discovery of participants in accordance with some embodiments;

[0016] Figure 7 is a block diagram of an example of sample ordering arrays in accordance with some embodiments;

[0017] Figure 8 is a sequence diagram illustrating an example training process in accordance with some embodiments;

[0018] Figure 9 is a sequence diagram illustrating an example for inter-domain VFL in accordance with some embodiments;

[0019] Figure 10 is a sequence diagram illustrating an example of a-priori sample alignment in accordance with some embodiments;

[0020] Figure 11 is a sequence diagram illustrating an example of a-priori sample alignment during a training process in accordance with some embodiments;

[0021] Figure 12 is a sequence diagram illustrating an example of online sample alignment during a training process in accordance with some embodiments;

[0022] Figures 13 and 14 are flow charts illustrating examples of operations performed by a computing device in accordance with some embodiments;

[0023] Figure 15 is a block diagram of a communication system in accordance with some embodiments;

[0024] Figure 16 is a block diagram of a computing device in accordance with some embodiments;

[0025] Figure 17 is a block diagram of a virtualization environment in accordance with some embodiments; and

[0026] Figure 18 is schematic diagram of an example O-RAN architecture in accordance with some embodiments.DETAILED DESCRIPTION

[0027] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art, in which examples of embodiments of the present disclosure are shown. The present disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0028] NWDAF uses different data sources to collect data for training and inference to generate analytics reports in response to a request from a NF service consumer. Data sources are used for data collection for different features that are used in ML models inside NWDAF. Using VFL, for example, ML models may be trained locally for a certain set of features at different domains such as, a core network (CN), a radio access network (RAN), a user equipment (UE), etc.; or within the same domain using different NFs inside the core network. In contrast to HFL, for example, collaborative distributed learning such as VFL may allow for the corresponding datasets to have the samesamples but different features which can be unique or contain some overlap between the VFL participants (e.g., NFs). As used herein, the term "collaborative distributed learning”, vertical federated learning”, or "VFL” refers to learning when the participants have different feature spaces. Such learning also may be referred to as parallel split learning (PSL).

[0029] In 3GPP TS 23.288 v18.3.0, HFL in NWDAF is standardized, where several instances of client NWDAFs (e.g., HFL clients), collaborate in training local ML models. The locally trained ML models are then sent to an instance of NWDAF (e.g., NWDAF server), to be merged and then provisioned as a single ML model to be used by NWDAF(AnLF) for inference. In HFL NWDAF, the HFL clients train the local ML models using the same features and have the same architecture. Thus, the locally trained ML models are exchanged / exposed among the clients.

[0030] There currently exist certain challenges. With regard to privacy, as opposed to HFL for example, where the feature space is known and shared collectively, in VFL the feature space can differ between the participants and as such remain concealed. This is because the respective clients only produce the corresponding embedding from their own feature space but with the same sample(s) without revealing what features have been used. A further challenge of HFL is that when the ML model is shared, it may be possible to obtain the ground truth that was used to train the ML model if secure aggregation is not used via gradient reversal attacks. VFL, for example, may overcome this challenge since it does not share the entire ML model but only the output of the last layer (previously referred to herein as embedding).

[0031] With regard to NWDAF, due to the way that HFL is standardized in 3GPP, the clients only include NWDAF instances and, thus, HFL is restricted to NWDAF. In VFL, for example, other entities in addition to NWDAF instances (e.g., a UE(s), RAN, or NFs other than or in addition to NWDAF), can collaborate in a distributed training process. It is noted, however, that in VFL a pattern can be followed that may be referred to as co-located NWDAF, where an instance of NWDAF (MTLF) is co-located with an NF instance and can collect NF specific data and train a local ML model.

[0032] With regard to a ML model, in VFL for example, a "tail” model residing on a computing device 100 (e.g., a NWDAF server) can be created that produces the output (e.g., classification or regression) from the input of the underlying heads of constituent "local” ML models of each client / participant 102a, 102b, as illustrated in Figure 1. The "tail” model may be re-used in other tasks that may benefit from the same or similar input as the input produced by the constituent models of client / participants 102a, 102b..

[0033] As previously discussed, distributed learning such as VFL can use multiple, different features which introduces a challenge in feature selection since different participants might have a different volume of data for each feature for the same samples. A further challenge includes that because of the nature of VFL, for example, features which are extracted from samples are distributed among the participants in the training process. Thus, the order of the features and samples needs to be preserved and aligned.

[0034] In some approaches, feature selection may include an a-priori task that takes place after a ML model has been trained. Techniques such as feature importance in gradient boosting models may be applied to determine important features that may have an impact on the output variable and / or a shapley additive explanations (SHAP) analysis may be employed in the types of ML models (e.g., including neural networks) to achieve the same result.Moreover, the challenge of feature selection also may be applicable to HFL, but it may be a simpler challenge since the feature space is the same for the respective clients.

[0035] In other approaches, sample alignment may be included in a setting where privacy is important, such as private sample intersection (PSI). While there may be such approaches for PSI, such approaches may rely on hash functions which may not be obtained in practice as extensive signaling is needed between participants to construct the hash function.

[0036] As used herein, different NFs or entities that participate in a collaborative distributed learning process (e.g., a VFL process) are referred to as participants. The respective collaborative learning (e.g., VFL) active participants have a local ML model, which is referred to herein as the collaborative learning active particiapant model or a VFL active participant model. The collaborative learning (e.g., VFL) passive participant is the tail, referred to herein as the collaborative learning passive participant model, a VFL passive participant model, or a ML model.

[0037] Examples of the present disclosure include collaborative distributed learning, such as VFL, in a NWDAF training process where different instances of NFs (also referred to herein as clients or participants) collaborate to train a ML model based on different features that are associated to each client by another instance of a NF (also referred to herein as a server or a computing device). When the collaborative learning active participant models are trained locally, either the server will collect the output from all collaborative learning active participant models from the clients and concatenate or a collaborative learning passive participant model that resides on the server is provisioned to an ML model consumer (e.g., NWDAF(AnLF)) for inference using as input the output of the constituent collaborative learning active participant models residing on the clients. Clients and the server can either be NWDAF(MTLF) instances and / or other NFs within the core network (referred to herein as intra-domain collaborative distributed learning) or a combination of NWDAF(MTLF) instances and other network entities (e.g., application function (AF), 0AM, etc.) which can reside in other domains such as a RAN, UE(s), etc., that collaborate in training an ML model (referred to herein as inter-domain collaborative distributed learning).

[0038] Examples further include a process to associate / align features to each client. This may reduce the cost of training and signaling and maintain the alignment between samples and features.

[0039] Examples of the present disclosure further include the following:• A feature distribution process that can measure a cost of training a ML model by a participant using a set of features and then minimize the cost by assigning features to the corresponding participants.• A sample alignment logical function used to align the samples either in advance or during the training process between different participants.• A logical function referred to herein as a "feature manager” inside NWDAF that can perform feature distribution and feature and sample alignment.• An architecture that enhances NWDAF to support VFL in a distributed training environment including: o Intra-domain VFL inside a 5GC among different NWDAF clients or other NFs that collaborate in the training process; and / or o Inter-domain VFL across domains, where NWDAF clients as well as other clients in, e.g., a UE or RAN domain collaborate in the training process.• Participants in VFL also may keep the information about what features they will use to train their collaborative learning active participant models and do not share either with other clients or with the server. This may be one by design, so the participants may be configured to be associated to a subset of feature space.• Transmission of the output of collaborative learning active participant models based on a specific set of features and then concatenate that output of the locally trained collaborative learning active participant models to become the input to a single collaborative learning passive participant model by NWDAF(MTLF) instances, which can be provisioned to and used by NWDF(AnLF) for inference.

[0040] An example architecture and examples are now discussed further.

[0041] In the following discussion, a process is described that associates a feature space(s) to different participants which includes preservation of features. The discussion further includes examples where participants are within the same domain (e.g., 5GC) and other examples where participants are distributed in different domains (e.g., RAN and 5GC).

[0042] Figures 2 and 3 illustrate differences between HFL and VFL, respectively. As shown in Figure 2, in HFL, a ML model is trained by clients 102a, 102b, 102c where all clients train their collaborative learning active participant model based on the same features while the training samples are different. As shown, client 102a uses training samples 1-3, client 102b uses training samples 4-6, and client 102c uses training samples 7-9. In this example, both the server 100 and the clients 102a, 102b, 102c are instances of NWDAF(MTLF).

[0043] As shown in Figure 3, in VFL, a ML model is trained by clients 102a, 102b, 102c where each client trains a collaborative learning active participant model based on the features that it has been assigned to by the server 100. Training samples 1-3 are the same for the clients 102a, 102b, 102c even though the features are different. This means that the clients 102a, 102b, 102c respectively train their collaborative learning active participant models with the same training samples 1-3 but with different features. In this example, the server 100 is NWDAF(MTLF), and the clients 102a, 102b, 102c can be any NF or entity in the same domain as the server 100 or in different domains in a network.

[0044] A feature sample distribution process is included in the examples that decides what features are associated to which participants to perform training of collaborative learning active participant models for the features. The feature sample distribution process can be implemented inside a logical unit (e.g., a feature manager that is embedded inside NWDAF). The feature sample distribution process receives different attributes about the data source(s) to provide features in the training data and the participant, and then evaluates the cost of training of a collaborative learning active participant model with the given features by a participant. Further, the feature sample distribution process is used by the server / computing device in distributed learning (e.g., VFL) to associate features to a participant and may include minimization of the training cost.

[0045] The feature sample distribution process can use a ML model that is trained offline and can determine which participant is assigned to which features to train a collaborative learning active participant model. The ML model can consider the cost of the training of a collaborative learning active participant model for the given features and available data source(s) to provide training data, as shown in the example in Figure 4.

[0046] Figure 4 illustrates an architecture for an entity, shown as feature manager 400 in this example, to calculate the cost of training a collaborative learning active participant model (shown as analytics Id 410) for specific feature(s) 406, and an ML model (shown as features-client assign ML model 404) to determine an output 416 that includes which feature(s) 406 will be assigned to which NF instances / workers based on the calculated cost 414 and available data sources 407. The feature manager 400 in this example is embedded inside a NWDAF as an logical function.

[0047] In Figure 4, the cost is estimated by estimated cost calculator 402. The output 416 of a features-client assign ML model 404 (e.g., a classifier) includes an identification, such as a list, of NF instances to operate as participants in a collaborative distributed learning process, such as VFL, orchestrated by an instance of NWDAF(MTLF) that acts as the server / computing device. In the output 416, the feature manager 400 of the NWDAF(MTLF) also associates the features to the identified NF instances or workers in other domains.

[0048] The estimated cost calculator 402 in Figure 4 is assigned to estimate the training cost per feature 406 in a distributed learning process, such as VFL. In this example, the cost is proportional to minimum number of samples per feature 406, type of the collaborative learning active participant model to train (e.g., analytics Id 410), and the number of features 406 to train the collaborative learning active participant model. Having a set of features 406, increasing a minimum number of samples per feature 406 may increase the cost as well the accuracy. On the other hand, decreasing a minimum number of samples per feature 406 may reduce the cost and, as a result, the accuracy. In this example, when the estimated cost 414 is calculated, it is sent to the features-client assign ML Model 404 which incorporates the estimated cost 414, a list of features 406, and available data sources 408 to provide data samples. The output 416 of the ML model 404, in this example, is a set of NFs in a core network or workers in other domains as candidates to participate in the collaborative distributed learning by training collaborative learning active participant models using the set of features 406. The NWDAF can then decide to choose participants and assign the feature space to the respective participants before triggering the training process.

[0049] In the example architecture in Figure 4, the logic of feature distribution, based on the estimated cost 414 is embedded into the logical function shown as feature manager 400. As discussed, feature manager 400 in this example includes as part of the NWDAF(MTLF) which acts as a server / computing device in the collaborative distributed learning process (e.g., a VFL process).

[0050] Figure 5 is a block diagram showing an example of the internal structure of an NWDAF 500 to distribute features among participants according to some embodiments. In Figure 5, NWDAF 500 includes the logical function referred to in, and discussed with respect to, Figure 4 as feature manager 400. In the architecture in Figure 5, a MTLF 502 queries the feature manager 400 to identify which features 406 will be assigned to which NFs / workers and in what domain(s). AnLF 504 is a consumer of the trained collaborative learning passive participant model to perform inference and generate predictions.

[0051] Alternatively, neither the server / computing device nor the participants know about which features have been associated to which participant. In this example, each participant is configured by design to train a collaborative learning active participant model for a specific subset of the features space. As a consequence, in this example, the feature manager 400 is not used.

[0052] A registration and discovery procedure for distributed learning, such as VFL, is now discussed.

[0053] In VFL, for example, each participant registers itself at a network repository function (NRF) by adding capability information about the "preferred” and / or "only” features for which the participant can train a VFL active participant model when collaborating in a VFL training process. The "preferred” and "only” features refer to an identity of the features that a NF prefers to be assigned to and / or the features for which the NF can only train a VFL active participant model, respectively. Figure 6 is a sequence diagram showing an example of a registration and discovery procedure. It is noted that the example process in Figure 6 covers a case when all participants are in the same domain (e.g., NFs that are within 5GC). For a cross-domain case, meaning that participants are distributed in different domains, a domain independent registry can be included that stores participant-feature information.

[0054] A challenge in VFL, for example, is that the participants should train their VFL active participant model using the same samples. This means, while each participant trains its respective VFL active participant model based on associated features, the same samples are used while features of each sample are split among the participants. Therefore, a coordination process is needed so the server / computing device will know how to aggregate features of each sample to compose, e.g., a correct row in the complete ML model where all features in a row belong to the same sample. Two strategies may be used to perform such synchronization between among the participants:• The participants will wait for the slowest participant to finish training for all its features for a specific sample. When all participants done with training for all features for a sample, a new training process is be started for a new sample. This process is referred to herein as a barrier-based process.• The participants train their respective collaborative learning active participant models independent of each other until convergence constraints, which are set by the server / computing device, are fulfilled. Each participant, upon finishing each training round, marks each set of trained features with a unique identifier. Later, when the collaborative learning active participant models are concatenated, the server / computing device will be able to associate all features with the same unique identifier to the same sample and correctly create a ML model. This process is referred to herein as a unique id-based process.

[0055] While the barrier-based process may not need extra meta data to express the relation between different parts of each sample, it uses a barrier as a synchronization point between the participants, which in turn may slow down the training process and degrade parallelism. Moreover, while the unique-id-based process may be faster because the participants train their collaborative learning active participant models in a parallel fashion, the unique-id-based process may need extra meta data to correlate each set of features to a specific sample.

[0056] Some examples of the present disclosure include that either the barrier-based process or the unique- id based process, or both, can be chosen as an agreed technique between the computing device and the participants.

[0057] When the participants train collaborative learning active participant models independently, the computing device identifies samples with different features when concatenating the collaborative learning active participant models. Thus, examples herein include that the computing device supports two techniques to identify samples:• Timestamp: The time that the samples are taken is used as a unique identifier and the computing device informs the participants about the granularity of the timestamp, e.g., minute, second, millisecond, etc.• Data property: The computing device informs the participant about a data property that can be used as a unique identifier, e.g., UE identifier (Id), subscription permanent identifier (SUPI), etc.

[0058] Figure 6 shows a sequence diagram of an example registration and discovery of different NF clients 102a . . . 102n that will operate as participants, where an NWDAF(MTLF) has the role of a server / computing device 500, in VFL.

[0059] In operation 1, the server NWDAF server 500 registers itself at NRF 600 by adding information to a NF profile. The server NWDAF server 500 also registers the supported methods for feature and samples alignment, e.g., a unique id-based process.

[0060] NF clients 102a . . . 102n, in operation 2, register themselves at NRF 600 by adding the capability of being able to participate in a VFL process as well as an identity of preferred features that each NF client 102a . . . 102n prefers to train its respective VFL active participant model or an identity of features that each NF client 102a . . . 102n can only train ML models for. If the identity of only features is empty, the NWDAF server 500 can assign the respective NF client 102 any feature even though the feature is not included in the preferred list of the features. The NF clients 102a . . . 102n also can register their profile at NRF 600 including the method(s) they support to collaborate with the NWDAF server 500 in features and samples alignment.

[0061] In operation 3, NWDAF server 500 sends a request to discover which NF clients 102a . . . 102n have the possibility to collaborate in VFL.

[0062] In operation 4, NRF 600 authorizes the NWDAF server 500.

[0063] NRF 600, in operation 5, sends the identified capable participants with the identification of preferred or only features that each participant has provided during registration. In this response signal, the NWDAF server 500 knows what training techniques are supported by each participant, e.g., barrier-, unique Id-based or both, as well as a unique identifier generation process.

[0064] Based on the received identification of the features, in operation 6, NWDAF server 500 sends a request to each identified participant 102 asking the participant to join a VFL process by assigning feature(s) to each participant 102 for training the VFL active participant model. The request is according to results from feature manager 400 (see Figure 4) that will be queried by MTLF of NWDAF server 500. In this request, the NWDAF server 500 indicates what technique is to be used to generate a unique identifier for each sample by each participant 102. Additionally, in this request, the NWDAF server 500 proposes to the identified participants 102 which training technique and unique identifier generation method to use. Further, the NWDAF server 500 includes the convergence constraints to the identified participants 102, e.g., number of samples used for training, accuracy requirements for the VFL active participant model, etc.

[0065] In operation 7, each participant 102 makes the decision to join the VFL process or decline.

[0066] In operation 8, the participants 102 send the response to the NWDAF server 500.

[0067] NWDAF server 500, in operation 9, selects participants 102 based on their response to the request to start the VFL process. At this point, the NWDAF server 500 has enough information about successful / failed feature distribution among NFs with different types. The process of the participant selection is based on the output from the feature manager 400 and incorporating the preferred and only features identification, registered by each participantin the NF participant 102 profile at NRF 600. The participant selection results are then used to further train the feature-client assign ML model 404 within the feature manager 400 of Figure 4.

[0068] Communication cost estimation is now discussed. Total communication cost in VFL, for example, is determined by the size of the cut layer, the number of samples, and the number of epochs. Since the cut layer is exchanged in forward and backward propagation the cost is multiplied by 2: totalcost= 2 * size(cutiayer) * epochs * samples

[0069] Communication cost per participant is determined by the size of the participants output (portion of the cut layer), the number of epochs, and the number of samples of each participant. The cost is multiplied by 2 since in the training process each participant also receives the same information when backward propagation takes place: cost; = 2 * size(cutlayer). * epochs * samples

[0070] An example training process of multiple participants and a computing device for VFL is now discussed. As discussed herein, the computing device and the participants can agree on the features and samples alignment method, e.g., unique identifier using timestamps for each sample. Information that can be used to specify the order of the samples in a VFL active participant model can include ML model meta data. The ML model metadata can be stored in a database, such as a sample order array (SOA).

[0071] A sample alignment logical function (SALF) assumes that for every VFL task every selected participant employs a feature which can be used to identify every sample. That feature is present in the dataset of each participant. The presence of this features allows for sample alignment, but this does not mean that every participant has the same samples. For example, as shown in Figure 7, participants may have some samples which are the same and some samples which are different. Additionally, the presence of this feature does not mean that the order of every sample included in the output of each VFL active participant model is the order that is expected. Arrays of samples can be stored in an analytics data repository function (ADRF) and can be used by the computing device when VFL active participant models are concatenated so the correct order of the samples is preserved.

[0072] Figure 7 is a block diagram of showing an example of SOAs that are stored by participants 102a, 102b, 102c in ADRF 700. The SOAs include unique identifiers (Ids), Ids 1-5, that describe the order of the trained samples without any need of synchronization between the participants 102a, 102b, 102c. The arrays are then used by the computing device (NWDAF server 500) when the VFL active participant models are concatenated to preserve a correct order between different partitions of the samples when the features are split among the participants 102a, 102b, 102c.

[0073] In another example, instead of ADRF 700, SOAs can also be stored locally at each participant 102a, 102b, 102c. Each participant 102a, 102b, 102c can decide whether to store the SOAs locally, considering memory / storage constraints, or whether ADRF 700 will be used:• If ADRF 700 is used to store SOAs, the NWDAF server 500 can be informed with the ADRF's 700 instance information, data set tag, etc., to fetch SOAs from ADRF 700.• If the participants use their internal memory / storage to store SOAs, then there is no need for the NWDAF server 500 to collect data, since the participants can send SOAs directly to the NWDAF server 500 when the training is completed.

[0074] Figure 8 is a sequence diagram of an example training process in VFL including agreement between the NWDAF server 500 and the participants 102a. . . 102n on the training process, and updating SOAs if the unique- id based process is used. The training process is repeated until the NWDAF server 500 decides to terminate the training process based on the local constraints that are set by the NWDAF server 500 and distributed to the participants 102a. . . 102n, or when a ML model consumer 504 is satisfied with the concatenated ML model.

[0075] The process of training VFL active participant models and concatenation by the NWDAF server 500 in VFL as shown in Figure 8 includes the following operations.

[0076] In operation 1, ML model consumer 504 subscribes to receive a ML model trained by NWDAF server 500.

[0077] In operation 2, NWDAF server 500 initiates a discovery process to find the participants that can collaborate in training a ML model, as described herein with reference to Figure 6.

[0078] At this point, the NWDAF server 500 and the participants 102a . . . 102n have agreed on what technique to use, e.g., barrier- or unique Id-based process, or both. Then, in operation 3, the NWDAF server 500 sends requests to the participants 102a. . . 102n to start the training process on the VFL active participant models. The NWDAF server 500 informs each participant 102a. . . 102n about the features to be used to train their respective VFL active participant models, and optionally constraints to be fulfilled by the participants 102a. . . 102n when the training is done.

[0079] In operation 4, during the training process, each participant 102 collects training data from relevant data sources 200.

[0080] Operation 5 is optional. If the participants 102 are using a barrier-based technique, synchronization is performed to make sure the same sample is used for training in each round.

[0081] In operation 6, which is optional, if the participants 102 are using a unique Id-based technique, unique Ids are generated and added to SOAs in ADRF 700.

[0082] Operation 7 is optional. In this operations, participants 102 check whether the constraints received from the NWDAF server 500 are met or not.

[0083] In operation 8, participants 102 notify the NWDAF server 500 about the training process and send the VFL active participant models to the NWDAF server 500, optionally together with the results of evaluation of the constraints in operation 7.

[0084] In operation 9, NWDAF server 500 concatenates the VFL active participant models to form a ML model that can be used by an ML model consumer 504 for inference. In this operation, the NWDAF server 500 can either collect SOAs from ADRF 700 or has already received them from the participants 102 if local storage at each participant is used.

[0085] Operation 10 is optional. NWDAF server 500 informs the ML model consumer 504 about the training status by provisioning the concatenated ML model to the ML model consumer 504.

[0086] In operation 11 , which is optional, ML model consumer 504 decides whether to terminate the training process, if the provisioned ML model is acceptable, or update the subscription (meaning that the training process continues).

[0087] In operation 12, which is optional, based on the response from the ML model consumer 504, the NWDAF server 500 decides to update or terminate the training process.

[0088] In operation 13, which is optional, if the NWDAF server 500 decides to update the training process, the concatenated ML model is distributed to the participants 102, with the updated constraints, for further training.

[0089] In an example of intra-domain VFL, participants are located only in a core network and features are allocated to the participants which can either be NWDAF instances or generally any NF in the network.

[0090] Referring to Figure 3, in intra-domain VFL, any client NF 102 (e.g., access and mobility management function (AMF), PCF, SMF, NWDAF(MTLF), etc.) can act as the client NF / participant while the server / computing device 100 is an instance of NWDAF. The NWDAF server 100 decides to use intra-domain or inter-domain as discussed herein, since the features for the ML model in this examples are all located inside the 5GC or distributed among several domains.

[0091] Alternatively, in another example, uses co-located NWDAF instances as participants in a VFL process. This means that for each NF (e.g., AMF, PCF, SMF, etc.), an instance of NWDAF(MTLF) is assigned to collects data for the features associated to that NF to perform VFL active participant model training.

[0092] In another example of inter-domain VFL, participants are distributed among domains so, instead of collecting the training data into a core network, the VFL active participant models train within each domain and are transferred to the server (e.g., NWDAF(MTLF)). This strategy can be used by the server if the features are distributed among several domains.

[0093] In yet another example, in domain-specific VFL, while features to train a single VFL active participant model are distributed based on the properties of each domain, a local server in each domain can then associate features into the participant's within each domain.

[0094] This example is an extension functionality and is shown in the sequence diagram in Figure 9 where the computing device / server 500a in 5GC (e.g., an instance of NWDAF(MTLF)) communicates with the servers 500b, 500c in other domains. In this example, domain servers 500b, 500c follow the operations in Figure 9 and inform the NWDAF(MTLF) 500a when the VFL active participant models are concatenated.

[0095] In Figure 9, the training process in each domain in the inter-domain training is summarized and includes the following operations.

[0096] In operation 1 , NWDAF server 500a decides what features belong to which domain; and then distributes the relevant features to each server 500b, 500c in each domain.

[0097] Each server 500b, 500c, in operation 2, asks the respective clients 102 in the respective domains to train collaborative learning active participant models as discussed herein, and concatenates the collaborative learning active participant models into a domain specific ML model.

[0098] In operation 3, the domain specific ML models are sent to NWDAF server 500a.

[0099] In operation 4, NWDAF server 500a concatenates the received domain specific ML models to create aML model that can be used by an ML model consumer.

[0100] A sample alignment logical function is now discussed. An important challenge in VFL, for example, is that of aligning samples between participants. During the training process, and also during inference, the activations that are received in the cut-layer (or the input to the head model) need to be aligned otherwise the collaborativelearning passive participant model may infer incorrectly by mixing up samples. This challenge is illustrated in the following table:[OO1O1] In the absence of sample alignment, the computing device / server may receive as input a concatenation of the activations from participant 1 related to sample id 1 and the activations from participant 2 for sample id 2 which would then require the collaborative learning passive participant model to infer about a 24 year old Male that has a PhD obtained in 2016. Since such information is not available in this example dataset, the collaborative learning passive participant model will not be able to perform its task which is that of producing a reliable estimation.

[0102] To that end, some examples include a sample alignment logical function that is tasked with aligning samples. Sample alignment can take place either before the training process begins (a-priori) or during the training process (online). In addition, sample alignment is also needed for inference purposes.

[0103] As shown in the example in Figure 10, in a-prioiri sample alignment, each participant receives a NWDAF sample alignment process 1002a, 1002b from the server 500 which describes which samples are to be selected and in what order.

[0104] The NWDAF sample alignment processes 1002a, 1002b in Figure 10 include:

[0105] In an online process, it may be assumed that a-priori alignment has not taken place. However, the output of every participant may still need to be marked to allow a VFL server to concatenate samples accordingly.

[0106] During the training process, the output of every participant which is received by the VFL server (e.g., when a VFL active participant model is hosted in the VFL server) is marked with the corresponding sample id. In this way, the VFL server can concatenate samples accordingly for forward propagation and for backward propagation.

[0107] Figure 11 is a sequence diagram of an example training process in VFL for a case of a-priori sample alignment during the training process. For example, if an a-priori sample alignment process 1102 applies during a training process 1100, loop 1104 is performed for every epoch or batch. In a loop 1104, in operation 1, client 102a performs forward propagation, and in operation 2 shares activations a1, a2, a3 with sample alignment logical function (LF) 500a. In operation 3, client 102b performs forward propagation, and in operation 4 shares activationsa4, a5, a6 with sample alignment logical function 500a. In operation 5, server 500 sends a request to sample alignment LF 500a to concatenate ([a1 , a4], [a2, a5], [a3, a6]). Sample alignment LF 500a, in operation 6, concatenates ([a1 , a4], [a2, a5], [a3, a6]) and forward propagates to server 500. In operations 7 and 8, server 500 calculates a loss and backward propagates. In operation 9, server 500 sends gradients 1 and 2 (g1 , g2) to sample alignment LF 500a. In operation 10, sample alignment LF 500a backward propagates g1 to client 102a; and in operation 11, sample alignment LF 500a backward propagates g2 to client 102b.

[0108] Figure 12 is a sequence diagram of an example training process in VFL for a case of online sample alignment during the training process. For example, if an online sample alignment process applies during a training process 1200, loop 1202 is performed for every epoch or batch. In a loop 1204, in operation 1, client 102a performs forward propagation, and in operation 2 shares indexes "1”, "2”, and "3” of each activation with sample alignment LF 500a as follows: activations 2:a2, 3: a3, 1 : a1. In operation 3, client 102b performs forward propagation, and in operation 4 shares indexes "1”, "2”, and "3” of each activation with sample alignment LF 500a as follows: activations 2:a5, 3: a6, 1 : a4. In operation 5, server 500 sends a request to sample alignment LF 500a to concatenate ([a1, a4], [a2, a5], [a3, a6]). Sample alignment LF 500a, in operation 6, concatenates ([a1 , a4], [a2, a5], [a3, a6]) and forward propagates to server 500. In operations 7 and 8, server 500 calculates a loss and backward propagates. In operation 9, server 500 sends gradient 1 (g1) for index 1 and gradient 2 (g2) for index 2 to sample alignment LF 500a. In operation 10, sample alignment LF 500a backward propagates g1 to client 102a; and in operation 11, sample alignment LF 500a backward propagates g2 to client 102b.

[0109] As shown in Figure 11, in the case of a-priori sample alignment, there is no need to share the index of each activation to the sample alignment logical function since activations between participants have already been aligned a-priori. As shown in Figure 12 in the online case however, the sample indices are needed to allow the sample alignment LF 500a to concatenate the samples in the correct sequence.

[0110] In another example in federated transfer learning, features are distributed among domains, while each server can then employ horizontal federated learning by assigning the same set of features (e.g., a subset of all features of the main ML model) to all participants in that domain. This example is another option of the example discussed with respect to Figure 9, where each server asks the clients to train collaborative learning active participant models on exactly the same set of features. In some further examples, VFL and HFL are combined. HFL is employed between active participants that have the same features and respective ML models of the active participants are aggregated into an aggregated ML model. The aggregated ML model is used with the collaborative learning active participant models in VFL.

[0111] Alternatively, in other examples, VFL and HFL are combined, where VFL is employed between active participants that have different features and respective collaborative learning active participant models are aggregated into an aggregated ML model. The aggregated ML model is used with the active participant models in HFL.

[0112] Additionally, or alternatively, a feature alignment function may decide to train two participants in an HFL fashion if they have exactly the same features.

[0113] As discussed herein, some examples include a process to align samples and features in large scale distributed learning, such as VFL, of network functions in a telecommunication core network assisted by NWDAF.

[0114] The method of various embodiments here may result in one or more of the following:• The signaling among different components e.g., UEs) and NFs in 5GC, may be reduced because the training data remains within the domain to which the data sources belong. Therefore, instead of exchanging training data, the collaborative learning trained active participant models are sent to the server to form an ML model with all features that can be provisioned for inference.• Privacy preferences may be preserved since private data, e.g., UE traffic information, remains with the same domain where the features exist, and can be used to train collaborative learning active participant models. Once, the collaborative learning active participant models are trained, the collaborative learning active participant models can be sent to the server in another domain, e.g., 5GC, and raw training data is not exposed to other components in a separate domain.• By assigning features to the participants, training collaborative learning active participant models may be done in an efficient way. For instance, if one participant, e.g., NWDAF(MTLF), is closer to a AMF or has privileges to use some kind of data, it may be more efficient to assign AMF-related feature to that instance of NWDAF(MTLF). Otherwise, the whole training data can be transferred from the AMF, which may be in another geographical area to the single instance of NWDAF(MTLF) for training.• Energy consumption may be reduced because of transferring less data for training since a large part of training is done at participants which may be close or in the same domain as the data source(s).

[0115] Some examples herein include associating features in an ML model to NF instances or domains by using a cost calculation and estimation process where an estimated cost calculator is utilized. In some examples, the estimated cost calculator receives the feature to be associated, NF instances or domains, available data sources to provide training data, and service level agreement (SLA)Zintent that possibly put constraints on the features to be associated to specific NFs or domains. Further, in some examples, the estimated cost is then sent to a ML model classifier which receives the features and the available data source(s) and can then determine which feature(s) to assign to which participants.

[0116] Yet other examples include identifying samples based on either a timestamp(s) or the use of a unique identifier determined by the server and usage of a sample order array (which can be stored in an ADRF) as the collaborative learning active participant models' metadata. The sample order arrays can be used by the server for features and samples alignment.

[0117] In still other examples, for inter-domain VFL, interfaces between a core network, UE(s) and RAN (e.g., N1, N2, and N5) can be updated to support feature association and collaborative learning active participant model exchange. Moreover, in intra-domain VFL, services in 5GC service-based architecture (SBA) can be updated to support feature distribution and collaborative learning active participant model exchange.

[0118] In a further example, in an open radio access network (O-RAN), xApps and / or rApps can act as the participants and train collaborative learning active participant models for features that are specific to the O-RAN domain. The process can then be coordinated by NWDAF(MTLF) through 0AM and the trained collaborative learning active participant models can be transferred to the server for inference. Some O-RAN specific features can be listed as uplink / downlink (UL / DL) packet drop, delay, and drop / loss which can be assigned to an xApp / rApp to train a collaborative learning active participant model.

[0119] Figure 18 is schematic diagram of an example 0-RAN architecture. This example 0-RAN architecture includes:(1) SMO framework 1800 comprises a component that oversees orchestration, management, and automation of RAN elements, and supports the 01, A1, and 02 interfaces;(2) Near-RT RIO 1804 comprises a logical function that enables near-real-time control and optimization of O-RAN elements and resources via fine-grained data collection and actions over an E2 interface; and includes an xApp 1820;(3) Non-RT RIO 1802 comprises a logical function that enables non-real-time control and optimization of RAN elements and resources, artificial intelligence (AI) / ML workflow including model training and updates, and policy-based guidance of applications / features in near-RT RIO; and includes an rApp 1818;(4) O-RAN Central Unit (O-CU) 1814 comprises a logical node hosting radio resource controls (RRC), service data adaptation protocol (SDAP) and packet data convergence protocol (PDCP) protocols;(5) O-RAN Central Unit - Control Plane (O-CU-CP) 1808 comprises a logical node hosting the RRC and the control plane part of the PDCP protocol;(6) O-RAN Central Unit - User Plane (O-CU-UP) 1810 comprises a logical node hosting the user plane part of the PDCP protocol and the SDAP protocol;(7) O-RAN Distributed Unit (O-DU) 1812 comprises a logical node hosting radio link control (RLC) / medium access control (M AC) / High-physical (PHY) layers based on a lower layer functional split;(8) O-RAN Radio Unit (O-RU) 1814 comprises a logical node hosting Low-PHY layer and radio frequency (RF) processing based on a lower layer functional split. This is similar, for example, to the Third Generation Partnership Project (3GPP's) transmission / reception points (“TRP”) or remote radio head (“RRH”) but more specific in including the Low-PHY layer (fast Fourier transform (FFT)Zinverse FFT (IFFT), physical random access channel (PRACH) extraction);(9) xAPP 1820 comprises a network function which, for example, can be an independent software plug-in to the Near-RT RIG 1804 platform to provide functional extensibility to a RAN by third parties;(10) rAPP 1818 comprises a network function which, for example, can be an independent software plug-in to the Non-RT RIG 1802 platform to provide functional extensibility to the RAN by third parties;(11) Computing device 1806 comprises, for example, an NWDAF(MTLF); and(12) O-Cloud 1816 comprises hardware and / or software components that can provide cloud computing capabilities to execute RAN network functions

[0120] Operations of a computing device (implemented using the structure of Figure 16) will now be discussed with reference to the flow charts of Figure 13 and 14 according to some embodiments of the present disclosure. For example, modules may be stored in memory 1604 or NWDAF 500 of Figure 16, and these modules may provide instructions so that when the instructions of a module are executed by respective computing device processing circuitry 1602, computing device 1600 performs respective operations of the flow chart.

[0121] In some embodiments, a computer-implemented method is provided that is performed by a computing device for collaborative distributed training of a ML model in a telecommunications core network. The method includes associating (operation 1308 in Figure 13) different features with a data sample in an aligned order for respective participants in the telecommunication core network where the participants respectively include a respective collaborative learning active participant model. The method further includes performing (operation 1314 in Figure 13) a training process including at least one of (i) accessing a respective output of the respective collaborative learning active participant models trained with the different features and concatenating the respective outputs into a concatenated input to the ML model, and (ii) provisioning a collaborative learning passive participant model to a ML model consumer for inference using the concatenated input provided by the respective participants.

[0122] In some embodiments, associating (operation 1308 in Figure 13) includes at least one of (i) identifying a plurality of common samples in the respective participants and selecting the different features from the common samples and (ii) selecting the different features and identifying a plurality of samples comprising the different features that are common for the respective participants.

[0123] In some embodiments, the method further includes deciding (1310) what features from the different features from a plurality of samples to associate to the respective participants to train the respective collaborative learning active participant models; and triggering (1312) training of the respective collaborative learning active participant models with the features from the different features associated to the respective participants. At least one of the concatenating and the concatenated input is based on a preserved order between the different features of the plurality of samples associated to the respective participants.

[0124] Associating (1308) can include least one of (i) a-priori alignment of a plurality of data samples and (ii) online alignment using indices to concatenate a plurality of data samples in the aligned order.

[0125] Associating (1308) can further include (i) receiving attributes about at least one source of respective data samples from a plurality of data samples, (ii) estimating an estimated cost of training a respective collaborative learning active participant model with at least one feature from the different features, and (iii) determining an assignment of the at least one feature to a respective participant based on the estimated cost and an availability of the at least one source of data. In some embodiments, the estimated cost is proportional to a minimum number of data samples per feature, a type of a respective collaborative learning active participant model, and a number of the different features to train a respective collaborative learning active participant model. Determining (in item (iii)) can include (i) using the estimated cost, the different features, and at least one available source of the respective data samples to obtain an identity of at least one participant and a domain of the at least one participant, and (ii) assigning the different features to the at least one participant

[0126] In some embodiments, a respective participant is configured to train a respective collaborative learning active participant model with at least one feature from the different features.

[0127] In some embodiments, the method further includes sending (1300) a first request to a database to discover participants that can collaborate in distributed learning. The method further includes receiving (1302) a response to the first request including an identifier of identified participants that can collaborate in distributed learning, at least one feature supported by the identified participants, and at least one of a barrier-based process and a unique identifier-based process supported per identified participant for aligning the different features to thedata sample. The method further includes sending (1304) a second request to the respective identified participants including (i) a request that the respective identified participants join a distributed learning process, and (ii) the different features of a data sample to train a respective collaborative learning active participant model of the respective identified participants; and selecting (1306) the respective participants for the distributed learning process based on responses to the second request from the identified participants.

[0128] In some embodiments, at least one of a barrier-based process and a unique identifier-based process includes processing information to identify an order of a plurality of data samples from a respective collaborative learning active participant model of the respective identified participants.

[0129] In yet other embodiments, the method further includes terminating (1318) the training process (i) based on local constraints set by the computing device and distributed to the respective participants, or (ii) responsive to an indication from a ML model consumer of satisfaction with the ML model for inference.

[0130] The training process can include use of a barrier-based process to align the different features to a same data sample used in each round of the training process.

[0131] In other embodiments, the training process includes an identity of a unique identifier based process to the respective participants to use to generate respective unique identifiers to mark at least one of the different features used to train the respective collaborative learning active participant models.

[0132] In still other embodiments, the method further includes informing (1316) the participants of constraints to be fulfilled by the respective participants when the training process is completed.

[0133] The training process can further include receiving the respective outputs pf the respective collaborative learning active participant models from the respective participants.

[0134] The training process can further include receiving an evaluation of the constraints.

[0135] In other embodiments, the training process includes updating the training process based on distributing the ML model to the respective participants with updated constraints for further training.

[0136] In some embodiments, collaborative distributed learning includes intra-domain collaborative distributed learning.

[0137] In other embodiments, collaborative distributed learning includes inter-domain collaborative distributed learning.

[0138] In yet other embodiments, the method further combining (1400) VFL and HFL. HFL is employed between active participants that have the same features and respective ML models of the active participants are aggregated (1402) into an aggregated ML model. The aggregated ML model is used (1404) with the collaborative learning active participant models in VFL.

[0139] In other embodiments, VFL and HFL are combined (1406), where VFL is employed between active participants that have different features and respective collaborative learning active participant models are aggregated (1408) into an aggregated ML model. The aggregated ML model is used (1410) with the active participant models in HFL.

[0140] Additionally, or alternatively, in other embodiments, a feature alignment function may decide to train two participants in an HFL fashion if they have exactly the same features.

[0141] Various operations from the flow charts of Figures 13-15 may be optional with respect to some embodiments of computing devices configured for collaborative distributed training of a ML model in a telecommunications core network and related methods. For example, the operations of blocks 1300-1306, 1310- 1312, 1316-1318, and 1400-1410 may be optional in some embodiments.

[0142] Figure 16 shows a computing device 1600 in accordance with some embodiments. The computing device 1600 includes a processing circuitry 1602, a memory 1604, a communication interface 1608, a power source 1610, and optionally NWDAF 500. It is noted that in some embodiments the computing device 1600 may be NWADF 500, and NWDAF 500 may include a processing circuitry 1602, a memory 1604, a communication interface 1608, a power source 1610. The computing device 1600 may be composed of multiple physically separate components, which may each have their own respective components. In certain scenarios in which the computing device 1600 comprises multiple separate components, one or more of the separate components may be shared among several computing devices. In some embodiments, the computing device 1600 may also include multiple sets of the various illustrated components for different wireless technologies integrated into computing device 1600.

[0143] The processing circuitry 1602 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 800 components, such as the memory 806 and / or network function manager component 808, to provide network node 800 functionality.

[0144] The memory 1604 and / or NWDAF 500 may comprise any form of volatile or non-volatile computer- readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 1602. The memory 1604 and / or NWDAF 500 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 1602 and utilized by the computing device 1600. The memory 1604 and / or NWDAF 500 may be used to store any calculations made by the processing circuitry 1602 and / or any data received via the communication interface 1608. In some embodiments, the processing circuitry 1602, memory 1604 and / or NWDAF 500 are integrated.

[0145] The communication interface 1608 is used in wired or wireless communication of signaling and / or data between a NF, client, participant, and / or server. The communication interface 1608 can comprise port(s) / terminal (s) to send and receive data, for example to and from a network over a wired connection. Digital data may be passed to the processing circuitry 1602. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0146] The power source 1610 provides power to the various components of computing device 1600 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respectivecomponent). The power source may further comprise, or be coupled to, power management circuitry to supply the components of the computing device 1600 with power for performing the functionality described herein. For example, the computing device 1600 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source. As a further example, the power source may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0147] Embodiments of the computing device 1600 may include additional components beyond those shown in Figure 16 for providing certain aspects of the computing device's functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the computing device 1600 may include user interface equipment to allow input of information into the computing device 1600 and to allow output of information from the computing device 1600. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the computing device 1600.

[0148] Figure 15 shows an example of a communication system 1500 in accordance with some embodiments.

[0149] In the example, the communication system 1500 includes a telecommunication network 1502 that includes an access network 1504, such as a RAN, and a core network 1506, which includes one or more core network nodes 1508. The access network 1504 includes one or more access network nodes, such as network nodes 1510a and 1510b (one or more of which may be generally referred to as network nodes 1510), or any other similar 3GPP access node or non-3GPP access point. Moreover, as will be appreciated by those of skill in the art, the network nodes 1510 are not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that the network nodes 1510 may include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunication network 1502 includes one or more ORAN network nodes. An ORAN network node is a node in the telecommunication network 1502 that supports an ORAN specification (e.g., a specification published by the O- RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the telecommunication network 1502, including one or more network nodes 1510 and / or core network nodes 1508.

[0150] Examples of an ORAN network node include an O-RU, an O-DU, an O-CU, including an O-CU-CP or an O-CU-UP, a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time RAN control application (e.g., xApp) or a non-real time RAN automation application (e.g., rApp), or any combination thereof (the adjective "open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an A1, F1, W1, E1, E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Intents and content-aware notifications described herein may be communicated from a 3GPP network node or an ORAN network node over 3GPP-defined interfaces (e.g., N2, N3) and / or ORAN Alliance- defined interfaces (e.g., A1, 01). Moreover, an ORAN network node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below)in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a SMO Framework via an 0-2 interface defined by the O-RAN Alliance. The network nodes 1510 facilitate direct or indirect connection of user equipment (UE), such as by connecting wireless devices 1512a, 1512b, 1512c, and 1512d (one or more of which may be generally referred to as UEs 1512) to the core network 1506 over one or more wireless connections. The network nodes 1510 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 1512a, 1512b, 1512c, and 1512d (one or more of which may be generally referred to as UEs 1512) to the core network 1506 over one or more wireless connections.

[0151] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 1500 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 1500 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0152] The UEs 1512 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 1510 and other communication devices. Similarly, the network nodes 1510 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 1512 and / or with other network nodes or equipment in the telecommunication network 1502 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 1502.

[0153] In the depicted example, the core network 1506 connects the network nodes 1510 to one or more hosts, such as host 1516. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 1506 includes one more core network nodes (e.g., core network node 1508) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 1508. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0154] The host 1516 may be under the ownership or control of a service provider other than an operator or provider of the access network 1504 and / or the telecommunication network 1502, and may be operated by the service provider or on behalf of the service provider. The host 1516 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs,analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0155] As a whole, the communication system 1500 of Figure 15 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WIMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LIFI, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0156] In some examples, the telecommunication network 1502 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 1502 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 1502. For example, the telecommunications network 1502 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC) / Massive loT services to yet further UEs.

[0157] In some examples, the UEs 1512 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 1504 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 1504. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0158] In the example, the hub 1514 communicates with the access network 1504 to facilitate indirect communication between one or more UEs (e.g., UE 1512c and / or 1512d) and network nodes (e.g., network node 1510b). In some examples, the hub 1514 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 1514 may be a broadband router enabling access to the core network 1506 for the UEs. As another example, the hub 1514 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 1510, or by executable code, script, process, or other instructions in the hub 1514. As another example, the hub 1514 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 1514 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 1514 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 1514 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 1514 acts as a proxy server or orchestrator for the UEs, in particular in if one or more of the UEs are low energy loT devices.

[0159] The hub 1514 may have a constant / persistent or intermittent connection to the network node 1510b. The hub 1514 may also allow for a different communication scheme and / or schedule between the hub 1514 and UEs (e.g., UE 1512c and / or 1512d), and between the hub 1514 and the core network 1506. In other examples, the hub 1514 is connected to the core network 1506 and / or one or more UEs via a wired connection. Moreover, the hub 1514 may be configured to connect to an M2M service provider over the access network 1504 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 1510 while still connected via the hub 1514 via a wired or wireless connection. In some embodiments, the hub 1514 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 1510b. In other embodiments, the hub 1514 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 1510b, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0160] Figure 17 is a block diagram illustrating a virtualization environment 1700 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1700 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 1700 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an O-2 interface.

[0161] Applications 1702 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0162] Hardware 1704 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1706 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1708a and 1708b (one or more of which may be generally referred to as VMs 1708), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 1706 may present a virtual operating platform that appears like networking hardware to the VMs 1708.

[0163] The VMs 1708 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1706. Different embodiments of the instance of a virtual appliance 1702 may be implemented on one or more of VMs 1708, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume serverhardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0164] In the context of NFV, a VM 1708 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1708, and that part of hardware 1704 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1708 on top of the hardware 1704 and corresponds to the application 1702.

[0165] Hardware 1704 may be implemented in a standalone network node with generic or specific components. Hardware 1704 may implement some functions via virtualization. Alternatively, hardware 1704 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1710, which, among others, oversees lifecycle management of applications 1702. In some embodiments, hardware 1704 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1712 which may alternatively be used for communication between hardware nodes and radio units.

[0166] Although computing devices described herein (e.g., NWDAFs, servers) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination.

[0167] Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0168] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on aseparate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

Claims

CLAIMSWhat is Claimed is:1 . A computer-implemented method performed by a computing device for collaborative distributed training of a machine learning, ML, model in a telecommunications core network, the method comprising: associating (1308) different features with a data sample in an aligned order for respective participants in the telecommunication core network where the participants respectively comprise a respective collaborative learning active participant model; and performing (1314) a training process comprising at least one of (I) accessing a respective output of the respective collaborative learning active participant models trained with the different features and concatenating the respective outputs into a concatenated input to the ML model, and (ii) provisioning a collaborative learning passive participant model to a ML model consumer for inference using the concatenated input provided by the respective participants.

2. The method of Claim 1, wherein the associating (1308) comprises at least one of (I) identifying a plurality of common samples in the respective participants and selecting the different features from the common samples and (ii) selecting the different features and identifying a plurality of samples comprising the different features that are common for the respective participants.

3. The method of any one of Claims 1 to 2, further comprising: deciding (1310) what features from the different features from a plurality of samples to associate to the respective participants to train the respective collaborative learning active participant models; and triggering (1312) training of the respective collaborative learning active participant models with the features from the different features associated to the respective participants, wherein at least one of the concatenating and the concatenated input is based on a preserved order between the different features of the plurality of samples associated to the respective participants.

4. The method of any one of Claims 1 to 3, wherein the associating (1308) comprises at least one of (I) a-priori alignment of a plurality of data samples and (ii) online alignment using indices to concatenate a plurality of data samples in the aligned order.

5. The method of any one of Claims 1 to 4, wherein the associating (1308) further comprises(I) receiving attributes about at least one source of respective data samples from a plurality of data samples,(ii) estimating an estimated cost of training a respective collaborative learning active participant model with at least one feature from the different features, and(ill) determining an assignment of the at least one feature to a respective participant based on the estimated cost and an availability of the at least one source of data.

6. The method of Claim 5, wherein the estimated cost is proportional to a minimum number of data samples per feature, a type of a respective collaborative learning active participant model, and a number of the different features to train a respective collaborative learning active participant model.

7. The method of any one of Claims 5 to 6, wherein the determining comprises (I) using the estimated cost, the different features, and at least one available source of the respective data samples to obtain an identity of at least one participant and a domain of the at least one participant, and (ii) assigning the different features to the at least one participant.

8. The method of any one of Claims 1 to 4, wherein a respective participant is configured to train a respective collaborative learning active participant model with at least one feature from the different features.

9. The method of any one of Claims 1 to 8, further comprising: sending (1300) a first request to a database to discover participants that can collaborate in distributed learning; receiving (1302) a response to the first request comprising an identifier of identified participants that can collaborate in distributed learning, at least one feature supported by the identified participants, and at least one of a barrier-based process and a unique identifier-based process supported per identified participant for aligning the different features to the data sample; sending (1304) a second request to the respective identified participants comprising (I) a request that the respective identified participants join a distributed learning process, and (ii) the different features of a data sample to train a respective collaborative learning active participant model of the respective identified participants; and selecting (1306) the respective participants for the distributed learning process based on responses to the second request from the identified participants.

10. The method of Claim 9, wherein the at least one of a barrier-based process and a unique identifier-based process comprises processing information to identify an order of a plurality of data samples from a respective collaborative learning active participant models of the respective identified participants.11 . The method of any one of Claims 1 to 10, further comprising: terminating (1318) the training process (I) based on local constraints set by the computing device and distributed to the respective participants, or (ii) responsive to an indication from a ML model consumer of satisfaction with the ML model for inference.

12. The method of any one of Claims 1 to 11, wherein the training process comprises use of a barrier-based process to align the different features to a same data sample used in each round of the training process.

13. The method of any one of Claims 1 to 12, wherein the training process comprises an identity of a unique identifier based process to the respective participants to use to generate respective unique identifiers to mark at least one of the different features used to train the respective collaborative learning active participant models.

14. The method of any one of Claims 1 to 13, further comprising: informing (1316) the participants of constraints to be fulfilled by the respective participants when the training process is completed.

15. The method of any one of Claims 1 to 14, wherein the training process further comprises receiving the respective outputs of the respective collaborative learning active participant models from the respective participants.

16. The method of any one of Claims 14 to 15, wherein the training process further comprises receiving an evaluation of the constraints.

17. The method of any one of Claims 1 to 16, wherein the training process further comprises updating the training process based on distributing the ML model to the respective participants with updated constraints for further training.

18. The method of any one of Claims 1 to 17, wherein the collaborative distributed learning comprises intradomain collaborative distributed learning.

19. The method of any one of Claims 1 to 17, wherein the collaborative distributed learning comprises interdomain collaborative distributed learning.

20. The method of any one of Claims 1 to 19, further comprising: combining (1400) vertical federated learning, VFL, and horizontal, HFL, wherein HFL is employed between active participants that have the same features; aggregating (1404) respective ML models of the active participants into an aggregated ML model; and using (1404) the aggregated ML model with the collaborative learning active participant models in VFL.21 . The method of any one of Claims 1 to 19, further comprising: combining (1406) vertical federated learning, VFL, and horizontal, HFL, wherein VFL is employed between active participants that have different features; aggregating (1408) respective collaborative learning active participant models into an aggregated ML model; and using (1410) the aggregated ML model with the active participant models in HFL.

22. The method of any one of Claims 20 to 21 , wherein a feature alignment function trains two participantsin an HFL fashion if they have exactly the same features.

23. A computing device (500, 1508, 1600, 1702, 1806) in a telecommunications core network comprising: processing circuitry (1602); memory (1604) coupled with the processing circuitry, wherein the memory includes instructions that when executed by the processing circuitry causes the computing device to perform operations comprising: associate different features with a data sample in an aligned order for respective participants in the telecommunication core network where the respective participants respectively comprise a respective collaborative learning active participant model; and perform a training process comprising at least one of (I) accessing a respective output of the respective collaborative learning active participant models trained with the different features and concatenating the respective outputs into a concatenated input to the ML model, and (ii) provisioning a collaborative learning passive participant model to a ML model consumer for inference using the concatenated input provided by the respective participants.

24. The computing device of Claim 23, wherein the operations further comprise any of the operations of Claims 2 to 22.

25. A non-transitory computer readable medium (1604) including program code to be executed by processing circuitry (1602) of a computing device (500, 1508, 1600, 1702, 1806) in a telecommunications core network, whereby execution of the program code causes the program code to perform operations comprising: associate different features with a data sample in an aligned order for respective participants in the telecommunication core network where the respective participants respectively comprise a respective collaborative learning active participant model; and perform a training process comprising at least one of (I) accessing a respective output of the respective collaborative learning active participant models trained with the different features and concatenating the respective outputs into a concatenated input to the ML model, and (ii) provisioning a collaborative learning passive participant model to a ML model consumer for inference using the concatenated input provided by the respective participants .

26. The non-transitory computer readable medium (1604) of Claim 25, the operations further comprising any of the operations of Claims 2 to 22.

Images