Verification scheme for machine learning model credibility evaluation in wireless communication system

Abstract

A method of wireless communication is provided. The method includes: receiving, by a first NWDAF entity including an analytic logic function (AnLF) from a consumer entity, a subscription request including consumer desired parameters; sending, by the first NWDAF entity, a verification request to a second NWDAF entity including a model training logic function (MTLF) to verify a machine learning (ML) model of the MTLF; receiving, by the first NWDAF entity from the second NWDAF entity, a verification response including the verification result; comparing the verification result with a consumer expectation parameter; and sending a comparison result to the consumer entity and the second NWDAF entity based on the comparison.

Description

Technical Field

[0001] This document relates to systems, devices, and technologies used for wireless communication. Background Technology

[0002] Wireless communication technology is driving the world towards an increasingly interconnected and networked society. The rapid growth and technological advancements in wireless communication have led to greater demands for capacity and connectivity. Other factors, such as energy consumption, equipment cost, spectrum efficiency, and latency, are also crucial for meeting the needs of various communication scenarios. Compared to existing wireless networks, next-generation systems and wireless communication technologies need to support a growing number of users and devices, and support an increasingly mobile society. Summary of the Invention

[0003] Various methods and apparatuses are provided for verifying the credibility of machine learning models in wireless communication systems.

[0004] In one example aspect, a method for wireless communication is disclosed. The method includes: a first NWDAF entity including an Analysis Logic Function (AnLF) receiving a subscription request including consumer expectation parameters from a consumer entity; the first NWDAF entity sending a verification request to a second NWDAF entity including a Model Training Logic Function (MTLF) to verify a machine learning (ML) model of the MTLF; the first NWDAF entity receiving a verification response including a verification result from the second NWDAF entity; comparing the verification result with the consumer expectation parameters; and sending a comparison result to the consumer entity and the second NWDAF entity based on the comparison.

[0005] In another example aspect, a method for wireless communication is disclosed. The method includes: a second NWDAF entity including a Model Training Logic Function (MTLF) receiving a verification request from a first NWDAF entity including an Analysis Logic Function (AnLF) to perform verification of a machine learning (ML) model; the second NWDAF entity performing the verification of the ML model; the second NWDAF entity sending a verification response including a verification result to the first NWDAF entity; the second NWDAF entity receiving feedback from the first NWDAF entity regarding the verification result; and retraining the ML model or selecting another ML model based on the feedback.

[0006] In another example aspect, a method for wireless communication is disclosed. The method includes: a consumer entity sending a subscription request for analytics to a first NWDAF entity including an Analytics Logic Function (AnLF), the subscription request for analytics including consumer expectation parameters, the consumer expectation parameters including at least one of: a plurality of validation metrics, a minimum acceptable threshold for each validation metric, or an accuracy reporting threshold parameter; the consumer entity receiving a validation check result from the first NWDAF entity, the validation check result indicating whether a machine learning (ML) model of a second NWDAF entity including a Model Training Logic Function (MTLF) satisfies the consumer expectation parameters, wherein the validation check result is obtained by comparing a validation result obtained from the second NWDAF entity with the consumer expectation parameters.

[0007] In another example aspect, a method for wireless communication is disclosed. The method includes: receiving a subscription request from a service consumer entity by a server NWDAF entity to retrieve a machine learning (ML) model, the ML model including ML model metrics selected by the service consumer entity, consumer expectation parameters of the ML model metrics, and a validation method for evaluating the ML model; sending one or more requests to one or more NWDAF entities containing MTLF to participate in joint learning as a joint learning client along with local ML model metrics; receiving a temporary model from the joint learning client having local model information, the local model information including numerical values ​​of the local ML model metrics; sending a request to the service consumer entity to obtain a validation dataset; receiving the validation dataset from the service consumer entity; performing model validation based on the validation method included in the subscription request or based on its local configuration to calculate a global ML model metric, or calculating a global ML model metric based on the local ML model metric; sending a message indicating the training status to the service consumer entity based on the model validation; and updating the global ML model metric to the service consumer entity.

[0008] In another example aspect, a method for wireless communication is disclosed. The method includes: a client NWDAF receiving a request from a server NWDAF to participate in joint learning to perform local model training and determine temporary local ML model information, the request including ML model metrics; calculating numerical values ​​for the local ML model metrics; and sending a temporary model with local model information, including numerical values ​​for the local ML model metrics, to the server NWDAF.

[0009] In another example aspect, a method for wireless communication is disclosed. The method includes: a service consumer entity sending a subscription request to a server NWDAF entity, the subscription request including ML (machine learning) model metrics and a validation method for the ML model, and consumer expectation parameters of the ML model metrics; receiving a message from the server NWDAF entity indicating the training status of the ML model; and determining, based on the global ML model metrics received from the server NWDAF entity, whether to stop or continue the training process of the ML model.

[0010] In yet another example aspect, a wireless communication device including a processor is disclosed. The processor is configured to implement the methods described herein.

[0011] In another example, the various techniques described herein can be embodied in processor-executable code and stored on a computer-readable program medium.

[0012] Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the specification, the drawings, and the claims. Attached Figure Description

[0013] Figure 1 An example architecture of a 5G system is shown.

[0014] Figure 2 An example data collection architecture for NWDAF used for data analysis and model training is shown.

[0015] Figure 3 An example network data analytics open architecture for NF consumers is shown.

[0016] Figure 4 An example data collection architecture for NWDAF used for data analysis and model training is shown.

[0017] Figure 5 An example network data analytics open architecture for NF consumers is shown.

[0018] Figure 6 An example verification process in general model training based on some implementations of the disclosed technology is shown.

[0019] Figure 7 An example verification process in joint learning among multiple NWDAFs based on some implementations of the disclosed technology is shown.

[0020] Figure 8 An example communication device based on some implementations of the disclosed technology is shown.

[0021] Figure 9-14This is an example flowchart of a wireless communication method based on some implementations of the disclosed technology. Detailed Implementation

[0022] The disclosed techniques provide implementation methods and examples for verifying the credibility of machine learning models in wireless communication systems.

[0023] In current machine learning systems for 5GS (5G system), the reliability of trained machine learning models in general training and joint learning is not well evaluated before generating analytical outputs for service consumers. Therefore, the predicted outputs of ML models may deviate from the baseline real data more than consumers expect, leading to inappropriate behaviors in the 5GC (5G core network) (e.g., the application of inappropriate policy control schemes). On the other hand, ML metrics for classification models are not comprehensive, and regression models within the 3GPP (3rd Generation Partnership Project) CT3 (core network and terminal 3) scope also lack ML metrics. This disclosure introduces a new validation process before and after the model begins generating analytical outputs, which will reduce the number of "incorrect decisions" triggered by ML model inference outputs. Furthermore, more metrics are introduced to rate the ML regression and classification models used for prediction in 5GS.

[0024] Figure 1 This is the existing architecture of 5G systems. Within this architecture, the following functions exist: 1) UE, i.e., User Equipment.

[0025] 2) RAN, or Radio Access Network. The RAN manages radio resources and transmits user data received through N3 to the UE, and also transmits user data from the UE through the N3 interface. The RAN performs mapping between DRB (Dedicated Radio Bearers) and QoS (Quality of Service) streams in the PDU session.

[0026] 3) AMF, or Access and Mobility Management function. This function includes the following: registration management, connection management, reachability management, and mobility management. It also performs access authentication and access authorization. The AMF is a NAS (Non-Access Stratum) secure terminal that relays SM NAS between the UE and the SMF.

[0027] 4) SMF, or Session Management Function, includes the following functions: session establishment, modification, and release; UE IP address allocation and management (including optional authentication functions); selection and control of UP (user plane) functions; and downlink data notification. SMF controls UPF (user plane function) through N4 association. SMF provides UPF with PDR (Packet Detection Rule) to instruct how to detect user data traffic; FAR (Forwarding Action Rule); QER (QoS Enforcement Rule); and URR (Usage Reporting Rule) to instruct UPF how to perform user data traffic forwarding, QoS processing, and usage reporting on user data traffic detected using PDR.

[0028] 5) UPF, or User Plane Function. This function includes the following: acting as an anchor point for intra / inter-Radio Access Technology (RAT) mobility, packet routing and forwarding, traffic usage reporting, user plane QoS processing, downlink packet buffering, and downlink data notification triggering. A GTP-U tunnel is used for the N3 interface between the RAN and the UPF. The GTP-U tunnel is per PDU session. For downlink traffic, the UPF binds the downlink traffic to the QoS flow within the GTP-U tunnel of the PDU session using the FAR received from the SMF. For uplink traffic, the RAN transmits user plane traffic to the QoS flow identified by the UE.

[0029] 6) PCF, or Policy Control Function. The PCF provides QoS policy rules to control plane functions to enforce these rules. One or more PCFs translate AF requests into PCC (Policy and Charging Control) rules applicable to PDU (packet data unit) sessions.

[0030] 7) UDM, or Unified Data Management. UDM performs tasks such as generating 3GPP AKA authentication credentials, access authentication based on subscription data, UE service NF (network function) registration management (e.g., storing AMF for UEs, storing SMF for UE PDU sessions), and subscription management. UDM accesses UDR to retrieve UE subscription data and stores the UE context in the UDR. UDM and UDR can be deployed together.

[0031] NWDAF (Network Data Analytics Function) is a 5GC NF residing in the control plane, performing statistical data analysis (i.e., distribution information of datasets) and machine learning-related tasks within 5GS. NWDAF may interact with different entities for various purposes: - Data collection based on subscriptions to events provided by AMF, SMF, UPF, PCF, UDM, NSACF, AF (directly or via NEF), and OAM; - [Optional] Use DCCF (Data Collection Coordination Function) for analysis and data collection; - Retrieve information from data repositories (e.g., retrieve user-related information via UDM, or retrieve PFD information via NEF (PFDF)). - Data collected from the LCS system for location information; - [Optional] Store and retrieve information from ADRF (Analytics Data Repository Function); - [Optional] Collect analytics and data from MFAF (Messaging Framework Adaptor Function); - Retrieve information about NF (e.g., obtain NF-related information from NRF); - Provide analytics to consumers on demand; - Provides bulk data associated with one or more analytics IDs (identities); - Provides information on the accuracy of the analysis ID; - Provides information about the accuracy of the ML model or the downgrade of the ML model's accuracy.

[0032] A single instance or multiple instances of NWDAF can be deployed in a PLMN (Public Land Mobile Network). NWDAF may contain the following logical functions: - Analysis Logic Function (AnLF): A logic function in NWDAF that performs inference, derives analytical information (e.g., derives statistics and / or forecasts based on Analytics Consumer Requests), and exposes analytics services, namely Nnwdaf_AnalyticsSubscription (NWDAF Analytics Subscription) or Nnwdaf_AnalyticsInfo (NWDAF Analytics Info).

[0033] - Model Training Logic Function (MTLF): A logic function in NWDAF used to train machine learning (ML) models and open up new training services (e.g., providing pre-trained ML models).

[0034] NWDAF can contain MTLF or AnLF or both of these logical functions.

[0035] The Data Collection Coordination and Function (DCCF) is also an NF on the 5GC control plane. DCCF coordinates the collection and distribution of data requested by NF consumers. It prevents data sources from having to process multiple subscriptions for the same data and from sending multiple notifications containing the same information due to incoordination of data consumer requests.

[0036] DCCF is applicable to: - NWDAF that requests data from a data source (e.g., for computational analysis).

[0037] - NF consumers requesting analysis from the NWDAF data source.

[0038] - An NF consumer that requests data from an ADRF data source.

[0039] - ADRF that receives data from NF data sources.

[0040] Figure 2 and Figure 4 Two data collection architectures for NWDAF are shown to enable data analysis and model training. Figure 3 and Figure 5 Two possible open architectures for network data analysis are shown for any NF consumer that subscribes to or requests analysis.

[0041] This section describes model performance metrics for classification tasks relevant to this disclosure, including accuracy, F1 score, and ROC curve. These metrics are constructed based on four types of classification results: true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). mlModelAcc (minimum value 0, maximum value 100, data type Uinteger), which indicates the accuracy of the ML model as a percentage, is widely considered a metric for predicting model effectiveness and can be calculated using the following formula:

[0042] mlModelPre expresses the precision of an ML model as a percentage (minimum value equals 0, maximum value equals 100, data type is Uinteger), measuring the percentage of correct results among all positive predictions made by the model. It describes the model's ability to predict positive classifications. Precision can be calculated using the following formula:

[0043] On the other hand, mlModelRec expresses the recall of the ML model as a percentage (minimum equal to 0, maximum equal to 100, data type Uinteger), measuring the percentage of correctly predicted results out of all positive results, as shown below:

[0044] The mlModelF1, which combines precision and recall and indicates the F1 score of an ML model as a percentage (minimum value equals 0, maximum value equals 100, data type is Uinteger), is generated according to the following formula:

[0045] Generally, there is a trade-off between precision and recall: higher precision usually results in lower recall. Therefore, to better evaluate a model, the F1 score is often considered. For multi-class tasks, micro-F1 is calculated by considering the total number of TP, FN, and FP. In contrast, macro-F1 is calculated by averaging the F1 scores across all classes.

[0046] mlModelCm represents the confusion matrix of an ML model, a tabular summary of the number of correct and incorrect predictions made by the classifier. It is formatted as an array. Each entry in the array should have attributes with a minimum value of 0, a maximum value equal to the number of data samples, and a data type of Uinteger. The confusion matrix shows how the classification model confuses itself when making predictions. It can be used to calculate performance metrics such as accuracy, precision, recall, and F1 score.

[0047] The Receiver Operating Characteristic (ROC) curve is a method for comparing diagnostic tests. It plots the relationship between the true positive rate and the false positive rate. mlRoc_Auc (minimum 50, maximum 100, data type Uinteger) represents the area under the ROC curve as a percentage and is also a widely used performance metric in binary classification tasks. It can serve as a summary of the model's sensitivity and specificity.

[0048] Three error metrics are commonly used to evaluate and report the performance of regression models: mlModelMSE (data type double (double precision) or float (single precision)) represents the mean squared error (MSE) of a regression ML model, which can be calculated using the following formula:

[0049] mlModelRMSE (data type double or float) represents the root mean squared error (RMSE) of a regression ML model, which can be calculated using the following formula:

[0050] mlModelMAE represents the Mean Absolute Error (MAE) of a regression ML model (data type double or float), which can be calculated using the following formula:

[0051] mlModelMAPE represents the Mean Absolute Percentage Error (MAPE) of the regression model (data type double or float), which can be calculated using the following formula:

[0052] mlModelRsquare represents the goodness of fit of the regression model (data type double or float), and can be calculated using the following formula:

[0053] Where n is the number of data points in the sample, and Y is the observed value (the baseline real data). This is the predicted output, and ym is the average value of the data sample.

[0054] The parameters mentioned above can be included in ML model metrics or accuracy report thresholds to evaluate ML models from different perspectives. ML model metrics can be used in the Nnwdaf_MLModelTrainingInfo_Request and Nnwdaf_MLModelMonitor_Notify service operations for the federated learning (FL) process of NWDAF and general model training / validation of MTLF.

[0055] In regression and classification machine learning tasks, validation is a mandatory process for evaluating the reliability of a trained model before invocation (e.g., to avoid overfitting; to ensure generalization ability). Model validation refers to the process of evaluating a trained model using a test dataset. A test dataset consists of different bits from a similar dataset from which the training set is inferred. The primary reason for using a test dataset is to test the inferential ability of the prepared model. Some commonly used validation techniques include K-fold cross-validation, leave-one-out cross-validation, random subsampling, and bootstrap ML validation. The output of the validation process includes the aforementioned model performance metrics.

[0056] Implementation Method—Validation Process in General Model Training Figure 6 An example verification process in general model training based on some implementations of the disclosed technology is shown. For example... Figure 6 The example validation process shown corresponds to the analysis and accuracy open process. Each step of the validation process will be discussed in detail below: 1. NWDAF service consumers use AnLF to select the appropriate NWDAF and subscribe to, modify, or unsubscribe from analytics accuracy information by invoking the Nnwdaf_AnalyticsSubscription_Subscribe service operation. If the subscription is not the initial subscription request, it may include analytics feedback information. Multiple validation metrics requested by NWDAF service consumers (e.g., NF, OAM (Operation, Administration, and Maintenance), UE) may include minimum acceptable thresholds for each validation metric (which may include accuracy, precision, recall, F1 score, area under the ROC curve, and confusion matrix for classification tasks; MSE, RMSE, MAE, MAPE, and R-squared for regression tasks). Accuracy reporting thresholds defined by the consumer may also include metrics such as accuracy, precision, recall, F1 score, and confusion matrix for classification tasks; and MSE, RMSE, and MAE for regression tasks.

[0057] The NWDAF service consumer can specify details of the validation method (e.g., 5x cross-validation, the proportion of the validation dataset in the training dataset). If no NF consumer is specified, MTLF can use its default validation method based on the local configuration.

[0058] 2. The NWDAF containing AnLF sends a validation request to the NWDAF containing MTLF (model producer). This request includes a unique model ID, analysis ID, validation method, and validation metrics specified by the consumer. If the NWDAF containing MTLF does not have validation results, it will begin validating the model using the validation method provided by the NF consumer and then send back the required validation parameters.

[0059] Note: The NWDAF containing MTLF does not know the minimum acceptable threshold of the verification metrics provided by the NF consumer in step 2.

[0060] 3. The NWDAF containing AnLF begins analyzing accuracy monitoring and generates analysis accuracy information associated with the analysis ID indicated in the subscription. To generate analysis accuracy information, the NWDAF containing AnLF compares the minimum acceptable threshold of validation metrics received from the consumer (which may include accuracy, precision, recall, F1 score, area under the ROC curve, and confusion matrix for classification tasks; MSE, RMSE, MAE, MAPE, and R-squared value for regression tasks) with the corresponding validation results received from the NWDAF containing MTLF.

[0061] If the expected results are not met, the NWDAF sends a "failure message" to both the service consumer and the MTLF, containing a series of validation metrics that failed to meet the consumer's expectations, along with the percentage of absolute difference between the expected and actual validation values ​​(e.g., the percentage by which the model needs improvement). The MTLF can then retrain the model based on the feedback. An NWDAF containing the AnLF can use the analytics ID to reselect a suitable NWDAF with the MTLF from the NRF and repeat steps 2 and 3 for validation checks.

[0062] If the comparison result meets the consumer's expectations, the NWDAF with AnLF sends a success notification to both the NWDAF with MTLF and the NWDAF service consumer. The model can then be used to predict outputs in real-world scenarios online and subsequently tested with benchmark real data.

[0063] 4a. After the verification process, the NWDAF containing AnLF performs data collection on the subscribed analytics ID and generates analytics output.

[0064] 4b. The NWDAF containing AnLF performs data collection (e.g., baseline real data collection) to generate accuracy information for the subscription analysis ID and related analysis accuracy information. Analysis accuracy information may include accuracy, precision, recall, F1 score, area under the ROC curve, and confusion matrix for classification tasks; and MSE, RMSE, MAE, MAPE, and R-squared value for regression tasks. If analysis feedback information is included in step 1, the NWDAF containing AnLF may consider this information and determine, through internal logic, whether it affects the baseline real data to generate analysis accuracy information.

[0065] 5a. When the subscription in step 1 does not contain analysis accuracy request information, the NWDAF containing AnLF provides analysis output based on the parameters defined in the analysis report information included in the subscription request.

[0066] 5b. NWDAF containing AnLF provides analysis accuracy information and analysis output for the analysis ID based on the parameters defined in the analysis accuracy request information included in the subscription request.

[0067] 5c. NWDAF containing AnLF provides analysis accuracy information only for the analysis ID, based on the parameters defined in the analysis accuracy request information included in the subscription request.

[0068] When the period for providing analysis accuracy information, as indicated in the analysis accuracy request message, differs from the period for providing analysis output, as indicated in the subscription request, the analysis accuracy information is provided in a separate notification.

[0069] 6. When the accuracy of the analysis ID is determined to be low or insufficient, for example, when the deviation between the output analysis using a trained ML model and the baseline real data (which is collected from the data producer NF and corresponds to the analysis ID requested at the time the prediction is indicated) exceeds the reporting thresholds (which are configured locally or received in the subscription request), the NWDAF containing AnLF can notify the NWDAF service consumer of the stop analysis output consumption instruction and the stop analysis output consumption time window.

[0070] 7. NWDAF service consumers can decide to stop consuming analytics output based on their own logic or based on a notification received from NWDAF with an instruction to stop consuming analytics output, without unsubscribing to the analytics ID. The NWDAF service consumer invokes the Nnwdaf_AnalyticsSubscription_Subscribe service operation (including subscribing to the relevance ID) to modify the existing subscription, providing the PauseAnalytics subscription flag in the analytics accuracy request information. Simultaneously, the NWDAF service consumer can choose to reselect NWDAF to obtain analytics information.

[0071] 8a. If the NWDAF service consumer decides to stop consuming analytics output, the AnLF containing the NWDAF can notify the MTLF containing the NWDAF that the model quality has degraded, with accuracy falling below the consumer's expectations. The MTLF can then decide to retrain the model or use a new model. The model retention here differs from the model retraining mentioned in step 3. The retraining in step 8a occurs because the service consumer decides to stop consuming analytics due to the deviation between the output analytics using the trained ML model and the baseline real data (collected from the data producer NF, corresponding to the analytics ID requested at the predicted time) exceeding one or more reporting thresholds. The retraining in step 3, however, is based on feedback from the validation check performed by the NWDAF containing the AnLF.

[0072] 8b. When the MTLF is confident in the model, it can send a request to the AnLF to initiate the verification process described in step 3.

[0073] 9. Based on the verification results provided by MTLF and the comparison results of AnLF, NWDAF service consumers can resume the consumption of analysis output.

[0074] Implementation Method—Validation Process in Joint Learning Among Multiple NWDAFs Figure 7 An example verification process in joint learning among multiple NWDAFs based on some implementations of the disclosed technology is illustrated. Figure 7The example validation process shown corresponds to the analysis and accuracy open process. Each step of the validation process will be discussed in detail below: 0. Consumers (NWDAFs containing AnLF or NWDAFs containing MTLF) use the Nnwdaf_MLModelProvision service to send subscription requests to the FL server NWDAF to retrieve ML models, including the analysis ID, ML model metrics (which may include accuracy, precision, recall, F-score, area under the ROC curve, and confusion matrix for classification tasks; and MSE, RMSE, MAE, MAPE, and R-squared values ​​for regression tasks), accuracy reporting intervals, and reservation status (ML model accuracy threshold or time required for the ML model). NWDAF service consumers can specify details of the validation method (e.g., 5x cross-validation). If no NF consumer is specified, MTLF can use its default validation method based on its local configuration.

[0075] The table below shows an example of ML model metrics. The text edits relative to the current draft of the 3GPP specification are highlighted (including bold) in the table below.

[0076]

[0077] The ML model accuracy threshold can be used to indicate the target ML model accuracy during the training process. When the ML model accuracy threshold is reached during training, the FL server NWDAF can stop the training process.

[0078] If a consumer (e.g., an NWDAF containing AnLF or an NWDAF containing MTLF) provides the time required for the ML model, the FL server NWDAF can take this information into account to determine the maximum response time for its (or one's) FL client NWDAFs.

[0079] 1. The FL server NWDAF selects (one or more) NWDAFs containing MTLF ((one or more) FL client NWDAFs).

[0080] 2. The FL server NWDAF sends either an Nnwdaf_MLModelTraining_Subscribe or an Nnwdaf_MLModelTrainingInfo_Request to the selected NWDAF containing the MTLF (FL client NWDAF). The selected NWDAF containing the MTLF participates in joint learning to perform local model training and determines provisional local ML model information based on the input parameters (including ML model metrics and the initial ML model) in the request from the FL server NWDAF. This request also includes a maximum response time; before this time, the FL client NWDAF must report the provisional local ML model information to the FL server NWDAF.

[0081] 3. If the client NWDAF has no local data available, each FL client NWDAF will use the current mechanism to collect its local data.

[0082] 4. During the joint learning training process, each FL client NWDAF further trains the ML model provided by the FL server NWDAF based on its own data, and reports temporary local ML model information to the FL server NWDAF in Nnwdaf_MLModelTraining_Notify or Nnwdaf_MLModelTrainingInfo_Response. Nnwdaf_MLModelTraining_Notify or Nnwdaf_MLModelTrainingInfo_Response may also include local ML model metrics calculated by the FL client NWDAF and training input data information in the FL client NWDAF (e.g., the region covered by the dataset, sampling rate, maximum / minimum values ​​of each dimension of the data, etc.).

[0083] During FL training, the ML model sent from the FL client NWDAF to the FL server NWDAF is the information required by the FL server NWDAF to build an aggregate model based on the locally trained ML model.

[0084] If the FL client NWDAF is unable to complete the training of the temporary local ML model within the maximum response time provided by the FL server NWDAF, the FL client NWDAF should send a delay event notification, which includes a delay event indication, an optional reason code (e.g., local ML model training failed, more time is required for local ML model training), and the expected time to complete training before the maximum response time has elapsed (if available).

[0085] 4a. [Optional] If the FL server NWDAF receives a notification / response from the FL client NWDAF that training cannot be completed within the maximum response time, the FL server NWDAF may send an extended maximum response time to the FL client NWDAF in `Nnwdaf_MLModelTraining_Subscribe` or `Nnwdaf_MLModelTrainingInfo_Request`. Before this, the FL client NWDAF must report temporary local ML model information to the FL server NWDAF. Otherwise, the FL server NWDAF may instruct the FL client NWDAF to skip reporting for this iteration. The FL server NWDAF includes the current iteration epoch ID in the message to indicate that the request is to modify the training parameters for the current iteration epoch. Alternatively, the FL server NWDAF may notify the FL client NWDAF to stop ML model training by sending a termination request and report the current local ML model update.

[0086] 5. The FL server NWDAF aggregates all local ML model information retrieved in step 4 to update the global ML model. The FL server NWDAF can also compute the global ML model metric using the ML model metric parameters provided by the consumer in step 0 (which may include accuracy, precision, recall, F-score, area under the ROC curve, and confusion matrix for classification tasks; MSE, RMSE, MAE, MAPE, and R-squared values ​​for regression tasks) or by applying the global model to the validation dataset (if available). The FL server NWDAF can compute the global model metric using a validation method (e.g., 5x cross-validation) with the ML model metric requested by the consumer in step 0. If the validation dataset is unavailable, the FL server NWDAF can request to collect the validation dataset from the consumer. If the consumer does not specify a validation method in its subscription request in step 0, the NWDAF server can choose a validation method based on its local configuration. The FL server NWDAF can update the global ML model each time the FL client NWDAF provides updated local ML model information as part of FL, or the FL server NWDAF can decide to wait for local ML model information from all FL client NWDAFs before updating the global ML model.

[0087] If the FL server NWDAF provides the FL client NWDAF with a maximum response time in step 2 to provide temporary local ML model information, or provides an extended maximum response time in step 4a, then the FL server NWDAF decides to wait for any FL client NWDAF that has not yet provided its temporary local ML model within the (extended) maximum response time, or simply aggregate the retrieved local ML model information instances to update the global ML model. The FL server NWDAF makes this decision based on a notification / response from the FL client NWDAF, or, if no notification is received, based on its local configuration.

[0088] 6a. [Optional] Based on the consumer request in step 0, the FL server NWDAF sends an Nnwdaf_MLModelProvision_Notify message to the consumer to dynamically update the global ML model metrics periodically (e.g., a certain number of training rounds or every 10 minutes) or when certain predetermined states are reached (e.g., when the ML model accuracy threshold is reached or the training time expires).

[0089] 6b. [Optional] The consumer determines whether the current model meets the requirements, for example, whether the calculated global ML model metrics or the validation results provided in step 0 are satisfactory, and decides to stop or continue the training process. If the consumer decides to continue the training process, but one or more ML model metrics during the validation process are lower than expected, a list of metrics that need improvement and the percentage of absolute difference from the expected values ​​can be provided. The consumer invokes the Nnwdaf_MLModelProvision_Subscribe (ML model provisioning subscription) service operation used in step 0 to stop or continue the training process. If one or more ML model metrics are lower than expected, the consumer can choose to select a new NWDAF server.

[0090] 6c. [Optional] Based on the subscription request sent from the consumer in step 6b, the FL server NWDAF updates or terminates the current FL training process. In step 6b, the FL server NWDAF may update the training model or reselect the client NWDAF based on feedback from the consumer. If the FL server NWDAF receives a request to stop the joint training process in step 6b, steps 7 and 8 are skipped.

[0091] If the FL process continues, the FL server NWDAF determines / reselects the FL client NWDAF and sends Nnwdaf_MLModelTraining_Subscribe or Nnwdaf_MLModelTrainingInfo_Request to the selected FL client NWDAF, which includes aggregated ML model information for the next round of joint training.

[0092] In step 7, each FL client NWDAF updates its own ML model based on the aggregated ML model information distributed by the FL server NWDAF.

[0093] Note: Steps 3-8 need to be repeated until the training termination condition is met (e.g., the maximum number of iterations, or the result of the loss function is below a threshold, or the validation result of the ML model metrics meets consumer expectations).

[0094] The following is a summary of some characteristics of the corresponding elements: NWDAF with MTLF: - Receive a verification request from AnLF, perform the verification using the method specified in the request, and respond to the request with the verification result.

[0095] - Verify the model using the verification method provided by the consumer. If the service consumer does not specify a verification method, it will perform verification based on its local configuration.

[0096] - If the validation results do not meet consumer expectations, or if consumers do not accept the deviation from the baseline real data, the ML model should be re-entered or re-selected based on AnLF feedback. If MTLF is confident in the model, it should send a request to AnLF to initiate the validation process.

[0097] NWDAF with AnLF: - Receive subscription requests from consumers containing details of the verification method, and send verification requests to MTLF accordingly.

[0098] - Compare the verification results received by MTLF with the consumer's expectations. Send the comparison results and verification feedback to the NWDAF and analytics service consumer that includes MTLF.

[0099] - If the verification result does not meet the consumer's expectations, a message is sent to the NRF to perform an MTLF reselection.

[0100] NWDAF service consumers (e.g., UE, NF, OAM): - In the subscription request, send validation metrics (which may include accuracy, precision, recall, F-score, area under the ROC curve, and confusion matrix for classification tasks; and MSE, RMSE, MAE, MAPE, and R-squared values ​​for regression tasks) and a minimum acceptable threshold for each validation metric to the NWDAF containing AnLF.

[0101] - Send accuracy report threshold parameters (which may include accuracy, precision, recall, F1 score, area under the ROC curve, and confusion matrix for classification tasks; and MSE, RMSE, and MAE for regression tasks) to NWDAF containing AnLF in the subscription request.

[0102] - Specify the verification method used by NWDAF with AnLF in detail in the subscription request.

[0103] - If the difference between the prediction and the baseline real data is greater than the accuracy threshold, a request is sent to the NRF to perform NWDAF and AnLF reselection.

[0104] Service consumers (e.g., UE, 5GC NF, OAM) in the FL learning process of multiple NWDAFs: - Send ML model metrics (which may include accuracy, precision, recall, F-score, area under the ROC curve, and confusion matrix for classification tasks; and MSE, RMSE, MAE, MAPE, and R-squared values ​​for regression tasks) and the specified validation method to the NWDAF server in the subscription request message.

[0105] - If the calculated validation results or global model metrics are lower than expected, you can request the NRF to reselect the NWDAF and send a subscription request to the newly selected NWDAF server.

[0106] - If the server requests NWDAF, the verification dataset will be sent to the server based on NWDAF.

[0107] - Send verification result feedback (e.g., the absolute difference of each metric) to the NWDAF server.

[0108] FL Learning Server NWDAF: - Request the consumer to send the verification dataset.

[0109] - Send the validation results generated based on ML model metrics to consumers.

[0110] If the validated ML model metrics fall short of consumer expectations, the FL server can send a request to the NRF to reselect an FL client to train the model.

[0111] - Repeat the training process until the validation results of the machine learning model metrics meet consumer expectations.

[0112] - Capable of performing verification.

[0113] - Calculate global ML model metrics.

[0114] FL learning client NWDAF: - Calculate ML local model metrics Figure 8 This is a block diagram representation of a portion of a device according to some embodiments of the present disclosure. Device 810, such as a network device, base station, or wireless device (or UE), may include processor electronics 820 (e.g., a microprocessor implementing one or more technologies presented in this document). Device 810 may include transceiver electronics 830 for transmitting and / or receiving wireless signals via one or more communication interfaces, such as antenna 840. Device 810 may include other communication interfaces for transmitting and receiving data. Device 810 may include one or more memories (not explicitly shown) configured to store information such as data and / or instructions. In some embodiments, processor electronics 820 may include at least a portion of transceiver electronics 830. In some embodiments, at least some of the disclosed technologies, modules, or functions are implemented using device 810.

[0115] Some preferred embodiments may include the following solutions.

[0116] 1. A method for wireless communication (e.g., such as...) Figure 9 The method shown (900) includes: receiving (910) a subscription request including consumer expectation parameters from a consumer entity by a first NWDAF entity including an analysis logic function (AnLF); sending (920) a verification request from the first NWDAF entity to a second NWDAF entity including a model training logic function (MTLF) to verify the machine learning (ML) model of the MTLF; receiving (930) a verification response including verification results from the second NWDAF entity by the first NWDAF entity; comparing the verification results with the consumer expectation parameters (9410); and sending (950) a comparison result to the consumer entity and the second NWDAF entity based on the comparison.

[0117] 2. The method according to Solution 1, wherein the consumer expectation parameter includes at least one of a plurality of verification metrics or accuracy reporting threshold parameters, the plurality of verification metrics including a minimum acceptable threshold for each verification metric.

[0118] 3. The method according to Solution 1, wherein the subscription request specifies the validation method used by the second NWDAF entity when validating the ML model.

[0119] 4. The method according to Solution 1, wherein the verification request includes at least one of the ML model's model ID, analysis ID, verification method, or multiple verification metrics, and the verification method and multiple verification metrics are included in the subscription request.

[0120] 5. The method according to Solution 1, wherein sending the comparison result includes: sending a failure message if the comparison result does not meet the consumer expectation parameter, the failure message including a list of verification metrics that do not meet the consumer expectation parameter and a percentage of the absolute difference between the consumer expectation parameter and the actual verification value.

[0121] 6. The method according to Solution 5 further includes: sending a message to the NRF to perform a selection of another second NWDAF including the Model Training Logic Function (MTLF).

[0122] 7. According to the method described in Solution 1, sending the comparison result includes: sending a success message if the comparison result meets the consumer's expected parameters.

[0123] 8. The method according to Solution 1 further includes: performing data collection and using an ML model to generate analysis output for the subscribed analysis ID.

[0124] 9. The method according to Solution 1 further includes: performing data collection and generating accuracy information for the subscribed analytics ID, the accuracy information including at least one of the following: accuracy for a classification task, precision for a classification task, recall for a classification task, F-score for a classification task, area under the ROC (Receiver Operating Characteristic) curve for a classification task, confusion matrix for a classification task, MSE (mean squared error) for a regression task, RMSE (root mean squared error) for a regression task, MAE (mean absolute error) for a regression task, MAPE (mean absolute percentage error) for a regression task, or R-squared value for a regression task.

[0125] 10. A method for wireless communication (e.g., Figure 10The method 1000 shown includes: receiving (1010) a verification request from a first NWDAF entity including an analysis logic function (AnLF) by a second NWDAF entity including a model training logic function (MTLF) to perform verification of a machine learning (ML) model; performing (1020) verification of the ML model by the second NWDAF entity; sending a verification response including (1030) verification results to the first NWDAF entity by the second NWDAF entity; receiving (1040) feedback on the verification results by the second NWDAF entity from the first NWDAF entity; and retraining (1050) the ML model or selecting another ML model based on the feedback.

[0126] 11. The method according to Solution 10, wherein if the feedback indicates that the verification result does not meet the consumer expectation parameter or the deviation from the baseline real data is greater than the accuracy reporting threshold parameter, another ML model is selected.

[0127] 12. The method according to solution 10 further includes: receiving a notification of quality degradation of the ML model from a first NWDAF entity, wherein retraining of the ML model or selection of another ML model is also based on the notification.

[0128] 13. The method according to solution 12 further includes: sending a request to the first NWDAF entity to perform another verification of the ML model if the second NWDAF entity is confident in the quality of the ML model.

[0129] 14. A method for wireless communication (e.g., Figure 11 The method 1100 shown includes: a consumer entity sending (1110) a subscription request for analytics to a first NWDAF entity including an analytics logic function (AnLF), the subscription request for analytics including consumer expectation parameters, the consumer expectation parameters including at least one of the following: a plurality of validation metrics, a minimum acceptable threshold for each validation metric, or an accuracy reporting threshold parameter; the consumer entity receiving (1120) a validation check result from the first NWDAF entity, the validation check result indicating whether a machine learning (ML) model of a second NWDAF entity including a model training logic function (MTLF) satisfies the consumer expectation parameters, wherein the validation check result is obtained by comparing the validation result obtained from the second NWDAF entity with the consumer expectation parameters.

[0130] 15. The method according to Solution 14, wherein at least one of the plurality of validation metrics or the accuracy reporting threshold parameter includes at least one of the following: accuracy for a classification task, precision for a classification task, recall for a classification task, F-score for a classification task, area under the ROC (Receiver Operating Characteristic) curve for a classification task, confusion matrix for a classification task, MSE (mean squared error) for a regression task, RMSE (root mean squared error) for a regression task, MAE (mean absolute error) for a regression task, MAPE (mean absolute percentage error), or R-squared value for a regression task.

[0131] 16. The method according to solution 14, wherein the subscription request specifies the validation method used by the second NWDAF entity when validating the ML model.

[0132] 17. The method according to Solution 14 further includes: sending a request to the NRF to perform a reselection of the Model Training Logic Function (MTLF) if the verification check result is lower than the consumer expectation parameter, or the difference between the prediction and the baseline real data is greater than the accuracy report threshold parameter provided by the consumer entity.

[0133] 18. The method according to solution 14 further includes: determining to stop consuming the analysis output; and determining to select another first NWDAF entity to obtain analysis information.

[0134] 19. A method for wireless communication (e.g., Figure 12 The method 1200 shown includes: receiving (1210) a subscription request from a service consumer entity by a server NWDAF entity to retrieve a machine learning (ML) model, the ML model including ML model metrics selected by the service consumer entity, consumer expectation parameters of the ML model metrics, and a validation method for evaluating the ML model; sending (1220) one or more requests to one or more NWDAF entities containing MTLF to participate in joint learning as a joint learning client along with local ML model metrics; receiving (1230) a temporary model with local model information from the joint learning client, the local model information including the values ​​of the local ML model metrics; sending (1240) a request to the service consumer entity to obtain a validation dataset; receiving (1250) the validation dataset from the service consumer entity; performing (1260) model validation based on the validation method included in the subscription request or based on its local configuration to calculate a global ML model metric, or calculating a global ML model metric based on the local ML model metric; sending (1270) a message indicating the training status to the service consumer entity based on the model validation; and updating (1280) the global ML model metric to the service consumer entity.

[0135] 20. The method according to solution 19 further includes: receiving feedback from a service consumer entity regarding whether to stop or continue the training process of the ML model; and updating the training process based on the feedback or selecting another NWDAF entity containing the MTLF.

[0136] 21. The method according to Solution 19 further includes: sending a request to the NRF to select another NWDAF entity containing the MTLF as a federated learning server to train the ML model.

[0137] 22. The method according to Solution 19 further includes: repeating the training process until the validation results of the ML model metrics meet the consumer's expected parameters.

[0138] 23. The method according to Solution 19, wherein the ML model metrics include at least one of the following: mlModelAcc (accuracy of the ML model) for classification tasks, mlModelPre (precision of the ML model) for classification tasks, mlModelRec (recall of the ML model) for classification tasks, mlModelF1 (F1 score of the ML model) for classification tasks, mlModelRoc_AUC (area under receiver operating characteristics of the ML model) for classification tasks, mlModelCm (confusion matrix of the ML model) for classification tasks; mlModelMse (mean squared error of the ML model) for regression tasks, mlModelRmse (root mean squared error of the ML model) for regression tasks, mlModelMae (mean absolute error of the ML model) for regression tasks, mlModelMape (mean absolute percentage error of the ML model) for regression tasks, or mlModelRsqure (R-squared value of the ML model) for regression tasks.

[0139] 24. A method for wireless communication (e.g., Figure 13 The method 1300 shown includes: receiving (1310) a request from the client NWDAF to participate in joint learning to perform local model training and determine temporary local ML model information, the request including ML model metrics; calculating (1320) the values ​​of the local ML model metrics; and sending (1330) a temporary model with local model information, including the values ​​of the local ML model metrics, to the server NWDAF.

[0140] 25. The method according to solution 24 further includes: collecting local data from at least one NF.

[0141] 26. A method for wireless communication (e.g., Figure 14 The method 1400 shown includes: sending a subscription request (1410) from a service consumer entity to a server NWDAF entity, the subscription request including ML (machine learning) model metrics and a validation method for the ML model, and consumer expectation parameters of the ML model metrics; receiving (1420) a message from the server NWDAF entity indicating the training status of the ML model; and determining (1430) whether to stop or continue the training process of the ML model based on the global ML model metrics received from the server NWDAF entity.

[0142] 27. The method according to solution 26 further includes: providing a list of ML model metrics and a percentage of the absolute difference from the consumer expected parameters if it is determined that the training process continues and one or more ML model metrics are below the consumer expected parameters.

[0143] 28. The method according to solution 26 further includes: sending a request to the NRF to select another NWDAF with MTLF as the federated learning server if one or more ML model metrics or the validation result of the ML model is lower than the consumer expectation parameter.

[0144] 29. The method according to any one of the above solutions, wherein the validation metric or ML model metric includes at least one of the following: F1 score, ROC curve or confusion matrix, wherein the validation metric or ML model metric is constructed based on four types of classification results, namely true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN).

[0145] 30. The method according to Solution 29, wherein the accuracy (mlModelAcc) of the ML model is expressed as a percentage (minimum value equal to 0; maximum value equal to 100; data type is unsigned integer; cardinality range is between 0 and 1), and the mlModelAcc is obtained by the following formula: .

[0146] 31. The method according to Solution 29, wherein the precision (mlModelPre) of the ML model is expressed as a percentage (minimum equal to 0; maximum equal to 100; data type is unsigned integer; cardinality range is between 0 and 1), the mlModelPre measuring the percentage of correct results among all positive predictions made by the ML model, and the recall (mlModelRec) of the ML model is expressed as a percentage (minimum equal to 0; maximum equal to 1100; data type is unsigned integer; cardinality range is between 0 and 1), the mlModelRec measuring the percentage of correct predictions among all positive results.

[0147] 32. The method according to solution 31, wherein the accuracy is obtained by the following formula: ,and The recall rate is obtained using the following formula: .

[0148] 33. The method according to Solution 29, wherein the F1 score (mlModelF1) of the ML model is expressed as a percentage (minimum value equal to 0; maximum value equal to 100; data type is unsigned integer; cardinality range is between 0 and 1), and the mlModelF1 is obtained by the following formula:

[0149] 34. The method according to any one of the above solutions, wherein the ML model metrics include a confusion matrix (mlModelCm) of the ML model in array format, and wherein each entry of the array has the following attributes: a minimum value equal to 0, a maximum value equal to the number of data samples, a data type of unsigned integer, and a cardinality ranging from 1 to N.

[0150] 35. The method according to any one of the above solutions, wherein the ML model metric includes the area under the receiver operating characteristic (ROC) curve (mlModelRoc_Auc) expressed as a percentage (minimum value equal to 50; maximum value equal to 100; data type is unsigned integer), with a base range between 0 and 1.

[0151] 36. The method according to any one of the above solutions, wherein the ML model metric includes at least one of the following: MSE (mlModelMse) for a regression task, RMSE (mlModelRmse) for a regression task, MAE (mlModelMae) for a regression task, MAPE (mlModelMape) for a regression task, or R-squared value (mlModelRsquare) for a regression task, and wherein the ML model metric has a cardinality ranging between 0 and 1 and a data type of double or float.

[0152] 37. A wireless communication device comprising a processor configured to implement the method described in any one of the above solutions.

[0153] 38. A computer storage medium having code stored thereon, which, when executed by a processor, causes the processor to implement the method described in any one of the above solutions.

[0154] The disclosed and other embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this document and their equivalents), or in a combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, that is, one or more modules implemented as computer program instructions encoded on a computer-readable medium, said one or more modules being executed by a data processing apparatus or used to control the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a component of a material that implements machine-readable propagation signals, or a combination of one or more of them. The term "data processing apparatus" includes all means, devices, and machines for processing data, such as a programmable processor, a computer, or a plurality of processors or a plurality of computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the mentioned computer program, for example, code that builds processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagating signal is a non-naturally generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0155] A computer program (also referred to as a program, software, software application, script, or code) can be written in any programming language, including compiled or interpreted languages, and can be invoked in any form, including as a standalone program, or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file containing other programs or data (e.g., one or more scripts stored in a markup language file), in a single file dedicated to the program in question, or in multiple co-located files (e.g., multiple files storing one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one or more computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0156] The processes and logic flows described in this document can be executed by one or more programmable processors, which execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuitry, and the device can also be implemented as dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0157] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. Essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operatively coupled to one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, to receive data from or to which data is transferred, or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM (Compact Disc Read-Only Memory) and DVD-ROM (Digital Video Disc Read-Only Memory) disks. Processors and memory may be supplemented by dedicated logic circuitry or incorporated into dedicated logic circuitry.

[0158] Although this document contains numerous details, these details should not be construed as limiting the scope of the claimed invention or any potentially claimed invention, but rather as descriptions of features specific to particular embodiments. Certain features described in the context of different embodiments in this document may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even if initially claimed in this way, in some cases one or more features from the claimed combination may be removed from that combination, and the claimed combination may be for sub-combinations or variations thereof. Similarly, although operations are depicted in a specific order in the drawings, this should not be construed as requiring the operations to be performed in the specific order shown or sequentially, or requiring the performance of all shown operations to achieve the desired result.

[0159] Only some examples and implementations are disclosed. Variations, modifications, and enhancements can be made to the described examples and implementations, as well as other implementations, based on the disclosed content.

Claims

1. A method for wireless communication, comprising: The first NWDAF entity, which includes the Analysis Logic Function (AnLF), receives a subscription request, including consumer expectation parameters, from the consumer entity. The first NWDAF entity sends a verification request to the second NWDAF entity, which includes the Model Training Logic Function (MTLF), to verify the machine learning (ML) model of the MTLF; The first NWDAF entity receives a verification response, including the verification result, from the second NWDAF entity; Compare the verification results with the consumer expectation parameters; as well as The comparison result is sent to the consumer entity and the second NWDAF entity based on the comparison.

2. The method according to claim 1, wherein, The consumer expectation parameter includes at least one of multiple verification metrics or accuracy reporting threshold parameters, wherein the multiple verification metrics include a minimum acceptable threshold for each verification metric.

3. The method according to claim 1, wherein, The subscription request specifies the verification method used by the second NWDAF entity when verifying the ML model.

4. The method according to claim 1, wherein, The verification request includes at least one of the ML model's model ID, analysis ID, verification method, or multiple verification metrics, wherein the verification method and the multiple verification metrics are included in the subscription request.

5. The method according to claim 1, wherein, Sending the comparison result includes: if the comparison result does not meet the consumer expectation parameter, sending a failure message, the failure message including a list of verification metrics that do not meet the consumer expectation parameter and a percentage of the absolute difference between the consumer expectation parameter and the actual verification value.

6. The method according to claim 5, further comprising: Send a message to the NRF to perform the selection of another second NWDAF, including the Model Training Logic Function (MTLF).

7. The method according to claim 1, wherein, Sending the comparison result includes sending a success message if the comparison result meets the consumer's expected parameters.

8. The method according to claim 1, further comprising: Perform data collection and use the ML model to generate analytics output for the subscribed analytics IDs.

9. The method according to claim 1, further comprising: Perform data collection and generate accuracy information for the subscribed analytics IDs, the accuracy information including at least one of the following: accuracy for a classification task, precision for a classification task, recall for a classification task, F-score for a classification task, area under the ROC (Receiver Operating Characteristic) curve for a classification task, confusion matrix for a classification task, MSE (mean squared error) for a regression task, RMSE (root mean squared error) for a regression task, MAE (mean absolute error) for a regression task, MAPE (mean absolute percentage error) for a regression task, or R-squared value for a regression task.

10. A method for wireless communication, comprising: The second NWDAF entity, which includes the Model Training Logic Function (MTLF), receives a verification request from the first NWDAF entity, which includes the Analysis Logic Function (AnLF), to perform verification of the machine learning (ML) model; The validation of the ML model is performed by the second NWDAF entity; The second NWDAF entity sends a verification response, including the verification result, to the first NWDAF entity; The second NWDAF entity receives feedback regarding the verification result from the first NWDAF entity; as well as Based on the feedback, the ML model can be retrained or another ML model can be selected.

11. The method according to claim 10, wherein, If the feedback indicates that the verification result does not meet the consumer expectation parameters or the deviation from the benchmark real data is greater than the accuracy report threshold parameter, the other ML model is selected.

12. The method of claim 10, further comprising: Receive notification of quality degradation of the ML model from the first NWDAF entity. The retraining of the ML model or the selection of another ML model is also based on the notification.

13. The method of claim 12, further comprising: If the second NWDAF entity is confident in the quality of the ML model, it sends a request to the first NWDAF entity to perform another verification of the ML model.

14. A method for wireless communication, comprising: A consumer entity sends a subscription request for analytics to a first NWDAF entity, which includes an Analytics Logic Function (AnLF). The subscription request includes consumer expectation parameters, which include at least one of the following: multiple validation metrics, a minimum acceptable threshold for each validation metric, or an accuracy reporting threshold parameter; and The consumer entity receives a verification check result from the first NWDAF entity. The verification check result indicates whether the machine learning (ML) model of the second NWDAF entity, which includes the Model Training Logic Function (MTLF), meets the consumer expectation parameters. The verification check result is obtained by comparing the verification result obtained from the second NWDAF entity with the consumer expectation parameters.

15. The method according to claim 14, wherein, At least one of the plurality of validation metrics or the accuracy reporting threshold parameters includes at least one of the following: accuracy for a classification task, precision for a classification task, recall for a classification task, F-score for a classification task, area under the ROC (Receiver Operating Characteristic) curve for a classification task, confusion matrix for a classification task, MSE (mean squared error) for a regression task, RMSE (root mean squared error) for a regression task, MAE (mean absolute error), MAPE (mean absolute percentage error) for a regression task, or R-squared value for a regression task.

16. The method of claim 14, wherein, The subscription request specifies the verification method used by the second NWDAF entity when verifying the ML model.

17. The method of claim 14, further comprising: If the verification check result is lower than the consumer expectation parameter, or the difference between the prediction and the baseline real data is greater than the accuracy report threshold parameter provided by the consumer entity, a request is sent to the NRF to perform a reselection of the Model Training Logic Function (MTLF).

18. The method of claim 14, further comprising: Determine to stop consuming the analysis output; as well as Select another first NWDAF entity to obtain analysis information.

19. A method for wireless communication, comprising: The server NWDAF entity receives a subscription request from the service consumer entity to retrieve a machine learning (ML) model, the ML model including ML model metrics selected by the service consumer entity, consumer expectation parameters of the ML model metrics, and a validation method for evaluating the ML model; Send one or more requests to one or more NWDAF entities containing MTLF to participate in federated learning as federated learning clients along with local ML model metrics; Receive a temporary model with local model information from the federated learning client, the local model information including the values ​​of the local ML model metrics; Send a request to the service consumer entity to obtain the verification dataset; Receive the verification dataset from the service consumer entity; Model validation is performed based on the validation method included in the subscription request or based on its local configuration to calculate global ML model metrics, or global ML model metrics are calculated based on the local ML model metrics. Based on the model validation, a message indicating the training status is sent to the service consumer entity; and Update the global ML model metrics to the service consumer entity.

20. The method of claim 19, further comprising: Receive feedback from the service consumer entity regarding whether to stop or continue the training process of the ML model; as well as The training process is updated based on the feedback, or another NWDAF entity containing MTLF is selected.

21. The method of claim 19, further comprising: A request is sent to the NRF to select another NWDAF entity containing MTLF as a federated learning server to train the ML model.

22. The method of claim 19, further comprising: Repeat the training process until the validation results of the ML model metrics reach the consumer's expected parameters.

23. The method according to claim 19, wherein, The ML model metrics include at least one of the following: mlModelAcc (ML model accuracy) for classification tasks, mlModelPre (ML model precision) for classification tasks, mlModelRec (ML model recall) for classification tasks, mlModelF1 (ML model F1 score) for classification tasks, mlModelRoc_AUC (ML model area under receiver operating characteristics) for classification tasks, and mlModelCm (ML model confusion matrix) for classification tasks; mlModelMse (ML model mean squared error) for regression tasks, mlModelRmse (ML model root mean squared error) for regression tasks, mlModelMae (ML model mean absolute error) for regression tasks, mlModelMape (ML model mean absolute percentage error) for regression tasks, or mlModelRsquare (ML model R-squared value) for regression tasks.

24. A method for wireless communication, comprising: The client NWDAF receives a request from the server NWDAF to participate in joint learning, to perform local model training and determine temporary local ML model information, the request including ML model metrics; Calculate the numerical values ​​of the local ML model metrics; and A temporary model with local model information, including the values ​​of the local ML model metrics, is sent to the server NWDAF.

25. The method of claim 24, further comprising: Collect local data from at least one NF.

26. A method for wireless communication, comprising: The service consumer entity sends a subscription request to the server NWDAF entity. The subscription request includes ML (machine learning) model metrics, ML model validation methods, and consumer expectation parameters of the ML model metrics. Receive a message indicating the training status of the ML model from the server NWDAF entity; as well as Based on the global ML model metrics received from the NWDAF server entity, it is determined whether to stop or continue the training process of the ML model.

27. The method of claim 26, further comprising: If it is determined that the training process should continue and one or more ML model metrics are below the consumer expected parameters, a list of ML model metrics and the percentage of the absolute difference from the consumer expected parameters are provided.

28. The method of claim 26, further comprising: If one or more ML model metrics or the validation results of the ML model are lower than the consumer's expected parameters, a request is sent to the NRF to select another NWDAF with MTLF as the federated learning server.

29. The method according to any one of the preceding claims, wherein, The validation metric or the ML model metric includes at least one of the following: F1 score, ROC curve, or confusion matrix. The validation metric or the ML model metric is constructed based on four types of classification results, namely, true positive (TP), true negative (TN), false positive (FP), and false negative (FN).

30. The method according to claim 29, wherein, The accuracy (mlModelAcc) of the ML model is expressed as a percentage (minimum value equals 0; maximum value equals 100; data type is unsigned integer; radix range is between 0 and 1), and mlModelAcc is obtained by the following formula: 。 31. The method according to claim 29, wherein, The precision (mlModelPre) of the ML model is expressed as a percentage (minimum value equal to 0; maximum value equal to 100; data type is unsigned integer; cardinality range is between 0 and 1), where mlModelPre measures the percentage of correct results out of all positive predictions made by the ML model, while the recall (mlModelRec) of the ML model is expressed as a percentage (minimum value equal to 0; maximum value equal to 1100; data type is unsigned integer; cardinality range is between 0 and 1), where mlModelRec measures the percentage of correct predictions out of all positive results.

32. The method according to claim 31, wherein, The accuracy is obtained using the following formula: ,and The recall rate is obtained using the following formula: 。 33. The method according to claim 29, wherein, The F1 score (mlModelF1) of the ML model is expressed as a percentage (minimum value equals 0; maximum value equals 100; data type is unsigned integer; cardinality range is between 0 and 1), and the mlModelF1 is obtained by the following formula: 。 34. The method according to any one of the preceding claims, wherein, The ML model metrics include a confusion matrix (mlModelCm) of the ML model in array format, wherein each entry of the array has the following attributes: minimum value equal to 0, maximum value equal to the number of data samples, data type is unsigned integer, and cardinality range is between 1 and N.

35. The method according to any one of the preceding claims, wherein, The ML model metrics include the area under the receiver operating characteristic (ROC) curve (mlModelRoc_Auc) expressed as a percentage (minimum value equals 50; maximum value equals 100; data type is unsigned integer), with a base range between 0 and 1.

36. The method according to any one of the preceding claims, wherein, The ML model metric includes at least one of the following: MSE (mlModelMse) for a regression task, RMSE (mlModelRmse) for a regression task, MAE (mlModelMae) for a regression task, MAPE (mlModelMape) for a regression task, or R-squared value (mlModelRsquare) for a regression task, wherein the ML model metric has a cardinality ranging between 0 and 1 and a data type of double or float.

37. A wireless communication device comprising a processor configured to implement the method of any one of the preceding claims.

38. A computer storage medium having code stored thereon, which, when executed by a processor, causes the processor to perform the method of any one of the preceding claims.

Images