Communication method, apparatus and system

Abstract

Embodiments of the present application provide a communication method, device and system. The method comprises: a first data analysis network element receiving first information from a second data analysis network element, the first information comprising a performance report of a model; the first data analysis network element updating first model information of the model according to the performance report of the model to obtain second model information of the model; and the first data analysis network element sending second information to the second data analysis network element, the second information comprising the second model information. In this scheme, when the second data analysis network element cannot complete model training, the second data analysis network element sends the performance report of the model to the first data analysis network element, the first data analysis network element updates the model according to the performance report of the model to obtain the second model information of the model, and sends the second model information to the second data analysis network element, so that the second data analysis network element updates the model based on the second model information, thereby achieving timely training and updating of the model when the performance of the model decreases, and further ensuring the performance of the model.

Description

[0001] Cross-references to related applications

[0002] This application is a divisional application. The original application has the application number 202180041882.4 and the original application date is April 2, 2021. The entire contents of the original application are incorporated herein by reference.

[0003] This application claims priority to Chinese Patent Application No. PCT / CN2020 / 117940, filed on September 25, 2020, entitled "Communication Method, Apparatus and System", the entire contents of which are incorporated herein by reference. Technical Field

[0004] This application relates to the field of communication technology, and in particular to communication methods, apparatus and systems. Background Technology

[0005] Machine learning models are typically trained by learning the mapping between a set of input features and the output target. This is achieved by optimizing certain loss functions to minimize the error between the model's output (predicted value) and the actual result (label / true value). Once the optimal model is trained, its output is used to predict future events. Ideally, it's assumed that the data used in the future will be similar to the data used during model training; specifically, it might be assumed that the distribution of the input features during training and prediction remains constant. However, in reality, this assumption usually doesn't hold. Data features change over time due to variations in network deployment, application layer business requirements, and the actual distribution of network users. Therefore, the model's performance (i.e., generalization ability) gradually declines over time. Specifically, this may manifest as a decrease in model accuracy, meaning a larger error between the model's predictions and the true values.

[0006] Taking a scenario where the training and inference functions of a data analysis network element are separated as an example, the data analysis network element supporting the training function (hereinafter referred to as the training data analysis network element) cannot perceive the model usage effect in the data analysis network element supporting the inference function (hereinafter referred to as the inference data analysis network element), and the inference data analysis network element is not capable of model training. Therefore, when model performance degrades, if the inference data analysis network element continues to use the degraded model for data analysis, it will lead to inaccurate data analysis results. Summary of the Invention

[0007] This application provides a communication method, apparatus, and system for retraining the model in a timely manner when the model performance degrades, thereby ensuring model performance.

[0008] In a first aspect, embodiments of this application provide a communication method, comprising: a first data analysis network element receiving first information from a second data analysis network element, the first information including a model performance report, the model performance report being used to indicate the performance evaluation result of the model, or the model performance report being used to indicate that the performance evaluation result of the model does not meet the performance index requirements of the model; the first data analysis network element updating first model information of the model according to the model performance report to obtain second model information of the model; and the first data analysis network element sending second information to the second data analysis network element, the second information including the second model information.

[0009] Based on the above scheme, when the second data analysis network element cannot complete model training, the second data analysis network element can send a model performance report to the first data analysis network element. The first data analysis network element can then update the model based on the performance report, obtain the second model information, and send the second model information to the second data analysis network element. This allows the second data analysis network element to update the model based on the second model information, thereby enabling timely model training when model performance deteriorates and ensuring model performance.

[0010] In one possible implementation, the first data analysis network element sends third information to the second data analysis network element, the third information including the performance indicators of the model, the performance indicators of the model being used to obtain the performance evaluation results of the model.

[0011] Based on the above scheme, the first data analysis network element can send the model's performance metrics to the second data analysis network element in advance, so that the second data analysis network element can generate a model performance report based on the model's performance metrics. This helps the first data analysis network element to determine whether to start model training and improve the model performance after training.

[0012] In one possible implementation, the first data analysis network element sends the second information to the third data analysis network element.

[0013] Based on the above scheme, the first data analysis network element can not only send the second information to the second data analysis network element, but also send the second information to other network elements that use the model, such as the third data analysis network element, so that the third data analysis network element can also use the second model information to update the model and improve the model usage effect.

[0014] In one possible implementation, the first data analysis network element receiving first information from the second data analysis network element includes: the first data analysis network element receiving the first information from the second data analysis network element through a network storage network element. The first data analysis network element sending second information to the second data analysis network element includes: the first data analysis network element sending the second information to the second data analysis network element through a network storage network element.

[0015] Based on the above scheme, a network storage network element can be used as an intermediate network element to realize the model update interaction between the first data analysis network element and the second data analysis network element. This can be applied to scenarios where there is no interface between the first data analysis network element and the second data analysis network element.

[0016] Secondly, embodiments of this application provide a communication method, comprising: a second data analysis network element sending first information to a first data analysis network element, the first information including a model performance report, the model performance report being used to indicate the performance evaluation result of the model, or the model performance report being used to indicate that the performance evaluation result of the model does not meet the performance index requirements of the model; the second data analysis network element receiving second information from the first data analysis network element, the second information including second model information of the model, the second model information being obtained by updating the first model information of the model based on the model performance report; and the second data analysis network element updating the model based on the second model information.

[0017] Based on the above scheme, when the second data analysis network element cannot complete model training, it can send a model performance report to the first data analysis network element. The first data analysis network element can then update the model based on the performance report, obtain the second model information, and send the second model information to the second data analysis network element. This allows the second data analysis network element to update the model based on the second model information, thereby enabling timely retraining of the model when its performance deteriorates, thus ensuring model performance.

[0018] In one possible implementation, the second data analysis network element receives third information from the first data analysis network element, the third information including the performance indicators of the model, the performance indicators of the model being used to obtain the performance evaluation results of the model.

[0019] Based on the above scheme, the first data analysis network element can send the model's performance metrics to the second data analysis network element in advance, so that the second data analysis network element can generate a model performance report based on the model's performance metrics. This helps the first data analysis network element determine whether to start model training and improves the accuracy of model training.

[0020] In one possible implementation, the second data analysis network element sends first information to the first data analysis network element, including: the second data analysis network element sends the first information to the first data analysis network element through a network storage network element; the second data analysis network element receives second information from the first data analysis network element, including: the second data analysis network element receives the second information from the first data analysis network element through a network storage network element.

[0021] Based on the above scheme, a network storage network element can be used as an intermediate network element to realize the model update interaction between the first data analysis network element and the second data analysis network element. This can be applied to scenarios where there is no interface between the first data analysis network element and the second data analysis network element.

[0022] Based on the first aspect, or any implementation of the first aspect, or the second aspect, or any implementation of the second aspect: In one possible implementation, the model performance metrics include one or more of the following: precision, accuracy, error rate, recall, F1 score, mean squared error, root mean squared error, root mean squared logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability.

[0023] In one possible implementation, the third information further includes one or more of the following: an analysis type identifier, an identifier of the model, and an identifier of the sub-model, wherein the analysis type identifier is used to indicate the analysis type of the model.

[0024] In one possible implementation, the third information further includes one or more of the following: a reporting period and threshold information, wherein the reporting period is used to indicate the time for reporting the performance report of the model, and the threshold information is used to indicate the conditions for reporting the performance report of the model.

[0025] Based on the above scheme, the first data analysis network element can instruct the second data analysis network element on the time and / or conditions for reporting the model's performance report, thereby achieving conditional reporting and saving resource consumption.

[0026] In one possible implementation, the first information may also include one or more of the following information corresponding to the performance report of the model: time, region, and slice.

[0027] Based on the above scheme, if the first information also includes the time, region or slice corresponding to the model's performance report, the model performance after the first data analysis network element executes the model for retraining can be improved.

[0028] In one possible implementation, the second information further includes one or more of the following: the identifier of the model, the identifier of the sub-model, the performance evaluation result of the model, the hardware capability information corresponding to the performance evaluation result of the model, the size of the model, and the inference time of the model.

[0029] Based on the above scheme, the first data analysis network element sends one or more of the following to the second data analysis network element: the model's performance evaluation result, the hardware capability information corresponding to the model's performance evaluation result, the model's size, or the model's inference time. This helps the second data analysis network element determine whether to use the model, thereby reducing the waste of resource consumption.

[0030] Thirdly, embodiments of this application provide a communication method, comprising: a first data analysis network element updating first information of a model to second information of the model; the first data analysis network element determining index information of the second information of the model, the index information of the second information including first identification information used to indicate the second information of the model; the first data analysis network element sending the index information of the second information to a second data analysis network element, the index information of the second information being used to acquire the second information of the model. The index information of the second information of the model can also be referred to as the model index information corresponding to the second information.

[0031] Based on the above scheme, the first data analysis network element updates the model and obtains the second information of the model. It can then send the index information of the second information to the second data analysis network element. The second data analysis network element can then obtain the new model information, i.e., the second information, based on the index information. The second data analysis network element can then update the model based on the new model information, thereby improving the model performance.

[0032] In one possible implementation, the index information of the second information further includes one or more of the following: the analysis type identifier corresponding to the model, the identifier of the model, and the version information of the second information of the model.

[0033] In one possible implementation, the first data analysis network element receives index information of the first information of the model from the second data analysis network element. The index information of the first information includes second identification information, which is used to indicate the first information of the model. The first data analysis network element obtains the first information of the model based on the index information of the first information.

[0034] In one possible implementation, the index information of the first information further includes one or more of the following: the analysis type identifier corresponding to the model, the identifier of the model, and the version information of the first information of the model.

[0035] In one possible implementation, the first data analysis network element updates the first information of the model to the second information of the model, including: the first data analysis network element obtaining a first request from the second data analysis network element, the first request being used to update the first information of the model, the first request including index information of the first information of the model; the first data analysis network element obtaining the first information of the model according to the index information of the first information; and the first data analysis network element updating the first information of the model to obtain the second information of the model.

[0036] In one possible implementation, the first data analysis network element receives index information of the first information of the model from the second data analysis network element, including: the first data analysis network element sending a second request to the second data analysis network element, the second request being used to request the index information of the first information of the model, the second request including an analysis type identifier corresponding to the model; the first data analysis network element receiving a second response from the second data analysis network element, the second response including the index information of the first information of the model.

[0037] In one possible implementation, the first data analysis network element receives the index information of the first information of the model from the second data analysis network element through the network storage network element.

[0038] In one possible implementation, the first data analysis network element sends the index information of the second information of the model to the second data analysis network element through the network storage network element.

[0039] In one possible implementation, the first data analysis network element is a client-side data analysis network element in distributed learning, and the second data analysis network element is a server-side data analysis network element in distributed learning.

[0040] In one possible implementation, the distributed learning is federated learning.

[0041] In one possible implementation, the first data analysis network element is a data analysis network element that supports inference functions, and the second data analysis network element is a data analysis network element that supports training functions.

[0042] Fourthly, embodiments of this application provide a communication method, comprising: a second data analysis network element sending a first request to a first data analysis network element, the first request carrying an analysis type identifier and second model requirement information, the first request being used to request index information corresponding to the analysis type identifier and satisfying the second model requirement information; and the second data analysis network element receiving the model index information from the first data analysis network element.

[0043] Based on the above scheme, the second model requirement information can help the second data analysis network element quickly obtain a model from the first data analysis network element that meets the second model requirement information as much as possible, thereby enabling the second data analysis network element to accurately provide a model.

[0044] In one possible implementation, the second data analysis network element sends a network function discovery request to the network storage network element. The network function discovery request includes the analysis type identifier and the first model requirement information. The network function discovery request is used to request a data analysis network element that can provide a model corresponding to the analysis type identifier and satisfying the first model requirement information. The second data analysis network element receives the address information of the first data analysis network element from the network storage network element.

[0045] In one possible implementation, the first model requirement information includes one or more of the following: analysis filtering information, analysis report target information, model performance information, or model deployment environment information; The analysis filtering information is used to indicate the scope of application of the model that the second data analysis network element needs to request. The analysis filtering information includes one or more of the following: region, time period, single network slice selection auxiliary information or data network name. The target information in the analysis report is used to indicate the terminal corresponding to the model that the second data analysis network element needs to request. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model requested by the second data analysis network element. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean square error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment for model deployment that the second data analysis network element needs to request. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

[0046] In one possible implementation, the second model requirement information includes some or all of the information in the first model requirement information.

[0047] In one possible implementation, the second model requirement information includes time information, which indicates the expected time to receive model index information from the first data analysis network element.

[0048] In one possible implementation, the second data analysis network element receives a first indication from the first data analysis network element, the first indication indicating that the model index information cannot be sent within the time period indicated by the time information.

[0049] Fifthly, embodiments of this application provide a communication method, comprising: a first data analysis network element receiving a first request from a second data analysis network element, the first request carrying an analysis type identifier and second model requirement information, the first request being used to request model index information of a model corresponding to the analysis type identifier and satisfying the second model requirement information; the first data analysis network element obtaining the model index information according to the second model requirement information and the analysis type identifier; and the first data analysis network element sending the model index information to the second data analysis network element.

[0050] Based on the above scheme, the second model requirement information can help the second data analysis network element quickly obtain a model from the first data analysis network element that meets the second model requirement information as much as possible, thereby enabling the second data analysis network element to accurately provide a model.

[0051] In one possible implementation, the second model requirement information includes time information, which indicates the expected time to receive model index information from the first data analysis network element.

[0052] In one possible implementation, the first data analysis network element sends a first indication message to the second data analysis network element, the first indication message indicating that the model index information cannot be sent within the time period indicated by the time information.

[0053] In one possible implementation, the first data analysis network element sends a network function registration request to the network storage network element. The network function registration request carries the analysis type identifier and model information. The model information includes second indication information, which indicates whether the model corresponding to the analysis type identifier has been trained or is ready to be used.

[0054] In one possible implementation, when the second indication information indicates that the model corresponding to the analysis type identifier has been trained or is ready to be completed, the model information further includes model description information, which includes one or more of the following: analysis filtering information, analysis report target information, model performance information, or model deployment environment information. The analysis filtering information is used to indicate the scope of application of the model corresponding to the analysis type identifier. The analysis filtering information includes one or more of the following: region, time period, single network slice selection auxiliary information, or data network name. The target information in the analysis report is used to indicate the terminal corresponding to the model corresponding to the analysis type identifier. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model corresponding to the analysis type identifier. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment of the model deployment corresponding to the analysis type identifier. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

[0055] Sixthly, embodiments of this application provide a communication method, comprising: a network storage network element receiving a network function discovery request from a second data analysis network element, the network function discovery request including an analysis type identifier and first model requirement information, the network function discovery request being used to request a data analysis network element capable of providing a model corresponding to the analysis type identifier and satisfying the first model requirement information; the network storage network element obtaining address information of a first data analysis network element based on the first model requirement information and the analysis type identifier; and the network storage network element sending the address information of the first data analysis network element to the second data analysis network element.

[0056] Based on the above scheme, the first model requirement information can help the second data analysis network element quickly obtain a first data analysis network element that meets the first model requirement information as much as possible from the network storage network element, thereby enabling the second data analysis network element to accurately provide the first data analysis network element.

[0057] In one possible implementation, the network storage element receives a network function registration request from the first data analysis element. The network function registration request carries the analysis type identifier and model information. The model information includes second indication information, which indicates whether the model corresponding to the analysis type identifier has been trained or is ready to be used.

[0058] In one possible implementation, when the second indication information indicates that the model corresponding to the analysis type identifier has been trained or is ready to be completed, the model information further includes model description information, which includes one or more of the following: analysis filtering information, analysis report target information, model performance information, or model deployment environment information. The analysis filtering information is used to indicate the scope of application of the model corresponding to the analysis type identifier. The analysis filtering information includes one or more of the following: region, time period, single network slice selection auxiliary information, or data network name. The target information in the analysis report is used to indicate the terminal corresponding to the model corresponding to the analysis type identifier. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model corresponding to the analysis type identifier. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment of the model deployment corresponding to the analysis type identifier. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

[0059] In one possible implementation, the first model requirement information includes one or more of the following: analysis filtering information, analysis report target information, model performance information, or model deployment environment information; The analysis filtering information is used to indicate the scope of application of the model that the second data analysis network element needs to request. The analysis filtering information includes one or more of the following: region, time period, single network slice selection auxiliary information or data network name. The target information in the analysis report is used to indicate the terminal corresponding to the model that the second data analysis network element needs to request. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model requested by the second data analysis network element. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean square error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment for model deployment that the second data analysis network element needs to request. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

[0060] In a seventh aspect, embodiments of this application provide a communication device, which can be a data analysis network element or a chip for a data analysis network element. The device has the function of implementing the first to sixth aspects, or various possible implementation methods of the first to sixth aspects. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0061] Eighthly, embodiments of this application provide a communication device, including a processor and a memory; the memory is used to store computer execution instructions, and when the device is running, the processor executes the computer execution instructions stored in the memory to cause the device to perform any of the methods described in the first to sixth aspects and any possible implementations of the first to sixth aspects.

[0062] Ninthly, embodiments of this application provide a communication apparatus, including units or means for performing the methods of the first to sixth aspects and any steps of any of the possible implementations of the first to sixth aspects.

[0063] In a tenth aspect, embodiments of this application provide a communication device, including a processor and an interface circuit. The processor is configured to communicate with other devices via the interface circuit and execute the methods described in the first to sixth aspects, as well as any possible implementations of the first to sixth aspects. The processor may include one or more processors.

[0064] Eleventhly, embodiments of this application provide a communication device, including a processor for connection to a memory, for calling a program stored in the memory to execute the methods described in the first to sixth aspects and any of the possible implementations of the first to sixth aspects. The memory may be located within or outside the device. The processor may include one or more processors.

[0065] In a twelfth aspect, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause a processor to perform the methods of the first to sixth aspects and any of the possible implementations of the first to sixth aspects.

[0066] In a thirteenth aspect, embodiments of this application also provide a computer program product, the computer product including a computer program that, when the computer program is run, enables any of the methods of the first to sixth aspects and any of the possible implementations of the first to sixth aspects.

[0067] In a fourteenth aspect, embodiments of this application also provide a chip system, including: a processor, configured to execute the methods described in the first to sixth aspects and any of the possible implementations of the first to sixth aspects.

[0068] In a fifteenth aspect, embodiments of this application also provide a communication system, including: a first data analysis network element for performing the first aspect or any implementation method of the first aspect, and a second data analysis network element for performing the second aspect or any implementation method of the second aspect.

[0069] In a sixteenth aspect, embodiments of this application also provide a communication system, including: a first data analysis network element for performing the fifth aspect or any implementation of the fifth aspect described above, and a second data analysis network element for performing the fourth aspect or any implementation of the fourth aspect described above. Optionally, the system further includes a network storage network element for performing the sixth aspect or any implementation of the sixth aspect described above. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of a 5G network architecture. Figure 2 A schematic diagram of the NF registration / discovery / update process in a 5G network; Figure 3 A schematic diagram illustrating the workflow of training NWDAF and inferring NWDAF under a training and inference separation architecture; Figure 4 This is a schematic diagram of the network architecture applicable to the embodiments of this application; Figures 5 to 12 Schematic diagrams of eight methods for ensuring model effectiveness in the training-inference separation scenario provided in the embodiments of this application; Figure 13 This is a schematic diagram of a communication method provided in an embodiment of this application; Figure 14(a) shows the training process of horizontal federated learning; Figure 14(b) is a schematic diagram of another communication method provided in an embodiment of this application; Figure 14(c) is a schematic diagram of another communication method provided in an embodiment of this application; Figure 15 A schematic diagram of a communication device provided in an embodiment of this application; Figure 16 This is a schematic diagram of another communication device provided in an embodiment of this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operating methods in the method embodiments can also be applied to the device embodiments or system embodiments. In the description of this application, unless otherwise stated, "multiple" means two or more.

[0072] The Wireless Machine Learning-based Network (wMLN) architecture primarily addresses the lifecycle management of machine learning models in wireless networks. Model training and model inference are two core functional modules closely related to the machine learning model. Model training demands significant computing power and a large amount of data, typically requiring deployment in centralized network elements with robust computing and data capabilities. Considering real-time inference requirements, model inference is usually deployed in local network elements closer to the service functions to reduce transmission and processing latency. Therefore, separating model training and inference is a typical deployment scenario.

[0073] The enabler of network automation (eNA) architecture is an intelligent network architecture based on the network data analytics function (NWDAF). For example... Figure 1 As shown, NWDAF is a standardized network element introduced by the 3rd Generation Partnership Project (3GPP). It is primarily used to collect network data (including one or more of the following: terminal data, base station data, transmission network data, core network data, network management data, and third-party application data), and to provide network data analysis services. It can output data analysis results for network, network management, and application execution strategy decisions. NWDAF can utilize machine learning models for data analysis. In 3GPP Release 17, the functionality of NWDAF was decomposed into data collection, model training, and model inference. In scenarios where training and inference functions are separated, the training and inference functions of the same model are deployed separately in different NWDAF instances. The NWDAF deploying the training function (referred to as the training NWDAF) provides the trained model, while the NWDAF deploying the inference function (referred to as the inference NWDAF) performs model inference by acquiring the model provided by the training NWDAF, thus providing data analysis services.

[0074] Machine learning models are typically trained by learning the mapping between a set of input features and the output target. This is achieved by optimizing certain loss functions to minimize the error between the model's output (predicted value) and the actual result (label / true value). Once the optimal model is trained, its output is used to predict future events. Ideally, it's assumed that the data used in the future will be similar to the data used during model training; specifically, it might be assumed that the distribution of the input features during training and prediction remains constant. However, in reality, this assumption usually doesn't hold. Data features change over time due to variations in network deployment, application layer business requirements, and the actual distribution of network users. Therefore, the model's performance (i.e., generalization ability) gradually declines over time. Specifically, this may manifest as a decrease in model accuracy, meaning a larger error between the model's predictions and the true values.

[0075] Figure 1 The 5G network architecture shown can be divided into three parts: terminal equipment, data network (DN), and operator network. The functions of some of these network elements are briefly described below.

[0076] The operator network may include one or more of the following network elements: Authentication Server Function (AUSF) network element, Network Exposure Function (NEF) network element, Policy Control Function (PCF) network element, Unified Data Management (UDM), Unified Data Repository (UDR), Network Repository Function (NRF) network element, Application Function (AF) network element, Access and Mobility Management Function (AMF) network element, Session Management Function (SMF) network element, RAN and User Plane Function (UPF) network element, NWDAF network element, etc. The portion of the operator network other than the radio access network can be referred to as the core network portion.

[0077] In specific implementations, the terminal device in this application embodiment can be a device used to implement wireless communication functions. Specifically, the terminal device can be a user equipment (UE), access terminal, terminal unit, terminal station, mobile station, mobile station, remote station, remote terminal, mobile device, wireless communication device, terminal agent, or terminal apparatus in a 5G network or a future evolved public land mobile network (PLMN). Access terminals can be cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices or wearable devices, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. Terminals can be mobile or fixed.

[0078] The aforementioned terminal devices can establish connections with the operator's network through interfaces provided by the operator's network (such as N1), and use data and / or voice services provided by the operator's network. The terminal devices can also access the DN (Network Provider) through the operator's network, and use operator services deployed on the DN, and / or services provided by third parties. These third parties can be service providers outside of the operator's network and terminal devices, and can provide other data and / or voice services to the terminal devices. The specific form of these third parties can be determined based on the actual application scenario and is not limited here.

[0079] RAN, as an access network element, is a sub-network of the operator's network and serves as the implementation system between service nodes and terminal equipment within the operator's network. For a terminal device to access the operator's network, it first passes through the RAN, and then connects to the operator's network's service nodes via the RAN. The RAN equipment in this application is a device that provides wireless communication functions for terminal equipment; RAN equipment is also called access network equipment. The RAN equipment in this application includes, but is not limited to: next-generation base stations (gnodeB, gNB) in 5G, evolved node B (eNB), radio network controller (RNC), node B (NB), base station controller (BSC), base transceiver station (BTS), home base station (e.g., home evolved nodeB, or homenode B, HNB), baseband unit (BBU), transmitting and receiving point (TRP), transmitting point (TP), mobile switching center, etc.

[0080] The AMF (Automatic Mobility Management) network element primarily performs functions such as mobility management and access authentication / authorization. Additionally, it is responsible for transmitting user policies between the UE and the PCF (Programmable Component Filter).

[0081] The SMF network element mainly performs functions such as session management, execution of control policies issued by the PCF, selection of the UPF, and allocation of Internet Protocol (IP) addresses for the UE.

[0082] UPF network elements, as the interface between the network and the data network, perform functions such as user plane data forwarding, session / flow-based billing statistics, and bandwidth limiting.

[0083] UDM network elements are mainly responsible for managing contract data, user access authorization, and other functions.

[0084] UDR is primarily responsible for storing and retrieving data of various types, such as contract data, strategy data, and application data.

[0085] NEF network elements are primarily used to support the opening of capabilities and events.

[0086] AF (Application Provider) network elements primarily convey application-side requests to the network side, such as Quality of Service (QoS) requirements or user state event subscriptions. AF can be a third-party functional entity or an application service deployed by the operator, such as the IP Multimedia Subsystem (IMS) voice call service.

[0087] The PCF network element is mainly responsible for policy control functions such as billing at the session and service flow levels, QoS bandwidth guarantee and mobility management, and UE policy decision-making.

[0088] NRF network elements can be used to provide network element discovery functionality, providing network element information corresponding to the network element type based on requests from other network elements. NRF also provides network element management services, such as network element registration, updates, deregistration, and network element status subscription and push.

[0089] AUSF network element: mainly responsible for authenticating users to determine whether to allow users or devices to access the network.

[0090] A Domain Provider (DN) is a network located outside of the carrier's network. A carrier's network can connect to multiple DNs, and various services can be deployed on a DN, providing data and / or voice services to terminal devices. For example, a DN might be the private network of a smart factory. Sensors installed in the workshop can act as terminal devices, and a control server for these sensors is deployed within the DN. The control server provides services to the sensors. Sensors can communicate with the control server, receive instructions from it, and transmit the collected sensor data back to the control server accordingly. Another example is a DN serving as an internal office network for a company. Employees' mobile phones or computers can act as terminal devices, accessing information and data resources on the company's internal office network.

[0091] Figure 1 Nnwdaf, Nausf, Nnef, Npcf, Nudm, Naf, Namf, Nsmf, N1, N2, N3, N4, and N6 are interface sequence numbers. The meanings of these interface sequence numbers can be found in the definitions in the 3GPP standard protocols, and are not limited here.

[0092] It should be noted that, in the embodiments of this application, the data analysis network element can be... Figure 1 The NWDAF network element shown can also be other network elements in future communication systems that have the functions of the aforementioned NWDAF network element. Network storage network elements can be... Figure 1The NRF network element shown can also be other network elements in future communication systems that have the functions of the aforementioned NRF network element. For ease of explanation, in this embodiment, the data analysis network element is taken as an NWDAF network element and the network storage network element is taken as an NRF network element. Furthermore, the NWDAF network element is further divided into training NWDAF network elements and inference NWDAF network elements.

[0093] like Figure 2 The diagram illustrates the NF registration / discovery / update process in a 5G network. In a 5G network, the NRF is primarily used for managing Network Functions (NFs), such as SMF, AMF, NEF, AUSF, NWDAF, PCF, etc. The functions supported by the NRF include: 1) NF Registration / Update / Deregistration: Available NF instances register the services they provide with the NRF. Registration information is described in NF profiles, which include information such as the NF type, NF service name, and NF address. The NRF is responsible for maintaining these NF profiles. When an NF needs to be updated or deleted, the NRF modifies or deletes the NF profile accordingly.

[0094] 2) NF Discovery: The NRF receives NF discovery requests from NF instances and provides the discovered NF instance information to the requesting NF instance. For example, an AMF requests the NRF to discover an SMF instance. Another example is an AMF requesting the NRF to discover another AMF instance.

[0095] 3) NF Status Notification: NRF notifies subscribed NF service consumers of newly registered / updated / deregistered NF instances and the NF services they provide.

[0096] exist Figure 2 In the process, the NF registration process includes steps 201 to 203.

[0097] Step 201: NF1 sends an NF registration request to NRF, carrying the NF configuration file.

[0098] This NF configuration file includes information such as NF type, NF service name, and NF address.

[0099] Step 202: NRF stores the NF configuration file.

[0100] Step 203: NRF sends an NF registration response to NF1.

[0101] This NF registration response is used to notify that the NF registration was successful.

[0102] The NF discovery process includes steps 204 to 205.

[0103] Step 204: NF2 sends an NF discovery request message to NRF, carrying the condition information of the NF to be found, such as the NF type.

[0104] Step 205: NRF sends an NF discovery response to NF2, carrying information about the matching NF instance, such as the NF identifier (NFID) or NF IP address.

[0105] The NF update process includes steps 206a to 210.

[0106] Step 206a: NF2 sends an NF state subscription request to NRF, carrying the NF instance, to request subscription to the state information of the NF instance.

[0107] Once NF2 subscribes to the state information of a certain NF instance from NRF (taking the subscription of NF1's state information as an example below), if NRF subsequently discovers that the state information of that NF instance has changed, NRF will send the updated state information of that NF instance to NF2.

[0108] Step 206b: NRF sends an NF state subscription response to NF2.

[0109] The NF state subscription response is used to notify that the NF state subscription was successful.

[0110] Step 207: NF1 sends an NF update request to NRF, carrying the updated NF configuration file.

[0111] Step 208: NRF updates the NF configuration file.

[0112] In other words, NRF updates the stored NF configuration file based on the received updated NF configuration file.

[0113] Step 209: NRF sends an NF update response to NF1.

[0114] The NF update response is used to indicate that the NF configuration file has been updated successfully.

[0115] Step 210: NRF sends an NF state change notification to NF2, carrying the updated NF configuration file.

[0116] In other words, NRF sends an NF state change notification to NF2, which had previously subscribed to NF1's state information.

[0117] Based on the above process, NRF can be combined to realize the functions of NRF registration, discovery, and updating.

[0118] It should be noted that the processes of NF registration, NF discovery, and NF update described above do not necessarily occur sequentially. This is just a process example to illustrate the usual order in which they occur.

[0119] like Figure 3 The diagram illustrates the workflow of training NWDAF and inferring NWDAF under a training and inference separation architecture. The functions of each network element are described below: NRF: Responsible for NF management, providing interface services including NF registration / deregistration / update, NF status subscription / notification, etc.

[0120] Training NWDAF: Responsible for model training. The trained model can be used by other NWDAFs (such as inference NWDAF).

[0121] Inference NWDAF: Responsible for model inference, using the inference results to perform data analysis, and outputting the data analysis results.

[0122] NF: Responsible for a specific business function, it can call the NWDAF inference service to obtain data analysis results.

[0123] Should Figure 3 The process shown includes the following steps: Step 301: Train NWDAF to send an NF registration request to NRF, carrying the NF configuration file.

[0124] The NF configuration file includes information such as NF type, NF service name, and analytics ID.

[0125] The NF type can be NWDAF.

[0126] The NF service name can be ModelProvision.

[0127] The analysis type identifier is used to indicate a specific analysis type provided by the NWDAF training, such as Service Experience, Network Performance, UE Mobility, etc.

[0128] Step 302: NRF stores the NF configuration file.

[0129] Step 303: NRF sends an NF registration response to the training NWDAF.

[0130] This NF registration response is used to notify that the NWDAF training registration was successful.

[0131] Step 304: It is inferred that NWDAF sends an NF discovery request to NRF, carrying the NF configuration file.

[0132] For example, if the NF configuration file contains the NF type (such as NWDAF), the NF service name (such as ModelProvision), and the Analytics ID, then the NF discovery request is used to request a training NWDAF corresponding to the Analytics ID from the NRF.

[0133] Step 305: NRF sends an NF discovery response to the inferred NWDAF, which carries the NWDAF instance.

[0134] The NWDAF instance carried is an instance of the trained NWDAF, which can be represented by the ID or IP address of the trained NWDAF.

[0135] Steps 301-305 above are optional. For example, if it is inferred that the NF configuration information for training NWDAF is configured on NWDAF, steps 301-305 may not be executed.

[0136] Step 306: Infer that the NWDAF sends a model request to the training NWDAF, which carries the Analytics ID.

[0137] Specifically, the inference NWDAF can send a model request to the training NWDAF based on the ID or IP address of the training NWDAF obtained from the NRF, and the Analytics ID carried is used to indicate that the request is for the model corresponding to the Analytics ID.

[0138] Step 307: The trained NWDAF sends a model response to the inference NWDAF, carrying model information.

[0139] The model (also known as a machine learning model, ML model) information describes the method used to determine the output data of a sample based on the input sample data. Model information may include, but is not limited to, one or more of the following: the feature type corresponding to the input data, the feature extraction method (functional relationship) for the feature type corresponding to the input data, the type of output data (category label, continuous numerical value, etc.), the type of algorithm used by the model, the model category (classification, regression, clustering, etc.), and the model parameters. Taking a cat-dog classification model as an example, this model can determine whether a sample is a cat or a dog based on the physical sample input data of an unknown animal. Here, the feature type of the input data can be animal weight, fur length, or vocalization; the feature extraction method for the animal weight feature type can be max-min normalization; the type of output data is cat or dog; the type of algorithm used by the model can be a deep neural network (DNN); the model category is classification; and the model parameters include, but are not limited to: the number of layers in the neural network, the activation function used in each layer, and one or more function parameter values ​​corresponding to each activation function. It is worth noting that all information related to model information (such as first model information, second model information, etc.) and model information (such as first model information, second model information, etc.) in this invention can be referred to in the description of model information, and will not be repeated elsewhere.

[0140] Steps 301 to 307 above describe the process of training the NWDAF to provide model services. Based on this process, the trained NWDAF registers the NF configuration file with the NRF. Subsequently, the inference NWDAF can obtain the trained NWDAF instance from the NRF, and thus the inference NWDAF can request specific types of model information from the trained NWDAF. In other words, the trained NWDAF can provide model services to the inference NWDAF.

[0141] Step 308: It is inferred that NWDAF sends an NF registration request to NRF, carrying the NF configuration file.

[0142] The NF configuration file includes information such as NF type, NF service name, and analytics ID.

[0143] The NF type can be NWDAF.

[0144] The NF service name can be "Analytics".

[0145] The analysis type identifier is used to indicate a specific analysis type provided by the NWDAF training, such as Service Experience, Network Performance, UE Mobility, etc.

[0146] Step 309: NRF stores the NF configuration file.

[0147] Step 310: NRF sends an NF registration response to the inferred NWDAF.

[0148] This NF registration response is used to notify that the NWDAF registration has been successfully inferred.

[0149] Step 311: The NF sends an NF discovery request to the NRF, carrying the NF configuration file.

[0150] The NF refers to an NF consumer, such as an SMF, AMF, or UPF, etc.

[0151] For example, if the NF configuration file carried contains the NF type (such as NWDAF), the NF service name (such as ModelProvision), and the Analytics ID, then the NF discovery request is used to request the inferred NWDAF corresponding to the Analytics ID from the NRF.

[0152] Step 312: NRF sends an NF discovery response to NF, which carries the NWDAF instance.

[0153] The NWDAF instance carried is an instance of the inferred NWDAF, which can be represented by the ID or IP address of the inferred NWDAF.

[0154] It is worth noting that steps 308-312 above are optional. For example, if the NF configuration information for inferring NWDAF is configured on the NF, steps 308-312 can be omitted.

[0155] Step 313: NF sends an analytics subscription to the inference NWDAF, which carries the Analytics ID.

[0156] In this context, NF can send an analysis subscription to the inferred NWDAF based on the ID or IP address obtained from NRF, and the Analytics ID carried is used to instruct the subscription to obtain the data analysis results corresponding to that Analytics ID.

[0157] Step 314: It is inferred that NWDAF sends an analysis result notification to NF, carrying the data analysis results.

[0158] Steps 308 to 314 above describe the process by which the inferred NWDAF provides analysis services. Based on this process, the inferred NWDAF registers the NF configuration file with the NRF. Subsequently, the NF can obtain the inferred NWDAF instance from the NRF, and thus the NF can request specific types of data analysis results from the inferred NWDAF. In other words, the inferred NWDAF can provide data analysis services to the NF.

[0159] As an alternative implementation method, steps 313 to 314 above can be replaced by steps 313' to 314': In step 313', NF sends an analysis request to the inferential NWDAF, which carries the Analytics ID.

[0160] In this context, NF can send an analysis subscription to the inferred NWDAF based on the ID or IP address obtained from NRF, carrying an Analytics ID to indicate a request for the data analysis results corresponding to that Analytics ID.

[0161] Step 314': It is inferred that NWDAF sends an analysis result response to NF, carrying the data analysis results.

[0162] Steps 313' to 314' require actively sending an analysis request each time, and it is inferred that NWDAF will send the data analysis results to NF only when it generates new data analysis results. In contrast, steps 313' to 314 only require a single subscription, and NWDAF will actively send the data analysis results to NF when it generates new data analysis results.

[0163] The above Figure 3 The problem with the model training and usage process illustrated is that, over time, the inference NWDAF can determine the inference result locally based on the inference data, and then determine the model's performance (i.e., the model performance evaluation result) based on the actual results of the inference data and the inference result. The inference NWDAF determines that the machine learning model's performance has declined based on the performance decline. However, in a training-inference separation scenario, the training NWDAF cannot perceive the model's performance decline in the inference NWDAF, and the inference NWDAF lacks the ability to train the model. Therefore, current technologies cannot achieve retraining and model updates when model performance declines, and thus cannot guarantee that the model's performance remains good throughout the running process. If the inference NWDAF continues to use the degraded model for data analysis, it may lead to inaccurate data analysis results, affecting model performance.

[0164] To address the aforementioned issues, this application proposes a model performance monitoring and feedback mechanism. This mechanism evaluates the performance of the model running in the inference NWDAF. When the model performance degrades to a certain level, the training NWDAF can detect this and retrain itself promptly. The inference NWDAF can then use the retrained, high-performing model to update (or replace) the model, ensuring its effectiveness. The monitoring, feedback, retraining, and updating mechanism can be implemented using NRF or through direct interaction between the training and inference NWDAFs.

[0165] The system architecture used in this application embodiment is the eNA architecture. Specifically, this application embodiment targets a scenario where model training and inference functions are deployed separately, that is, the training function and the inference function are deployed in different NWDAF instances. Figure 4 The diagram shown illustrates a network architecture applicable to an embodiment of this application. Training the NWDAF, inferring the NWDAF, and NF all require registration in the NRF via the Nnrf interface service. The inferring NWDAF requests the model from the training NWDAF through the Nnwdaf interface service, and the NF requests data analysis results from the inferring training NWDAF through the Nnwdaf interface service.

[0166] The solutions provided in the embodiments of this application will be described below.

[0167] Example 1

[0168] like Figure 5 The diagram shown is a flowchart illustrating a method for ensuring model effectiveness in a training-inference separation scenario provided by an embodiment of this application.

[0169] This first embodiment considers updating registration information via NRF to achieve performance monitoring and model updates. It mainly involves: 1. Model performance monitoring and feedback: The registration information of the inferred NWDAF at the NRF is added to the model state information. The inferred NWDAF performs model performance monitoring. When it is determined that the model needs to be retrained, the model state information is updated through the NRF. The NRF notifies the training NWDAF to update the model state, triggering the training NWDAF to retrain the model.

[0170] 2. Model Update: The registration information of the NWDAF at the NRF during training is updated with model index information. After the NWDAF is trained again to obtain a new model, the model index information is updated through the NRF. The NRF notifies the inference NWDAF that a new model is available. The inference NWDAF actively requests the new model from the training NWDAF and completes the model update.

[0171] This embodiment includes the following steps: Step 501: Train NWDAF to register with NRF.

[0172] The NWDAF is trained to send an NF registration request to the NRF, carrying an NF configuration file. This NF configuration file includes information such as the NF type, NF service name (e.g., NF Service), analysis type identifier (e.g., Analytics ID), and model index information. This model index information can be a model version number (e.g., version), location information (e.g., location), or a Uniform Resource Locator (URL). Here, version represents the model version, and location or URL represents the model's storage location; any of these three can be used. Optionally, when the model index information is location information or a URL, the location information or URL can also contain the model version. Optionally, the location information can be an IP address.

[0173] Accordingly, the NRF stores the NF configuration file and sends NF registration responses to the training NWDAF.

[0174] Step 502: Infer that NWDAF is registered with NRF.

[0175] It is inferred that NWDAF sends an NF registration request to NRF, carrying an NF configuration file. This NF configuration file includes information such as NF type, NF service name (e.g., NF Service), analysis type identifier (e.g., Analytics ID), and model status information. This model status information is used to indicate the model's usage status.

[0176] The selectable values ​​for the model's state information include, but are not limited to: 1) 'null': indicates that no model can be used.

[0177] 2) Allow 'ok': This indicates that the model performs well and can provide analysis services to external parties.

[0178] 3) 'limited': This indicates that the model performance has decreased, but it can still provide services and requires retraining.

[0179] 4) Stopped: This indicates that the model is closed and has stopped providing services.

[0180] During the registration process, the model status information carried in the NF registration request is 'null', which means that there is currently no model available on NWDAF.

[0181] Accordingly, the NRF stores the NF configuration file and sends an NF registration response to the inferred NWDAF.

[0182] Step 503: Infer NWDAF discovery and train NWDAF, and request model information from the train NWDAF.

[0183] This process can be referenced. Figure 3 Steps 304 to 307 in the embodiment will not be repeated. Based on this process, it is inferred that NWDAF can obtain model information from training NWDAF.

[0184] Step 504: Infer the state of NWDAF subscribing to NRF for training NWDAF.

[0185] When the NF configuration file registered in NRF for NWDAF is updated during subsequent training, NRF notifies the inference of NWDAF.

[0186] Step 505: Train NWDAF to infer the state of NWDAF from NRF subscription.

[0187] It is subsequently inferred that when the NF configuration file registered by NWDAF in NRF is updated, NRF notifies the training of NWDAF.

[0188] There is no fixed order between steps 504 and 505.

[0189] Step 506: It is inferred that NWDAF sends an NF update request to NRF, carrying the updated NF configuration file.

[0190] The updated NF configuration file must at least carry updated model status information, such as "ok". Optionally, the updated NF configuration file may also carry an Analytics ID to identify the model to be updated. Optionally, the updated NF configuration file may also carry the NF type, NF service name, etc.

[0191] Step 507: NRF updates the NF configuration file.

[0192] In other words, NRF updates the stored NF configuration file based on the received updated NF configuration file.

[0193] Step 508: NRF sends an NF update response to the inferred NWDAF.

[0194] This NF update response is used to notify that the NF configuration file for inferring NWDAF has been successfully updated.

[0195] Step 509: NRF sends an NF state update notification to the training NWDAF, which carries the updated model state information.

[0196] The updated model status information could be, for example, 'ok'.

[0197] Optionally, the NF state update notification also carries indication information to indicate that the type of update is a model state information update.

[0198] Since the NWDAF training subscribed to the state of the inferred NWDAF from the NRF in step 505 above, the NRF notifies the NWDAF training when the NF configuration file of the inferred NWDAF stored in the NRF is updated.

[0199] Step 510: Infer that NWDAF determines the model needs to be retrained.

[0200] The criteria for judgment can be that the model performance evaluation results do not meet the model performance requirements (e.g., the model accuracy drops below 80%, where 80% is the model accuracy requirement), or that the business key performance indicator (KPI) reported by NF does not meet the KPI requirements (e.g., the KPI drops below the KPI requirement). Other methods for determining whether the model needs retraining or updating, as described in other embodiments of this invention, can be referred to here and will not be repeated.

[0201] It should be noted that step 510 occurs during the model execution process in NWDAF inference, and the timing of its occurrence is not fixed.

[0202] Step 511: It is inferred that NWDAF sends an NF update request to NRF, carrying the updated NF configuration file.

[0203] The updated NF configuration file must at least carry updated model state information, which may be, for example, 'limited'. Optionally, the updated NF configuration file may also carry an Analytics ID to identify the model to be updated. Optionally, the updated NF configuration file may also carry the NF type, NF service name, etc.

[0204] The NRF updates the stored NF configuration file and then sends an NF update response to the inferred NWDAF.

[0205] Step 512: NRF sends an NF state update notification to the training NWDAF, which carries the updated model state information.

[0206] The updated model state information can be 'limited'.

[0207] Optionally, the NF state update notification also carries indication information to indicate that the type of update is a model state information update.

[0208] Since the NWDAF training subscribed to the state of the inferred NWDAF from the NRF in step 505 above, the NRF notifies the NWDAF training when the NF configuration file of the inferred NWDAF stored in the NRF is updated.

[0209] Step 513, Train NWDAF to start retraining the model.

[0210] Training NWDAF retrains the model to obtain the trained model and corresponding model index information, such as model version number, location information or URL.

[0211] Step 514: Train NWDAF to send an NF update request to NRF, carrying the updated NF configuration file.

[0212] The updated NF configuration file must at least carry updated model index information, which may include updated model version information, updated model location information, or updated model URL. Optionally, the updated NF configuration file may also carry an Analytics ID to identify the model to be updated. Optionally, the updated NF configuration file may also carry the NF type, NF service name, etc.

[0213] The NRF updates the stored NF configuration file and then sends an NF update response to the training NWDAF.

[0214] Step 515: NRF sends an NF state update notification to the inference NWDAF, which carries the updated model index information.

[0215] The updated model index information could be, for example, updated model version information, updated model location information, or updated model URL.

[0216] Optionally, the NF status update notification also carries indication information to indicate that the type of update is a model index information update.

[0217] Since in step 504 above, the inferred NWDAF subscribed to the state of the trained NWDAF from the NRF, the NRF notifies the inferred NWDAF when the NF configuration file of the trained NWDAF stored in the NRF is updated.

[0218] Step 516: Infer that the NWDAF sends a model request to the training NWDAF, carrying the Analytics ID and updated model index information.

[0219] The Analytics ID is used to indicate the model corresponding to that Analytics ID.

[0220] Step 517: Train the NWDAF and send the model response to the inference NWDAF, carrying model information.

[0221] This model information includes the model information corresponding to the updated model index information, that is, the model information corresponding to the new model.

[0222] Optionally, the model information carried in the model response may be the parameter values ​​of the new model, or the new model (such as a model file or an image file containing the model), or the new model address (such as a URL or IP address).

[0223] The model file is a persistent file of the model saved using a third-party framework, such as a .pb format model file saved using the artificial intelligence framework TensorFlow. The model image file is an image package containing the model, which may include the model file and several other files related to model usage.

[0224] It should be noted that if the model index information carried in step 516 is a new model address, then the inferred NWDAF can directly obtain the new model information based on this address information without executing steps 516-517. For example, the inferred NWDAF can obtain a file containing new model information (such as a file containing the parameter values ​​of the new model, or a new model file, or an image file containing the new model) based on the URL via the File Transfer Protocol (FTP).

[0225] If the model index information carried in step 516 is a model version number, then the model information carried in the model response can be the parameter values ​​of the new model, the new model (model file or image containing the model), or the address of the new model (such as a URL or IP address). If the model information carried in the model response is the address of the new model (such as a URL or IP address), it can be inferred that NWDAF can further obtain new model information based on this address information.

[0226] Step 518: Infer NWDAF to update the model.

[0227] In other words, the inference NWDAF updates or replaces the old model in use based on the new model information received.

[0228] Optionally, NWDAF can perform local testing on the new model information before updating the model, and only update or replace it after the test is passed.

[0229] Step 519: It is inferred that NWDAF sends an NF update request to NRF, carrying the updated NF configuration file.

[0230] The updated NF configuration file must at least carry updated model status information, which could be, for example, 'ok'. Optionally, the updated NF configuration file may also carry an Analytics ID to identify the model to be updated. Optionally, the updated NF configuration file may also carry the NF type, NF service name, etc.

[0231] The NRF updates the stored NF configuration file and then sends an NF update response to the inferred NWDAF.

[0232] Step 520: NRF sends an NF state update notification to the training NWDAF, which carries the updated model state information.

[0233] The updated model status information can be 'ok'.

[0234] Optionally, the NF state update notification also carries indication information to indicate that the type of update is a model state information update.

[0235] Steps 511-520 above are optional. For example, if step 510 infers that the NWDAF determines the model does not need to be retrained or that the NWDAF can tolerate a decrease in model performance below the required level, steps 511-520 may not be performed.

[0236] Since the NWDAF training subscribed to the state of the inferred NWDAF from the NRF in step 505 above, the NRF notifies the NWDAF training when the NF configuration file of the inferred NWDAF stored in the NRF is updated.

[0237] Based on the above embodiments, on the one hand, when the performance of the model used in the inference NWDAF degrades, the NRF can be used to notify the training NWDAF to retrain the model. After the training is completed, the inference NWDAF can use the new model to update or replace the old model, thus ensuring the effectiveness of the model.

[0238] On the other hand, when there are multiple inference NWDAFs, if only the performance of one inference NWDAF X degrades, and the training NWDAF is notified to retrain, after training, in addition to inference NWDAF X being able to update or replace the old model with the new model, any other inference NWDAF among the multiple inference NWDAFs can also use the above mechanism to obtain a new model and update or replace the old model with the new model. This ensures the effectiveness of the models across multiple inference NWDAFs. For a detailed example of the process for multiple inference NWDAFs, please refer to Example 2.

[0239] Example 2

[0240] like Figure 6 The diagram shown is a flowchart illustrating another method for ensuring model effectiveness in a training-inference separation scenario provided by an embodiment of this application.

[0241] This second embodiment, based on the first embodiment, considers a scenario where multiple inferential NWDAFs exist. We will use two inferential NWDAFs (denoted as inferential NWDAF1 and inferential NWDAF2) as an example. Both inferential NWDAF1 and inferential NWDAF2 use the same model for data analysis, and this model comes from the same trained NWDAF. At a certain point, the model performance in NWDAF1 deteriorates and requires retraining, while the model in inferential NWDAF2 does not require retraining. This example only uses inferential NWDAF2 to illustrate that besides inferential NWDAF1 which requests retraining, other inferential NWDAFs using the same model also exist.

[0242] Since the inference NWDAF2 also subscribes to the training NWDAF state, it will receive a notification from the NRF when the training NWDAF is retrained to obtain a new model. On one hand, if the new model performs better than the model in inference NWDAF2, inference NWDAF2 can use the new model to further improve its data analysis performance. On the other hand, since the model in inference NWDAF2 does not necessarily need to be updated at this point, it needs to acquire the new model and perform local evaluation before determining whether an update is necessary. If inference NWDAF2 ultimately decides not to update after acquiring the new model, some transmission resources are wasted. This embodiment considers further adding model performance information, including accuracy and required computation, to the registration information of the training NWDAF to help other inference NWDAFs that do not currently need updates determine whether to request a new model.

[0243] This embodiment includes the following steps: Step 601: Train NWDAF to register with NRF.

[0244] The NWDAF is trained to send an NF registration request to the NRF, carrying an NF configuration file. This file includes information such as the NF type, NF service name, and Analytics ID, as well as model index and performance information. The model index can be a model version number, location information, or URL. Here, version indicates the model version, and location or URL indicates the model's storage location; any of these three can be used. Optionally, when the model index is a location or URL, the location or URL can also contain the version. The model performance information indicates the model's performance, and may include, for example, model accuracy, hardware capabilities required to achieve that accuracy, computational cost for inference, inference time, and model size.

[0245] Optionally, the NF configuration file can also include information such as the algorithm used by the model, the artificial intelligence framework, and the model's input features.

[0246] Accordingly, the NRF stores the NF configuration file and sends NF registration responses to the training NWDAF.

[0247] Steps 602 to 613 are similar to steps 502 to 513 in the above embodiment 1.

[0248] It should be noted that the operations related to inferring NWDAF1 and NWDAF2 in steps 602 to 613 can be referred to the operations related to inferring NWDAF in steps 502 to 513 above. Furthermore, in steps 610 to 611 (refer to steps 510 to 611), NWDAF1 is inferred to determine that the model needs retraining, and then an NF update request is sent to the NRF, thereby triggering the training of NWDAF to start retraining the model.

[0249] Step 614: Train NWDAF to send an NF update request to NRF, carrying the updated NF configuration file.

[0250] The updated NF configuration file carries at least updated model index information and updated model performance information. The updated model index information may include, for example, updated model version information, updated model location information, or updated model URL. Optionally, the updated NF configuration file also carries an Analytics ID to identify the model to be updated. The updated model performance information may include, for example, model accuracy, hardware capabilities required to achieve that accuracy, computational cost for model inference, inference time, and model size.

[0251] Optionally, the updated NF configuration file also includes the NF type, NF service name, etc.

[0252] The NRF updates the stored NF configuration file and then sends an NF update response to the training NWDAF.

[0253] In step 615, NRF sends NF state update notifications to inference NWDAF1 and inference NWDAF2 respectively, which carry updated model index information and updated model performance information.

[0254] The updated model index information could be, for example, updated model version information, updated model location information, or updated model URL.

[0255] The updated model performance information may include, for example, model accuracy, hardware capabilities required to achieve that accuracy, computational cost of model inference, inference time, and model size.

[0256] Optionally, the NF status update notification also carries indication information to indicate whether the update type is a model index information update or a model performance information update.

[0257] Since in the above steps, inferred NWDAF1 and inferred NWDAF2 subscribe to the state of the trained NWDAF from the NRF, when the NF configuration file of the trained NWDAF stored in the NRF is updated, the NRF notifies inferred NWDAF1 and inferred NWDAF2.

[0258] Step 616: Infer NWDAF2 to determine whether the model needs to be updated.

[0259] Since inferenceNWDAF2 is not the trigger for model training, it needs to determine whether the model needs to be updated after inferenceNWDAF2 receives updated model index information.

[0260] Optionally, the inference-based NWDAF2 can determine whether a model needs updating based on its own computing power, model performance requirements, and received updated model performance information. Alternatively, the inference-based NWDAF2 can also determine whether a model needs updating based on the performance status of the currently used model and received updated model performance information.

[0261] Step 617: Infer that NWDAF1 sends a model request to the training NWDAF, carrying the Analytics ID and updated model index information.

[0262] The Analytics ID is used to indicate the model corresponding to that Analytics ID.

[0263] Since inference NWDAF1 is the trigger for model training, it needs to update the model after receiving updated model index information.

[0264] Step 618: Train NWDAF sends a model response to inference NWDAF1, carrying model information.

[0265] This model information includes the model information corresponding to the updated model identifier, that is, the model information corresponding to the new model.

[0266] The specific implementation of the model information can be found in the description of the foregoing embodiments.

[0267] Step 619: Infer NWDAF1 and update the model.

[0268] In other words, the inference NWDAF updates or replaces the old model in use based on the new model information received.

[0269] Optionally, NWDAF can perform local testing on the new model information before updating the model, and only update or replace it after the test is passed.

[0270] It should be noted that if, in step 616 above, NWDAF2 determines that the model needs to be updated, then NWDAF2 also needs to perform operations similar to steps 617 to 619 above to request updated model information from the training NWDAF, and then update the old model in use based on the received new model information. If, in step 616 above, NWDAF2 determines that the model does not need to be updated, then this model update process does not need to be performed.

[0271] Based on the above embodiments, the inference NWDAF that subscribes to the same model can determine whether a new model needs to be requested based on the model performance information, so as to avoid unnecessary model transmission and local evaluation processes, thereby improving the efficiency of the model update process and saving resources.

[0272] Example 3

[0273] like Figure 7 The diagram shown is a flowchart illustrating another method for ensuring model effectiveness in a training-inference separation scenario provided by an embodiment of this application.

[0274] This third embodiment builds upon the first embodiment, considering a scenario where multiple sub-models need to collaborate to complete the analysis for the same Analytics ID. In this scenario, a performance degradation in any sub-model will lead to a performance degradation in the model corresponding to the Analytics ID. If model monitoring is performed solely based on the Analytics ID, the performance of the sub-models cannot be accurately located, resulting in the retraining and updating of all sub-models corresponding to the Analytics ID. However, some sub-models may be performing well and do not require updating, leading to unnecessary training and updates.

[0275] This embodiment considers further adding model IDs to represent each sub-model.

[0276] This embodiment includes the following steps: Step 701: Train NWDAF to register with NRF.

[0277] The NWDAF is trained to send an NF registration request to the NRF, carrying the NF configuration file. This file includes information such as the NF type, NF service name, and Analytics ID, as well as a model ID and model index information. The model index information can be a model version number, location information, or URL. Here, version represents the model version, and location or URL represents the model's storage location; any of these three can be used. Optionally, when the model index information is a location or URL, the location or URL can also contain the version. The model ID is used to uniquely identify a model; for example, the model ID can consist of the NWDAF address, PLMN ID, and a modelID unique within a specific NWDAF scope.

[0278] It should be noted that the NF configuration file can carry multiple model index information, with each model identifier corresponding to one model index information.

[0279] Accordingly, the NRF stores the NF configuration file and sends NF registration responses to the training NWDAF.

[0280] Optionally, the updated NF configuration file may also carry multiple model identifiers, each used to identify one of the multiple updated models.

[0281] Step 702: Infer that NWDAF is registered with NRF.

[0282] It is inferred that NWDAF sends an NF registration request to NRF, carrying an NF configuration file. This NF configuration file includes information such as NF type, NF service name, and analytics ID, as well as model status information and model identifiers. Each model identifier corresponds to a model status information.

[0283] Model status information indicates the usage status of the model corresponding to the model identifier. Selectable values ​​for this model status information include, but are not limited to: 1) 'null': indicates that no model can be used.

[0284] 2) 'ok': indicates that the model performs well and can provide analysis services to external parties.

[0285] 3) 'limited': This indicates that the model performance has decreased, but it can still provide services and requires retraining.

[0286] 4) 'stopped': This means the model is closed and has stopped providing services.

[0287] During the registration process, the model status information carried in the NF registration request is 'null', which means that there is currently no model available on NWDAF.

[0288] For example, the model state information and model identifier carried in the NF configuration file are as follows: {(model ID 1, null), (model ID 2, null), (model ID 3, null)}.

[0289] Alternatively, the model state information and model identifier carried in the NF configuration file are as follows: {(model ID 1, model ID 2, model ID 3), (null, null, null)}.

[0290] Optionally, in practical applications, the model state information and model identifier carried by the NF configuration file can be a list containing multiple items, each containing a model state information and a model identifier.

[0291] Accordingly, the NRF stores the NF configuration file and sends an NF registration response to the inferred NWDAF.

[0292] Step 703: Infer NWDAF discovery and train NWDAF, and request model information from the train NWDAF.

[0293] This process can be referenced. Figure 5Step 503 in the embodiment will not be repeated. Based on this process, it is inferred that NWDAF can obtain model information from training NWDAF.

[0294] Step 704: Infer the state of NWDAF subscribing to NRF for training NWDAF.

[0295] When the NF configuration file registered in NRF for NWDAF is updated during subsequent training, NRF notifies the inference of NWDAF.

[0296] Step 705: Train NWDAF to infer the state of NWDAF from NRF subscription.

[0297] It is subsequently inferred that when the NF configuration file registered by NWDAF in NRF is updated, NRF notifies the training of NWDAF.

[0298] There is no fixed order between steps 704 and 705.

[0299] Step 706: It is inferred that NWDAF sends an NF update request to NRF, carrying the updated NF configuration file.

[0300] The updated NF configuration file must at least carry updated model state information, which could be, for example, 'ok'. Each model identifier corresponds to one updated model state value.

[0301] Optionally, the updated NF configuration file also includes an Analytics ID to identify the model to be updated. Optionally, the updated NF configuration file also includes the NF type, NF service name, etc.

[0302] Optionally, the updated NF configuration file can also carry a model identifier to identify the updated model.

[0303] Step 707: NRF updates the stored NF configuration file.

[0304] Step 708: NRF sends an NF update response to the inferred NWDAF.

[0305] This NF update response is used to notify that the NF configuration file has been updated successfully.

[0306] Step 709: NRF sends an NF state update notification to the training NWDAF, which carries the updated model state information.

[0307] Each model identifier corresponds to an updated model state information. The updated model state information could be, for example, 'ok'.

[0308] Optionally, the NF state update notification also carries indication information to indicate that the type of update is a model state information update.

[0309] Optionally, the NF state update notification may also carry a model identifier to identify the updated model.

[0310] Since the NWDAF training subscribed to the state of the inferred NWDAF from the NRF in step 705 above, the NRF notifies the NWDAF training when the NF configuration file of the inferred NWDAF stored in the NRF is updated.

[0311] Step 710: Infer that NWDAF determines the model needs to be retrained.

[0312] The criteria for judgment can be the evaluation results of model performance (such as a decrease in model accuracy) or the business KPIs reported by NF (such as a decrease in KPIs).

[0313] It should be noted that step 710 occurs during the model execution process in NWDAF inference, and the timing of its occurrence is not fixed.

[0314] It should be noted that in this step, the result of the judgment can be that one or more sub-models need to be retrained. For example, for a model corresponding to a certain Analytics ID, there are a total of 10 sub-models, represented by model ID 1 to model ID 10. The judgment result of step 710 is, for example, that the sub-models corresponding to model ID 1 to model ID 3 need to be retrained, while the sub-models corresponding to model ID 4 to model ID 10 do not need to be retrained.

[0315] Step 711: It is inferred that NWDAF sends an NF update request to NRF, carrying the updated NF configuration file.

[0316] The updated NF configuration file must at least carry updated model state information, where each model identifier corresponds to one updated model state information. This updated model state information could be, for example, 'limited'. Optionally, the updated NF configuration file may also carry an Analytics ID to identify the model to be updated. Optionally, the updated NF configuration file may also carry the NF type, NF service name, etc.

[0317] Optionally, the updated NF configuration file can also carry a model identifier to identify the updated model.

[0318] The NRF updates the stored NF configuration file and then sends an NF update response to the inferred NWDAF.

[0319] It should be noted that the model identifier carried in the updated NF configuration file in step 711 is the identifier information of the sub-model that needs to be retrained as determined in step 710 above, and the updated model state information is the updated model state information corresponding to the identifier information of the sub-model that needs to be retrained.

[0320] Step 712: NRF sends an NF state update notification to the training NWDAF, which carries the updated model state information.

[0321] Each model identifier corresponds to an updated model state information. This updated model state information can be 'limited'.

[0322] Optionally, the NF state update notification also carries indication information to indicate that the type of update is a model state information update.

[0323] Optionally, the NF state update notification may also carry a model identifier to identify the updated model.

[0324] Since the NWDAF training subscribed to the state of the inferred NWDAF from the NRF in step 705 above, the NRF notifies the NWDAF training when the NF configuration file of the inferred NWDAF stored in the NRF is updated.

[0325] Step 713, Train NWDAF to start retraining the model.

[0326] Training NWDAF retrains the model to obtain the trained model and corresponding model index information, such as model version number, location information or URL.

[0327] It should be noted that in this step, retraining is only initiated for the received sub-models that require training. For example, if the received model identifiers are model ID 1 to model ID 3, then the sub-models corresponding to model ID 1 to model ID 3 will be retrained.

[0328] Step 714: Train NWDAF to send an NF update request to NRF, carrying the updated NF configuration file.

[0329] The updated NF configuration file must at least carry updated model index information, with each model identifier corresponding to updated model status information. This updated model index information may include, for example, updated model version information, updated model location information, or updated model URL. Optionally, the updated NF configuration file may also carry an Analytics ID to identify the model to be updated. Optionally, the updated NF configuration file may also carry the NF type, NF service name, etc.

[0330] Optionally, the updated NF configuration file can also carry a model identifier to identify the updated model.

[0331] The NRF updates the stored NF configuration file and then sends an NF update response to the training NWDAF.

[0332] Step 715: NRF sends an NF state update notification to the inference NWDAF, which carries the updated model index information.

[0333] The updated model index information could be, for example, updated model version information, updated model location information, or updated model URL.

[0334] Optionally, the NF status update notification also carries indication information to indicate that the type of update is a model index information update.

[0335] Optionally, the NF status update notification also carries a model identifier to identify the updated model.

[0336] Since in step 704 above, the inferred NWDAF subscribed to the state of the trained NWDAF from the NRF, the NRF notifies the inferred NWDAF when the NF configuration file of the trained NWDAF stored in the NRF is updated.

[0337] Step 716: Infer that the NWDAF sends a model request to the training NWDAF, carrying the Analytics ID and updated model index information.

[0338] The Analytics ID is used to indicate the model corresponding to that Analytics ID.

[0339] The model identifier is used to indicate the sub-models in the model corresponding to this Analytics ID.

[0340] Step 717: Train the NWDAF and send the model response to the inference NWDAF, carrying model information.

[0341] This model information includes the model information corresponding to the updated model identifier, that is, the model information corresponding to the new model.

[0342] The specific implementation of the model information can be found in the description of the foregoing embodiments.

[0343] Step 718: Infer NWDAF to update the model.

[0344] In other words, the inference NWDAF updates the old model (specifically the corresponding sub-model that needs to be updated) based on the new model information received.

[0345] Optionally, NWDAF can perform local testing on the new model information before updating the model, and only update or replace it after the test is passed.

[0346] Step 719: It is inferred that NWDAF sends an NF update request to NRF, carrying the updated NF configuration file.

[0347] The updated NF configuration file must at least carry updated model status information, with one updated model status information corresponding to each model identifier. This updated model status information could be, for example, 'ok'. Optionally, the updated NF configuration file may also carry an Analytics ID to identify the model to be updated. Optionally, the updated NF configuration file may also carry the NF type, NF service name, etc.

[0348] The NRF updates the stored NF configuration file and then sends an NF update response to the inferred NWDAF.

[0349] Optionally, the updated NF configuration file can also carry a model identifier to identify the updated model.

[0350] Step 720: NRF sends an NF state update notification to the training NWDAF, which carries the updated model state information.

[0351] Each model identifier corresponds to an updated model status information. The updated model status information can be 'ok'.

[0352] Optionally, the NF state update notification also carries indication information to indicate that the type of update is a model state information update.

[0353] Optionally, the NF state update notification may also carry a model identifier to identify the updated model.

[0354] Steps 714-720 above are optional. For example, if in step 713, training the NWDAF determines that the model does not need to be retrained, or training the NWDAF can tolerate a decrease in model performance below the required level, or training the NWDAF currently does not have the capability for model retraining (e.g., limited hardware resources), steps 714-720 can be omitted.

[0355] Since the NWDAF training subscribed to the state of the inferred NWDAF from the NRF in step 705 above, the NRF notifies the NWDAF training when the NF configuration file of the inferred NWDAF stored in the NRF is updated.

[0356] Based on the above embodiments, a model identifier (also known as a sub-model identifier) ​​is added. Performance monitoring is performed at the sub-model granularity. In scenarios where one Analytics ID corresponds to multiple sub-models, accurate model retraining and updates can be achieved, avoiding waste of training and transmission resources.

[0357] Example 4

[0358] like Figure 8 The diagram shown is a flowchart illustrating another method for ensuring model effectiveness in a training-inference separation scenario provided by an embodiment of this application.

[0359] The above embodiments one to three consider using NRF to realize information interaction between training NWDAF and inferring NWDAF. This fourth embodiment considers adding an operation to the interface between training NWDAF and inferring NWDAF to directly interact with information.

[0360] This embodiment includes the following steps: Step 801: Train NWDAF to register with NRF.

[0361] The NWDAF is trained to send an NF registration request to the NRF, carrying an NF configuration file, which includes information such as NF type, NF service name, and analytics ID.

[0362] Accordingly, the NRF stores the NF configuration file and sends NF registration responses to the training NWDAF.

[0363] Step 802a: It is inferred that NWDAF sends an NF discovery request to NRF, carrying the NF configuration file.

[0364] For example, if the NF configuration file contains the NF type (such as NWDAF), the NF service name (such as ModelProvision), and the Analytics ID, then the NF discovery request is used to request the training NWDAF corresponding to the Analytics ID from the NRF.

[0365] In step 802b, the NRF sends an NF discovery response to the inferred NWDAF, which carries the NWDAF instance.

[0366] The NWDAF instance carried is an instance of the trained NWDAF, which can be represented by the ID or IP address of the trained NWDAF.

[0367] Step 803a: Infer that the NWDAF sends a model request to the training NWDAF, which carries the Analytics ID.

[0368] Specifically, the inference NWDAF can send a model request to the training NWDAF based on the ID or IP address of the training NWDAF obtained from the NRF, and the Analytics ID carried is used to indicate that the request is for the model corresponding to the Analytics ID.

[0369] Step 803b: The trained NWDAF sends a model response to the inference NWDAF, carrying model information.

[0370] The specific implementation of the model information can be found in the description of the foregoing embodiments.

[0371] For example, steps 801-803b above are optional. For example, if the NWDAF is configured with an NF configuration file for training the NWDAF, then steps 801-803b may not be executed.

[0372] Step 804a: The training NWDAF sends a model performance information subscription request to the inference NWDAF, which carries the Analytics ID, model performance metrics (such as Precision, Accuracy, Error Rate, Recall, F1 score, Mean Square Error (MSE), Root Mean Squared Error (RMSE), Root Mean Squared Logarithmic Error (RMSLE), Mean Absolute Error (MAE), model inference time, model robustness, model scalability, and model interpretability), and reporting cycle.

[0373] Precision, accuracy, error rate, recall, and F1 score are used to indicate the performance of classification or labeling models. Mean squared error, root mean square error, root mean square logarithmic error, and mean absolute error are used to indicate the performance of regression models. Model inference time indicates the time required for the model to make predictions. Model robustness indicates the model's ability to handle missing and outlier values. Model scalability indicates the model's ability to handle large datasets. Model interpretability indicates the understandability of the model's prediction criteria; for example, decision tree models have high interpretability due to the rules or tree structure they generate, while neural network models have low interpretability due to the large number of model parameters.

[0374] Step 804b: Infer that the NWDAF sends a model performance information notification to the training NWDAF, which carries the AnalyticsID, the model performance metric, and the corresponding value of the model performance metric.

[0375] Among them, the inference NWDAF is based on the reporting period and periodically sends model performance information notifications to the training NWDAF.

[0376] Based on steps 804a to 804b above, it is inferred that NWDAF can periodically report model performance information to the training NWDAF.

[0377] Optionally, the model performance information notification may also include the model performance requirements of the inference NWDAF, and / or the data used for model evaluation by the inference NWDAF. The model performance requirements can assist in determining whether retraining is needed for training the NWDAF and whether the performance of the retrained model meets the requirements of the inference NWDAF. The data used for model evaluation by the inference NWDAF includes the model's input data, the model's output data (inference results), and the actual network measurements corresponding to the inference results (network data), which can be used when retraining the model for training the NWDAF.

[0378] Optionally, steps 804a to 804b above can also be replaced by steps 804a' to 804b'.

[0379] Step 804a': The training NWDAF sends a model performance information subscription request to the inference NWDAF, which carries the Analytics ID, model performance metrics (such as precision, accuracy, error rate, recall, F1 score, mean squared error, root mean squared error, root mean squared logarithmic error, mean absolute error, model inference time, model robustness, model scalability, model interpretability) and performance threshold values.

[0380] Step 804b': Infer that the NWDAF sends a model performance retraining notification to the training NWDAF, which carries the Analytics ID.

[0381] Based on steps 804a' to 804b' above, if the inference NWDAF determines that the value corresponding to the model performance metric has reached the performance threshold, it will report a model performance information notification to the training NWDAF. This model performance retraining notification is used to trigger the training NWDAF to retrain the model. Optionally, step 804b' may not include a performance threshold, in which case the inference NWDAF can determine the performance threshold itself. This model performance retraining notification in step 804b' can also be called a model performance threshold reaching notification or a model performance information notification.

[0382] Optionally, the model performance information notification may also include the model performance requirements of the inference NWDAF, and / or the data used by the inference NWDAF for model evaluation. The model performance requirements can be threshold values ​​determined by the inference NWDAF itself, used to assist the training NWDAF in determining whether retraining is needed and whether the performance of the retrained model meets the inference NWDAF requirements. The data used by the inference NWDAF for model evaluation includes the model's input data, the model's output data (inference results), and the actual network measurements corresponding to the inference results (network data), which can be used by the training NWDAF when retraining the model.

[0383] Optionally, steps 804a to 804b above can also be replaced by steps 804a'' to 804b''.

[0384] Step 804a'': The training NWDAF sends a model performance information request to the inference NWDAF, which carries the AnalyticsID and model performance metrics (such as precision, accuracy, error rate, recall, F1 score, mean squared error, root mean squared error, root mean squared logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability).

[0385] Step 804b'': Infer that the NWDAF sends a model performance information response to the training NWDAF, which carries the AnalyticsID, the model performance metric, and the corresponding value of the model performance metric.

[0386] Based on steps 804a'' to 804b'' above, the training NWDAF can periodically send model performance information requests to the inference NWDAF. Each time the inference NWDAF receives a model performance information request, it performs a model performance evaluation based on the model performance metrics and sends a model performance information response to the training NWDAF.

[0387] Optionally, the model performance information response may also carry the model performance requirements of the inferred NWDAF, and / or the data used for model evaluation by the inferred NWDAF. The model performance requirements can assist the NWDAF training in determining whether retraining is necessary and whether the retrained model performance meets the inferred NWDAF requirements. The data used for model evaluation by the inferred NWDAF includes the model's input data, the model's output data (inference results), and the actual network measurements corresponding to the inference results (network data), which can be used by the NWDAF training team when retraining the model.

[0388] Optionally, steps 804a to 804b above can also be replaced by steps 804a''' to 804b'''.

[0389] Step 804a''', The training NWDAF sends a model performance data subscription request to the inference NWDAF, which carries the Analytics ID and the reporting period.

[0390] Step 804b''', infer that the NWDAF sends a model performance data notification to the training NWDAF, which carries the Analytics ID and model performance evaluation reference information.

[0391] The model performance evaluation reference information includes at least one of the following: the model's input data, the model's output data (inference results), or the actual network measurements corresponding to the inference results.

[0392] Based on steps 804a''' to 804b''' above, it is inferred that the NWDAF periodically sends model performance data notifications to the training NWDAF based on the reporting cycle, that is, it is inferred that the NWDAF can periodically report model performance evaluation reference information to the training NWDAF.

[0393] The actual network measurements (network data) corresponding to the inference results can be collected from the existing network by the inference NWDAF and reported to the training NWDAF, or the training NWDAF can collect them from the existing network itself.

[0394] Optionally, the model performance data notification may also include the model performance requirements of the inferred NWDAF for the model.

[0395] The NWDAF can be trained by constructing a test set and evaluating the model performance based on the inferred NWDAF periodically reported model performance evaluation reference information.

[0396] Optionally, steps 804a to 804b above can also be replaced by steps 804a'''' to 804b''''.

[0397] Step 804a'''', The training NWDAF sends a model performance data request to the inference NWDAF, which carries the Analytics ID.

[0398] Optionally, the model performance data request may also include a time range, indicating that performance data is requested within that time range.

[0399] Step 804b'''', infer that the NWDAF sends a model performance data response to the training NWDAF, which carries the Analytics ID and model performance evaluation reference information.

[0400] The model performance evaluation reference information includes at least one of the following: the model's input data, the model's output data (inference results), or the actual network measurement (network data) value corresponding to the inference results.

[0401] Based on steps 804a'''' to 804b'''' above, the training NWDAF can send a model performance data request to the inference NWDAF, and the inference NWDAF can send a model performance data response to the training NWDAF. That is, the inference NWDAF sends model performance evaluation reference information to the training NWDAF, and the model performance evaluation reference information can be within a certain time range.

[0402] The actual network measurements corresponding to the inference results can be collected from the existing network by the inference NWDAF and reported to the training NWDAF, or they can be collected from the existing network by the training NWDAF itself.

[0403] Optionally, the model performance data response may also carry the inferred NWDAF's model performance requirements.

[0404] The NWDAF can be trained by constructing a test set and evaluating the model performance based on the model performance evaluation reference information sent by the inferred NWDAF.

[0405] Step 805: Train NWDAF to determine the start of model retraining.

[0406] For example, if steps 804a to 804b above are executed, if the value of the performance index corresponding to the NWDAF training model reaches the preset performance threshold value of the NWDAF training or does not meet the model performance requirements of the inference NWDAF, then it is determined to start model retraining.

[0407] For example, if steps 804a' to 804b' are executed, the NWDAF training receives a notification of model performance information and then determines to start model retraining.

[0408] For example, if steps 804a'' to 804b'' are executed, the value of the NWDAF training model performance indicator reaches the preset performance threshold of the NWDAF training or does not meet the model performance requirements of the inference NWDAF, then the model retraining is initiated.

[0409] For example, if steps 804a''' to 804b''' are executed, or steps 804a'''' to 804b'''' are executed, then the NWDAF training determines, based on the model performance evaluation reference information, whether the model performance has reached the preset performance threshold for training the NWDAF, or does not meet the model performance requirements for inference NWDAF, and then determines to start model retraining.

[0410] Step 806: The trained NWDAF sends a model update request to the inference NWDAF, carrying the Analytics ID and the new model information.

[0411] Optionally, the new model information in the model update request can be the parameter values ​​of the new model, the new model file or the image file containing the new model, or the address of the new model (such as a URL or IP address).

[0412] It should be noted that if step 806 carries the address of the new model, it can be inferred that NWDAF can obtain a file containing the new model information based on the address. This file can be a file containing the parameter values ​​of the new model, a model file, or an image file containing the new model.

[0413] Step 807: Infer that the NWDAF sends a model update response to the training NWDAF.

[0414] Step 808: Infer NWDAF and update the model.

[0415] In other words, the inference NWDAF updates or replaces the old model in use based on the new model information received.

[0416] Optionally, NWDAF can perform local testing on the new model information before updating the model, and only update or replace it after the test is passed.

[0417] Steps 806-808 above are optional. For example, if in step 805, training the NWDAF determines that the model does not need to be trained again, or training the NWDAF can tolerate a decrease in model performance below the required level, or training the NWDAF currently does not have the capability for model retraining (e.g., limited hardware resources), steps 806-808 can be omitted.

[0418] Based on the above embodiments, the training NWDAF sends model performance subscriptions or model performance requests to the inference NWDAF to monitor the model performance in the inference NWDAF. When the performance degrades to the point that the retraining condition is met, the training NWDAF is retrained in a timely manner and sends the new model to the inference NWDAF for updates, thus ensuring the model performance of the model in the inference NWDAF.

[0419] Example 5

[0420] like Figure 9 The diagram shown is a flowchart illustrating another method for ensuring model effectiveness in a training-inference separation scenario provided by an embodiment of this application.

[0421] This fifth embodiment builds upon the fourth embodiment, considering a scenario where multiple inference NWDAFs exist. The specific scenario is the same as in the second embodiment, and can be found in the scenario description of the second embodiment. Similar to the second embodiment, this embodiment considers adding parameters indicating model performance information, including accuracy and required computation, to the model update request to help other inference NWDAFs that do not currently need updating determine whether a new model request is required.

[0422] This embodiment includes the following steps: Steps 901 to 905 are similar to steps 801 to 805 in the above embodiment four.

[0423] It should be noted that the operations related to inferring NWDAF1 and NWDAF2 in steps 901 to 905 can be referred to the operations related to inferring NWDAF in steps 802a to 804b above. Furthermore, in step 905, training NWDAF is triggered by starting model retraining based on the model performance information notification or response sent by inferred NWDAF1.

[0424] Next, training the NWDAF requires informing the inference NWDAF to perform model updates.

[0425] The first approach is to not distinguish between inference NWDAFs, meaning that the training NWDAF always sends the newly trained model information to all inference NWDAFs. This approach refers to steps 906a to 906b.

[0426] The second approach is to differentiate between different inference NWDAFs and only send new model information to the inference NWDAF that triggered the training NWDAF to perform model training. This approach refers to steps 907a to 907c.

[0427] It should be noted that either the first or the second option mentioned above should be chosen for implementation.

[0428] Option 1: Step 906a: The training NWDAF sends a model update request to the inference NWDAF1, carrying the Analytics ID, new model information, and model performance information.

[0429] Step 906b: Infer NWDAF1 to determine whether the model needs to be updated.

[0430] Upon receiving a model update request, NWDAF1 can determine whether an update is necessary based on model performance information and / or local test results of the new model information. If an update is determined to be needed, the new model information will be used to update or replace the old model.

[0431] Step 906c: The trained NWDAF sends a model update request to the inference NWDAF2, carrying the Analytics ID, new model information, and model performance information.

[0432] Step 906d: Infer NWDAF2 to determine whether the model needs to be updated.

[0433] Upon receiving a model update request, NWDAF2 can determine whether an update is necessary based on model performance information and / or local test results of the new model. If an update is determined to be required, the new model information will be used to update or replace the old model.

[0434] The second option: Step 907a: The trained NWDAF sends a model update request to the inference NWDAF1, carrying the Analytics ID and new model information.

[0435] Step 907b: Infer NWDAF1 to update the model.

[0436] It is inferred that after NWDAF1 receives a model update request, it updates or replaces the old model with the new model information.

[0437] Optionally, NWDAF can perform local testing on the new model information before updating the model, and only update or replace it after the test is passed.

[0438] Step 907c: The trained NWDAF sends a model training completion notification to the inference NWDAF2, carrying the Analytics ID and model performance information.

[0439] Step 907d: Infer NWDAF2 to determine whether the model needs to be updated.

[0440] For example, NWDAF2 can determine whether to update the model based on its own computing power, model performance requirements, and the received model performance information.

[0441] If the NWDAF2 inference determines that the model needs to be updated, then steps 907e to 907g are executed; otherwise, steps 907e to 907g are not executed.

[0442] Step 907e, optionally, infers that NWDAF2 sends a model request to the training NWDAF, carrying the Analytics ID.

[0443] The Analytics ID is used to indicate the model corresponding to that Analytics ID.

[0444] Step 907f, optional (depending on whether step 907e is performed), train NWDAF sends the model response to infer NWDAF2, carrying new model information.

[0445] Step 907g: Infer NWDAF2 update model.

[0446] In other words, it is inferred that NWDAF2 updates the old model being used based on the new model information received.

[0447] Optionally, it is inferred that NWDAF2 performs local testing on the new model information before updating the model, and updates or replaces the model only after the test is passed.

[0448] Based on the above embodiments, the inference NWDAF using the same model can obtain information about the new model and determine whether a new model needs to be requested based on the model performance information, thus avoiding unnecessary model transfer and local evaluation processes.

[0449] Example 6

[0450] like Figure 10 The diagram shown is a flowchart illustrating another method for ensuring model effectiveness in a training-inference separation scenario provided by an embodiment of this application.

[0451] This embodiment builds upon Embodiment 4, considering a scenario where multiple sub-models need to collaborate to complete the analysis for the same Analytics ID. Similar to the solution in Embodiment 3, this embodiment further considers adding a modelID to identify each sub-model, training NWDAF to assign different model IDs to the sub-models, and using the model IDs to accurately monitor the performance of each sub-model.

[0452] This embodiment includes the following steps: Step 1001: Train NWDAF to register with NRF.

[0453] The NWDAF is trained to send an NF registration request to the NRF, carrying an NF configuration file, which includes information such as NF type, NF service name, and analytics ID.

[0454] Accordingly, the NRF stores the NF configuration file and sends NF registration responses to the training NWDAF.

[0455] Step 1002a: It is inferred that NWDAF sends an NF discovery request to NRF, carrying the NF configuration file.

[0456] For example, if the NF configuration file contains the NF type (such as NWDAF), the NF service name (such as ModelProvision), and the Analytics ID, then the NF discovery request is used to request the training NWDAF corresponding to the Analytics ID from the NRF.

[0457] Step 1002b: The NRF sends an NF discovery response to the inferred NWDAF, which carries the NWDAF instance.

[0458] The NWDAF instance carried is an instance of the trained NWDAF, which can be represented by the ID or IP address of the trained NWDAF.

[0459] Step 1003a: Infer that the NWDAF sends a model request to the training NWDAF, which carries the Analytics ID.

[0460] Specifically, the inference NWDAF can send a model request to the training NWDAF based on the ID or IP address of the training NWDAF obtained from the NRF, and the Analytics ID carried is used to indicate that the request is for the model corresponding to the Analytics ID.

[0461] Step 1003b: The trained NWDAF sends a model response to the inference NWDAF, carrying model information and model identifier.

[0462] The specific implementation of the model information can be found in the description of the foregoing embodiments.

[0463] Each model identifier corresponds to a model information.

[0464] Optionally, model information and model identifiers can be implemented in the form of a model list, that is, the model response carries a model list, which contains model information and model identifiers, as well as the correspondence between them. For example, the model list includes: <model information 1, model identifier 1>, <model information 2, model identifier 2>, ...

[0465] Steps 1001-1003b above are optional. For example, if the NWDAF inference is configured with an NF configuration file for training the NWDAF, then steps 1001-1003b may not be executed.

[0466] Step 1004a: The training NWDAF sends a model performance information subscription request to the inference NWDAF, which carries the Analytics ID, model performance metrics (such as precision, accuracy, error rate, recall, F1 score, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, model interpretability), reporting cycle, and model identifier.

[0467] It should be noted that a model performance information subscription request can carry multiple model identifiers, along with the corresponding model performance metrics and reporting periods for each model identifier. Specifically, if the reporting periods for each model identifier are the same, only one reporting period needs to be carried.

[0468] Step 1004b: Infer that the NWDAF sends a model performance information notification to the training NWDAF, which carries the AnalyticsID, the model performance metric, and the corresponding value of the model performance metric.

[0469] Among them, the inference NWDAF is based on the reporting period and periodically sends model performance information notifications corresponding to each sub-model to the training NWDAF.

[0470] Based on steps 1004a to 1004b above, it is inferred that NWDAF can periodically report the model performance information corresponding to each sub-model to the training NWDAF.

[0471] Optionally, the model performance information notification may also include the model performance requirements of the inference NWDAF for each sub-model, and / or the data used by the inference NWDAF to evaluate each sub-model. The model performance requirements can help the training NWDAF determine whether retraining is needed and whether the performance of the retrained model meets the requirements of the inference NWDAF. The data used by the inference NWDAF to evaluate the model includes the model's input data, the model's output data (inference results), and the actual network measurements corresponding to the inference results, which can be used by the training NWDAF to retrain the model.

[0472] Optionally, steps 1004a to 1004b above can also be replaced by steps 1004a' to 1004b'.

[0473] Step 1004a': The training NWDAF sends a model performance information subscription request to the inference NWDAF, which carries the Analytics ID, model performance metrics (such as precision, accuracy, error rate, recall, F1 score, mean squared error, root mean squared error, root mean squared logarithmic error, mean absolute error, model inference time, model robustness, model scalability, model interpretability), performance threshold, and model identifier.

[0474] It should be noted that a model performance information subscription request can carry multiple model identifiers, as well as the corresponding model performance metrics and performance thresholds for each model identifier. In particular, when the performance thresholds for each model identifier are the same, only one performance threshold can be carried.

[0475] Step 1004b': Infer that the NWDAF sends a model performance retraining notification to the training NWDAF, which carries the Analytics ID.

[0476] Based on steps 1004a' to 1004b' above, if the inference NWDAF determines that the value corresponding to the model performance index of the sub-model has reached the performance threshold, it will report the corresponding model performance information notification to the training NWDAF. This model performance retraining notification is used to trigger the training NWDAF to retrain the sub-model. Optionally, step 1004b' may not include the performance threshold, in which case the inference NWDAF can determine the performance threshold itself. This model performance retraining notification in step 1004b' can also be called a model performance threshold reaching notification or a model performance information notification.

[0477] Optionally, the model performance information notification may also include the model performance requirements of the inference NWDAF for the sub-model, and / or the data used by the inference NWDAF for model evaluation. The model performance requirements can be threshold values ​​determined by the inference NWDAF itself, used to assist the training NWDAF in determining whether retraining is needed and whether the performance of the retrained model meets the inference NWDAF requirements. The data used by the inference NWDAF for model evaluation includes the model's input data, the model's output data (inference results), and the actual network measurements corresponding to the inference results, which can be used by the training NWDA when retraining the model.

[0478] Optionally, steps 1004a to 1004b above can also be replaced by steps 1004a'' to 1004b''.

[0479] Step 1004a'': The training NWDAF sends a model performance information request to the inference NWDAF, which carries the Analytics ID, model performance metrics (such as precision, accuracy, error rate, recall, F1 score, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, model interpretability) and model identifier.

[0480] It should be noted that a model performance information request can carry multiple model identifiers and the corresponding model performance metrics for each model identifier.

[0481] Step 1004b'': Infer that the NWDAF sends a model performance information response to the training NWDAF, which carries the Analytics ID, the model performance metric, and the corresponding value of the model performance metric.

[0482] Based on steps 1004a'' to 1004b'' above, the training NWDAF can periodically send model performance information requests to the inference NWDAF. Each time the inference NWDAF receives a model performance information request, it performs a model performance evaluation based on the model performance metrics and sends the model performance information response corresponding to the sub-model to the training NWDAF.

[0483] Optionally, the model performance information response may also include the model performance requirements of the inferred NWDAF for each sub-model, and / or the data used by the inferred NWDAF for model evaluation. The model performance requirements can assist the NWDAF training in determining whether retraining is necessary and whether the retrained model performance meets the inferred NWDAF requirements. The data used by the inferred NWDAF for model evaluation includes the model's input data, the model's output data (inference results), and the actual network measurements corresponding to the inference results (network data), which can be used by the NWDAF training team when retraining the model.

[0484] Optionally, steps 1004a to 1004b above can also be replaced by steps 1004a''' to 1004b'''.

[0485] Step 1004a''', The training NWDAF sends a model performance data subscription request to the inference NWDAF, which carries the Analytics ID, reporting period and model identifier.

[0486] It should be noted that a model performance data subscription request can carry multiple model identifiers and the corresponding reporting period for each model identifier.

[0487] Step 1004b''', infer that the NWDAF sends a model performance data notification to the training NWDAF, which carries the Analytics ID and model performance evaluation reference information.

[0488] The model performance evaluation reference information includes at least one of the following: the model's input data, the model's output data (inference results), or the actual network measurements (network data) corresponding to the inference results.

[0489] It should be noted that there can be multiple model performance evaluation reference information here. Specifically, each model identifier corresponds to one model performance evaluation reference information.

[0490] Based on steps 1004a''' to 1004b''' above, it is inferred that the NWDAF periodically sends model performance data notifications to the training NWDAF based on the reporting cycle, that is, it is inferred that the NWDAF can periodically report model performance evaluation reference information corresponding to each sub-model to the training NWDAF.

[0491] The actual network measurements (network data) corresponding to the inference results can be collected from the existing network by the inference NWDAF and reported to the training NWDAF, or the training NWDAF can collect them from the existing network itself.

[0492] Optionally, the model performance data notification may also include the inferred NWDAF model performance requirements for each sub-model.

[0493] The NWDAF can be trained by constructing a test set and evaluating the model performance based on the inferred NWDAF periodically reported model performance evaluation reference information.

[0494] Optionally, steps 1004a to 1004b above can also be replaced by steps 1004a'''' to 1004b''''.

[0495] Step 1004a'''', The training NWDAF sends a model performance data request to the inference NWDAF, which carries the Analytics ID and model identifier.

[0496] It should be noted that a model performance data subscription request can carry multiple model identifiers.

[0497] Optionally, the model performance data request may also include a time range, indicating that performance data is requested within that time range. Specifically, each model identifier can correspond to a time range.

[0498] Step 1004b'''', infer that the NWDAF sends a model performance data response to the training NWDAF, which carries the Analytics ID and model performance evaluation reference information.

[0499] The model performance evaluation reference information includes at least one of the following: the model's input data, the model's output data (inference results), and / or, the actual network measurements (network data) corresponding to the inference results.

[0500] It should be noted that there can be multiple model performance evaluation reference information here. Specifically, each model identifier corresponds to one model performance evaluation reference information.

[0501] Based on steps 1004a'''' to 1004b'''' above, the training NWDAF can send a model performance data request to the inference NWDAF, and the inference NWDAF sends a model performance data response to the training NWDAF. That is, the inference NWDAF sends model performance evaluation reference information to the training NWDAF, and the model performance evaluation reference information can be within a certain time range.

[0502] The actual network measurements (network data) corresponding to the inference results can be collected from the existing network by the inference NWDAF and reported to the training NWDAF, or the training NWDAF can collect them from the existing network itself.

[0503] Optionally, the model performance data response may also carry the inferred NWDAF's model performance requirements.

[0504] The NWDAF can be trained by constructing a test set and evaluating the model performance based on the model performance evaluation reference information sent by the inferred NWDAF.

[0505] Step 1005: Train NWDAF to determine the start of model retraining.

[0506] For example, if steps 1004a to 1004b above are executed, and the value of the performance index corresponding to the NWDAF training model reaches the preset performance threshold value of the NWDAF training or does not meet the model performance requirements of the inference NWDAF, then it is determined to start model retraining.

[0507] For example, if steps 1004a' to 1004b' are executed, the NWDAF training receives a notification of model performance information and then determines to start model retraining.

[0508] For example, if steps 1004a'' to 1004b'' are executed, the value of the NWDAF training model performance indicator reaches the preset performance threshold of the NWDAF training or does not meet the model performance requirements of the inference NWDAF, then the model retraining is initiated.

[0509] For example, if steps 1004a''' to 1004b''' are executed, or steps 1004a'''' to 1004b'''' are executed, then the NWDAF training determines, based on the model performance evaluation reference information, whether the model performance has reached the preset performance threshold for training the NWDAF, or does not meet the model performance requirements for inference NWDAF, and then determines to start model retraining.

[0510] Step 1006: The trained NWDAF sends a model update request to the inference NWDAF, carrying the Analytics ID, new model information, and model identifier.

[0511] It should be noted that a model update request can carry multiple model identifiers and the new model information corresponding to each model identifier.

[0512] Step 1007: Infer that the NWDAF sends a model update response to the training NWDAF.

[0513] Step 1008: Infer NWDAF to update the model.

[0514] In other words, the inference NWDAF updates or replaces the old model (specifically the old sub-model) that is currently in use based on the new model information received.

[0515] Optionally, NWDAF can perform local testing on the new model information before updating the model, and only update or replace it after the test is passed.

[0516] Steps 1006-1008 above are optional. For example, if in step 805, training the NWDAF determines that the model does not need to be trained again, or training the NWDAF can tolerate a decrease in model performance below the required level, or training the NWDAF currently does not have the capability to retrain the model (e.g., limited hardware resources), steps 1006-1008 can be omitted.

[0517] Based on the above embodiments, by adding sub-model identifiers and performing performance monitoring at the model granularity, accurate model retraining and updating can be achieved in scenarios where one Analytics ID corresponds to multiple sub-models, avoiding the waste of training and transmission resources.

[0518] Example 7

[0519] like Figure 11 The diagram shown is a flowchart illustrating another method for ensuring model effectiveness in a training-inference separation scenario provided by an embodiment of this application.

[0520] This embodiment considers periodically retraining the trained NWDAF and notifying the inference NWDAF that a new model is available. This embodiment is suitable for scenarios where the inference NWDAF lacks evaluation capabilities, i.e., real-time feedback on model performance from the inference NWDAF is unavailable. To maintain model performance, the trained NWDAF can be periodically retrained.

[0521] This embodiment includes the following steps: Step 1101: Train NWDAF to register with NRF.

[0522] The NWDAF is trained to send an NF registration request to the NRF, carrying an NF configuration file. This NF configuration file includes information such as the NF type, NF service name, and Analytics ID, as well as model index information. This model index information can be a model version number, location information, or a Uniform Resource Locator (URL). Here, version represents the model version, and location or URL represents the model's storage location; any of these three can be used. Optionally, when the model index information is a location or URL, the location or URL can also contain the version number.

[0523] Accordingly, the NRF stores the NF configuration file and sends NF registration responses to the training NWDAF.

[0524] Step 1102: Infer NWDAF discovery and train NWDAF, and request model information from the train NWDAF.

[0525] This process can be referenced. Figure 3 Steps 304 to 307 in the embodiment will not be repeated. Based on this process, it is inferred that NWDAF can obtain model information from training NWDAF.

[0526] Step 1103: Infer the state of NWDAF subscribing to NRF for training NWDAF.

[0527] When the NF configuration file registered in NRF for NWDAF is updated during subsequent training, NRF notifies the inference of NWDAF.

[0528] Step 1104: Train NWDAF by periodically restarting the model for retraining.

[0529] For example, when training NWDAF, a timer can be set to train it again at fixed intervals.

[0530] Step 1105: Train NWDAF to send an NF update request to NRF, carrying the updated NF configuration file.

[0531] The updated NF configuration file carries at least the updated model index information. Optionally, the updated NF configuration file may also carry the NF type, NF service name, etc.

[0532] Optionally, the updated NF configuration file can also carry updated model performance information, such as model accuracy, hardware capabilities required to achieve that accuracy, computational cost for model inference, inference time, and model size.

[0533] Step 1106: NRF updates the stored NF configuration file.

[0534] Step 1107: NRF sends an NF update response to the training NWDAF.

[0535] This NF update response is used to notify that the NF configuration file has been updated successfully.

[0536] Step 1108: NRF sends an NF state update notification to the inference NWDAF, which carries the updated model index information.

[0537] Optionally, the NF status update notification also carries indication information to indicate that the type of update is a model index information update.

[0538] Step 1109: Infer that the NWDAF sends a model request to the training NWDAF, carrying the Analytics ID and updated model index information.

[0539] The Analytics ID is used to indicate the model corresponding to that Analytics ID.

[0540] Step 1110: The trained NWDAF sends the model response to the inference NWDAF, carrying model information.

[0541] This model information includes the model information corresponding to the updated model index information, that is, the model information corresponding to the new model.

[0542] Step 1111: Infer NWDAF to update the model.

[0543] In other words, it is inferred that NWDAF updates the old model being used based on the new model information received.

[0544] Optionally, NWDAF can perform local testing on the new model information before updating the model, and only update or replace it after the test is passed.

[0545] Steps 1109-1111 above are optional. For example, if the inference NWDAF after step 1108 can tolerate a decrease in model performance to the required level, steps 1109-1111 may not be performed.

[0546] It should be noted that steps 1104-1108 above are performed periodically, therefore steps 1109-1111 are optional, because training NWDAF is only responsible for periodic retraining, and whether to request a new model for updating is determined by inference NWDAF itself.

[0547] Based on this embodiment, in scenarios where the inferred NWDAF lacks evaluation functionality, i.e., when training the NWDAF cannot obtain real-time feedback on the model performance from the inferred NWDAF, periodic retraining can be performed to ensure the model's performance.

[0548] It should be noted that, based on the above embodiment seven, when there are multiple inference NWDAFs, after step 1107 above, the NRF can send NF state update notifications to multiple inference NWDAFs, so that multiple inference NWDAFs can send model requests to the training NWDAF, thereby realizing model updates for multiple inference NWDAFs.

[0549] It should be noted that, based on the above embodiment seven, when an analysis type identifier corresponds to multiple sub-models, and each sub-model is identified by a model identifier, then step 1105 above can also carry one or more model identifiers, and step 1108 above can also carry one or more model identifiers, and then step 1109 above can carry one or more model identifiers, thereby realizing the update of one or more sub-models in the inference NWDAF.

[0550] Example 8

[0551] like Figure 12 The diagram shown is a flowchart illustrating another method for ensuring model effectiveness in a training-inference separation scenario provided by an embodiment of this application.

[0552] This embodiment eight is the same as the scenario in embodiment seven, that is, when the inferred NWDAF does not have the evaluation function, the training NWDAF is periodically retrained and a model update message is sent to the inferred NWDAF.

[0553] This embodiment includes the following steps: Step 1201: Train NWDAF to register with NRF.

[0554] The NWDAF is trained to send an NF registration request to the NRF, carrying an NF configuration file, which includes information such as NF type, NF service name, and analytics ID.

[0555] Accordingly, the NRF stores the NF configuration file and sends NF registration responses to the training NWDAF.

[0556] Step 1202: Infer NWDAF discovery and train NWDAF, and request the model from the train NWDAF.

[0557] This process can be referenced. Figure 3 Steps 304 to 307 in the embodiment will not be repeated. Based on this process, it is inferred that NWDAF can obtain model information from training NWDAF.

[0558] Step 1203: Train NWDAF by periodically starting model retraining.

[0559] For example, when training NWDAF, a timer can be set to train it again at fixed intervals.

[0560] Step 1204: The trained NWDAF sends a model update request to the inference NWDAF, carrying the Analytics ID and the new model information.

[0561] Optionally, the model update request may also carry performance information of the new model, such as model accuracy, hardware capabilities required to achieve that accuracy, computational cost required for model inference, inference time, and model size.

[0562] Step 1205: Infer that the NWDAF sends a model update response to the training NWDAF.

[0563] Step 1206: Infer NWDAF to update the model.

[0564] In other words, it is inferred that NWDAF updates the old model being used based on the new model information received.

[0565] Optionally, NWDAF can perform local testing on the new model information before updating the model, and only update or replace it after the test is passed.

[0566] Steps 1204-1206 above are optional. For example, if the model performance evaluation result of the trained and updated model determined in step 1203 is less than or equal to the model performance evaluation result provided by the trained NWDAF to the inference NWDAF in step 1202, steps 1204-1206 may not be executed.

[0567] Based on the above embodiments, for scenarios where the inference NWDAF does not have an evaluation function, when the training NWDAF cannot obtain real-time feedback on the model performance from the inference NWDAF, periodic retraining can be performed to ensure the model performance.

[0568] It should be noted that, based on the above embodiment eight, when there are multiple inference NWDAFs, in step 1204 above, the training NWDAF can send model update requests to multiple inference NWDAFs, thereby realizing the model update of multiple inference NWDAFs.

[0569] It should be noted that, based on the above embodiment eight, when one analysis type identifier corresponds to multiple sub-models, and each sub-model is identified by a model identifier, then step 1204 above can also carry one or more model identifiers, thereby realizing the update of one or more sub-models in the inference NWDAF.

[0570] Example 9

[0571] The relationship between Embodiment Nine and Embodiments One to Eight described above is that Embodiments One to Eight are various specific implementations of Embodiment Nine. For example... Figure 13 The diagram shown is a schematic flowchart of a communication method provided in an embodiment of this application. It should be noted that the first NWDAF in this embodiment nine can be the training NWDAF from embodiments one to eight above, the second NWDAF can be the inferred NWDAF1 from embodiments one to eight above, and the third NWDAF can be the inferred NWDAF2 from embodiments one to eight above.

[0572] The method includes the following steps: Step 1301: The first NWDAF sends the third information to the second NWDAF. Accordingly, the second NWDAF receives the third information.

[0573] The third piece of information includes model performance metrics, which are used to obtain performance evaluation results for the model. Optionally, model performance metrics include one or more of the following: precision, accuracy, error rate, recall, F1 score, mean squared error, root mean squared error, root mean squared logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. That is, the second NWDAF evaluates the performance of the model in use based on the received model performance metrics, thereby obtaining performance evaluation results and generating a model performance report.

[0574] Optionally, the third information may also include one or more of the following: Analysis Type ID, Model ID, and Sub-Model ID. The Analysis Type ID (Analytics ID) indicates the analysis type of the model, such as Service Experience, Network Performance, UE Mobility, etc. The Model ID identifies the model. The Sub-Model ID identifies the sub-models of the model. It should be noted that if the model has no sub-models, the third information may carry the Model ID but not the Sub-Model ID, or it may carry neither the Model ID nor the Sub-Model ID. If the model has sub-models, it must carry both the Model ID and one or more Sub-Model IDs. It should be noted that when the third information carries the Sub-Model ID, the model's performance metrics are used to obtain the performance evaluation results of the model's sub-models.

[0575] For specific examples of the sub-models, please refer to the descriptions of Embodiments 3 and 6 above.

[0576] Optionally, the third information may also include one or more of the following: reporting period and threshold information. The reporting period indicates the time for reporting the model's performance report, that is, it instructs the second NWDAF to report the model's performance report to the first NWDAF based on this reporting period. The threshold information indicates the conditions for reporting the model's performance report; that is, when the evaluation result of the model obtained by the second NWDAF reaches the threshold value corresponding to this threshold information, the second NWDAF reports the model's performance report to the first NWDAF.

[0577] It should be noted that step 1301 is an optional step. If step 1301 is not performed, the aforementioned third information can be pre-configured on the second NWDAF, or the aforementioned third information can be configured to the second NWDAF by other network elements.

[0578] In step 1302, the second NWDAF sends the first information to the first NWDAF. Accordingly, the first NWDAF receives the first information.

[0579] The first information includes a model performance report, which indicates the evaluation results of the model's performance, or indicates that the evaluation results of the model's performance do not meet the requirements of the model's performance indicators.

[0580] Optionally, the first information may also include one or more of the following information corresponding to the model's performance report: time, region, and slice. The time refers to the time range in which the model's performance report was generated; the region refers to the regional range corresponding to the model's performance report; and the slice refers to the slice information corresponding to the model's performance report.

[0581] Step 1303: The first NWDAF updates the first model information of the model based on the model's performance report to obtain the second model information of the model.

[0582] In step 1304, the first NWDAF sends the second information to the second NWDAF. Accordingly, the second NWDAF receives the second information.

[0583] The second information includes the second model information.

[0584] Optionally, the second information may also include one or more of the following: model identifier, sub-model identifier, model performance evaluation result, hardware capability information corresponding to the model performance evaluation result, model size, and model inference time. The hardware capability information corresponding to the model performance evaluation result refers to the hardware requirements for running the model, such as requiring a Graphics Processing Unit (GPU) acceleration capability. The model inference time refers to the latency between receiving input and generating output. Optionally, each type of hardware capability information corresponds to a specific inference time; the stronger the hardware capability, the shorter the inference time.

[0585] Step 1305: The second NWDAF updates the model based on the second information.

[0586] For example, the second NWDAF uses the second model information to replace the first model information based on the second information, thereby updating the model.

[0587] Optionally, the first NWDAF can also send second information to other NWDAFs besides the second NWDAF (such as the third NWDAF). That is, the second NWDAF triggers the first NWDAF to update the model and obtain the second model information. However, the first NWDAF not only sends the second information to the second NWDAF, but also to the third NWDAF, so that the third NWDAF can update the model. This avoids the third NWDAF needing to request a model update from the first NWDAF separately, thus saving signaling overhead.

[0588] For a specific example of the first NWDAF sending the second information to the third NWDAF, please refer to the descriptions of Embodiments 2 and 5 above.

[0589] As one implementation method, step 1301 above can specifically be: the first NWDAF sends third information to the second NWDAF through the NRF; step 1302 above can specifically be: the first NWDAF receives first information from the second NWDAF through the NRF; and step 1304 above can specifically be: the first NWDAF sends second information to the second NWDAF through the NRF. That is, when there is no interface between the first NWDAF and the second NWDAF, the NRF can be used as an intermediate network element to achieve interaction between the first NWDAF and the second NWDAF.

[0590] For specific examples of using NRF as an intermediate network element, please refer to the descriptions of Embodiments 1 to 3 above.

[0591] Based on the above scheme, when the second NWDAF cannot complete model training, the second NWDAF can send the model's performance report to the first NWDAF. The first NWDAF can then update the model based on the performance report to obtain the second model information and send the second model information to the second NWDAF. This allows the second NWDAF to update the model based on the second model information, thereby enabling timely model training when model performance degrades and ensuring model performance.

[0592] For example, assuming the model is a service experience model, this model can be used to evaluate the service experience of a service flow based on network data corresponding to the service flow (such as the air interface quality of the terminal corresponding to the service flow at the base station, and the bandwidth, latency, jitter, etc. of the service quality flow of the session of the terminal corresponding to the service flow on the user plane management network element). The network-side policy charging function (PCF) can determine whether the experience requirements of the service flow are met based on the service experience output of this model. If not, the QoS parameters of the service can be adjusted. Here, the prerequisite for the PCF to adjust the QoS parameters is that the performance of the service experience model is good enough; otherwise, it will affect the service experience. For example, consider the service experience of voice services, specifically the Mean Opinion Score (MOS). A MOS score of 3.0 is required. If the actual MOS score of the service flow is 2.5, but the model's output MOS score is 3.5, the Process Control Function (PCF) will not adjust the service's QoS parameters, resulting in a poor service experience. If the model's performance is good enough, its output MOS score should be 2.5, prompting the PCF to adjust the QoS parameters to achieve a MOS score of 3.0 or higher. In this example, model performance impacts service experience. Furthermore, if model performance continues to decline, it may eventually deteriorate to the point where the model becomes completely unusable, causing an extremely poor service experience or service interruption.

[0593] Federated learning, as a novel artificial intelligence technology, enables cross-domain joint training of models without the original data remaining within its own domain. This improves training efficiency and, most importantly, avoids security issues that arise when data is aggregated at a data analysis center (e.g., the original data is intercepted during transmission or misused by the data center). Horizontal federated learning, as a federated learning technique, is suitable for training data scenarios where "feature repetition is very high, but the differences between data samples are significant."

[0594] Figure 14(a) illustrates the training process of horizontal federated learning (taking linear regression as an example). As can be seen, horizontal federation includes a central server node and multiple edge client nodes (e.g., client node A, client node B, and client node K). The original data is distributed across the client nodes; the server node does not possess the original data, and client nodes are not allowed to send original data to the server node.

[0595] First, the datasets on each client node (assuming a total of...) There are client nodes, which means there are... K The datasets are as follows:

[0596] in, For sample data, This refers to the label data corresponding to the sample data. In horizontal federated learning, each sample data includes a label; that is, the label and data are stored together.

[0597] Then, the data analysis module on each client node can train its own model based on the linear regression algorithm, which is called a sub-model. .

[0598] Assuming the loss function used in linear regression is the mean squared error (MSE), then the objective function for training each sub-model (the entire training process aims to minimize the value of the aforementioned loss function) is: min

[0599] The training process then truly begins, with each iteration taking place. (1) The gradient of the sub-model generated by each client node is as follows:

[0600] (2) Each client reports the number of samples and the local gradient value, i.e.: as well as ,in, Indicates the number of samples. This represents the local gradient value.

[0601] (3) After receiving the above information, the server node aggregates the gradients as follows: , in, The number of client nodes. .

[0602] (4) The server node sends the aggregated gradients to each client node participating in the training, and then the client nodes update the model parameters locally, as follows:

[0603] (5) After the client node updates the model parameters, it calculates the loss function value. Proceed to step (1).

[0604] In the training process described above, the server node can control the end of training by controlling the number of iterations, such as terminating training after 10,000 iterations, or by setting a threshold for the loss function, for example... The training session ended at that time.

[0605] After training, each client node retains the same model (which can come from the server node or be further customized locally based on the model from the server node) for local inference.

[0606] This application embodiment combines horizontal federated learning with NWDAF to realize the model training and update process. Specifically, the first NWDAF (also known as Server NWDAF) can train or aggregate the model, while the second NWDAF (also known as Client NWDAF) can train, update, and use the model for inference.

[0607] Figure 14(b) shows a schematic flowchart of another communication method provided in an embodiment of this application. The method includes the following steps: Step 1401b: The first NWDAF registers with the NRF.

[0608] The first NWDAF sends an NF registration request to the NRF, carrying the NF configuration file, which includes information such as NF type, NF service name (NF Service, such as ModelProvision), and analytics type identifier (Analytics ID).

[0609] Accordingly, the NRF stores the NF configuration file and sends an NF registration response to the first NWDAF.

[0610] Step 1402b: The second NWDAF registers with the NRF.

[0611] The second NWDAF sends an NF registration request to the NRF, carrying an NF configuration file. This NF configuration file includes information such as the NF type, NF service name (NF Service, such as ModelUpdate), and analytics type identifier (Analytics ID).

[0612] Accordingly, the NRF stores the NF configuration file and sends an NF registration response to the second NWDAF.

[0613] Step 1403b: The second NWDAF sends an NF discovery request to the NRF, carrying the NF configuration file.

[0614] For example, if the NF configuration file contains the NF type (such as NWDAF), the NF service name (NF Service, such as ModelProvision), and the Analytics ID, then the NF discovery request is used to request the Server NWDAF corresponding to the Analytics ID from the NRF.

[0615] In step 1404b, the NRF sends an NF discovery response to the second NWDAF, which carries the NWDAF instance.

[0616] The NWDAF instance carried is an instance of Server NWDAF, which can be represented by the Server NWDAF ID or IP address.

[0617] Step 1405b: The first NWDAF sends an NF discovery request to the NRF, carrying the NF configuration file.

[0618] For example, if the NF configuration file carried contains the NF type (such as NWDAF), the NF service name (NF Service, such as ModelUpdate), and the Analytics ID, then the NF discovery request is used to request the Client NWDAF corresponding to the Analytics ID from the NRF.

[0619] Step 1406b: The NRF sends an NF discovery response to the first NWDAF, which carries the NWDAF instance.

[0620] The NWDAF instance carried is an instance of Client NWDAF, which can be represented by the ID or IP address of Client NWDAF.

[0621] It should be noted that the NF discovery response can contain one or more Client NWDAF instances.

[0622] It should be noted that "Steps 1403b-1404b" and "Steps 1405b-1406b" can be executed only one of them. In this way, in federated learning, the client NWDAF can actively trigger the horizontal federated training of the server NWDAF, or the server NWDAF can actively trigger the horizontal federated training of the client NWDAF.

[0623] Steps 1401b-1406b are optional. For example, if the first NWDAF has an NF configuration file for the second NWDAF and / or the second NWDAF has an NF configuration file for the first NWDAF, steps 1401b-1406b may not be executed.

[0624] Step 1407b: The second NWDAF sends a model subscription request to the first NWDAF, carrying the Analytics ID.

[0625] This model subscription request is used to subscribe to the model index information corresponding to the Analytics ID from the first NWDAF.

[0626] Step 1408b: The first NWDAF sends model notification 1 to the second NWDAF, carrying model index information 1.

[0627] The model index information 1 is the index information of the model corresponding to the Analytics ID.

[0628] This model notification is the model notification corresponding to the model subscription request in step 1707.

[0629] Furthermore, the second NWDAF can obtain the first information of the corresponding model based on the model index information 1.

[0630] Step 1409b: The first NWDAF sends a model subscription request to the second NWDAF, carrying model index information 1.

[0631] The model subscription request is used to request the second NWDAF to update the first information of the model corresponding to model index information 1, and to subscribe to the updated model information.

[0632] Step 1410b: Model update.

[0633] Specifically, the second NWDAF uses the model information corresponding to model index information 1 for local training, obtains the second information of the model, and determines the model index information 2 corresponding to the second information of the model.

[0634] Step 1411b: The second NWDAF sends a model notification to the first NWDAF, carrying model index information 2.

[0635] This model notification is the model notification corresponding to the model subscription request in step 1409b.

[0636] Step 1412b, model update.

[0637] Specifically, the first NWDAF uses the second information of the model corresponding to the model index information 2 for local training, obtains the third information of the model, and determines the model index information 3 corresponding to the third information of the model.

[0638] Optionally, the second NWDAF in steps 1407b-1410b can be an instance of multiple Client NWDAFs. In this case, the first NWDAF can receive model index information from multiple second NWDAF instances in step 1411b. The first NWDAF obtains multiple corresponding model information based on the multiple model index information and aggregates and trains the multiple model information to obtain updated model information.

[0639] Step 1413b: The first NWDAF sends a model notification to the second NWDAF, carrying model index information 3.

[0640] This model notification is the model notification corresponding to model subscription request 1 in step 1407b.

[0641] Subsequently, steps 1410b to 1413b can be repeated, and the model index information continues to change until the first NWDAF determines to stop iteration. Optionally, the first NWDAF can send a model unsubscribe message to the second NWDAF, that is, cancel the model subscription request corresponding to step 1409b, thereby stopping iteration.

[0642] It should be noted that, in this embodiment, the model index information may include identification information, which is used to indicate the information of the model corresponding to the model index information. Optionally, the model index information may also include one or more of the following: the analysis type identifier corresponding to the model, the model identifier, and the version information of the model information.

[0643] Based on the above scheme, both the first NWDAF and the second NWDAF can update the model to obtain new model information and send the model index information corresponding to the new model information to each other. In this way, the model can be iterated repeatedly, thereby gradually improving the model performance and finally obtaining a model with optimal performance, thus ensuring the model performance.

[0644] Figure 14(c) shows a schematic flowchart of another communication method provided in an embodiment of this application. In this embodiment, the first NWDAF can also be called the training NWDAF, and the second NWDAF can also be called the inference NWDAF or other training NWDAFs different from the first NWDAF.

[0645] The method includes the following steps: Step 1401c: The first NWDAF registers with the NRF.

[0646] The first NWDAF sends an NF registration request to the NRF, carrying the NF configuration file, which includes information such as NF type, NF service name, and analytics ID.

[0647] The NF type can be NWDAF.

[0648] The NF service name can be ModelProvision.

[0649] The analysis type identifier is used to indicate a specific analysis type provided by the NWDAF training, such as Service Experience, Network Performance, UE Mobility, etc.

[0650] Accordingly, the NRF stores the NF configuration file and sends an NF registration response to the first NWDAF.

[0651] As one implementation method, for a given Analytics ID, the first NWDAF can be registered with the NRF once the model training capability corresponding to that Analytics ID is available. Alternatively, for a given Analytics ID, the first NWDAF can also be registered with the NRF once the model corresponding to that Analytics ID has already been trained.

[0652] As one implementation method, the first NWDAF can also carry second indication information in the aforementioned NF configuration file. This second indication information indicates whether the model corresponding to each Analytics ID has been trained or is ready to be used. Alternatively, the second indication information can be carried in the model information within the NF configuration file. Optionally, when the second indication information indicates that the model corresponding to a certain Analytics ID has been trained or is ready to be used, the NF configuration file can also carry model description information corresponding to that Analytics ID.

[0653] As another implementation method, the NF configuration file carries model description information corresponding to the Analytics ID. The model description information is used to indicate whether the model corresponding to the Analytics ID has been trained or is ready to be used.

[0654] The model description information includes one or more of the following: analytics filter, target of analytics reporting, model performance, or model deployment environment. The analytics filter information indicates the scope of applicability of the model corresponding to the Analytics ID and includes one or more of the following: region, time period, single network slice selection assistance information (S-NSSAI), or data network name (DNN).

[0655] The target information in the analysis report is used to identify the terminal corresponding to the model associated with the Analytics ID. This target information includes one or more of the following: a terminal identifier, a terminal group identifier, or information indicating any terminal. Specifically, the information indicating any terminal means that the terminal corresponding to the model associated with the Analytics ID can be any terminal.

[0656] Model performance information is used to represent the performance of the model corresponding to the Analytics ID. Model performance information includes one or more of the following: accuracy, precision or recall, error rate, F1 score, mean square error, standard deviation, logarithm standard deviation, mean absolute deviation, model inference delay, model robustness, model expandability, and model interpretability.

[0657] Model deployment environment information is used to represent the hardware environment in which the model corresponding to the Analytics ID is deployed. Model deployment environment information includes one or more of the following: number of central processing units (CPUs), number of graphics processing units (GPUs), memory size, or hard disk size.

[0658] Step 1402c: The second NWDAF sends an NF discovery request to the NRF, carrying NF requirement information.

[0659] For example, an NF discovery request is used to retrieve a training NWDAF from an NRF based on NF requirement information. This NF requirement information might include the NF type (e.g., NWDAF), the NF service name (e.g., ModelProvision), the Analytics ID, and the first model requirement information corresponding to that Analytics ID. The NF discovery request is used to request from the NRF a model that can provide a training NWDAF corresponding to that Analytics ID, and a model trained on that NWDAF based on that Analytics ID that satisfies the first model requirement information. Here, a model that satisfies the first model requirement information can be understood as: a model trained on that NWDAF for that Analytics ID that fully or partially matches the first model requirement information.

[0660] The first model requirement information includes one or more of the following: analytics filter requirements, target of analytics reporting, model performance information, or model deployment environment information.

[0661] For example, a model trained with Analytics ID that fully matches the first model requirement information can be understood as follows: the analysis filtering information, analysis report target information, model performance information, and model deployment environment information corresponding to this model completely satisfy the analysis filtering information, analysis report target information, model performance information, and model deployment environment information in the first model requirement information. A model trained with Analytics ID that partially matches the first model requirement information can be understood as follows: the analysis filtering information, analysis report target information, model performance information, and model deployment environment information corresponding to this model completely satisfy one or more of the analysis filtering information, analysis report target information, model performance information, or model deployment environment information in the first model requirement information.

[0662] The analytics filter information is used to indicate the scope of the model that the second NWDAF needs to request. The analytics filter includes one or more of the following: region, time period, S-NSSAI, and DNN.

[0663] The target information in the analysis report is used to indicate the terminal corresponding to the model requested by the second NWDAF. The target information includes one or more of the following: the terminal identifier, the terminal group identifier, or information indicating any terminal. Specifically, the information indicating any terminal indicates that the terminal corresponding to the model with the Analytics ID can be any terminal.

[0664] Model performance information is used to represent the performance of the model requested by the second NWDAF. Model performance information includes one or more of the following: accuracy, precision or recall, error rate, F1 score, mean square error, standard deviation, logarithm standard deviation, mean absolute deviation, model inference delay, model robustness, model expandability, and model interpretability.

[0665] The model deployment environment information is used to indicate the hardware environment in which the second NWDAF needs to request model deployment. The model deployment environment information includes one or more of the following: number of CPUs, number of GPUs, memory size, or hard disk size.

[0666] Step 1403c: The NRF sends an NF discovery response to the second NWDAF, which carries the address information of the first NWDAF.

[0667] For example, the address information can be an ID, an IP address, or a fully qualified domain name (FQDN).

[0668] Optionally, before the NRF returns the address information of the first NWDAF to the second NWDAF, the NRF determines that the first NWDAF is the training NWDAF corresponding to the Analytics ID in step 1402c, and the model trained by the first NWDAF according to the Analytics ID satisfies the first model requirement information in step 1402c.

[0669] The first NWDAF carried is an instance of the training NWDAF, which can be represented by the ID or IP address of the training NWDAF instance.

[0670] Specifically, the NRF obtains the first NWDAF based on the first model requirement information sent by the second NWDAF. The obtained first NWDAF satisfies the first model requirement information. This can be understood as the first NWDAF completely or partially matching the first model requirement information for the model trained on the Analytics ID.

[0671] For example, a model trained with Analytics ID that fully matches the first model requirement information can be understood as follows: the analysis filtering information, analysis report target information, model performance information, and model deployment environment information corresponding to this model are the same as those in the first model requirement information, respectively. A model trained with Analytics ID that partially matches the first model requirement information can be understood as satisfying any one or more of the following conditions: the analysis filtering information corresponding to this model is the same as the analysis filtering information in the first model requirement information; the analysis report target information corresponding to this model is the same as that in the first model requirement information; the model performance information corresponding to this model is the same as that in the first model requirement information; or the model deployment environment information corresponding to this model is the same as the analysis filtering information in the first model requirement information.

[0672] Step 1404c: The second NWDAF sends a first request to the first NWDAF.

[0673] Optionally, the first request is a model subscription request, which is used to subscribe to the model index information of the model corresponding to the Analytics ID and which meets the second model requirement information from the first NWDAF.

[0674] The first request carries an Analytics ID and second model requirement information. This first request is used to request the model index information of the model corresponding to the Analytics ID that satisfies the second model requirement information from the first NWDAF. A model that satisfies the second model requirement information can be understood as a model trained by the first NWDAF for the Analytics ID that fully or partially matches the second model requirement information.

[0675] For example, a model trained with Analytics ID that fully matches the requirements of the second model can be understood as follows: the analysis filtering information, analysis report target information, model performance information, and model deployment environment information corresponding to this model completely satisfy the analysis filtering information, analysis report target information, model performance information, and model deployment environment information in the requirements of the second model. A model trained with Analytics ID that partially matches the requirements of the second model can be understood as follows: the analysis filtering information, analysis report target information, model performance information, and model deployment environment information corresponding to this model completely satisfy one or more of the analysis filtering information, analysis report target information, model performance information, or model deployment environment information in the requirements of the second model.

[0676] The model index information can be one or more of the following: model ID, model version number (e.g., version), location information (e.g., location), or address information. The model ID identifies the model, the model version number indicates the model version, and the location or address information indicates the model's storage location. Optionally, when the model index information is location or address information, the location or address information may also include the model version. Optionally, the address information may include one or more of the following: IP address, FQDN, or URL.

[0677] The second model requirement information here includes time information, as well as some or all of the information in the first model requirement information mentioned above. This time information indicates the expected time when model index information from the first NWDAF is expected to be received. This time information can be absolute or relative.

[0678] As one implementation method, the second NWDAF can send a model subscription request to the first NWDAF by triggering the Nnwdaf_MLModelProvision_Subscribe service operation or the Nnwdaf_MLModelProvision_Unsubscribe service operation to the first NWDAF.

[0679] Step 1405c: The first NWDAF obtains the model index information based on the second model requirement information and the Analytics ID.

[0680] The first NWDAF retrieves the model index information of the model that corresponds to the Analytics ID and meets the second model requirement information.

[0681] The Analytics ID here is the same as the Analytics ID in the model subscription request.

[0682] The first method for NWDAF to obtain model index information is: 1) If the first NWDAF determines that there is a local model corresponding to the Analytics ID and that the local model meets the requirements of the second model, then the first NWDAF does not need to collect data from other network elements, but can directly obtain the model index information of the local model. 2) If the first NWDAF determines that a local model corresponding to the Analytics ID exists, but the local model does not meet the second model requirement information, then the first NWDAF further collects data from other network elements, and retrains the local model based on the collected data to obtain a new model, and obtains the model index information of the new model; wherein, the local model not meeting the second model requirement information can be understood as the local model's corresponding analysis filtering information, analysis report target information, model performance information, and model deployment environment information not meeting any one of the analysis filtering information, analysis report target information, model performance information, and model deployment environment information in the first model requirement information.

[0683] 3) If the first NWDAF determines that there is no local model corresponding to the Analytics ID, the first NWDAF further collects data from other network elements, trains an initial model based on the collected data, and obtains the model index information of the initial model.

[0684] Specifically, the time required for the second NWDAF to obtain model index information for the above three operations increases sequentially. This is because the time required for model retraining is much shorter than the time required for initial model training.

[0685] Step 1406c: The first NWDAF sends model index information to the second NWDAF.

[0686] The model corresponding to the model index information is the model that the first NWDAF obtains, which corresponds to the Analytics ID and meets the requirements of the second model.

[0687] The second NWDAF can obtain the model corresponding to the Analytics ID based on the model index information, and use it to infer the data analysis results corresponding to the Analytics ID.

[0688] Optionally, if the first NWDAF determines that it cannot send model index information to the second NWDAF within the time period indicated by the time information in the second model requirement information, the first NWDAF may send a first indication message to the second NWDAF, which indicates that it cannot send model index information within the time period indicated by the time information. This first indication message may be called an Error Response or Error Notification.

[0689] Based on the above scheme, the first model requirement information can assist the second NWDAF in quickly obtaining a first NWDAF that as closely meets the first model requirement information from the NRF, thereby enabling the accurate provision of the first NWDAF to the second NWDAF. Similarly, the second model requirement information can assist the second NWDAF in quickly obtaining a model that as closely meets the second model requirement information from the first NWDAF, thereby enabling the accurate provision of the model to the second NWDAF.

[0690] refer to Figure 15 This is a schematic diagram of a communication device provided in an embodiment of this application. The communication device 1500 includes a transceiver unit 1510 and a processing unit 1520.

[0691] In the first embodiment, the communication device is used to implement the steps corresponding to the first data analysis network element in the above embodiments: The transceiver unit 1510 is configured to receive first information from the second data analysis network element, the first information including a model performance report, the model performance report indicating the performance evaluation result of the model, or the model performance report indicating that the performance evaluation result of the model does not meet the performance index requirements of the model; and to send second information to the second data analysis network element, the second information including second model information of the model. The processing unit 1520 is configured to update the first model information of the model according to the model performance report and obtain the second model information.

[0692] In one possible implementation, the transceiver unit 1510 is further configured to send third information to the second data analysis network element, the third information including the performance indicators of the model, the performance indicators of the model being used to obtain the performance evaluation results of the model.

[0693] In one possible implementation, the transceiver unit 1510 is further configured to send the second information to a third data analysis network element.

[0694] In one possible implementation, the transceiver unit 1510 is configured to receive first information from the second data analysis network element, specifically including: receiving the first information from the second data analysis network element through a network storage network element. The transceiver unit 1510 is also configured to send second information to the second data analysis network element, specifically including: sending the second information to the second data analysis network element through a network storage network element.

[0695] In one possible implementation, the model performance metrics include one or more of the following: precision, accuracy, error rate, recall, F1 score, mean squared error, root mean squared error, root mean squared logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability.

[0696] In one possible implementation, the third information further includes one or more of the following: an analysis type identifier, an identifier of the model, and an identifier of the sub-model, wherein the analysis type identifier is used to indicate the analysis type of the model.

[0697] In one possible implementation, the third information further includes one or more of the following: a reporting period and threshold information, wherein the reporting period is used to indicate the time for reporting the performance report of the model, and the threshold information is used to indicate the conditions for reporting the performance report of the model.

[0698] In one possible implementation, the first information may also include one or more of the following information corresponding to the performance report of the model: time, region, and slice.

[0699] In one possible implementation, the second information further includes one or more of the following: the identifier of the model, the identifier of the sub-model, the performance evaluation result of the model, the hardware capability information corresponding to the performance evaluation result of the model, the size of the model, and the inference time of the model.

[0700] In the second embodiment, the communication device is used to implement the steps corresponding to the second data analysis network element in the above embodiments: The transceiver unit 1510 is configured to send first information to a first data analysis network element, the first information including a model performance report, the model performance report indicating the performance evaluation result of the model, or the model performance report indicating that the performance evaluation result of the model does not meet the performance index requirements of the model; and to receive second information from the first data analysis network element, the second information including second model information of the model, the second model information being obtained by updating the first model information of the model based on the model performance report. The processing unit 1520 is configured to update the model based on the second model information.

[0701] In one possible implementation, the transceiver unit 1510 is further configured to receive third information from the first data analysis network element, the third information including the performance indicators of the model, the performance indicators of the model being used to obtain the performance evaluation results of the model.

[0702] In one possible implementation, the transceiver unit 1510 is configured to send first information to the first data analysis network element, specifically including: sending the first information to the first data analysis network element via a network storage network element. The transceiver unit 1510 is also configured to receive second information from the first data analysis network element, specifically including: receiving the second information from the first data analysis network element via a network storage network element.

[0703] In one possible implementation, the model performance metrics include one or more of the following: precision, accuracy, error rate, recall, F1 score, mean squared error, root mean squared error, root mean squared logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability.

[0704] In one possible implementation, the third information further includes one or more of the following: an analysis type identifier, an identifier of the model, and an identifier of the sub-model, wherein the analysis type identifier is used to indicate the analysis type of the model.

[0705] In one possible implementation, the third information further includes one or more of the following: a reporting period and threshold information, wherein the reporting period is used to indicate the time for reporting the performance report of the model, and the threshold information is used to indicate the conditions for reporting the performance report of the model.

[0706] In one possible implementation, the first information may also include one or more of the following information corresponding to the performance report of the model: time, region, and slice.

[0707] In one possible implementation, the second information further includes one or more of the following: the identifier of the model, the identifier of the sub-model, the performance evaluation result of the model, the hardware capability information corresponding to the performance evaluation result of the model, the size of the model, and the inference time of the model.

[0708] In the third embodiment, the communication device is used to implement the steps corresponding to the first data analysis network element in the above embodiments: Processing unit 1520 is used to update the first information of the model to the second information of the model; the first data analysis network element determines the index information of the second information of the model, the index information of the second information includes first identification information, the first identification information is used to indicate the second information of the model; transceiver unit 1510 is used to send the index information of the second information to the second data analysis network element, the index information of the second information is used to obtain the second information of the model. The index information of the second information of the model can also be referred to as the model index information corresponding to the second information.

[0709] In one possible implementation, the index information of the second information further includes one or more of the following: the analysis type identifier corresponding to the model, the identifier of the model, and the version information of the second information of the model.

[0710] In one possible implementation, the transceiver unit 1510 is used to receive index information of the first information of the model from the second data analysis network element. The index information of the first information includes second identification information, which is used to indicate the first information of the model. The processing unit 1520 is used to obtain the first information of the model according to the index information of the first information.

[0711] In one possible implementation, the index information of the first information further includes one or more of the following: the analysis type identifier corresponding to the model, the identifier of the model, and the version information of the first information of the model.

[0712] In one possible implementation, the processing unit 1520 is configured to obtain a first request from the second data analysis network element, the first request being used to update the first information of the model, the first request including index information of the first information of the model; obtain the first information of the model according to the index information of the first information; and the first data analysis network element updates the first information of the model to obtain the second information of the model.

[0713] In one possible implementation, the transceiver unit 1510 is configured to send a second request to the second data analysis network element, the second request being used to request index information of the first information of the model, the second request including an analysis type identifier corresponding to the model; and to receive a second response from the second data analysis network element, the second response including index information of the first information of the model.

[0714] In one possible implementation, the transceiver unit 1510 is used to receive index information of the first information of the model from the second data analysis network element through the network storage network element.

[0715] In one possible implementation, the transceiver unit 1510 is used to send index information of the second information of the model to the second data analysis network element through the network storage network element.

[0716] In one possible implementation, the first data analysis network element is a client-side data analysis network element in distributed learning, and the second data analysis network element is a server-side data analysis network element in distributed learning.

[0717] In one possible implementation, the distributed learning is federated learning.

[0718] In one possible implementation, the first data analysis network element is a data analysis network element that supports inference functions, and the second data analysis network element is a data analysis network element that supports training functions.

[0719] In the fourth embodiment, the communication device is used to implement the steps corresponding to the second data analysis network element in the above embodiments: The transceiver unit 1510 is used to send a first request to a first data analysis network element. The first request carries an analysis type identifier and second model requirement information. The first request is used to request model index information of the model corresponding to the analysis type identifier and satisfying the second model requirement information. The transceiver unit 1510 is also used to receive the model index information from the first data analysis network element.

[0720] In one possible implementation, the transceiver unit 1510 is configured to send a network function discovery request to a network storage network element. The network function discovery request includes the analysis type identifier and the first model requirement information. The network function discovery request is used to request the acquisition of a data analysis network element that can provide a model corresponding to the analysis type identifier and satisfying the first model requirement information. The transceiver unit 1510 also receives the address information of the first data analysis network element from the network storage network element.

[0721] In one possible implementation, the first model requirement information includes one or more of the following: analysis filtering information, analysis report target information, model performance information, or model deployment environment information; The analysis filtering information is used to indicate the scope of application of the model that the second data analysis network element needs to request. The analysis filtering information includes one or more of the following: region, time period, single network slice selection auxiliary information or data network name. The target information in the analysis report is used to indicate the terminal corresponding to the model that the second data analysis network element needs to request. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model requested by the second data analysis network element. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean square error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment for model deployment that the second data analysis network element needs to request. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

[0722] In one possible implementation, the second model requirement information includes some or all of the information in the first model requirement information.

[0723] In one possible implementation, the second model requirement information includes time information, which indicates the expected time to receive model index information from the first data analysis network element.

[0724] In one possible implementation, the transceiver unit 1510 is configured to receive first indication information from the first data analysis network element, the first indication information indicating that the model index information cannot be sent within the time period indicated by the time information.

[0725] In the fifth embodiment, the communication device is used to implement the steps corresponding to the first data analysis network element in the above embodiments: The transceiver unit 1510 is configured to receive a first request from the second data analysis network element, the first request carrying an analysis type identifier and second model requirement information, the first request being used to request model index information of the model corresponding to the analysis type identifier and satisfying the second model requirement information; and to send the model index information to the second data analysis network element; the processing unit 1520 is configured to obtain the model index information according to the second model requirement information and the analysis type identifier.

[0726] In one possible implementation, the second model requirement information includes time information, which indicates the expected time to receive model index information from the first data analysis network element.

[0727] In one possible implementation, the transceiver unit 1510 is used to send a first indication information to the second data analysis network element, the first indication information indicating that the model index information cannot be sent within the time indicated by the time information.

[0728] In one possible implementation, the transceiver unit 1510 is configured to send a network function registration request to the network storage element. The network function registration request carries the analysis type identifier and model information. The model information includes second indication information, which indicates whether the model corresponding to the analysis type identifier has been trained or is ready to be used.

[0729] In one possible implementation, when the second indication information indicates that the model corresponding to the analysis type identifier has been trained or is ready to be completed, the model information further includes model description information, which includes one or more of the following: analysis filtering information, analysis report target information, model performance information, or model deployment environment information. The analysis filtering information is used to indicate the scope of application of the model corresponding to the analysis type identifier. The analysis filtering information includes one or more of the following: region, time period, single network slice selection auxiliary information, or data network name. The target information in the analysis report is used to indicate the terminal corresponding to the model corresponding to the analysis type identifier. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model corresponding to the analysis type identifier. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment of the model deployment corresponding to the analysis type identifier. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

[0730] In the sixth embodiment, the communication device is used to implement the steps of the corresponding network storage element in the above embodiments: The transceiver unit 1510 is configured to receive a network function discovery request from a second data analysis network element, the network function discovery request including an analysis type identifier and first model requirement information, the network function discovery request being used to request the acquisition of a data analysis network element capable of providing a model corresponding to the analysis type identifier and satisfying the first model requirement information; and to send the address information of the first data analysis network element to the second data analysis network element; the processing unit 1520 is configured to acquire the address information of the first data analysis network element according to the first model requirement information and the analysis type identifier; In one possible implementation, the transceiver unit 1510 is configured to receive a network function registration request from the first data analysis network element. The network function registration request carries the analysis type identifier and model information. The model information includes second indication information, which indicates whether the model corresponding to the analysis type identifier has been trained or is ready to be used.

[0731] In one possible implementation, when the second indication information indicates that the model corresponding to the analysis type identifier has been trained or is ready to be completed, the model information further includes model description information, which includes one or more of the following: analysis filtering information, analysis report target information, model performance information, or model deployment environment information. The analysis filtering information is used to indicate the scope of application of the model corresponding to the analysis type identifier. The analysis filtering information includes one or more of the following: region, time period, single network slice selection auxiliary information, or data network name. The target information in the analysis report is used to indicate the terminal corresponding to the model corresponding to the analysis type identifier. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model corresponding to the analysis type identifier. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment of the model deployment corresponding to the analysis type identifier. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

[0732] In one possible implementation, the first model requirement information includes one or more of the following: analysis filtering information, analysis report target information, model performance information, or model deployment environment information; The analysis filtering information is used to indicate the scope of application of the model that the second data analysis network element needs to request. The analysis filtering information includes one or more of the following: region, time period, single network slice selection auxiliary information or data network name. The target information in the analysis report is used to indicate the terminal corresponding to the model that the second data analysis network element needs to request. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model requested by the second data analysis network element. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean square error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment for model deployment that the second data analysis network element needs to request. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

[0733] Optionally, the communication device may further include a storage unit for storing data or instructions (also referred to as code or program). The various units described above may interact with or be coupled to the storage unit to implement corresponding methods or functions. For example, the processing unit 1520 may read data or instructions from the storage unit, enabling the communication device to implement the methods described in the above embodiments.

[0734] It should be understood that the division of units in the above communication device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, all units in the communication device can be implemented entirely through software calls from processing elements; all units can be implemented entirely in hardware; or some units can be implemented through software calls from processing elements, while others are implemented in hardware. For example, each unit can be a separate processing element, or it can be integrated into a chip within the communication device. Alternatively, it can be stored as a program in memory, called and executed by a processing element of the communication device. Moreover, these units can be fully or partially integrated together, or implemented independently. The processing element mentioned here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be implemented through integrated logic circuits in the processor element or through software calls from processing elements.

[0735] In one example, a unit in any of the above communication devices can be one or more integrated circuits configured to implement the above methods, such as: one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms. As another example, when a unit in the communication device can be implemented in the form of a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Furthermore, these units can be integrated together and implemented as a system-on-a-chip (SOC).

[0736] Referring to Figure 16, this is a schematic diagram of a communication device provided in an embodiment of this application, used to implement the operation of the first data analysis network element or the second data analysis network element in the above embodiments. For example... Figure 16As shown, the communication device includes a processor 1610 and an interface 1630. Optionally, the communication device also includes a memory 1620. The interface 1630 is used to enable communication with other devices.

[0737] The method executed by the first or second data analysis network element in the above embodiments can be implemented by the processor 1610 calling a program stored in memory (which can be memory 1620 in the first or second data analysis network element, or external memory). That is, the first or second data analysis network element may include the processor 1610, which executes the method executed by the first or second data analysis network element in the above method embodiments by calling a program in memory. The processor here can be an integrated circuit with signal processing capabilities, such as a CPU. The first or second data analysis network element can be implemented by one or more integrated circuits configured to implement the above methods. For example: one or more ASICs, or one or more microprocessors (DSPs), or one or more FPGAs, or a combination of at least two of these integrated circuit forms. Alternatively, the above implementation methods can be combined.

[0738] Specifically, Figure 15 The functions / implementation of the transceiver unit 1510 and the processing unit 1520 can be understood through... Figure 16 The processor 1610 in the communication device 1600 shown calls computer-executable instructions stored in memory 1620 to implement the communication. Alternatively, Figure 15 The function / implementation process of the processing unit 1520 can be achieved through... Figure 16 The processor 1610 in the communication device 1600 shown calls computer execution instructions stored in memory 1620 to implement this. Figure 15 The function / implementation process of the transceiver unit 1510 in the middle can be obtained through Figure 16 The interface 1630 in the communication device 1600 shown is used to implement this. For example, the function / implementation process of the transceiver unit 1510 can be implemented by the processor calling program instructions in memory to drive the interface 1630.

[0739] Those skilled in the art will understand that the various numerical designations, such as "first," "second," etc., used in this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application, nor do they indicate a sequential order. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" refers to one or more. "At least two" refers to two or more. "At least one," "any one," or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. "Multiple" refers to two or more, and other quantifiers are similar.

[0740] It should be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this invention.

[0741] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0742] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0743] The various illustrative logic units and circuits described in the embodiments of this application can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can also be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0744] The steps of the methods or algorithms described in the embodiments of this application can be directly embedded in hardware, software units executed by a processor, or a combination of both. The software units can be stored in random access memory (RAM), flash memory, read-only memory (ROM), EPROM, EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC.

[0745] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0746] In one or more exemplary designs, the functions described herein can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media includes computer storage media and communication media that facilitate the transfer of computer programs from one location to another. Storage media can be any available media accessible to a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other formats readable by a general-purpose or special-purpose computer or processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server, or other remote resource via a coaxial cable, fiber optic computer, twisted pair, digital subscriber line (DSL), or wirelessly, such as infrared, wireless, and microwave, it is also included in the definition of a computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while discs typically copy data optically using lasers. Combinations of these can also be contained in computer-readable media.

[0747] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely illustrative descriptions of the application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Thus, if such modifications and modifications fall within the scope of the claims and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A communication method, characterized in that, include: The second data analysis network element sends a first request to the first data analysis network element. The first request carries an analysis type identifier and second model requirement information. The first request is used to request the model index information of the model corresponding to the analysis type identifier and satisfying the second model requirement information. The second model requirement information includes analysis filtering information, which includes one or more of the following: region, time period, single network slice selection auxiliary information or data network name. The second data analysis network element receives the model index information from the first data analysis network element.

2. The method as described in claim 1, characterized in that, Also includes: The second data analysis network element sends a network function discovery request to the network storage network element. The network function discovery request includes the analysis type identifier and the first model requirement information. The network function discovery request is used to request a data analysis network element that can provide a model that corresponds to the analysis type identifier and meets the first model requirement information. The second data analysis network element receives the address information of the first data analysis network element from the network storage network element.

3. The method as described in claim 2, characterized in that, The first model requirement information includes one or more of the following: analysis filtering information, analysis report target information, model performance information, or model deployment environment information; The analysis filtering information is used to indicate the scope of application of the model that the second data analysis network element needs to request. The analysis filtering information includes one or more of the following: region, time period, single network slice selection auxiliary information or data network name. The target information in the analysis report is used to indicate the terminal corresponding to the model that the second data analysis network element needs to request. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model requested by the second data analysis network element. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean square error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment for model deployment that the second data analysis network element needs to request. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

4. The method as described in claim 3, characterized in that, The second model requirement information includes some or all of the information in the first model requirement information.

5. The method according to any one of claims 1-4, characterized in that, The second model requirement information includes time information, which indicates the expected time to receive model index information from the first data analysis network element.

6. The method as described in claim 5, characterized in that, Also includes: The second data analysis network element receives a first indication from the first data analysis network element, the first indication indicating that the model index information cannot be sent within the time period indicated by the time information.

7. A communication method, characterized in that, include: The first data analysis network element receives a first request from the second data analysis network element. The first request carries an analysis type identifier and second model requirement information. The first request is used to request the model index information of the model corresponding to the analysis type identifier and satisfying the second model requirement information. The second model requirement information includes analysis filtering information, which includes one or more of the following: region, time period, single network slice selection auxiliary information, or data network name. The first data analysis network element obtains the model index information based on the second model requirement information and the analysis type identifier; The first data analysis network element sends the model index information to the second data analysis network element.

8. The method as described in claim 7, characterized in that, The second model requirement information includes time information, which indicates the expected time to receive model index information from the first data analysis network element.

9. The method as described in claim 8, characterized in that, Also includes: The first data analysis network element sends a first indication message to the second data analysis network element, the first indication message indicating that the model index information cannot be sent within the time period indicated by the time information.

10. The method according to any one of claims 7-9, characterized in that, Also includes: The first data analysis network element sends a network function registration request to the network storage network element. The network function registration request carries the analysis type identifier and model information. The model information includes second indication information, which indicates whether the model corresponding to the analysis type identifier has been trained or is ready to be used.

11. The method as described in claim 10, characterized in that, When the second indication information indicates that the model corresponding to the analysis type identifier has been trained or is ready to be completed, the model information further includes model description information, which includes one or more of the following: analysis filtering information, analysis report target information, model performance information, or model deployment environment information; The analysis filtering information is used to indicate the scope of application of the model corresponding to the analysis type identifier. The analysis filtering information includes one or more of the following: region, time period, single network slice selection auxiliary information, or data network name. The target information in the analysis report is used to indicate the terminal corresponding to the model corresponding to the analysis type identifier. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model corresponding to the analysis type identifier. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment of the model deployment corresponding to the analysis type identifier. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

12. A communication method, characterized in that, include: The network storage element receives a network function registration request from the first data analysis element. The network function registration request carries an analysis type identifier and model information. The model information includes analysis filtering information, which includes one or more of the following: region, time period, single network slice selection auxiliary information, or data network name. The network storage element receives a network function discovery request from the second data analysis element. The network function discovery request includes the analysis type identifier and the first model requirement information. The network function discovery request is used to request a data analysis element that can provide a model corresponding to the analysis type identifier and meets the first model requirement information. The first model requirement information includes the analysis filtering information. The network storage element obtains the address information of the first data analysis element based on the first model requirement information and the analysis type identifier; The network storage element sends the address information of the first data analysis element to the second data analysis element.

13. The method as described in claim 12, characterized in that, The model description information also includes one or more of the following: analysis report target information, model performance information, or model deployment environment information; The target information in the analysis report is used to indicate the terminal corresponding to the model corresponding to the analysis type identifier. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model corresponding to the analysis type identifier. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment of the model deployment corresponding to the analysis type identifier. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

14. The method as described in claim 12 or 13, characterized in that, Also includes: The model information also includes second indication information, which indicates whether the model corresponding to the analysis type identifier has been trained or is ready to be used.

15. The method according to any one of claims 12-14, characterized in that, The first model requirement information also includes one or more of the following: analysis report target information, model performance information, or model deployment environment information; The target information in the analysis report is used to indicate the terminal corresponding to the model that the second data analysis network element needs to request. The target information in the analysis report includes one or more of the following: the identifier of the terminal, the identifier of the terminal group, or information indicating any terminal. The model performance information is used to represent the performance of the model requested by the second data analysis network element. The model performance information includes one or more of the following: precision, accuracy, error rate, recall, F1 score, mean square error, root mean square error, root mean square logarithmic error, mean absolute error, model inference time, model robustness, model scalability, and model interpretability. The model deployment environment information is used to represent the hardware environment for model deployment that the second data analysis network element needs to request. The model deployment environment information includes one or more of the following: number of central processing units, number of graphics processing units, memory size, or hard disk size.

16. A communication device, characterized in that, include: A processor coupled to a memory for storing programs or instructions that, when executed by the processor, cause the device to perform the method as described in any one of claims 1-6.

17. A communication device, characterized in that, include: A processor coupled to a memory for storing programs or instructions that, when executed by the processor, cause the device to perform the method as described in any one of claims 7-11.

18. A communication device, characterized in that, include: A processor coupled to a memory for storing programs or instructions that, when executed by the processor, cause the device to perform the method as described in any one of claims 12-15.

19. A chip system, characterized in that, include: The chip system includes at least one processor and an interface circuit coupled to the at least one processor, the processor executing instructions to perform the method according to any one of claims 1-6, 7-11, and 12-15.

20. A communication system, characterized in that, It includes a first data analysis network element for performing the method of any one of claims 1-6, and a second data analysis network element for performing the method of any one of claims 7-11.

21. The system as claimed in claim 20, characterized in that, The system further includes a network storage element for performing the method of any one of claims 12-15.

Images