Communication method and apparatus

Abstract

The present application belongs to the technical field of communications. Provided are a communication method and apparatus, which are used for improving the reliability of ML model evaluation. In the method, data of different ML models can be respectively stored in different data sets, and the reliability of ML model evaluation is improved by using the data in the data sets corresponding to the ML models. For example, after an AnLF entity receives information of a first ML model, the AnLF entity can configure related data of the first ML model to be stored in a first data set, i.e., setting a correspondence between the first ML model and the first data set, and the AnLF entity sends the correspondence to an MTLF entity by means of carrying in a first message the information of the first ML model and information of the first data set, so that the MTLF entity can obtain the related data of the first ML model from the first data set, so as to evaluate the first ML model on the basis of the related data, thus improving the reliability of the evaluation of the first ML model.

Description

Communication method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 6, 2023, with application number 202310981040.8 and application name “Communication Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of communications, and in particular to a communication method and device. Background Art

[0003] The network data analytics function (NWDAF) network element has data collection, model training, data analysis and model reasoning functions. That is to say, NWDAF can be used to collect relevant data from network network elements, third-party service servers, terminal equipment or network management systems, and perform model training based on the relevant data to obtain an artificial intelligence (AI) model or a machine learning (ML) model, and provide the AI ​​model or ML model to other NWDAF network elements. The AI ​​model or ML model can be used to assist the NWDAF network element in generating data analysis results based on the relevant data. In addition, the NWDAF containing the Model Training logical function (MTLF) can be used to train machine learning (ML) models and expose new training services, such as providing trained ML models.

[0004] In large-scale scenarios, NWDAF with MTLF typically evaluates the ML model provided to NWDAF with AnLF (analytics logical function), such as accuracy and robustness. Improving the reliability of ML model evaluation is currently a hot topic.

[0005] Summary of the Invention

[0006] Embodiments of the present application provide a communication method and apparatus for improving the reliability of ML model evaluation.

[0007] To achieve the above objectives, this application adopts the following technical solutions:

[0008] In a first aspect, a communication method is provided, comprising: an analysis logic function (AnLF) functional entity receiving information about a first machine learning (ML) model from a model training logic function (MTLF) functional entity; and sending a first message to the MTLF functional entity. The information about the first ML model is used to obtain the first ML model, and the first message carries the information about the first ML model and information about a first dataset, where the first dataset is used to store data related to the first ML model.

[0009] Based on the method described in the first aspect, it can be seen that in a large-scale scenario, the relevant data of each ML model (such as the data generated by the ML model during use) is usually stored in the same data set. Because in the data set, it is impossible to distinguish which ML model each data belongs to, the MTLF functional entity usually evaluates each ML model based on all the data in the data set, but this method makes the reliability of the evaluation low. Therefore, the data of different ML models can be stored in different data sets respectively, and the reliability of the ML model evaluation can be improved by using the data in the data set corresponding to the ML model. For example, after receiving the information of the first ML model, the AnLF functional entity can set the relevant data of the first ML model to be saved in the first data set, that is, set the corresponding relationship between the first ML model and the first data set, and carry the information of the first ML model and the information of the first data set in the first message, and send the corresponding relationship to the MTLF functional entity, so that the MTLF functional entity can obtain the relevant data of the first ML model from the first data set to evaluate the first ML model based on the relevant data, thereby improving the reliability of the evaluation of the first ML model.

[0010] In one possible design, the first ML model is a plurality of ML models, the information of the first ML model is the information of the plurality of ML models, the information of the first data set is the information of the plurality of data sets, the plurality of ML models correspond one-to-one with the information of the plurality of ML models, and the information of the plurality of ML models corresponds one-to-one with the information of the plurality of data sets. That is, when the MTLF functional entity sends the plurality of ML models to the AnLF functional entity, the AnLF functional entity may set a correspondence between the plurality of ML models and the plurality of data sets, and send the correspondence to the MTLF functional entity, so that the MTLF functional entity obtains a data set corresponding to any one of the plurality of ML models and evaluates the ML model, thereby improving the reliability of the ML model evaluation.

[0011] In one possible design, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator (URL) address of the first ML model, or a fully qualified domain name (FQDN) of the first ML model. That is, by carrying this information, the first ML model can be accurately identified, ensuring that the MTLF functional entity successfully obtains the correspondence between the first ML model and the first dataset.

[0012] In one possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis storage function ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the first data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the first data set and the identification information of the first ADRF functional entity in the first message, indicating that the first data set is stored in the first ADRF functional entity. In this way, the content contained in the information of the first data set can be flexibly set according to actual conditions.

[0013] In one possible design, before the AnLF functional entity receives information about the first ML model from the MTLF functional entity, the method described in the first aspect further includes: the AnLF functional entity sends a request message to the MTLF functional entity, where the request message is used to request the ML model; accordingly, the AnLF functional entity receives information about the first ML model from the MTLF functional entity, including: in response to the request message, the AnLF functional entity receives information about the first ML model from the MTLF functional entity. That is, when the AnLF functional entity requests the ML model from the MTLF functional entity, it may not carry information about the first data set in the request message, but after receiving the first ML model, it may send the first message based on the first ML model. In this way, inaccurate information about the first data set carried in the request message can be avoided, reducing communication overhead.

[0014] In one possible design, before the AnLF functional entity receives information about the first ML model from the MTLF functional entity, the method described in the first aspect further includes: the AnLF functional entity sends a request message to the MTLF functional entity; in response to the request message, the AnLF functional entity receives information about a second ML model from the MTLF functional entity; the request message carries information about a second data set, the second data set being used to store relevant data about the ML model sent by the MTLF functional entity after receiving the request message, and the information about the second ML model being used to obtain the second ML model. It will be understood that when the AnLF functional entity requests an ML model from the MTLF functional entity, the information about the data set corresponding to the ML model is carried in the request message, which can avoid the AnLF functional entity from sending a message indicating the correspondence between the second ML model and the second data set after receiving the second ML model, thereby reducing communication overhead.

[0015] Optionally, the information of the second data set includes identification information of the second data set and / or identification information of the second ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the second data set and the second ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the second data set and the identification information of the second ADRF functional entity in the request message, indicating that the second data set is stored in the second ADRF functional entity. In this way, the content contained in the information of the second data set can be flexibly set according to actual conditions.

[0016] Optionally, the first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

[0017] In one possible design scheme, the method described in the first aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0018] Alternatively, in another possible design scheme, the method described in the first aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, and the request for monitoring the ML model carries information of the first ML model; in response to the request for monitoring the ML model, the AnLF functional entity sends information of a third data set to the MTLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0019] Alternatively, in another possible design scheme, the method described in the first aspect also includes: the AnLF functional entity monitors the analysis accuracy of the first ML model; when the analysis accuracy of the first ML model is insufficient, the AnLF functional entity sends a notification request message to the MTLF functional entity, and the notification request message carries information of a third data set, and the third data set is used to save relevant data of the first ML model.

[0020] In the process of the AnLF functional entity monitoring the accuracy of the first ML model, the relevant data of the first ML model (such as the data of the deviation sent during prediction) can be stored in the third data set, so that the MTLF functional entity can subsequently evaluate, retrain or regenerate the ML model based on the data of the third data set. It can be seen that the correspondence between the first ML model and the third data set can be carried in the request for monitoring the ML model; or, it can be carried in the response for monitoring the ML model; or, it can be carried in the notification request message. That is to say, according to the actual situation, the AnLF functional entity or the MTLF functional entity can determine the third data set, and send the corresponding relationship between the relevant data of the first ML model and the third data set to the other party.

[0021] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the third data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the third data set and the identification information of the third ADRF functional entity in the request message, indicating that the third data set is stored in the third ADRF functional entity. In this way, the content contained in the information of the third data set can be flexibly set according to actual conditions.

[0022] In one possible design, the relevant data includes at least one of the following: input data, predicted data, ground truth data during prediction, and deviation data during prediction. The ground truth data during the ML model's prediction refers to the actual results of the ML model's input data, not the predicted results; the deviation data during the ML model's prediction refers to data where the predicted results deviate. In other words, the relevant data includes data generated during the use of the ML model.

[0023] In one possible design, the AnLF functional entity is hosted on a network data analysis function (NWDAF) network element; and / or the MTLF functional entity is hosted on an NWDAF network element. It is understood that the AnLF functional entity and the MTLF functional entity can be hosted on different NWDAF network elements or on the same NWDAF network element, and this embodiment of the present application does not impose any limitation thereto.

[0024] According to a second aspect, a communication method is provided, which includes: a model training logic function MTLF functional entity sends information of a first machine learning ML model to an analysis logic function AnLF functional entity, wherein the information of the first ML model is used to indicate the first ML model; the MTLF functional entity receives a first message from the AnLF functional entity, wherein the first message carries information of the first ML model and information of a first data set, and the first data set is used to store relevant data of the first ML model.

[0025] In one possible design scheme, the first ML model is multiple ML models, the information of the first ML model is the information of multiple ML models, the information of the first data set is the information of multiple data sets, the multiple ML models correspond one-to-one to the information of the multiple ML models, and the information of the multiple ML models corresponds one-to-one to the information of the multiple data sets.

[0026] In a possible design solution, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator (URL) address of the first ML model, and a fully qualified domain name (FQDN) of the first ML model.

[0027] In a possible design solution, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0028] In one possible design scheme, before the MTLF functional entity sends information about the first ML model to the AnLF functional entity, the method described in the second aspect also includes: the MTLF functional entity receives a request message from the AnLF functional entity, and the request message is used to request the ML model; accordingly, the MTLF functional entity sends information about the first ML model to the AnLF functional entity, including: upon receiving the request message, the MTLF functional entity sends information about the first ML model to the AnLF functional entity.

[0029] In one possible design scheme, before the MTLF functional entity sends information of the first ML model to the AnLF functional entity, the method described in the second aspect also includes: the MTLF functional entity receives a request message from the AnLF functional entity; upon receiving the request message, the MTLF functional entity sends information of the second ML model to the AnLF functional entity; the request message is used to request the ML model, and the request message carries information of a second data set. The second data set is used to store relevant data of the ML model sent by the MTLF functional entity after receiving the request message, and the information of the second ML model is used to obtain the second ML model.

[0030] Optionally, the information of the second data set includes identification information of the second data set and / or identification information of the second ADRF functional entity.

[0031] Optionally, the first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

[0032] Optionally, the method of the second aspect further includes: the MTLF functional entity evaluating the second ML model based on the second dataset. Because the second dataset only stores data related to the second ML model, using the second dataset to evaluate the second ML model can improve the reliability of the evaluation of the second ML model.

[0033] In one possible design, the method of the second aspect further includes: the MTLF functional entity evaluating the first ML model based on the first dataset. Because the first dataset only stores data relevant to the first ML model, using the first dataset to evaluate the first ML model can improve the reliability of the evaluation of the first ML model.

[0034] In one possible design scheme, the method described in the second aspect also includes: when the MTLF functional entity receives a registration request for ML model monitoring, the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity, and the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0035] Alternatively, in another possible design scheme, the method described in the second aspect also includes: the MTLF functional entity receives information of a third data set from the AnLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0036] Optionally, information of the third data set is carried in a response and / or notification request message for ML model monitoring.

[0037] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity.

[0038] In one possible design solution, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0039] In a possible design solution, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0040] In addition, other technical effects of the method described in the second aspect can also refer to the technical effects of the method described in the first aspect, and will not be repeated here.

[0041] According to a third aspect, a communication method is provided, comprising: a model training logic function (MTLF) functional entity receiving a request message from an analysis logic function (AnLF) functional entity; and sending a first message to the AnLF functional entity. The request message is used to request a machine learning (ML) model, and the first message carries information about the first ML model and information about a first dataset, the information about the first ML model being used to indicate the first ML model, and the first dataset being used to store data related to the first ML model.

[0042] Based on the method described in the third aspect, it can be seen that in a large-scale scenario, the relevant data of each ML model (such as the data generated by the ML model during use) is usually stored in the same data set. Because in the data set, it is impossible to distinguish which ML model each data belongs to, the MTLF functional entity usually evaluates each ML model based on all the data in the data set, but this method makes the reliability of the evaluation low. Therefore, the data of different ML models can be stored in different data sets respectively, and the reliability of the ML model evaluation can be improved by using the data in the data set corresponding to the ML model. For example, after the MTLF functional entity receives a request message for requesting an ML model from the AnLF functional entity, it can be set to save the relevant data of the first ML model in the first data set, that is, set the corresponding relationship between the first ML model and the first data set, and carry the information of the first ML model and the information of the first data set in the first message, and send the first ML model and the corresponding relationship to the AnLF functional entity, so that the AnLF functional entity can obtain the first ML model according to the information of the first ML model, and save the relevant information of the first ML model in the first data set according to the corresponding relationship. Furthermore, the MTLF functional entity may obtain relevant data of the first ML model from the first data set to evaluate the first ML model based on the relevant data, thereby improving the reliability of the evaluation of the first ML model.

[0043] In one possible design, the first ML model is a plurality of ML models, the information of the first ML model is the information of the plurality of ML models, the information of the first data set is the information of the plurality of data sets, the plurality of ML models correspond one-to-one with the information of the plurality of ML models, and the information of the plurality of ML models corresponds one-to-one with the information of the plurality of data sets. That is, when the MTLF functional entity sends the plurality of ML models to the AnLF functional entity, the AnLF functional entity may set a correspondence between the plurality of ML models and the plurality of data sets, and send the correspondence to the MTLF functional entity, so that the MTLF functional entity obtains a data set corresponding to any one of the plurality of ML models and evaluates the ML model, thereby improving the reliability of the ML model evaluation.

[0044] In one possible design, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator (URL) address of the first ML model, or a fully qualified domain name (FQDN) of the first ML model. That is, by carrying this information, the first ML model can be accurately identified, ensuring that the MTLF functional entity successfully obtains the correspondence between the first ML model and the first dataset.

[0045] In one possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis storage function ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the first data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the first data set and the identification information of the first ADRF functional entity in the first message, indicating that the first data set is stored in the first ADRF functional entity. In this way, the content contained in the information of the first data set can be flexibly set according to actual conditions.

[0046] In one possible design, the method of the third aspect further includes: the MTLF functional entity sends a request to the first ADRF functional entity to establish the first data set in the first ADRF functional entity. In this way, the MTLF functional entity can set a correspondence between the first ML model and the first data set.

[0047] In one possible design, the method of the third aspect further includes: the MTLF functional entity evaluating the first ML model based on the first dataset. Because the first dataset only stores data relevant to the first ML model, using the first dataset to evaluate the first ML model can improve the reliability of the evaluation of the first ML model.

[0048] In one possible design scheme, the method described in the third aspect also includes: when the MTLF functional entity receives a registration request for ML model monitoring, the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity, and the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0049] Alternatively, in another possible design scheme, the method described in the third aspect also includes: the MTLF functional entity receives information of a third data set from the AnLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0050] During the process of the AnLF functional entity monitoring the accuracy of the first ML model, the relevant data of the first ML model can be stored in the third data set, so that the MTLF functional entity can subsequently evaluate, retrain, or regenerate the first ML model based on the data of the third data set. It can be seen that the correspondence between the first ML model and the third data set can be carried in the request for monitoring the ML model, or sent by the AnLF functional entity to the MTLF functional entity. In other words, according to actual conditions, the third data set can be determined by the AnLF functional entity or the MTLF functional entity, and the correspondence between the relevant data of the first ML model and the third data set can be sent to the other party.

[0051] Optionally, information about the third dataset is included in a response and / or notification request message for ML model monitoring. That is, the AnLF functional entity can send the correspondence between the first ML model and the third dataset to the MTLF functional entity at different stages of monitoring the accuracy of the first ML model. This allows for flexible scheduling of the timing of sending this correspondence.

[0052] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the third data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the third data set and the identification information of the third ADRF functional entity in the request message, indicating that the third data set is stored in the third ADRF functional entity. In this way, the content contained in the information of the third data set can be flexibly set according to actual conditions.

[0053] In one possible design, the relevant data includes at least one of the following: input data, predicted data, ground truth data during prediction, and deviation data during prediction. The ground truth data during the ML model's prediction refers to the actual results of the ML model's input data, not the predicted results; the deviation data during the ML model's prediction refers to data where the predicted results deviate. In other words, the relevant data includes data generated during the use of the ML model.

[0054] In one possible design, the AnLF functional entity is hosted on a network data analysis function (NWDAF) network element; and / or the MTLF functional entity is hosted on an NWDAF network element. It is understood that the AnLF functional entity and the MTLF functional entity can be hosted on different NWDAF network elements or on the same NWDAF network element, and this embodiment of the present application does not impose any limitation thereto.

[0055] In a fourth aspect, a communication method is provided, comprising: an analysis logic function (AnLF) functional entity sending a request message to a model training logic function (MTLF) functional entity; and receiving a first message from the MTLF functional entity. The request message is used to request a machine learning (ML) model, and the first message carries information about the first ML model and information about a first dataset, the information about the first ML model being used to obtain the first ML model, and the first dataset being used to store data related to the first ML model.

[0056] In one possible design scheme, the first ML model is multiple ML models, the information of the first ML model is the information of multiple ML models, the information of the first data set is the information of multiple data sets, the multiple ML models correspond one-to-one to the information of the multiple ML models, and the information of the multiple ML models corresponds one-to-one to the information of the multiple data sets.

[0057] In a possible design solution, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator (URL) address of the first ML model, and a fully qualified domain name (FQDN) of the first ML model.

[0058] In a possible design solution, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0059] In one possible design scheme, the method of the fourth aspect further includes: the AnLF functional entity sends relevant data of the first ML model and information of the first dataset to the first ADRF functional entity based on information of the first dataset. In this way, the first ADRF functional entity can accurately save the relevant data of the first ML model in the first dataset based on the information of the first dataset.

[0060] In one possible design scheme, the method described in the fourth aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0061] Alternatively, in another possible design scheme, the method described in the fourth aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, and the request for monitoring the ML model carries information of the first ML model; in response to the request for monitoring the ML model, the AnLF functional entity sends information of a third data set to the MTLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0062] Alternatively, in another possible design scheme, the method described in the fourth aspect also includes: the AnLF functional entity monitors the analysis accuracy of the first ML model; when the analysis accuracy of the first ML model is insufficient, the AnLF functional entity sends a notification request message to the MTLF functional entity, and the notification request message carries information of a third data set, and the third data set is used to save relevant data of the first ML model.

[0063] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity.

[0064] In one possible design solution, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0065] In a possible design solution, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0066] In addition, other technical effects of the method described in the fourth aspect can also refer to the technical effects of the method described in the third aspect, and will not be repeated here.

[0067] In a fifth aspect, a communication method is provided, comprising: a model training logic function (MTLF) functional entity receiving a request message from an analysis logic function (AnLF) functional entity; the MTLF functional entity sending a request to a data analysis and storage function (ADRF) functional entity for obtaining relevant data of a first ML model; and receiving relevant data of the first ML model from the ADRF functional entity. The request message carries information of a first data set, the first data set being used to store relevant data of multiple ML models, the multiple ML models including the first ML model; and the request for obtaining relevant data of the first ML model carries information of the first ML model and information of the first data set.

[0068] Based on the method described in the first aspect, it can be seen that in a large-scale scenario, the relevant data of each ML model (such as the data generated by the ML model during use) is usually stored in the same data set. Because in this data set, it is impossible to distinguish which ML model each data belongs to, the MTLF functional entity usually evaluates each ML model based on all the data in the data set, but this method makes the reliability of the evaluation low. Therefore, the data of different ML models can be marked with corresponding model labels and then stored in the same data set; by using data with model labels corresponding to the ML model, the reliability of the evaluation of the ML model can be improved. For example, the MTLF functional entity can obtain the data set label from the received request message, that is, it knows that the data of each ML model is stored in the data set corresponding to the data set label; then send a request for obtaining the relevant data of the first ML model to the ADRF functional entity, the request carries the information of the first ML model and the information of the first data set, so that the ADRF functional entity can obtain the relevant data with the information of the first ML model in the first data set from its stored content, and send the relevant data to the MTLF functional entity, so that the MTLF functional entity evaluates the first ML model based on the relevant data, thereby improving the reliability of the evaluation of the first ML model.

[0069] In one possible design, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator (URL) address of the first ML model, or a fully qualified domain name (FQDN) of the first ML model. That is, by carrying this information, the first ML model can be accurately identified, ensuring that the MTLF functional entity successfully obtains the correspondence between the first ML model and the first dataset.

[0070] In one possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis storage function ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the first data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the first data set and the identification information of the first ADRF functional entity in the first message, indicating that the first data set is stored in the first ADRF functional entity. In this way, the content contained in the information of the first data set can be flexibly set according to actual conditions.

[0071] In one possible design, the method of aspect 5 further includes: the MTLF functional entity sending a first message to the AnLF functional entity, where the first message carries information about the first ML model, where the information about the first ML model is used to indicate the first ML model. In this way, the AnLF functional entity can obtain the first ML model based on the information about the first ML model.

[0072] In one possible design scheme, the method described in the fifth aspect also includes: the MTLF functional entity retrains or regenerates the second ML model based on relevant data of the first ML model.

[0073] Optionally, the method described in the fifth aspect further includes: the MTLF functional entity sending information about the second ML model to the AnLF functional entity, where the information about the second ML model is used to indicate the second ML model. That is, after retraining or regenerating the second ML model, the MTLF functional entity may send the second ML model to the AnLF functional entity, so that the AnLF functional entity can update the ML model used, i.e., use the latest ML model, thereby improving the accuracy of ML model prediction.

[0074] In one possible design scheme, the method described in the fifth aspect also includes: when the MTLF functional entity receives a registration request for ML model monitoring, the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity, and the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0075] Alternatively, in another possible design scheme, the method described in the fifth aspect also includes: the MTLF functional entity receives information of a third data set from the AnLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0076] During the process of the AnLF functional entity monitoring the accuracy of the first ML model, the relevant data of the first ML model can be stored in the third data set, so that the MTLF functional entity can subsequently evaluate, retrain, or regenerate the first ML model based on the data of the third data set. It can be seen that the correspondence between the first ML model and the third data set can be carried in the request for monitoring the ML model, or sent by the AnLF functional entity to the MTLF functional entity. In other words, according to actual conditions, the third data set can be determined by the AnLF functional entity or the MTLF functional entity, and the correspondence between the relevant data of the first ML model and the third data set can be sent to the other party.

[0077] Optionally, information about the third dataset is included in a response and / or notification request message for ML model monitoring. That is, the AnLF functional entity can send the correspondence between the first ML model and the third dataset to the MTLF functional entity at different stages of monitoring the accuracy of the first ML model. This allows for flexible scheduling of the timing of sending this correspondence.

[0078] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the third data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the third data set and the identification information of the third ADRF functional entity in the request message, indicating that the third data set is stored in the third ADRF functional entity. In this way, the content contained in the information of the third data set can be flexibly set according to actual conditions.

[0079] In one possible design, the relevant data includes at least one of the following: input data, predicted data, ground truth data during prediction, and deviation data during prediction. The ground truth data during the ML model's prediction refers to the actual results of the ML model's input data, not the predicted results; the deviation data during the ML model's prediction refers to data where the predicted results deviate. In other words, the relevant data includes data generated during the use of the ML model.

[0080] In one possible design, the AnLF functional entity is hosted on a network data analysis function (NWDAF) network element; and / or the MTLF functional entity is hosted on an NWDAF network element. It is understood that the AnLF functional entity and the MTLF functional entity can be hosted on different NWDAF network elements or on the same NWDAF network element, and this embodiment of the application does not impose any limitation.

[0081] In a sixth aspect, a communication method is provided, which includes: a data analysis and storage function ADRF functional entity receives relevant data of a first machine learning ML model, information of the first ML model, and information of a first data set from an analysis logic function AnLF functional entity; and stores relevant data of the first ML model based on the information of the first ML model and the information of the first data set; the first data set is used to save the relevant data of the first ML model.

[0082] In a possible design solution, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator (URL) address of the first ML model, and a fully qualified domain name (FQDN) of the first ML model.

[0083] In a possible design solution, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0084] In a possible design scheme, the method described in the sixth aspect also includes: the ADRF functional entity receives a request from the model training logic function MTLF functional entity for obtaining relevant data of the first ML model, and the request for obtaining relevant data of the first ML model includes information of the first ML model and information of the first data set; the ADRF functional entity sends the relevant data of the first ML model to the MTLF functional entity based on the information of the first ML model and the information of the first data set.

[0085] In one possible design solution, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0086] In a possible design solution, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0087] In addition, other technical effects of the method described in the sixth aspect can also refer to the technical effects of the method described in the fifth aspect, and will not be repeated here.

[0088] In a seventh aspect, a communication method is provided, comprising: an analysis logic function (AnLF) functional entity receiving a first message from a model training logic function (MTLF) functional entity; and sending relevant data of a first ML model, information about the first ML model, and information about a first dataset to a data analysis and storage function (ADRF) functional entity. The first message carries the information about the first ML model, the information about the first ML model is used to obtain the first ML model, and the first dataset is used to store relevant data of multiple ML models, the multiple ML models including the first ML model.

[0089] In a possible design solution, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator (URL) address of the first ML model, and a fully qualified domain name (FQDN) of the first ML model.

[0090] In a possible design solution, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0091] In a possible design solution, the method of the seventh aspect further includes: the AnLF functional entity establishing the first data set. That is, the AnLF functional entity may send a request to the ADRF functional entity to establish the first data set at the ADRF functional entity, thereby establishing the first data set.

[0092] In one possible design scheme, the method described in the seventh aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, the request for monitoring the ML model carries information of the first ML model and information of the third data set, and the third data set is used to store relevant data of the first ML model.

[0093] Alternatively, in another possible design scheme, the method described in the seventh aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, and the request for monitoring the ML model carries information of the first ML model; in response to the request for monitoring the ML model, the AnLF functional entity sends information of a third data set to the MTLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0094] Alternatively, in another possible design scheme, the method described in the seventh aspect also includes: the AnLF functional entity monitors the analysis accuracy of the first ML model; when the analysis accuracy of the first ML model is insufficient, the AnLF functional entity sends a notification request message to the MTLF functional entity, and the notification request message carries information of a third data set, and the third data set is used to save relevant data of the first ML model.

[0095] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity.

[0096] In one possible design solution, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0097] In a possible design solution, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0098] In addition, other technical effects of the method described in the seventh aspect can also refer to the technical effects of the method described in the fifth aspect, and will not be repeated here.

[0099] In an eighth aspect, a communication device is provided. The communication device includes: a module for executing the method described in any possible implementation of aspects 1 to 7, such as a transceiver module and a processing module. For example, the transceiver module is configured to indicate the transceiver function of the communication device, and the processing module is configured to execute functions of the communication device other than the transceiver function.

[0100] Optionally, the transceiver module may include a sending module and a receiving module, wherein the sending module is used to implement the sending function of the communication device described in the eighth aspect, and the receiving module is used to implement the receiving function of the communication device described in the eighth aspect.

[0101] Optionally, the communication device described in the eighth aspect may further include a storage module, wherein the storage module stores a program or instruction. When the processing module executes the program or instruction, the communication device can execute the method described in any possible implementation manner of the first to seventh aspects.

[0102] It can be understood that the communication device described in the eighth aspect can be a network device, or a chip (system) or other parts or components that can be set in the network device, or a device that includes a network device. This application does not limit this.

[0103] In addition, the technical effects of the communication device described in the eighth aspect can refer to the technical effects of the method described in any possible implementation method in the first to seventh aspects, and will not be repeated here.

[0104] In a ninth aspect, a communication device is provided, comprising: a processor configured to execute the method as described in any possible implementation of the first to fifth aspects.

[0105] In one possible design solution, the communication device described in aspect 9 may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the communication device described in aspect 9 to communicate with other communication devices.

[0106] In one possible design, the communication device described in aspect 9 may further include a memory. The memory may be integrated with the processor or provided separately. The memory may be used to store computer programs and / or data involved in the method described in any possible implementation of aspects 1 to 7.

[0107] In an embodiment of the present application, the communication device described in the ninth aspect may be a terminal device or a network device, or a chip (system) or other parts or components that can be set in the terminal device or network device, or a device that includes the terminal device or network device.

[0108] In addition, the technical effects of the communication device described in the ninth aspect can refer to the technical effects of the method described in any possible implementation method of the first to seventh aspects, and will not be repeated here.

[0109] In a tenth aspect, a communication device is provided. The communication device includes: a processor coupled to a memory, the processor being configured to execute a computer program stored in the memory, so that the communication device performs the method described in any possible implementation of the first to seventh aspects.

[0110] In one possible design solution, the communication device described in the tenth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the communication device described in the tenth aspect to communicate with other communication devices.

[0111] In an embodiment of the present application, the communication device described in the tenth aspect may be a terminal device or a network device, or a chip (system) or other parts or components that can be set in the terminal device or network device, or a device that includes the terminal device or network device.

[0112] In addition, the technical effects of the communication device described in the tenth aspect can refer to the technical effects of the method described in any possible implementation method of the first aspect to the seventh aspect, and will not be repeated here.

[0113] In the eleventh aspect, a communication device is provided, comprising: a processor and a memory; the memory is used to store a computer program, and when the processor executes the computer program, the communication device executes the method described in any possible implementation method of the first to seventh aspects.

[0114] In one possible design solution, the communication device described in the eleventh aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the communication device described in the eleventh aspect to communicate with other communication devices.

[0115] In an embodiment of the present application, the communication device described in the eleventh aspect may be a terminal device or a network device, or a chip (system) or other parts or components that can be set in the terminal device or network device, or a device that includes the terminal device or network device.

[0116] In addition, the technical effects of the communication device described in the eleventh aspect can refer to the technical effects of the method described in any possible implementation method of the first aspect to the seventh aspect, and will not be repeated here.

[0117] In the twelfth aspect, a communication system is provided. The communication system includes: an analysis logic function (AnLF) functional entity, a model training logic function (MTLF) functional entity, and a data analysis and storage function (ADRF) functional entity network element. The AnLF functional entity is configured to execute the method described in any one of the first, fourth, or seventh aspects; the MTLF functional entity is configured to execute the method described in any one of the second, third, or fifth aspects; and the ADRF functional entity network element is configured to execute the method described in the sixth aspect.

[0118] In a thirteenth aspect, a communication chip is provided, in which instructions are stored. When the chip runs on a communication device, the method described in any one of the implementation methods of the first to seventh aspects is implemented.

[0119] In a fourteenth aspect, a computer-readable storage medium is provided, comprising: a computer program or instructions; when the computer program or instructions are run on a computer, the computer is caused to execute the method described in any possible implementation method of the first to fifth aspects.

[0120] In the fifteenth aspect, a computer program product is provided, comprising a computer program or instructions, which, when executed on a computer, enables the computer to execute the method described in any one of the possible implementations of aspects one to five. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] FIG1 is a schematic diagram of the architecture of a fifth-generation mobile communication system 5GS provided in an embodiment of the present application;

[0122] FIG2 is a schematic diagram of a process for monitoring the accuracy of a machine learning ML model based on a model training logic function MTLF according to an embodiment of the present application;

[0123] FIG3 is a schematic diagram of a process for monitoring the accuracy of ML model analysis according to an embodiment of the present application;

[0124] FIG4 is a schematic diagram of the architecture of a communication system provided in an embodiment of the present application;

[0125] FIG5 is a flow chart of a communication method according to an embodiment of the present application;

[0126] FIG6 is a second flow chart of the communication method provided in an embodiment of the present application;

[0127] FIG7 is a third flow chart of the communication method provided in an embodiment of the present application;

[0128] FIG8 is a fourth flow chart of a communication method according to an embodiment of the present application;

[0129] FIG9 is a fifth flow chart of a communication method according to an embodiment of the present application;

[0130] FIG10 is a first structural diagram of a communication device provided in an embodiment of the present application;

[0131] FIG11 is a second structural diagram of the communication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0132] For ease of understanding, the technical terms involved in the embodiments of this application are first introduced below.

[0133] 1. Fifth generation (5G) mobile communication system (abbreviated as 5G system (5G system, 5GS)):

[0134] Figure 1 is a schematic diagram of the 5GS architecture. As shown in Figure 1, 5GS includes: an access network (AN) and a core network (CN), and may also include: terminal equipment.

[0135] The terminal device may be a terminal device with transceiver functions, or a chip or chip system that can be provided in the terminal device. The terminal device may also be referred to as user equipment (UE), access terminal, subscriber unit, user station, mobile station (MS), mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device. The terminal in the embodiments of the present application can be a mobile phone, a cellular phone, a smart phone, a tablet computer, a wireless data card, a personal digital assistant (PDA), a wireless modem, a handset, a laptop computer, a machine type communication (MTC) terminal, a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, a vehicle-mounted terminal, a road side unit (RSU) with terminal function, etc. The terminal device of the present application may also be an on-board module, on-board module, on-board component, on-board chip or on-board unit built into the vehicle as one or more components or units.

[0136] The AN is used to implement access-related functions. It can provide network access for authorized users in a specific area and determine transmission links of varying quality to transmit user data based on user level and service requirements. The AN forwards control signals and user data between terminals and the CN. The AN may include access network equipment, also known as radio access network (RAN) equipment. The CN is primarily responsible for maintaining mobile network subscription data and providing terminal devices with functions such as session management, mobility management, policy management, and security authentication. CN mainly includes the following: user plane function (UPF), authentication server function (AUSF), access and mobility management function (AMF), session management function (SMF), network slice selection function (NSSF), network exposure function (NEF), network repository function (NRF), policy control function (PCF), unified data management (UDM), unified data repository (UDR), application function (AF), network data analytics function (NWDAF), and analytics data repository function (ADRF).

[0137] As shown in Figure 1, the UE accesses the 5G network through the RAN equipment. The UE communicates with the AMF through the N1 interface (referred to as N1); the RAN communicates with the AMF through the N2 interface (referred to as N2); the RAN communicates with the UPF through the N3 interface (referred to as N3); the SMF communicates with the UPF through the N4 interface (referred to as N4), and the UPF accesses the data network (DN) through the N6 interface (referred to as N6). In addition, the control plane functions shown in Figure 1, such as AUSF, AMF, SMF, NSSF, NEF, NRF, PCF, UDM, UDR, AF, NWDAF, and ADRF, interact using service-based interfaces. For example, the service-oriented interface provided by AUSF to the outside world includes Nausf; the service-oriented interface provided by AMF to the outside world includes Namf; the service-oriented interface provided by SMF to the outside world includes Nsmf; the service-oriented interface provided by NSSF to the outside world includes Nnssf; the service-oriented interface provided by NEF to the outside world includes Nnef; the service-oriented interface provided by NRF to the outside world includes Nnrf; the service-oriented interface provided by PCF to the outside world includes Npcf; the service-oriented interface provided by UDM to the outside world includes Nudm; the service-oriented interface provided by UDR to the outside world includes Nudr; the service-oriented interface provided by AF to the outside world includes Naf; the service-oriented interface provided by NWDAF to the outside world includes Nnwdaf; the service-oriented interface provided by ADRF to the outside world includes Nadrf.

[0138] RAN equipment can be equipment that provides access to terminal equipment. For example, the RAN equipment may include: a next-generation mobile communication system, such as an access network device of 6G, such as a 6G base station, or in the next-generation mobile communication system, the network equipment may also have other naming methods, which are all included in the protection scope of the embodiments of this application, and this application does not impose any restrictions on this. Alternatively, the RAN equipment may also include 5G, such as a gNB in ​​a new radio (NR) system, or one or a group of (including multiple antenna panels) antenna panels of a base station in 5G, or a network node constituting a gNB, a transmission point (TRP or transmission point, TP) or a transmission measurement function (TMF), such as a baseband unit (building base band unit, BBU), or a centralized unit (CU) or a distributed unit (DU), an RSU with base station function, or a wired access gateway, or a 5G core network. Alternatively, RAN devices may also include access points (APs) in wireless fidelity (WiFi) systems, wireless relay nodes, wireless backhaul nodes, various forms of macro base stations, micro base stations (also known as small stations), relay stations, access points, wearable devices, vehicle-mounted devices, and the like.

[0139] UPF is mainly responsible for user data processing (forwarding, receiving, billing, etc.).

[0140] AUSF is mainly used to perform security authentication of terminal devices.

[0141] AMF is mainly used for mobility management in mobile networks, such as user location update, user network registration, and user handover.

[0142] The SMF is primarily used for session management in mobile networks, such as session establishment, modification, and release. Specific functions include allocating Internet Protocol (IP) addresses to users and selecting the UPF that provides packet forwarding capabilities.

[0143] PCF primarily supports providing a unified policy framework to control network behavior, providing policy rules to the control layer network functions, and is responsible for obtaining user subscription information related to policy decisions. PCF can provide policies to AMF and SMF, such as quality of service (QoS) policies and slice selection policies.

[0144] NSSF is mainly used to select network slices for terminal devices.

[0145] NEF is mainly used to support the opening of capabilities and events. For example, NEF can open some capabilities of the 5G network to third-party applications through application program interfaces (APIs). Third-party applications can obtain some capabilities of the 5G network by calling the APIs provided by NEF through AF, allowing third-party applications to control certain behaviors of the 5G network and terminal devices.

[0146] UDM is mainly used to store user data, such as contract data, authentication / authorization data, etc.

[0147] UDR is mainly used to store structured data, including contract data and policy data, externally exposed structured data, and application-related data.

[0148] AF mainly supports interaction with CN to provide services, such as influencing data routing decisions, policy control functions, or providing some third-party services to the network side.

[0149] The NWDAF network element supports collecting data from other network functions and AF network elements; supports collecting data from operation administration and maintenance (OAM) network elements; and supports providing analysis information to other network functions and AF network elements. In other words, the NWDAF network element has the functions of data collection, model training, data analysis, and model reasoning, that is, it can be used to collect relevant data from network network elements, third-party service servers, terminal devices, or network management systems; perform data analysis based on the relevant data to obtain analysis results; and provide analysis results to network network elements, third-party service servers, terminal devices, or network management systems. The analysis results can assist the network in selecting service quality parameters for the service, or assist the network in executing traffic routing, or assist the network in selecting background data transmission strategies, etc. In addition, the NWDAF network element can also be used to collect relevant data from network network elements, third-party service servers, terminal devices, or network management systems, and perform model training based on the relevant data to obtain an AI model or ML model, and provide the AI ​​model or ML model to other NWDAF network elements. The AI ​​model or ML model can be used to assist the NWDAF network element in generating data analysis results based on the relevant data.

[0150] NWDAF with AnLF can be used to perform model inference, export analytical information (i.e., export statistics and / or predictions based on analytical consumer requests), and expose analytical services. NWDAF with MTLF can be used to train ML models and expose new training services, such as providing trained ML models.

[0151] ADRF network elements can provide storage and retrieval services for data and analytics for consumers. For example, a consumer sends a Nadrf Data Management Storage Request to an ADRF network element containing the data or analytics to be stored; or a consumer sends a Nadrf Data Management Storage Subscription Request to an ADRF network element, requesting the ADRF network element to subscribe to receive data or analytics for storage. The ADRF network element then subscribes to an NWDAF network element or a data collection coordination function (DCCF) network element to obtain the data or analytics, providing the ADRF network element notification address or notification association ID. The analytics or data are then provided as notifications using the DCCF Data Management (Ndccf_DataManagement) service, the NWDAF Data Management (Nnwdaf_DataManagement) service, or the MFAF (messaging framework adaptor NF) Data Management (Nmfaf_3caDataManagement) service.

[0152] It can be understood that the functions mentioned in the embodiments of the present application can also be expressed as functional network elements or functional entities. For example, UPF can be expressed as UPF network element, AMF can be expressed as AMF network element, SMF can be expressed as SMF network element, PCF can be expressed as PCF network element, and so on, without limitation.

[0153] 2. Process of ML model accuracy monitoring based on MTLF:

[0154] Figure 2 illustrates the process of monitoring the accuracy of a configured ML model using newly collected data. The NWDAF with AnLF can provide inference data to the NWDAF with MTLF for model accuracy monitoring, and the NWDAF with MTLF can determine whether to retrain or reconfigure the ML model. The following details the MTLF-based ML model accuracy monitoring process.

[0155] S201: An analytics consumer initiates an analytics exposure service subscription (Nnwdaf_AnalyticsSubscription_subscribe) service operation to the NWDAF including the AnLF.

[0156] S202 , the NWDAF including the AnLF requests the ML model from the NWDAF including the MTLF using the ML model provision subscription (Nnwdaf_MLModelProvision_Subscribe) service operation.

[0157] NWDAFs with AnLF can include analysis accuracy thresholds as indicators for performing accuracy monitoring operations. NWDAFs with AnLF may include dataset tags (DataSetTag) and / or ADRFID for storing inference data and obtaining inference data from ADRF. Inference data is relevant for monitoring the accuracy of ML models, as well as for retraining or reconfiguring them. This inference data can include input data, prediction data, and ground truth data at the time of prediction.

[0158] S203 , the NWDAF including the MTLF provides the trained ML model to the NWDAF including the AnLF.

[0159] The NWDAF including the MTLF may include accuracy information indicating the accuracy of the ML model during training. The NWDAF including the MTLF may send the ML model to the NWDAF including the AnLF via the ML model provision notification (Nnwdaf_MLModelProvision_Notify) service operation.

[0160] S204 , the NWDAF including the AnLF registers the use of the ML model with the NWDAF including the MTLF to indicate its ability to send analysis feedback information and / or ML model accuracy information from analysis consumers.

[0161] NWDAF including AnLF can register the use of ML models with NWDAF including MTLF through the ML Model Monitor Registration (Nnwdaf_MLModelMonitor_Register) service operation.

[0162] S205, due to the registration in the previous step, if the NWDAF containing ANLF supports the service operation, the NWDAF containing MTLF can subscribe to the NWDAF containing ANLF by calling the ML model monitoring subscription (Nnwdaf_MLModelMonitor_Subscribe) service operation to obtain analysis feedback information and / or ML model accuracy information from the analysis consumer.

[0163] S206 , the analytics consumer may send analytics feedback information in an analytics subscription (Nnwdaf_AnalyticsSubscription_Subscribe) message, as described in Section 6.1.1.

[0164] In step S207, the NWDAF including the AnLF may send the analysis feedback information received from the analysis consumer and / or the provided ML model accuracy information by calling the ML model monitoring notification (Nnwdaf_MLModelMonitor_Notify) service operation according to the request in step S205. When the NWDAF including the MTLF receives the analysis feedback information or the ML model accuracy information, the NWDAF including the MTLF may trigger steps S208 to S2013 to improve the ML model accuracy.

[0165] S208a to S208f, the NWDAF including the MTLF determines whether to perform ML model accuracy monitoring and ML model retraining / reconfiguration based on requests from one or more NWDAFs including the AnLF or its local policy by collecting new data from various data sources:

[0166] 1) NWDAF with MTLF can collect new data for ML model accuracy monitoring, retraining, and reconfiguration from data sources NF and DCCF by calling the Data Exposure Subscription (Nnf_EventExposure_Subscribe) and Data Management Subscription (Ndccf_DataManagement_Susubscribe) service operations respectively.

[0167] 2) When S202 indicates an ADRFID and / or DataSetTag, the NWDAF with the MTLF can retrieve historical data from the ADRF indicated by the NWDAF with the AnLF in S202 by calling the Nadrf_DataManagementRetrievalRequest or Nadrf_DataManagementRetrieval_Subscribe service operations. Otherwise, the NWDAF with the MTLF can retrieve historical data from the DCCF or NWDAF with the AnLF by calling the Ndccf_DataManagement_Subscribe or Nnwdaf_DataManagement_Subsubscribe service operations, respectively.

[0168] 3) If the NWDAF containing AnLF in S202 does not include a DataSetTag with ADRFID, the NWDAF containing MLTF may use the procedure defined in Section 6.2B.3 to request the ADRF to subscribe to the collection of analytics and data corresponding to the analytics generated by the ML model provided in S203.

[0169] 4) The NWDAF including the MTLF may subscribe to the UDM by calling the SDM subscription (Nudm_SDM_subscribe) service operation of the UDM to obtain notifications of changes in subscription data of the target reported by the ML model, and the UDM may subscribe to the UDR by calling the DM subscription (Nudr_DM_subscribe) service operation of the UDR to obtain notifications of modifications of UE subscription data.

[0170] 5) NWDAF including MTLF can determine the status of data source NF by collecting fault prediction analysis data from MDAS using management data analytics (MDA) requests, thereby taking data quality into account in accuracy monitoring.

[0171] If the NWDAF containing MTLF has collected new test data and performed ML model accuracy monitoring and retraining triggered by other NWDAFs containing AnLF (for ML model accuracy monitoring and retraining), the NWDAF containing MTLF determines whether to use the data for subscription based on its internal logic.

[0172] S209a-S209f, the NWDAF including the MTLF receives the requested data from the various sources requested in S208a-S208f.

[0173] At S2010, based on the analysis and data collected from S209a-S209f, the NWDAF with MTLF uses the predictions and the actual measured data observed at the time of the prediction reference to calculate accuracy. If the data quality of a data source is detected to be poor, the NWDAF with MTLF may discard the data from that source. If only input data and ground truth data are available, the NWDAF with MTLF can generate predictions using the collected input data to calculate accuracy.

[0174] S2011: An accuracy report is sent to the NWDAF including the AnLF, for example, when the reporting threshold is reached by calling the NWDAF's ML Model Provision Notification (Nnwdaf_MLModelProvision_Notify) service operation.

[0175] S2012, based on the calculated accuracy, NWDAF including MTLF can decide to re-train or re-provision the ML model.

[0176] In step S2013, when the newly generated or retrained ML model is ready, the NWDAF including the MTLF sends the new or retrained ML model to the NWDAF including the AnLF by calling the NWDAF ML Model Provision Notification (Nnwdaf_MLModelProvision_Notify) service operation. The NWDAF including the MTLF can send an accuracy report of the new or retrained ML model to the NWDAF including the AnLF.

[0177] It can be understood that the above-mentioned process of MTLF-based ML model accuracy monitoring can refer to the relevant content in the 3rd Generation Partnership Project (3GPP) TS23.288V18.2.0, which will not be repeated here.

[0178] 3. Process for monitoring ML model analysis accuracy:

[0179] Due to the registration for monitoring the analysis accuracy of the ML model received from the NWDAF including AnLF and local policy, the NWDAF including MTLF subscribes to the NWDAF including AnLF to receive notifications of the accuracy of the ML model or analysis feedback information of the ML model. The NWDAF including MTLF can obtain the subscription endpoint address of the NWDAF including AnLF from the information received in the previous registration or through the service discovery process at the NRF.

[0180] Figure 3 illustrates the process of monitoring ML model analysis accuracy or delivering ML model analysis feedback. The NWDAF ML Model Monitor Subscription (Nnwdaf_MLModelMonitor_Subscribe) and NWDAF ML Model Monitor Notification (Nnwdaf-MLModelMonitor_Notify) service operations are used for this purpose. The service consumer (i.e., the NWDAF with the MTLF) subscribes to the service producer (i.e., the NWDAF with the AnLF) to be notified when the analysis accuracy of the previously provided ML model is insufficient or when analysis feedback information is retrieved from the analysis consumer NF.

[0181] S300: After receiving the ML model monitoring registration (Nnwdaf_MLModelMonitor_Register) request from NWDAF, NWDAF including MTLF decides to subscribe to the analysis accuracy monitoring of the ML model based on local policies.

[0182] S301, the NWDAF including the MTLF sends an NWDAF ML model monitoring subscription (Nnwdaf_MLModelMonitor_Subscribe) request (analysis ID, unique identifier of the ML model to be monitored, accuracy metric to be monitored, optional reporting threshold or reporting period) to the NWDAF including the AnLF subscription endpoint.

[0183] S302: The NWDAF including the AnLF sends a response to the NWDAF including the MTLF.

[0184] S303 , the analysis consumer NF may send analysis feedback information to the NWDAF including the AnLF.

[0185] At step S304, when step S301 is triggered, the NWDAF including AnLF may start monitoring the analysis accuracy of the ML model (if it has not already started). The NWDAF including AnLF may monitor the analysis accuracy in a variety of ways, such as comparing the ML model's predictions with their corresponding real data, comparing changes in the internal configuration of the analysis ID generation, and comparing pre-existing records of analysis accuracy information.

[0186] S305, the NWDAF including the AnLF determines whether the analysis accuracy of the ML model is insufficient, that is, the deviation of the output analysis of the trained ML model from the ground truth data (collected from the data producer NF, corresponding to the analysis ID requested when predicting the reference) is greater than the reporting threshold (configured locally or received in the subscription request), or the reporting period indicated in the subscription request is reached.

[0187] In step S306, if the analysis feedback information is retrieved in step S303 or if the NWDAF including AnLF detects that the analysis accuracy of the ML model is insufficient in step S305, the NWDA including AnLF sends a NWDAF ML model monitoring notification (Nnwdaf_MLModelMonitor_Notify) request to the notification endpoint (e.g., the NWDAF including MTLF). The notification request includes the analysis feedback information or the analysis accuracy information of the ML model (e.g., a deviation value indicating the deviation between the prediction generated using the ML model and the ground truth data, and the network data when the deviation occurs (which can be used by the NWDAF including MTLF for possible ML model retraining), as well as the number of inferences performed during the time interval between the NWDAF ML model monitoring registration (Nnwdaf_MLModelMonitor_Register) request and the notification request, or between the time of the previous notification message and the time of the current notification message, indicating that the analysis accuracy of the ML model does not meet the accuracy requirement of the ML model.

[0188] S307: The NWDAF including the MTLF sends a response.

[0189] At step S308, the NWDAF including the MTLF determines whether the ML model is degraded based on the notification in step S306. If the notification includes analysis feedback information, the NWDAF including the MTLF may determine that the ML model is degraded according to the procedure described in section 6.2E.2. Otherwise, when the NWDAF including the MTLF has received multiple pieces of analysis accuracy information from one or more NWDAFs including the AnLF, it may consider the ML model degraded / needs to be updated, i.e., it has received a sufficient amount of analysis accuracy information from one or more NWDAFs including the AnLF, indicating that the analysis accuracy is insufficient.

[0190] It is understood that the actual mechanism used by the NWDAF with MTLF to determine ML model degradation is an internal process of the NWDAF with MTLF. For example, the NWDAF with MTLF calculates global accuracy based on the analysis accuracy information and the number of inferences received from multiple NWDAFs with AnLF.

[0191] In S309, when the ML model is deemed downgraded / updated in S308, the NWDAF including the MTLF retrains the existing ML model or selects a new ML model. If the network data is not included in the NWDAF's ML Model Monitoring Notification (Nnwdaf_MLModelMonitor_Notify) request in S305, the NWDAF including the MTLF may request data from the NWDAF including the AnLF, ADRF, and / or other 5GS entities specified in Section 6.2, and use the collected data for ML model retraining. As described in Section 6.2A, the NWDAF including the MTLF notifies the NWDAF including the AnLF of the updated trained ML model information by calling the NWDAF's ML Model Provision Notification (Nnwdaf_MLModelProvision_Notify) service operation.

[0192] It can be understood that the above-mentioned process of MTLF-based ML model accuracy monitoring can refer to the relevant content in 3GPP TS23.288V18.2.0, which will not be repeated here.

[0193] It can also be understood that in the embodiments of the present application, "monitoring" and "monitoring" may refer to the same action and the two may be expressed interchangeably.

[0194] As shown in Figures 2 and 3 above, the MTLF-based ML model accuracy monitoring and ML model analysis accuracy monitoring processes both monitor or analyze the accuracy of a single ML model. In other words, when there is only one ML model, the corresponding dataset can be used directly to evaluate the ML model.

[0195] Research has found that when multiple ML models exist, their relevant data is stored in a single dataset. Consequently, when evaluating any one of these models, all the data in that dataset is used. This means that it's impossible to retrieve relevant data for any of the multiple models from the dataset, reducing the reliability of the evaluation of that ML model. Therefore, improving the reliability of ML model evaluation in large-scale scenarios is a hot topic of discussion.

[0196] In response to the above technical problems, the embodiments of the present application propose the following technical solutions to improve the reliability of ML model evaluation.

[0197] The technical solution in this application will be described below with reference to the accompanying drawings.

[0198] The technical solutions of the embodiments of the present application can be applied to various communication systems, such as wireless network (Wi-Fi) systems, vehicle to everything (V2X) communication systems, device-to-device (D2D) communication systems, Internet of Vehicles communication systems, fourth-generation (4G) mobile communication systems, such as long-term evolution (LTE) systems, world-wide interoperability for microwave access (WiMAX) communication systems, fifth-generation (5G) systems, such as new radio (NR) systems, and future communication systems.

[0199] This application will present various aspects, embodiments, or features in the context of systems that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Furthermore, combinations of these aspects may also be used.

[0200] Additionally, in the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0201] In the embodiments of the present application, "information", "signal", "message", "channel" and "signaling" can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings they intend to express are matched. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings they intend to express are matched. In addition, the " / " mentioned in this application can be used to express an "or" relationship.

[0202] The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field will know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0203] To facilitate understanding of the embodiments of the present application, a communication system applicable to the embodiments of the present application is first described in detail using the communication system shown in Figure 4 as an example. For example, Figure 4 is a schematic diagram of the architecture of a communication system applicable to the communication method provided in the embodiments of the present application.

[0204] As shown in Figure 4, the communication system can be applied to the above-mentioned 5GS, including: AnLF functional entity and MTLF functional entity. The AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0205] It can be understood that the AnLF functional entity and the MTLF functional entity can be set on the same NWDAF network element, that is, one NWDAF network element includes the AnLF functional entity and the MTLF functional entity; or they can be set separately on different NWDAF network elements, that is, different NWDAF network elements include the AnLF functional entity and the MTLF functional entity, and the embodiments of the present application do not limit this. In addition, the "AnLF functional entity" and "MTLF functional entity" in the embodiments of the present application are only exemplary expressions, and the "AnLF functional entity" can also be replaced with any possible expression, such as "NWDAF including AnLF", etc., and the "MTLF functional entity" can also be replaced with any possible expression, such as "NWDAF including MTLF", etc., and the embodiments of the present application do not limit this.

[0206] Optionally, the communication system may further include an ADRF functional entity. It is understood that in the embodiment of the present application, the ADRF functional entity and the ADRF network element are different expressions of the same thing, that is, the ADRF functional entity may also be referred to as an ADRF network element.

[0207] In a communication system, relevant information of different ML models can be stored in different data sets, that is, relevant data of each ML model can be distinguished by the data set. For example, after receiving the information of the first ML model sent by the MTLF functional entity, the AnLF functional entity can obtain the first ML model based on the information, and set the relevant information of the first ML model to be saved in the first data set, that is, set the correspondence between the first ML model and the first data set. Then, by carrying the information of the first ML model and the information of the first data set in the first message, the correspondence can be sent to the MTLF functional entity, so that the MTLF functional entity can subsequently obtain the relevant data of the first ML model from the first data set, and evaluate the first ML model based on the relevant data, such as robustness evaluation. Since there is no relevant data of other ML models in the first data set, that is, there is only relevant data of the first ML model, the reliability of the evaluation can be improved in this way.

[0208] The following will describe the interaction process between network elements / devices in the above communication system in detail through a method embodiment in conjunction with Figure 5. The communication method provided in the embodiment of the present application can be applied to the above communication system and will be described in detail below.

[0209] Figure 5 is a flow chart of a communication method according to an embodiment of the present application. The communication method is applicable to the above communication system and involves interaction between an AnLF functional entity and a MTLF functional entity.

[0210] Specifically, as shown in FIG5 , the process of the communication method is as follows:

[0211] S501: The MTLF functional entity sends information of a first ML model to the AnLF functional entity. Correspondingly, the AnLF functional entity receives the information of the first ML model from the MTLF functional entity.

[0212] The information of the first ML model can identify the first ML model, that is, the information of the first ML model can be used to indicate the first ML model and can be used to obtain the first ML model. The information of the first ML model can be identification information (model identifier) ​​of the first ML model, such as the ID (unique ML Model identifier) ​​of the first ML model, the uniform resource locator (URL) address of the first ML model, or the fully qualified domain name (FQDN) of the first ML model, or other information that can identify the first ML model, without limitation.

[0213] The first ML model is an ML model trained by the MTLF functional entity. It can be a new ML model generated by the MTLF functional entity based on a data set, or it can be an ML model retrained by the MTLF functional entity based on an already generated ML model (such as a second ML model, which will be described below), or it can be a regenerated ML model. The first ML model can be one ML model, or multiple ML models. That is, the MTLF functional entity can send information about one or more ML models to the AnLF functional entity so that the AnLF functional entity obtains one or more ML models. Specifically, when the first ML model is one ML model, the information about the first ML model can be information about one ML model, such as an ID of an ML model; conversely, when the first ML model is multiple ML models, the information about the first ML model can be information about multiple ML models, and the multiple ML models correspond one-to-one to the information about the multiple ML models. For example, when the first ML model is ML model #1, ML model #2, and model #3, the information about the first ML model is information about ML model #1, information about ML model #2, and information about model #3.

[0214] After receiving the information about the first ML model, the AnLF functional entity can retrieve the ML model corresponding to the information from the location where the ML model is stored, based on the information about the first ML model, thereby obtaining the first ML model. For example, the AnLF functional entity sends an ML model management retrieval (Nadrf_MLModelManagement_Retrieval) service request carrying the ID of the first ML model to the ADRF functional entity. The ADRF functional entity can store the ID and model file of the first ML model at a URL or FQDN in the ADRF functional entity. The AnLF functional entity can obtain the first ML model based on the URL or FQDN, such as by using HTTP protocol technology.

[0215] The trigger condition for the MTLF functional entity to send information about the first ML model to the AnLF functional entity may be that the AnLF functional entity requests the ML model from the MTLF functional entity, or that the MTLF functional entity generates a new ML model after sending the ML model to the AnLF functional entity, such as the MTLF functional entity retrains a new ML model based on the sent ML model, or regenerates a new ML model.

[0216] Specifically, before the MTLF functional entity sends the first ML model information to the AnLF functional entity (S501), the AnLF functional entity may send a request message for the ML model to the MTLF functional entity. Alternatively, if the MTLF functional entity has already sent the ML model information to the AnLF functional entity, the updated ML model information, i.e., the first ML model information, is sent in S501. The following describes different situations.

[0217] Case 1:

[0218] Before the MTLF functional entity sends the first ML model information to the AnLF functional entity (S501), the communication method may further include: the AnLF functional entity sends a request message to the MTLF functional entity. Accordingly, the MTLF functional entity receives the request message from the AnLF functional entity (S500a in FIG5 ). The AnLF functional entity receives the first ML model information from the MTLF functional entity (S501), including: in response to the request message, the AnLF functional entity receives the first ML model information from the MTLF functional entity.

[0219] The request message is used to request an ML model. The request message may not carry the information of the dataset. Alternatively, the request message may carry the information of the dataset, and the number of datasets corresponding to the information of the dataset is smaller than the number of ML models corresponding to the information of the ML model sent by the MTLF functional entity after receiving the request message. For example: the request message carries the information of dataset #1, and the MTLF functional entity sends the information of 2 ML models after receiving the request message. At this time, the number of datasets is smaller than the number of ML models. For another example: the request message carries the information of dataset #11 and the information of dataset #22, that is, the request message carries the information of 2 datasets, and the MTLF functional entity sends the information of 3 ML models after receiving the request message. At this time, the number of datasets is smaller than the number of ML models.

[0220] Sending the request message may be implemented through an ML model provision subscription (MLModelProvision_Subscribe) service operation, or other operations, which is not limited in the embodiment of the present application.

[0221] In an embodiment of the present application, the AnLF functional entity can send a request message for requesting an ML model to the MTLF functional entity. After receiving the request message, the MTLF functional entity sends information of the first ML model to the AnLF functional entity, that is, provides an ML model for the AnLF functional entity so that the AnLF functional entity obtains the first ML model based on the information of the first ML model.

[0222] It can be understood that the AnLF functional entity and the MTLF functional entity can pre-set or pre-define a data set by agreement, so that after the AnLF functional entity sends a request message to the MTLF functional entity, the MTLF functional entity can generate a first ML model based on the data set and send it to the AnLF functional entity.

[0223] Case 2:

[0224] Before the MTLF functional entity sends the first ML model information to the AnLF functional entity (S501), the communication method may further include: the AnLF functional entity sending a request message to the MTLF functional entity. Accordingly, the MTLF functional entity receives the request message from the AnLF functional entity (S500b1 in FIG5 ). Upon receiving the request message, the MTLF functional entity sends the second ML model information to the AnLF functional entity. Accordingly, in response to the request message, the AnLF functional entity receives the second ML model information from the MTLF functional entity (S500b2 in FIG5 ).

[0225] The request message is used to request the ML model and carries information about the second data set. For details, please refer to the aforementioned related introduction and will not be repeated here.

[0226] The information of the second data set can be used to identify the second data set, and the location of the second data set, that is, the location where the second data set is stored. The information of the second data set may include identification information (DateSetTag) of the second data set and / or identification information of the second ADRF functional entity. The identification information of the second data set can be the ID of the second data set, or other information that can identify the second data set, without limitation. The identification information of the second ADRF functional entity can be the ID of the second ADRF functional entity, or other information that can identify the second ADRF functional entity, without limitation.

[0227] It can be understood that when there is one ADRF functional entity in the communication system, each data set is stored in the ADRF functional entity. In this case, the information of the second data set includes the identification information of the second data set, that is, there is no need to indicate the correspondence between the second data set and the ADRF functional entity, that is, the correspondence between each data set and the ADRF functional entity can be preset or predefined by protocol. When there are multiple ADRF functional entities in the communication system, it is necessary to indicate the correspondence between the second data set and the second ADRF functional entity that stores the second data set. In this case, the information of the second data set includes the identification information of the second data set and the identification information of the second ADRF functional entity.

[0228] The second data set is used to store relevant data of the ML model (second ML model) sent by the MTLF functional entity after receiving the request message. The second data set can be established by the AnLF functional entity in the ADRF functional entity (such as the second ADRF functional entity). It is understood that the relevant data can be sent by the AnLF functional entity to the ADRF functional entity and stored in the second data set by the ADRF functional entity.

[0229] The relevant data of the second ML model may include at least one of the following: the input data of the second ML model, the predicted data, the ground truth data at the time of prediction, and the deviation data at the time of prediction. The predicted data is the output of the second ML model. The ground truth data at the time of prediction is the actual result for the input data of the second ML model, not the predicted result. In other words, if the predicted result is consistent with the actual result, it can be considered that the prediction result is correct. The deviation data at the time of prediction is data where the prediction result deviates, such as when the model's predicted result for a certain input data is inconsistent with the ground truth result corresponding to the input data. The consistency standard here can be complete consistency, such as numerical equality, or it can be a predefined standard, that is, the predicted result meets specific requirements, such as if the difference between the predicted result and the ground truth result is within a certain range, which is considered consistent. It is understood that the input data, predicted data, and ground truth data at the time of prediction can also be referred to as inference data. That is, when the relevant data of the second ML model includes the input data, predicted data, and ground truth data of the second ML model, the relevant data of the second ML model can also be referred to as the inference data of the second ML model.

[0230] The second ML model is an ML model trained by the MTLF functional entity, and the second ML model is an ML model sent before the first ML model. It is understood that the first ML model can be an ML model retrained based on the second ML model, or the first ML model can be a newly generated ML model (S500b3 in Figure 5).

[0231] The information of the second ML model can identify the second ML model, that is, the information of the second ML model can be used to indicate the second ML model and can be used to obtain the second ML model. The information of the second ML model can be identification information of the second ML model, such as the ID of the second ML model, the URL address of the second ML model, or the Full Qualified Domain Name (FQDN) of the second ML model, or other information that can represent the second ML model, without limitation.

[0232] In this embodiment of the present application, the information about the second data set carried in the request message can be used to indicate that the second data set corresponds to the ML model sent by the MTLF functional entity after receiving the request message. In other words, the information about the second data set can be used to indicate that the relevant data of the ML model sent by the MTLF functional entity after receiving the request message will be stored in the second data set. It is understood that in scenario 2, the number of data sets corresponding to the information about the second data set is greater than or equal to the number of ML models corresponding to the information about the ML model sent by the MTLF functional entity after receiving the request message, so that each ML model provided by the MTLF functional entity has a data set corresponding to that ML model. It is also understood that the correspondence between the information about the data set carried in the request message and the information about the ML model subsequently sent by the MTLF functional entity can be preset or predefined by protocol, such as associating the information about the nth data set in the request message with the information about the nth ML model subsequently sent by the MTLF functional entity, where n is a positive integer. In this way, the AnLF functional entity and the MTLF functional entity determine the correspondence between each data set and each ML model based on the information about the second data set and the information about the ML model subsequently sent by the MTLF functional entity. In addition, the correspondence between the information of the data set carried in the request message and the information of the ML model subsequently sent by the MTLF functional entity can also be set by other methods. For example, when the MTLF functional entity sends the information of the ML model to AnLF, it indicates the correspondence between the information of each model and the data set carried in the request message, that is, the MTLF functional entity determines which data set each ML model corresponds to, without any restriction.

[0233] It can be seen that when the AnLF functional entity first requests an ML model from the MTLF functional entity, it can carry the dataset information in the request message, that is, it informs the MTLF functional entity that the relevant data of the requested ML model will be stored in the dataset, so that the MTLF functional entity can subsequently obtain the relevant data of the ML model from the dataset and use the relevant data. In this way, the AnLF functional entity does not need to send the corresponding relationship between the ML model and the dataset to the MTLF functional entity after receiving the ML model, thereby reducing communication overhead.

[0234] Optionally, after the MTLF functional entity sends information about the second ML model to the AnLF functional entity, the communication method may further include: the MTLF functional entity evaluating the second ML model based on the second dataset (S503 in FIG. 5 ). In other words, the MTLF functional entity may use the second dataset to evaluate the second ML model. Because the second dataset stores data related to the second ML model and does not store data related to other ML models, using the data in the second dataset to evaluate the second ML model, such as for accuracy and robustness, can improve the reliability of the evaluation of the second ML model.

[0235] Based on the above cases 1 and 2, it can be seen that when the AnLF functional entity requests the MTLF functional entity for an ML model, it can choose to carry the dataset information in the request message or not. That is, when the request message does not carry the dataset information, the AnLF functional entity can send the correspondence between the ML model and the dataset to the MTLF functional entity after receiving the ML model; when the request message carries the dataset information, and the number of datasets corresponding to the dataset information is less than the number of ML models corresponding to the ML model information sent by the MTLF functional entity after receiving the request message, the AnLF functional entity can send the correspondence between the ML model and the dataset to the MTLF functional entity after receiving the ML model; when the request message carries the dataset information, and the number of datasets corresponding to the dataset information is greater than or equal to the number of ML models corresponding to the ML model information sent by the MTLF functional entity after receiving the request message, the AnLF functional entity does not need to send the correspondence between the ML model and the dataset to the MTLF functional entity after receiving the ML model.

[0236] S502: The AnLF functional entity sends a first message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the first message from the AnLF functional entity.

[0237] The first message carries information about the first ML model and the first dataset, and can be used to indicate the corresponding relationship between the first ML model and the first dataset. The first message can be preconfigured or predefined by the protocol, or can be dynamically generated based on actual conditions, which is not limited in this embodiment of the present application. The first message can be sent through the ML Model Provision_Subscribe_Update service operation or other operations, which is not limited in this embodiment of the present application.

[0238] The information of the first data set can be used to identify the first data set and the location of the first data set. The information of the first data set may include identification information (DateSetTag) of the first data set and / or identification information of the first ADRF functional entity. The identification information of the first data set can be the ID of the first data set, or other information that can identify the first data set, without limitation. The identification information of the first ADRF functional entity can be the ID of the first ADRF functional entity, or other information that can identify the first ADRF functional entity, without limitation.

[0239] It can be understood that when there is an ADRF functional entity in the communication system, each data set is stored in the ADRF functional entity. In this case, the information of the first data set includes the identification information of the first data set, that is, there is no need to indicate the correspondence between the first data set and the ADRF functional entity, that is, the correspondence between each data set and the ADRF functional entity can be preset or predefined by protocol. When there are multiple ADRF functional entities in the communication system, it is necessary to indicate the correspondence between the first data set and the first ADRF functional entity that stores the first data set. In this case, the information of the first data set includes the identification information of the first data set and the identification information of the first ADRF functional entity.

[0240] The first dataset is used to store data related to the first ML model and can be established by the AnLF functional entity in an ADRF functional entity (e.g., the first ADRF functional entity). It is understood that the data related to the first ML model can be sent from the AnLF functional entity to the ADRF functional entity and stored in the first dataset by the ADRF functional entity. The first dataset corresponds to the first ML model and can be one or more datasets. Generally, one ML model corresponds to one dataset. That is, when the first ML model is one ML model, the first dataset is one dataset, and the ML model corresponds to the dataset; when the first ML model is multiple ML models, the first dataset is multiple datasets, and the multiple ML models correspond one-to-one to the multiple datasets. For example, if the first dataset is dataset #11 and the first ML model is ML model #11, then dataset #11 corresponds to ML model #11, that is, the relevant data of ML model #11 is stored in dataset #11; for another example, if the first dataset is dataset #a, dataset #b, and dataset #c, and the first ML model is ML model #A, ML model #B, and ML model #C, then dataset #a corresponds to ML model #A, dataset #b corresponds to ML model #B, and dataset #c corresponds to ML model #C, that is, the relevant data of ML model #A is stored in dataset #a, the relevant data of ML model #B is stored in dataset #b, and the relevant data of ML model #C is stored in dataset #c. It is understandable that an ML model can also correspond to multiple datasets, that is, the relevant data of an ML model can be stored in multiple datasets respectively. In this case, there may be one ML model corresponding to multiple datasets, or there may be multiple ML models corresponding to multiple datasets. The corresponding settings can be made according to the actual situation without any restrictions.

[0241] It can be seen that the first data set corresponds to the first ML model, the information of the first data set corresponds to the first data set, and the information of the first ML model corresponds to the first ML model. Therefore, the information of the first data set corresponds to the information of the first ML model. When the information of the first data set is information of multiple data sets, and the information of the first ML model is information of multiple ML models, the information of the multiple data sets corresponds to the information of the multiple ML models. In addition, the information of the multiple data sets and the information of the multiple ML models can correspond one-to-one, or the information of at least two data sets can correspond to the information of one ML model. This is not limited in the embodiments of the present application.

[0242] For example, the information about the first dataset includes information about multiple datasets, including the IDs of dataset #1, dataset #2, and dataset #3. The information about the first ML model includes information about multiple ML models, including the IDs of ML model #1, ML model #2, and ML model #3. The ID of dataset #1 corresponds to the ID of ML model #1, the ID of dataset #2 corresponds to the ID of ML model #2, and the ID of dataset #3 corresponds to the ID of ML model #3. That is, the information about the multiple datasets corresponds one-to-one to the information about the multiple ML models.

[0243] For another example, the information of the first dataset includes information on multiple datasets, including the IDs of dataset #11, dataset #22, and dataset #33. The information of the first ML model includes information on multiple ML models, including the IDs of ML model #11 and ML model #22. The ID of dataset #11 corresponds to the ID of ML model #11, and the IDs of dataset #22 and dataset #33 correspond to the ID of ML model #22. This means that data related to ML model #22 can be stored in both dataset #22 and dataset #33. In other words, information on at least two datasets corresponds to information on one ML model.

[0244] The relevant data of the first ML model may include at least one of the following: the input data of the first ML model, the predicted data, the ground truth data during the prediction, and the deviation data during the prediction. For details about each of these data, please refer to the description of "S501" above and will not be repeated here. It is understood that when the relevant data of the first ML model includes the input data of the first ML model, the predicted data, and the ground truth data during the prediction, the relevant data of the first ML model may also be referred to as the inference data of the first ML model.

[0245] In an embodiment of the present application, when the first ML model is a single ML model, by carrying information about the first ML model and information about the first dataset in the first message, such as placing the information about the first ML model and the information about the first dataset in a tuple, it can be indicated that the information about the first ML model corresponds to the information about the first dataset, i.e., the relevant data of the first ML model is stored in the first dataset. When the first ML model is multiple ML models, the information about each ML model and the information about the dataset corresponding to the ML model can be placed in the same tuple in the first message, i.e., the correspondence between different ML models and datasets can be distinguished by different tuples. For example, [Model ID1, DataSetTag1] can indicate that the first ML model corresponds to the first dataset, i.e., the AnLF functional entity can store the relevant data of the first ML model in the first dataset; [Model ID2, DataSetTag2, DataSetTag3] can indicate that the second ML model corresponds to the second and third datasets, i.e., the AnLF functional entity can store the relevant data of the second ML model in the second and third datasets. In other words, by carrying information about the first dataset and the first ML model in the first message, the correspondence between the first dataset and the first ML model can be indicated. It can be understood that the above-mentioned method of placing the information of the first data set and the information of the first ML model in the form of at least one tuple is only an example, and there may be other ways to implement it, which is not limited in the embodiments of the present application.

[0246] After receiving the first message, the MTLF functional entity can determine the first dataset corresponding to the first ML model based on the information carried in the first message. That is, it knows that the AnLF functional entity has stored the relevant data of the first ML model in the first dataset. In this way, the MTLF functional entity can use the data in the first dataset when evaluating the first ML model or retraining the model.

[0247] In summary, the MTLF functional entity sends information about the first ML model to the AnLF functional entity so that the AnLF functional entity obtains the first ML model based on the information about the first ML model, that is, the MTLF functional entity provides the first ML model to the AnLF functional entity. After receiving the information about the first ML model, the AnLF functional entity can set the correspondence between the first ML model and the first data set based on the information about the first ML model, that is, save the relevant data of the first ML model in the first data set. In addition, the correspondence is sent to the MTLF functional entity via a first message carrying information about the first ML model and information about the first data set. That is, when the AnLF functional entity receives the ML model, the AnLF functional entity can send a set of tuples to the MTLF functional entity, which include identification information of the ML model, identification information of the data set (DataSetTag) and / or identification information of the ADRF (ADRFID), and use the ML model provision subscription (Nnwdaf_MLModelProvision_Subscribe) service operation to indicate the mapping relationship between the identification information of the data set and the identification information of the ML model. It can be understood that other service operations can also be used for implementation without limitation. Accordingly, after receiving the tuple, the MTLF functional entity can determine that the data corresponding to the dataset identification information in the tuple is relevant data for the model corresponding to the ML model identification information. In other words, the MTLF functional entity can obtain relevant data for the ML model from the dataset corresponding to the dataset identification information. In this way, the MTLF functional entity can evaluate the first ML model using the data from the first dataset, rather than using a dataset containing data from multiple ML models, thereby improving the reliability of the evaluation of the first ML model.

[0248] It is understood that in the embodiment of the present application, the first dataset is used to store the data of the first ML model. In other words, the first dataset may store the data of the first ML model. Therefore, the information of the first ML model (ML Model Identifier) ​​may also indicate the model corresponding to the data corresponding to the identifier (DataSetTag) of the first dataset. In this example, the corresponding model is the first ML model.

[0249] It can also be understood that in an embodiment of the present application, after the AnLF functional entity sends a request message to the MTLF functional entity (such as S500a in Figure 5), the MTLF functional entity can be triggered to send information about the first ML model to the AnLF functional entity (such as S501 in Figure 5), that is, to provide the first ML model to the AnLF functional entity; after the AnLF functional entity receives the information about the first ML model from the MTLF functional entity, the AnLF functional entity can be triggered to send information about the first ML model and information about the first data set to the MTLF functional entity (such as S502 in Figure 5), that is, to provide the MTLF functional entity with a correspondence between the first ML model and the first data set. In other words, these three steps (such as S500a, S501, and S502 in Figure 5) can be strongly associated. In which, the request message is a message that does not carry information about the data set, or the request message is a message that carries information about the data set, but the number of data sets corresponding to the information about the data set is less than the number of ML models corresponding to the information about the ML model sent by the MTLF functional entity after receiving the request message.

[0250] Optionally, after the MTLF functional entity receives the first message from the AnLF functional entity, the communication method may further include: the MTLF functional entity evaluating the first ML model based on the first dataset (S503 in FIG. 5 ). In other words, the MTLF functional entity may use the first dataset to evaluate the first ML model. Because the first dataset is used to store data related to the first ML model and does not store data related to other ML models, using the data in the first dataset to evaluate the first ML model, such as accuracy and robustness, can improve the reliability of the evaluation of the first ML model.

[0251] Optionally, after the AnLF functional entity obtains the first ML model, the MTLF functional entity may request the AnLF functional entity to monitor the analysis accuracy of the first ML model (S504 to S5013 in Figure 5). In the process of the AnLF functional entity monitoring the ML model analysis accuracy of the first ML model, the MTLF functional entity or the AnLF functional entity may set a third data set for storing relevant data of the first ML model (such as data that deviates during prediction), and send information about the third data set to the AnLF functional entity or the MTLF functional entity so that the MTLF functional entity can use the data of the third data set when updating the ML model. The following describes the methods of setting and sending the third data set respectively.

[0252] Mode 1: The MTLF functional entity sets and sends the third data set.

[0253] In this case, the communication method may further include: when the MTLF functional entity receives the registration request for ML model monitoring, the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity. Correspondingly, the AnLF functional entity receives the request for monitoring the ML model from the MTLF functional entity (S505 in FIG. 5 ).

[0254] The request for monitoring the ML model carries information about the first ML model and information about the third dataset, indicating a corresponding relationship between the first ML model and the third dataset. Sending the request for monitoring the ML model can be implemented through an ML model monitoring subscription request (Nnwdaf_MLModelMonitor_Subscribe request) service operation or other operations, and is not limited in this embodiment of the application.

[0255] The information of the third data set can be used to identify the third data set and the storage location of the third data set. The information of the third data set may include the identification information of the third data set and / or the identification information of the third ADRF functional entity. The identification information of the third data set can be the ID of the third data set, or other information that can identify the third data set, without limitation. The identification information of the third ADRF functional entity can be the ID of the third ADRF functional entity, or other information that can identify the third ADRF functional entity, without limitation.

[0256] It can be understood that when there is one ADRF functional entity in the communication system, each data set is stored in the ADRF functional entity. In this case, the information of the third data set includes the identification information of the third data set, that is, there is no need to indicate the correspondence between the third data set and the ADRF functional entity, that is, the correspondence between each data set and the ADRF functional entity can be pre-set or pre-defined by protocol. When there are multiple ADRF functional entities in the communication system, it is necessary to indicate the correspondence between the third data set and the third ADRF functional entity that stores the third data set. In this case, the information of the third data set includes the identification information of the third data set and the identification information of the third ADRF functional entity.

[0257] The third data set is used to store the relevant data of the first ML model, and the third data set can be established by the MTLF functional entity in the ADRF functional entity (such as the third ADRF functional entity). It can be understood that the AnLF functional entity can store the relevant data of the first ML model in the third data set.

[0258] The relevant data of the first ML model includes deviation data during prediction, which refers to data where the prediction result deviates. For example, each group of data of the first ML model includes 100 data, and there are 3 groups of data, namely data group #1, data group #2, and data group #3. There are 20 deviated data in data group #1, 10 deviated data in data group #2, and 15 deviated data in data group #3. At this time, the deviation data during prediction of the first ML model is 20 deviated data in data group #1, 10 deviated data in data group #2, and 15 deviated data in data group #3, for a total of 45 deviated data.

[0259] It can be understood that in the prior art, when the analysis accuracy of the ML model is insufficient, the network data at the time of deviation sent by the AnLF functional entity to the MTLF functional entity includes data that has not deviated, and not all deviated data. For example: each set of data of the first ML model includes 100 data, and the accuracy of the first set of data is 90%, which is greater than the accuracy threshold (80%), then the first set of data is determined not to be network data at the time of deviation; the accuracy of the second set of data is 70%, which is less than the accuracy threshold (80%), then the second set of data is determined to be network data at the time of deviation, that is, the second set of data is sent to the MTLF functional entity. It can be seen that the network data at the time of deviation in the prior art is inaccurate, that is, it does not include all deviated data, and includes normal data. Therefore, saving the deviation data at the time of prediction in the third data set can guide the MTLF functional entity to subsequently update the ML model and improve the prediction accuracy of the updated ML model.

[0260] It can be seen that in method 1, the MTLF functional entity can set a third data set in the third ADRF functional entity, and when requesting the AnLF functional entity to monitor the analysis accuracy of the ML model, the information of the first ML model and the information of the third data set are sent to the AnLF functional entity, so that the AnLF functional entity saves the relevant data of the first ML model in the third data set for the MTLF functional entity to use when updating the ML model.

[0261] Method 2: The AnLF functional entity sets and sends the third data set according to the request of the monitoring ML model.

[0262] In this case, the communication method may further include: the MTLF functional entity sending a request for monitoring the ML model to the AnLF functional entity. Accordingly, the AnLF functional entity receives the request for monitoring the ML model from the MTLF functional entity. In response to the request for monitoring the ML model, the AnLF functional entity sends information about the third data set to the MTLF functional entity. Accordingly, the MTLF functional entity receives information about the third data set from the AnLF functional entity (S506 in FIG. 5 ). In other words, the information about the third data set is included in the response for monitoring the ML model.

[0263] The request for monitoring the ML model carries information about the first ML model, which is used to indicate the accuracy of the analysis of the first ML model. The request for monitoring the ML model and the third dataset can be found in the aforementioned "Method 1" and will not be repeated here.

[0264] It can be seen that in method 2, after receiving the request for monitoring the ML model from the MTLF functional entity, the AnLF functional entity can set a third data set in the ADRF functional entity (such as the third ADRF functional entity) and send the correspondence between the first ML model and the third data set to the MTLF functional entity, so that the MTLF functional entity can use the data in the third data set when updating the ML model.

[0265] Mode 3: The AnLF functional entity sets the third data set and sends information of the third data set to the MTLF functional entity through a notification request message.

[0266] In this case, the communication method may further include: the AnLF functional entity monitoring the analysis accuracy of the first ML model; and when the analysis accuracy of the first ML model is insufficient, the AnLF functional entity sending a notification request message to the MTLF functional entity. In response, the MTLF functional entity receives the notification request message from the AnLF functional entity (S509 in FIG. 5 ).

[0267] The analysis accuracy of the first ML model can represent the accuracy of the prediction result using the first ML model. The insufficient analysis accuracy of the first ML model can be determined by the accuracy of the prediction result of the first ML model and a preset threshold, that is, when the accuracy of the prediction result of the first ML model is less than the preset threshold, it can be determined that the analysis accuracy of the first ML model is insufficient. For example, the accuracy of the prediction result of the first ML model is 80%, which is less than the threshold (90%). At this time, the analysis accuracy of the first ML model is insufficient. It can be understood that the way in which the AnLF functional entity monitors the analysis accuracy of the first ML model and the way in which the analysis accuracy of the first ML model is determined to be insufficient can reuse the methods of the prior art and will not be repeated here.

[0268] The notification request message carries information about the third data set to indicate the correspondence between the first ML model and the third data set. Sending the notification request message can be implemented through the ML model monitoring notification request (Nnwdaf_MLModelMonitoring_Notify request) service operation or other operations, which is not limited in this embodiment of the application.

[0269] The third data set can refer to the relevant introduction in the aforementioned "Method 1" and will not be repeated here.

[0270] It can be seen that in method 2, after receiving the request from the MTLF functional entity for monitoring the ML model, the AnLF functional entity may temporarily not send the information of the third data set to the MTLF functional entity. Instead, after monitoring the analysis accuracy of the first ML model, when the analysis accuracy of the first ML model is insufficient and a notification request message is sent to the MTLF functional entity, the information of the third data set is sent to instruct the relevant information of the first ML model to be saved in the third data set.

[0271] It can be understood that in Methods 1 and 2, when the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity, it carries information about the first ML model to indicate the accuracy of monitoring and analysis of the first ML model. Therefore, when the AnLF functional entity sends information about the third data set to the MTLF functional entity, it does not need to carry information about the first ML model. In other words, the correspondence between the third data set and the first ML model can be indicated by sending information about the third data set.

[0272] In addition, the other steps of the AnLF functional entity monitoring the ML model analysis accuracy of the first ML model (S504, S507-S508, S5010-S5013 in Figure 5) can refer to the relevant introduction of the aforementioned "3. Process of monitoring ML model analysis accuracy" and will not be repeated here.

[0273] It is also understood that after the AnLF functional entity obtains the first ML model, the MTLF functional entity may also monitor the analysis accuracy of the first ML model. In other words, in this case, the aforementioned steps (S504-S5013) of monitoring the ML model analysis accuracy of the first ML model by the AnLF functional entity are not required.

[0274] Figure 6 is a second flow chart of a communication method provided in an embodiment of the present application. This communication method is applicable to the above communication system and involves interaction between an AnLF functional entity and a MTLF functional entity.

[0275] Specifically, as shown in FIG6 , the process of the communication method is as follows:

[0276] S601: AnLF functional entity sends a request message to MTLF functional entity. Correspondingly, MTLF functional entity receives the request message from AnLF functional entity.

[0277] The request message is used to request the ML model. For details, please refer to the relevant introduction in the aforementioned "S501 Case 1", which will not be repeated here.

[0278] S602: The MTLF functional entity sends a first message to the AnLF functional entity. Correspondingly, the AnLF functional entity receives the first message from the MTLF functional entity.

[0279] The first message carries information about the first ML model and information about the first data set. The information about the first ML model is used to indicate the first ML model. The information about the first ML model and the first model can refer to the relevant introduction at the aforementioned "S501" and will not be repeated here. The first data set is used to store relevant data of the first ML model, and the first data set is established by the MTLF functional entity in the ADRF functional entity (such as the first ADRF functional entity). The information of the first data set is used to identify the first data set. The first data set, the information of the first data set and the related information can refer to the relevant introduction at the aforementioned "S502" and will not be repeated here.

[0280] The information about the first ML model and the first dataset carried in the first message can indicate the corresponding relationship between the first ML model and the first dataset, that is, instruct the AnLF functional entity to save the relevant data of the first ML model to the first dataset. In this way, the MTLF functional entity can evaluate the first ML model based on the data saved in the first dataset (the relevant data of the first ML model), thereby improving the reliability of the evaluation of the first ML model.

[0281] Optionally, before the MTLF functional entity sends the first message (S602) to the AnLF functional entity, the communication method may further include: the MTLF functional entity sends a request to the first ADRF functional entity to establish a first data set in the first ADRF functional entity. Accordingly, the first ADRF functional entity receives the request from the MTLF functional entity to establish the first data set in the first ADRF functional entity.

[0282] The request for establishing the first data set in the first ADRF functional entity may carry information about the first data set to instruct the first ADRF functional entity to establish the first data set based on the information about the first data set. In other words, the first data set may be established by the MTLF functional entity. It is understood that when the first data set is multiple data sets, the MTLF functional entity may establish multiple data sets in the first ADRF functional entity, that is, information about multiple ML models may be carried in the request for establishing the first data set in the first ADRF functional entity.

[0283] Optionally, after the AnLF functional entity receives the first message from the MTLF functional entity (S602), the communication method may further include: the AnLF functional entity sending, based on the information of the first data set, relevant data of the first ML model and information of the first data set to the first ADRF functional entity. Correspondingly, the first ADRF functional entity receives the relevant data of the first ML model and information of the first data set from the AnLF functional entity.

[0284] It can be seen that when the AnLF functional entity saves the relevant data of the first ML model, it can send the relevant data and information of the first data set to the first ADRF functional entity to instruct the first ADRF functional entity to store the relevant data in the first data set corresponding to the information of the first data set. In this case, the first ADRF functional entity can store the relevant data of the first ML model based on the information of the first data set.

[0285] Optionally, after the MTLF functional entity sends the first message (S602) to the AnLF functional entity, the above-mentioned communication method may further include: the MTLF functional entity evaluates the first ML model based on the data in the first data set. That is, the MTLF functional entity can use the data in the first data set to evaluate the first ML model. Specifically, after the AnLF functional entity saves the relevant data of the first ML model in the first ADRF functional entity, the MTLF functional entity can obtain the relevant data of the first ML model from the first ADRF functional entity, for example: the MTLF functional entity sends a request for obtaining the relevant data of the first ML model to the first ADRF functional entity, and the request may carry information of the first data set to instruct the first ADRF functional entity to return the data in the first data set (i.e., the relevant data of the first ML model) to the MTLF functional entity through the information of the first data set. After the MTLF functional entity obtains the relevant data of the first ML model, the relevant data can be used to evaluate the first ML model, such as accuracy evaluation, robustness evaluation, etc.

[0286] Furthermore, after the MTLF functional entity evaluates the first ML model, the evaluation result may be sent to the AnLF functional entity. In addition, the MTLF functional entity may also send information about a new ML model (such as a second ML model) and information about a data set corresponding to the ML model (such as a second data set) to the AnLF functional entity, so that the AnLF functional entity acquires and uses the second ML model and stores relevant data of the second ML model in the second data set.

[0287] It can be understood that when multiple AnLF functional entities use the first ML model, multiple AnLF functional entities can save the relevant data of the first ML model they use in the first data set. In this case, after the first ADRF functional entity receives the relevant data of the first ML model and the information of the first data set sent by multiple AnLF functional entities, the first ADRF functional entity can record the source of each data, that is, which AnLF functional entity each data comes from, in order to distinguish each data.

[0288] For example, AnLF functional entity #1 and AnLF functional entity #2 both use ML model #aa, and the data of ML model #aa is stored in data set #aa. AnLF functional entity #1 sends the relevant data of ML model #aa (recorded as relevant data #1) and the information of data set #aa to the ADRF functional entity, and AnLF functional entity #2 also sends the relevant data of ML model #aa (recorded as relevant data #2) and the information of data set #aa to the ADRF functional entity. In this case, the ADRF functional entity can save the relevant data #1 in the data set #aa and record the relevant data #1 as data from AnLF functional entity #1, such as marking the relevant data #1 with the label of AnLF functional entity #1; correspondingly, the ADRF functional entity can also save the relevant data #2 in the data set #aa and record the relevant data #2 as data from AnLF functional entity #2, such as marking the relevant data #2 with the label of AnLF functional entity #2. In this way, entities from different AnLF functional entities can be distinguished.

[0289] It can also be understood that when multiple AnLF functional entities all use the first ML model, the MTLF can also set a different data set for each of the multiple AnLF functional entities, that is, send information about different data sets to each AnLF functional entity, so that each AnLF functional entity stores the relevant data of the first ML model in a different data set. In this case, after the ADRF functional entity receives the relevant data of the first ML model sent by each AnLF functional entity, it does not need to record the source of each data, that is, record which AnLF functional entity each data comes from.

[0290] In addition, the MTLF functional entity may also request the AnLF functional entity to monitor the analysis accuracy of the first ML model. In the process of the AnLF functional entity monitoring the ML model analysis accuracy of the first ML model, the MTLF functional entity or the AnLF functional entity may set a third data set for storing relevant data of the first ML model (such as data that deviates during prediction), and send the information of the third data set to the AnLF functional entity or the MTLF functional entity, so that the MTLF functional entity can use the data of the third data set when updating the ML model. For details, please refer to the relevant introduction of the aforementioned "S502 Method 1-Method 3", which will not be repeated here.

[0291] In summary, before providing the first ML model to the AnLF functional entity, the MTLF functional entity can set the correspondence between the first ML model and the first data set, that is, set the relevant information of the first ML model to be saved in the first data set. Furthermore, when the MTLF functional entity sends a first message carrying information about the first ML model to the AnLF functional entity, it can also carry information about the first data set in the first message, so that the AnLF functional entity can obtain the first ML model based on the information of the first ML model, and save the relevant information of the first ML model in the first data set based on the correspondence. In this way, the MTLF functional entity can obtain the relevant data of the first ML model from the first data set to evaluate the first ML model based on the relevant data, thereby improving the reliability of the evaluation of the first ML model.

[0292] Scenario 1:

[0293] FIG7 is a flow chart of the communication method provided in an embodiment of the present application. The communication method is applicable to the above-mentioned communication system and mainly involves a first AnLF functional entity, a second AnLF functional entity, a MTLF functional entity, and an ADRF functional entity. In scenario 1, the AnLF functional entity can request an ML model from the MTLF functional entity, and the MTLF functional entity can provide the ML model to the AnLF functional entity and indicate in which data set the relevant data of the ML model is stored.

[0294] As shown in Figure 7, the process of the communication method is as follows:

[0295] S700: The MTLF functional entity sends a request for establishing a data set in the ADRF functional entity to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the request for establishing a data set in the ADRF functional entity from the MTLF functional entity.

[0296] The request for establishing a data set in the ADRF functional entity may carry information of the first data set and the second data set to instruct the ADRF functional entity to establish the first data set and the second data set. For details, please refer to the relevant introduction of the aforementioned "S602" and will not be repeated here.

[0297] S701a: The first AnLF functional entity sends a request message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the request message from the AnLF functional entity.

[0298] S701b: The second AnLF functional entity sends a request message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the request message from the AnLF functional entity.

[0299] S702a: The MTLF functional entity sends a first message to the first AnLF functional entity. Correspondingly, the first AnLF functional entity receives the first message from the MTLF functional entity.

[0300] The first message carries information about the first ML model and information about the first dataset.

[0301] S702b: The MTLF functional entity sends a second message to the second AnLF functional entity. Correspondingly, the second AnLF functional entity receives the second message from the MTLF functional entity.

[0302] The second message carries information about the first ML model and information about the first dataset.

[0303] S703a: The first AnLF functional entity sends the relevant data of the first ML model and the information of the first data set to the ADRF functional entity based on the information of the first data set. Correspondingly, the ADRF functional entity receives the relevant data of the first ML model and the information of the first data set from the first AnLF functional entity.

[0304] S703b: The second AnLF functional entity sends the relevant data of the first ML model and the information of the first data set to the ADRF functional entity based on the information of the second data set. Correspondingly, the ADRF functional entity receives the relevant data of the first ML model and the information of the first data set from the second AnLF functional entity.

[0305] S704: The ADRF functional entity records the source of the relevant data of the first ML model and saves the first data set.

[0306] It can be understood that the source of the relevant data of the first data set is the first AnLF functional entity or the second AnLF functional entity.

[0307] S705: The MTLF functional entity sends a request for obtaining relevant data of the first ML model to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the request for obtaining relevant data of the first ML model from the MTLF functional entity.

[0308] The request for obtaining relevant data of the first ML model may carry information of the first data set, so as to instruct the ADRF functional entity to return the data in the first data set, i.e., the relevant data of the first ML model, to the MTLF functional entity through the information of the first data set.

[0309] S706: The ADRF functional entity sends the data in the first data set to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the data in the first data set from the ADRF functional entity.

[0310] S707 , the MTLF functional entity performs an accuracy evaluation on the first ML model based on the data in the first dataset.

[0311] The MTLF functional entity can reuse the existing method to evaluate the accuracy of the first ML model, which will not be described here.

[0312] It can be understood that after the MTLF functional entity performs accuracy evaluation on the first ML model, it can obtain accuracy information of the first ML model.

[0313] S708: The MTLF functional entity sends the accuracy information of the first ML model and the information of the first ML model to the first AnLF functional entity. Correspondingly, the first AnLF functional entity receives the accuracy information of the first ML model and the information of the first ML model from the MTLF functional entity.

[0314] It can be understood that the accuracy information of the first ML model and the information of the first ML model sent by the MTLF functional entity may indicate that the accuracy information is the accuracy information of the first ML model, and S708 may be implemented through the ML model provision notification (Nnwdaf_MLModelProvision_Notify) service operation.

[0315] S709: The MTLF functional entity sends information about the second ML model and the second data set to the first AnLF functional entity. Correspondingly, the first AnLF functional entity receives information about the second ML model and the third data set from the MTLF functional entity.

[0316] The information of the second ML model is used to obtain the second ML model, that is, the first AnLF functional entity can obtain the second ML model based on the information of the second ML model. The information of the second ML model and the information of the second data set can instruct the first AnLF functional entity to save the relevant data of the second ML model in the second data set.

[0317] The second ML model may be an ML model retrained by the MTLF functional entity based on the first ML model, or may be a regenerated ML model.

[0318] The specific implementation of S700-S709 can refer to the related introduction of S601-S602 mentioned above, which will not be repeated here.

[0319] It is understood that the above-mentioned S701a-S703a and S701b-S703b can be arranged in different orders. That is, the order in which the first AnLF functional entity and the second AnLF functional entity obtain the ML model and save the relevant data of the ML model can be set according to actual circumstances, and the embodiments of the present application are not limited thereto. For example, after the first AnLF functional entity receives the first message, the second AnLF functional entity sends a request message, that is, S701b is performed after S702a. For another example, the first AnLF functional entity and the second AnLF functional entity send request messages simultaneously, that is, S701a and S701b are performed simultaneously.

[0320] 8 is a fourth flow chart of a communication method according to an embodiment of the present application. The communication method is applicable to the above communication system and involves interaction between AnLF functional entities, MTLF functional entities, and ADRF functional entities.

[0321] Specifically, as shown in FIG8 , the process of the communication method is as follows:

[0322] S801: AnLF functional entity sends a request message to MTLF functional entity. Correspondingly, MTLF functional entity receives the request message from AnLF functional entity.

[0323] The request message is used to request the ML model and carries information about the first data set. For details, please refer to the relevant introduction at "S501" above, which will not be repeated here.

[0324] The first dataset is used to store data related to multiple ML models. The multiple ML models include the ML model sent by the MTLF functional entity after receiving the request message, such as the first ML model. For example, the first dataset is used to store data related to three ML models: ML model #1, ML model #2, and ML model #3, with the first ML model being ML model #1. The information in the first dataset represents the first dataset. For information about the first dataset, refer to the aforementioned description in "S501" and will not be repeated here.

[0325] By carrying the information of the first data set in the request message, it can be indicated that the AnLF functional entity stores the relevant data of the ML model subsequently received in the first data set. For the MTLF functional entity, it can obtain the relevant data of the ML model from the first data set.

[0326] Optionally, before the AnLF functional entity sends a request message to the MTLF functional entity, the communication method may further include: the AnLF functional entity establishing a first data set. For example, the AnLF functional entity may send a request to the ADRF functional entity to establish the first data set in the ADRF functional entity. Accordingly, the ADRF functional entity receives the request from the AnLF functional entity to establish the first data set in the ADRF functional entity. After receiving the request, the ADRF functional entity may establish the first data set.

[0327] It can be understood that for the ADRF functional entity, after establishing the first data set, the ADRF functional entity does not know what data the first data set is used to store. After subsequently receiving the data to be stored and the information of the first data set, it can store the data in the first data set according to the information of the first data set.

[0328] S802: The MTLF functional entity sends a first message to the AnLF functional entity. Correspondingly, the AnLF functional entity receives the first message from the MTLF functional entity.

[0329] The first message carries information about the first ML model. The information about the first ML model is used to indicate the first ML model and can be used to obtain the first ML model. For details, please refer to the relevant introduction at "S501" above and will not be repeated here. In other words, the MTLF functional entity can send the first ML model to the AnLF functional entity through the first message.

[0330] By sending the first message carrying the information of the first ML model to the AnLF functional entity, the AnLF functional entity can obtain the first ML model according to the information of the first ML model.

[0331] It is understood that the first data set may also be established by the MTLF functional entity in the ADRF functional entity. In this case, the request message sent by the AnLF functional entity to the MTLF functional entity will not carry information about the first data set, and the MTLF functional entity may carry information about the first data set in the first message sent to the AnLF functional entity.

[0332] S803: The AnLF functional entity sends the relevant data of the first ML model, the information of the first ML model, and the information of the first data set to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the relevant data of the first ML model, the information of the first ML model, and the information of the first data set from the AnLF functional entity.

[0333] For the relevant data of the first ML model, please refer to the relevant introduction at "S502" above, which will not be repeated here.

[0334] The relevant data of the first ML model and the information of the first ML model sent by the AnLF functional entity may indicate that the relevant data is the relevant data of the first ML model; the information of the first data set sent by the AnLF functional entity may indicate that the relevant data is stored in the first data set.

[0335] S804 : The ADRF functional entity stores relevant data of the first ML model according to the information of the first ML model and the information of the first data set.

[0336] After receiving the relevant data of the first ML model, the information of the first ML model and the information of the first data set, the ADRF functional entity can save the relevant data in the first data set and classify the relevant data as data belonging to the first ML model. That is, the ADRF functional entity can use additional ML model information (such as the ID of the ML model) to further identify the relevant data.

[0337] S805: The MTLF functional entity sends a request for obtaining relevant data of the first ML model to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the request for obtaining relevant data of the first ML model from the MTLF functional entity.

[0338] The request for obtaining relevant data of the first ML model carries information of the first ML model and information of the first data set, so that after receiving the request, the ADRF functional entity can determine the location of the data requested by the MTLF functional entity based on the information of the first data set, and filter out relevant data belonging to the first ML model from the first data set based on the information of the first ML model.

[0339] It can be understood that since the first dataset stores data of multiple ML models, when requesting the relevant data of a certain ML model, in addition to carrying the dataset information in the message, it is also necessary to carry the ML model information so that the relevant data of the ML model can be obtained.

[0340] S806: The ADRF functional entity sends the relevant data of the first ML model to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the relevant data of the first ML model from the ADRF functional entity.

[0341] That is, after the ADRF functional entity filters out the relevant data of the first ML model according to the information of the first ML model and the information of the first data set, it can send the relevant data to the MTLF functional entity.

[0342] Optionally, after the MTLF functional entity obtains relevant data of the first ML model, the above communication method may further include: the MTLF functional entity retraining or regenerating the second ML model based on the relevant data.

[0343] The MTLF functional entity can reuse the methods in the prior art to retrain or regenerate the second ML model based on the relevant data of the first ML model, which will not be described in detail here.

[0344] Furthermore, after the MTLF functional entity generates the second ML model, the communication method may further include: the MTLF functional entity sending information about the second ML model to the AnLF functional entity. In response, the AnLF functional entity receives the information about the second ML model from the MTLF functional entity. This allows the AnLF functional entity to obtain the second ML model, i.e., the latest ML model, based on the information about the second ML model. This allows the AnLF functional entity to use the latest ML model, thereby improving the accuracy of predictions made by the AnLF functional entity using the ML model.

[0345] It can be understood that the MTLF functional entity can also request the AnLF functional entity to monitor the analysis accuracy of the first ML model. In the process of the AnLF functional entity monitoring the ML model analysis accuracy of the first ML model, the MTLF functional entity or the AnLF functional entity can set a third data set for storing relevant data of the first ML model (such as data that deviates during prediction), and send the information of the third data set to the AnLF functional entity or the MTLF functional entity, so that the MTLF functional entity can use the data of the third data set when updating the ML model. For details, please refer to the relevant introduction of "S502 Method 1-Method 3" mentioned above, which will not be repeated here.

[0346] In summary, the relevant data of each ML model can be stored in a data set, and by labeling the relevant data with the corresponding ML model label, it is possible to distinguish which ML model each relevant data belongs to. In other words, the MTLF functional entity can obtain the data set label from the received request message, that is, it knows that the data of each ML model is stored in the data set corresponding to the data set label; and sends a request for obtaining the relevant data of the first ML model carrying the information of the first ML model and the information of the first data set to the ADRF functional entity, so that the ADRF functional entity can find the data set that stores the relevant data based on the information of the first data set, and filter out the relevant data of the first ML model from the first data set based on the information of the first ML model, and send the relevant data to the MTLF functional entity. In this way, the MTLF functional entity can evaluate the first ML model based on the relevant data, thereby improving the reliability of the evaluation of the first ML model.

[0347] It can be understood that the difference between the two aforementioned embodiments (the embodiment shown in Figure 5 and the embodiment shown in Figure 6) and the embodiment of the present application is that: the two aforementioned embodiments allocate different data sets to different ML models, and distinguish the relevant data of each ML model by storing the relevant data of different ML models in different data sets respectively; the embodiment of the present application stores the relevant data of multiple ML models in one data set, and distinguishes the relevant data of each ML model by the information of the ML model on each data in the data set. The definition of each information, the sending and receiving of messages, the evaluation of ML models, etc. in these three embodiments are similar implementation principles, so they can be understood by reference and will not be repeated here. In addition, the three embodiments can be used in combination with each other without limitation.

[0348] Scenario 2:

[0349] FIG9 is a flowchart diagram of the communication method provided in an embodiment of the present application. This communication method is applicable to the above-mentioned communication system and mainly involves the AnLF functional entity, the MTLF functional entity, and the ADRF functional entity. In scenario 5, the relevant data of each ML model is stored in the same dataset. The MTLF functional entity can use the information of the ML model and the dataset to obtain the relevant data of the ML model from the ADRF functional entity.

[0350] As shown in Figure 9, the process of the communication method is as follows:

[0351] S901: The AnLF functional entity sends a request message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the request message from the AnLF functional entity.

[0352] The request message is used to request the ML model and carries information about the first data set. For details, please refer to the relevant introduction at "S801" above, which will not be repeated here.

[0353] S902: The MTLF functional entity sends a first message to the AnLF functional entity. Correspondingly, the AnLF functional entity receives the first message from the MTLF functional entity.

[0354] The first message carries information about the first ML model and information about the second ML model, so that the AnLF functional entity obtains the first ML model and the second ML model at corresponding positions according to the information about the first ML model and the information about the second ML model.

[0355] S903a: The AnLF functional entity sends the relevant data of the first ML model, the information of the first ML model, and the information of the first data set to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the relevant data of the first ML model, the information of the first ML model, and the information of the first data set from the AnLF functional entity.

[0356] S903b: The AnLF functional entity sends the relevant data of the second ML model, the information of the second ML model, and the information of the first data set to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the relevant data of the second ML model, the information of the second ML model, and the information of the first data set from the AnLF functional entity.

[0357] S904 , the ADRF functional entity stores relevant data of the first ML model based on the information of the first ML model and the information of the first data set, and stores relevant data of the second ML model based on the information of the second ML model and the information of the first data set.

[0358] When the ADRF functional entity stores the relevant data of the first ML model (denoted as relevant data #a) and the relevant data of the second ML model (denoted as relevant data #b), it can use the ML model ID to further identify each relevant data, such as using the first ML model to identify the relevant data #a and using the second ML model to identify the relevant data #b.

[0359] S905: The MTLF functional entity sends a request for obtaining relevant data of the first ML model to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the request for obtaining relevant data of the first ML model from the MTLF functional entity.

[0360] The request for obtaining relevant data of the first ML model carries information of the first data set and information of the first ML model, that is, the MTLF functional entity requests the relevant data of the first ML model from the ADRF functional entity.

[0361] S906: The ADRF functional entity sends the relevant data of the first ML model to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the relevant data of the first ML model from the ADRF functional entity.

[0362] S907 , the MTLF functional entity retrains or regenerates a third ML model based on relevant data of the first ML model.

[0363] S908: The MTLF functional entity sends information about the third ML model to the AnLF functional entity. Correspondingly, the AnLF functional entity receives information about the third ML model from the MTLF functional entity.

[0364] After receiving the information of the third ML model, the AnLF functional entity may obtain the third ML model, that is, the latest ML model, from the corresponding position according to the information of the third ML model, and use the third ML model.

[0365] For the specific implementation of S901-S908, please refer to the related introduction of S801-S806 mentioned above, which will not be repeated here.

[0366] It is understandable that S903a and S903b can be performed simultaneously or in sequence, such as performing S903a first and then S903b; or performing S903b first and then S903a, which is not limited in the embodiment of the present application.

[0367] Figure 10 is a structural diagram of a communication device according to an embodiment of the present application. As shown in Figure 10 , the communication device 1000 includes a transceiver module 1001 and a processing module 1002. For ease of illustration, Figure 10 only shows the main components of the communication device.

[0368] In some embodiments, the communication device 1000 may be applicable to the communication system shown in FIG. 4 , and perform the functions of the AnLF functional entity in the method shown in FIG. 5 .

[0369] Transceiver module 1001 is configured to receive information about a first ML model from a MTLF functional entity, where the information about the first ML model is used to obtain the first ML model. Processing module 1002 is configured to control transceiver module 1001 to send a first message to the MTLF functional entity, where the first message carries information about the first ML model and information about a first data set, where the first data set is used to store data related to the first ML model.

[0370] In one possible design scheme, the first ML model is multiple ML models, the information of the first ML model is the information of multiple ML models, the information of the first data set is the information of multiple data sets, the multiple ML models correspond one-to-one to the information of the multiple ML models, and the information of the multiple ML models corresponds one-to-one to the information of the multiple data sets.

[0371] In one possible design, the information of the first ML model is any one of the following: identification information of the first ML model, a URL address of the first ML model, or an FQDN of the first ML model.

[0372] In a possible design solution, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0373] In one possible design scheme, the processing module 1002 is further used to control the transceiver module 1001 to send a request message to the MTLF functional entity, where the request message is used to request the ML model; in response to the request message, the transceiver module 1001 is further used to receive information about the first ML model from the MTLF functional entity.

[0374] In one possible design scheme, the processing module 1002 is further used to control the transceiver module 1001 to send a request message to the MTLF functional entity, where the request message is used to request an ML model; in response to the request message, the transceiver module 1001 is further used to receive information about a second ML model from the MTLF functional entity, where the information about the second ML model is used to obtain the second ML model, and the information of the second data set carried in the request message is used to indicate that the second data set corresponds to the ML model sent by the MTLF functional entity after receiving the request message, and the second data set is used to store relevant data of the second ML model.

[0375] Optionally, the information of the second data set includes identification information of the second data set and / or identification information of the second ADRF functional entity.

[0376] Optionally, the first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

[0377] In one possible design scheme, the transceiver module 1001 is further used to receive a request for monitoring the ML model from the MTLF functional entity. The request for monitoring the ML model carries information of the first ML model and information of a third data set. The third data set is used to store relevant data of the first ML model.

[0378] In one possible design scheme, the transceiver module 1001 is further used to receive a request for monitoring the ML model from the MTLF functional entity, where the request for monitoring the ML model carries information of the first ML model; in response to the request for monitoring the ML model, the transceiver module 1001 is further used to send information of a third data set to the MTLF functional entity, where the third data set is used to store relevant data of the first ML model.

[0379] In one possible design scheme, the processing module 1002 is further used to monitor the analysis accuracy of the first ML model; when the analysis accuracy of the first ML model is insufficient, the processing module 1002 is further used to control the transceiver module 1001 to send a notification request message to the MTLF functional entity, where the notification request message carries information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0380] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity.

[0381] In one possible design solution, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0382] In a possible design solution, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0383] Optionally, the transceiver module 1001 may include a sending module (not shown in FIG10 ) and a receiving module (not shown in FIG10 ). The sending module is used to implement the sending function of the communication device 1000 , and the receiving module is used to implement the receiving function of the communication device 1000 .

[0384] Optionally, the communication device 1000 may further include a storage module (not shown in FIG10 ) storing a program or instruction. When the processing module 1002 executes the program or instruction, the communication device 1000 may perform the functions of the AnLF functional entity in the method shown in FIG5 in the above method.

[0385] It can be understood that the communication device 1000 can be a network device, or a chip (system) or other parts or components that can be set in the network device, or a device that includes a network device, which is not limited in this application.

[0386] In addition, the technical effects of the communication device 1000 can refer to the technical effects of the communication method shown in Figure 5, and will not be repeated here.

[0387] In some other embodiments, the communication device 1000 may be applicable to the communication system shown in FIG. 4 , and execute the functions of the MTLF functional entity in the method shown in FIG. 5 .

[0388] The processing module 1002 is used to control the transceiver module 1001 to send information about the first ML model to the AnLF functional entity, where the information about the first ML model is used to indicate the first ML model. The transceiver module 1001 is used to receive a first message from the AnLF functional entity, where the first message carries information about the first ML model and information about a first data set, where the first data set is used to store relevant data of the first ML model.

[0389] In one possible design scheme, the first ML model is multiple ML models, the information of the first ML model is the information of multiple ML models, the information of the first data set is the information of multiple data sets, the multiple ML models correspond one-to-one to the information of the multiple ML models, and the information of the multiple ML models corresponds one-to-one to the information of the multiple data sets.

[0390] In one possible design, the information of the first ML model is any one of the following: identification information of the first ML model, a URL address of the first ML model, or an FQDN of the first ML model.

[0391] In a possible design solution, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0392] In one possible design scheme, the transceiver module 1001 is also used to receive a request message from the AnLF functional entity, where the request message is used to request an ML model; upon receiving the request message, the processing module 1002 is further used to control the transceiver module 1001 to send information of the first ML model to the AnLF functional entity.

[0393] In one possible design scheme, the transceiver module 1001 is further used to receive a request message from the AnLF functional entity, where the request message is used to request an ML model. The request message carries information of a second data set, and the second data set is used to store relevant data of the ML model sent by the MTLF functional entity after receiving the request message. Upon receiving the request message, the processing module 1002 is further used to control the transceiver module 1001 to send information of the second ML model to the AnLF functional entity, where the information of the second ML model is used to obtain the second ML model.

[0394] Optionally, the information of the second data set includes identification information of the second data set and / or identification information of the second ADRF functional entity.

[0395] Optionally, the first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

[0396] Optionally, the processing module 1002 is further configured to evaluate a second ML model using a second data set.

[0397] In one possible design, the processing module 1002 is further configured to evaluate the first ML model using the first dataset.

[0398] In one possible design scheme, when the MTLF functional entity receives a registration request for ML model monitoring, the processing module 1002 is further used to control the transceiver module 1001 to send a request for monitoring the ML model to the AnLF functional entity. The request for monitoring the ML model carries information about the first ML model and information about a third data set. The third data set is used to store relevant data of the first ML model.

[0399] In one possible design, the transceiver module 1001 is further configured to receive information of a third data set from the AnLF functional entity, where the third data set is configured to store relevant data of the first ML model.

[0400] Optionally, information of the third data set is carried in a response and / or notification request message for ML model monitoring.

[0401] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity.

[0402] In one possible design solution, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0403] In a possible design solution, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0404] Optionally, the transceiver module 1001 may include a sending module (not shown in FIG10 ) and a receiving module (not shown in FIG10 ). The sending module is used to implement the sending function of the communication device 1000 , and the receiving module is used to implement the receiving function of the communication device 1000 .

[0405] Optionally, the communication device 1000 may further include a storage module (not shown in FIG10 ) storing a program or instruction. When the processing module 1002 executes the program or instruction, the communication device 1000 may perform the functions of the MTLF functional entity in the method shown in FIG5 in the above method.

[0406] It can be understood that the communication device 1000 can be a network device, or a chip (system) or other parts or components that can be set in the network device, or a device that includes a network device, which is not limited in this application.

[0407] In addition, the technical effects of the communication device 1000 can refer to the technical effects of the communication method shown in Figure 5, and will not be repeated here.

[0408] In other embodiments, the transceiver module 1001 is used to perform the transceiver function of the method shown in Figures 6 to 9 above, and the processing module 1002 is used to perform other functions of the method shown in Figures 6 to 9 except the transceiver function.

[0409] Optionally, the transceiver module 1001 may include a sending module (not shown in FIG10 ) and a receiving module (not shown in FIG10 ). The sending module is used to implement the sending function of the communication device 1000 , and the receiving module is used to implement the receiving function of the communication device 1000 .

[0410] Optionally, the communication device 1000 may further include a storage module (not shown in FIG10 ) that stores a program or instruction. When the processing module 1002 executes the program or instruction, the communication device 1000 may perform the functions of the MTLF functional entity, the AnLF functional entity, or the ADRF functional entity in the methods shown in FIG6 to FIG9 in the above method.

[0411] It can be understood that the communication device 1000 can be a network device, or a chip (system) or other parts or components that can be set in the network device, or a device that includes a network device, which is not limited in this application.

[0412] Figure 11 is a second structural diagram of a communication device provided in an embodiment of the present application. Exemplarily, the communication device may be a terminal, or a chip (system) or other component or assembly that can be provided in a terminal. As shown in Figure 11, the communication device 1100 may include a processor 1101. Optionally, the communication device 1100 may further include a memory 1102 and / or a transceiver 1103. The processor 1101 is coupled to the memory 1102 and the transceiver 1103, such as by a communication bus.

[0413] The following is a detailed introduction to the various components of the communication device 1100 with reference to FIG11 :

[0414] The processor 1101 is the control center of the communication device 1100 and can be a single processor or a collective term for multiple processing elements. For example, the processor 1101 can be one or more central processing units (CPUs), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0415] Optionally, the processor 1101 can execute various functions of the communication device 1100 by running or executing software programs stored in the memory 1102 and calling data stored in the memory 1102, such as executing the communication method shown in Figures 5 to 8 above.

[0416] In a specific implementation, as an embodiment, the processor 1101 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG11 .

[0417] In a specific implementation, as an embodiment, the communication device 1100 may also include multiple processors, such as the processor 1101 and the processor 1104 shown in FIG11 . Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0418] Among them, the memory 1102 is used to store the software program for executing the solution of this application, and the execution is controlled by the processor 1101. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0419] Alternatively, the memory 1102 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1102 may be integrated with the processor 1101 or exist independently and be coupled to the processor 1101 via an interface circuit (not shown in FIG. 11 ) of the communication device 1100, which is not specifically limited in this embodiment of the present application.

[0420] Transceiver 1103 is used for communication with other communication devices. For example, if communication device 1100 is a terminal, transceiver 1103 can be used to communicate with a network device or another terminal device. For another example, if communication device 1100 is a network device, transceiver 1103 can be used to communicate with a terminal or another network device.

[0421] Optionally, the transceiver 1103 may include a receiver and a transmitter (not shown separately in FIG11 ), wherein the receiver is used to implement a receiving function, and the transmitter is used to implement a transmitting function.

[0422] Optionally, the transceiver 1103 may be integrated with the processor 1101 or exist independently and be coupled to the processor 1101 through an interface circuit (not shown in FIG. 11 ) of the communication device 1100 . This embodiment of the present application does not specifically limit this.

[0423] It is understandable that the structure of the communication device 1100 shown in FIG11 does not constitute a limitation on the communication device, and an actual communication device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0424] In addition, the technical effects of the communication device 1100 can refer to the technical effects of the methods described in the above method embodiments, and will not be repeated here.

[0425] An embodiment of the present application also provides a communication chip having instructions stored therein. When the chip is run on a communication device, the method provided in the embodiment of the present application is implemented.

[0426] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0427] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0428] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (such as infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0429] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0430] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0431] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0432] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0433] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0434] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0435] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0436] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0437] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0438] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A communication method, characterized in that: The method comprises: The analysis logic function AnLF functional entity receives information of a first machine learning ML model from the model training logic function MTLF functional entity, and the information of the first ML model is used to obtain the first ML model; The AnLF functional entity sends a first message to the MTLF functional entity, where the first message carries information about the first ML model and information about a first data set, where the first data set is used to store relevant data of the first ML model.

2. The method according to claim 1, characterized in that The method further comprises: The AnLF functional entity sends a request message to the MTLF functional entity, where the request message is used to request an ML model; The AnLF functional entity receives information of the first ML model from the MTLF functional entity, including: In response to the request message, the AnLF functional entity receives information of the first ML model from the MTLF functional entity.

3. The method according to claim 1, characterized in that The method further comprises: The AnLF functional entity sends a request message to the MTLF functional entity, where the request message is used to request an ML model, and the request message carries information of a second data set; In response to the request message, the AnLF functional entity receives information about a second ML model from the MTLF functional entity, the information about the second ML model is used to obtain the second ML model, and the second data set is used to store relevant data of the ML model sent by the MTLF functional entity after receiving the request message.

4. The method according to claim 3, characterized in that The first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The AnLF functional entity receives a request for monitoring an ML model from the MTLF functional entity, where the request for monitoring an ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The AnLF functional entity receives a request for monitoring an ML model from the MTLF functional entity, where the request for monitoring an ML model carries information of the first ML model; In response to the request for monitoring the ML model, the AnLF functional entity sends information of a third data set to the MTLF functional entity, where the third data set is used to store relevant data of the first ML model.

7. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The AnLF functional entity monitors the analysis accuracy of the first ML model; In the case that the analysis accuracy of the first ML model is insufficient, the AnLF functional entity sends a notification request message to the MTLF functional entity, where the notification request message carries information of a third data set, where the third data set is used to store relevant data of the first ML model.

8. A communication method, characterized in that: The method comprises: The model training logic function MTLF functional entity sends information of a first machine learning ML model to the analysis logic function AnLF functional entity, where the information of the first ML model is used to indicate the first ML model; The MTLF functional entity receives a first message from the AnLF functional entity, where the first message carries information about the first ML model and information about a first data set, where the first data set is used to store relevant data of the first ML model.

9. The method according to claim 8, characterized in that The method further comprises: The MTLF functional entity receives a request message from the AnLF functional entity, where the request message is used to request an ML model; The MTLF functional entity sends information of the first ML model to the AnLF functional entity, including: Upon receiving the request message, the MTLF functional entity sends information of the first ML model to the AnLF functional entity.

10. The method according to claim 8, characterized in that The method further comprises: The MTLF functional entity receives a request message from the AnLF functional entity, where the request message is used to request an ML model, and the request message carries information of a second data set; Upon receiving the request message, the MTLF functional entity sends information of a second ML model to the AnLF functional entity, the information of the second ML model is used to obtain the second ML model, and the second data set is used to store relevant data of the ML model sent by the MTLF functional entity after receiving the request message.

11. The method according to claim 10, characterized in that The first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

12. The method according to claim 10 or 11, characterized in that: The method further comprises: The MTLF functional entity evaluates the second ML model based on the second data set.

13. The method according to any one of claims 8 to 12, characterized in that: The method further comprises: The MTLF functional entity evaluates the first ML model based on the first data set.

14. The method according to any one of claims 8 to 13, characterized in that: The method further comprises: In a case where the MTLF functional entity receives a registration request for ML model monitoring, the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity, where the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

15. The method according to any one of claims 8 to 13, characterized in that: The method further comprises: The MTLF functional entity receives information of a third data set from the AnLF functional entity, where the third data set is used to store relevant data of the first ML model.

16. The method according to claim 15, characterized in that The information of the third data set is carried in a response and / or notification request message for ML model monitoring.

17. A communication method, characterized in that: The method comprises: The model training logic function MTLF functional entity receives a request message from the analysis logic function AnLF functional entity, wherein the request message is used to request a machine learning ML model; The MTLF functional entity sends a first message to the AnLF functional entity, where the first message carries information about a first ML model and information about a first data set, where the information about the first ML model is used to indicate the first ML model, and the first data set is used to store relevant data of the first ML model.

18. The method according to claim 17, characterized in that The method further comprises: The MTLF functional entity sends a request for establishing the first data set in the first ADRF functional entity to the first ADRF functional entity.

19. The method according to claim 17 or 18, characterized in that The method further comprises: The MTLF functional entity evaluates the first ML model based on the first data set.

20. A communication method, characterized in that: The method comprises: The analysis logic function AnLF functional entity sends a request message to the model training logic function MTLF functional entity, wherein the request message is used to request a machine learning ML model; The AnLF functional entity receives a first message from the MTLF functional entity, where the first message carries information of a first ML model and information of a first data set, where the information of the first ML model is used to obtain the first ML model, and the first data set is used to store relevant data of the first ML model.

21. A communication method, characterized in that: The method comprises: The model training logic function MTLF functional entity receives a request message from the analysis logic function AnLF functional entity, wherein the request message carries information of a first data set, where the first data set is used to store relevant data of a plurality of ML models, and the plurality of ML models include a first ML model; The MTLF functional entity sends a request for obtaining relevant data of the first ML model to the data analysis and storage function ADRF functional entity, where the request for obtaining relevant data of the first ML model carries information of the first ML model and information of the first data set; The MTLF functional entity receives relevant data of the first ML model from the ADRF functional entity.

22. The method according to claim 21, characterized in that The method further comprises: The MTLF functional entity sends a first message to the AnLF functional entity, where the first message carries information of a first ML model, where the information of the first ML model is used to indicate the first ML model.

23. The method according to claim 21 or 22, characterized in that The method further comprises: The MTLF functional entity retrains or regenerates a second ML model based on relevant data of the first ML model.

24. The method according to claim 23, characterized in that The method further comprises: The MTLF functional entity sends information about the second ML model to the AnLF functional entity, where the information about the second ML model is used to indicate the second ML model.

25. A communication method, characterized in that: The method comprises: The data analysis and storage function ADRF functional entity receives the relevant data of the first machine learning ML model, the information of the first ML model and the information of the first data set from the analysis logic function AnLF functional entity, where the first data set is used to store the relevant data of the first ML model; The ADRF functional entity stores relevant data of the first ML model according to information of the first ML model and information of the first data set.

26. The method according to claim 25, characterized in that The method further comprises: The ADRF functional entity receives a request from the model training logic function MTLF functional entity for obtaining relevant data of the first ML model, wherein the request for obtaining relevant data of the first ML model includes information of the first ML model and information of the first data set; The ADRF functional entity sends relevant data of the first ML model to the MTLF functional entity based on information of the first ML model and information of the first data set.

27. A communication method, characterized in that: The method comprises: The analysis logic function AnLF functional entity receives a first message from the model training logic function MTLF functional entity, where the first message carries information of a first ML model, and the information of the first ML model is used to obtain the first ML model; The AnLF functional entity sends the relevant data of the first ML model, information of the first ML model and information of a first data set to the data analysis and storage function ADRF functional entity, where the first data set is used to store relevant data of multiple ML models, and the multiple ML models include the first ML model.

28. The method according to claim 27, characterized in that The method further comprises: The AnLF functional entity establishes the first data set.

29. The method according to any one of claims 1 to 28, characterized in that The information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator URL address of the first ML model, and a fully qualified domain name FQDN of the first ML model.

30. The method according to any one of claims 1 to 29, characterized in that The information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

31. The method according to any one of claims 1 to 30, characterized in that The relevant data includes at least one of the following: input data, predicted data, true value data during prediction, and deviation data during prediction.

32. The method according to any one of claims 1 to 31, characterized in that The AnLF functional entity is carried on a network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on a NWDAF network element.

33. A communication device, characterized in that: The apparatus comprises: a module for executing the method as claimed in any one of claims 1-32.

34. A communication device, characterized in that: The communication device comprises: a processor and a memory; the memory is used to store computer instructions, and when the processor executes the instructions, the communication device executes the method according to any one of claims 1-32.

35. A communication system, characterized in that: The communication system includes at least one of the following: an analysis logic function AnLF functional entity, a model training logic function MTLF functional entity, and a data analysis and storage function ADRF functional entity network element, wherein the AnLF functional entity is used to execute the method as described in any one of claims 1-7, or 20, or 27-28, the MTLF functional entity is used to execute the method as described in any one of claims 8-16, or 17-19, or 21-24, and the ADRF functional entity network element is used to execute the method as described in claim 25 or 26.

36. A communication chip, characterized in that: Instructions are stored therein, and when the chip runs on a communication device, the method according to any one of claims 1 to 32 is implemented.

37. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 32.

38. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are executed by a communication device, the method according to any one of claims 1 to 32 is executed.

Images