Communication method and device

Abstract

The invention provides a communication method and device, belongs to the technical field of communication, and is used for improving the reliability of ML model evaluation. According to the method, the data of different ML models can be stored in different data sets, and the reliability of ML model evaluation is improved by using the data in the data set corresponding to the ML model. For example, after an AnLF functional entity receives information of a first ML model, the AnLF functional entity can set to store related data of the first ML model in a first data set, namely, set a corresponding relation between the first ML model and the first data set, and send the corresponding relation to an MTLF functional entity by carrying the information of the first ML model and the information of the first data set in a first message; therefore, the MTLF functional entity can obtain the related data of the first ML model from the first data set so as to evaluate the first ML model based on the related data, thereby improving the reliability of evaluation of the first ML model.

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular to a communication method and device. Background Art

[0002] The network data analytics function (NWDAF) network element has data collection, model training, data analysis and model reasoning functions. That is to say, NWDAF can be used to collect relevant data from network elements, third-party service servers, terminal devices or network management systems, and perform model training based on the relevant data to obtain an artificial intelligence (AI) model or a machine learning (ML) model, and provide the AI ​​model or ML model to other NWDAF network elements. The AI ​​model or ML model can be used to assist the NWDAF network element in generating data analysis results based on the relevant data. In addition, NWDAF containing a model training logical function (MTLF) can be used to train a machine learning (ML) model and expose new training services, such as providing trained ML models.

[0003] In large-scale scenarios, NWDAF containing MTLF usually evaluates the ML model provided to NWDAF containing AnLF (analytics logical function, AnLF), such as accuracy evaluation and robustness evaluation. How to improve the reliability of ML model evaluation is currently a hot topic under discussion. Summary of the invention

[0004] Embodiments of the present application provide a communication method and apparatus for improving the reliability of ML model evaluation.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a communication method is provided, the method comprising: an analysis logic function AnLF functional entity receives information of a first machine learning ML model from a model training logic function MTLF functional entity; and sends a first message to the MTLF functional entity. The information of the first ML model is used to obtain the first ML model, the first message carries the information of the first ML model and information of a first data set, and the first data set is used to store relevant data of the first ML model.

[0007] Based on the method described in the first aspect, it can be known that in a large-scale scenario, the relevant data of each ML model (such as the data generated by the ML model during use) is usually stored in the same data set. Because in the data set, it is impossible to distinguish which ML model each data belongs to, so that the MTLF functional entity usually evaluates each ML model based on all the data in the data set, but this method makes the reliability of the evaluation low. Therefore, the data of different ML models can be stored in different data sets respectively, and the reliability of the evaluation of the ML model can be improved by using the data in the data set corresponding to the ML model. For example, after receiving the information of the first ML model, the AnLF functional entity can set the relevant data of the first ML model to be saved in the first data set, that is, set the corresponding relationship between the first ML model and the first data set, and send the corresponding relationship to the MTLF functional entity by carrying the information of the first ML model and the information of the first data set in the first message, so that the MTLF functional entity can obtain the relevant data of the first ML model from the first data set, so as to evaluate the first ML model based on the relevant data, thereby improving the reliability of the evaluation of the first ML model.

[0008] In a possible design scheme, the first ML model is a plurality of ML models, the information of the first ML model is the information of a plurality of ML models, the information of the first data set is the information of a plurality of data sets, the plurality of ML models correspond one-to-one with the information of the plurality of ML models, and the information of the plurality of ML models corresponds one-to-one with the information of the plurality of data sets. That is to say, when the MTLF functional entity sends a plurality of ML models to the AnLF functional entity, the AnLF functional entity can set a correspondence between the plurality of ML models and the plurality of data sets, and send the correspondence to the MTLF functional entity, so that the MTLF functional entity obtains a data set corresponding to any ML model among the plurality of ML models, and evaluates the ML model, thereby improving the reliability of the evaluation of the ML model.

[0009] In a possible design, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator URL address of the first ML model, and a fully qualified domain name FQDN of the first ML model. That is, by carrying this information, the first ML model can be accurately indicated, ensuring that the MTLF functional entity successfully obtains the corresponding relationship between the first ML model and the first data set.

[0010] In a possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis storage function ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the first data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the first data set and the identification information of the first ADRF functional entity in the first message, indicating that the first data set is stored in the first ADRF functional entity. In this way, the content contained in the information of the first data set can be flexibly set according to actual conditions.

[0011] In a possible design scheme, before the AnLF functional entity receives the information of the first ML model from the MTLF functional entity, the method described in the first aspect also includes: the AnLF functional entity sends a request message to the MTLF functional entity, and the request message is used to request the ML model; accordingly, the AnLF functional entity receives the information of the first ML model from the MTLF functional entity, including: in response to the request message, the AnLF functional entity receives the information of the first ML model from the MTLF functional entity. That is, when the AnLF functional entity requests the ML model from the MTLF functional entity, the information of the first data set may not be carried in the request message, and after receiving the first ML model, the first message may be sent according to the first ML model. In this way, inaccurate information of the first data set carried in the request message can be avoided, and communication overhead can be reduced.

[0012] In a possible design scheme, before the AnLF functional entity receives the information of the first ML model from the MTLF functional entity, the method described in the first aspect further includes: the AnLF functional entity sends a request message to the MTLF functional entity; in response to the request message, the AnLF functional entity receives the information of the second ML model from the MTLF functional entity; the request message carries the information of the second data set, the second data set is used to store the relevant data of the ML model sent by the MTLF functional entity after receiving the request message, and the information of the second ML model is used to obtain the second ML model. It can be understood that when the AnLF functional entity requests the MTLF functional entity for the ML model, the information of the data set corresponding to the ML model is carried in the request message, which can avoid the AnLF functional entity from sending a message indicating the correspondence between the second ML model and the second data set after receiving the second ML model, thereby reducing communication overhead.

[0013] Optionally, the information of the second data set includes identification information of the second data set and / or identification information of the second ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the second data set and the second ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the second data set and the identification information of the second ADRF functional entity in the request message, indicating that the second data set is stored in the second ADRF functional entity. In this way, the content contained in the information of the second data set can be flexibly set according to the actual situation.

[0014] Optionally, the first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

[0015] In a possible design scheme, the method described in the first aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0016] Alternatively, in another possible design scheme, the method described in the first aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, and the request for monitoring the ML model carries information of the first ML model; in response to the request for monitoring the ML model, the AnLF functional entity sends information of a third data set to the MTLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0017] Alternatively, in another possible design scheme, the method described in the first aspect also includes: the AnLF functional entity monitors the analysis accuracy of the first ML model; when the analysis accuracy of the first ML model is insufficient, the AnLF functional entity sends a notification request message to the MTLF functional entity, and the notification request message carries information of a third data set, and the third data set is used to save relevant data of the first ML model.

[0018] In the process of the AnLF functional entity monitoring the accuracy of the first ML model, the relevant data of the first ML model (such as the data of the deviation sent during prediction) can be stored in the third data set, so that the MTLF functional entity can subsequently evaluate, retrain or regenerate the ML model based on the data of the third data set. It can be seen that the correspondence between the first ML model and the third data set can be carried in the request for monitoring the ML model; or, it can be carried in the response for monitoring the ML model; or, it can be carried in the notification request message. In other words, according to the actual situation, the third data set can be determined by the AnLF functional entity or the MTLF functional entity, and the corresponding relationship between the relevant data of the first ML model and the third data set can be sent to the other party.

[0019] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the third data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the third data set and the identification information of the third ADRF functional entity in the request message, indicating that the third data set is stored in the third ADRF functional entity. In this way, the content contained in the information of the third data set can be flexibly set according to the actual situation.

[0020] In one possible design, the relevant data includes at least one of the following: input data, predicted data, true value data during prediction, and deviation data during prediction. The true value data during prediction of the ML model is the actual result of the input data of the ML model, not the predicted result; the deviation data during prediction of the ML model is the data where the predicted result deviates. In other words, the relevant data includes the data generated when the ML model is used.

[0021] In a possible design scheme, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element. It can be understood that the AnLF functional entity and the MTLF functional entity can be carried on different NWDAF network elements respectively, or on the same NWDAF network element, and the embodiment of the present application does not limit this.

[0022] In a second aspect, a communication method is provided, the method comprising: a model training logic function MTLF functional entity sends information of a first machine learning ML model to an analysis logic function AnLF functional entity, wherein the information of the first ML model is used to indicate the first ML model; the MTLF functional entity receives a first message from the AnLF functional entity, the first message carries information of the first ML model and information of a first data set, and the first data set is used to store relevant data of the first ML model.

[0023] In one possible design scheme, the first ML model is multiple ML models, the information of the first ML model is information of multiple ML models, the information of the first data set is information of multiple data sets, the multiple ML models correspond one-to-one to the information of the multiple ML models, and the information of the multiple ML models corresponds one-to-one to the information of the multiple data sets.

[0024] In a possible design scheme, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator URL address of the first ML model, and a fully qualified domain name FQDN of the first ML model.

[0025] In a possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0026] In a possible design scheme, before the MTLF functional entity sends information about the first ML model to the AnLF functional entity, the method described in the second aspect also includes: the MTLF functional entity receives a request message from the AnLF functional entity, and the request message is used to request the ML model; accordingly, the MTLF functional entity sends information about the first ML model to the AnLF functional entity, including: upon receiving the request message, the MTLF functional entity sends information about the first ML model to the AnLF functional entity.

[0027] In a possible design scheme, before the MTLF functional entity sends information about the first ML model to the AnLF functional entity, the method described in the second aspect also includes: the MTLF functional entity receives a request message from the AnLF functional entity; upon receiving the request message, the MTLF functional entity sends information about the second ML model to the AnLF functional entity; the request message is used to request the ML model, the request message carries information about a second data set, the second data set is used to store relevant data of the ML model sent by the MTLF functional entity after receiving the request message, and the information of the second ML model is used to obtain the second ML model.

[0028] Optionally, the information of the second data set includes identification information of the second data set and / or identification information of the second ADRF functional entity.

[0029] Optionally, the first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

[0030] Optionally, the method of the second aspect further includes: the MTLF functional entity evaluates the second ML model based on the second data set. Since the second data set only stores relevant data of the second ML model, using the second data set to evaluate the second ML model can improve the reliability of the evaluation of the second ML model.

[0031] In a possible design scheme, the method of the second aspect further includes: the MTLF functional entity evaluates the first ML model based on the first data set. Because the first data set only stores relevant data of the first ML model, using the first data set to evaluate the first ML model can improve the reliability of the evaluation of the first ML model.

[0032] In a possible design scheme, the method described in the second aspect also includes: when the MTLF functional entity receives a registration request for ML model monitoring, the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity, and the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0033] Alternatively, in another possible design scheme, the method described in the second aspect also includes: the MTLF functional entity receives information of a third data set from the AnLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0034] Optionally, information of the third data set is carried in a response and / or notification request message for ML model monitoring.

[0035] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity.

[0036] In a possible design scheme, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0037] In a possible design scheme, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0038] In addition, other technical effects of the method described in the second aspect can also refer to the technical effects of the method described in the first aspect, and will not be repeated here.

[0039] In a third aspect, a communication method is provided, the method comprising: a model training logic function MTLF functional entity receives a request message from an analysis logic function AnLF functional entity; and sends a first message to the AnLF functional entity. The request message is used to request a machine learning ML model, the first message carries information about the first ML model and information about a first data set, the information about the first ML model is used to indicate the first ML model, and the first data set is used to store relevant data of the first ML model.

[0040] Based on the method described in the third aspect, it can be known that in a large-scale scenario, the relevant data of each ML model (such as the data generated by the ML model during use) is usually stored in the same data set. Because in the data set, it is impossible to distinguish which ML model each data belongs to, so that the MTLF functional entity usually evaluates each ML model based on all the data in the data set, but this method makes the reliability of the evaluation low. Therefore, the data of different ML models can be stored in different data sets respectively, and the reliability of the ML model evaluation can be improved by using the data in the data set corresponding to the ML model. For example, after the MTLF functional entity receives a request message for requesting an ML model from the AnLF functional entity, it can be set to save the relevant data of the first ML model in the first data set, that is, set the corresponding relationship between the first ML model and the first data set, and carry the information of the first ML model and the information of the first data set in the first message, and send the first ML model and the corresponding relationship to the AnLF functional entity, so that the AnLF functional entity can obtain the first ML model according to the information of the first ML model, and save the relevant information of the first ML model in the first data set according to the corresponding relationship. Furthermore, the MTLF functional entity may obtain relevant data of the first ML model from the first data set to evaluate the first ML model based on the relevant data, thereby improving the reliability of the evaluation of the first ML model.

[0041] In a possible design scheme, the first ML model is a plurality of ML models, the information of the first ML model is the information of a plurality of ML models, the information of the first data set is the information of a plurality of data sets, the plurality of ML models correspond one-to-one with the information of the plurality of ML models, and the information of the plurality of ML models corresponds one-to-one with the information of the plurality of data sets. That is to say, when the MTLF functional entity sends a plurality of ML models to the AnLF functional entity, the AnLF functional entity can set a correspondence between the plurality of ML models and the plurality of data sets, and send the correspondence to the MTLF functional entity, so that the MTLF functional entity obtains a data set corresponding to any ML model among the plurality of ML models, and evaluates the ML model, thereby improving the reliability of the evaluation of the ML model.

[0042] In a possible design, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator URL address of the first ML model, and a fully qualified domain name FQDN of the first ML model. That is, by carrying this information, the first ML model can be accurately indicated, ensuring that the MTLF functional entity successfully obtains the corresponding relationship between the first ML model and the first data set.

[0043] In a possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis storage function ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the first data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the first data set and the identification information of the first ADRF functional entity in the first message, indicating that the first data set is stored in the first ADRF functional entity. In this way, the content contained in the information of the first data set can be flexibly set according to actual conditions.

[0044] In a possible design scheme, the method described in the third aspect further includes: the MTLF functional entity sends a request to the first ADRF functional entity to establish a first data set in the first ADRF functional entity. In this way, the MTLF functional entity can set a correspondence between the first ML model and the first data set.

[0045] In a possible design scheme, the method described in the third aspect further includes: the MTLF functional entity evaluates the first ML model based on the first data set. Because the first data set only stores relevant data of the first ML model, using the first data set to evaluate the first ML model can improve the reliability of the evaluation of the first ML model.

[0046] In a possible design scheme, the method described in the third aspect also includes: when the MTLF functional entity receives a registration request for ML model monitoring, the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity, and the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0047] Alternatively, in another possible design scheme, the method described in the third aspect also includes: the MTLF functional entity receives information of a third data set from the AnLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0048] In the process of the AnLF functional entity monitoring the accuracy of the first ML model, the relevant data of the first ML model can be stored in the third data set, so that the MTLF functional entity can subsequently evaluate, retrain or regenerate the ML model based on the data of the third data set. It can be seen that the correspondence between the first ML model and the third data set can be carried in the request for monitoring the ML model, or sent by the AnLF functional entity to the MTLF functional entity. That is, according to the actual situation, the third data set can be determined by the AnLF functional entity or the MTLF functional entity, and the corresponding relationship between the relevant data of the first ML model and the third data set can be sent to the other party.

[0049] Optionally, the information of the third data set is carried in a response and / or notification request message for ML model monitoring. That is, the AnLF functional entity can send the correspondence between the first ML model and the third data set to the MTLF functional entity at different stages of monitoring the accuracy of the first ML model. In this way, the time for sending the correspondence can be flexibly set.

[0050] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the third data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the third data set and the identification information of the third ADRF functional entity in the request message, indicating that the third data set is stored in the third ADRF functional entity. In this way, the content contained in the information of the third data set can be flexibly set according to the actual situation.

[0051] In one possible design, the relevant data includes at least one of the following: input data, predicted data, true value data during prediction, and deviation data during prediction. The true value data during prediction of the ML model is the actual result of the input data of the ML model, not the predicted result; the deviation data during prediction of the ML model is the data where the predicted result deviates. In other words, the relevant data includes the data generated when the ML model is used.

[0052] In a possible design scheme, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element. It can be understood that the AnLF functional entity and the MTLF functional entity can be carried on different NWDAF network elements respectively, or on the same NWDAF network element, and the embodiment of the present application does not limit this.

[0053] In a fourth aspect, a communication method is provided, the method comprising: an analysis logic function AnLF functional entity sends a request message to a model training logic function MTLF functional entity; and receiving a first message from the MTLF functional entity. The request message is used to request a machine learning ML model, the first message carries information about the first ML model and information about a first data set, the information about the first ML model is used to obtain the first ML model, and the first data set is used to store relevant data of the first ML model.

[0054] In one possible design scheme, the first ML model is multiple ML models, the information of the first ML model is information of multiple ML models, the information of the first data set is information of multiple data sets, the multiple ML models correspond one-to-one to the information of the multiple ML models, and the information of the multiple ML models corresponds one-to-one to the information of the multiple data sets.

[0055] In a possible design scheme, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator URL address of the first ML model, and a fully qualified domain name FQDN of the first ML model.

[0056] In a possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0057] In a possible design scheme, the method of the fourth aspect also includes: the AnLF functional entity sends the relevant data of the first ML model and the information of the first data set to the first ADRF functional entity based on the information of the first data set. In this way, the first ADRF functional entity can accurately save the relevant data of the first ML model in the first data set based on the information of the first data set.

[0058] In a possible design scheme, the method described in the fourth aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0059] Alternatively, in another possible design scheme, the method described in the fourth aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, and the request for monitoring the ML model carries information of the first ML model; in response to the request for monitoring the ML model, the AnLF functional entity sends information of a third data set to the MTLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0060] Alternatively, in another possible design scheme, the method described in the fourth aspect also includes: the AnLF functional entity monitors the analysis accuracy of the first ML model; when the analysis accuracy of the first ML model is insufficient, the AnLF functional entity sends a notification request message to the MTLF functional entity, and the notification request message carries information of a third data set, and the third data set is used to save relevant data of the first ML model.

[0061] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity.

[0062] In a possible design scheme, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0063] In a possible design scheme, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0064] In addition, other technical effects of the method described in the fourth aspect can also refer to the technical effects of the method described in the third aspect, which will not be repeated here.

[0065] In a fifth aspect, a communication method is provided, the method comprising: a model training logic function MTLF functional entity receives a request message from an analysis logic function AnLF functional entity; the MTLF functional entity sends a request for obtaining relevant data of a first ML model to a data analysis storage function ADRF functional entity; and receives relevant data of the first ML model from the ADRF functional entity. The request message carries information of a first data set, the first data set is used to store relevant data of multiple ML models, the multiple ML models include the first ML model; the request for obtaining relevant data of the first ML model carries information of the first ML model and information of the first data set.

[0066] Based on the method described in the first aspect, it can be known that in a large-scale scenario, the relevant data of each ML model (such as the data generated by the ML model during use) is usually stored in the same data set. Because in the data set, it is impossible to distinguish which ML model each data belongs to, so that the MTLF functional entity usually evaluates each ML model based on all the data in the data set, but this method makes the reliability of the evaluation low. Therefore, the data of different ML models can be marked with corresponding model labels and then stored in the same data set; by using the data with the model label corresponding to the ML model, the reliability of the evaluation of the ML model can be improved. For example, the MTLF functional entity can obtain the data set label from the received request message, that is, it is known that the data of each ML model is stored in the data set corresponding to the data set label; then send a request for obtaining the relevant data of the first ML model to the ADRF functional entity, the request carries the information of the first ML model and the information of the first data set, so that the ADRF functional entity can obtain the relevant data with the information of the first ML model in the first data set from its stored content, and send the relevant data to the MTLF functional entity, so that the MTLF functional entity evaluates the first ML model based on the relevant data, thereby improving the reliability of the evaluation of the first ML model.

[0067] In a possible design, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator URL address of the first ML model, and a fully qualified domain name FQDN of the first ML model. That is, by carrying this information, the first ML model can be accurately indicated, ensuring that the MTLF functional entity successfully obtains the corresponding relationship between the first ML model and the first data set.

[0068] In a possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis storage function ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the first data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the first data set and the identification information of the first ADRF functional entity in the first message, indicating that the first data set is stored in the first ADRF functional entity. In this way, the content contained in the information of the first data set can be flexibly set according to actual conditions.

[0069] In a possible design scheme, the method described in the fifth aspect further includes: the MTLF functional entity sends a first message to the AnLF functional entity, the first message carrying information of the first ML model, and the information of the first ML model is used to indicate the first ML model. In this way, the AnLF functional entity can obtain the first ML model based on the information of the first ML model.

[0070] In a possible design scheme, the method described in the fifth aspect also includes: the MTLF functional entity retrains or regenerates the second ML model based on relevant data of the first ML model.

[0071] Optionally, the method described in the fifth aspect further includes: the MTLF functional entity sends information of the second ML model to the AnLF functional entity, and the information of the second ML model is used to indicate the second ML model. That is, after retraining or regenerating the second ML model, the MTLF functional entity can send the second ML model to the AnLF functional entity, so that the AnLF functional entity can update the ML model used, that is, use the latest ML model, thereby improving the accuracy of ML model prediction.

[0072] In a possible design scheme, the method described in the fifth aspect also includes: when the MTLF functional entity receives a registration request for ML model monitoring, the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity, and the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0073] Alternatively, in another possible design scheme, the method described in the fifth aspect also includes: the MTLF functional entity receives information of a third data set from the AnLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0074] In the process of the AnLF functional entity monitoring the accuracy of the first ML model, the relevant data of the first ML model can be stored in the third data set, so that the MTLF functional entity can subsequently evaluate, retrain or regenerate the ML model based on the data of the third data set. It can be seen that the correspondence between the first ML model and the third data set can be carried in the request for monitoring the ML model, or sent by the AnLF functional entity to the MTLF functional entity. That is, according to the actual situation, the third data set can be determined by the AnLF functional entity or the MTLF functional entity, and the corresponding relationship between the relevant data of the first ML model and the third data set can be sent to the other party.

[0075] Optionally, the information of the third data set is carried in a response and / or notification request message for ML model monitoring. That is, the AnLF functional entity can send the correspondence between the first ML model and the third data set to the MTLF functional entity at different stages of monitoring the accuracy of the first ML model. In this way, the time for sending the correspondence can be flexibly set.

[0076] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity. That is to say, when each data set is stored in one ADRF, the correspondence between the data set and the ADRF functional entity can be preset or predefined, so that only the correspondence between the third data set and the first ML model can be indicated; and when there are multiple ADRFs, the correspondence between the data set and the ADRF functional entity can be indicated, such as by carrying the identification information of the third data set and the identification information of the third ADRF functional entity in the request message, indicating that the third data set is stored in the third ADRF functional entity. In this way, the content contained in the information of the third data set can be flexibly set according to the actual situation.

[0077] In one possible design, the relevant data includes at least one of the following: input data, predicted data, true value data during prediction, and deviation data during prediction. The true value data during prediction of the ML model is the actual result of the input data of the ML model, not the predicted result; the deviation data during prediction of the ML model is the data where the predicted result deviates. In other words, the relevant data includes the data generated when the ML model is used.

[0078] In a possible design scheme, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element. It can be understood that the AnLF functional entity and the MTLF functional entity can be carried on different NWDAF network elements respectively, or on the same NWDAF network element, and the embodiment of the present application does not limit this.

[0079] In a sixth aspect, a communication method is provided, the method comprising: a data analysis and storage function ADRF functional entity receives relevant data of a first machine learning ML model, information of the first ML model and information of a first data set from an analysis logic function AnLF functional entity; and stores relevant data of the first ML model based on the information of the first ML model and the information of the first data set; the first data set is used to save relevant data of the first ML model.

[0080] In a possible design scheme, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator URL address of the first ML model, and a fully qualified domain name FQDN of the first ML model.

[0081] In a possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0082] In a possible design scheme, the method described in the sixth aspect also includes: the ADRF functional entity receives a request from the model training logic function MTLF functional entity for obtaining relevant data of the first ML model, and the request for obtaining relevant data of the first ML model includes information of the first ML model and information of the first data set; the ADRF functional entity sends the relevant data of the first ML model to the MTLF functional entity based on the information of the first ML model and the information of the first data set.

[0083] In a possible design scheme, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0084] In a possible design scheme, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0085] In addition, other technical effects of the method described in the sixth aspect can also refer to the technical effects of the method described in the fifth aspect, and will not be repeated here.

[0086] In a seventh aspect, a communication method is provided, the method comprising: an analysis logic function AnLF functional entity receives a first message from a model training logic function MTLF functional entity; and sends relevant data of a first ML model, information of the first ML model, and information of a first data set to a data analysis storage function ADRF functional entity. The first message carries the information of the first ML model, the information of the first ML model is used to obtain the first ML model, and the first data set is used to store relevant data of multiple ML models, and the multiple ML models include the first ML model.

[0087] In a possible design scheme, the information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator URL address of the first ML model, and a fully qualified domain name FQDN of the first ML model.

[0088] In a possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0089] In a possible design, the method of the seventh aspect further includes: the AnLF functional entity establishes the first data set. That is, the AnLF functional entity may send a request to the ADRF functional entity to establish the first data set in the ADRF functional entity, thereby establishing the first data set.

[0090] In a possible design scheme, the method described in the seventh aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0091] Alternatively, in another possible design scheme, the method described in the seventh aspect also includes: the AnLF functional entity receives a request for monitoring the ML model from the MTLF functional entity, and the request for monitoring the ML model carries information of the first ML model; in response to the request for monitoring the ML model, the AnLF functional entity sends information of a third data set to the MTLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0092] Alternatively, in another possible design scheme, the method described in the seventh aspect also includes: the AnLF functional entity monitors the analysis accuracy of the first ML model; when the analysis accuracy of the first ML model is insufficient, the AnLF functional entity sends a notification request message to the MTLF functional entity, and the notification request message carries information of a third data set, and the third data set is used to save relevant data of the first ML model.

[0093] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity.

[0094] In a possible design scheme, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0095] In a possible design scheme, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0096] In addition, other technical effects of the method described in the seventh aspect can also refer to the technical effects of the method described in the fifth aspect, and will not be repeated here.

[0097] In an eighth aspect, a communication device is provided. The communication device includes: a module for executing the method described in any possible implementation of the first aspect to the seventh aspect, such as a transceiver module and a processing module. For example, the transceiver module is used to indicate the transceiver function of the communication device, and the processing module is used to execute functions of the communication device other than the transceiver function.

[0098] Optionally, the transceiver module may include a sending module and a receiving module, wherein the sending module is used to implement the sending function of the communication device described in the eighth aspect, and the receiving module is used to implement the receiving function of the communication device described in the eighth aspect.

[0099] Optionally, the communication device described in the eighth aspect may further include a storage module, which stores a program or instruction. When the processing module executes the program or instruction, the communication device can execute the method described in any possible implementation manner of the first aspect to the seventh aspect.

[0100] It can be understood that the communication device described in the eighth aspect can be a network device, or a chip (system) or other parts or components that can be set in the network device, or a device that includes a network device, and this application does not limit this.

[0101] In addition, the technical effects of the communication device described in the eighth aspect can refer to the technical effects of the method described in any possible implementation method of the first aspect to the seventh aspect, and will not be repeated here.

[0102] In a ninth aspect, a communication device is provided, comprising: a processor, the processor being configured to execute the method as described in any possible implementation manner in the first to fifth aspects.

[0103] In a possible design scheme, the communication device described in the ninth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the communication device described in the ninth aspect to communicate with other communication devices.

[0104] In a possible design scheme, the communication device described in the ninth aspect may also include a memory. The memory may be integrated with the processor or may be separately provided. The memory may be used to store the computer program and / or data involved in the method described in any possible implementation of the first aspect to the seventh aspect.

[0105] In an embodiment of the present application, the communication device described in the ninth aspect may be a terminal device or a network device, or a chip (system) or other parts or components that can be set in the terminal device or network device, or a device that includes the terminal device or network device.

[0106] In addition, the technical effects of the communication device described in the ninth aspect can refer to the technical effects of the method described in any possible implementation method of the first aspect to the seventh aspect, and will not be repeated here.

[0107] In a tenth aspect, a communication device is provided, comprising: a processor, the processor being coupled to a memory, the processor being configured to execute a computer program stored in the memory, so that the communication device executes the method described in any possible implementation manner in the first to seventh aspects.

[0108] In a possible design solution, the communication device described in the tenth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the communication device described in the tenth aspect to communicate with other communication devices.

[0109] In an embodiment of the present application, the communication device described in the tenth aspect may be a terminal device or a network device, or a chip (system) or other parts or components that can be set in the terminal device or network device, or a device that includes the terminal device or network device.

[0110] In addition, the technical effects of the communication device described in the tenth aspect can refer to the technical effects of the method described in any possible implementation method of the first aspect to the seventh aspect, and will not be repeated here.

[0111] In the eleventh aspect, a communication device is provided, comprising: a processor and a memory; the memory is used to store a computer program, and when the processor executes the computer program, the communication device executes the method described in any possible implementation method of the first aspect to the seventh aspect.

[0112] In a possible design solution, the communication device described in the eleventh aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the communication device described in the eleventh aspect to communicate with other communication devices.

[0113] In an embodiment of the present application, the communication device described in the eleventh aspect may be a terminal device or a network device, or a chip (system) or other parts or components that can be set in the terminal device or the network device, or a device that includes the terminal device or the network device.

[0114] In addition, the technical effects of the communication device described in the eleventh aspect can refer to the technical effects of the method described in any possible implementation method of the first aspect to the seventh aspect, and will not be repeated here.

[0115] In the twelfth aspect, a communication system is provided. The communication includes: an analysis logic function AnLF functional entity, a model training logic function MTLF functional entity, and a data analysis storage function ADRF functional entity network element. Among them, the AnLF functional entity is used to execute the method described in any one of the first aspect, the fourth aspect, or the seventh aspect; the MTLF functional entity is used to execute the method described in any one of the second aspect, the third aspect, or the fifth aspect; the ADRF functional entity network element is used to execute the method described in the sixth aspect.

[0116] In a thirteenth aspect, a communication chip is provided, in which instructions are stored, and when the chip runs on a communication device, the method described in any one of the implementation methods of the first to seventh aspects is implemented.

[0117] In a fourteenth aspect, a computer-readable storage medium is provided, comprising: a computer program or instructions; when the computer program or instructions are executed on a computer, the computer executes the method described in any possible implementation method of the first to fifth aspects.

[0118] In a fifteenth aspect, a computer program product is provided, comprising a computer program or instructions, which, when executed on a computer, enables the computer to execute a method as described in any possible implementation of aspects one to five. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 A schematic diagram of the architecture of the fifth generation mobile communication system 5GS provided in an embodiment of the present application;

[0120] Figure 2 A schematic diagram of a process for monitoring the accuracy of a machine learning ML model based on a model training logic function MTLF provided in an embodiment of the present application;

[0121] Figure 3 A schematic diagram of a process for monitoring the accuracy of ML model analysis provided in an embodiment of the present application;

[0122] Figure 4 A schematic diagram of the architecture of a communication system is provided for an embodiment of the present application;

[0123] Figure 5 Schematic diagram of the communication method provided in the embodiment of the present application Figure 1 ;

[0124] Figure 6 Schematic diagram of the communication method provided in the embodiment of the present application Figure 2 ;

[0125] Figure 7 Schematic diagram of the communication method provided in the embodiment of the present application Figure 3 ;

[0126] Figure 8 Schematic diagram of the communication method provided in the embodiment of the present application Figure 4 ;

[0127] Fig. 9 Schematic diagram of the communication method provided in the embodiment of the present application Figure 5 ;

[0128] Fig.10 This is a schematic diagram of the structure of the communication device provided in the embodiment of the present application. Figure 1 ;

[0129] Fig.11 This is a schematic diagram of the structure of the communication device provided in the embodiment of the present application. Figure 2 . DETAILED DESCRIPTION

[0130] For ease of understanding, the technical terms involved in the embodiments of the present application are first introduced below.

[0131] 1. The fifth generation (5G) mobile communication system (referred to as 5G system (5G system, 5GS)):

[0132] Figure 1 The schematic diagram of the 5GS architecture is shown in Figure 2. Figure 1 As shown, 5GS includes: access network (AN) and core network (CN), and may also include: terminal equipment.

[0133] The terminal device may be a terminal device with transceiver functions, or a chip or chip system that can be set in the terminal device. The terminal device may also be called user equipment (UE), access terminal, subscriber unit, user station, mobile station (MS), mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device. The terminal in the embodiments of the present application can be a mobile phone, a cellular phone, a smart phone, a tablet computer, a wireless data card, a personal digital assistant (PDA), a wireless modem, a handset, a laptop computer, a machine type communication (MTC) terminal, a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a vehicle-mounted terminal, a road side unit (RSU) with terminal function, etc. The terminal device of the present application may also be a vehicle-mounted module, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip or a vehicle-mounted unit built into the vehicle as one or more components or units.

[0134] The above-mentioned AN is used to implement access-related functions, can provide network access functions for authorized users in a specific area, and can determine transmission links of different qualities to transmit user data according to the user's level, business requirements, etc. AN forwards control signals and user data between the terminal and CN. AN may include: access network equipment, also known as radio access network (RAN) equipment. CN is mainly responsible for maintaining the subscription data of the mobile network and providing session management, mobility management, policy management, and security authentication functions for terminal equipment. CN mainly includes the following: user plane function (UPF), authentication server function (AUSF), access and mobility management function (AMF), session management function (SMF), network slice selection function (NSSF), network exposure function (NEF), network storage function (NRF), policy control function (PCF), unified data management (UDM), unified data storage (UDR), application function (AF), network data analytics function (NWDAF), and analytics data repository function (ADRF).

[0135] like Figure 1 As shown, the UE accesses the 5G network through the RAN device, the UE communicates with the AMF through the N1 interface (referred to as N1); the RAN communicates with the AMF through the N2 interface (referred to as N2); the RAN communicates with the UPF through the N3 interface (referred to as N3); the SMF communicates with the UPF through the N4 interface (referred to as N4), and the UPF accesses the data network (DN) through the N6 interface (referred to as N6). In addition, Figure 1The control plane functions shown in the figure, such as AUSF, AMF, SMF, NSSF, NEF, NRF, PCF, UDM, UDR, AF, NWDAF, and ADRF, interact with each other using service-oriented interfaces. For example, the service-oriented interface provided by AUSF to the outside includes Nausf; the service-oriented interface provided by AMF to the outside includes Namf; the service-oriented interface provided by SMF to the outside includes Nsmf; the service-oriented interface provided by NSSF to the outside includes Nnssf; the service-oriented interface provided by NEF to the outside includes Nnef; the service-oriented interface provided by NRF to the outside includes Nnrf; the service-oriented interface provided by PCF to the outside includes Npcf; the service-oriented interface provided by UDM to the outside includes Nudm; the service-oriented interface provided by UDR to the outside includes Nudr; the service-oriented interface provided by AF to the outside includes Naf; the service-oriented interface provided by NWDAF to the outside includes Nnwdaf; and the service-oriented interface provided by ADRF to the outside includes Nadrf.

[0136] RAN equipment can be equipment that provides access to terminal equipment. For example, RAN equipment may include: a next-generation mobile communication system, such as an access network device of 6G, such as a 6G base station, or in the next-generation mobile communication system, the network equipment may also have other naming methods, which are all included in the protection scope of the embodiments of the present application, and the present application does not make any restrictions on this. Alternatively, the RAN equipment may also include 5G, such as a gNB in ​​a new radio (NR) system, or one or a group of (including multiple antenna panels) antenna panels of a base station in 5G, or a network node constituting a gNB, a transmission point (transmission and reception point, TRP or transmission point, TP) or a transmission measurement function (transmission measurement function, TMF), such as a baseband unit (building baseband unit, BBU), or a centralized unit (centralized unit, CU) or a distributed unit (distributed unit, DU), an RSU with base station function, or a wired access gateway, or a 5G core network. Alternatively, RAN equipment may also include access points (APs) in wireless fidelity (WiFi) systems, wireless relay nodes, wireless backhaul nodes, various forms of macro base stations, micro base stations (also called small stations), relay stations, access points, wearable devices, vehicle-mounted devices, and the like.

[0137] UPF is mainly responsible for user data processing (forwarding, receiving, billing, etc.).

[0138] AUSF is mainly used to perform security authentication of terminal devices.

[0139] AMF is mainly used for mobility management in mobile networks, such as user location update, user network registration, user switching, etc.

[0140] SMF is mainly used for session management in mobile networks, such as session establishment, modification, and release. Specific functions include allocating Internet Protocol (IP) addresses to users and selecting UPFs that provide data packet forwarding functions.

[0141] PCF mainly supports providing a unified policy framework to control network behavior, provides policy rules to the control layer network functions, and is responsible for obtaining user subscription information related to policy decisions. PCF can provide policies to AMF and SMF, such as quality of service (QoS) policy, slice selection policy, etc.

[0142] NSSF is mainly used to select network slices for terminal devices.

[0143] NEF is mainly used to support the opening of capabilities and events. For example, NEF can open some capabilities of the 5G network to third-party applications through the application program interface (API). Third-party applications obtain some capabilities of the 5G network through AF calling the API provided by NEF, so that third-party applications can control certain behaviors of the 5G network and terminal devices.

[0144] UDM is mainly used to store user data, such as contract data, authentication / authorization data, etc.

[0145] UDR is mainly used to store structured data, including contract data and policy data, externally exposed structured data, and application-related data.

[0146] AF mainly supports interaction with CN to provide services, such as influencing data routing decisions, policy control functions, or providing some third-party services to the network side.

[0147] The NWDAF network element supports data collection from other network functions and AF network elements; supports data collection from operation administration and maintenance (OAM) network elements; supports providing analysis information to other network functions and AF network elements. In other words, the NWDAF network element has the functions of data collection, model training, data analysis, and model reasoning, that is, it can be used to collect relevant data from network network elements, third-party service servers, terminal devices, or network management systems; perform data analysis based on the relevant data to obtain analysis results; and provide analysis results to network network elements, third-party service servers, terminal devices, or network management systems. The analysis results can assist the network in selecting service quality parameters for the service, or assist the network in performing traffic routing, or assist the network in selecting background data transmission strategies, etc. In addition, the NWDAF network element can also be used to collect relevant data from network network elements, third-party service servers, terminal devices, or network management systems, and perform model training based on the relevant data to obtain an AI model or an ML model, and provide an AI model or an ML model to other NWDAF network elements, which can be used to assist the NWDAF network element in generating data analysis results based on the relevant data.

[0148] NWDAF with AnLF can be used to perform model inference, export analytical information, i.e., export statistics and / or predictions based on analytical consumer requests, and expose analytical services. NWDAF with MTLF can be used to train ML models, and expose new training services, such as providing trained ML models.

[0149] ADRF network elements can provide storage and retrieval services for data and analysis for consumers. For example, a consumer sends a data management storage request (Nadrf Data Management StorageRequest) containing the data or analysis to be stored to an ADRF network element; or, a consumer sends a data management storage subscription request (Nadrf Data ManagementStorage Subscription Request) to an ADRF network element, requesting the ADRF network element to subscribe to receive data or analysis for storage, and then the ADRF network element subscribes to the NWDAF network element or the data collection coordination function (DCCF) network element to obtain the data or analysis, providing the ADRF network element notification address, or notification association identity (ID). The analysis or data is then provided as a notification using the DCCF data management (Ndccf_DataManagement) service, the NWDAF data management (Nnwdaf_DataManagement) service, or the MFAF (messaging framework adaptor NF) data management (Nmfaf_3caDataManagement) service.

[0150] It can be understood that the functions mentioned in the embodiments of the present application can also be expressed as functional network elements or functional entities. For example, UPF can be expressed as UPF network element, AMF can be expressed as AMF network element, SMF can be expressed as SMF network element, PCF can be expressed as PCF network element, and so on, without limitation.

[0151] 2. Process of ML model accuracy monitoring based on MTLF:

[0152] like Figure 2 As shown, Figure 2 The process of monitoring the accuracy of the configured ML model using newly collected data is described. The NWDAF including AnLF can provide inference data to the NWDAF including MTLF for model accuracy monitoring, and the NWDAF including MTLF determines the retraining or reconfiguration of the ML model. The following specifically describes the process of MTLF-based ML model accuracy monitoring.

[0153] S201, an analytics consumer initiates an analytics exposure service subscription (Nnwdaf_AnalyticsSubscription_subscribe) service operation to the NWDAF including the AnLF.

[0154] S202 , the NWDAF including the AnLF requests the ML model from the NWDAF including the MTLF using the ML model provision subscription (Nnwdaf_MLModelProvision_Subscribe) service operation.

[0155] The NWDAF including AnLF may include an analysis accuracy threshold as an indicator for performing accuracy monitoring operations. The NWDAF including AnLF may include a dataset tag (DataSetTag) and / or ADRFID for storing inference data and obtaining inference data from ADRF. The inference data is related to the accuracy monitoring of the ML model, as well as retraining or reconfiguration, and the inference data may include input data, prediction data, and ground truth data at the time of prediction.

[0156] S203, the NWDAF including the MTLF provides the trained ML model to the NWDAF including the AnLF.

[0157] The NWDAF including the MTLF may include accuracy information indicating the accuracy of the ML model during training. The NWDAF including the MTLF may send the ML model to the NWDAF including the AnLF through the ML model provision notification (Nnwdaf_MLModelProvision_Notify) service operation.

[0158] S204 , the NWDAF including the AnLF registers the use of the ML model with the NWDAF including the MTLF to indicate its ability to send analysis feedback information and / or ML model accuracy information from analysis consumers.

[0159] The NWDAF including AnLF can register the use of ML models to the NWDAF including MTLF through the ML Model Monitor Registration (Nnwdaf_MLModelMonitor_Register) service operation.

[0160] S205, due to the registration in the previous step, if the NWDAF containing ANLF supports the service operation, the NWDAF containing MTLF can subscribe to the NWDAF containing ANLF by calling the ML model monitoring subscription (Nnwdaf_MLModelMonitor_Subscribe) service operation to obtain analysis feedback information and / or ML model accuracy information from the analysis consumer.

[0161] S206, the analytics consumer may send analytics feedback information in an analytics subscription (Nnwdaf_AnalyticsSubscription_Subscribe) message, as described in Section 6.1.1.

[0162] S207, the NWDAF including AnLF may send the analysis feedback information received from the analysis consumer and / or the provided ML model accuracy information by calling the ML model monitoring notification (Nnwdaf_MLModelMonitor_Notify) service operation according to the request in S205. When the NWDAF including MTLF receives the analysis feedback information or the ML model accuracy information, the NWDAF including MTLF may trigger from S208 to S2013 to improve the ML model accuracy.

[0163] S208a to S208f, the NWDAF including the MTLF determines whether to perform ML model accuracy monitoring and retraining / reconfiguration of the ML model by collecting new data from various data sources based on requests from one or more NWDAFs including the AnLF or its local policy:

[0164] 1) NWDAF including MTLF can collect new data for ML model accuracy monitoring, retraining and reconfiguration from data sources NF and DCCF by calling data exposure subscription (Nnf_EventExposure_Subscribe) and data management subscription (Ndccf_DataManagement_Susubscribe) service operations respectively.

[0165] 2) When S202 gives ADRFID and / or DataSetTag, in S202, the NWDAF containing MTLF can retrieve historical data from the ADRF indicated by the NWDAF containing AnLF by calling the service operation of data management retrieval request (Nadrf_DataManagementRetrievalRequest) or data management retrieval subscription (Nadrf_DataManagementRetrieval_Subscribe). Otherwise, the NWDAF containing MTLF can retrieve historical data from the DCCF or the NWDAF containing AnLF by calling the data management subscription (Ndccf_DataManagement_Subscribe) of DCCF or the data management subscription (Nnwdaf_DataManagement_Subsubscribe) of NWDAF, respectively.

[0166] 3) If the NWDAF containing AnLF in S202 does not include a DataSetTag with ADRFID, the NWDAF containing MLTF may request the ADRF to subscribe to a collection of analysis and data corresponding to the analysis generated by the ML model provided in S203 using the procedure defined in Section 6.2B.3.

[0167] 4) The NWDAF including the MTLF may subscribe to the UDM by calling the SDM subscription (Nudm_SDM_subscribe) service operation of the UDM to obtain notifications of changes in subscription data of the target reported by the ML model, and the UDM may subscribe to the UDR by calling the DM subscription (Nudr_DM_subscribe) service operation of the UDR to obtain notifications of modifications in UE subscription data.

[0168] 5) NWDAF including MTLF can determine the status of data source NF by collecting fault prediction analysis data from MDAS using management data analytics (MDA) requests, thereby taking data quality into account for accuracy monitoring.

[0169] If the NWDAF containing MTLF has collected new test data and performed ML model accuracy monitoring and retraining triggered by other NWDAFs containing AnLF (for ML model accuracy monitoring and retraining), the NWDAF containing MTLF determines whether to use the data for subscription based on its internal logic.

[0170] S209a-S209f, the NWDAF including the MTLF receives the requested data from the various sources requested in S208a-S208f.

[0171] S2010, based on the analysis and data collected from S209a-S209f, the NWDAF including MTLF uses the prediction and the actual measurement data observed at the time of prediction reference to calculate accuracy. If the data quality of a data source is detected to be poor, the NWDAF including MTLF may discard the data of the data source. If only input data and ground truth data are available, the NWDAF including MTLF can generate predictions using the collected input data to calculate accuracy.

[0172] S2011, an accuracy report is sent to the NWDAF including the AnLF, for example, when a reporting threshold is reached by calling the NWDAF's ML model provision notification (Nnwdaf_MLModelProvision_Notify) service operation.

[0173] S2012, based on the calculated accuracy, the NWDAF including the MTLF may decide to re-train or re-provision the ML model.

[0174] S2013, when the newly generated or retrained ML model is ready, the NWDAF including the MTLF sends the new or retrained ML model to the NWDAF including the AnLF by calling the NWDAF's ML model provision notification (Nnwdaf_MLModelProvision_Notify) service operation. The NWDAF including the MTLF may send an accuracy report of the new or retrained ML model to the NWDAF including the AnLF.

[0175] It can be understood that the above-mentioned process of MTLF-based ML model accuracy monitoring can refer to the relevant content in the 3rd Generation Partnership Project (3GPP) TS23.288V18.2.0, which will not be repeated here.

[0176] 3. Process for monitoring ML model analysis accuracy:

[0177] Due to the registration of monitoring of the analysis accuracy of the ML model received from the NWDAF containing AnLF and local policy, the NWDAF containing MTLF subscribes to the NWDAF containing AnLF to receive notifications of the accuracy of the ML model or analysis feedback information of the ML model. The NWDAF containing MTLF can obtain the subscription endpoint address of the NWDAF containing AnLF from the information received in the previous registration or through the service discovery process at the NRF.

[0178] Figure 3 The procedure for monitoring the ML model analysis accuracy or delivering ML model analysis feedback information is described. The NWDAF-MLModelMonitor_Subscribe and NWDAF-MLModelMonitor_Notify service operations are used for this purpose. The service consumer (i.e., the NWDAF containing the MTLF) subscribes at the service producer (i.e., the NWDAF containing the AnLF) to be notified when the analysis accuracy of the previously provided ML model is not sufficient or when analysis feedback information is retrieved from the analysis consumer NF.

[0179] S300 , after receiving the ML model monitoring registration (Nnwdaf_MLModelMonitor_Register) request from NWDAF, based on the local policy, the NWDAF including the MTLF decides to subscribe to the analysis accuracy monitoring of the ML model.

[0180] S301, the NWDAF including the MTLF sends a NWDAF ML model monitoring subscription (Nnwdaf_MLModelMonitor_Subscribe) request (analysis ID, unique identifier of the ML model to be monitored, accuracy metric to be monitored, optional reporting threshold or reporting period) to the NWDAF including the AnLF subscription endpoint.

[0181] S302: The NWDAF including the AnLF sends a response to the NWDAF including the MTLF.

[0182] S303, the analysis consumer NF may send analysis feedback information to the NWDAF including the AnLF.

[0183] S304, when S301 is triggered, the NWDAF including AnLF may start monitoring the analysis accuracy of the ML model (if it has not started yet). Among them, the NWDAF including AnLF can monitor the analysis accuracy in various ways, such as comparing the prediction of the ML model and its corresponding real data, comparing the changes in the internal configuration of the analysis ID generation, the previously existing records of the analysis accuracy information, etc.

[0184] S305, the NWDAF including the AnLF determines whether the analysis accuracy of the ML model is insufficient, that is, the deviation of the output analysis using the trained ML model from the basic fact data (collected from the data producer NF, corresponding to the analysis ID requested when predicting the reference) is greater than the reporting threshold (configured or received locally in the subscription request), or the reporting period indicated in the subscription request is reached.

[0185] S306, if the analysis feedback information is retrieved in S303, or if the NWDAF including AnLF detects that the analysis accuracy of the ML model is insufficient in S305, the NWDA including AnLF sends a NWDAF ML model monitoring notification (Nnwdaf_MLModelMonitor_Notify) request to the notification endpoint (such as the NWDAF including MTLF). The notification request includes the analysis feedback information or the analysis accuracy information of the ML model (such as a deviation value indicating the deviation of the prediction generated using the ML model from the ground truth data, and the network data when the deviation occurs (which can be used by the NWDAF including MTLF for possible ML model retraining), and the number of inferences performed during the time interval between the NWDAF ML model monitoring registration (Nnwdaf_MLModelMonitor_Register) request and the notification request, or between the time of the last notification message and the time of the current notification message, indicating that the analysis accuracy of the ML model does not meet the accuracy requirements of the ML model.

[0186] S307, the NWDAF including the MTLF sends a response.

[0187] S308, the NWDAF including the MTLF determines whether the ML model is degraded based on the notification of S306. If the notification includes analysis feedback information, the NWDAF including the MTLF may determine that the ML model is degraded according to the procedure described in Section 6.2E.2. Otherwise, when the NWDAF including the MTLF has received a plurality of analysis accuracy information from one or more NWDAFs including the AnLF, it may consider that the ML model is degraded / needs to be updated, i.e., a sufficient number of analysis accuracy information is received from one or more NWDAs including the AnLF, indicating that the analysis accuracy is insufficient.

[0188] It is understood that the actual mechanism used by the NWDAF including the MTLF to determine the degradation of the ML model is an internal process of the NWDAF including the MTLF. For example, the NWDAF within the MTLF-calculates the global accuracy based on the analysis accuracy information and the number of inferences received from multiple NWDAFs including the AnLF.

[0189] S309, when the ML model is considered to be downgraded / to be updated in S308, the NWDAF including the MTLF retrains the existing ML model or selects a new ML model. If the network data is not included in the NWDAF's ML model monitoring notification (Nnwdaf_MLModelMonitor_Notify) request in S305, the NWDAF including the MTLF may request data from the NWDAF including the AnLF, ADRF and / or other 5GS entities specified in Section 6.2, and use the collected data for ML model retraining. As described in Section 6.2A, the NWDAF including the MTLF notifies the NWDAF including the AnLF of the updated trained ML model information by calling the NWDAF's ML model provision notification (Nnwdaf_MLModelProvision_Notify) service operation.

[0190] It can be understood that the above-mentioned process of MTLF-based ML model accuracy monitoring can refer to the relevant content in 3GPPTS23.288V18.2.0, which will not be repeated here.

[0191] It can also be understood that in the embodiments of the present application, "monitoring" and "monitoring" may refer to the same action and the two may be expressed interchangeably.

[0192] According to the above Figure 2 and Figure 3As can be seen from the introduction, the aforementioned MTLF-based ML model accuracy monitoring process and the ML model analysis accuracy monitoring process are both for the accuracy monitoring or accuracy analysis of one ML model. In other words, when there is only one ML model, the data set corresponding to the ML model can be directly used to complete the evaluation of the ML model.

[0193] Through research, it is found that when there are multiple ML models, the relevant data of the multiple ML models will be stored in one data set, so when evaluating any of the multiple ML models, all the data in the data set is used, that is, it is impossible to obtain the relevant data corresponding to any of the multiple ML models from the data set, which reduces the reliability of the evaluation of the ML model. Therefore, how to improve the reliability of ML model evaluation in large-scale scenarios is currently a hot topic for discussion.

[0194] In response to the above technical problems, the embodiments of the present application propose the following technical solutions to improve the reliability of ML model evaluation.

[0195] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0196] The technical solutions of the embodiments of the present application can be applied to various communication systems, such as wireless network (Wi-Fi) systems, vehicle to everything (V2X) communication systems, device-to-device (D2D) communication systems, Internet of Vehicles communication systems, fourth generation (4G) mobile communication systems, such as long term evolution (LTE) systems, worldwide interoperability for microwave access (WiMAX) communication systems, fifth generation (5G) systems, such as new radio (NR) systems, and future communication systems.

[0197] The present application will present various aspects, embodiments or features around a system that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. In addition, combinations of these schemes may also be used.

[0198] In addition, in the embodiments of the present application, words such as "exemplary" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0199] In the embodiments of the present application, "information", "signal", "message", "channel" and "signaling" can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are matched. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are matched. In addition, the " / " mentioned in the present application can be used to express an "or" relationship.

[0200] The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0201] To facilitate understanding of the embodiments of the present application, first Figure 4 The communication system shown in the example is used to describe in detail the communication system applicable to the embodiment of the present application. Figure 4 A schematic diagram of the architecture of a communication system applicable to the communication method provided in an embodiment of the present application.

[0202] like Figure 4 As shown, the communication system can be applicable to the above 5GS, including: AnLF functional entity and MTLF functional entity. The AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or, the MTLF functional entity is carried on the NWDAF network element.

[0203] It can be understood that the AnLF functional entity and the MTLF functional entity can be set on the same NWDAF network element, that is, one NWDAF network element includes the AnLF functional entity and the MTLF functional entity; they can also be separately set on different NWDAF network elements, that is, different NWDAF network elements include the AnLF functional entity and the MTLF functional entity, which is not limited in the embodiments of the present application. In addition, the "AnLF functional entity" and "MTLF functional entity" in the embodiments of the present application are only an exemplary expression, and the "AnLF functional entity" can also be replaced by any possible expression, such as "NWDAF including AnLF", etc., and the "MTLF functional entity" can also be replaced by any possible expression, such as "NWDAF including MTLF", etc., which is not limited in the embodiments of the present application.

[0204] Optionally, the communication system may further include an ADRF functional entity. It can be understood that in the embodiment of the present application, the ADRF functional entity and the ADRF network element are different expressions of the same thing, that is, the ADRF functional entity may also be referred to as an ADRF network element.

[0205] In a communication system, relevant information of different ML models can be stored in different data sets, that is, relevant data of each ML model can be distinguished by the data set. For example, after receiving the information of the first ML model sent by the MTLF functional entity, the AnLF functional entity can obtain the first ML model according to the information, and set the relevant information of the first ML model to be saved in the first data set, that is, set the correspondence between the first ML model and the first data set. Then, by carrying the information of the first ML model and the information of the first data set in the first message, the corresponding relationship can be sent to the MTLF functional entity, so that the MTLF functional entity can subsequently obtain the relevant data of the first ML model from the first data set, and evaluate the first ML model based on the relevant data, such as robustness evaluation. Because there is no relevant data of other ML models in the first data set, that is, there is only relevant data of the first ML model, the reliability of the evaluation can be improved in this way.

[0206] The following will be combined Figure 5 The communication method provided in the embodiment of the present application can be applied to the above communication system, which is described in detail below.

[0207] Figure 5 Schematic diagram of the communication method provided in the embodiment of the present application Figure 1 The communication method is applicable to the above communication system and involves the interaction between the AnLF functional entity and the MTLF functional entity.

[0208] Specifically, Figure 5 As shown, the process of the communication method is as follows:

[0209] S501, the MTLF functional entity sends information of a first ML model to the AnLF functional entity. Correspondingly, the AnLF functional entity receives information of the first ML model from the MTLF functional entity.

[0210] The information of the first ML model can identify the first ML model, that is, the information of the first ML model can be used to indicate the first ML model, and it can be used to obtain the first ML model. The information of the first ML model can be identification information (model identifier) ​​of the first ML model, such as the ID (unique ML Model identifier) ​​of the first ML model, the uniform resource locator (URL) address of the first ML model, or the fully qualified domain name (FQDN) of the first ML model, and can also be other information that can identify the first ML model, without limitation.

[0211] The first ML model is an ML model trained by the MTLF functional entity, which can be a new ML model generated by the MTLF functional entity according to the data set, or a ML model retrained by the MTLF functional entity according to the generated ML model (such as the second ML model, the second ML model is described below), or a regenerated ML model. The first ML model can be one ML model, or multiple ML models. That is, the MTLF functional entity can send information of one or more ML models to the AnLF functional entity so that the AnLF functional entity obtains one or more ML models. Specifically, when the first ML model is one ML model, the information of the first ML model can be information of one ML model, such as the ID of an ML model; conversely, when the first ML model is multiple ML models, the information of the first ML model can be information of multiple ML models, and the multiple ML models correspond one to one with the information of the multiple ML models, for example: when the first ML model is ML model #1, ML model #2 and model #3, the information of the first ML model is information of ML model #1, information of ML model #2 and information of model #3.

[0212] After receiving the information of the first ML model, the AnLF functional entity can obtain the ML model corresponding to the information at the location where the ML model is stored according to the information of the first ML model, thereby obtaining the first ML model. For example, the AnLF functional entity carries the ID of the first ML model in the ML model management retrieval (Nadrf_MLModelManagement_Retrieval) service request and sends it to the ADRF functional entity. The ADRF functional entity can store the ID and model file of the first ML model at the URL or FQDN in the ADRF functional entity; the AnLF functional entity can obtain the first ML model according to the URL or the FQDN, such as obtaining the first ML model according to the Http protocol technology.

[0213] The trigger condition for the MTLF functional entity to send information about the first ML model to the AnLF functional entity may be that the AnLF functional entity requests the ML model from the MTLF functional entity, or that the MTLF functional entity generates a new ML model after sending the ML model to the AnLF functional entity, such as the MTLF functional entity retrains a new ML model based on the sent ML model, or regenerates a new ML model.

[0214] Specifically, before the MTLF functional entity sends the information of the first ML model to the AnLF functional entity (S501), the AnLF functional entity may send a request message for requesting the ML model to the MTLF functional entity, or the MTLF functional entity has sent the information of the ML model to the AnLF functional entity, and sends the updated ML model information, i.e., the first ML model information, in S501. The following describes the different situations.

[0215] Case 1:

[0216] Before the MTLF functional entity sends the information of the first ML model to the AnLF functional entity (S501), the communication method may further include: the AnLF functional entity sends a request message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the request message from the AnLF functional entity ( Figure 5 The AnLF functional entity receives information of a first ML model from the MTLF functional entity (S501), including: in response to the request message, the AnLF functional entity receives information of a first ML model from the MTLF functional entity.

[0217] The request message is used to request ML models. The request message may not carry the information of the data set. Alternatively, the request message may carry the information of the data set, and the number of data sets corresponding to the information of the data set is less than the number of ML models corresponding to the information of the ML model sent by the MTLF functional entity after receiving the request message. For example: the request message carries the information of data set #1, and the MTLF functional entity sends the information of 2 ML models after receiving the request message. At this time, the number of data sets is less than the number of ML models. For another example: the request message carries the information of data set #11 and the information of data set #22, that is, the request message carries the information of 2 data sets, and the MTLF functional entity sends the information of 3 ML models after receiving the request message. At this time, the number of data sets is less than the number of ML models.

[0218] Sending the request message may be implemented through an ML model provision subscription (MLModelProvision_Subscribe) service operation, or other operations, which is not limited in the embodiments of the present application.

[0219] In an embodiment of the present application, the AnLF functional entity may send a request message for requesting an ML model to the MTLF functional entity. After receiving the request message, the MTLF functional entity sends information of a first ML model to the AnLF functional entity, that is, provides an ML model for the AnLF functional entity, so that the AnLF functional entity obtains the first ML model based on the information of the first ML model.

[0220] It can be understood that the AnLF functional entity and the MTLF functional entity can preset or predefine a data set by protocol, so that after the AnLF functional entity sends a request message to the MTLF functional entity, the MTLF functional entity can generate a first ML model based on the data set and send it to the AnLF functional entity.

[0221] Case 2:

[0222] Before the MTLF functional entity sends the information of the first ML model to the AnLF functional entity (S501), the communication method may further include: the AnLF functional entity sends a request message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the request message from the AnLF functional entity ( Figure 5 In S500b1). When the MTLF functional entity receives the request message, the MTLF functional entity sends the information of the second ML model to the AnLF functional entity. Accordingly, in response to the request message, the AnLF functional entity receives the information of the second ML model from the MTLF functional entity ( Figure 5 (S500b2).

[0223] The request message is used to request the ML model and carries information about the second data set. For details, please refer to the above-mentioned related introduction, which will not be repeated here.

[0224] The information of the second data set can be used to identify the second data set, and the location of the second data set, that is, the location where the second data set is stored. The information of the second data set may include identification information (DateSetTag) of the second data set and / or identification information of the second ADRF functional entity. The identification information of the second data set can be the ID of the second data set, or other information that can identify the second data set, without limitation. The identification information of the second ADRF functional entity can be the ID of the second ADRF functional entity, or other information that can identify the second ADRF functional entity, without limitation.

[0225] It can be understood that when there is one ADRF functional entity in the communication system, each data set is stored in the ADRF functional entity. In this case, the information of the second data set includes the identification information of the second data set, that is, there is no need to indicate the correspondence between the second data set and the ADRF functional entity, that is, the correspondence between each data set and the ADRF functional entity can be preset or predefined by protocol. When there are multiple ADRF functional entities in the communication system, it is necessary to indicate the correspondence between the second data set and the second ADRF functional entity storing the second data set. In this case, the information of the second data set includes the identification information of the second data set and the identification information of the second ADRF functional entity.

[0226] The second data set is used to store the relevant data of the ML model (second ML model) sent by the MTLF functional entity after receiving the request message. And the second data set can be established by the AnLF functional entity in the ADRF functional entity (such as the second ADRF functional entity). It can be understood that the relevant data can be sent by the AnLF functional entity to the ADRF functional entity and stored in the second data set by the ADRF functional entity.

[0227] The relevant data of the second ML model may include at least one of the following: input data of the second ML model, predicted data, true value data at the time of prediction, and deviation data at the time of prediction. Among them, the predicted data is the output result of the second ML model. The true value data at the time of prediction is the actual result for the input data of the second ML model, not the predicted result. In other words, if the predicted result is consistent with the actual result, it can be shown that the predicted result is correct. The deviation data at the time of prediction is the data in which the predicted result deviates, such as the predicted result of the model for a certain input data and the true value result corresponding to the input data are inconsistent. The standard of consistency here can be complete consistency, such as numerical equality, or it can be a certain predefined standard, that is, the predicted result meets specific requirements, such as: the difference between the predicted result and the true value result is within a certain range, which can be understood as consistent. It can be understood that the input data, predicted data, and true value data at the time of prediction can also be referred to as inference data. That is, when the relevant data of the second ML model includes the input data, predicted data, and true value data at the time of prediction of the second ML model, the relevant data of the second ML model can also be referred to as inference data of the second ML model.

[0228] The second ML model is an ML model trained by the MTLF functional entity, and the second ML model is an ML model sent before the first ML model. It can be understood that the first ML model can be an ML model retrained based on the second ML model, or the first ML model can be a regenerated ML model.

[0229] The information of the second ML model can identify the second ML model, that is, the information of the second ML model can be used to indicate the second ML model, and it can be used to obtain the second ML model. The information of the second ML model can be identification information of the second ML model, such as the ID of the second ML model, the URL address of the second ML model, or the FQDN of the second ML model, and can also be other information that can represent the second ML model, without limitation.

[0230] In the embodiment of the present application, the information of the second data set carried in the request message can be used to indicate that the second data set corresponds to the ML model sent by the MTLF functional entity after receiving the request message, that is, the information of the second data set can be used to indicate that the relevant data of the ML model sent by the MTLF functional entity after receiving the request message will be stored in the second data set. It can be understood that in case 2, the number of data sets corresponding to the information of the second data set is greater than or equal to the number of ML models corresponding to the information of the ML model sent by the MTLF functional entity after receiving the request message, so that the ML models provided by the MTLF functional entity all have data sets corresponding to the ML model. It can also be understood that the correspondence between the information of the data set carried in the request message and the information of the ML model subsequently sent by the MTLF functional entity can be preset or predefined by protocol, such as matching the information of the nth data set in the request message with the information of the nth ML model subsequently sent by the MTLF functional entity, where n is a positive integer. In this way, the AnLF functional entity and the MTLF functional entity determine the correspondence between each data set and each ML model based on the information of the second data set and the information of the ML model subsequently sent by the MTLF functional entity. In addition, the correspondence between the information of the data set carried in the request message and the information of the ML model subsequently sent by the MTLF functional entity can also be set by other methods. For example, when the MTLF functional entity sends the information of the ML model to AnLF, it indicates the correspondence between the information of each model and the data set carried in the request message, that is, the MTLF functional entity determines which data set each ML model corresponds to without any restriction.

[0231] It can be seen that when the AnLF functional entity requests the ML model from the MTLF functional entity for the first time, the information of the data set can be carried in the request message, that is, the MTLF functional entity is informed that the relevant data of the requested ML model will be stored in the data set, so that the MTLF functional entity can subsequently obtain the relevant data of the ML model from the data set and use the relevant data. In this way, the AnLF functional entity does not need to send the corresponding relationship between the ML model and the data set to the MTLF functional entity after receiving the ML model, thereby reducing communication overhead.

[0232] Optionally, after the MTLF functional entity sends information about the second ML model to the AnLF functional entity, the communication method may further include: the MTLF functional entity evaluates the second ML model based on the second data set ( Figure 5 In S503). That is, the MTLF functional entity can use the second data set to evaluate the second ML model. Because the second data set stores relevant data of the second ML model and does not store relevant data of other ML models, the second ML model is evaluated using the data in the second data set, such as accuracy evaluation and robustness evaluation, to improve the reliability of the evaluation of the second ML model.

[0233] Based on the above situation 1 and situation 2, it can be seen that when the AnLF functional entity requests the MTLF functional entity for the ML model, it can choose to carry the information of the data set in the request message, or not to carry the information of the data set. That is, when the request message does not carry the information of the data set, the AnLF functional entity can send the corresponding relationship between the ML model and the data set to the MTLF functional entity after receiving the ML model; when the request message carries the information of the data set, and the number of data sets corresponding to the information of the data set is less than the number of ML models corresponding to the information of the ML model sent by the MTLF functional entity after receiving the request message, the AnLF functional entity can send the corresponding relationship between the ML model and the data set to the MTLF functional entity after receiving the ML model; when the request message carries the information of the data set, and the number of data sets corresponding to the information of the data set is greater than or equal to the number of ML models corresponding to the information of the ML model sent by the MTLF functional entity after receiving the request message, the AnLF functional entity does not need to send the corresponding relationship between the ML model and the data set to the MTLF functional entity after receiving the ML model.

[0234] S502: The AnLF functional entity sends a first message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the first message from the AnLF functional entity.

[0235] The first message carries information about the first ML model and information about the first data set, and can be used to indicate the correspondence between the first ML model and the first data set. The first message can be preconfigured or predefined by the protocol, or can be dynamically generated according to actual conditions, which is not limited in the embodiments of the present application. And sending the first message can be implemented through the ML Model Provision_Subscribe_Update service operation, or other operations, which is not limited in the embodiments of the present application.

[0236] The information of the first data set can be used to identify the first data set, and the location of the first data set. The information of the first data set may include identification information (DateSetTag) of the first data set and / or identification information of the first ADRF functional entity. The identification information of the first data set may be the ID of the first data set, or other information that can identify the first data set, without limitation. The identification information of the first ADRF functional entity may be the ID of the first ADRF functional entity, or other information that can identify the first ADRF functional entity, without limitation.

[0237] It can be understood that when there is one ADRF functional entity in the communication system, each data set is stored in the ADRF functional entity. In this case, the information of the first data set includes the identification information of the first data set, that is, there is no need to indicate the correspondence between the first data set and the ADRF functional entity, that is, the correspondence between each data set and the ADRF functional entity can be preset or predefined by protocol. When there are multiple ADRF functional entities in the communication system, it is necessary to indicate the correspondence between the first data set and the first ADRF functional entity storing the first data set. In this case, the information of the first data set includes the identification information of the first data set and the identification information of the first ADRF functional entity.

[0238] The first data set is used to store the relevant data of the first ML model, which can be established by the AnLF functional entity in the ADRF functional entity (such as the first ADRF functional entity). It can be understood that the relevant data of the first ML model can be sent by the AnLF functional entity to the ADRF functional entity and stored in the first data set by the ADRF functional entity. The first data set corresponds to the first ML model, and the first data set can be one or more data sets. Normally, one ML model corresponds to one dataset, that is, when the first ML model is one ML model, the first dataset is one dataset, and the ML model corresponds to the dataset; when the first ML model is multiple ML models, the first dataset is multiple datasets, and the multiple ML models correspond one-to-one to the multiple datasets, for example: the first dataset is dataset #11, and the first ML model is ML model #11, then dataset #11 corresponds to ML model #11, that is, the relevant data of ML model #11 is stored in dataset #11; for another example: the first dataset is dataset #a, dataset #b, and dataset #c, and the first ML model is ML model #A, ML model #B, and ML model #C, then dataset #a corresponds to ML model #A, dataset #b corresponds to ML model #B, and dataset #c corresponds to ML model #C, that is, the relevant data of ML model #A is stored in dataset #a, the relevant data of ML model #B is stored in dataset #b, and the relevant data of ML model #C is stored in dataset #c. It can be understood that an ML model can also correspond to multiple data sets, that is, the relevant data of an ML model can be stored in multiple data sets respectively. In this case, there may be one ML model corresponding to multiple data sets, or there may be multiple ML models corresponding to multiple data sets. The corresponding settings can be made according to the actual situation without any restrictions.

[0239] It can be seen that the first data set corresponds to the first ML model, the information of the first data set corresponds to the first data set, and the information of the first ML model corresponds to the first ML model, and therefore, the information of the first data set corresponds to the information of the first ML model. When the information of the first data set is the information of multiple data sets, and the information of the first ML model is the information of multiple ML models, the information of the multiple data sets corresponds to the information of the multiple ML models. In addition, the information of multiple data sets can correspond to the information of multiple ML models one by one, or the information of at least two data sets can correspond to the information of one ML model, which is not limited in the embodiments of the present application.

[0240] For example, the information of the first data set is information of multiple data sets, and the information of the multiple data sets is respectively: ID of data set #1, ID of data set #2, and ID of data set #3; the information of the first ML model is information of multiple ML models, and the information of the multiple ML models is respectively: ID of ML model #1, ID of ML model #2, and ID of ML model #3. Among them, the ID of data set #1 corresponds to the ID of ML model #1, the ID of data set #2 corresponds to the ID of ML model #2, and the ID of data set #3 corresponds to the ID of ML model #3, that is, the information of multiple data sets corresponds to the information of multiple ML models one by one.

[0241] For another example, the information of the first data set is information of multiple data sets, and the information of the multiple data sets is respectively: ID of data set #11, ID of data set #22, and ID of data set #33; the information of the first ML model is information of multiple ML models, and the information of the multiple ML models is respectively: ID of ML model #11 and ID of ML model #22. Among them, the ID of data set #11 corresponds to the ID of ML model #11, and the ID of data set #22 and the ID of data set #33 both correspond to the ID of ML model #22, that is, the relevant data of ML model #22 can be stored in data set #22 and data set #33, that is, the information of at least two data sets corresponds to the information of one ML model.

[0242] The relevant data of the first ML model may include at least one of the following: input data of the first ML model, prediction data, true value data during prediction, and deviation data during prediction. For each data, reference may be made to the relevant introduction at "S501" above, and no further description is given here. It can be understood that when the relevant data of the first ML model includes the input data of the first ML model, prediction data, and true value data during prediction, the relevant data of the first ML model may also be referred to as the inference data of the first ML model.

[0243] In an embodiment of the present application, when the first ML model is one ML model, by carrying the information of the first ML model and the information of the first data set in the first message, such as placing the information of the first ML model and the information of the first data set in a tuple, it can be indicated that the information of the first ML model corresponds to the information of the first data set, that is, the relevant data of the first ML model is stored in the first data set. When the first ML model is a plurality of ML models, the information of each ML model and the information of the data set corresponding to the ML model can be placed in the same tuple in the first message, that is, the correspondence between different ML models and data sets is distinguished by different tuples, for example: [Model ID1, DataSetTag1] can indicate that the first ML model corresponds to the first data set, that is, the AnLF functional entity can store the relevant data of the first ML model in the first data set; [Model ID2, DataSetTag2, DataSetTag3] can indicate that the second ML model corresponds to the second data set and the third data set, that is, the AnLF functional entity can store the relevant data of the second ML model in the second data set and the third data set. In other words, by carrying the information of the first data set and the information of the first ML model in the first message, the correspondence between the first data set and the first ML model can be indicated. It can be understood that the above-mentioned method of placing the information of the first data set and the information of the first ML model in the form of at least one tuple is only an example, and there may be other ways to implement it, which is not limited in the embodiments of the present application.

[0244] After receiving the first message, the MTLF functional entity can determine the first data set corresponding to the first ML model based on the information carried in the first message, that is, it knows that the AnLF functional entity stores the relevant data of the first ML model in the first data set. In this way, the MTLF functional entity can use the data in the first data set when evaluating the first ML model or retraining the model.

[0245] In summary, the MTLF functional entity sends the information of the first ML model to the AnLF functional entity, so that the AnLF functional entity obtains the first ML model according to the information of the first ML model, that is, the MTLF functional entity provides the first ML model to the AnLF functional entity. After receiving the information of the first ML model, the AnLF functional entity can set the correspondence between the first ML model and the first data set according to the information of the first ML model, that is, save the relevant data of the first ML model in the first data set. And, the correspondence is sent to the MTLF functional entity through a first message carrying the information of the first ML model and the information of the first data set. That is to say, when the AnLF functional entity receives the ML model, the AnLF functional entity can send a set of tuples to the MTLF functional entity, which includes the identification information of the ML model, the identification information of the data set (DataSetTag) and / or the identification information of the ADRF (ADRFID), and use the ML model supply subscription (Nnwdaf_MLModelProvision_Subscribe) service operation to indicate the mapping relationship between the identification information of the data set and the identification information of the ML model. It can be understood that other service operations can also be used to implement it without limitation. Accordingly, after receiving the tuple, the MTLF functional entity can determine that the data corresponding to the identification information of the data set in the tuple is the relevant data of the model corresponding to the identification information of the ML model, that is, the MTLF functional entity can obtain the relevant data of the ML model from the data set corresponding to the identification information of the data set. In this way, the MTLF functional entity can evaluate the first ML model using the data of the first data set instead of using the data set containing data of multiple ML models to evaluate the first ML model, thereby improving the reliability of the evaluation of the first ML model.

[0246] It can be understood that in the embodiment of the present application, the first data set is used to store the data of the first ML model, that is, the first data set can store the data of the first ML model. Then, the information of the first ML model (ML ModelIdentifier) ​​can also represent the model corresponding to the data corresponding to the identifier (DataSetTag) of the first data set. In the example, the corresponding model is the first ML model.

[0247] It can also be understood that in the embodiment of the present application, the AnLF functional entity sends a request message (such as Figure 5 After S500a), the MTLF functional entity may be triggered to send information of the first ML model to the AnLF functional entity (eg Figure 5In S501), the first ML model is provided to the AnLF functional entity; when the AnLF functional entity receives the information of the first ML model from the MTLF functional entity, the AnLF functional entity can be triggered to send the information of the first ML model and the information of the first data set to the MTLF functional entity (such as Figure 5 In S502), the corresponding relationship between the first ML model and the first data set is provided to the MTLF functional entity. In other words, these three steps (such as Figure 5 S500a, S501, and S502) may be strongly associated. The request message is a message that does not carry the information of the data set, or the request message is a message that carries the information of the data set, but the number of data sets corresponding to the information of the data set is less than the number of ML models corresponding to the information of the ML model sent by the MTLF functional entity after receiving the request message.

[0248] Optionally, after the MTLF functional entity receives the first message from the AnLF functional entity, the communication method may further include: the MTLF functional entity evaluates the first ML model ( Figure 5 In S503). That is, the MTLF functional entity can use the first data set to evaluate the first ML model. Because the first data set is used to store relevant data of the first ML model and does not store relevant data of other ML models, the first ML model is evaluated using the data in the first data set, such as accuracy evaluation and robustness evaluation, to improve the reliability of the evaluation of the first ML model.

[0249] Optionally, after the AnLF functional entity obtains the first ML model, the MTLF functional entity may request the AnLF functional entity to monitor the analysis accuracy of the first ML model ( Figure 5 In the process of the AnLF functional entity monitoring the ML model analysis accuracy of the first ML model, the MTLF functional entity or the AnLF functional entity may set a third data set for storing relevant data of the first ML model (such as data that deviates during prediction), and send information of the third data set to the AnLF functional entity or the MTLF functional entity, so that the MTLF functional entity can use the data of the third data set when updating the ML model. The following describes the methods of setting and sending the third data set respectively.

[0250] Mode 1: The MTLF functional entity sets and sends the third data set.

[0251] In this case, the communication method may further include: when the MTLF functional entity receives a registration request for ML model monitoring, the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity. Accordingly, the AnLF functional entity receives the request for monitoring the ML model from the MTLF functional entity ( Figure 5 (S505).

[0252] The request for monitoring the ML model carries information about the first ML model and information about the third data set to indicate the corresponding relationship between the first ML model and the third data set. Sending the request for monitoring the ML model can be implemented through an ML model monitoring subscription request (Nnwdaf_MLModelMonitor_Subscibe request) service operation, or other operations, which is not limited in the embodiments of the present application.

[0253] The information of the third data set can be used to identify the third data set, and the storage location of the third data set. The information of the third data set may include identification information of the third data set and / or identification information of the third ADRF functional entity. The identification information of the third data set can be the ID of the third data set, or other information that can identify the third data set, without limitation. The identification information of the third ADRF functional entity can be the ID of the third ADRF functional entity, or other information that can identify the third ADRF functional entity, without limitation.

[0254] It can be understood that when there is one ADRF functional entity in the communication system, each data set is stored in the ADRF functional entity. In this case, the information of the third data set includes the identification information of the third data set, that is, there is no need to indicate the correspondence between the third data set and the ADRF functional entity, that is, the correspondence between each data set and the ADRF functional entity can be preset or predefined by protocol. When there are multiple ADRF functional entities in the communication system, it is necessary to indicate the correspondence between the third data set and the third ADRF functional entity storing the third data set. In this case, the information of the third data set includes the identification information of the third data set and the identification information of the third ADRF functional entity.

[0255] The third data set is used to store the relevant data of the first ML model, and the third data set can be established by the MTLF functional entity in the ADRF functional entity (such as the third ADRF functional entity). It can be understood that the AnLF functional entity can store the relevant data of the first ML model in the third data set.

[0256] The relevant data of the first ML model includes deviation data during prediction, and the deviation data refers to data with deviation in the prediction result. For example, each group of data of the first ML model includes 100 data, and there are 3 groups of data, namely data group #1, data group #2 and data group #3. There are 20 deviated data in data group #1, 10 deviated data in data group #2, and 15 deviated data in data group #3. At this time, the deviation data during prediction of the first ML model is 20 deviated data in data group #1, 10 deviated data in data group #2, and 15 deviated data in data group #3, totaling 45 deviated data.

[0257] It can be understood that in the prior art, when the analysis accuracy of the ML model is insufficient, the network data when the deviation occurs sent by the AnLF functional entity to the MTLF functional entity includes data that has not deviated, and not all deviation data, for example: each group of data of the first ML model includes 100 data, and the accuracy of the first group of data is 90%, which is greater than the accuracy threshold (80%), then the first group of data is determined not to be network data when the deviation occurs; the accuracy of the second group of data is 70%, which is less than the accuracy threshold (80%), then the second group of data is determined to be network data when the deviation occurs, that is, the second group of data is sent to the MTLF functional entity. It can be seen that the network data when the deviation occurs in the prior art is inaccurate, that is, it does not include all deviation data, and includes normal data. Therefore, saving the deviation data when predicting in the third data set can guide the MTLF functional entity to subsequently update the ML model and improve the prediction accuracy of the updated ML model.

[0258] It can be seen that in method 1, the MTLF functional entity can set a third data set in the third ADRF functional entity, and when requesting the AnLF functional entity to monitor the analysis accuracy of the ML model, the MTLF functional entity sends the information of the first ML model and the information of the third data set to the AnLF functional entity, so that the AnLF functional entity saves the relevant data of the first ML model in the third data set for use by the MTLF functional entity when updating the ML model.

[0259] Mode 2: The AnLF functional entity sets and sends the third data set according to the request of the monitoring ML model.

[0260] In this case, the communication method may further include: the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity. Accordingly, the AnLF functional entity receives the request for monitoring the ML model from the MTLF functional entity. In response to the request for monitoring the ML model, the AnLF functional entity sends information of the third data set to the MTLF functional entity. Accordingly, the MTLF functional entity receives information of the third data set from the AnLF functional entity ( Figure 5 In other words, the information of the third data set is carried in the response for ML model monitoring.

[0261] The request for monitoring the ML model carries information of the first ML model, and the information of the first ML model is used to indicate the analysis accuracy of the first ML model. The request for monitoring the ML model and the third data set can refer to the relevant introduction in the aforementioned "Method 1", which will not be repeated here.

[0262] It can be seen that in method 2, after receiving the request for monitoring the ML model from the MTLF functional entity, the AnLF functional entity can set a third data set in the ADRF functional entity (such as the third ADRF functional entity), and send the correspondence between the first ML model and the third data set to the MTLF functional entity, so that the MTLF functional entity can use the data in the third data set when updating the ML model.

[0263] Mode 3: The AnLF functional entity sets the third data set and sends information of the third data set to the MTLF functional entity through a notification request message.

[0264] In this case, the communication method may further include: the AnLF functional entity monitors the analysis accuracy of the first ML model; when the analysis accuracy of the first ML model is insufficient, the AnLF functional entity sends a notification request message to the MTLF functional entity. Accordingly, the MTLF functional entity receives the notification request message from the AnLF functional entity ( Figure 5 (S509).

[0265] The analysis accuracy of the first ML model can represent the accuracy of the prediction results using the first ML model. The insufficient analysis accuracy of the first ML model can be determined by the accuracy of the prediction results of the first ML model and a preset threshold, that is, when the accuracy of the prediction results of the first ML model is less than the preset threshold, it can be determined that the analysis accuracy of the first ML model is insufficient. For example, the accuracy of the prediction results of the first ML model is 80%, which is less than the threshold (90%). At this time, the analysis accuracy of the first ML model is insufficient. It can be understood that the way in which the AnLF functional entity monitors the analysis accuracy of the first ML model and the way in which it determines whether the analysis accuracy of the first ML model is insufficient can reuse the methods of the prior art, which will not be repeated here.

[0266] The notification request message carries information of the third data set to indicate the correspondence between the first ML model and the third data set. Sending the notification request message can be implemented through an ML model monitoring notification request (Nnwdaf_MLModelMonitoring_Notify request) service operation, or other operations, which is not limited in the embodiment of the present application.

[0267] For the third data set, reference may be made to the related introduction in the aforementioned “Method 1” and will not be repeated here.

[0268] It can be seen that in method 2, after receiving the request for monitoring the ML model from the MTLF functional entity, the AnLF functional entity may temporarily not send the information of the third data set to the MTLF functional entity. Instead, after monitoring the analysis accuracy of the first ML model, when the analysis accuracy of the first ML model is insufficient and a notification request message is sent to the MTLF functional entity, the information of the third data set is sent to indicate that the relevant information of the first ML model is saved in the third data set.

[0269] It can be understood that in Mode 1 and Mode 2, when the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity, it will carry the information of the first ML model to indicate the accuracy of monitoring and analyzing the first ML model. Therefore, when the AnLF functional entity sends the information of the third data set to the MTLF functional entity, it is not necessary to carry the information of the first ML model, that is, the correspondence between the third data set and the first ML model can be indicated by sending the information of the third data set.

[0270] In addition, the AnLF functional entity monitors the other steps of the ML model analysis accuracy of the first ML model ( Figure 5 For S504, S507-S508, S5010-S5013), please refer to the relevant introduction of "3. Process of monitoring ML model analysis accuracy" mentioned above, which will not be repeated here.

[0271] It can also be understood that after the AnLF functional entity obtains the first ML model, the MTLF functional entity can also monitor the analysis accuracy of the first ML model. That is, in this case, there is no need to perform the above-mentioned steps (S504-S5013) of monitoring the ML model analysis accuracy of the first ML model by the AnLF functional entity.

[0272] Figure 6 Schematic diagram of the communication method provided in the embodiment of the present application Figure 2 The communication method is applicable to the above communication system and involves the interaction between the AnLF functional entity and the MTLF functional entity.

[0273] Specifically, Figure 6 As shown, the process of the communication method is as follows:

[0274] S601: AnLF functional entity sends a request message to MTLF functional entity. Correspondingly, MTLF functional entity receives the request message from AnLF functional entity.

[0275] The request message is used to request the ML model. For details, please refer to the relevant introduction in the aforementioned "S501 Case 1", which will not be repeated here.

[0276] S602: The MTLF functional entity sends a first message to the AnLF functional entity. Correspondingly, the AnLF functional entity receives the first message from the MTLF functional entity.

[0277] The first message carries information about the first ML model and information about the first data set. The information about the first ML model is used to indicate the first ML model. The information about the first ML model and the first model can refer to the relevant introduction at the aforementioned "S501", which will not be repeated here. The first data set is used to save the relevant data of the first ML model, and the first data set is established by the MTLF functional entity in the ADRF functional entity (such as the first ADRF functional entity). The information of the first data set is used to identify the first data set. The first data set, the information of the first data set and the related information can refer to the relevant introduction at the aforementioned "S502", which will not be repeated here.

[0278] The information of the first ML model and the information of the first data set carried in the first message can indicate the corresponding relationship between the first ML model and the first data set, that is, instruct the AnLF functional entity to save the relevant data of the first ML model into the first data set. In this way, the MTLF functional entity can evaluate the first ML model based on the data (the relevant data of the first ML model) saved in the first data set, thereby improving the reliability of the evaluation of the first ML model.

[0279] Optionally, before the MTLF functional entity sends the first message (S602) to the AnLF functional entity, the communication method may further include: the MTLF functional entity sends a request to the first ADRF functional entity to establish a first data set in the first ADRF functional entity. Correspondingly, the first ADRF functional entity receives the request from the MTLF functional entity to establish the first data set in the first ADRF functional entity.

[0280] The request for establishing the first data set in the first ADRF functional entity may carry information about the first data set to instruct the first ADRF functional entity to establish the first data set based on the information about the first data set. In other words, the first data set may be established by the MTLF functional entity. It is understood that when the first data set is a plurality of data sets, the MTLF functional entity may establish a plurality of data sets in the first ADRF functional entity, i.e., information about a plurality of ML models may be carried in the request for establishing the first data set in the first ADRF functional entity.

[0281] Optionally, after the AnLF functional entity receives the first message from the MTLF functional entity (S602), the communication method may further include: the AnLF functional entity sends the relevant data of the first ML model and the information of the first data set to the first ADRF functional entity based on the information of the first data set. Correspondingly, the first ADRF functional entity receives the relevant data of the first ML model and the information of the first data set from the AnLF functional entity.

[0282] It can be seen that when the AnLF functional entity saves the relevant data of the first ML model, the relevant data and the information of the first data set can be sent to the first ADRF functional entity to instruct the first ADRF functional entity to store the relevant data in the first data set corresponding to the information of the first data set. In this case, the first ADRF functional entity can store the relevant data of the first ML model according to the information of the first data set.

[0283] Optionally, after the MTLF functional entity sends the first message to the AnLF functional entity (S602), the communication method may further include: the MTLF functional entity evaluates the first ML model based on the data in the first data set. That is, the MTLF functional entity may use the data in the first data set to evaluate the first ML model. Specifically, after the AnLF functional entity saves the relevant data of the first ML model in the first ADRF functional entity, the MTLF functional entity may obtain the relevant data of the first ML model from the first ADRF functional entity, such as: the MTLF functional entity sends a request to the first ADRF functional entity for obtaining the relevant data of the first ML model, and the request may carry information of the first data set to instruct the first ADRF functional entity to return the data in the first data set (i.e., the relevant data of the first ML model) to the MTLF functional entity through the information of the first data set. After the MTLF functional entity obtains the relevant data of the first ML model, the relevant data may be used to evaluate the first ML model, such as accuracy evaluation, robustness evaluation, etc.

[0284] Further, after the MTLF functional entity evaluates the first ML model, the evaluation result can be sent to the AnLF functional entity. In addition, the MTLF functional entity can also send information about a new ML model (such as a second ML model) and information about a data set (such as a second data set) corresponding to the ML model to the AnLF functional entity, so that the AnLF functional entity acquires and uses the second ML model, and saves the relevant data of the second ML model in the second data set.

[0285] It can be understood that when multiple AnLF functional entities use the first ML model, multiple AnLF functional entities can save the relevant data of the first ML model they use in the first data set. In this case, after the first ADRF functional entity receives the relevant data of the first ML model and the information of the first data set sent by multiple AnLF functional entities, the first ADRF functional entity can record the source of each data, that is, which AnLF functional entity each data comes from, in order to distinguish each data.

[0286] For example, AnLF functional entity #1 and AnLF functional entity #2 both use ML model #aa, and the data of ML model #aa is stored in data set #aa. AnLF functional entity #1 sends the relevant data of ML model #aa (recorded as relevant data #1) and the information of data set #aa to the ADRF functional entity, and AnLF functional entity #2 also sends the relevant data of ML model #aa (recorded as relevant data #2) and the information of data set #aa to the ADRF functional entity. In this case, the ADRF functional entity can save relevant data #1 in data set #aa, and record relevant data #1 as data from AnLF functional entity #1, such as marking relevant data #1 with the label of AnLF functional entity #1; correspondingly, the ADRF functional entity can also save relevant data #2 in data set #aa, and record relevant data #2 as data from AnLF functional entity #2, such as marking relevant data #2 with the label of AnLF functional entity #2. In this way, entities from different AnLF functional entities can be distinguished.

[0287] It can also be understood that when multiple AnLF functional entities all use the first ML model, MTLF can also set different data sets for each of the multiple AnLF functional entities, that is, send information of different data sets to each AnLF functional entity, so that each AnLF functional entity saves the relevant data of the first ML model in a different data set. In this case, after the ADRF functional entity receives the relevant data of the first ML model sent by each AnLF functional entity, it is not necessary to record the source of each data, that is, to record which AnLF functional entity each data comes from.

[0288] In addition, the MTLF functional entity may also request the AnLF functional entity to monitor the analysis accuracy of the first ML model. In the process of the AnLF functional entity monitoring the ML model analysis accuracy of the first ML model, the MTLF functional entity or the AnLF functional entity may set a third data set for storing relevant data of the first ML model (such as data that deviates during prediction), and send the information of the third data set to the AnLF functional entity or the MTLF functional entity, so that the MTLF functional entity can use the data of the third data set when updating the ML model. For details, please refer to the relevant introduction of the aforementioned "S502 Method 1-Method 3", which will not be repeated here.

[0289] In summary, before providing the first ML model to the AnLF functional entity, the MTLF functional entity can set the correspondence between the first ML model and the first data set, that is, set the relevant information of the first ML model to be saved in the first data set. Furthermore, when the MTLF functional entity sends a first message carrying information about the first ML model to the AnLF functional entity, the MTLF functional entity can also carry information about the first data set in the first message, so that the AnLF functional entity can obtain the first ML model based on the information of the first ML model, and save the relevant information of the first ML model in the first data set according to the correspondence. In this way, the MTLF functional entity can obtain the relevant data of the first ML model from the first data set, so as to evaluate the first ML model based on the relevant data, thereby improving the reliability of the evaluation of the first ML model.

[0290] Scenario 1:

[0291] Figure 7 Schematic diagram of the communication method provided in the embodiment of the present application Figure 3 The communication method is applicable to the above communication system, and mainly involves a first AnLF functional entity, a second AnLF functional entity, a MTLF functional entity, and an ADRF functional entity. In scenario 1, the AnLF functional entity can request an ML model from the MTLF functional entity, and the MTLF functional entity can provide the ML model to the AnLF functional entity and indicate in which data set the relevant data of the ML model is stored.

[0292] like Figure 7 As shown, the process of the communication method is as follows:

[0293] S700: The MTLF functional entity sends a request for establishing a data set in the ADRF functional entity to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the request for establishing a data set in the ADRF functional entity from the MTLF functional entity.

[0294] The request for establishing a data set in the ADRF functional entity may carry information of the first data set and information of the second data set to instruct the ADRF functional entity to establish the first data set and the second data set. For details, please refer to the relevant introduction of the aforementioned "S602" and will not be repeated here.

[0295] S701a, the first AnLF functional entity sends a request message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the request message from the AnLF functional entity.

[0296] S701b, the second AnLF functional entity sends a request message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the request message from the AnLF functional entity.

[0297] S702a, the MTLF functional entity sends a first message to the first AnLF functional entity. Correspondingly, the first AnLF functional entity receives the first message from the MTLF functional entity.

[0298] The first message carries information about the first ML model and information about the first data set.

[0299] S702b, the MTLF functional entity sends a second message to the second AnLF functional entity. Correspondingly, the second AnLF functional entity receives the second message from the MTLF functional entity.

[0300] The second message carries information about the first ML model and information about the first data set.

[0301] S703a, the first AnLF functional entity sends the relevant data of the first ML model and the information of the first data set to the ADRF functional entity according to the information of the first data set. Correspondingly, the ADRF functional entity receives the relevant data of the first ML model and the information of the first data set from the first AnLF functional entity.

[0302] S703b, the second AnLF functional entity sends the relevant data of the first ML model and the information of the first data set to the ADRF functional entity according to the information of the second data set. Correspondingly, the ADRF functional entity receives the relevant data of the first ML model and the information of the first data set from the second AnLF functional entity.

[0303] S704, the ADRF functional entity records the source of the relevant data of the first ML model and saves the first data set.

[0304] It can be understood that the source of the relevant data of the first data set is the first AnLF functional entity or the second AnLF functional entity.

[0305] S705: The MTLF functional entity sends a request for obtaining relevant data of the first ML model to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the request for obtaining relevant data of the first ML model from the MTLF functional entity.

[0306] The request for obtaining relevant data of the first ML model may carry information of the first data set, so as to instruct the ADRF functional entity to return the data in the first data set, that is, the relevant data of the first ML model, to the MTLF functional entity through the information of the first data set.

[0307] S706: The ADRF functional entity sends the data in the first data set to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the data in the first data set from the ADRF functional entity.

[0308] S707, the MTLF functional entity evaluates the accuracy of the first ML model based on the data in the first data set.

[0309] The MTLF functional entity can reuse the method in the prior art to evaluate the accuracy of the first ML model, which will not be described in detail here.

[0310] It can be understood that after the MTLF functional entity evaluates the accuracy of the first ML model, it can obtain the accuracy information of the first ML model.

[0311] S708, the MTLF functional entity sends the accuracy information of the first ML model and the information of the first ML model to the first AnLF functional entity. Correspondingly, the first AnLF functional entity receives the accuracy information of the first ML model and the information of the first ML model from the MTLF functional entity.

[0312] It can be understood that the accuracy information of the first ML model and the information of the first ML model sent by the MTLF functional entity may indicate that the accuracy information is the accuracy information of the first ML model. And S708 may be implemented through an ML model provision notification (Nnwdaf_MLModelProvision_Notify) service operation.

[0313] S709, the MTLF functional entity sends information about the second ML model and information about the second data set to the first AnLF functional entity. Correspondingly, the first AnLF functional entity receives information about the second ML model and information about the third data set from the MTLF functional entity.

[0314] The information of the second ML model is used to obtain the second ML model, that is, the first AnLF functional entity can obtain the second ML model according to the information of the second ML model. The information of the second ML model and the information of the second data set can instruct the first AnLF functional entity to save the relevant data of the second ML model in the second data set.

[0315] The second ML model may be an ML model retrained by the MTLF functional entity based on the first ML model, or may be a regenerated ML model.

[0316] For the specific implementation of S700-S709, reference may be made to the related introduction of S601-S602 mentioned above, which will not be repeated here.

[0317] It can be understood that the above-mentioned S701a-S703a and S701b-S703b can have different orders, that is, the order in which the first AnLF functional entity and the second AnLF functional entity obtain the ML model and save the relevant data of the ML model can be set according to the actual situation, and the embodiment of the present application does not limit it. For example, after the first AnLF functional entity receives the first message, the second AnLF functional entity sends a request message again, that is, S701b is performed after S702a. For another example, the first AnLF functional entity and the second AnLF functional entity send request messages at the same time, that is, S701a and S701b are performed at the same time.

[0318] Figure 8 Schematic diagram of the communication method provided in the embodiment of the present application Figure 4 The communication method is applicable to the above communication system, and involves the interaction between the AnLF functional entity, the MTLF functional entity, and the ADRF functional entity.

[0319] Specifically, Figure 8 As shown, the process of the communication method is as follows:

[0320] S801, the AnLF functional entity sends a request message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the request message from the AnLF functional entity.

[0321] The request message is used to request the ML model and carries information of the first data set. For details, please refer to the relevant introduction at the aforementioned "S501", which will not be repeated here.

[0322] The first data set is used to store relevant data of multiple ML models, including the ML model sent by the MTLF functional entity after receiving the request message, such as the first ML model. For example, the first data set is used to store relevant data of three ML models, namely: relevant data of ML model #1, ML model #2 and ML model #3, and the first ML model is ML model #1. The information of the first data set is used to represent the first data set. The first data set and the information of the first data set can refer to the relevant introduction at the aforementioned "S501", which will not be repeated here.

[0323] By carrying the information of the first data set in the request message, it can be indicated that the AnLF functional entity stores the relevant data of the ML model subsequently received in the first data set. For the MTLF functional entity, it can obtain the relevant data of the ML model from the first data set.

[0324] Optionally, before the AnLF functional entity sends a request message to the MTLF functional entity, the communication method may further include: the AnLF functional entity establishes a first data set. For example, the AnLF functional entity may send a request to the ADRF functional entity to establish a first data set in the ADRF functional entity. Accordingly, the ADRF functional entity receives a request from the AnLF functional entity to establish a first data set in the ADRF functional entity. After receiving the request, the ADRF functional entity may establish the first data set.

[0325] It can be understood that for the ADRF functional entity, after establishing the first data set, the ADRF functional entity does not know what data the first data set is used to store. After subsequently receiving the data to be stored and the information of the first data set, it can store the data in the first data set according to the information of the first data set.

[0326] S802: The MTLF functional entity sends a first message to the AnLF functional entity. Correspondingly, the AnLF functional entity receives the first message from the MTLF functional entity.

[0327] The first message carries information about the first ML model, and the information about the first ML model is used to indicate the first ML model, and the information about the first ML model can be used to obtain the first ML model. For details, please refer to the relevant introduction at "S501" above, which will not be repeated here. That is, the MTLF functional entity can send the first ML model to the AnLF functional entity through the first message.

[0328] By sending the first message carrying the information of the first ML model to the AnLF functional entity, the AnLF functional entity can obtain the first ML model according to the information of the first ML model.

[0329] It is understandable that the first data set may also be established by the MTLF functional entity in the ADRF functional entity. In this case, the request message sent by the AnLF functional entity to the MTLF functional entity will not carry the information of the first data set, and the MTLF functional entity may carry the information of the first data set in the first message sent to the AnLF functional entity.

[0330] S803, the AnLF functional entity sends the relevant data of the first ML model, the information of the first ML model and the information of the first data set to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the relevant data of the first ML model, the information of the first ML model and the information of the first data set from the AnLF functional entity.

[0331] For the relevant data of the first ML model, please refer to the relevant introduction at "S502" mentioned above, which will not be repeated here.

[0332] The relevant data of the first ML model and the information of the first ML model sent by the AnLF functional entity may indicate that the relevant data is the relevant data of the first ML model; the information of the first data set sent by the AnLF functional entity may indicate that the relevant data is stored in the first data set.

[0333] S804: The ADRF functional entity stores relevant data of the first ML model according to information of the first ML model and information of the first data set.

[0334] After receiving the relevant data of the first ML model, the information of the first ML model and the information of the first data set, the ADRF functional entity can save the relevant data in the first data set and classify the relevant data as data belonging to the first ML model, that is, the ADRF functional entity can use additional ML model information (such as the ID of the ML model) to further identify the relevant data.

[0335] S805: The MTLF functional entity sends a request for obtaining relevant data of the first ML model to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the request for obtaining relevant data of the first ML model from the MTLF functional entity.

[0336] The request for obtaining relevant data of the first ML model carries information of the first ML model and information of the first data set, so that after receiving the request, the ADRF functional entity can determine the location where the data requested by the MTLF functional entity is stored based on the information of the first data set, and filter out relevant data belonging to the first ML model from the first data set based on the information of the first ML model.

[0337] It can be understood that since the first data set stores data of multiple ML models, when requesting relevant data of a certain ML model, in addition to carrying the information of the data set in the message, it is also necessary to carry the information of the ML model, so that the relevant data of the ML model can be obtained.

[0338] S806: The ADRF functional entity sends the relevant data of the first ML model to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the relevant data of the first ML model from the ADRF functional entity.

[0339] That is, after the ADRF functional entity screens out relevant data of the first ML model according to the information of the first ML model and the information of the first data set, the ADRF functional entity may send the relevant data to the MTLF functional entity.

[0340] Optionally, after the MTLF functional entity acquires relevant data of the first ML model, the communication method may further include: the MTLF functional entity retrains or regenerates the second ML model based on the relevant data.

[0341] The MTLF functional entity can reuse the methods in the prior art to retrain or regenerate the second ML model based on the relevant data of the first ML model, which will not be described in detail here.

[0342] Furthermore, after the MTLF functional entity generates the second ML model, the communication method may further include: the MTLF functional entity sends information about the second ML model to the AnLF functional entity. Accordingly, the AnLF functional entity receives information about the second ML model from the MTLF functional entity. In this way, the AnLF functional entity can obtain the second ML model, that is, the latest ML model, based on the information about the second ML model, so that the AnLF functional entity can use the latest ML model, thereby improving the accuracy of prediction when the AnLF functional entity uses the ML model.

[0343] It can be understood that the MTLF functional entity can also request the AnLF functional entity to monitor the analysis accuracy of the first ML model. In the process of the AnLF functional entity monitoring the ML model analysis accuracy of the first ML model, the MTLF functional entity or the AnLF functional entity can set a third data set for storing relevant data of the first ML model (such as data that deviates during prediction), and send the information of the third data set to the AnLF functional entity or the MTLF functional entity, so that the MTLF functional entity can use the data of the third data set when updating the ML model. For details, please refer to the relevant introduction of the aforementioned "S502 Mode 1-Mode 3", which will not be repeated here.

[0344] In summary, the relevant data of each ML model can be stored in a data set, and by labeling the relevant data with the corresponding ML model label, it is possible to distinguish which ML model each relevant data belongs to. In other words, the MTLF functional entity can obtain the data set label from the received request message, that is, it knows that the data of each ML model is stored in the data set corresponding to the data set label; and sends a request for obtaining the relevant data of the first ML model carrying the information of the first ML model and the information of the first data set to the ADRF functional entity, so that the ADRF functional entity can find the data set storing the relevant data according to the information of the first data set, and filter out the relevant data of the first ML model from the first data set according to the information of the first ML model, and send the relevant data to the MTLF functional entity. In this way, the MTLF functional entity can evaluate the first ML model based on the relevant data, thereby improving the reliability of the evaluation of the first ML model.

[0345] It can be understood that the above two embodiments ( Figure 5 The embodiments shown and Figure 6 The difference between the embodiments shown in the preceding claims and the embodiments of the present application is that: the two aforementioned embodiments allocate different data sets to different ML models, and distinguish the relevant data of each ML model by saving the relevant data of different ML models in different data sets respectively; the embodiments of the present application save the relevant data of multiple ML models in one data set, and distinguish the relevant data of each ML model by the information of the ML model on each data in the data set. The definition of each information, the sending and receiving of messages, the evaluation of ML models, etc. in these three embodiments are similar implementation principles, so they can be understood by reference and will not be repeated here. In addition, these three embodiments can be used in combination with each other without limitation.

[0346] Scenario 2:

[0347] Fig. 9 Schematic diagram of the communication method provided in the embodiment of the present application Figure 5 The communication method is applicable to the above communication system, and mainly involves AnLF functional entity, MTLF functional entity and ADRF functional entity. In scenario 5, the relevant data of each ML model is stored in the same data set, and the MTLF functional entity can use the information of the ML model and the information of the data set to obtain the relevant data of the ML model from the ADRF functional entity.

[0348] like Fig. 9 As shown, the process of the communication method is as follows:

[0349] S901, the AnLF functional entity sends a request message to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the request message from the AnLF functional entity.

[0350] The request message is used to request the ML model and carries information of the first data set. For details, please refer to the relevant introduction at the aforementioned "S801", which will not be repeated here.

[0351] S902: The MTLF functional entity sends a first message to the AnLF functional entity. Correspondingly, the AnLF functional entity receives the first message from the MTLF functional entity.

[0352] The first message carries information about the first ML model and information about the second ML model, so that the AnLF functional entity obtains the first ML model and the second ML model at corresponding positions according to the information about the first ML model and the information about the second ML model.

[0353] S903a, the AnLF functional entity sends the relevant data of the first ML model, the information of the first ML model and the information of the first data set to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the relevant data of the first ML model, the information of the first ML model and the information of the first data set from the AnLF functional entity.

[0354] S903b, the AnLF functional entity sends the relevant data of the second ML model, the information of the second ML model and the information of the first data set to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the relevant data of the second ML model, the information of the second ML model and the information of the first data set from the AnLF functional entity.

[0355] S904, the ADRF functional entity stores relevant data of the first ML model according to information of the first ML model and information of the first data set, and stores relevant data of the second ML model according to information of the second ML model and information of the first data set.

[0356] When the ADRF functional entity stores relevant data of the first ML model (denoted as relevant data #a) and relevant data of the second ML model (denoted as relevant data #b), the ML model ID can be used to further identify each relevant data, such as using the first ML model to identify relevant data #a and using the second ML model to identify relevant data #b.

[0357] S905: The MTLF functional entity sends a request for obtaining relevant data of the first ML model to the ADRF functional entity. Correspondingly, the ADRF functional entity receives the request for obtaining relevant data of the first ML model from the MTLF functional entity.

[0358] The request for obtaining relevant data of the first ML model carries information of the first data set and information of the first ML model, that is, the MTLF functional entity requests the relevant data of the first ML model from the ADRF functional entity.

[0359] S906: The ADRF functional entity sends the relevant data of the first ML model to the MTLF functional entity. Correspondingly, the MTLF functional entity receives the relevant data of the first ML model from the ADRF functional entity.

[0360] S907, the MTLF functional entity retrains or regenerates a third ML model based on relevant data of the first ML model.

[0361] S908, the MTLF functional entity sends the information of the third ML model to the AnLF functional entity. Correspondingly, the AnLF functional entity receives the information of the third ML model from the MTLF functional entity.

[0362] After receiving the information of the third ML model, the AnLF functional entity may obtain the third ML model, that is, the latest ML model, from the corresponding position according to the information of the third ML model, and use the third ML model.

[0363] For the specific implementation of S901-S908, please refer to the related introduction of S801-S806 mentioned above, which will not be repeated here.

[0364] It is understandable that S903a and S903b may be performed simultaneously or in sequence, such as performing S903a first and then S903b; or performing S903b first and then S903a, which is not limited in the embodiments of the present application.

[0365] Fig.10 This is a schematic diagram of the structure of the communication device provided in the embodiment of the present application. Figure 1 For example, Fig.10 As shown, the communication device 1000 includes: a transceiver module 1001 and a processing module 1002. For ease of description, Fig.10 Only the main components of the communication device are shown.

[0366] In some embodiments, the communication device 1000 may be suitable for Figure 4 In the communication system shown in FIG. , the above Figure 5 The functionality of the AnLF functional entity in the method shown.

[0367] The transceiver module 1001 is used to receive information of a first ML model from the MTLF functional entity, and the information of the first ML model is used to obtain the first ML model. The processing module 1002 is used to control the transceiver module 1001 to send a first message to the MTLF functional entity, and the first message carries information of the first ML model and information of a first data set, and the first data set is used to store relevant data of the first ML model.

[0368] In one possible design scheme, the first ML model is multiple ML models, the information of the first ML model is information of multiple ML models, the information of the first data set is information of multiple data sets, the multiple ML models correspond one-to-one to the information of the multiple ML models, and the information of the multiple ML models corresponds one-to-one to the information of the multiple data sets.

[0369] In a possible design scheme, the information of the first ML model is any one of the following: identification information of the first ML model, a URL address of the first ML model, and an FQDN of the first ML model.

[0370] In a possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0371] In a possible design scheme, the processing module 1002 is further used to control the transceiver module 1001 to send a request message to the MTLF functional entity, where the request message is used to request the ML model; in response to the request message, the transceiver module 1001 is further used to receive information about the first ML model from the MTLF functional entity.

[0372] In one possible design scheme, the processing module 1002 is further used to control the transceiver module 1001 to send a request message to the MTLF functional entity, where the request message is used to request the ML model; in response to the request message, the transceiver module 1001 is further used to receive information about a second ML model from the MTLF functional entity, where the information about the second ML model is used to obtain the second ML model, and the information about the second data set carried by the request message is used to indicate that the second data set corresponds to the ML model sent by the MTLF functional entity after receiving the request message, and the second data set is used to store relevant data of the second ML model.

[0373] Optionally, the information of the second data set includes identification information of the second data set and / or identification information of the second ADRF functional entity.

[0374] Optionally, the first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

[0375] In a possible design scheme, the transceiver module 1001 is further used to receive a request for monitoring the ML model from the MTLF functional entity, where the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0376] In one possible design scheme, the transceiver module 1001 is further used to receive a request for monitoring the ML model from the MTLF functional entity, where the request for monitoring the ML model carries information of the first ML model; in response to the request for monitoring the ML model, the transceiver module 1001 is further used to send information of a third data set to the MTLF functional entity, where the third data set is used to store relevant data of the first ML model.

[0377] In one possible design scheme, the processing module 1002 is also used to monitor the analysis accuracy of the first ML model; when the analysis accuracy of the first ML model is insufficient, the processing module 1002 is also used to control the transceiver module 1001 to send a notification request message to the MTLF functional entity, where the notification request message carries information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0378] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity.

[0379] In a possible design scheme, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0380] In a possible design scheme, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0381] Optionally, the transceiver module 1001 may include a sending module ( Fig.10 ) and a receiving module ( Fig.10 The sending module is used to implement the sending function of the communication device 1000, and the receiving module is used to implement the receiving function of the communication device 1000.

[0382] Optionally, the communication device 1000 may further include a storage module ( Fig.10 (not shown), the storage module stores a program or instruction. When the processing module 1002 executes the program or instruction, the communication device 1000 can execute the above method. Figure 5 Functions of the AnLF functional entity in the method shown.

[0383] It can be understood that the communication device 1000 can be a network device, or a chip (system) or other parts or components that can be set in the network device, or a device that includes a network device, which is not limited in this application.

[0384] In addition, the technical effects of the communication device 1000 can be referred to Figure 5 The technical effects of the communication method shown will not be repeated here.

[0385] In some other embodiments, the communication device 1000 may be suitable for Figure 4 In the communication system shown in FIG. , the above Figure 5 Functions of the MTLF functional entity in the method shown.

[0386] The processing module 1002 is used to control the transceiver module 1001 to send information of the first ML model to the AnLF functional entity, where the information of the first ML model is used to indicate the first ML model; the transceiver module 1001 is used to receive a first message from the AnLF functional entity, where the first message carries information of the first ML model and information of a first data set, where the first data set is used to store relevant data of the first ML model.

[0387] In one possible design scheme, the first ML model is multiple ML models, the information of the first ML model is information of multiple ML models, the information of the first data set is information of multiple data sets, the multiple ML models correspond one-to-one to the information of the multiple ML models, and the information of the multiple ML models corresponds one-to-one to the information of the multiple data sets.

[0388] In a possible design scheme, the information of the first ML model is any one of the following: identification information of the first ML model, a URL address of the first ML model, and an FQDN of the first ML model.

[0389] In a possible design scheme, the information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

[0390] In a possible design scheme, the transceiver module 1001 is also used to receive a request message from an AnLF functional entity, where the request message is used to request an ML model; when the request message is received, the processing module 1002 is also used to control the transceiver module 1001 to send information of the first ML model to the AnLF functional entity.

[0391] In a possible design scheme, the transceiver module 1001 is also used to receive a request message from the AnLF functional entity, where the request message is used to request the ML model, and the request message carries information of a second data set, where the second data set is used to store relevant data of the ML model sent by the MTLF functional entity after receiving the request message; when the request message is received, the processing module 1002 is further used to control the transceiver module 1001 to send information of the second ML model to the AnLF functional entity, and the information of the second ML model is used to obtain the second ML model.

[0392] Optionally, the information of the second data set includes identification information of the second data set and / or identification information of the second ADRF functional entity.

[0393] Optionally, the first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

[0394] Optionally, the processing module 1002 is further configured to evaluate a second ML model using a second data set.

[0395] In one possible design, the processing module 1002 is further configured to evaluate the first ML model using the first data set.

[0396] In a possible design scheme, when the MTLF functional entity receives a registration request for ML model monitoring, the processing module 1002 is also used to control the transceiver module 1001 to send a request for monitoring the ML model to the AnLF functional entity, where the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

[0397] In a possible design scheme, the transceiver module 1001 is further used to receive information of a third data set from the AnLF functional entity, and the third data set is used to store relevant data of the first ML model.

[0398] Optionally, information of the third data set is carried in a response and / or notification request message for ML model monitoring.

[0399] Optionally, the information of the third data set includes identification information of the third data set and / or identification information of the third ADRF functional entity.

[0400] In a possible design scheme, the relevant data includes at least one of the following: input data, predicted data, true value data at the time of prediction, and deviation data at the time of prediction.

[0401] In a possible design scheme, the AnLF functional entity is carried on the network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on the NWDAF network element.

[0402] Optionally, the transceiver module 1001 may include a sending module ( Fig.10 ) and a receiving module ( Fig.10 The sending module is used to implement the sending function of the communication device 1000, and the receiving module is used to implement the receiving function of the communication device 1000.

[0403] Optionally, the communication device 1000 may further include a storage module ( Fig.10 (not shown), the storage module stores a program or instruction. When the processing module 1002 executes the program or instruction, the communication device 1000 can execute the above method. Figure 5 The method shown in MTLF

[0404] It can be understood that the communication device 1000 can be a network device, or a chip (system) or other parts or components that can be set in the network device, or a device that includes a network device, which is not limited in this application.

[0405] In addition, the technical effects of the communication device 1000 can be referred to Figure 5 The technical effects of the communication method shown will not be repeated here.

[0406] In some other embodiments, the transceiver module 1001 is used to perform the above Figure 6-Figure 9 The processing module 1002 is used to perform the above-mentioned Figure 6-Figure 9 The method shown has other functions besides the sending and receiving functions.

[0407] Optionally, the transceiver module 1001 may include a sending module ( Fig.10 ) and a receiving module ( Fig.10 The sending module is used to implement the sending function of the communication device 1000, and the receiving module is used to implement the receiving function of the communication device 1000.

[0408] Optionally, the communication device 1000 may further include a storage module ( Fig.10 (not shown), the storage module stores a program or instruction. When the processing module 1002 executes the program or instruction, the communication device 1000 can execute the above method. Figure 6-Figure 9 The functions of the MTLF functional entity, the AnLF functional entity, or the ADRF functional entity in the method shown.

[0409] It can be understood that the communication device 1000 can be a network device, or a chip (system) or other parts or components that can be set in the network device, or a device that includes a network device, which is not limited in this application.

[0410] Fig.11 A schematic diagram of the structure of the communication device provided in the embodiment of the present application Figure 2 For example, the communication device may be a terminal, or a chip (system) or other components or assemblies that may be provided in the terminal. Fig.11 As shown, the communication device 1100 may include a processor 1101. Optionally, the communication device 1100 may further include a memory 1102 and / or a transceiver 1103. The processor 1101 is coupled to the memory 1102 and the transceiver 1103, such as being connected via a communication bus.

[0411] Combine the following Fig.11 The components of the communication device 1100 are described in detail:

[0412] The processor 1101 is the control center of the communication device 1100, and may be a processor or a general term for multiple processing elements. For example, the processor 1101 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present application, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0413] Optionally, the processor 1101 may execute various functions of the communication device 1100 by running or executing a software program stored in the memory 1102 and calling data stored in the memory 1102, such as executing the above Figure 5-Figure 8 The communication method shown.

[0414] In a specific implementation, as an embodiment, the processor 1101 may include one or more CPUs, such as Fig.11 CPU0 and CPU1 are shown in FIG.

[0415] In a specific implementation, as an embodiment, the communication device 1100 may also include multiple processors, such as Fig.11 1 and 1104. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0416] The memory 1102 is used to store the software program for executing the solution of the present application, and the execution is controlled by the processor 1101. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0417] Optionally, the memory 1102 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1102 may be integrated with the processor 1101, or may exist independently and be accessed through the interface circuit ( Fig.11 1101) is coupled to the processor 1101, which is not specifically limited in the embodiments of the present application.

[0418] The transceiver 1103 is used for communication with other communication devices. For example, the communication device 1100 is a terminal, and the transceiver 1103 can be used to communicate with a network device, or to communicate with another terminal device. For another example, the communication device 1100 is a network device, and the transceiver 1103 can be used to communicate with a terminal, or to communicate with another network device.

[0419] Optionally, the transceiver 1103 may include a receiver and a transmitter ( Fig.11 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0420] Optionally, the transceiver 1103 may be integrated with the processor 1101, or may exist independently and communicate with the processor 1101 through the interface circuit ( Fig.11 1101) is coupled to the processor 1101, which is not specifically limited in the embodiments of the present application.

[0421] Understandably, Fig.11 The structure of the communication device 1100 shown in the figure does not constitute a limitation on the communication device. The actual communication device may include more or fewer components than those shown in the figure, or combine certain components, or arrange the components differently.

[0422] In addition, the technical effects of the communication device 1100 can refer to the technical effects of the method described in the above method embodiment, which will not be repeated here.

[0423] An embodiment of the present application also provides a communication chip, in which instructions are stored. When the chip runs on a communication device, the method provided in the embodiment of the present application is implemented.

[0424] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0425] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0426] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0427] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0428] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0429] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0430] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0431] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0432] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0433] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0434] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0435] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0436] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A communication method, characterized in that: The method comprises: The analysis logic function AnLF functional entity receives information of a first machine learning ML model from the model training logic function MTLF functional entity, and the information of the first ML model is used to obtain the first ML model; The AnLF functional entity sends a first message to the MTLF functional entity, where the first message carries information about the first ML model and information about a first data set, where the first data set is used to store relevant data of the first ML model.

2. The method according to claim 1, characterized in that: The method further comprises: The AnLF functional entity sends a request message to the MTLF functional entity, where the request message is used to request an ML model; The AnLF functional entity receives information of the first ML model from the MTLF functional entity, including: In response to the request message, the AnLF functional entity receives information of the first ML model from the MTLF functional entity.

3. The method according to claim 1, characterized in that The method further comprises: The AnLF functional entity sends a request message to the MTLF functional entity, where the request message is used to request an ML model, and the request message carries information of a second data set; In response to the request message, the AnLF functional entity receives information about a second ML model from the MTLF functional entity, the information about the second ML model is used to obtain the second ML model, and the second data set is used to store relevant data of the ML model sent by the MTLF functional entity after receiving the request message.

4. The method according to claim 3, characterized in that The first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The AnLF functional entity receives a request for monitoring an ML model from the MTLF functional entity, where the request for monitoring an ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The AnLF functional entity receives a request for monitoring an ML model from the MTLF functional entity, where the request for monitoring an ML model carries information of the first ML model; In response to the request for monitoring the ML model, the AnLF functional entity sends information of a third data set to the MTLF functional entity, where the third data set is used to store relevant data of the first ML model.

7. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The AnLF functional entity monitors the analysis accuracy of the first ML model; In the case that the analysis accuracy of the first ML model is insufficient, the AnLF functional entity sends a notification request message to the MTLF functional entity, where the notification request message carries information of a third data set, where the third data set is used to store relevant data of the first ML model.

8. A communication method, characterized in that: The method comprises: The model training logic function MTLF functional entity sends information of a first machine learning ML model to the analysis logic function AnLF functional entity, where the information of the first ML model is used to indicate the first ML model; The MTLF functional entity receives a first message from the AnLF functional entity, where the first message carries information about the first ML model and information about a first data set, where the first data set is used to store relevant data of the first ML model.

9. The method according to claim 8, characterized in that The method further comprises: The MTLF functional entity receives a request message from the AnLF functional entity, where the request message is used to request an ML model; The MTLF functional entity sends information of the first ML model to the AnLF functional entity, including: Upon receiving the request message, the MTLF functional entity sends information of the first ML model to the AnLF functional entity.

10. The method according to claim 8, characterized in that The method further comprises: The MTLF functional entity receives a request message from the AnLF functional entity, where the request message is used to request an ML model, and the request message carries information of a second data set; Upon receiving the request message, the MTLF functional entity sends information of a second ML model to the AnLF functional entity, the information of the second ML model is used to obtain the second ML model, and the second data set is used to store relevant data of the ML model sent by the MTLF functional entity after receiving the request message.

11. The method according to claim 10, characterized in that The first ML model is an ML model retrained based on the second ML model, or the first ML model is a regenerated ML model.

12. The method according to claim 10 or 11, characterized in that: The method further comprises: The MTLF functional entity evaluates the second ML model based on the second data set.

13. The method according to any one of claims 8 to 12, characterized in that: The method further comprises: The MTLF functional entity evaluates the first ML model based on the first data set.

14. The method according to any one of claims 8 to 13, characterized in that: The method further comprises: In a case where the MTLF functional entity receives a registration request for ML model monitoring, the MTLF functional entity sends a request for monitoring the ML model to the AnLF functional entity, where the request for monitoring the ML model carries information of the first ML model and information of a third data set, and the third data set is used to store relevant data of the first ML model.

15. The method according to any one of claims 8 to 13, characterized in that: The method further comprises: The MTLF functional entity receives information of a third data set from the AnLF functional entity, where the third data set is used to store relevant data of the first ML model.

16. The method according to claim 15, characterized in that The information of the third data set is carried in a response and / or notification request message for ML model monitoring.

17. A communication method, characterized in that: The method comprises: The model training logic function MTLF functional entity receives a request message from the analysis logic function AnLF functional entity, wherein the request message is used to request a machine learning ML model; The MTLF functional entity sends a first message to the AnLF functional entity, where the first message carries information about a first ML model and information about a first data set, where the information about the first ML model is used to indicate the first ML model, and the first data set is used to store relevant data of the first ML model.

18. The method according to claim 17, characterized in that The method further comprises: The MTLF functional entity sends a request for establishing the first data set in the first ADRF functional entity to the first ADRF functional entity.

19. The method according to claim 17 or 18, characterized in that The method further comprises: The MTLF functional entity evaluates the first ML model based on the first data set.

20. A communication method, characterized in that: The method comprises: The analysis logic function AnLF functional entity sends a request message to the model training logic function MTLF functional entity, wherein the request message is used to request a machine learning ML model; The AnLF functional entity receives a first message from the MTLF functional entity, where the first message carries information of a first ML model and information of a first data set, where the information of the first ML model is used to obtain the first ML model, and the first data set is used to store relevant data of the first ML model.

21. A communication method, characterized in that: The method comprises: The model training logic function MTLF functional entity receives a request message from the analysis logic function AnLF functional entity, wherein the request message carries information of a first data set, where the first data set is used to store relevant data of a plurality of ML models, and the plurality of ML models include a first ML model; The MTLF functional entity sends a request for obtaining relevant data of the first ML model to the data analysis and storage function ADRF functional entity, where the request for obtaining relevant data of the first ML model carries information of the first ML model and information of the first data set; The MTLF functional entity receives relevant data of the first ML model from the ADRF functional entity.

22. The method according to claim 21, characterized in that The method further comprises: The MTLF functional entity sends a first message to the AnLF functional entity, where the first message carries information of a first ML model, where the information of the first ML model is used to indicate the first ML model.

23. The method according to claim 21 or 22, characterized in that The method further comprises: The MTLF functional entity retrains or regenerates a second ML model based on relevant data of the first ML model.

24. The method according to claim 23, characterized in that The method further comprises: The MTLF functional entity sends information about the second ML model to the AnLF functional entity, where the information about the second ML model is used to indicate the second ML model.

25. A communication method, characterized in that: The method comprises: The data analysis and storage function ADRF functional entity receives the relevant data of the first machine learning ML model, the information of the first ML model and the information of the first data set from the analysis logic function AnLF functional entity, where the first data set is used to store the relevant data of the first ML model; The ADRF functional entity stores relevant data of the first ML model according to information of the first ML model and information of the first data set.

26. The method according to claim 25, characterized in that The method further comprises: The ADRF functional entity receives a request from the model training logic function MTLF functional entity for obtaining relevant data of the first ML model, wherein the request for obtaining relevant data of the first ML model includes information of the first ML model and information of the first data set; The ADRF functional entity sends relevant data of the first ML model to the MTLF functional entity based on information of the first ML model and information of the first data set.

27. A communication method, characterized in that: The method comprises: The analysis logic function AnLF functional entity receives a first message from the model training logic function MTLF functional entity, where the first message carries information of a first ML model, and the information of the first ML model is used to obtain the first ML model; The AnLF functional entity sends the relevant data of the first ML model, information of the first ML model and information of a first data set to the data analysis and storage function ADRF functional entity, where the first data set is used to store relevant data of multiple ML models, and the multiple ML models include the first ML model.

28. The method according to claim 27, characterized in that The method further comprises: The AnLF functional entity establishes the first data set.

29. The method according to any one of claims 1 to 28, characterized in that The information of the first ML model is any one of the following: identification information of the first ML model, a uniform resource locator URL address of the first ML model, and a fully qualified domain name FQDN of the first ML model.

30. The method according to any one of claims 1 to 29, characterized in that The information of the first data set includes identification information of the first data set and / or identification information of the first data analysis and storage function ADRF functional entity.

31. The method according to any one of claims 1 to 30, characterized in that The relevant data includes at least one of the following: input data, predicted data, true value data during prediction, and deviation data during prediction.

32. The method according to any one of claims 1 to 31, characterized in that The AnLF functional entity is carried on a network data analysis function NWDAF network element; and / or the MTLF functional entity is carried on a NWDAF network element.

33. A communication device, characterized in that: The apparatus comprises: a module for executing the method as claimed in any one of claims 1-32.

34. A communication device, characterized in that: The communication device comprises: a processor and a memory; the memory is used to store computer instructions, and when the processor executes the instructions, the communication device executes the method according to any one of claims 1-32.

35. A communication system, characterized in that: The communication system includes at least one of the following: an analysis logic function AnLF functional entity, a model training logic function MTLF functional entity, and a data analysis and storage function ADRF functional entity network element, wherein the AnLF functional entity is used to execute the method as described in any one of claims 1-7, or 20, or 27-28, the MTLF functional entity is used to execute the method as described in any one of claims 8-16, or 17-19, or 21-24, and the ADRF functional entity network element is used to execute the method as described in claim 25 or 26.

36. A communication chip, characterized in that: Instructions are stored therein, and when the chip runs on a communication device, the method according to any one of claims 1 to 32 is implemented.

37. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 32.

38. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are executed by a communication device, the method according to any one of claims 1 to 32 is executed.

Images