Evaluation information determination method, related system and storage medium

By passing preset parameters between the first network element and the second network element to generate evaluation information, the problem that consumers cannot choose suitable NWDAF services is solved, and the ability to select suitable services according to the evaluation criteria is realized, and the needs of consumers are met. And ensures the accuracy and performance of the service.

WO2024230375A1PCT designated stage expired Publication Date: 2024-11-14HUAWEI TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/086061
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-11
Filing Date
2024-04-03
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Consumers cannot choose appropriate network data analysis function (NWDAF) services based on accuracy information because there is a lack of clear criteria for generating accuracy information, and different consumers have different criteria for evaluating the same service or model.

Method used

Evaluation information is generated by passing preset parameters between the first network element and the second network element, such as the classification error cost matrix and the category weight, so that the second network element can select the appropriate first network element to provide services.

Benefits of technology

Allow consumers to select appropriate NWDAF services based on their evaluation information generation criteria to meet their needs and ensure the accuracy and performance of the services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024086061_14112024_PF_FP_ABST
    Figure CN2024086061_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the embodiments of the present application are an evaluation information determination method, a related system and a storage medium. The method comprises: receiving a first request sent by a second network element, the first request comprising first information, the first information comprising preset parameters, and the preset parameters being used for generating evaluation information; acquiring the evaluation information according to the first information, the evaluation information being used for evaluating a service or a model; and returning the evaluation information to the second network element. By using the means, the preset parameters in the first information sent by the second network element can be used for generating the evaluation information, so that the second network element knows an evaluation information generation standard, thereby facilitating the second network element to select a proper first network element.
Need to check novelty before this filing date? Find Prior Art

Description

Evaluation information determination method and related system and storage medium

[0001] This application claims priority to the Chinese patent application with application number 202310534752.5 filed with the State Intellectual Property Office of China on May 11, 2023, and priority to the Chinese patent application with the invention name “Evaluation Information Determination Method and Related Systems, Storage Medium”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of communication technology, and in particular to a method for determining evaluation information and related systems and storage media. Background Art

[0003] Among them, the machine learning (ML) model combined with the network data analytics function (NWDAF) can provide prediction services in different scenarios. Currently, service consumers select an appropriate NWDAF with a model analysis logical function (AnLF) and subscribe to analytics accuracy information by calling the network data analytics function Nnwdaf_AnalyticsSubscription_Subscribe service operation so that consumers can check and deploy accuracy information. However, the services (such as the analysis identification (ID)) or model accuracy information obtained by consumers include the preferred level of accuracy of the analytics ("Low", "Medium", "High" or "Highest"), which only defines the accuracy level and accuracy information value, but does not clarify the generation standard of accuracy information.

[0004] Since consumers do not know the accuracy information generation criteria, they cannot select a more appropriate NWDAF service based on this information. At the same time, for example, for the same analysis ID, different consumers may have different evaluation criteria. Therefore, different consumers cannot select a suitable NWDAF based solely on accuracy information.

[0005] Summary of the Invention

[0006] The present application discloses an evaluation information determination method and related system and storage medium, which can help service consumers select a suitable NWDAF based on the evaluation information generation criteria determined by them.

[0007] In a first aspect, an embodiment of the present application provides a method for determining evaluation information, comprising: a first network element receiving a first request sent by a second network element. The first request includes first information. The first information includes preset parameters. The preset parameters are used to generate evaluation information. The first network element then obtains evaluation information based on the first information. The evaluation information is used to evaluate a service or model. The first network element then returns the evaluation information to the second network element.

[0008] In an embodiment of the present application, a first network element obtains evaluation information based on first information from a second network element, and then returns the evaluation information to the second network element. With this approach, because the preset parameters used to generate the evaluation information are set by the second network element, the first network element obtains the evaluation information based on the preset parameters. This facilitates the second network element's selection of an appropriate first network element based on the received evaluation information.

[0009] This example is introduced by taking the first network element as an example. The first network element can be understood as a single network element or a combination of multiple network elements, and this solution does not impose any limitation on this.

[0010] In one possible implementation, the preset parameter is a classification error cost matrix. The classification error cost can be understood as a measure of the cost to consumers of receiving incorrect class predictions. This classification error cost matrix can be used for classification tasks in various analytical services, such as network element load analysis, abnormal behavior prediction, and trend measurement.

[0011] Different consumers may have different prediction error cost matrices for the same service or model. Therefore, by clarifying the preset parameters, we can obtain evaluation information that meets the needs of the consumer.

[0012] In another possible implementation, the preset parameter is a category weight. The category weight is a measure of the importance of a service (such as an analysis ID) or the output category of a model to a consumer. The specific standard for the measure of importance is not limited in this solution, and it can be a relative measure or an absolute measure. Optionally, for abnormal behavior analysis, it supports mobility-related anomalies and session-related anomalies. Mobility-related anomalies include: abnormal UE location, abnormal UE wake-up, abnormal wireless connection errors, etc. Session-related anomalies include: abnormally long or high-rate traffic, abnormal wake-up, suspected distributed denial of service attack (DDoS), wrong destination address, too frequent service access, etc.

[0013] For example, the prediction category of a certain analysis service is the above abnormal behavior category. Since different consumer network elements have different concerns about the above abnormal behavior category, the category weights in their setting services are different.

[0014] In yet another possible implementation, the preset parameters are a classification error cost matrix and a category weight.

[0015] In a possible implementation, the first information further includes an evaluation information calculation method, the evaluation information calculation method includes the preset parameters, and the evaluation information calculation method is associated with the preset parameters.

[0016] The evaluation information calculation method can be understood as a formula or text for calculating the evaluation information. The evaluation information calculation method corresponds to, for example, one or more of the accuracy, precision, recall rate, classification report, loss function, etc. of the evaluation service or model.

[0017] It is understandable that the first information may correspond to an information element (IE). For example, an IE includes an accuracy calculation formula, a loss function, a classification report, and the loss value calculation method includes category weights, etc.

[0018] Alternatively, the first information may correspond to multiple IEs, for example, two IEs, one of which includes a loss value calculation method and the other includes a category weight.

[0019] Of course, other methods may be used to indicate the first information, and this solution does not limit this.

[0020] Optionally, this evaluation information can be calculated using methods such as F1-score, loss function, and classification report. Loss function types include mean squared error loss, weighted mean squared error loss, cross entropy loss, and cost loss. The classification report indicates the classification results of NWDAF on the test dataset. Based on the classification report and internal configuration, consumers can evaluate the performance of the model. For example, the classification report can be used to calculate the model's accuracy and recall. Preset parameters can include limits on the number of samples per category, such as the ratio of samples per category and the minimum number of samples.

[0021] The evaluation information can be used to measure the performance of the service or model, such as accuracy, precision, recall, classification report, loss value, etc. It is understandable that the service is an analysis service provided by the first network element. For example, it can analyze the quality of service (QoS), analyze the network element load, analyze abnormal behavior, etc. The model can be a machine learning model used by the first network element to provide the analysis service. For example, the model is a model obtained by the first network element from a third network element.

[0022] In a possible implementation manner, the first network element further obtains evaluation information according to the first information and the test data set.

[0023] Among them, test data refers to data used to independently evaluate trained and verified artificial intelligence systems (services or models) in order to confirm their expected performance before using the service or model.

[0024] In a possible implementation, the test data set is obtained according to a test data set identifier, wherein the first request further includes the test data set identifier.

[0025] In a possible implementation manner, the first network element returns the correlation degree between the evaluation information and the second network element to the second network element.

[0026] Optionally, the degree of correlation is the proportion of data related to the second network element in the test data set.

[0027] In a possible implementation, the data related to the second network element is real data of a service subscribed to by the second network element or a prediction result of a model, and the relevant proportion is the proportion of the real data in the entire test data set.

[0028] In another possible implementation manner, the test data set is a data set related to the second network element.

[0029] In a possible implementation, the first network element sends the first information to the third network element, and then the first network element receives the evaluation information sent by the third network element.

[0030] That is, the evaluation information is calculated by the third network element.

[0031] In one possible implementation, the first request further includes second information, where the second information is used to instruct the return of evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units within the time window. Alternatively, the evaluation-related information includes statistical information about the evaluation information corresponding to multiple time units within the time window. Alternatively, the evaluation-related information includes evaluation information corresponding to multiple time units within the time window and statistical information about the evaluation information corresponding to the multiple time units. The first network element returns the evaluation-related information to the second network element based on the second information.

[0032] This time window, also known as the interval [start.end], can be understood as a start time and end time in the past or a start time and end time in the future. The interval can be expressed as an actual start time and an actual end time (for example, using Coordinated Universal Time Coordinated (UTC) time). By setting the start and end times to the same value, analysis users can request or subscribe to analysis for a specific time.

[0033] The time unit can be understood as a plurality of time periods obtained by dividing the above-mentioned time window. The multiple time units can be of the same size or different, and this solution does not limit this. For example, the above-mentioned time window is: the start time is 12:00 on January 1, 2022, and the end time is 12:00 on February 1, 2022. The time unit can be 1 day, 6 hours, 7 days, one day, one week, one month, etc. Alternatively, the time unit is a series of time points used to divide the above-mentioned time window. For example, if the time point is 00:00 on January 14, 2022, then the above-mentioned time window is divided into two time periods. By analogy, there can be multiple time points.

[0034] The statistical information may be information obtained by processing the evaluation information corresponding to the multiple time units. For example, the statistical information may be the variance, mean, etc. of the evaluation information corresponding to the multiple time units. Other statistical information may also be used, and this solution does not impose any restrictions on this.

[0035] In this way, the evaluation association information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element select a suitable first network element based on the fluctuations.

[0036] In a second aspect, an embodiment of the present application provides a method for determining evaluation information, comprising: a second network element sending a first request to a first network element. The first request includes first information. The first information includes preset parameters. The preset parameters are used to generate evaluation information. The second network element then receives evaluation information returned by the first network element. The evaluation information is obtained by the first network element based on the first information. The evaluation information is used to evaluate a service or model.

[0037] In this embodiment of the present application, a second network element sends a first request to a first network element. The first request includes first information, which includes preset parameters used to generate evaluation information. The second network element then receives the evaluation information returned by the first network element. This approach generates evaluation information based on the preset parameters in the first information sent by the second network element. This allows the second network element to understand the criteria for generating the evaluation information, which can help it select a suitable first network element.

[0038] In a possible implementation, the preset parameter is a classification error cost matrix and / or a category weight.

[0039] In a possible implementation, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameter.

[0040] In a possible implementation manner, the evaluation information is acquired by the first network element according to the first information and a test data set.

[0041] In a possible implementation, the test data set is obtained according to a test data set identifier, wherein the first request further includes the test data set identifier.

[0042] In a possible implementation manner, the second network element further receives the correlation degree between the evaluation information returned by the first network element and the second network element.

[0043] In a possible implementation, the correlation degree is a proportion of data related to the second network element in the test data set, for example, a proportion size.

[0044] In another possible implementation manner, the test data set is a data set related to the second network element.

[0045] In one possible implementation, the first request further includes second information, which is used to indicate the return of evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units within the time window. Alternatively, the evaluation-related information includes statistical information about the evaluation information corresponding to multiple time units within the time window. Alternatively, the evaluation-related information includes evaluation information corresponding to multiple time units within the time window and statistical information about the evaluation information corresponding to the multiple time units. The second network element also receives the evaluation-related information returned by the first network element.

[0046] In this way, the evaluation association information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element select a suitable first network element based on the fluctuations.

[0047] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.

[0048] In a third aspect, an embodiment of the present application provides a method for determining evaluation information, comprising: a first network element receives a second request sent by a second network element. The second request includes second information. The second information is used to indicate the return of evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units within a time window. Alternatively, the evaluation-related information includes statistical information of the evaluation information corresponding to multiple time units within a time window. Alternatively, the evaluation-related information includes evaluation information corresponding to multiple time units within a time window, and statistical information of the evaluation information corresponding to the multiple time units. The evaluation information is used to evaluate a service or model. The first network element then returns the evaluation-related information to the second network element based on the second information.

[0049] In this way, the evaluation association information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element select a suitable first network element based on the fluctuations.

[0050] This time window, also known as the interval [start.end], can be understood as a start time and end time in the past or a start time and end time in the future. The interval can be expressed as an actual start time and an actual end time (for example, using Coordinated Universal Time Coordinated (UTC) time). By setting the start and end times to the same value, analysis users can request or subscribe to analysis for a specific time.

[0051] The time unit can be understood as a plurality of time periods obtained by dividing the above-mentioned time window. The multiple time units can be of the same size or different, and this solution does not limit this. For example, the above-mentioned time window is: the start time is 12:00 on January 1, 2022, and the end time is 12:00 on February 1, 2022. The time unit can be 1 day, 6 hours, 7 days, one day, one week, one month, etc. Alternatively, the time unit is a series of time points used to divide the above-mentioned time window. For example, if the time point is 00:00 on January 14, 2022, then the above-mentioned time window is divided into two time periods. By analogy, there can be multiple time points.

[0052] The statistical information may be information obtained by processing the evaluation information corresponding to the multiple time units. For example, the statistical information may be the variance, mean, etc. of the evaluation information corresponding to the multiple time units. Other statistical information may also be used, and this solution does not impose any restrictions on this.

[0053] In a possible implementation, the second request further includes the time window.

[0054] Optionally, the second information may also indicate periodically returning the evaluation-related information.

[0055] In one possible implementation, the second request further includes first information. The first information includes preset parameters. The preset parameters are used to generate evaluation information. The first network element further obtains evaluation information based on the first information.

[0056] In a possible implementation, the preset parameter is a classification error cost matrix and / or a category weight.

[0057] In a possible implementation, the first information further includes an evaluation information calculation method, wherein the evaluation information calculation method includes the above-mentioned preset parameters.

[0058] In a possible implementation manner, the first network element obtains evaluation information according to the first information and the test data set.

[0059] In a possible implementation manner, the first network element sends the first information to the third network element, and then the first network element receives the evaluation information sent by the third network element.

[0060] In one possible implementation, the above-mentioned service is an analysis service provided by the first network element, and the above-mentioned model is a machine learning model used by the first network element to provide the analysis service.

[0061] In a fourth aspect, an embodiment of the present application provides a method for determining evaluation information, comprising: a second network element sends a second request to a first network element. The second request includes second information. The second information is used to indicate the return of evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units within a time window. Alternatively, the evaluation-related information includes statistical information of the evaluation information corresponding to multiple time units within a time window. Alternatively, the evaluation-related information includes evaluation information corresponding to multiple time units within a time window, and statistical information of the evaluation information corresponding to the multiple time units. The evaluation information is used to evaluate a service or model. The second network element then receives the evaluation-related information returned by the first network element.

[0062] In an embodiment of the present application, a second network element sends a request to a first network element, the request including second information, the second information being used to instruct the return of evaluation-related information, the evaluation-related information including evaluation information corresponding to multiple time units within a time window and / or statistical information of the evaluation information corresponding to the multiple time units, the evaluation information being used to evaluate a service or model; and the first network element then returns the evaluation-related information to the second network element based on the second information. In this way, the evaluation-related information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element select an appropriate first network element based on the fluctuations.

[0063] In a possible implementation, the second request further includes a time window.

[0064] In one possible implementation, the second request further includes first information. The first information includes preset parameters. The preset parameters are used to generate evaluation information. The evaluation information is obtained by the first network element based on the first information.

[0065] In a possible implementation, the preset parameter is a classification error cost matrix and / or a category weight.

[0066] In a possible implementation, the first information further includes an evaluation information calculation method, which includes the aforementioned preset parameters.

[0067] In a possible implementation manner, the evaluation information is acquired by the first network element according to the first information and a test data set.

[0068] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.

[0069] In a fifth aspect, the present application provides a method for determining evaluation information, comprising: a second network element sending a first request to a first network element. The first request includes first information. The first information includes preset parameters. The preset parameters are used to generate evaluation information. The first network element receives the first request sent by the second network element. The first network element then obtains evaluation information based on the first information, the evaluation information being used to evaluate a service or model, and returns the evaluation information to the second network element. The second network element receives the evaluation information returned by the first network element.

[0070] In this embodiment of the present application, a second network element sends a request to a first network element. The request includes first information, which includes preset parameters used to generate evaluation information. The first network element obtains evaluation information based on the first information, which is used to evaluate a service or model, and then returns the evaluation information to the second network element. This method generates evaluation information based on the preset parameters in the first information sent by the second network element. This allows the second network element to know the criteria for generating the evaluation information, which can help it select an appropriate first network element.

[0071] In a possible implementation, the preset parameter is a classification error cost matrix and / or a category weight.

[0072] In a possible implementation, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

[0073] In one possible implementation, the first request also includes second information. The second information is used to indicate the return of evaluation-related information, where the evaluation-related information includes evaluation information corresponding to multiple time units within a time window and / or statistical information of the evaluation information corresponding to the multiple time units. The first network element also returns the evaluation-related information to the second network element. The second network element also receives the evaluation-related information returned by the first network element.

[0074] In a sixth aspect, the present application provides a method for determining evaluation information, comprising: a second network element sending a second request to a first network element. The second request includes second information. The second information is used to instruct the first network element to return evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units within a time window and / or statistical information of the evaluation information corresponding to the multiple time units. The evaluation information is used to evaluate a service or model. The first network element receives the second request sent by the second network element. The first network element returns the evaluation-related information to the second network element based on the second information. The second network element then receives the evaluation-related information returned by the first network element.

[0075] In an embodiment of the present application, a second network element sends a request to a first network element, the request including second information, the second information being used to instruct the return of evaluation-related information, the evaluation-related information including evaluation information corresponding to multiple time units within a time window and / or statistical information of the evaluation information corresponding to the multiple time units, the evaluation information being used to evaluate a service or model; and the first network element then returns the evaluation-related information to the second network element based on the second information. In this way, the evaluation-related information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element select an appropriate first network element based on the fluctuations.

[0076] In a possible implementation, the second request further includes first information. The first information includes preset parameters, and the preset parameters are used to generate evaluation information. The first network element further obtains evaluation information based on the first information.

[0077] In a possible implementation, the preset parameter is a classification error cost matrix and / or a category weight.

[0078] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

[0079] In a seventh aspect, the present application provides an evaluation information determination device, which can be any one of NWDAF, NF, AF, or a chip thereof. The evaluation information determination device includes a unit and / or module for executing the evaluation information determination method provided by any one of the first to sixth aspects, or any possible implementation of any one of them, such as a transceiver unit and / or a processing unit. The transceiver unit is used to send and receive various information or signaling, and thus can also achieve the beneficial effects (or advantages) of the evaluation information determination method provided in any of the above aspects.

[0080] In an eighth aspect, the present application provides an evaluation information determination device, which may include a processor and a transceiver. The transceiver is used to send and receive various information or signaling, and the computer program includes program instructions. When the processor runs the program instructions, the evaluation information determination device executes the communication method described in any one of the first to sixth aspects, or any possible implementation of any one of them. The transceiver may be a radio frequency module in a communication device, or a combination of a radio frequency module and an antenna, or an input and output interface of a chip or circuit. Optionally, the evaluation information determination device also includes a memory for storing the computer program.

[0081] In the ninth aspect, the present application provides a readable storage medium having program instructions stored thereon, which, when executed on a computer, enables the computer to execute the evaluation information determination method described in any one of the first to sixth aspects, or any possible implementation of any one of the aspects.

[0082] In a tenth aspect, the present application provides a program product comprising instructions, which, when executed, enables the evaluation information determination method described in any possible implementation of any one of the first to sixth aspects to be executed.

[0083] In an eleventh aspect, the present application provides an apparatus, which may be implemented in the form of a chip or a device, comprising a processor. The processor is configured to read and execute a program stored in a memory to perform the evaluation information determination method provided by one or more of any one of the first to sixth aspects, or one or more of any possible implementations of any one of these aspects.

[0084] Optionally, the device further includes a memory, which is connected to the processor via a circuit.

[0085] Further optionally, the apparatus further includes a communication interface, the processor being connected to the communication interface. The communication interface is configured to receive information and / or signaling to be processed, the processor obtaining the information and / or signaling from the communication interface, processing the information and / or signaling, and outputting the processing results via the communication interface. The communication interface may be an input / output interface.

[0086] Optionally, the processor and memory may be physically independent units, or the memory may be integrated with the processor.

[0087] In a twelfth aspect, the present application provides an evaluation information determination system, which includes an apparatus corresponding to the evaluation information determination method described in the first to second aspects, or an apparatus corresponding to the evaluation information determination method described in the third to fourth aspects.

[0088] It is understandable that the above-mentioned devices, computer storage media, or computer program products are all used to perform any of the methods provided in the first aspect, any of the methods provided in the second aspect, any of the methods provided in the third aspect, any of the methods provided in the fourth aspect, any of the methods provided in the fifth aspect, and any of the methods provided in the sixth aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The following is an introduction to the drawings used in the embodiments of this application.

[0090] FIG1 is a schematic diagram of an architecture of an NWDAF network element interacting with other network elements provided in an embodiment of the present application;

[0091] FIG2 is a flow chart of a method for determining evaluation information provided in an embodiment of the present application;

[0092] FIG3 is a flow chart of another method for determining evaluation information provided in an embodiment of the present application;

[0093] FIG4 is a flow chart of another method for determining evaluation information provided in an embodiment of the present application;

[0094] FIG5 is a flow chart of another method for determining evaluation information provided in an embodiment of the present application;

[0095] FIG6 is a schematic diagram of the structure of an evaluation information determination device provided in an embodiment of the present application;

[0096] FIG7 is a schematic structural diagram of another device for determining evaluation information provided in an embodiment of the present application;

[0097] FIG8 is a schematic structural diagram of another device for determining evaluation information provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the embodiments of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0099] The technical solution provided in this application can be applied to various communication systems, such as: long term evolution (LTE) system, LTE frequency division duplex (FDD) system, LTE time division duplex (TDD), Internet of Things (IoT) system, narrowband Internet of Things (NB-IoT) system, wireless fidelity (Wi-Fi), fifth generation (5G) communication system or new radio (NR) and other future communication systems, such as the sixth generation mobile communication system.

[0100] The network data analytics function (NWDAF) network element supports data collection from other network function and application function (AF) network elements; supports data collection from operation administration and maintenance (OAM) network elements; and supports the provision of analytical information to other network function and AF network elements.

[0101] The NWDAF network element has the functions of data collection, model training, data analysis, and model reasoning. It can be used to collect relevant data from network network elements, third-party service servers, terminal devices, or network management systems, perform data analysis based on the relevant data to obtain analysis results, and provide analysis results to network network elements, third-party service servers, terminal devices, or network management systems. The analysis results can assist the network in selecting service quality parameters for the service, or assist the network in executing traffic routing, or assist the network in selecting background data transmission strategies, etc. In addition, the NWDAF network element can also be used to collect relevant data from network network elements, third-party service servers, terminal devices, or network management systems, and perform model training based on the relevant data to obtain an artificial intelligence (AI) model or a machine learning (ML) model, and provide the AI ​​model / ML model to other NWDAF network elements. The AI ​​model / ML model can be used to assist the NWDAF network element in generating data analysis results based on the relevant data.

[0102] In this embodiment of the present application, ANLF NWDAF refers to an NWDAF that includes an analytics logical function (AnLF), which is used to perform inference, derive analytical information (i.e., derive statistical information and / or predictions based on analysis consumer requests), and expose analytical services.

[0103] MTLF NWDAF represents an NWDAF that includes a model training logical function (MTLF). It is used to train machine learning (ML) models and expose new training services, such as providing trained ML models.

[0104] The following will describe the system architecture of the embodiment of the present application in detail with reference to the accompanying drawings. Please refer to Figure 1, which is a schematic diagram of the architecture of an NWDAF network element interacting with other network elements applicable to the embodiment of the present application. As shown in Figure 1(a), it shows the architecture used when collecting data, that is, the architecture used when the NWDAF network element calls the services of other 5G core network network elements to collect the input information required for network analysis. As shown in Figure 1(b), it shows the architecture used when data is open, that is, the architecture in which the NWDAF network element sends analysis data to the network elements that subscribe to or request the analysis data by providing NWDAF services. Among them, NWDAF represents the network analysis logic function managed by the operator. The NWDAF network element provides slice-level network data analysis for other network elements, and provides network analysis information (i.e., load level information) at the network slice instance level without knowing the current subscribers using the slice. "Any NF" shown in Figure 1 represents any network function network element, "Nnf" and "Nnwdaf" represent the names of the service-based interfaces. For details, please refer to the relevant description in the 3GPP standard protocol, which will not be explained here.

[0105] The above describes the architecture of the embodiment of the present application. The following describes the method of the embodiment of the present application in detail.

[0106] Referring to Figure 2, it is a flow chart of an evaluation information determination method provided by an embodiment of the present application. Optionally, the method can be applied to the aforementioned evaluation information determination system, such as the evaluation information determination system shown in Figure 1. The evaluation information determination method shown in Figure 2 may include steps 201-205. It should be understood that this application is described in the order of 201-205 for the convenience of description, and is not intended to limit execution to the above order. The embodiment of the present application does not limit the order of execution, execution time, number of executions, etc. of the above one or more steps. The following description is based on the example that the execution subject of step 201 and step 205 of the evaluation information determination method is the second network element, such as a service consumer, and the execution subject of steps 202-204 is the first network element, such as ANLF NWDAF. This application is also applicable to other execution subjects. Steps 201-205 are as follows:

[0107] 201. A second network element sends a first request to a first network element, where the first request includes first information, and the first information includes preset parameters, where the preset parameters are used to generate evaluation information.

[0108] The first request may be, for example, a request to subscribe to analysis and evaluation information, such as the analysis and evaluation information subscription service Nnwdaf_AnalyticsSubscription_Subscribe, or other requests, which are not limited in this solution.

[0109] The preset parameters can be used to generate evaluation information. The second network element transmits the preset parameters to the first network element so that the first network element generates evaluation information according to the preset parameters.

[0110] In one possible implementation, the preset parameter is a classification error cost matrix. The classification error cost can be understood as a measure of the cost to consumers of receiving a prediction that is incorrectly predicted, as shown in Table 1.

[0111] Table 1

[0112] Among them, C12 represents the cost of the model incorrectly predicting that a sample with the true label of 1 is classified as category 2. C21 represents the cost of the model incorrectly predicting that a sample with the true label of 2 is classified as category 1. The specific error cost calculation method is not limited in this solution. It can be a relative metric or an absolute metric. For example, for a relative metric, for a certain category, the sum of the prediction error costs of other categories is 1. For an absolute metric, the consumer can generate a classification error cost matrix based on pre-configured information according to local policy.

[0113] This misclassification cost matrix can be used to evaluate various analytical services or models. For example, network element load analysis, abnormal behavior category prediction, and abnormal trend level measurement. Furthermore, service consumers can customize the specific numerical values ​​and ratios of the cost of mispredicting the output category of an analysis or model. For example, for abnormal behavior analysis, the predicted output is the category of abnormal behavior, which is divided into four levels: high, medium, low, and normal. For a given analysis or model, the cost of predicting a high-level anomaly as a low-level anomaly is 1, but the cost of predicting a high-level anomaly as normal may be different, such as 3.

[0114] Furthermore, different network elements may have different security requirements. For example, if a certain abnormal behavior has a high anomaly level but a low prediction result, the cost of a misprediction is relatively low for the Unified Data Management (UDM) function, as it likely has high security requirements. However, for network elements with lower security requirements, which are only sensitive to high-level abnormal behavior, when the prediction level is low, the abnormal behavior will be ignored, potentially causing security issues and a high cost of a misprediction. Therefore, the cost of a misprediction varies for network elements with different security requirements.

[0115] For another example, in a binary classification problem, the output is two categories. For example, the 5QI of the GBR resource type meets or exceeds the reported QoS flow retention threshold. C1 indicates satisfied, and C2 indicates unsatisfied. C12 represents the cost of the model predicting a satisfied result as unsatisfied. C21 represents the cost of the model predicting an unsatisfied result as satisfied.

[0116] It is understandable that different consumers may have different prediction error cost matrices for the same service or model. Therefore, by clarifying the preset parameters, evaluation information that meets the needs of the consumer can be obtained.

[0117] In another possible implementation, the preset parameter is a category weight. The category weight is a measure of the importance of the service (such as analysis ID) or the output category of the model to the consumer. The specific standard for measuring this importance is not limited in this solution, and it can be a relative measure or an absolute measure. For example, for a relative measure, the sum of the importance measure values ​​of the output category of a certain analysis or model is 1.

[0118] For example, the abnormal behavior analysis service supports both mobility-related and session-related anomalies. Mobility-related anomalies include abnormal UE location, abnormal UE wakeup, and abnormal wireless connection errors. Session-related anomalies include abnormally long or high-rate traffic, abnormal wakeup, suspected distributed denial of service (DDoS) attacks, incorrect destination addresses, and excessively frequent service access. Because different consumer network elements have different levels of interest in these abnormal behavior categories, the category weights they set in the service vary. For example, the Access and Mobility Management Function (AMF) focuses more on the accuracy of unexpected wakeups and unexpected radio link failures, while the Session Management Function (SMF) focuses more on the accuracy of unexpected long-lived / high-rate flows. Therefore, different consumers have different accuracy requirements for different anomaly types. The AnLF NWDAF needs to select appropriate models based on consumer preferences to provide more appropriate prediction results.

[0119] In another possible implementation, the preset parameters are a classification error cost matrix and a category weight. The consumer considers both the classification error cost and the category weight.

[0120] The above is only an example. Of course, other parameters are also possible and this solution does not limit this.

[0121] In a possible implementation, the first information further includes an evaluation information calculation method. For example, an evaluation information generation standard may be determined based on the evaluation information calculation method and preset parameters.

[0122] The evaluation information calculation method can be understood as a formula or text for calculating the evaluation information, or instruction information (indicating the evaluation information calculation method), etc.

[0123] The evaluation information calculation method includes the above-mentioned preset parameters. It can be understood that, for example, the evaluation information calculation method is a formula. The formula includes terms corresponding to the above-mentioned preset parameters. For example, if the preset parameter is a=2, the formula is L=ax.

[0124] It can also be understood that, for example, the evaluation information calculation method is a formula. The formula includes the above-mentioned preset parameters. For example, if the preset parameter is a=2, the formula is L=2x.

[0125] That is to say, the evaluation information calculation method is associated with the above-mentioned preset parameters. The above description is only an example, and it can also be other ways, which are not limited by this solution.

[0126] The receiver can uniquely determine the evaluation information calculation method based on the evaluation information calculation method. If the evaluation information calculation method includes pre-configured parameters, the first information also includes the corresponding pre-configured parameters.

[0127] In a possible implementation, the evaluation information calculation method is indication information. The network element pre-configures a correspondence between the indication information and the evaluation information calculation method, and the receiver can uniquely determine the evaluation information calculation method based on the indication information.

[0128] The evaluation information calculation method, for example, one or more of the accuracy, precision, recall rate, classification report, loss function, robustness, and resilience of the evaluation service or model.

[0129] It is understandable that the first information may correspond to an information element (IE). That is, an information element includes both the evaluation information calculation method and the preset parameters. For example, an IE includes the accuracy calculation formula, the loss function, the weighted loss function, and the classification report. Since the weighted loss function has a preset parameter: the category weight, the IE will also include the specific value of the category weight.

[0130] Alternatively, the first information may correspond to multiple IEs. For example, it may correspond to two IEs. One IE includes the evaluation information calculation method, and the other IE includes preset parameters. For example, IE1 includes a weighted loss function, and IE2 includes a category weight.

[0131] Of course, other methods may be used to indicate the first information, and this solution does not limit this.

[0132] Optionally, the evaluation information is calculated using an F1-score, a loss function, or the like. Examples of loss functions include mean squared error loss, weighted mean squared error loss, cross entropy loss, cost loss, and weighted loss. For example, a weighted loss function may have a preset parameter, which is a class weight. The class weight may serve as a weight for the weighted loss function.

[0133] For example, for the weighted loss function, its calculation method can be expressed as:

[0134] It represents a test dataset D, where N is the total number of samples, Xi is a sample in the test dataset, Yi is the true category of the sample, and Yi' is the predicted label of the sample, where Yi is a category element in the category set C. Wci represents the weight of category Ci. Discance[Yi,Yi'] represents the distance between the predicted category and the true category of sample Xi. It can be the mean squared error, absolute error, Hamming distance, etc., which is not limited in this solution. The preset parameter can be Wci.

[0135] For another example, for the cost loss function, its calculation method can be expressed as:

[0136] It represents a test dataset D, where N is the total number of samples, Xi is a sample in the test dataset, Yi is the true class of the sample, and Yi' is the predicted label of the sample, where Yi is a class element in the class set C. Cost_cij represents the cost of incorrectly predicting a sample of true class Ci as Cj. Its preset parameter can be a classification error cost matrix, with Cost_cij being an element in the classification error cost matrix.

[0137] For another example, for the weighted cost loss function, its calculation method can be expressed as:

[0138] It represents the cost of incorrectly predicting a sample of class Ci as Cj for a test dataset D, where N is the total number of samples, Xi is a sample in the test dataset, Yi is the true class of the sample, and Yi' is the predicted label of the sample, where Yi is a class element in the class set C. Cost_cij represents the cost of incorrectly predicting a sample of class Ci as Cj, and Wci represents the weight of class Ci. The preset parameters can be Cost_cij and / or Wci.

[0139] For another example, the evaluation information calculation method is a classification report. The classification report indicates the classification results of NWDAF on the test data set. Consumers can evaluate the performance of the model based on the internal configuration according to the classification report. For example, the accuracy and recall rate of the model are calculated according to the classification report. Its preset parameters can be the number limit of samples in each category. For example, the proportion of samples in each category, the minimum number of samples, etc. For example, the set of categories class of samples in the test data set is C, where Ci is an element in the set. The classification report contains the number of samples of category Ci predicted by the model as category Cj. Optionally, the classification report can be an N*N matrix, as shown in Table 2, where N is the number of sample categories, and Cij represents the number of category i predicted as category j.

[0140] Table 2

[0141] The above is only an example, and other calculation methods are also possible, for example, the accuracy information calculation method is the ratio of the number of correctly predicted samples to (the sum of the number of correctly predicted samples and the number of incorrectly predicted samples), etc. This solution does not impose any restrictions on this.

[0142] 202. The first network element receives the first request sent by the second network element.

[0143] 203. The first network element obtains evaluation information based on the first information, where the evaluation information is used to evaluate a service or model.

[0144] Accordingly, the above evaluation information can be used to measure the performance of the service or model, such as accuracy, precision, recall, classification report, loss value, etc. Alternatively, the evaluation information can also be used for model training.

[0145] It is understandable that the service is an analysis service provided by the first network element, for example, analyzing quality of service (QoS), analyzing network element load, analyzing abnormal behavior, etc. The model may be a machine learning model.

[0146] In a possible implementation manner, the first network element calculates the evaluation information according to the first information and a test data set.

[0147] Among them, test data refers to data used to independently evaluate trained and verified artificial intelligence systems (services or models) in order to confirm their expected performance before using the service or model.

[0148] Specifically, when the preset parameter is a classification error cost matrix, the first network element can calculate the evaluation information based on the classification error cost matrix and a test dataset. Optionally, the first network element and the second network element pre-configure an evaluation information calculation method, or the second network element stores a corresponding evaluation information calculation method. The first network element can then calculate the evaluation information based on the evaluation information calculation method, the classification error cost matrix, and the test dataset. Of course, other methods can also be used, and this solution does not limit this.

[0149] The test data set is obtained by the first network element according to the test data set identifier. The first request sent by the second network element also includes the test data set identifier.

[0150] Of course, when the first request does not include the test dataset identifier, the test dataset may also be proactively collected by the first network element, and this solution does not limit this. The first network element obtains the percentage of data in the test dataset that is relevant to the second network element and returns the percentage to the second network element. This allows the second network element to understand the relevance of the evaluation information to the second network element, thereby facilitating the selection of an appropriate NWDAF.

[0151] In one possible implementation, the consumer-related data is the real data of the service subscribed by the consumer or the prediction result of the model. The relevant proportion is the proportion of the above real data in the entire test data set.

[0152] Of course, the first network element may also only collect the data set related to the second network element as the test data set.

[0153] In another possible implementation manner, the first network element sends the first information to the third network element, and then receives evaluation information returned by the third network element.

[0154] Optionally, the first network element sends the first information and a test data set to a third network element, and then receives evaluation information returned by the third network element. For an introduction to the test data set, please refer to the aforementioned description and will not be repeated here. The third network element may evaluate the trained model based on the first information and the test data set.

[0155] Optionally, the third network element may be a MTLF NWDAF.

[0156] For example, MTLF NWDAF sets the first information as a non-learnable parameter during model training. Specifically, MTLF NWDAF uses, for example, class weights as parameters in a weighted loss function during model training. MTLF NWDAF then obtains a model based on the first information.

[0157] The third network element may also obtain an evaluation information calculation method based on the first information, and then obtain the evaluation information of the model based on the test data set.

[0158] It should be noted that, in this solution, the first network element and the third network element can be the same network element or different network elements, and this solution does not impose any restrictions on this.

[0159] 204. The first network element returns the evaluation information to the second network element;

[0160] After obtaining the evaluation information, the first network element sends it to the second network element.

[0161] In a possible implementation manner, the first network element further returns to the second network element the degree of relevance between the evaluation information and the second network element.

[0162] For example, the first network element returns the proportion of data related to the second network element in the above test data set, or other relevant information, which is not limited in this solution.

[0163] 205. The second network element receives the evaluation information returned by the first network element.

[0164] In one possible implementation, the second network element determines whether the first network element meets the requirements based on the evaluation information. For example, the second network element selects a suitable first network element and its corresponding model, service, etc. based on the evaluation information returned by multiple first network elements.

[0165] In one possible implementation, the first request also includes second information, and the second information is used to instruct the first network element to return evaluation-related information, the evaluation-related information including evaluation information corresponding to multiple time units within the time window and / or statistical information of the evaluation information corresponding to the multiple time units, and the evaluation information is used to evaluate the service or model.

[0166] This time window, also known as the interval [start.end], can be understood as a start time and end time in the past or a start time and end time in the future. The interval can be expressed as an actual start time and an actual end time (for example, using Coordinated Universal Time Coordinated (UTC) time). By setting the start and end times to the same value, analysis users can request or subscribe to analysis for a specific time.

[0167] The time unit can be understood as a plurality of time periods obtained by dividing the above-mentioned time window. The multiple time units can be of the same size or different, and this solution does not limit this. For example, the above-mentioned time window is: the start time is 12:00 on January 1, 2022, and the end time is 12:00 on February 1, 2022. The time unit can be 1 day, 6 hours, 7 days, one day, one week, one month, etc. Alternatively, the time unit is a series of time points used to divide the above-mentioned time window. For example, if the time point is 00:00 on January 14, 2022, then the above-mentioned time window is divided into two time periods. By analogy, there can be multiple time points.

[0168] The statistical information may be information obtained by processing the evaluation information corresponding to the multiple time units. For example, the statistical information may be the variance, mean, etc. of the evaluation information corresponding to the multiple time units. Other statistical information may also be used, and this solution does not impose any restrictions on this.

[0169] Optionally, the evaluation association information may also be evaluation information corresponding to multiple time units within the periodically returned time window and / or statistical information of the evaluation information corresponding to the multiple time units, etc., which is not limited in this solution.

[0170] Of course, the above second information can also be carried in other requests, and this solution does not limit this.

[0171] Accordingly, step 204 may include: the first network element returning the evaluation association information to the second network element.

[0172] The first network element obtains evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units.

[0173] The first network element may calculate the evaluation association information by itself, or may obtain the evaluation association information from a third network element, and this solution does not impose any limitation on this.

[0174] That is, the first network element may first obtain the evaluation information corresponding to the multiple time units within the above time window, and then return the obtained evaluation information corresponding to the multiple time units to the second network element.

[0175] Alternatively, the first network element may first obtain evaluation information corresponding to multiple time units within the above time window, and then process the evaluation information corresponding to the multiple time units to obtain statistical information, and then return the statistical information to the second network element.

[0176] Step 205 may include: the second network element receiving the evaluation association information returned by the first network element.

[0177] In a possible implementation, the first request further includes the above-mentioned time window.

[0178] In this embodiment of the present application, a second network element sends a request to a first network element. The request includes first information, which includes preset parameters used to generate evaluation information. The first network element obtains evaluation information based on the first information, which is used to evaluate a service or model, and then returns the evaluation information to the second network element. This method generates evaluation information based on the preset parameters in the first information sent by the second network element. This allows the second network element to know the criteria for generating the evaluation information, which can help it select an appropriate first network element.

[0179] The embodiment shown in FIG2 above introduces the method for determining evaluation information of this solution.

[0180] Example 1

[0181] 3 is a schematic diagram of an evaluation information determination scheme provided in an embodiment of the present application. The method may include steps 301-305, as follows:

[0182] 301. The service consumer sends a first request to the ANLF NWDAF, where the first request includes first information.

[0183] The first information includes an evaluation information calculation method and a category weight, and the evaluation information calculation method includes the category weight.

[0184] For example, the evaluation information calculation method includes a weighted cost loss function, which can be expressed as:

[0185] It represents the test dataset D, N is the total number of samples, Xi is a sample in the test dataset, Yi is the true category of the sample, Yi' is the predicted label of the sample, where Yi is a category element in the category set C. Cost_cij represents the cost of incorrectly predicting a sample of true category Ci as Cj, and Wci represents the weight of category Ci.

[0186] The preset parameters may be Cost_cij and / or Wci.

[0187] 302. The ANLF NWDAF receives the first request and sends the evaluation information calculation method and category weight to the MTLF NWDAF.

[0188] The ANLF NWDAF may also obtain a test dataset and send the test dataset to the MTLF NWDAF.

[0189] 303. MTLF NWDAF generates evaluation information based on the received test data set, evaluation information calculation method and category weights.

[0190] Since the consumer has configured the preset parameters, the MTLF NWDAF can obtain the evaluation information according to the above evaluation information calculation method, the value of the preset weight, and the above test data set.

[0191] For the implementation process, please refer to the description of the embodiment shown in FIG2 , which will not be described in detail here.

[0192] 304. The MTLF NWDAF sends the evaluation information to the ANLF NWDAF.

[0193] 305. ANLF NWDAF returns the evaluation information to the service consumer.

[0194] In this example, the service consumer provides the ANLF NWDAF with the evaluation information calculation method and category weights, so that the service consumer knows the evaluation information generation standard, which can help the service consumer select an appropriate NWDAF based on the evaluation information.

[0195] Example 2

[0196] 4 is a schematic diagram of an evaluation information determination scheme provided in an embodiment of the present application. The method may include steps 401-410, as follows:

[0197] 401. The service consumer sends a first request to the ANLF NWDAF, where the first request includes a classification report.

[0198] This example takes the classification report as an example for introduction. For the introduction of the classification report, please refer to the description of the embodiment shown in FIG2 , which will not be repeated here.

[0199] 402. The ANLF NWDAF receives the first request and sends the first request to the MTLF NWDAF to obtain an ML model.

[0200] Optionally, the first request may include analysis ID, model metrics (e.g., accuracy, precision, recall, classification report, loss, etc.), model metric configuration (category weights, cost metrics, etc.), predetermined status (e.g., requested accuracy level and / or total training time) and / or periodic reporting conditions (e.g., training rounds / time), and test dataset.

[0201] For example, NWDAF containing ANLF uses the Nnwdaf_MLModelProvision service to send a subscription request to NWDAF containing MTLF.

[0202] If the service consumer's request indicates a test dataset, the ANLF NWDAF sends the test dataset to the MTLF NWDAF. If the service consumer does not indicate a test dataset, the ANLF NWDAF obtains the test dataset itself. For details on this aspect, please refer to the description of the embodiment shown in Figure 2 above and will not be repeated here.

[0203] When an accuracy metric is requested, the accuracy level may be used to indicate a target accuracy level for the training process. When the requested accuracy level is reached during training, the federated learning (FL) server NWDAF included in the MTLF NWDAF may stop the training process.

[0204] If the NWDAF containing AnLF or the user of the NWDAF containing AnLF provides a time required for analyzing information, the server NWDAF may determine the maximum response time of its client NWDAF based on the time.

[0205] 403. The server NWDAF sends a first request to the client NWDAF participating in the federated learning to perform local model training of the federated learning.

[0206] The request may include the above-mentioned model metrics (such as classification reports, etc.) and model metric configuration.

[0207] Furthermore, the first request may also include a maximum response time within which the client NWDAF must report the temporary local ML model information to the server NWDAF.

[0208] 404. The multiple client NWDAFs receive the first request and collect their local data.

[0209] 405. Each client NWDAF trains the ML model obtained from the server NWDAF based on the collected data, and sends temporary local ML model information to the server NWDAF.

[0210] During the federated learning training process, each client NWDAF trains the model based on the data it collects.

[0211] Each client NWDAF also calculates local model metrics (such as the classification report mentioned above) and sends them to the server NWDAF.

[0212] For example, the client NWDAF generates a model based on its local data and generates the above classification report based on the local model and the local test dataset. The classification report represents the performance measurement results of each local model on the corresponding local test dataset.

[0213] If the server NWDAF specifies a test data set in the first request, the client NWDAF generates the classification report according to the local model and the specified test data set.

[0214] The local model metrics can be included in the local training status report. The local model metrics may include a report on the model training dataset, the number of samples, and a classification report on the test dataset. The local training status report may also include other information about the local training in the client NWDAF, such as the characteristics of the local training dataset (e.g., area, sampling ratio, maximum / minimum values ​​of each dimension of the data), etc.

[0215] The ML model is sent from the client NWDAF to the server NWDAF during the FL training process. This ML model provides the information needed by the server NWDAF to build an aggregate model based on the locally trained model.

[0216] If the client NWDAF fails to complete the training of the temporary local ML model within the maximum response time provided by the server NWDAF, the client NWDAF notifies the server NWDAF of the delay event and provides appropriate reason information, such as local ML model training failure and the local ML model training requires more time before the maximum response time expires.

[0217] 406. The server NWDAF aggregates the obtained ML model information of all clients to update the global ML model.

[0218] The server NWDAF aggregates all the above model information according to the classification report returned by the client NWDAF to obtain the global ML model.

[0219] The server NWDAF can also calculate global model metrics, such as those based on local model metrics or by applying the global model on a test dataset. For example, the server NWDAF performs validation on a test dataset provided by the consumer and generates evaluation information.

[0220] If the server NWDAF provides the client NWDAF with the aforementioned maximum response time, the server NWDAF may decide to wait for the client NWDAF that has not provided its temporary local ML model within the maximum response time, or to aggregate only the acquired local ML model information instances to update the global ML model, etc. This solution does not impose any restrictions on this.

[0221] In one possible implementation, based on consumer requests, the server NWDAF dynamically updates the training status (e.g., global model metrics) to the consumer periodically (e.g., a certain number of training rounds or every 10 minutes) or when certain predetermined states are reached (e.g., the requested accuracy level is reached or the training time expires).

[0222] In one possible implementation, the consumer determines whether the current model meets the requirements, for example, the global model metrics meet the requirements for the consumer. The consumer can then modify the subscription (stop or continue the training process).

[0223] In a possible implementation, according to the received request, the server NWDAF updates or terminates the current FL training process.

[0224] If the FL process continues, the server NWDAF sends the aggregated ML model information to each client NWDAF for the next round of model training.

[0225] 407. The server NWDAF sends the global ML model to each client NWDAF.

[0226] 408. Each client NWDAF receives the global ML model to update the ML model of each client NWDAF.

[0227] Then it is trained through multiple iterations.

[0228] 409. After the training is completed, the server NWDAF returns the trained model to the ANLF NWDAF.

[0229] 410. ANLF NWDAF returns the model to the service consumer.

[0230] In this example, the service consumer sends a classification report to the ANLF NWDAF so that the MTLF NWDAF generates a global ML model based on the classification report.

[0231] Referring to Figure 5, it is a flow chart of another evaluation information determination method provided by an embodiment of the present application. Optionally, the method can be applied to the aforementioned interactive system, such as the interactive system shown in Figure 1. The evaluation information determination method shown in Figure 5 may include steps 501-504. It should be understood that this application is described in the order of 501-504 for the convenience of description, and is not intended to limit execution to the above order. The embodiment of the present application does not limit the order of execution, execution time, number of executions, etc. of the above one or more steps. The following description is based on the example that the execution subject of steps 501 and 504 of the evaluation information determination method is the second network element, such as a service consumer, and the execution subject of steps 502 and 503 is the first network element, such as ANLF NWDAF. This application is also applicable to other execution subjects. Steps 501-504 are as follows:

[0232] 501. The second network element sends a second request to the first network element, where the second request includes second information. The second information is used to instruct the first network element to return evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units within a time window and / or statistical information of the evaluation information corresponding to the multiple time units. The evaluation information is used to evaluate a service or model.

[0233] The second request may be, for example, a request to subscribe to analysis and evaluation information.

[0234] The evaluation information can be used to evaluate the accuracy, precision, recall rate, etc. of a service or model. Alternatively, the evaluation information can also be used for model training.

[0235] The service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.

[0236] This time window, also known as the interval [start.end], can be understood as a start time and end time in the past or a start time and end time in the future. The interval can be expressed as an actual start time and an actual end time (for example, using Coordinated Universal Time Coordinated (UTC) time). By setting the start and end times to the same value, analysis users can request or subscribe to analysis for a specific time.

[0237] The time unit can be understood as a plurality of time periods obtained by dividing the above-mentioned time window. The multiple time units can be of the same size or different, and this solution does not limit this. For example, the above-mentioned time window is: the start time is 12:00 on January 1, 2022, and the end time is 12:00 on February 1, 2022. The time unit can be 1 day, 6 hours, 7 days, one day, one week, one month, etc. Alternatively, the time unit is a series of time points used to divide the above-mentioned time window. For example, if the time point is 00:00 on January 14, 2022, then the above-mentioned time window is divided into two time periods. By analogy, there can be multiple time points.

[0238] The statistical information may be information obtained by processing the evaluation information corresponding to the multiple time units. For example, the statistical information may be the variance, mean, etc. of the evaluation information corresponding to the multiple time units. Other statistical information may also be used, and this solution does not impose any restrictions on this.

[0239] Optionally, the evaluation association information may also be evaluation information corresponding to multiple time units within the periodically returned time window and / or statistical information of the evaluation information corresponding to the multiple time units, etc., which is not limited in this solution.

[0240] In a possible implementation, the second request further includes the above-mentioned time window.

[0241] 502. The first network element receives the second request sent by the second network element.

[0242] 503. The first network element returns evaluation association information to the second network element according to the second information.

[0243] The first network element obtains evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units.

[0244] The first network element may calculate the evaluation association information by itself, or may obtain the evaluation association information from a third network element, and this solution does not impose any limitation on this.

[0245] That is, the first network element may first obtain the evaluation information corresponding to the multiple time units within the above time window, and then return the obtained evaluation information corresponding to the multiple time units to the second network element.

[0246] Alternatively, the first network element may first obtain evaluation information corresponding to multiple time units within the above time window, and then process the evaluation information corresponding to the multiple time units to obtain statistical information, and then return the statistical information to the second network element.

[0247] For the specific implementation of obtaining the evaluation information, please refer to the description in the embodiment shown in FIG2 , which will not be repeated here.

[0248] 504. The second network element receives the evaluation association information returned by the first network element.

[0249] In one possible implementation, the second network element determines whether the first network element meets the requirements based on the evaluation association information. For example, the second network element selects an appropriate first network element and its corresponding model, service, etc. based on the evaluation association information returned by multiple first network elements. For example, the second network element selects the first network element with the smallest variance among the variances of the evaluation information returned by the multiple first network elements for service and cancels subscriptions to analysis services from other first network elements.

[0250] In a possible implementation, the second request further includes first information, where the first information includes preset parameters.

[0251] For example, the preset parameter is one or more of a classification error cost matrix and a category weight.

[0252] For the introduction of this part, please refer to the description of the embodiment shown in FIG2 , which will not be repeated here.

[0253] In an embodiment of the present application, a second network element sends a request to a first network element, the request including second information, the second information being used to instruct the return of evaluation-related information, the evaluation-related information including evaluation information corresponding to multiple time units within a time window and / or statistical information of the evaluation information corresponding to the multiple time units, the evaluation information being used to evaluate a service or model; and the first network element then returns the evaluation-related information to the second network element based on the second information. In this way, the evaluation-related information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element select an appropriate first network element based on the fluctuations.

[0254] It should be noted that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0255] The above describes in detail the method of the embodiment of the present application, and the following provides the device of the embodiment of the present application. It will be understood that in the various device embodiments of the present application, the division of multiple units or modules is only a logical division based on function, and is not intended to limit the specific structure of the device. In a specific implementation, some functional modules may be subdivided into more small functional modules, and some functional modules may be combined into one functional module, but no matter whether these functional modules are subdivided or combined, the general process performed by the device is the same. For example, some devices include a receiving unit and a sending unit. In some designs, the sending unit and the receiving unit can also be integrated into a communication unit, which can implement the functions implemented by the receiving unit and the sending unit. Typically, each unit corresponds to its own program code (or program instructions), and when the program code corresponding to each of these units runs on the processor, the unit is controlled by the processing unit to execute the corresponding process to implement the corresponding function.

[0256] Embodiments of the present application also provide an apparatus for implementing any of the above methods. For example, an apparatus for determining evaluation information includes a module (or means) for implementing each step performed by a first network element (e.g., an ANLF NWDAF) in any of the above methods. For another example, another apparatus for determining evaluation information includes a module (or means) for implementing each step performed by a second network element (e.g., a service consumer) in any of the above methods.

[0257] For example, FIG6 is a schematic diagram of a structure of an evaluation information determination device provided in an embodiment of the present application. The evaluation information determination device is used to implement the aforementioned evaluation information determination method, such as the evaluation information determination method shown in FIG2, FIG3 and FIG4.

[0258] As shown in FIG6 , the apparatus may include a transceiver module 601 and a processing module 602 , specifically as follows:

[0259] The transceiver module 601 is configured to receive a first request sent by a second network element, where the first request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information;

[0260] A processing module 602 is configured to obtain evaluation information based on the first information, where the evaluation information is used to evaluate a service or model;

[0261] The transceiver module 601 is further configured to return the evaluation information to the second network element.

[0262] In a possible implementation, the preset parameters are a classification error cost matrix and / or a category weight.

[0263] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

[0264] In a possible implementation, the processing module 602 is configured to obtain the evaluation information according to the first information and a test data set.

[0265] In a possible implementation, the test data set is obtained according to a test data set identifier, wherein the first request further includes the test data set identifier.

[0266] In a possible implementation, the transceiver module 601 is further configured to return the correlation degree between the evaluation information and the second network element to the second network element.

[0267] In a possible implementation, the degree of correlation is a proportion of data related to the second network element in a test data set.

[0268] In a possible implementation manner, the test data set is a data set related to the second network element.

[0269] In a possible implementation, the processing module 602 is configured to send the first information to a third network element; and receive evaluation information sent by the third network element.

[0270] In one possible implementation, the first request also includes second information, and the second information is used to indicate the return of evaluation-related information, wherein the evaluation-related information includes evaluation information corresponding to multiple time units within the time window and / or statistical information of the evaluation information corresponding to the multiple time units; the transceiver module 601 is also used to return the evaluation-related information to the second network element based on the second information.

[0271] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.

[0272] 7 is a schematic diagram of another apparatus for determining evaluation information provided in an embodiment of the present application. The apparatus is used to implement the aforementioned evaluation information determination method, such as the evaluation information determination method shown in FIG2, FIG3, and FIG4.

[0273] As shown in FIG7 , the device may include a transceiver module 701 , specifically as follows:

[0274] The transceiver module 701 is configured to send a first request to a first network element, where the first request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information;

[0275] The transceiver module 701 is further used to receive evaluation information returned by the first network element, where the evaluation information is obtained by the first network element based on the first information, and is used to evaluate a service or model.

[0276] In a possible implementation, the preset parameters are a classification error cost matrix and / or a category weight.

[0277] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

[0278] In a possible implementation manner, the evaluation information is acquired by the first network element according to the first information and a test data set.

[0279] In a possible implementation, the test data set is obtained according to a test data set identifier, wherein the first request further includes the test data set identifier.

[0280] In a possible implementation, the transceiver module 701 is further configured to receive the evaluation information returned by the first network element and the correlation degree between the evaluation information and the second network element.

[0281] In a possible implementation, the degree of correlation is a proportion of data related to the second network element in a test data set.

[0282] In a possible implementation manner, the test data set is a data set related to the second network element.

[0283] In a possible implementation, the first request further includes second information, where the second information is used to instruct to return evaluation-related information, where the evaluation-related information includes evaluation information corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information corresponding to the plurality of time units.

[0284] The transceiver module 701 is further configured to receive the evaluation association information returned by the first network element.

[0285] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.

[0286] For another example, reusing FIG. 7 , the evaluation information determination apparatus is used to implement the aforementioned evaluation information determination method, such as the evaluation information determination method shown in FIG. 5 .

[0287] The transceiver module 701 is configured to receive a second request sent by a second network element, where the second request includes second information, where the second information is used to instruct the return of evaluation-related information, where the evaluation-related information includes evaluation information corresponding to multiple time units within a time window and / or statistical information of the evaluation information corresponding to the multiple time units, where the evaluation information is used to evaluate a service or model.

[0288] The transceiver module 701 is further configured to return evaluation association information to the second network element according to the second information.

[0289] In a possible implementation, the second request further includes the time window.

[0290] In a possible implementation, the second request further includes first information, where the first information includes preset parameters, and the preset parameters are used to generate evaluation information; and further includes a processing module, which is used to obtain evaluation information according to the first information.

[0291] In a possible implementation, the preset parameters are a classification error cost matrix and / or a category weight.

[0292] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

[0293] In a possible implementation manner, the processing module is configured to obtain the evaluation information according to the first information and a test data set.

[0294] In a possible implementation manner, the processing module is configured to: send the first information to a third network element; and receive evaluation information sent by the third network element.

[0295] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.

[0296] For another example, Figure 7 is a schematic diagram of another evaluation information determination device provided in an embodiment of the present application. The evaluation information determination device is used to implement the aforementioned evaluation information determination method, such as the evaluation information determination method shown in Figure 5 .

[0297] The transceiver module 701 is configured to send a second request to the first network element, where the second request includes second information, where the second information is used to instruct the return of evaluation-related information, where the evaluation-related information includes evaluation information corresponding to multiple time units within a time window and / or statistical information of the evaluation information corresponding to the multiple time units, where the evaluation information is used to evaluate a service or model;

[0298] The transceiver module 701 is further configured to receive the evaluation association information returned by the first network element.

[0299] In a possible implementation, the second request further includes the time window.

[0300] In a possible implementation, the second request further includes first information, where the first information includes preset parameters, and the preset parameters are used to generate evaluation information, where the evaluation information is obtained by the first network element according to the first information.

[0301] In a possible implementation, the preset parameters are a classification error cost matrix and / or a category weight.

[0302] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

[0303] In a possible implementation manner, the evaluation information is acquired by the first network element according to the first information and a test data set.

[0304] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.

[0305] For the introduction of the above modules, please refer to the introduction of the above methods, which will not be repeated here.

[0306] It should be understood that the division of the modules in the above-mentioned devices is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or they can be physically separated. In addition, the modules in the evaluation information determination device can be implemented in the form of a processor calling software; for example, the evaluation information determination device includes a processor, the processor is connected to a memory, and the memory stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the modules of the device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory within the device or a memory outside the device. Alternatively, the modules in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), which realizes the functions of some or all of the above units by designing the logical relationship of the components in the circuit. For another example, in another implementation, the hardware circuit can be implemented by a programmable logic device (PLD). Taking a field programmable gate array (FPGA) as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, thereby realizing the functions of some or all of the above units. All modules of the above devices can be implemented in the form of software called by the processor, or in the form of hardware circuits, or in part by the form of software called by the processor, and the rest by hardware circuits.

[0307] Referring to FIG8 , a schematic diagram of the hardware structure of another device for determining evaluation information provided in an embodiment of the present application is shown. The device 800 for determining evaluation information shown in FIG8 (which may be a computer device) includes a memory 801, a processor 802, a communication interface 803, and a bus 804. The memory 801, processor 802, and communication interface 803 are interconnected via bus 804.

[0308] The memory 801 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).

[0309] The memory 801 can store programs. When the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are used to perform the various steps of the evaluation information determination method of the embodiment of the present application.

[0310] The processor 802 is a circuit with signal processing capabilities. In one implementation, the processor 802 can be a circuit with instruction reading and execution capabilities, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP). In another implementation, the processor 802 can implement certain functions through the logical relationship of a hardware circuit. The logical relationship of the hardware circuit is fixed or reconfigurable. For example, the processor 802 is a hardware circuit implemented by an ASIC or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration file and implementing the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. The processor 802 is used to execute relevant programs to implement the functions required to be performed by the units in the evaluation information determination device of the embodiment of the present application, or to perform the evaluation information determination method of the method embodiment of the present application.

[0311] It can be seen that each module in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0312] In addition, the modules in the above device can be fully or partially integrated together, or can be implemented independently. In one implementation, these modules are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the modules of the device. The type of the at least one processor can be different, for example, including a CPU and FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0313] The communication interface 803 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 800 and other devices or a communication network. For example, data can be obtained through the communication interface 803 .

[0314] The bus 804 may include a path for transmitting information between various components of the device 800 (eg, the memory 801 , the processor 802 , and the communication interface 803 ).

[0315] It should be noted that although the device 800 shown in FIG8 only shows a memory, a processor, and a communication interface, during the specific implementation process, those skilled in the art will understand that the device 800 also includes other components necessary for normal operation. At the same time, according to specific needs, those skilled in the art will understand that the device 800 may also include hardware components that implement other additional functions. In addition, those skilled in the art will understand that the device 800 may also include only the components necessary to implement the embodiments of the present application, and does not necessarily include all the components shown in FIG8.

[0316] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or processor executes one or more steps in any of the above methods.

[0317] The present application also provides a computer program product comprising instructions, which, when executed on a computer or processor, causes the computer or processor to execute one or more steps in any of the above methods.

[0318] It should be understood that in the description of this application, unless otherwise specified, " / " indicates that the objects associated with each other are in an "or" relationship. For example, A / B can mean A or B; where A and B can be singular or plural. Also, in the description of this application, unless otherwise specified, "multiple" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural. In addition, to facilitate the clear description of the technical solutions of the embodiments of this application, in the embodiments of this application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity or execution order, and words such as "first" and "second" do not necessarily mean different. At the same time, in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner to facilitate understanding.

[0319] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, and can be electrical, mechanical or other forms.

[0320] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0321] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic medium such as a floppy disk, a hard disk, a tape, a magnetic disk, or an optical medium such as a digital versatile disc (DVD), or a semiconductor medium such as a solid state disk (SSD).

[0322] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for determining evaluation information, characterized in that: include: Sending a first request to a first network element, where the first request includes first information, where the first information includes preset parameters, and where the preset parameters are used to generate evaluation information; Receive evaluation information returned by the first network element, where the evaluation information is obtained by the first network element based on the first information, and the evaluation information is used to evaluate a service or model.

2. The method according to claim 1, characterized in that The preset parameters are classification error cost matrix and / or category weights.

3. The method according to claim 1 or 2, characterized in that: The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

4. The method according to any one of claims 1 to 3, characterized in that: The evaluation information is acquired by the first network element according to the first information and a test data set.

5. The method according to claim 4, characterized in that The test data set is obtained according to a test data set identifier, wherein the first request also includes the test data set identifier.

6. The method according to claim 4, characterized in that The method further comprises: Receive the correlation degree between the evaluation information returned by the first network element and the second network element, wherein the correlation degree is the proportion of data related to the second network element in the test data set.

7. The method according to claim 4, characterized in that The test data set is a data set related to the second network element.

8. The method according to any one of claims 1 to 7, characterized in that: The first request further includes second information, where the second information is used to indicate that evaluation-related information is returned, where the evaluation-related information includes evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units; The receiving the evaluation information returned by the first network element includes: Receive the evaluation association information returned by the first network element.

9. The method according to any one of claims 1 to 8, characterized in that: The service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide analysis services.

10. A method for determining evaluation information, characterized in that: include: receiving a first request sent by a second network element, wherein the first request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information; Acquire evaluation information according to the first information, wherein the evaluation information is used to evaluate a service or a model; The evaluation information is returned to the second network element.

11. The method according to claim 10, characterized in that The preset parameters are classification error cost matrix and / or category weights.

12. The method according to claim 10 or 11, characterized in that: The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

13. The method according to any one of claims 10 to 12, characterized in that: The obtaining of evaluation information according to the first information includes: The evaluation information is acquired according to the first information and a test data set.

14. The method according to claim 13, characterized in that The test data set is obtained according to a test data set identifier, wherein the first request also includes the test data set identifier.

15. The method according to claim 13, characterized in that The method further comprises: Return the relevance degree of the evaluation information to the second network element to the second network element, wherein the relevance degree is the proportion of data related to the second network element in the test data set.

16. The method according to claim 13, characterized in that The test data set is a data set related to the second network element.

17. The method according to any one of claims 10 to 16, characterized in that The obtaining of evaluation information according to the first information includes: Sending the first information to a third network element; Receive evaluation information sent by the third network element.

18. The method according to any one of claims 10 to 17, characterized in that The first request further includes second information, where the second information is used to indicate that evaluation-related information is returned, where the evaluation-related information includes evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units; The returning the evaluation information to the second network element includes: Evaluation association information is returned to the second network element according to the second information.

19. The method according to any one of claims 10 to 18, characterized in that The service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.

20. A method for determining evaluation information, characterized in that: include: Sending a second request to the first network element, the second request including second information, the second information being used to instruct the first network element to return evaluation-related information, the evaluation-related information including evaluation information respectively corresponding to a plurality of time units within a time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units, the evaluation information being used to evaluate a service or model; Receive the evaluation association information returned by the first network element.

21. The method according to claim 20, characterized in that The second request also includes the time window.

22. The method according to claim 20 or 21, characterized in that The second request also includes first information, the first information includes preset parameters, the preset parameters are used to generate evaluation information, and the evaluation information is obtained by the first network element according to the first information.

23. The method according to claim 22, characterized in that The preset parameters are classification error cost matrix and / or category weights.

24. The method according to claim 22 or 23, characterized in that The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

25. The method according to any one of claims 22 to 24, characterized in that The evaluation information is acquired by the first network element according to the first information and a test data set.

26. The method according to any one of claims 20 to 25, characterized in that The service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide analysis services.

27. A method for determining evaluation information, characterized in that: include: receiving a second request sent by a second network element, the second request including second information, the second information being used to indicate returning evaluation-related information, the evaluation-related information including evaluation information respectively corresponding to a plurality of time units within a time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units, the evaluation information being used to evaluate a service or model; Evaluation association information is returned to the second network element according to the second information.

28. The method according to claim 27, characterized in that The second request also includes the time window.

29. The method according to claim 27 or 28, characterized in that The second request further includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information. The method further includes: Evaluation information is acquired according to the first information.

30. The method according to claim 29, characterized in that The preset parameters are classification error cost matrix and / or category weights.

31. The method according to claim 29 or 30, characterized in that The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

32. The method according to any one of claims 29 to 31, characterized in that The obtaining of evaluation information according to the first information includes: The evaluation information is acquired according to the first information and a test data set.

33. The method according to any one of claims 29 to 32, characterized in that The obtaining of evaluation information according to the first information includes: Sending the first information to a third network element; Receive evaluation information sent by the third network element.

34. The method according to any one of claims 27 to 33, characterized in that The service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.

35. A method for determining evaluation information, characterized in that: include: The second network element sends a first request to the first network element, where the first request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information; The first network element receives the first request sent by the second network element; The first network element obtains evaluation information according to the first information, where the evaluation information is used to evaluate a service or a model; The first network element returns the evaluation information to the second network element; The second network element receives the evaluation information returned by the first network element.

36. The method according to claim 35, characterized in that The preset parameters are classification error cost matrix and / or category weights.

37. The method according to claim 35 or 36, characterized in that The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

38. The method according to any one of claims 35 to 37, characterized in that The first request further includes second information, where the second information is used to indicate that evaluation-related information is returned, where the evaluation-related information includes evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units; The first network element returns the evaluation information to the second network element, including: The first network element returns the evaluation association information to the second network element; The second network element receiving the evaluation information returned by the first network element includes: The second network element receives the evaluation association information returned by the first network element.

39. A method for determining evaluation information, characterized in that: include: The second network element sends a second request to the first network element, where the second request includes second information, where the second information is used to instruct the first network element to return evaluation-related information, where the evaluation-related information includes evaluation information corresponding to a plurality of time units within a time window and / or statistical information of the evaluation information corresponding to the plurality of time units, respectively, where the evaluation information is used to evaluate a service or a model; The first network element receives the second request sent by the second network element; The first network element returns evaluation association information to the second network element according to the second information; The second network element receives the evaluation association information returned by the first network element.

40. The method according to claim 39, characterized in that The second request further includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information. The method further includes: The first network element obtains evaluation information according to the first information.

41. The method according to claim 40, characterized in that The preset parameters are classification error cost matrix and / or category weights.

42. The method according to claim 40 or 41, characterized in that The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

43. An evaluation information determination device, characterized in that: Comprising a unit or module for executing the method according to any one of claims 1 to 34.

44. An evaluation information determination system, characterized in that Comprising a unit or module for executing the method described in any one of claims 1 to 9, and a unit or module for executing the method described in any one of claims 10 to 19; or, comprising a unit or module for executing the method described in any one of claims 20 to 26, and a unit or module for executing the method described in any one of claims 27 to 34.

45. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program instructions, and when the program instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 34.

46. ​​A computer program product comprising program instructions, characterized in that When the program instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 34.

Citation Information

Patent Citations

  • Evaluation information determination method, related system and storage medium

    CN118945068A

  • NWDAF network element selection method and device, electronic equipment and readable storage medium

    CN110798360A

  • Model training method and device

    CN113128686A

  • Data analysis method, device and system

    CN115004653A

  • Performance evaluation method and device

    CN115835266A