Evaluation information determination method, related system and storage medium
By passing preset parameters between the first network element and the second network element to generate evaluation information, the problem that consumers cannot choose suitable NWDAF services is solved, and the function of selecting NWDAF according to consumer evaluation standards is realized, which improves the accuracy of services. Sex and efficiency.
Patent Information
- Application Number
- CN202310534752.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-12
AI Technical Summary
Consumers cannot choose appropriate network data analysis function (NWDAF) services based on accuracy information because there is a lack of clear criteria for generating accuracy information, and different consumers have different criteria for evaluating the same service or model.
By passing preset parameters, such as classification error cost matrix and category weights between the first network element and the second network element, evaluation information is generated, thereby helping consumers choose NWDAF services that meet their needs.
It realizes the selection of appropriate NWDAF services based on consumer-specific evaluation criteria, improves the accuracy and efficiency of services, and meets the diversified needs of different consumers for accuracy and performance.
Smart Images

Figure CN118945068A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of communication technology, and in particular to a method for determining evaluation information and a related system and storage medium. Background Art
[0002] Among them, the machine learning (ML) model combined with the network data analytics function (NWDAF) can provide prediction services in different scenarios. Currently, service consumers select an appropriate NWDAF with a model analysis logical function (AnLF) and subscribe to the analysis accuracy information by calling the network data analytics function Nnwdaf_AnalyticsSubscription_Subscribe service operation so that consumers can check and deploy the accuracy information. However, the service (such as the analysis identification (ID)) or the accuracy information of the model obtained by the consumer contains the preferred level of accuracy of the analytics ("Low", "Medium", "High" or "Highest"), which only defines the accuracy level and the accuracy information value, but does not clarify the generation standard of the accuracy information.
[0003] Since consumers do not know the generation standard of accuracy information, they cannot select a more appropriate NWDAF to provide services based on the information. At the same time, for example, for the same analysis ID, different consumers may have different evaluation standards, so different consumers cannot select a suitable NWDAF based only on accuracy information. Summary of the invention
[0004] The present application discloses an evaluation information determination method and a related system and storage medium, which can help service consumers select a suitable NWDAF based on evaluation information generation standards determined by them.
[0005] In a first aspect, an embodiment of the present application provides a method for determining evaluation information, comprising: a first network element receives a first request sent by a second network element. The first request includes first information. The first information includes preset parameters. The preset parameters are used to generate evaluation information. Then, the first network element obtains evaluation information based on the first information. The evaluation information is used to evaluate a service or model. Then, the first network element returns the evaluation information to the second network element.
[0006] In the embodiment of the present application, the first network element obtains evaluation information based on the first information from the second network element, and then returns the evaluation information to the second network element. With this method, since the preset parameters are used to generate the evaluation information, which are set by the second network element, the first network element obtains the evaluation information based on the preset parameters, which can help the second network element select a suitable first network element based on the received evaluation information.
[0007] In this example, the first network element is used as an example for introduction. The first network element can be understood as one network element, or as a combination of multiple network elements, etc., and this solution does not limit this.
[0008] In a possible implementation, the preset parameter is a classification error cost matrix. The classification error cost can be understood as a measure of the loss of consumers for receiving a category prediction error. The classification error cost matrix can be used for classification tasks of various analysis services. For example, network element load analysis, abnormal behavior prediction, measurement trend, etc.
[0009] Different consumers may have different prediction error cost matrices for the same service or model. Therefore, by clarifying the preset parameters, evaluation information that meets the needs of the consumer can be obtained.
[0010] In another possible implementation, the preset parameter is a category weight. The category weight is a measure of the importance of a service (such as an analysis ID) or an output category of a model to a consumer. The specific standard for measuring the importance is not limited in this solution, and it can be a relative measure or an absolute measure. Optionally, for abnormal behavior analysis, it supports mobility-related anomalies and session-related anomalies. Mobility-related anomalies include: abnormal UE location, abnormal UE wakeup, abnormal wireless connection errors, etc. Session-related anomalies include: abnormally long or high-rate traffic, abnormal wakeup, suspected distributed denial of service attack (DDoS), wrong destination address, too frequent service access, etc.
[0011] For example, the prediction category of a certain analysis service is the above abnormal behavior category. Since different consumer network elements have different concerns about the above abnormal behavior category, the category weights in their setting services are different.
[0012] In yet another possible implementation, the preset parameters are a classification error cost matrix and a category weight.
[0013] In a possible implementation, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameter. The evaluation information calculation method is associated with the preset parameter.
[0014] The evaluation information calculation method can be understood as a formula or text for calculating the evaluation information. The evaluation information calculation method corresponds to one or more of the accuracy, precision, recall rate, classification report, loss function, etc. of the evaluation service or model.
[0015] It is understandable that the first information may correspond to an information element (IE). For example, an IE includes an accuracy calculation formula, a loss function, a classification report, and the loss value calculation method includes a category weight, etc.
[0016] Alternatively, the first information may correspond to multiple information element IEs. For example, it corresponds to two IEs. One of the IEs contains the loss value calculation method, and the other IE contains the category weight.
[0017] Of course, other methods may be used to indicate the first information, and this solution does not limit this.
[0018] Optionally, the evaluation information calculation method is F1-score, loss function, classification report, etc. The types of loss functions include: mean square error loss, weighted mean square error loss, cross entropy loss, cost loss, etc. The classification report indicates the classification results of NWDAF on the test data set. Consumers can evaluate the performance of the model based on the classification report and internal configuration. For example, the accuracy and recall rate of the model are calculated according to the classification report. The preset parameters can be the number limit of samples in each category. For example, the proportion of samples in each category, the minimum number of samples, etc.
[0019] The evaluation information can be used to measure the performance of the service or model, such as accuracy, precision, recall rate, classification report, loss value, etc. It can be understood that the service is an analysis service provided by the first network element. For example, analyzing the quality of service (Quality of Service, QoS), analyzing the network element load, analyzing abnormal behavior, etc. The model can be a machine learning model used by the first network element to provide analysis services. For example, the model is a model obtained by the first network element from the third network element.
[0020] In a possible implementation manner, the first network element further obtains evaluation information according to the first information and the test data set.
[0021] Among them, test data refers to data used to independently evaluate trained and verified artificial intelligence systems (services or models) in order to confirm their expected performance before using the service or model.
[0022] In a possible implementation manner, the test data set is obtained according to a test data set identifier, wherein the first request also includes the test data set identifier.
[0023] In a possible implementation manner, the first network element returns the correlation degree between the evaluation information and the second network element to the second network element.
[0024] Optionally, the degree of relevance is a proportion of data related to the second network element in the test data set.
[0025] In a possible implementation, the data related to the second network element is real data of a service subscribed to by the second network element or a prediction result of a model, and the relevant proportion is the proportion of the real data in the entire test data set.
[0026] In another possible implementation manner, the test data set is a data set related to the second network element.
[0027] In a possible implementation manner, the first network element sends the first information to the third network element. Furthermore, the first network element receives the evaluation information sent by the third network element.
[0028] That is to say, the evaluation information is calculated by the third network element.
[0029] In a possible implementation, the first request also includes second information, and the second information is used to indicate the return of evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units in the time window. Alternatively, the evaluation-related information includes statistical information of the evaluation information corresponding to multiple time units in the time window. Alternatively, the evaluation-related information includes evaluation information corresponding to multiple time units in the time window, and statistical information of the evaluation information corresponding to the multiple time units. The first network element returns the evaluation-related information to the second network element based on the second information.
[0030] This time window, also called the time interval [start.end], can be understood as a start time and an end time in the past, or a start time and an end time in the future. The time interval can be expressed in terms of the actual start time and the actual end time (for example, in Coordinated Universal Time Coordinated (UTC) time). By setting the start time and the end time to the same value, the user of the analysis can request or subscribe to analysis for a specific time.
[0031] The time unit can be understood as a plurality of time periods obtained by dividing the above time window. The multiple time units can be of the same size or different sizes, and this scheme does not limit this. For example, the above time window is: the start time is 12:00 on January 1, 2022, and the end time is 12:00 on February 1, 2022. The time unit can be 1 day, 6 hours, 7 days, one day, one week, one month, etc. Alternatively, the time unit is a series of time points to divide the above time window, for example: the time point is 00:00 on January 14, 2022, then the above time window is divided into two time periods. By analogy, there can be multiple time points.
[0032] The statistical information may be information obtained by processing the evaluation information corresponding to the multiple time units. For example, the statistical information is the variance, mean, etc. of the evaluation information corresponding to the multiple time units. It may also be other statistical information, which is not limited in this solution.
[0033] In this way, the evaluation association information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element to select a suitable first network element based on the fluctuations.
[0034] In a second aspect, an embodiment of the present application provides a method for determining evaluation information, comprising: a second network element sends a first request to a first network element. The first request includes first information. The first information includes preset parameters. The preset parameters are used to generate evaluation information. Then, the second network element receives the evaluation information returned by the first network element. The evaluation information is obtained by the first network element based on the first information. The evaluation information is used to evaluate a service or model.
[0035] In an embodiment of the present application, the second network element sends a first request to the first network element, the first request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information; and then receives the evaluation information returned by the first network element. By adopting this means, the evaluation information is generated based on the preset parameters in the first information sent by the second network element. In this way, the second network element knows the evaluation information generation standard, which can help the second network element select a suitable first network element.
[0036] In a possible implementation manner, the preset parameter is a classification error cost matrix and / or a category weight.
[0037] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameter.
[0038] In a possible implementation manner, the evaluation information is acquired by the first network element according to the first information and a test data set.
[0039] In a possible implementation manner, the test data set is obtained according to a test data set identifier, wherein the first request also includes the test data set identifier.
[0040] In a possible implementation manner, the second network element further receives the correlation degree between the evaluation information returned by the first network element and the second network element.
[0041] In a possible implementation, the correlation degree is a proportion of data related to the second network element in the test data set, for example, a proportion size.
[0042] In another possible implementation manner, the test data set is a data set related to the second network element.
[0043] In a possible implementation, the first request also includes second information, and the second information is used to indicate the return of evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units in the time window. Alternatively, the evaluation-related information includes statistical information of the evaluation information corresponding to multiple time units in the time window. Alternatively, the evaluation-related information includes evaluation information corresponding to multiple time units in the time window, and statistical information of the evaluation information corresponding to the multiple time units. The second network element also receives the evaluation-related information returned by the first network element.
[0044] In this way, the evaluation association information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element to select a suitable first network element based on the fluctuations.
[0045] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.
[0046] In a third aspect, an embodiment of the present application provides a method for determining evaluation information, comprising: a first network element receives a second request sent by a second network element. The second request includes second information. The second information is used to indicate the return of evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units in a time window. Alternatively, the evaluation-related information includes statistical information of evaluation information corresponding to multiple time units in a time window. Alternatively, the evaluation-related information includes evaluation information corresponding to multiple time units in a time window, and statistical information of evaluation information corresponding to the multiple time units. The evaluation information is used to evaluate a service or model. Then, the first network element returns the evaluation-related information to the second network element based on the second information.
[0047] In this way, the evaluation association information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element to select a suitable first network element based on the fluctuations.
[0048] This time window, also called the time interval [start.end], can be understood as a start time and an end time in the past, or a start time and an end time in the future. The time interval can be expressed in terms of the actual start time and the actual end time (for example, in Coordinated Universal Time Coordinated (UTC) time). By setting the start time and the end time to the same value, the user of the analysis can request or subscribe to analysis for a specific time.
[0049] The time unit can be understood as a plurality of time periods obtained by dividing the above time window. The multiple time units can be of the same size or different sizes, and this scheme does not limit this. For example, the above time window is: the start time is 12:00 on January 1, 2022, and the end time is 12:00 on February 1, 2022. The time unit can be 1 day, 6 hours, 7 days, one day, one week, one month, etc. Alternatively, the time unit is a series of time points to divide the above time window, for example: the time point is 00:00 on January 14, 2022, then the above time window is divided into two time periods. By analogy, there can be multiple time points.
[0050] The statistical information may be information obtained by processing the evaluation information corresponding to the multiple time units. For example, the statistical information is the variance, mean, etc. of the evaluation information corresponding to the multiple time units. It may also be other statistical information, which is not limited in this solution.
[0051] In a possible implementation manner, the second request also includes the time window.
[0052] Optionally, the second information may also indicate periodically returning the above-mentioned evaluation-related information.
[0053] In a possible implementation, the second request further includes first information. The first information includes preset parameters. The preset parameters are used to generate evaluation information. The first network element further obtains evaluation information according to the first information.
[0054] In a possible implementation manner, the preset parameter is a classification error cost matrix and / or a category weight.
[0055] In a possible implementation, the first information further includes an evaluation information calculation method, wherein the evaluation information calculation method includes the above-mentioned preset parameters.
[0056] In a possible implementation manner, the first network element obtains evaluation information according to the first information and the test data set.
[0057] In a possible implementation manner, the first network element sends the first information to the third network element, and then the first network element receives the evaluation information sent by the third network element.
[0058] In one possible implementation, the above-mentioned service is an analysis service provided by the first network element, and the above-mentioned model is a machine learning model used by the first network element to provide the analysis service.
[0059] In a fourth aspect, an embodiment of the present application provides a method for determining evaluation information, comprising: a second network element sends a second request to a first network element. The second request includes second information. The second information is used to indicate the return of evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units within a time window. Alternatively, the evaluation-related information includes statistical information of evaluation information corresponding to multiple time units within a time window. Alternatively, the evaluation-related information includes evaluation information corresponding to multiple time units within a time window, and statistical information of evaluation information corresponding to the multiple time units. The evaluation information is used to evaluate a service or model. Then, the second network element receives the evaluation-related information returned by the first network element.
[0060] In an embodiment of the present application, the second network element sends a request to the first network element, the request includes second information, the second information is used to indicate the return of evaluation-related information, the evaluation-related information includes evaluation information corresponding to multiple time units in a time window and / or statistical information of the evaluation information corresponding to the multiple time units, the evaluation information is used to evaluate a service or model; and then the first network element returns the evaluation-related information to the second network element according to the second information. In this way, the evaluation-related information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element select a suitable first network element based on the fluctuations.
[0061] In a possible implementation manner, the second request also includes a time window.
[0062] In a possible implementation, the second request further includes first information. The first information includes preset parameters. The preset parameters are used to generate evaluation information. The evaluation information is obtained by the first network element according to the first information.
[0063] In a possible implementation manner, the preset parameter is a classification error cost matrix and / or a category weight.
[0064] In a possible implementation, the first information further includes an evaluation information calculation method. The evaluation information calculation method includes the above-mentioned preset parameters.
[0065] In a possible implementation manner, the evaluation information is acquired by the first network element according to the first information and a test data set.
[0066] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.
[0067] In a fifth aspect, the present application provides a method for determining evaluation information, comprising: a second network element sends a first request to a first network element. The first request includes first information. The first information includes preset parameters. The preset parameters are used to generate evaluation information. The first network element receives the first request sent by the second network element. Then, the first network element obtains evaluation information based on the first information, the evaluation information is used to evaluate a service or model, and returns the evaluation information to the second network element. The second network element receives the evaluation information returned by the first network element.
[0068] In an embodiment of the present application, the second network element sends a request to the first network element, the request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information; the first network element obtains evaluation information based on the first information, and the evaluation information is used to evaluate the service or model; and then returns the evaluation information to the second network element. By adopting this method, the evaluation information is generated based on the preset parameters in the first information sent by the second network element. In this way, the second network element knows the evaluation information generation standard, which can help the second network element select a suitable first network element.
[0069] In a possible implementation manner, the preset parameter is a classification error cost matrix and / or a category weight.
[0070] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
[0071] In a possible implementation, the first request also includes second information. The second information is used to indicate the return of evaluation association information, wherein the evaluation association information includes evaluation information corresponding to multiple time units in a time window and / or statistical information of the evaluation information corresponding to the multiple time units. The first network element also returns the evaluation association information to the second network element. The second network element also receives the evaluation association information returned by the first network element.
[0072] In a sixth aspect, the present application provides a method for determining evaluation information, comprising: a second network element sends a second request to a first network element. The second request includes second information. The second information is used to instruct the first network element to return evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units in a time window and / or statistical information of the evaluation information corresponding to the multiple time units. The evaluation information is used to evaluate a service or model. The first network element receives the second request sent by the second network element. The first network element returns the evaluation-related information to the second network element based on the second information. Then, the second network element receives the evaluation-related information returned by the first network element.
[0073] In an embodiment of the present application, the second network element sends a request to the first network element, the request includes second information, the second information is used to indicate the return of evaluation-related information, the evaluation-related information includes evaluation information corresponding to multiple time units in a time window and / or statistical information of the evaluation information corresponding to the multiple time units, the evaluation information is used to evaluate a service or model; and then the first network element returns the evaluation-related information to the second network element according to the second information. In this way, the evaluation-related information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element select a suitable first network element based on the fluctuations.
[0074] In a possible implementation, the second request further includes first information. The first information includes preset parameters, and the preset parameters are used to generate evaluation information. The first network element further obtains evaluation information according to the first information.
[0075] In a possible implementation manner, the preset parameter is a classification error cost matrix and / or a category weight.
[0076] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
[0077] In a seventh aspect, the present application provides an evaluation information determination device, which may be any one of NWDAF, NF, AF, or a chip thereof. The evaluation information determination device includes a unit and / or module for executing the evaluation information determination method provided by any one of the first to sixth aspects, or any possible implementation of any one of the aspects, such as a transceiver unit and / or a processing unit, and the transceiver unit is used to send and receive various information or signaling, so that the beneficial effects (or advantages) of the evaluation information determination method provided in any of the above aspects can also be achieved.
[0078] In an eighth aspect, the present application provides an evaluation information determination device, which may include a processor and a transceiver. The transceiver is used to send and receive various information or signaling, and the computer program includes program instructions. When the processor runs the program instructions, the evaluation information determination device executes the communication method described in any one of the first to sixth aspects, or any possible implementation of any one of the aspects. The transceiver may be a radio frequency module in a communication device, or a combination of a radio frequency module and an antenna, or an input and output interface of a chip or circuit. Optionally, the evaluation information determination device also includes a memory, which is used to store the computer program.
[0079] In a ninth aspect, the present application provides a readable storage medium having program instructions stored thereon, which, when executed on a computer, enables the computer to execute the evaluation information determination method described in any one of the first to sixth aspects, or any possible implementation of any one of the aspects.
[0080] In a tenth aspect, the present application provides a program product comprising instructions, which, when executed, enables the evaluation information determination method described in any possible implementation of any one of the first to sixth aspects to be executed.
[0081] In an eleventh aspect, the present application provides a device, which can be implemented in the form of a chip or in the form of a device, and the device includes a processor. The processor is used to read and execute a program stored in a memory to execute one or more of any aspects of the first to sixth aspects, or one or more of any possible implementations of any aspect, and a method for determining evaluation information provided by one or more of them.
[0082] Optionally, the device also includes a memory, and the memory is connected to the processor via a circuit.
[0083] Further optionally, the device further includes a communication interface, and the processor is connected to the communication interface. The communication interface is used to receive information and / or signaling to be processed, and the processor obtains the information and / or signaling from the communication interface, processes the information and / or signaling, and outputs the processing result through the communication interface. The communication interface can be an input and output interface.
[0084] Optionally, the processor and memory mentioned above may be physically independent units, or the memory may be integrated with the processor.
[0085] In a twelfth aspect, the present application provides an evaluation information determination system, which includes a device corresponding to the evaluation information determination method described in the first aspect to the second aspect, or a device corresponding to the evaluation information determination method described in the third aspect to the fourth aspect.
[0086] It can be understood that the above-mentioned devices, computer storage media or computer program products are used to execute any of the methods provided in the first aspect, any of the methods provided in the second aspect, any of the methods provided in the third aspect, any of the methods provided in the fourth aspect, any of the methods provided in the fifth aspect, and any of the methods provided in the sixth aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0087] The following is an introduction to the drawings used in the embodiments of the present application.
[0088] Figure 1 This is a schematic diagram of an architecture of an NWDAF network element interacting with other network elements provided in an embodiment of the present application;
[0089] Figure 2 It is a flowchart of a method for determining evaluation information provided in an embodiment of the present application;
[0090] Figure 3 It is a flowchart of another method for determining evaluation information provided in an embodiment of the present application;
[0091] Figure 4 It is a flowchart of another method for determining evaluation information provided in an embodiment of the present application;
[0092] Figure 5 It is a flowchart of another method for determining evaluation information provided in an embodiment of the present application;
[0093] Figure 6 is a structural diagram of an evaluation information determination device provided in an embodiment of the present application;
[0094] Figure 7 is a structural schematic diagram of another device for determining evaluation information provided in an embodiment of the present application;
[0095] Figure 8 It is a structural diagram of another evaluation information determination device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0096] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0097] The technical solution provided in the present application can be applied to various communication systems, such as: long term evolution (LTE) system, LTE frequency division duplex (FDD) system, LTE time division duplex (TDD), Internet of Things (IoT) system, narrowband Internet of Things (NB-IoT) system, wireless fidelity (Wi-Fi), fifth generation (5G) communication system or new radio (NR) and other future communication systems, such as the sixth generation mobile communication system.
[0098] The network data analytics function (NWDAF) network element supports collecting data from other network function and application function (AF) network elements; supports collecting data from operation administration and maintenance (OAM) network elements; and supports providing analysis information to other network function and AF network elements.
[0099] The NWDAF network element has the functions of data collection, model training, data analysis, and model reasoning. It can be used to collect relevant data from network network elements, third-party service servers, terminal devices, or network management systems, perform data analysis based on the relevant data to obtain analysis results, and provide analysis results to network network elements, third-party service servers, terminal devices, or network management systems. The analysis results can assist the network in selecting service quality parameters for services, or assist the network in executing traffic routing, or assist the network in selecting background data transmission strategies, etc. In addition, the NWDAF network element can also be used to collect relevant data from network network elements, third-party service servers, terminal devices, or network management systems, and perform model training based on the relevant data to obtain an artificial intelligence (AI) model or a machine learning (ML) model, and provide AI models / ML models to other NWDAF network elements. The AI models / ML models can be used to assist the NWDAF network element in generating data analysis results based on relevant data.
[0100] In the embodiment of the present application, ANLF NWDAF represents an NWDAF including an analytics logical function (AnLF), which is used to perform reasoning, derive analysis information (i.e., derive statistical information and / or prediction according to analysis consumer requests) and disclose analysis services.
[0101] MTLF NWDAF represents a NWDAF that contains a model training logical function (MTLF). It is used to train machine learning ML models and expose new training services, such as providing trained ML models.
[0102] The following will describe the system architecture of the embodiment of the present application in detail with reference to the accompanying drawings. Figure 1 , Figure 1 Schematic diagram of an architecture for NWDAF network element to interact with other network elements applicable to the embodiment of the present application. Figure 1 1(a) shows the architecture used when collecting data, that is, the architecture used when the NWDAF network element calls the services of other 5G core network elements to collect the input information required for network analysis. Figure 1 1(b) shows the architecture used when data is open, that is, the NWDAF network element sends analysis data to network elements that subscribe to or request the analysis data by providing NWDAF services. Among them, NWDAF represents the network analysis logic function managed by the operator. The NWDAF network element provides slice-level network data analysis for other network elements, and provides network analysis information (i.e., load level information) at the network slice instance level without knowing the current subscribers using the slice. Figure 1 The "Any NF" shown in the figure represents any network function element, and "Nnf" and "Nnwdaf" represent the names of service-oriented interfaces. For details, please refer to the relevant description in the 3GPP standard protocol, which will not be elaborated here.
[0103] The architecture of the embodiment of the present application is described above, and the method of the embodiment of the present application is described in detail below.
[0104] Reference Figure 2 FIG. 1 is a flow chart of a method for determining evaluation information provided by an embodiment of the present application. Optionally, the method can be applied to the aforementioned evaluation information determination system, for example Figure 1 The evaluation information determination system shown. Figure 2The evaluation information determination method shown may include steps 201-205. It should be understood that for the convenience of description, this application is described in the order of 201-205, and is not intended to limit the execution to the above order. The embodiment of the present application does not limit the order of execution, execution time, number of executions, etc. of the above one or more steps. The following description takes the execution subject of step 201 and step 205 of the evaluation information determination method as the second network element, such as a service consumer, and the execution subject of steps 202-204 as the first network element, such as ANLF NWDAF as an example. This application is also applicable to other execution subjects. Steps 201-205 are as follows:
[0105] 201. A second network element sends a first request to a first network element, where the first request includes first information, and the first information includes preset parameters, where the preset parameters are used to generate evaluation information.
[0106] The first request may be, for example, a request to subscribe to analysis and evaluation information, such as the analysis and evaluation information subscription service Nnwdaf_AnalyticsSubscription_Subscribe, or other requests, which are not limited in this solution.
[0107] The preset parameters can be used to generate evaluation information. The second network element transmits the preset parameters to the first network element so that the first network element generates evaluation information according to the preset parameters.
[0108] In a possible implementation, the preset parameter is a classification error cost matrix, wherein the classification error cost can be understood as a measure of the cost loss of consumers for receiving a prediction of a category prediction error, as shown in Table 1.
[0109] Table 1
[0110] Category 1 Category 2 Category 1 0 C12 Category 2 C21 0
[0111] Among them, C12 represents: the cost of the model mistakenly predicting that the sample with the true label of 1 is category 2. C21 represents: the cost of the model mistakenly predicting that the sample with the true label of 2 is category 1. The specific error cost calculation method is not limited in this scheme. It can be a relative measure or an absolute measure. For example, for a relative measure, for a certain category, the sum of the prediction error costs of other categories is 1. For an absolute measure, the consumer can generate a classification error cost matrix based on pre-configured information according to the local strategy.
[0112] This classification error cost matrix can be used to evaluate various analysis services or models. For example, network element load analysis, abnormal behavior category prediction, abnormal trend level measurement, etc. At the same time, service consumers can customize the specific values and size ratios of the cost loss of the output category prediction error of the analysis or model. For example, for abnormal behavior analysis, the predicted output is the category of abnormal behavior, which is divided into four levels: high, medium, low, and normal. For a certain analysis or model, the cost loss of predicting a high-level anomaly as a low-level anomaly is 1, but the cost of predicting a high-level anomaly as normal may be different, perhaps 3, etc.
[0113] At the same time, different network elements may have different security requirements. For example, for a certain abnormal behavior, its abnormal level is high, but the prediction result is low. For the Unified Data Management (UDM), because it may have a high security requirement, the cost of prediction error is small. However, for some network elements with lower security requirements, they are only sensitive to high-level abnormal behaviors. When the prediction level is low, the abnormal behavior will be ignored, which will cause security problems and the cost of prediction error is very high. Therefore, the cost of prediction error is different for network elements with different security requirements.
[0114] For another example, for a binary classification problem, the output is two categories. For example, the 5QI of the GBR resource type meets or exceeds the reported QoS flow retention threshold. C1 means satisfied, and C2 means not satisfied. C12 means the cost of the model predicting a satisfied result as an unsatisfied result. C21 means the cost of the model predicting an unsatisfied result as a satisfied result.
[0115] It is understandable that different consumers may have different prediction error cost matrices for the same service or model. Therefore, by clarifying the preset parameters, evaluation information that meets the needs of the consumer can be obtained.
[0116] In another possible implementation, the preset parameter is a category weight. The category weight is a measure of the importance of a service (such as an analysis ID) or a model's output category to a consumer. The specific standard for measuring the importance is not limited in this solution, and it can be a relative measure or an absolute measure. For example, for a relative measure, the sum of the importance measure values of an analysis or model's output category is 1.
[0117] For example, for abnormal behavior analysis services, it supports mobility-related anomalies and session-related anomalies. Mobility-related anomalies include: abnormal UE location, abnormal UE wakeup, abnormal wireless connection errors, etc. Session-related anomalies include: abnormal long or high-rate traffic, abnormal wakeup, suspected distributed denial of service attack (DDoS), wrong destination address, too frequent service access, etc. Since different consumer network elements have different concerns about the above abnormal behavior categories, the category weights in their service settings are different. For example, AMF (Access and mobility management function) is more concerned about the accuracy of unexpected wakeup and unexpected wireless link failure, while Session Management Function (SMF) is more concerned about the accuracy of unexpected long-lived / high-rate flows. Therefore, different consumers have different accuracy requirements for different anomaly types, and AnLF NWDAF needs to select appropriate models based on consumer preference criteria to provide more suitable prediction results.
[0118] In another possible implementation, the preset parameters are a classification error cost matrix and a category weight. The consumer considers both the classification error cost and the category weight.
[0119] The above is only an example. Of course, other parameters are possible and this solution does not limit this.
[0120] In a possible implementation, the first information further includes an evaluation information calculation method. For example, an evaluation information generation standard may be determined based on the evaluation information calculation method and preset parameters.
[0121] The evaluation information calculation method may be understood as a formula or text for calculating the evaluation information, or instruction information (indicating the evaluation information calculation method), etc.
[0122] The evaluation information calculation method includes the above-mentioned preset parameters. It can be understood that, for example, the evaluation information calculation method is a formula. The formula includes items corresponding to the above-mentioned preset parameters. If the preset parameter is a=2, the formula is L=ax.
[0123] It can also be understood that, for example, the evaluation information calculation method is a formula. The formula includes the above preset parameters. If the preset parameter is a=2, the formula is L=2x.
[0124] That is to say, the evaluation information calculation method is related to the above-mentioned preset parameters. The above introduction is only an example, and it can also be other ways, which are not limited by this solution.
[0125] The receiver can uniquely determine the evaluation information calculation method according to the evaluation information calculation method. If the evaluation information calculation method includes pre-configured parameters, the first information also includes the corresponding pre-configured parameters.
[0126] In a possible implementation, the evaluation information calculation method is indication information. The network element preconfigures the correspondence between the indication information and the evaluation information calculation method, and the receiver can uniquely determine the evaluation information calculation method according to the indication information.
[0127] The evaluation information calculation method, for example: one or more of the accuracy, precision, recall rate, classification report, loss function, robustness, resilience, etc. of the evaluation service or model.
[0128] It is understandable that the first information may correspond to an information element (IE). That is, an information element includes both the evaluation information calculation method and the preset parameters. For example, an IE includes an accuracy calculation formula, a loss function, a weighted loss function, and a classification report. Since the weighted loss function has a preset parameter: category weight, the IE will also include the specific value of the category weight.
[0129] Alternatively, the first information may correspond to multiple information elements IE. For example, it corresponds to two IEs. One IE includes the evaluation information calculation method, and the other IE includes preset parameters. For example, IE1 includes a weighted loss function, and IE2 includes a category weight.
[0130] Of course, other methods may be used to indicate the first information, and this solution does not limit this.
[0131] Optionally, the evaluation information calculation method is F1-score, loss function, etc. Types of loss functions include: mean square error loss, weighted mean square error loss, cross entropy loss, cost loss, weighted loss, etc. For example, for a weighted loss function, its preset parameter is a category weight. The category weight can be used as the weight of the weighted loss function.
[0132] For example, for the weighted loss function, its calculation method can be expressed as:
[0133]
[0134] It means that for the test data set D, N is the total number of samples, Xi is a sample in the test data set, Yi is the true category of the sample, Yi' is the predicted label of the sample, where Yi is a category element in the category set C. Wci represents the weight of category Ci. Discance[Yi,Yi'] represents the distance between the predicted category and the true category of sample Xi, which can be mean square error, absolute value error, Hamming distance, etc., which is not limited in this scheme. Its preset parameter can be Wci.
[0135] For another example, for the cost loss function, its calculation method can be expressed as:
[0136]
[0137] It means that for the test data set D, N is the total number of samples, Xi is a sample in the test data set, Yi is the true category of the sample, Yi' is the predicted label of the sample, where Yi is a category element in the category set C. Cost_cij represents the cost of incorrectly predicting a sample with the true category Ci as Cj. Its preset parameter can be a classification error cost matrix, and Cost_cij is an element in the classification error cost matrix.
[0138] For another example, for the weighted cost loss function, its calculation method can be expressed as:
[0139]
[0140] It means that for the test data set D, N is the total number of samples, Xi is a sample in the test data set, Yi is the true category of the sample, Yi' is the predicted label of the sample, where Yi is a category element in the category set C. Cost_cij represents the cost of incorrectly predicting a sample with the true category Ci as Cj, and Wci represents the weight of category Ci. The preset parameters can be Cost_cij and / or Wci.
[0141] For another example, the evaluation information calculation method is a classification report. The classification report indicates the classification results of NWDAF on the test data set. Consumers can evaluate the performance of the model based on the internal configuration according to the classification report. For example, the accuracy and recall rate of the model are calculated according to the classification report. Its preset parameters can be the number limit of samples in each category. For example, the proportion of samples in each category, the minimum number of samples, etc. For example, the set of categories class of samples in the test data set is C, where Ci is an element in the set. The classification report contains the number of samples of category Ci predicted by the model as category Cj. Optionally, the classification report can be an N*N matrix, as shown in Table 2, where N is the number of sample categories, and Cij represents the number of samples of category i predicted as category j.
[0142] Table 2
[0143] Prediction / Real class1 class2 class3 class1 C11 C12 C13 class2 C21 C22 C23 class3 C31 C32 C33
[0144] The above is only an example, and other calculation methods may also be used, such as the accuracy information calculation method being the ratio of the number of correctly predicted samples to (the sum of the number of correctly predicted samples and the number of incorrectly predicted samples), etc. This solution does not limit this.
[0145] 202. The first network element receives the first request sent by the second network element.
[0146] 203. The first network element obtains evaluation information according to the first information, where the evaluation information is used to evaluate a service or a model.
[0147] Accordingly, the above evaluation information can be used to measure the performance of the service or model, such as accuracy, precision, recall rate, classification report, loss value, etc. Alternatively, the evaluation information can also be used for model training.
[0148] It can be understood that the service is an analysis service provided by the first network element, for example, analyzing quality of service (Quality of Service, QoS), analyzing network element load, analyzing abnormal behavior, etc. The model can be a machine learning model.
[0149] In a possible implementation manner, the first network element calculates the evaluation information according to the first information and a test data set.
[0150] Among them, test data refers to data used to independently evaluate trained and verified artificial intelligence systems (services or models) in order to confirm their expected performance before using the service or model.
[0151] Specifically, when the preset parameter is a classification error cost matrix, the first network element can calculate the evaluation information based on the classification error cost matrix and the test data set. Optionally, the first network element and the second network element pre-configure the evaluation information calculation method, or the second network element stores the corresponding evaluation information calculation method. Then, the first network element can calculate the evaluation information based on the evaluation information calculation method and the classification error cost matrix and the test data set. Of course, other methods can also be used, and this solution does not limit this.
[0152] The test data set is obtained by the first network element according to the test data set identifier. The first request sent by the second network element also includes the test data set identifier.
[0153] Of course, when the first request does not include the above test data set identifier, the test data set may also be actively collected by the first network element, and this solution does not limit this. The first network element obtains the proportion of data related to the second network element in the test data set, and returns the proportion to the second network element. So that the second network element knows the relevance of the evaluation information to the second network element, which helps to select a suitable NWDAF.
[0154] In a possible implementation, the data related to the consumer is the real data of the service subscribed by the consumer or the prediction result of the model. The relevant proportion is the proportion of the above real data in the entire test data set.
[0155] Of course, the first network element may also only collect the data set related to the second network element as the test data set.
[0156] In another possible implementation manner, the first network element sends the first information to the third network element, and then receives evaluation information returned by the third network element.
[0157] Optionally, the first network element sends the first information and the test data set to the third network element, and then receives the evaluation information returned by the third network element. For the introduction of the test data set, please refer to the above record, which will not be repeated here. The third network element can evaluate the trained model based on the first information and the test data set.
[0158] Optionally, the third network element may be a MTLF NWDAF.
[0159] For example, MTLF NWDAF sets the first information as a non-learnable parameter during model training. Specifically, MTLF NWDAF uses, for example, the category weight as a parameter in the weighted loss function during the model training phase. Then, MTLF NWDAF obtains a model based on the first information.
[0160] The third network element may also obtain an evaluation information calculation method based on the first information, and then obtain the evaluation information of the model based on the test data set.
[0161] It should be noted that, in this solution, the first network element and the third network element may be the same network element or different network elements, and this solution does not impose any restriction on this.
[0162] 204. The first network element returns the evaluation information to the second network element;
[0163] After acquiring the evaluation information, the first network element sends it to the second network element.
[0164] In a possible implementation manner, the first network element further returns to the second network element the degree of relevance of the evaluation information and the second network element.
[0165] For example, the first network element returns the proportion of data related to the second network element in the above test data set, or it can also be other relevant information, which is not limited in this solution.
[0166] 205. The second network element receives the evaluation information returned by the first network element.
[0167] In a possible implementation, the second network element determines whether the first network element is a network element that meets the requirements based on the evaluation information. For example, the second network element selects a suitable first network element and its corresponding model, service, etc. based on the evaluation information returned from multiple first network elements.
[0168] In a possible implementation, the first request also includes second information, and the second information is used to instruct the first network element to return evaluation-related information, the evaluation-related information including evaluation information corresponding to multiple time units within a time window and / or statistical information of the evaluation information corresponding to the multiple time units, and the evaluation information is used to evaluate a service or model.
[0169] This time window, also called the time interval [start.end], can be understood as a start time and an end time in the past, or a start time and an end time in the future. The time interval can be expressed in terms of the actual start time and the actual end time (for example, in Coordinated Universal Time Coordinated (UTC) time). By setting the start time and the end time to the same value, the user of the analysis can request or subscribe to analysis for a specific time.
[0170] The time unit can be understood as a plurality of time periods obtained by dividing the above time window. The multiple time units can be of the same size or different sizes, and this scheme does not limit this. For example, the above time window is: the start time is 12:00 on January 1, 2022, and the end time is 12:00 on February 1, 2022. The time unit can be 1 day, 6 hours, 7 days, one day, one week, one month, etc. Alternatively, the time unit is a series of time points to divide the above time window, for example: the time point is 00:00 on January 14, 2022, then the above time window is divided into two time periods. By analogy, there can be multiple time points.
[0171] The statistical information may be information obtained by processing the evaluation information corresponding to the multiple time units. For example, the statistical information is the variance, mean, etc. of the evaluation information corresponding to the multiple time units. It may also be other statistical information, which is not limited in this solution.
[0172] Optionally, the evaluation association information may also be evaluation information corresponding to multiple time units in the periodically returned time window and / or statistical information of the evaluation information corresponding to the multiple time units, etc., which is not limited in this solution.
[0173] Of course, the second information mentioned above may also be carried in other requests, and this solution does not limit this.
[0174] Accordingly, step 204 may include: the first network element returns the evaluation association information to the second network element.
[0175] The first network element obtains evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units.
[0176] The first network element may calculate the above-mentioned evaluation-related information by itself, or may obtain the above-mentioned evaluation-related information from a third network element, and this solution does not impose any limitation on this.
[0177] That is to say, the first network element may first obtain the evaluation information respectively corresponding to the multiple time units within the above time window, and then return the obtained evaluation information respectively corresponding to the multiple time units to the second network element.
[0178] Alternatively, the first network element may first obtain the evaluation information corresponding to the multiple time units within the above time window, and then process the evaluation information corresponding to the multiple time units to obtain statistical information, and then return the statistical information to the second network element.
[0179] Step 205 may include: the second network element receiving the evaluation association information returned by the first network element.
[0180] In a possible implementation manner, the first request also includes the above-mentioned time window.
[0181] In an embodiment of the present application, the second network element sends a request to the first network element, the request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information; the first network element obtains evaluation information based on the first information, and the evaluation information is used to evaluate the service or model; and then returns the evaluation information to the second network element. By adopting this method, the evaluation information is generated based on the preset parameters in the first information sent by the second network element. In this way, the second network element knows the evaluation information generation standard, which can help the second network element select a suitable first network element.
[0182] Above Figure 2 The embodiment shown introduces the method for determining the evaluation information of the present solution. The technical solution is introduced below in combination with Example 1 and Example 2.
[0183] Example 1
[0184] Reference Figure 3 FIG. 1 is a schematic diagram of an evaluation information determination scheme provided in an embodiment of the present application. The method may include steps 301-305, which are as follows:
[0185] 301. The service consumer sends a first request to the ANLF NWDAF, where the first request includes first information.
[0186] The first information includes an evaluation information calculation method and a category weight, and the evaluation information calculation method includes the category weight.
[0187] For example, the evaluation information calculation method includes a weighted cost loss function, which can be expressed as:
[0188]
[0189] It means that for the test data set D, N is the total number of samples, Xi is a sample in the test data set, Yi is the true category of the sample, Yi' is the predicted label of the sample, where Yi is a category element in the category set C. Cost_cij represents the cost of incorrectly predicting a sample with the true category Ci as Cj, and Wci represents the weight of category Ci.
[0190] The preset parameters may be Cost_cij and / or Wci.
[0191] 302. The ANLF NWDAF receives the first request, and sends the evaluation information calculation method and category weight to the MTLF NWDAF.
[0192] The ANLF NWDAF may also obtain a test data set and send the test data set to the MTLF NWDAF.
[0193] 303. The MTLF NWDAF generates evaluation information according to the received test data set, the evaluation information calculation method and the category weight.
[0194] Since the consumer has configured the preset parameters, the MTLF NWDAF can obtain the evaluation information according to the above evaluation information calculation method, the value of the preset weight, and the above test data set.
[0195] For the implementation process, please refer to the above Figure 2 The description of the illustrated embodiment will not be repeated here.
[0196] 304. The MTLF NWDAF sends the evaluation information to the ANLF NWDAF.
[0197] 305. ANLF NWDAF returns the evaluation information to the service consumer.
[0198] In this example, the service consumer provides the ANLF NWDAF with the evaluation information calculation method and category weights, so that the service consumer knows the evaluation information generation standard, which can help the service consumer select a suitable NWDAF according to the evaluation information.
[0199] Example 2
[0200] Reference Figure 4 FIG. 4 is a schematic diagram of an evaluation information determination scheme provided in an embodiment of the present application. The method may include steps 401-410, which are as follows:
[0201] 401. The service consumer sends a first request to the ANLF NWDAF, where the first request includes a classification report.
[0202] This example uses the classification report as an example. For an introduction to the classification report, please refer to the Figure 2 The description of the illustrated embodiment will not be repeated here.
[0203] 402. The ANLF NWDAF receives the first request, and sends the first request to the MTLF NWDAF to obtain an ML model.
[0204] Optionally, the first request may include an analysis ID, model metrics (e.g., accuracy, precision, recall, classification report, loss, etc.), model metric configuration (category weights, cost metrics, etc.), predetermined status (e.g., requested accuracy level and / or total training time) and / or periodic reporting conditions (e.g., training rounds / time), and a test dataset.
[0205] For example, the NWDAF containing ANLF uses the Nnwdaf_MLModelProvision service to send a subscription request to the NWDAF containing MTLF.
[0206] When the request sent by the service consumer indicates that there is a test data set, the ANLF NWDAF sends the test data set to the MTLF NWDAF. When the service consumer does not indicate a test data set, the ANLF NWDAF obtains the test data set itself. For an introduction to this part, please refer to the above Figure 2 The description of the illustrated embodiment will not be repeated here.
[0207] Wherein, when an accuracy metric is requested, the accuracy level may be used to indicate a target accuracy level for the training process. When the requested accuracy level is reached during the training process, the federated learning (FL) server NWDAF included in the MTLF NWDAF may stop the training process.
[0208] If the NWDAF containing AnLF or the user of the NWDAF containing AnLF provides a time required for analyzing information, the server NWDAF may determine the maximum response time of its client NWDAF based on the time.
[0209] 403. The server NWDAF sends a first request to the client NWDAF participating in the federated learning to perform local model training of the federated learning.
[0210] The request may include the above-mentioned model metrics (such as classification reports, etc.) and model metric configuration.
[0211] Further, the first request may also include a maximum response time that the client NWDAF must take to report the temporary local ML model information to the server NWDAF.
[0212] 404. Multiple client NWDAFs receive the first request and collect their local data.
[0213] 405. Each client NWDAF trains the ML model obtained from the server NWDAF according to the collected data, and sends temporary local ML model information to the server NWDAF.
[0214] During the federated learning training process, each client NWDAF trains the model based on the data it collects.
[0215] Each client NWDAF also calculates local model metrics (such as the above-mentioned classification report, etc.) and sends them to the server NWDAF.
[0216] For example, the client NWDAF generates a model based on its local data, and generates the above classification report based on the local model and the local test data set. The classification report represents the performance measurement results of each local model on the corresponding local test data set.
[0217] If the server NWDAF specifies a test data set in the first request, the client NWDAF generates the classification report according to the local model and the specified test data set.
[0218] The local model metrics may be carried in the local training status report. The local model metrics may include a report of the model training data set, the number of samples, and a classification report of the test data set. The local training status report may also include other information about the local training in the client NWDAF, such as the characteristics of the local training data set (e.g., area, sampling ratio, maximum / minimum value of each dimension of the data), etc.
[0219] The above ML model is sent from the client NWDAF to the server NWDAF during the FL training process. The ML model is the information required by the server NWDAF to build an aggregate model based on the locally trained model.
[0220] If the client NWDAF cannot complete the training of the temporary local ML model within the maximum response time provided by the server NWDAF, the client NWDAF notifies the server NWDAF of the delay event and provides appropriate reason information, such as local ML model training failure and more time is needed for local ML model training before the maximum response time ends.
[0221] 406. The server NWDAF aggregates the acquired ML model information of all clients to update the global ML model.
[0222] The server NWDAF aggregates all the above model information according to the classification report returned by the client NWDAF to obtain the global ML model.
[0223] The server NWDAF can also calculate the global model metric, such as based on the local model metric or by applying the global model on the test dataset. For example, the server NWDAF performs verification on the test dataset provided by the consumer to generate evaluation information.
[0224] If the server NWDAF provides the above maximum response time for the client NWDAF, the server NWDAF decides to wait for the client NWDAF that has not provided its temporary local ML model within the maximum response time, or only aggregate the acquired local ML model information instances to update the global ML model, etc. This solution does not impose any restrictions on this.
[0225] In one possible implementation, based on consumer requests, the server NWDAF dynamically updates the training status (e.g., global model metrics) to the consumer periodically (e.g., a certain number of training rounds or every 10 minutes) or when certain predetermined states are reached (e.g., the requested accuracy level is reached or the training time expires).
[0226] In one possible implementation, the consumer determines whether the current model meets the requirements, for example, the global model metrics meet the requirements for the consumer. Then the consumer can modify the subscription (stop or continue the training process).
[0227] In a possible implementation, according to the received request, the server NWDAF updates or terminates the current FL training process.
[0228] If the FL process continues, the server NWDAF sends the aggregated ML model information to each client NWDAF for the next round of model training.
[0229] 407. The server NWDAF sends the global ML model to each client NWDAF.
[0230] 408. Each client NWDAF receives the global ML model to update the ML model of each client NWDAF.
[0231] Then it goes through multiple iterations of training.
[0232] 409. After the training is completed, the server NWDAF returns the trained model to the ANLF NWDAF.
[0233] 410. ANLF NWDAF returns the model to the service consumer.
[0234] In this example, the service consumer sends a classification report to the ANLF NWDAF so that the MTLF NWDAF generates a global ML model based on the classification report.
[0235] Reference Figure 5 FIG. 2 is a flow chart of another method for determining evaluation information provided by an embodiment of the present application. Optionally, the method can be applied to the aforementioned interactive system, for example Figure 1 The interactive system shown in Figure 1. Figure 5 The evaluation information determination method shown may include steps 501-504. It should be understood that for the convenience of description, this application is described in the order of 501-504, and is not intended to be limited to execution in the above order. The embodiment of the present application does not limit the order of execution, execution time, number of executions, etc. of the above one or more steps. The following description takes the execution subject of step 501 and step 504 of the evaluation information determination method as the second network element, such as a service consumer, and the execution subject of steps 502 and 503 as the first network element, such as ANLF NWDAF as an example. This application is also applicable to other execution subjects. Steps 501-504 are as follows:
[0236] 501. The second network element sends a second request to the first network element, where the second request includes second information, and the second information is used to instruct the first network element to return evaluation-related information. The evaluation-related information includes evaluation information corresponding to multiple time units within a time window and / or statistical information of the evaluation information corresponding to the multiple time units, and the evaluation information is used to evaluate a service or model.
[0237] The second request may be, for example, a request to subscribe to analysis and evaluation information.
[0238] The above evaluation information can be used to evaluate the accuracy, precision, recall rate, etc. of the service or model. Alternatively, the evaluation information can also be used for model training.
[0239] The service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.
[0240] This time window, also called the time interval [start.end], can be understood as a start time and an end time in the past, or a start time and an end time in the future. The time interval can be expressed in terms of the actual start time and the actual end time (for example, in Coordinated Universal Time Coordinated (UTC) time). By setting the start time and the end time to the same value, the user of the analysis can request or subscribe to analysis for a specific time.
[0241] The time unit can be understood as a plurality of time periods obtained by dividing the above time window. The multiple time units can be of the same size or different sizes, and this scheme does not limit this. For example, the above time window is: the start time is 12:00 on January 1, 2022, and the end time is 12:00 on February 1, 2022. The time unit can be 1 day, 6 hours, 7 days, one day, one week, one month, etc. Alternatively, the time unit is a series of time points to divide the above time window, for example: the time point is 00:00 on January 14, 2022, then the above time window is divided into two time periods. By analogy, there can be multiple time points.
[0242] The statistical information may be information obtained by processing the evaluation information corresponding to the multiple time units. For example, the statistical information is the variance, mean, etc. of the evaluation information corresponding to the multiple time units. It may also be other statistical information, which is not limited in this solution.
[0243] Optionally, the evaluation association information may also be evaluation information corresponding to multiple time units in the periodically returned time window and / or statistical information of the evaluation information corresponding to the multiple time units, etc., which is not limited in this solution.
[0244] In a possible implementation manner, the second request also includes the above-mentioned time window.
[0245] 502. The first network element receives the second request sent by the second network element.
[0246] 503. The first network element returns evaluation association information to the second network element according to the second information.
[0247] The first network element obtains evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units.
[0248] The first network element may calculate the above-mentioned evaluation-related information by itself, or may obtain the above-mentioned evaluation-related information from a third network element, and this solution does not impose any limitation on this.
[0249] That is to say, the first network element may first obtain the evaluation information respectively corresponding to the multiple time units within the above time window, and then return the obtained evaluation information respectively corresponding to the multiple time units to the second network element.
[0250] Alternatively, the first network element may first obtain the evaluation information corresponding to the multiple time units within the above time window, and then process the evaluation information corresponding to the multiple time units to obtain statistical information, and then return the statistical information to the second network element.
[0251] For the specific implementation of obtaining the above evaluation information, please refer to Figure 2 The description in the illustrated embodiment will not be repeated here.
[0252] 504. The second network element receives the evaluation association information returned by the first network element.
[0253] In a possible implementation, the second network element determines whether the first network element is a network element that meets the requirements according to the evaluation association information. For example, the second network element selects a suitable first network element and its corresponding model, service, etc. according to the evaluation association information returned from multiple first network elements. For example, the second network element selects the first network element with the smallest variance from the variance of the evaluation information returned by multiple first network elements to provide services, and cancels the subscription to the analysis services of other first network elements.
[0254] In a possible implementation manner, the second request further includes first information, and the first information includes preset parameters.
[0255] For example, the preset parameter is one or more of a classification error cost matrix and a category weight.
[0256] For an introduction to this part, please refer to the above Figure 2The description of the illustrated embodiment will not be repeated here.
[0257] In an embodiment of the present application, the second network element sends a request to the first network element, the request includes second information, the second information is used to indicate the return of evaluation-related information, the evaluation-related information includes evaluation information corresponding to multiple time units in a time window and / or statistical information of the evaluation information corresponding to the multiple time units, the evaluation information is used to evaluate a service or model; and then the first network element returns the evaluation-related information to the second network element according to the second information. In this way, the evaluation-related information returned by the first network element to the second network element includes fluctuations in the evaluation information, which helps the second network element select a suitable first network element based on the fluctuations.
[0258] It should be noted that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced mutually, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.
[0259] The above detailed description of the method of the embodiment of the present application, the following provides a device of the embodiment of the present application. It can be understood that in the various device embodiments of the present application, the division of multiple units or modules is only a logical division according to function, and is not used as a limitation on the specific structure of the device. In a specific implementation, some functional modules may be subdivided into more small functional modules, and some functional modules may also be combined into one functional module, but no matter whether these functional modules are subdivided or combined, the general process performed by the device is the same. For example, some devices contain a receiving unit and a sending unit. In some designs, the sending unit and the receiving unit can also be integrated into a communication unit, which can implement the functions implemented by the receiving unit and the sending unit. Usually, each unit corresponds to its own program code (or program instruction), and when the program code corresponding to each of these units is run on the processor, the unit is controlled by the processing unit to execute the corresponding process to implement the corresponding function.
[0260] The embodiment of the present application also provides a device for implementing any of the above methods, for example, providing an evaluation information determination device including a module (or means) for implementing each step performed by the first network element (such as ANLF NWDAF) in any of the above methods. For another example, another evaluation information determination device is provided, including a module (or means) for implementing each step performed by the second network element (such as a service consumer) in any of the above methods.
[0261] For example, refer to Figure 6 FIG. 1 is a schematic diagram of a structure of an evaluation information determination device provided in an embodiment of the present application. The evaluation information determination device is used to implement the aforementioned evaluation information determination method, for example Figure 2 , Figure 3 and Figure 4 The method for determining the evaluation information is shown.
[0262] like Figure 6 As shown, the device may include a transceiver module 601 and a processing module 602, which are specifically as follows:
[0263] The transceiver module 601 is configured to receive a first request sent by a second network element, where the first request includes first information, where the first information includes preset parameters, and where the preset parameters are used to generate evaluation information;
[0264] A processing module 602 is used to obtain evaluation information according to the first information, where the evaluation information is used to evaluate a service or a model;
[0265] The transceiver module 601 is further configured to return the evaluation information to the second network element.
[0266] In a possible implementation manner, the preset parameter is a classification error cost matrix and / or a category weight.
[0267] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
[0268] In a possible implementation manner, the processing module 602 is configured to obtain the evaluation information according to the first information and a test data set.
[0269] In a possible implementation manner, the test data set is obtained according to a test data set identifier, wherein the first request also includes the test data set identifier.
[0270] In a possible implementation, the transceiver module 601 is further configured to return the correlation degree between the evaluation information and the second network element to the second network element.
[0271] In a possible implementation manner, the degree of correlation is the proportion of data related to the second network element in the test data set.
[0272] In a possible implementation manner, the test data set is a data set related to the second network element.
[0273] In a possible implementation, the processing module 602 is configured to send the first information to a third network element; and receive evaluation information sent by the third network element.
[0274] In a possible implementation, the first request also includes second information, and the second information is used to indicate the return of evaluation-related information, and the evaluation-related information includes evaluation information corresponding to multiple time units within the time window and / or statistical information of the evaluation information corresponding to the multiple time units; the transceiver module 601 is also used to return the evaluation-related information to the second network element according to the second information.
[0275] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.
[0276] For example, refer to Figure 7 FIG. 1 is a schematic diagram of another evaluation information determination device provided in an embodiment of the present application. The evaluation information determination device is used to implement the aforementioned evaluation information determination method, for example Figure 2 , Figure 3 , Figure 4 The method for determining the evaluation information is shown.
[0277] like Figure 7 As shown, the device may include a transceiver module 701, which is specifically as follows:
[0278] The transceiver module 701 is configured to send a first request to a first network element, where the first request includes first information, where the first information includes preset parameters, and where the preset parameters are used to generate evaluation information;
[0279] The transceiver module 701 is also used to receive evaluation information returned by the first network element, where the evaluation information is obtained by the first network element based on the first information, and the evaluation information is used to evaluate the service or model.
[0280] In a possible implementation manner, the preset parameter is a classification error cost matrix and / or a category weight.
[0281] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
[0282] In a possible implementation manner, the evaluation information is acquired by the first network element according to the first information and a test data set.
[0283] In a possible implementation manner, the test data set is obtained according to a test data set identifier, wherein the first request also includes the test data set identifier.
[0284] In a possible implementation, the transceiver module 701 is further configured to receive the evaluation information returned by the first network element and the correlation degree between the evaluation information and the second network element.
[0285] In a possible implementation manner, the degree of correlation is the proportion of data related to the second network element in the test data set.
[0286] In a possible implementation manner, the test data set is a data set related to the second network element.
[0287] In a possible implementation, the first request further includes second information, where the second information is used to indicate that evaluation-related information is returned, and the evaluation-related information includes evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units;
[0288] The transceiver module 701 is further configured to receive the evaluation association information returned by the first network element.
[0289] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.
[0290] For example, reuse Figure 7 The evaluation information determination device is used to implement the aforementioned evaluation information determination method, for example Figure 5 The method for determining the evaluation information is shown.
[0291] The transceiver module 701 is configured to receive a second request sent by a second network element, where the second request includes second information, where the second information is used to indicate that evaluation-related information is returned, where the evaluation-related information includes evaluation information corresponding to a plurality of time units in a time window and / or statistical information of the evaluation information corresponding to the plurality of time units, respectively, where the evaluation information is used to evaluate a service or model;
[0292] The transceiver module 701 is further configured to return evaluation-related information to the second network element according to the second information.
[0293] In a possible implementation manner, the second request also includes the time window.
[0294] In a possible implementation, the second request further includes first information, the first information includes preset parameters, the preset parameters are used to generate evaluation information, and further includes a processing module, which is used to obtain evaluation information according to the first information.
[0295] In a possible implementation manner, the preset parameter is a classification error cost matrix and / or a category weight.
[0296] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
[0297] In a possible implementation manner, the processing module is used to obtain the evaluation information according to the first information and a test data set.
[0298] In a possible implementation manner, the processing module is used to: send the first information to a third network element; and receive evaluation information sent by the third network element.
[0299] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.
[0300] For example, reuse Figure 7 , is a schematic diagram of the structure of another evaluation information determination device provided in an embodiment of the present application. The evaluation information determination device is used to implement the aforementioned evaluation information determination method, for example Figure 5 The method for determining the evaluation information is shown.
[0301] The transceiver module 701 is configured to send a second request to the first network element, where the second request includes second information, where the second information is used to indicate the return of evaluation-related information, where the evaluation-related information includes evaluation information corresponding to a plurality of time units in a time window and / or statistical information of the evaluation information corresponding to the plurality of time units, respectively, where the evaluation information is used to evaluate a service or model;
[0302] The transceiver module 701 is further configured to receive the evaluation association information returned by the first network element.
[0303] In a possible implementation manner, the second request also includes the time window.
[0304] In a possible implementation, the second request further includes first information, the first information includes preset parameters, the preset parameters are used to generate evaluation information, and the evaluation information is acquired by the first network element according to the first information.
[0305] In a possible implementation manner, the preset parameter is a classification error cost matrix and / or a category weight.
[0306] In a possible implementation manner, the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
[0307] In a possible implementation manner, the evaluation information is acquired by the first network element according to the first information and a test data set.
[0308] In one possible implementation, the service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.
[0309] For the introduction of the above modules, please refer to the introduction in the above method, which will not be repeated here.
[0310] It should be understood that the division of each module in each of the above devices is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. In addition, the modules in the evaluation information determination device can be implemented in the form of a processor calling software; for example, the evaluation information determination device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each module of the device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory inside the device or a memory outside the device. Alternatively, the modules in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits, and the hardware circuits can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be realized by a programmable logic device (PLD), taking a field programmable gate array (FPGA) as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all of the above units. All modules of the above devices can be realized in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the rest by hardware circuits.
[0311] Reference Figure 8 , which is a schematic diagram of the hardware structure of another evaluation information determination device provided in an embodiment of the present application. Figure 8 The evaluation information determination device 800 shown (the device 800 may be a computer device) includes a memory 801, a processor 802, a communication interface 803 and a bus 804. The memory 801, the processor 802 and the communication interface 803 are connected to each other through the bus 804.
[0312] The memory 801 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).
[0313] The memory 801 may store a program. When the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are used to execute each step of the evaluation information determination method of the embodiment of the present application.
[0314] The processor 802 is a circuit with signal processing capability. In one implementation, the processor 802 may be a circuit with instruction reading and running capability, such as a central processing unit CPU, a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), or a digital signal processor (DSP), etc.; in another implementation, the processor 802 may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as the processor 802 being a hardware circuit implemented by an ASIC or a programmable logic device PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and implementing the hardware circuit configuration may be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules. In addition, it may also be a hardware circuit designed for artificial intelligence, which may be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. The processor 802 is used to execute relevant programs to implement the functions that need to be executed by the units in the evaluation information determination device of the embodiment of the present application, or to execute the evaluation information determination method of the method embodiment of the present application.
[0315] It can be seen that each module in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.
[0316] In addition, the modules in the above device can be fully or partially integrated together, or can be implemented independently. In one implementation, these modules are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the modules of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.
[0317] The communication interface 803 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 800 and other devices or a communication network. For example, data can be obtained through the communication interface 803.
[0318] The bus 804 may include a path for transmitting information between various components of the device 800 (eg, the memory 801 , the processor 802 , and the communication interface 803 ).
[0319] It should be noted that although Figure 8 The device 800 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the device 800 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 800 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 800 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 8 All devices shown in .
[0320] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.
[0321] The embodiment of the present application further provides a computer program product including instructions. When the computer program product is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.
[0322] It should be understood that in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; wherein A and B can be singular or plural. Also, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, wherein a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be necessarily different. Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.
[0323] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0324] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0325] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a magnetic disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.
[0326] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.
Claims
1. A method for determining evaluation information, characterized in that: include: Sending a first request to a first network element, where the first request includes first information, where the first information includes preset parameters, and where the preset parameters are used to generate evaluation information; Receive evaluation information returned by the first network element, where the evaluation information is obtained by the first network element based on the first information, and the evaluation information is used to evaluate a service or model.
2. The method according to claim 1, characterized in that The preset parameters are classification error cost matrix and / or category weight.
3. The method according to claim 1 or 2, characterized in that: The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
4. The method according to any one of claims 1 to 3, characterized in that: The evaluation information is acquired by the first network element according to the first information and a test data set.
5. The method according to claim 4, characterized in that The test data set is obtained according to a test data set identifier, wherein the first request also includes the test data set identifier.
6. The method according to claim 4, characterized in that The method further comprises: Receive the correlation degree between the evaluation information returned by the first network element and the second network element, wherein the correlation degree is the proportion of data related to the second network element in the test data set.
7. The method according to claim 4, characterized in that The test data set is a data set related to the second network element.
8. The method according to any one of claims 1 to 7, characterized in that: The first request further includes second information, where the second information is used to indicate that evaluation-related information is returned, where the evaluation-related information includes evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units; The receiving the evaluation information returned by the first network element includes: Receive the evaluation association information returned by the first network element.
9. The method according to any one of claims 1 to 8, characterized in that: The service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide analysis services.
10. A method for determining evaluation information, characterized in that: include: receiving a first request sent by a second network element, wherein the first request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information; Acquire evaluation information according to the first information, wherein the evaluation information is used to evaluate a service or a model; The evaluation information is returned to the second network element.
11. The method according to claim 10, characterized in that The preset parameters are classification error cost matrix and / or category weights.
12. The method according to claim 10 or 11, characterized in that: The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
13. The method according to any one of claims 10 to 12, characterized in that: The obtaining of evaluation information according to the first information includes: The evaluation information is acquired according to the first information and a test data set.
14. The method according to claim 13, characterized in that The test data set is obtained according to a test data set identifier, wherein the first request also includes the test data set identifier.
15. The method according to claim 13, characterized in that The method further comprises: Return the relevance degree of the evaluation information to the second network element to the second network element, wherein the relevance degree is the proportion of data related to the second network element in the test data set.
16. The method according to claim 13, characterized in that The test data set is a data set related to the second network element.
17. The method according to any one of claims 10 to 16, characterized in that The obtaining of evaluation information according to the first information includes: sending the first information to a third network element; Receive evaluation information sent by the third network element.
18. The method according to any one of claims 10 to 17, characterized in that The first request further includes second information, where the second information is used to indicate that evaluation-related information is returned, where the evaluation-related information includes evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units; The returning the evaluation information to the second network element includes: Evaluation association information is returned to the second network element according to the second information.
19. The method according to any one of claims 10 to 18, characterized in that The service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.
20. A method for determining evaluation information, characterized in that: include: Sending a second request to the first network element, the second request including second information, the second information being used to instruct the first network element to return evaluation-related information, the evaluation-related information including evaluation information respectively corresponding to a plurality of time units within a time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units, the evaluation information being used to evaluate a service or model; Receive the evaluation association information returned by the first network element.
21. The method according to claim 20, characterized in that The second request also includes the time window.
22. The method according to claim 20 or 21, characterized in that The second request also includes first information, the first information includes preset parameters, the preset parameters are used to generate evaluation information, and the evaluation information is obtained by the first network element according to the first information.
23. The method according to claim 22, characterized in that The preset parameters are classification error cost matrix and / or category weights.
24. The method according to claim 22 or 23, characterized in that The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
25. The method according to any one of claims 22 to 24, characterized in that The evaluation information is acquired by the first network element according to the first information and a test data set.
26. The method according to any one of claims 20 to 25, characterized in that The service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide analysis services.
27. A method for determining evaluation information, characterized in that: include: receiving a second request sent by a second network element, the second request including second information, the second information being used to indicate returning evaluation-related information, the evaluation-related information including evaluation information respectively corresponding to a plurality of time units within a time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units, the evaluation information being used to evaluate a service or model; Evaluation association information is returned to the second network element according to the second information.
28. The method according to claim 27, characterized in that The second request also includes the time window.
29. The method according to claim 27 or 28, characterized in that The second request further includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information. The method further includes: Evaluation information is acquired according to the first information.
30. The method according to claim 29, characterized in that The preset parameters are classification error cost matrix and / or category weights.
31. The method according to claim 29 or 30, characterized in that The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
32. The method according to any one of claims 29 to 31, characterized in that The obtaining of evaluation information according to the first information includes: The evaluation information is acquired according to the first information and a test data set.
33. The method according to any one of claims 29 to 32, characterized in that The obtaining of evaluation information according to the first information includes: Sending the first information to a third network element; Receive evaluation information sent by the third network element.
34. The method according to any one of claims 27 to 33, characterized in that The service is an analysis service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analysis service.
35. A method for determining evaluation information, characterized in that: include: The second network element sends a first request to the first network element, where the first request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information; The first network element receives the first request sent by the second network element; The first network element obtains evaluation information according to the first information, where the evaluation information is used to evaluate a service or a model; The first network element returns the evaluation information to the second network element; The second network element receives the evaluation information returned by the first network element.
36. The method according to claim 35, characterized in that The preset parameters are classification error cost matrix and / or category weights.
37. The method according to claim 35 or 36, characterized in that The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
38. The method according to any one of claims 35 to 37, characterized in that The first request further includes second information, where the second information is used to indicate that evaluation-related information is returned, where the evaluation-related information includes evaluation information respectively corresponding to a plurality of time units within the time window and / or statistical information of the evaluation information respectively corresponding to the plurality of time units; The first network element returns the evaluation information to the second network element, including: The first network element returns the evaluation association information to the second network element; The second network element receiving the evaluation information returned by the first network element includes: The second network element receives the evaluation association information returned by the first network element.
39. A method for determining evaluation information, characterized in that: include: The second network element sends a second request to the first network element, where the second request includes second information, where the second information is used to instruct the first network element to return evaluation-related information, where the evaluation-related information includes evaluation information corresponding to a plurality of time units within a time window and / or statistical information of the evaluation information corresponding to the plurality of time units, respectively, where the evaluation information is used to evaluate a service or a model; The first network element receives the second request sent by the second network element; The first network element returns evaluation association information to the second network element according to the second information; The second network element receives the evaluation association information returned by the first network element.
40. The method according to claim 39, characterized in that The second request further includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information. The method further includes: The first network element obtains evaluation information according to the first information.
41. The method according to claim 40, characterized in that The preset parameters are classification error cost matrix and / or category weights.
42. The method according to claim 40 or 41, characterized in that The first information also includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.
43. An evaluation information determination device, characterized in that: Comprising a unit or module for executing the method according to any one of claims 1 to 34.
44. An evaluation information determination system, characterized in that Comprising a unit or module for executing the method described in any one of claims 1 to 9, and a unit or module for executing the method described in any one of claims 10 to 19; or, comprising a unit or module for executing the method described in any one of claims 20 to 26, and a unit or module for executing the method described in any one of claims 27 to 34.
45. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program instructions, and when the program instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 34.
46. A computer program product comprising program instructions, characterized in that When the program instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 34.
Citation Information
Cited By
Evaluation information determination method, related system and storage medium
EP4694066A1
Evaluation information determination method, related system and storage medium
WO2024230375A1
