Evaluation information determination method, related systems, and storage media

Abstract

Embodiments of this application provide a method for determining evaluation information, a related system, and a storage medium. The method includes the steps of: receiving a first request sent by a second network element, wherein the first request includes first information, wherein the first information includes preset parameters, which are used to generate evaluation information; obtaining evaluation information based on the first information, wherein the evaluation information is used to evaluate a service or model; and returning the evaluation information to the second network element. In this scheme, the preset parameters in the first information sent by the second network element are used to generate evaluation information. In this way, the second network element knows the standard for generating evaluation information. This can help the second network element select an appropriate first network element.

Claims

1. A method for determining evaluation information, A step of sending a first request to a first network element, wherein the first request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information. A step of receiving evaluation information returned by the first network element, wherein the evaluation information is obtained by the first network element based on the first information, and the evaluation information is used to evaluate a service or model. A method for determining evaluation information, including the method described above.

2. The method according to claim 1, wherein the preset parameters are a classification error cost matrix and / or class weights.

3. The method according to claim 1 or 2, wherein the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

4. The method according to any one of claims 1 to 3, wherein the evaluation information is obtained by the first network element based on the first information and the test dataset.

5. The method according to claim 4, wherein the test dataset is obtained based on a test dataset identifier, and the first request further includes the test dataset identifier.

6. The aforementioned method, A step of receiving a correlation between the evaluation information and a second network element, wherein the correlation is the ratio of data related to the second network element in the test dataset. The method according to claim 4, further comprising:

7. The method according to claim 4, wherein the test dataset is a dataset related to a second network element.

8. The first request further includes second information, the second information indicating that evaluation-related information returns evaluation information corresponding to each of several time units in a time window and / or statistical information of the evaluation information corresponding to each of the several time units. The step of receiving the evaluation information returned by the first network element is: Step of receiving the evaluation-related information returned by the first network element. The method according to any one of claims 1 to 7, including the method described in any one of claims 1 to 7.

9. The method according to any one of claims 1 to 8, wherein the service is an analytical service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analytical service.

10. A method for determining evaluation information, A step of receiving a first request sent by a second network element, wherein the first request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information. A step of obtaining evaluation information based on the first information, wherein the evaluation information is used to evaluate a service or model, The steps include returning the evaluation information to the second network element, A method for determining evaluation information, including the method described above.

11. The method according to claim 10, wherein the preset parameters are a classification error cost matrix and / or class weights.

12. The method according to claim 10 or 11, wherein the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

13. The step of obtaining the evaluation information based on the first information is: Steps to obtain the evaluation information based on the first information and the test dataset. The method according to any one of claims 10 to 12, including the method described in any one of claims 10 to 12.

14. The method according to claim 13, wherein the test dataset is obtained based on a test dataset identifier, and the first request further includes the test dataset identifier.

15. The aforementioned method, A step of returning the degree of association between the evaluation information and the second network element to the second network element, wherein the degree of association is the ratio of data related to the second network element in the test dataset. The method according to claim 13, further comprising:

16. The method according to claim 13, wherein the test dataset is a dataset related to the second network element.

17. The step of obtaining the evaluation information based on the first information is: The steps include sending the first information to a third network element, The steps include receiving the evaluation information sent by the third network element, The method according to any one of claims 10 to 16, including the method described in any one of claims 10 to 16.

18. The first request further includes second information, the second information indicating that evaluation-related information returns evaluation information corresponding to each of several time units in a time window and / or statistical information of the evaluation information corresponding to each of the several time units. The step of returning the evaluation information to the second network element is: Step of returning the evaluation-related information to the second network element based on the second information. The method according to any one of claims 10 to 17, including the method described in any one of claims 10 to 17.

19. The method according to any one of claims 10 to 18, wherein the service is an analytical service provided by a first network element, and the model is a machine learning model used by the first network element to provide the analytical service.

20. A method for determining evaluation information, A step of sending a second request to a first network element, wherein the second request includes second information, which indicates to the first network element that it should return evaluation-related information, which includes evaluation information corresponding to a plurality of time units in a time window and / or statistical information of the evaluation information corresponding to the plurality of time units, and which is used to evaluate a service or model. The steps include receiving the evaluation-related information returned by the first network element, A method for determining evaluation information, including the method described above.

21. The method of claim 20, wherein the second requirement further includes the time window.

22. The method according to claim 20 or 21, wherein the second requirement further includes first information, the first information includes preset parameters, the preset parameters are used to generate evaluation information, and the evaluation information is obtained by the first network element based on the first information.

23. The method according to claim 22, wherein the preset parameters are a classification error cost matrix and / or class weights.

24. The method according to claim 22 or 23, wherein the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

25. The method according to any one of claims 22 to 24, wherein the evaluation information is obtained by the first network element based on the first information and the test dataset.

26. The method according to any one of claims 20 to 25, wherein the service is an analytical service provided by the first network element, and the model is a machine learning model used by the first network element to provide the analytical service.

27. A method for determining evaluation information, A step of receiving a second request sent by a second network element, wherein the second request includes second information, the second information indicates that evaluation-related information should be returned, the evaluation-related information includes evaluation information corresponding to each of a plurality of time units in a time window and / or statistical information of the evaluation information corresponding to each of the plurality of time units, and the evaluation information is used to evaluate a service or model. A step of returning the evaluation-related information to the second network element based on the second information, A method for determining evaluation information, including the method described above.

28. The method of claim 27, wherein the second requirement further includes the time window.

29. The second requirement further includes the first information, the first information includes preset parameters, the preset parameters are used to generate evaluation information, and the method is Steps to obtain evaluation information based on the first information described above. The method according to claim 27 or 28, further comprising:

30. The method according to claim 29, wherein the preset parameters are a classification error cost matrix and / or class weights.

31. The method according to claim 29 or 30, wherein the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

32. The step of obtaining the evaluation information based on the first information is: Steps to obtain the evaluation information based on the first information and the test dataset. The method according to any one of claims 29 to 31, including the method described in that claim.

33. The step of obtaining the evaluation information based on the first information is: The steps include sending the first information to a third network element, The steps include receiving the evaluation information sent by the third network element, The method according to any one of claims 29 to 32, including the method described in that claim.

34. The method according to any one of claims 27 to 33, wherein the service is an analytical service provided by a first network element, and the model is a machine learning model used by the first network element to provide the analytical service.

35. A method for determining evaluation information, A step of sending a first request to a first network element via a second network element, wherein the first request includes first information, the first information includes preset parameters, and the preset parameters are used to generate evaluation information. The first network element receives the first request sent by the second network element, A step of obtaining evaluation information based on the first information using the first network element, wherein the evaluation information is used to evaluate a service or model. The first network element returns the evaluation information to the second network element, The second network element receives the evaluation information returned by the first network element, A method for determining evaluation information, including the method described above.

36. The method according to claim 35, wherein the preset parameters are a classification error cost matrix and / or class weights.

37. The method according to claim 35 or 36, wherein the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

38. The first request further includes second information, the second information indicating that evaluation-related information returns evaluation information corresponding to each of several time units in a time window and / or statistical information of the evaluation information corresponding to each of the several time units. The step of returning the evaluation information to the second network element by the first network element is: The first network element includes the step of returning the evaluation-related information to the second network element, The step of receiving the evaluation information returned by the first network element by the second network element is as follows: The second network element includes the step of receiving the evaluation-related information returned by the first network element, The method according to any one of claims 35 to 37.

39. A method for determining evaluation information, A step of sending a second request to a first network element via a second network element, wherein the second request includes second information, which indicates to the first network element that it will return evaluation-related information, which includes evaluation information corresponding to a plurality of time units in a time window and / or statistical information of the evaluation information corresponding to the plurality of time units, and which is used to evaluate a service or model. The first network element receives the second request sent by the second network element, The first network element returns the evaluation-related information to the second network element based on the second information, The second network element receives the evaluation-related information returned by the first network element, A method for determining evaluation information, including the method described above.

40. The second requirement further includes the first information, the first information includes preset parameters, the preset parameters are used to generate evaluation information, and the method is The first network element obtains evaluation information based on the first information. The method according to claim 39, further comprising:

41. The method according to claim 40, wherein the preset parameters are a classification error cost matrix and / or class weights.

42. The method according to claim 40 or 41, wherein the first information further includes an evaluation information calculation method, and the evaluation information calculation method includes the preset parameters.

43. An evaluation information determination device comprising a unit or module configured to carry out the method described in any one of claims 1 to 34.

44. An evaluation information determination system comprising a unit or module configured to carry out the method described in any one of claims 1 to 9, and a unit or module configured to carry out the method described in any one of claims 10 to 19, or a unit or module configured to carry out the method described in any one of claims 20 to 26, and a unit or module configured to carry out the method described in any one of claims 27 to 34.

45. A computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed on a computer, the computer is able to carry out the method according to any one of claims 1 to 34.

46. A computer program product comprising program instructions, wherein when the program instructions are executed on a computer, the computer becomes capable of carrying out the method according to any one of claims 1 to 34.

Images