Performance measurement method, system and related equipment

Abstract

The invention provides a performance measurement method and system and related equipment, and relates to the technical field of communication. The method comprises the following steps: acquiring interaction information when a first network function and / or a network element acquires target data from a second network function and / or a network element through multiple iterations; and determining a performance measurement result of the first network function and / or the network element and / or the second network function and / or the network element according to the interaction information. The process that the first network function and / or the network element obtains the target data from the second network function and / or the network element through multiple iterations can be effectively monitored, and reference data is provided for an operator to optimize the network function and / or the network element.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of communication, and particularly relates to a performance measurement method, system and related equipment. BACKGROUND

[0002] In a communication network, for a network node supporting aggregation capability, it is often necessary to obtain corresponding data from other network nodes providing data for aggregation through multiple iterations, wherein the network node supporting aggregation capability can be a network function and / or network element of a core network, or a network function and / or network element of an access network (such as a base station and the like), or a user equipment.

[0003] In the multiple iteration process, if a certain network node providing data fails to respond in time, it indicates that the performance of the network node supporting aggregation capability in obtaining data from the network node for aggregation is poor, and at this time, it is often necessary to adjust the configuration information of the network node or select a suitable network node to provide data in order to respond in time.

[0004] Therefore, effective monitoring of the process of the network node supporting aggregation capability obtaining data from multiple network nodes through multiple iterations for aggregation, and understanding the performance of the network node supporting aggregation capability in obtaining data from other network nodes, can provide necessary data reference for operators to optimize the performance of the network node.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The present disclosure provides a performance measurement method, system and related equipment, which at least partially solves the technical problem of lack of effective monitoring of the aggregation process between network nodes in the related art, resulting in no quantitative data reference for network node optimization.

[0007] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a performance measurement method is provided, which comprises: obtaining interaction information when a first network function and / or network element obtains target data from a second network function and / or network element through multiple iterations; and determining a performance measurement result of the first network function and / or network element, and / or the second network function and / or network element according to the interaction information.

[0009] In some embodiments, the first network function and / or network element is a network function and / or network element supporting data analysis function.

[0010] In some embodiments, the first network function and / or network element is a network function and / or network element supporting aggregation capability, the aggregation capability comprising: an analytics information aggregation capability and / or a model information aggregation capability.

[0011] In some embodiments, the interaction information comprises at least one of: interaction information of the first network function and / or network element supporting the analytics information aggregation capability when obtaining the analytics information from the second network function and / or network element through multiple iterations; and interaction information of the first network function and / or network element supporting the model information aggregation capability when obtaining the model information from the second network function and / or network element through multiple iterations.

[0012] In some embodiments, the target data comprises: analytics information and / or model information.

[0013] In some embodiments, the interaction information comprises: a response and / or a notification.

[0014] In some embodiments, the method further comprises at least one of: obtaining interaction information of the network side device when obtaining the target data from the user equipment through multiple iterations; and obtaining interaction information of the user equipment when obtaining the target data from other user equipment through multiple iterations.

[0015] In some embodiments, the method further comprises: determining, according to the performance measurement result, whether the target data obtained by the first network function and / or network element from the second network function and / or network element is complete or missing.

[0016] In some embodiments, the method further comprises: determining, according to the performance measurement result, a completeness or a missing degree of the target data obtained by the first network function and / or network element from the second network function and / or network element.

[0017] In some embodiments, the performance measurement result comprises: a number of responses and / or response number distribution information.

[0018] In some embodiments, the number of responses comprises at least one of: a number of successful responses, a maximum number of successful responses, a minimum number of successful responses, and an average number of successful responses, received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations; and the response number distribution information comprises: a distribution of the number of successful responses received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations.

[0019] In some embodiments, the method further comprises: determining a maximum value of the number of successful responses and a minimum value of the number of successful responses according to the number of successful responses received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations; determining a range interval of the number of successful responses according to the maximum value of the number of successful responses and the minimum value of the number of successful responses; dividing the range interval of the number of successful responses into multiple sub-range intervals, and matching the number of successful responses received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations to a corresponding sub-range interval to obtain a distribution of the number of successful responses received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations.

[0020] In some embodiments, the method further comprises: determining a mean value of the number of successful responses according to the number of successful responses received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations; calculating a variance of the number of successful responses according to the number of successful responses received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations and the mean value of the number of successful responses; and determining a distribution of the number of successful responses received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations according to the variance of the number of successful responses.

[0021] In some embodiments, the method further comprises: determining whether the target data obtained by the first network function and / or network element from the second network function and / or network element is complete according to the number of successful responses and the distribution of the number of successful responses when the first network function and / or network element obtains the target data from the second network function and / or network element through multiple iterations.

[0022] In some embodiments, the method further comprises: determining a degree of completeness of the target data obtained by the first network function and / or network element from the second network function and / or network element according to the number of successful responses and the distribution of the number of successful responses when the first network function and / or network element obtains the target data from the second network function and / or network element through multiple iterations.

[0023] In some embodiments, the response quantity comprises at least one of: a number of unsuccessful responses, a maximum number of unsuccessful responses, a minimum number of unsuccessful responses, an average number of unsuccessful responses, in which the first network function and / or network element does not receive a successful response or receives an error response when obtaining the target data from the second network function and / or network element through multiple iterations; and the response quantity distribution information comprises a distribution of the number of unsuccessful responses in which the first network function and / or network element does not receive a successful response or receives an error response when obtaining the target data from the second network function and / or network element through multiple iterations.

[0024] In some embodiments, the method further comprises: determining the maximum number of unsuccessful responses and the minimum number of unsuccessful responses according to the number of unsuccessful responses in which the first network function and / or network element does not receive a successful response or receives an error response when obtaining the target data from the second network function and / or network element through multiple iterations; determining a range interval of the number of unsuccessful responses according to the maximum number of unsuccessful responses and the minimum number of unsuccessful responses; dividing the range interval of the number of unsuccessful responses into multiple sub-range intervals, and matching the number of unsuccessful responses in which the first network function and / or network element receives an unsuccessful response when obtaining the target data from the second network function and / or network element through multiple iterations to a corresponding sub-range interval to obtain the distribution of the number of unsuccessful responses in which the first network function and / or network element does not receive a successful response or receives an error response when obtaining the target data from the second network function and / or network element through multiple iterations.

[0025] In some embodiments, the method further comprises: determining the average number of unsuccessful responses according to the number of unsuccessful responses in which the first network function and / or network element does not receive a successful response or receives an error response when obtaining the target data from the second network function and / or network element through multiple iterations; calculating a variance of the number of unsuccessful responses according to the number of unsuccessful responses in which each second network function and / or network element receives an unsuccessful response during the multiple iterations and the average number of unsuccessful responses; and determining the distribution of the number of unsuccessful responses in which the first network function and / or network element does not receive a successful response or receives an error response when obtaining the target data from the second network function and / or network element through multiple iterations according to the variance of the number of unsuccessful responses.

[0026] In some embodiments, the method further comprises: determining whether the target data obtained by the first network function and / or network element from the second network function and / or network element is missing according to the number of unsuccessful responses in which the first network function and / or network element does not receive a successful response or receives an error response when obtaining the target data from the second network function and / or network element through multiple iterations and the distribution of the number of unsuccessful responses.

[0027] In some embodiments, the method further comprises: determining the degree of absence of the target data acquired by the first network function and / or network element from the second network function and / or network element according to the number of unsuccessful responses and the distribution of the number of unsuccessful responses when the second network function and / or network element does not receive a successful response or receives an error response from the first network function and / or network element through multiple iterations of acquiring the target data.

[0028] In some embodiments, the interaction information when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations comprises: a performance measurement service request; and the interaction information when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations according to the performance measurement service request.

[0029] In some embodiments, the method further comprises: determining whether to update / optimize the configuration information of the first network function and / or network element, and / or the second network function and / or network element according to the performance measurement result.

[0030] In some embodiments, the method further comprises: determining whether to reselect the second network function and / or network element that provides the target data for the first network function and / or network element according to the performance measurement result.

[0031] According to another aspect of the present disclosure, a performance measurement method is also provided, which can be applied to a performance measurement service user device, and comprises: initiating a performance measurement service request to a performance measurement service provider device, wherein the performance measurement service request is used to request interaction information when a first network function and / or network element acquires target data from a second network function and / or network element through multiple iterations; receiving the interaction information returned by the performance measurement service provider device according to the performance measurement service request, and determining a performance measurement result of the first network function and / or network element, and / or the second network function and / or network element according to the interaction information when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations.

[0032] According to another aspect of the present disclosure, there is also provided a performance measurement method applicable to a performance measurement service provider device, comprising: receiving a performance measurement service request from the performance measurement service provider device, wherein the performance measurement service request is used to request interaction information when a first network function and / or network element acquires target data from a second network function and / or network element through multiple iterations; acquiring, according to the performance measurement service request, the interaction information when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations; and determining a performance measurement result of the first network function and / or network element, and / or the second network function and / or network element according to the interaction information when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations.

[0033] According to another aspect of the present disclosure, there is also provided a performance measurement system, comprising: a performance measurement service provider device and a performance measurement service user device; wherein the performance measurement service user device is configured to initiate a performance measurement service request to the performance measurement service provider device, wherein the performance measurement service request is used to request interaction information when a first network function and / or network element acquires target data from a second network function and / or network element through multiple iterations; and the performance measurement service provider device is configured to acquire, according to the performance measurement service request, the interaction information when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations, and feed back to the performance measurement service user device; wherein the performance measurement service user device is further configured to determine a performance measurement result of the first network function and / or network element, and / or the second network function and / or network element according to the interaction information when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations.

[0034] According to another aspect of the present disclosure, there is also provided a performance measurement system, comprising: a performance measurement service provider device and a performance measurement service user device; wherein the performance measurement service user device is configured to initiate a performance measurement service request to the performance measurement service provider device, wherein the performance measurement service request is configured to request interaction information of a first network function and / or network element when acquiring target data from a second network function and / or network element through multiple iterations; and the performance measurement service provider device is configured to acquire the interaction information of the first network function and / or network element when acquiring the target data from the second network function and / or network element through multiple iterations according to the performance measurement service request, and determine a performance measurement result of the first network function and / or network element, and / or the second network function and / or network element according to the interaction information of the first network function and / or network element when acquiring the target data from the second network function and / or network element through multiple iterations, and feed back the performance measurement result to the performance measurement service user device.

[0035] According to another aspect of the present disclosure, there is also provided an electronic device, comprising: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to perform the performance measurement method according to any one of the preceding aspects by executing the executable instructions.

[0036] According to another aspect of the present disclosure, there is also provided a computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the performance measurement method according to any one of the preceding aspects.

[0037] According to another aspect of the present disclosure, there is also provided a computer program product, comprising: a computer program or instructions, which, when executed by a processor, implements the performance measurement method according to any one of the preceding aspects.

[0038] The performance measurement method, system and related device provided in the embodiments of the present disclosure can effectively monitor the process of the first network function and / or network element acquiring the target data from the second network function and / or network element through multiple iterations, and provide reference data for the operator to optimize the network function and / or network element.

[0039] It should be understood that the general description above and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure. It is apparent that the accompanying drawings described below are only some embodiments of the present disclosure, and other drawings can be obtained by those of ordinary skill in the art without creative effort on the basis of these drawings.

[0041] Figure 1 An application system architecture diagram in an embodiment of the present disclosure is shown;

[0042] Figure 2 A scenario diagram in which a network function and / or network element supporting aggregation capability acquires performance measurement of data from other network functions and / or network elements providing data in an embodiment of the present disclosure is shown;

[0043] Figure 3 A scenario diagram in which a network side device supporting aggregation capability acquires performance measurement of data from user equipment providing data for aggregation in an embodiment of the present disclosure is shown;

[0044] Figure 4 A scenario diagram in which user equipment supporting aggregation capability acquires performance measurement of data from other user equipment providing data for aggregation in an embodiment of the present disclosure is shown;

[0045] Figure 5 A performance measurement method flow chart in an embodiment of the present disclosure is shown;

[0046] Figure 6 An optional performance measurement method flow chart in an embodiment of the present disclosure is shown;

[0047] Figure 7 A performance measurement method flow chart based on the number of successful responses in an embodiment of the present disclosure is shown;

[0048] Figure 8 A performance measurement method flow chart based on the number of unsuccessful responses in an embodiment of the present disclosure is shown;

[0049] Figure 9 Another optional performance measurement method flow chart in an embodiment of the present disclosure is shown;

[0050] Figure 10 A performance measurement method flow chart applied to a performance measurement service user equipment in an embodiment of the present disclosure is shown;

[0051] Figure 11 A performance measurement method flow chart applied to a performance measurement service provider equipment in an embodiment of the present disclosure is shown;

[0052] Figure 12Fig. 1 shows a schematic diagram of a performance measurement system according to an embodiment of the present disclosure;

[0053] Figure 13 Fig. 2 shows a schematic diagram of an interaction process of a performance measurement system according to an embodiment of the present disclosure;

[0054] Figure 14 Fig. 3 shows a schematic diagram of an interaction process of another performance measurement system according to an embodiment of the present disclosure;

[0055] Figure 15 Fig. 4 shows a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art. Features described in one implementation can be combined with features described in another implementation.

[0057] In addition, the drawings are to be considered in the illustrative mode only and not necessarily to scale. The same or similar reference symbols are used in the drawings to indicate the same or similar parts, and thus repeated description thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities, which do not necessarily have to correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0058] For ease of understanding, before introducing the embodiments of the present disclosure, first, several terms involved in the embodiments of the present disclosure are explained as follows:

[0059] 5GC: 5G Core, 5G core network.

[0060] NWDAF (Network Data Analytics Function): a network function in the 5GC.

[0061] Aggregator NWDAF: an NWDAF supporting an analytics aggregation capability, which can collect and aggregate analytics information from other NWDAFs supporting an analytics metadata provisioning capability;

[0062] FL Client NWDAF: Federated Learning Client NWDAF, NWDAF as a client in federated learning, used to provide model information in federated learning.

[0063] FL Server NWDAF: Federated Learning Server NWDAF, NWDAF as a server in federated learning, used to aggregate model information provided by FL Client NWDAF in federated learning.

[0064] PM Service Consumer: Performance Measurement Service Producer, performance measurement service consumer;

[0065] PM Service Producer: Performance Measurement Service Producer, performance measurement service producer / provider, receives performance measurement service requests from PM Service Consumer and provides performance measurement services to PM Service Consumer.

[0066] The specific implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0067] Figure 1 An application system architecture diagram in the embodiments of the present disclosure is shown, as shown in the figure, the system architecture includes a user equipment 10, an access network equipment 20 and a core network 30, wherein the core network includes a plurality of network functions and / or network elements. Figure 1

[0068] Optionally, the medium providing a communication link between the user equipment 10 and the access network equipment 20 can be a wired network or a wireless network.

[0069] ​Optionally, the wireless or wired networks described above use standard communications technologies and / or protocols. The networks typically carry Internet traffic, but can also include, without limitation, any combination of local, regional, or global packet-switched and / or circuit-switched networks, including an electrical network, a provider's network, and the Internet, or a virtual private network. In some embodiments, technologies used include and / or are compliant with standards and / or protocols such as Hypertext Markup Language (HTML), Extensible Markup Language (XML), and others. Additionally, all or some links can be encrypted using conventional encryption technologies, such as the Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), Internet Protocol Security (IPSec), and others. In other embodiments, custom and / or proprietary data communications technologies can be employed instead of, or in addition to, the ones described.

[0070] Optionally, the user equipment (UE) in the embodiments of the present disclosure can also be referred to as terminal equipment. In specific implementation, the terminal equipment can be a mobile phone, a Tablet Personal Computer, a Laptop Computer, a Personal Digital Assistant (PDA), a Mobile Internet Device (MID), a Wearable Device, or a vehicle-mounted device, and the like. It should be noted that the specific type of the terminal equipment is not limited in the embodiments of the present disclosure.

[0071] Optionally, the access network device 20 in the embodiments of the present disclosure can be a base station, a relay, or an access point, and the like. In some embodiments, the base station can be a base station of 5G and later versions (for example, a 5G NR NB), or a base station in other communication systems (for example, an eNB base station). It should be noted that the specific type of the network side device is not limited in the embodiments of the present disclosure.

[0072] Optionally, the core network 30 in the embodiments of the present disclosure can be a 5G or later version core network, or a core network after the 5G or later version core network is fused with a 4G or previous version core network. With the continuous updating of the core network version, the network functions and / or network elements in the core network can also change.

[0073] Those skilled in the art can know that, Figure 1 The number of user equipment, access network equipment, and network functions and / or network elements in the core network in the above embodiments is only illustrative, and any number of user equipment, access network equipment, and network functions and / or network elements in the core network can be provided according to actual needs. The embodiments of the present disclosure do not limit this.

[0074] Under the above system architecture, the embodiments of the present disclosure provide a performance measurement method, which can be executed by any electronic device with computing processing capability.

[0075] The performance measurement method provided in the embodiments of the present disclosure can include the following steps: obtaining interaction information when a first network node obtains target data from a second network node through multiple iterations; and determining a performance measurement result of the first network node and / or the second network node according to the interaction information.

[0076] It should be noted that the first network node in the embodiments of the present disclosure is a network node (which can be a network function and / or network element of a core network, or a base station and the like of an access network equipment of an access network, or a user equipment of a terminal side) supporting aggregation capability (including but not limited to analysis information aggregation capability and / or model information aggregation capability); the second network node in the embodiments of the present disclosure is a network node (which can be a network function and / or network element of a core network, or a base station and the like of an access network equipment of an access network, or a user equipment of a terminal side) providing data to be aggregated (including but not limited to analysis information and / or model information).

[0077] The performance measurement method provided in the embodiments of the present disclosure can be applied to but not limited to the following three application scenarios.

[0078] The first application scenario, as shown in Figure 2 For the scenario that a certain network function and / or network element in the core network supporting aggregation capability obtains data (including but not limited to analysis information and / or model information) from other network functions and / or network elements in the core network providing data and performs aggregation, the first network node in the embodiments of the present disclosure can be the network function and / or network element in the core network supporting aggregation capability, and the second network node can be the network function and / or network element in the core network providing data to be aggregated.

[0079] The second application scenario, as shown in Figure 3As shown, for the scenario that the network-side device (which can be a network function and / or network element of the core network, and can also be an access device of the access network) supporting the aggregation capability acquires data (including but not limited to analysis information and / or model information) from the user equipment providing the data and performs aggregation, the first network node in the embodiment of the disclosure can be a network function and / or network element in the core network supporting the aggregation capability, and the second network node in the embodiment of the disclosure can be a network function and / or network element in the core network providing the data to be aggregated.

[0080] The third application scenario is as follows: Figure 4 As shown, for the scenario that the user equipment supporting the aggregation capability acquires data (including but not limited to analysis information and / or model information) from other user equipment providing the data and performs aggregation, the first network node in the embodiment of the disclosure can be the user equipment supporting the aggregation capability, and the second network node can be the other user equipment providing the data.

[0081] It should be noted that the above three application scenarios are only examples. In some aggregation scenarios (for example, the network function and / or network element in the core network supporting the aggregation capability not only acquires data from the network function and / or network element in the core network providing the data, but also acquires data from the access network device providing the data, and even acquires data from the user equipment providing the data), the first network node can be the network function and / or network element in the core network supporting the aggregation capability, and the second network node can be at least one of the following providing the data: the network function and / or network element in the core network providing the data, the access network device, and the user equipment.

[0082] Under the above system architecture, the embodiment of the disclosure provides a performance measurement method, as shown in Figure 5 The performance measurement method provided in the embodiment of the disclosure includes the following steps:

[0083] S502, acquiring interaction information when the first network function and / or network element acquires target data from the second network function and / or network element through multiple iterations.

[0084] It should be noted that the first network function and / or network element in the embodiments of the present disclosure is any network function and / or network element (which can be a network function and / or network element of a core network, or a base station and the like of an access network device of an access network) in a communication network supporting aggregation capability (including but not limited to analysis information aggregation capability and / or model information aggregation capability); the second network function and / or network element in the embodiments of the present disclosure is a network function and / or network element (which can be a network function and / or network element of a core network, or a base station and the like of an access network device of an access network) in a communication network providing data to be aggregated (i.e. target data, including but not limited to analysis information and / or model information). The interaction information in the embodiments of the present disclosure can be information exchanged between the first network function and / or network element and the second network function and / or network element in the process of the first network function and / or network element aggregating data obtained from the second network function and / or network element through multiple iterations, including information sent by the first network function and / or network element to the second network function and / or network element, and / or information sent by the second network function and / or network element to the first network function and / or network element.

[0085] It should be noted here that the network function (such as the first network function and the second network function) in the embodiments of the present disclosure can be a logical function module or a function interface for providing network services in a core network; the network element in the embodiments of the present disclosure can be a physical device for providing network services in a core network. The above S502 includes but is not limited to the following cases: ① obtaining interaction information when the first network function obtains target data from the second network function through multiple iterations; ② obtaining interaction information when the first network function obtains target data from the second network element through multiple iterations; ③ obtaining interaction information when the first network function obtains target data from the second network function and the second network element through multiple iterations; ④ obtaining interaction information when the first network element obtains target data from the second network function through multiple iterations; ⑤ obtaining interaction information when the first network element obtains target data from the second network element through multiple iterations; ⑥ obtaining interaction information when the first network element obtains target data from the second network function and the second network element through multiple iterations.

[0086] S504, determining the performance measurement result of the first network function and / or network element, and / or the second network function and / or network element according to the interaction information.

[0087] It should be noted that after obtaining the interaction information when the first network function and / or network element obtains target data from the second network function and / or network element through multiple iterations, the performance measurement result of the first network function and / or network element can be determined according to the interaction information; the performance measurement result of the second network function and / or network element can also be determined according to the interaction information; the performance measurement result of the first network function and / or network element and the second network function and / or network element can also be determined according to the interaction information.

[0088] In some embodiments, the first network function and / or network element in the embodiments of the present disclosure is a network function and / or network element supporting a data analytics function. When the first network function is a network function of a 5G core network, the first network function can be a Network Data Analytics Function (NWDAF).

[0089] In some embodiments, when the first network function and / or network element is a network function and / or network element supporting an aggregation capability, the aggregation capability can include, but is not limited to, an analytics information aggregation capability and / or a model information aggregation capability.

[0090] Based on the above embodiments, in some embodiments, the above S502, when implemented in detail, can include, but is not limited to, at least one of the following: obtaining, by the first network function and / or network element supporting the analytics information aggregation capability, interaction information when the second network function and / or network element obtains the analytics information through multiple iterations; and obtaining, by the first network function and / or network element supporting the model information aggregation capability, interaction information when the second network function and / or network element obtains the model information through multiple iterations.

[0091] In some embodiments, the target data in the embodiments of the present disclosure can include, but is not limited to, analytics information and / or model information.

[0092] In some embodiments, the interaction information in the embodiments of the present disclosure can include, but is not limited to, a response and / or a notification.

[0093] In some embodiments, the performance measurement method in the embodiments of the present disclosure can further include, but is not limited to, at least one of the following: obtaining, by the network side device, interaction information when the user equipment obtains the target data through multiple iterations; and obtaining, by the user equipment, interaction information when the user equipment obtains the target data through multiple iterations. In this embodiment, the network side device can be a core network device (various network elements of the core network or devices providing various network functions) or an access network device (such as a base station). The user equipment in the embodiments of the present disclosure can be any terminal device.

[0094] In some embodiments, as shown in FIG. 6, the performance measurement method in the embodiments of the present disclosure can further include the following steps: Figure 6

[0095] S506, determining, according to the performance measurement result, whether the target data obtained by the first network function and / or network element from the second network function and / or network element is complete or missing.

[0096] In some embodiments, as shown in FIG. 6, the performance measurement method in the embodiments of the present disclosure can further include the following steps: Figure 6

[0097] ​​S508, determining, according to the performance measurement result, a completeness or a deficiency of the target data acquired by the first network function and / or network element from the second network function and / or network element.

[0098] In some embodiments, the performance measurement result in the embodiments of the present disclosure can include, but is not limited to, the number of responses and / or response number distribution information.

[0099] Further, in some embodiments, the number of responses includes at least one of the following: the number of successful responses received by the first network function and / or network element from the second network function and / or network element when acquiring the target data through multiple iterations, the maximum number of successful responses, the minimum number of successful responses, and the average number of successful responses; and the response number distribution information includes a distribution of the number of successful responses received by the first network function and / or network element from the second network function and / or network element when acquiring the target data through multiple iterations.

[0100] In some embodiments, as shown in FIG. 7a, the performance measurement method provided in the embodiments of the present disclosure can further include the following steps: Figure 7

[0101] S702a, determining, according to the number of successful responses received by the first network function and / or network element from the second network function and / or network element when acquiring the target data through multiple iterations, the maximum number of successful responses and the minimum number of successful responses;

[0102] S704a, determining, according to the maximum number of successful responses and the minimum number of successful responses, a successful response number range interval;

[0103] S706a, dividing the successful response number range interval into multiple sub-range intervals, and matching the number of successful responses received by the first network function and / or network element from the second network function and / or network element when acquiring the target data through multiple iterations to a corresponding sub-range interval to obtain the distribution of the number of successful responses received by the first network function and / or network element from the second network function and / or network element when acquiring the target data through multiple iterations.

[0104] In some other embodiments, as shown in FIG. 7b, the performance measurement method provided in the embodiments of the present disclosure can further include the following steps: Figure 7

[0105] S702b, determining, according to the number of successful responses received by the first network function and / or network element from the second network function and / or network element when acquiring the target data through multiple iterations, the average number of successful responses;

[0106] ​​S704b, calculating a variance of the number of successful responses according to the number of successful responses and the average of the number of successful responses when the first network function and / or network element receives successful responses from the second network function and / or network element through multiple iterations to obtain the target data;

[0107] S706b, determining a distribution of the number of successful responses when the first network function and / or network element receives successful responses from the second network function and / or network element through multiple iterations to obtain the target data according to the variance of the number of successful responses.

[0108] In some embodiments, as shown in FIG. 7B, the performance measurement method provided in the embodiments of the present disclosure can further include the following steps: Figure 7

[0109] S708, determining whether the target data obtained by the first network function and / or network element from the second network function and / or network element is complete according to the number of successful responses and the distribution of the number of successful responses when the first network function and / or network element receives successful responses from the second network function and / or network element through multiple iterations to obtain the target data.

[0110] In some embodiments, as shown in FIG. 7B, the performance measurement method provided in the embodiments of the present disclosure can further include the following steps: Figure 7

[0111] S710, determining a completeness of the target data obtained by the first network function and / or network element from the second network function and / or network element according to the number of successful responses and the distribution of the number of successful responses when the first network function and / or network element receives successful responses from the second network function and / or network element through multiple iterations to obtain the target data.

[0112] In some embodiments, the number of responses includes at least one of the following: a number of unsuccessful responses, a maximum value of the number of unsuccessful responses, a minimum value of the number of unsuccessful responses, and an average of the number of unsuccessful responses, when the first network function and / or network element does not receive a successful response or receives an error response through multiple iterations to obtain the target data from the second network function and / or network element; and the distribution information of the number of responses includes a distribution of the number of unsuccessful responses when the first network function and / or network element does not receive a successful response or receives an error response through multiple iterations to obtain the target data from the second network function and / or network element.

[0113] It should be noted that the number of unsuccessful responses in the embodiments of the present disclosure can be the number of error responses, or the number of times of not receiving a successful response.

[0114] In some embodiments, as shown in FIG. 7B, the performance measurement method provided in the embodiments of the present disclosure can further include the following steps: Figure 8

[0115] ​​​S802a, determining a maximum value of the number of unsuccessful responses and a minimum value of the number of unsuccessful responses according to the number of unsuccessful responses when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations;

[0116] S804a, determining a range interval of the number of unsuccessful responses according to the maximum value of the number of unsuccessful responses and the minimum value of the number of unsuccessful responses;

[0117] S806a, dividing the range interval of the number of unsuccessful responses into multiple sub-range intervals, and matching the number of unsuccessful responses when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations to a corresponding sub-range interval to obtain a distribution of the number of unsuccessful responses when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations.

[0118] In some embodiments, as shown in Figure 8 the performance measurement method provided in the embodiments of the present disclosure can further include the following steps:

[0119] S802b, determining an average value of the number of unsuccessful responses according to the number of unsuccessful responses when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations;

[0120] S804b, calculating a variance of the number of unsuccessful responses according to the number of unsuccessful responses of each second network function in the multiple iterations and the average value of the number of unsuccessful responses;

[0121] S806b, determining a distribution of the number of unsuccessful responses when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations according to the variance of the number of unsuccessful responses.

[0122] Further, in some embodiments, as shown in Figure 8 the performance measurement method provided in the embodiments of the present disclosure can further include the following steps:

[0123] S808, determining whether the target data acquired by the first network function and / or network element from the second network function and / or network element is missing according to the number of unsuccessful responses when the first network function and / or network element acquires the target data from the second network function and / or network element through multiple iterations and the distribution of the number of unsuccessful responses.

[0124] Still further, in some embodiments, as shown inFigure 8 As shown, the performance measurement method provided in the embodiments of the present disclosure can further include the following steps:

[0125] S810, determining the missing degree of the target data acquired by the first network function and / or network element from the second network function and / or network element according to the number of unsuccessful responses and the distribution of the number of unsuccessful responses when the first network function and / or network element does not receive a successful response or receives an error response from the second network function and / or network element through multiple iterations to acquire the target data.

[0126] Figure 9 As shown, the performance measurement method provided in the embodiments of the present disclosure can further include the following steps: Figure 9

[0127] S900, receiving a performance measurement service request;

[0128] S902, acquiring interaction information when the first network function and / or network element acquires target data from the second network function and / or network element through multiple iterations according to the performance measurement service request;

[0129] S904, determining the performance measurement result of the first network function and / or network element and / or the second network function and / or network element according to the interaction information when the first network function and / or network element acquires target data from the second network function and / or network element through multiple iterations.

[0130] Further, in some embodiments, as shown, the performance measurement method provided in the embodiments of the present disclosure can further include the following steps: Figure 9

[0131] S906, determining whether to update / optimize the configuration information of the first network function and / or network element and / or the second network function and / or network element according to the performance measurement result.

[0132] Still further, in some embodiments, as shown, the performance measurement method provided in the embodiments of the present disclosure can further include the following steps: Figure 9

[0133] S908, determining whether to reselect the second network function and / or network element providing the target data for the first network function and / or network element according to the performance measurement result.

[0134] The performance measurement method provided in the embodiments of the present disclosure will be described in detail below taking the first network function and / or network element acquiring target data from the second network function and / or network element through multiple iterations in the 5GC as an example.

[0135] ​​​NWDAF (Network Data Analytics Function) is a function in 5GC. When the service requested by an NWDAF service consumer needs to be completed by multiple NWDAF instances, the results of multiple NWDAF instances need to be aggregated. This aggregation process may require multiple rounds of iterations. In the current 5GC, an NWDAF that supports aggregation capabilities can obtain analysis and / or model information from multiple other NWDAFs for aggregation, and then provide the aggregated information to its service consumers. Among them, an NWDAF with analytics aggregation capability (Analytics Aggregation capability) can serve as an aggregation node (Aggregator NWDAF) to collect analysis information from other NWDAFs with analytics metadata provision capability (AnalyticsMetadata Provisioning Capability) for aggregation. An NWDAF with model aggregation capability (Model Aggregation Capability) can serve as an FL Server NWDAF to collect model information from FL Client NWDAF for aggregation. Similarly, similar scenarios may exist on the wireless side. For example, an access network device (such as a base station) acts as an aggregation node, collecting analysis and / or model information from other access network devices (such as base stations) or user equipment UE for aggregation.

[0136] In some embodiments, when a NWDAF supporting aggregation capability requests analysis and / or model information from other NWDAFs, the parameters carried in the request message may indicate the latest time that the NWDAF supporting aggregation capability expects to receive analysis and / or model information provided by other NWDAFs. If the expected latest time is reached but the analysis and / or model information is not yet ready, the NWDAF providing the analysis and / or model information will send an error response or notification to the NWDAF supporting aggregation capability. When a NWDAF supporting aggregation capability requires multiple rounds of iterations to obtain analysis and / or model information from multiple other NWDAFs for aggregation, if a certain NWDAF fails to respond in time for multiple times during these multiple rounds of iterations, it means that the performance of the NWDAF supporting aggregation capability in obtaining information from the NWDAF for aggregation is poor. Therefore, it may be necessary to reselect a suitable NWDAF to obtain information for aggregation, or to adjust the configuration information so that the NWDAF can respond in time.

[0137] It can be known from the analysis that, if the process of the network node supporting the aggregation capability (such as the NWDAF supporting the analysis information aggregation capability, the NWDAF acting as a server in the federated learning, the base station supporting the aggregation capability or the user equipment supporting the aggregation capability) aggregating the analysis and / or model information from other network nodes is monitored, the performance of the network node supporting the aggregation capability (such as the NWDAF supporting the analysis information aggregation capability, the NWDAF acting as a server in the federated learning, the base station supporting the aggregation capability or the user equipment supporting the aggregation capability) successfully obtaining the analysis and / or model information from other network nodes is understood, and the performance of the network node (such as the NWDAF supporting the analysis information aggregation capability, the NWDAF acting as a server in the federated learning, the base station supporting the aggregation capability or the user equipment supporting the aggregation capability) of the operator can be optimized.

[0138] In some embodiments, the performance measurement method provided in the embodiments of the present disclosure can be that a PM Service Consumer (measurement service producer / provider) initiates a service request to a PM Service Producer (performance measurement service consumer), and then the performance of the network function / network element supporting the aggregation capability obtaining the analysis and / or model information from other network functions / network elements is monitored by the PM Service Producer. In specific implementation, the following steps can be included:

[0139] 1) The PM Service Producer receives the service request of the PM Service Consumer, and requests to monitor the performance of the network function / network element supporting the aggregation capability obtaining the analysis and / or model information from other network functions / network elements, including the distribution of the number of successful responses in multiple rounds of iterations or the distribution of the number of unsuccessful times. The service request can be in any of the following modes, including but not limited to: subscription, request.

[0140] In some embodiments, the network function / network element supporting the aggregation capability can include but is not limited to: the NWDAF supporting the analysis information aggregation capability, the NWDAF acting as a server in the federated learning, the base station supporting the aggregation capability or the user equipment supporting the aggregation capability.

[0141] 2) The distribution of the responses received by the PM Service Producer from other network functions / network elements in multiple rounds of iterations when the network function / network element supporting aggregation capability acquires analytics and / or model information from other network functions / network elements. The number of successful / error responses received from each of the other network functions / network elements in multiple rounds of iterations can be used to analyze whether the analytics and / or model information acquired by the network function / network element supporting aggregation capability from each of the other network functions / network elements in multiple rounds of iterations is complete, i.e., whether each of the other network functions / network elements that accepted the service request successfully provided the analytics and / or model information to the network function / network element supporting aggregation capability in time in each of the multiple rounds of iterations. The distribution (e.g., variance) of the number of successful / error responses can be used to analyze the completeness / incompleteness of the analytics and / or model information acquired by the network function / network element supporting aggregation capability from multiple network functions / network elements in multiple rounds of iterations. For example, the more error responses received by the network function / network element supporting aggregation capability from a certain network function / network element in multiple rounds of iterations, the less complete the analytics and / or model information acquired from the certain network function / network element.

[0142] 2.1) The number of responses received by the network function / network element supporting aggregation capability from a certain network function / network element in multiple rounds of iterations when the network function / network element supporting aggregation capability acquires analytics and / or model information from the certain network function / network element. This includes the value of a cumulative counter (CC) related to the number of analytics / model service responses received by the network function / network element supporting aggregation capability. The network function / network element supporting aggregation capability acquires analytics / model service responses from a certain network function / network element in multiple rounds of iterations. When the network function / network element supporting aggregation capability receives an analytics / model service response from the certain network function / network element in multiple rounds of iterations, the value of the related CC is incremented (e.g., increased by 1). The value of the CC represents the number of analytics / model service responses received by the network function / network element supporting aggregation capability from the certain network function / network element in multiple rounds of iterations.

[0143] 2.2) The number of successful responses received by the network function / network element supporting aggregation capability from some other network function / network element when obtaining analytics and / or model information through multiple rounds of iteration: includes the value of a cumulative counter (CC) related to the number of normal analytics / model service responses received by the network function / network element supporting aggregation capability from some other network function / network element. The network function / network element supporting aggregation capability obtains analytics / model service responses from some other network function / network element through multiple rounds of iteration, and the value of the related CC is incremented (e.g., incremented by one) each time the network function / network element supporting aggregation capability receives a normal response from the analytics / model service of that network function / network element. The value of the CC represents the number of normal analytics / model service responses received by the network function / network element supporting aggregation capability from some other network function / network element after multiple rounds of iteration.

[0144] 2.3) The maximum value of the number of successful responses received by the network function / network element supporting aggregation capability from multiple network functions / network elements when obtaining analytics and / or model information through multiple rounds of iteration: the network function / network element supporting aggregation capability obtains analytics and / or model information from multiple network functions / network elements through multiple rounds of iteration, and after measuring the number of successful responses received by the network function / network element supporting aggregation capability from each of the multiple network functions / network elements when obtaining analytics and / or model information from each of the multiple network functions / network elements through multiple rounds of iteration, the maximum value of the measured values represents the maximum number of normal analytics / model service responses received by the network function / network element supporting aggregation capability from some other network function / network element after multiple rounds of iteration.

[0145] 2.4) The minimum value of the number of successful responses received by the network function / network element supporting aggregation capability from multiple network functions / network elements when obtaining analytics and / or model information through multiple rounds of iteration: the network function / network element supporting aggregation capability obtains analytics and / or model information from multiple network functions / network elements through multiple rounds of iteration, and after measuring the number of successful responses received by the network function / network element supporting aggregation capability from each of the multiple network functions / network elements when obtaining analytics and / or model information from each of the multiple network functions / network elements through multiple rounds of iteration, the minimum value of the measured values represents the minimum number of normal analytics / model service responses received by the network function / network element supporting aggregation capability from some other network function / network element after multiple rounds of iteration.

[0146] 2.5) The average of the number of successful responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from other network functions / network elements through multiple rounds of iteration: The average of the number of successful responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from other network functions / network elements through multiple rounds of iteration is obtained by taking the arithmetic average of the measured values of the number of successful responses received from each of the other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from each of the other network functions / network elements through multiple rounds of iteration.

[0147] 2.6) The distribution of the number of successful responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from each of the other network functions / network elements through multiple rounds of iteration: The distribution of the number of successful responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from each of the other network functions / network elements through multiple rounds of iteration is obtained by dividing the interval between the minimum value of the number of successful responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from other network functions / network elements through multiple rounds of iteration and the maximum value of the number of successful responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from other network functions / network elements through multiple rounds of iteration into sub-intervals, and then matching the number of successful responses received from each of the other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from each of the other network functions / network elements through multiple rounds of iteration to the corresponding sub-interval.

[0148] 2.7) The number of error responses received by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from a certain one of the other network functions / network elements through multiple rounds of iteration: The value of a cumulative counter (CC) related to the number of abnormal analytics / model service responses received by the network function / network element supporting aggregation capability. The value of the CC is incremented (e.g., incremented by 1) when the network function / network element supporting aggregation capability receives an abnormal response from the analytics / model service of the certain one of the other network functions / network elements through multiple rounds of iteration. The value of the CC represents the number of abnormal analytics / model service responses received by the network function / network element supporting aggregation capability from the certain one of the other network functions / network elements through multiple rounds of iteration.

[0149] 2.8) The maximum value of the number of error responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from other network functions / network elements through multiple iterations: The maximum value of the number of error responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from other network functions / network elements through multiple iterations, after taking the maximum value from the measured values of the number of error responses received in the multiple iterations when obtaining analytics and / or model information from each of the other network functions / network elements by the network function / network element supporting aggregation capability, indicates that the number of abnormal analytics / model service responses obtained by the network function / network element supporting aggregation capability from some of the other network functions / network elements through the multiple iterations is the most.

[0150] 2.9) The minimum value of the number of error responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from other network functions / network elements through multiple iterations: The minimum value of the number of error responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from other network functions / network elements through multiple iterations, after taking the minimum value from the measured values of the number of error responses received in the multiple iterations when obtaining analytics and / or model information from each of the other network functions / network elements by the network function / network element supporting aggregation capability, indicates that the number of abnormal analytics / model service responses obtained by the network function / network element supporting aggregation capability from some of the other network functions / network elements through the multiple iterations is the least.

[0151] 2.10) The average value of the number of error responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from other network functions / network elements through multiple iterations: The average value of the number of error responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analytics and / or model information from other network functions / network elements through multiple iterations, after taking the arithmetic average value from the measured values of the number of error responses received in the multiple iterations when obtaining analytics and / or model information from each of the other network functions / network elements by the network function / network element supporting aggregation capability, indicates that the average value of the number of abnormal analytics / model service responses obtained by the network function / network element supporting aggregation capability from other network functions / network elements through the multiple iterations.

[0152] 2.11) Distribution of the number of error responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analysis and / or model information from each of the other network functions / network elements through multiple rounds of iterations: sub-intervals are divided within the interval of the minimum value of the number of error responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analysis and / or model information from the other network functions / network elements through multiple rounds of iterations and the maximum value of the number of error responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analysis and / or model information from the other network functions / network elements through multiple rounds of iterations, and then the number of error responses received by the network function / network element supporting aggregation capability when obtaining analysis and / or model information from a certain network function / network element through multiple rounds of iterations is matched to the corresponding sub-interval, so as to obtain the distribution of the number of error responses received from other network functions / network elements by the network function / network element supporting aggregation capability when obtaining analysis and / or model information from each of the other network functions / network elements through multiple rounds of iterations.

[0153] 3) The PM Service Producer responds to the PM Service Consumer with the measurement results, and the response mode includes at least one of notification and response.

[0154] 4) Based on the above measurement values, it is analyzed whether the analysis and / or model information obtained by the network function / network element supporting aggregation capability from other network functions / network elements is complete / missing, and the completeness / missing degree; for example, the more the number of error responses obtained from a certain network function / network element, the less complete the analysis and / or model information obtained from the network function / network element.

[0155] 5) Based on the above measurement values and analysis results, the configuration or strategy of the NWDAF is optimized.

[0156] It is evaluated whether the missing degree of the analysis and / or model information obtained by the network function / network element supporting aggregation capability from other network functions / network elements is greater than or equal to a preconfigured threshold value, and based on the evaluation result, it is judged whether the configuration of the network function / network element needs to be updated or optimized, or based on the evaluation result, it is judged whether the analysis and / or model information needs to be obtained from a new network function / network element.

[0157] It should be noted that the analysis method of the dispersion degree in the embodiments of the present disclosure is not limited, and the variance can be used to measure the dispersion degree, or the dispersion degree can be measured based on the ratio of the performance indicators between the NWDAF supporting aggregation capability and different NWDAFs.

[0158] It should also be noted that the process of analysis and evaluation is not limited to being performed by the PM Service Producer or the PM Service Consumer in the embodiments of the present disclosure.

[0159] In one embodiment, it is assumed that the PM Service Producer is a network function / network element supporting aggregation capability (e.g., an Aggregator), the PM Service Consumer is an OSS (Operation Support Systems) / BSS (Business Support Systems), and the performance measurement service is a kind of management service (MnS). The performance measurement method provided in the embodiments of the present disclosure can include the following steps:

[0160] 1) The Aggregator receives a performance measurement service request from the OSS / BSS, requests to monitor abnormal situations when the Aggregator acquires analysis information from other different network functions / network elements, and the request mode includes at least one of subscription and request.

[0161] In one embodiment, the Aggregator includes an Aggregator NWDAF, and the other network functions include other NWDAFs. In another embodiment, the Aggregator includes an FL Server NWDAF, and the other network functions / network elements include FL Client NWDAFs. In another embodiment, the Aggregator includes a base station, and the other network functions / network elements include other base stations or UEs. In another embodiment, the Aggregator includes a UE, and the other network functions / network elements include other UEs.

[0162] 2)PM Service When the Producer obtains the analysis and / or model information obtained by the Aggregator from other network functions / network elements, the distribution of responses received in multiple rounds of iterations includes but is not limited to: the number of successful / error responses received from each other network function / network element in multiple rounds of iterations and the distribution of the number of successful / error responses; the number of successful / error responses can be used to analyze whether the analysis and / or model information obtained by the Aggregator from each other network function / network element in multiple rounds of iterations is complete, that is, whether each other network function / network element that accepts the service request has successfully provided the analysis and / or model information to the Aggregator on time in each round of iteration; the distribution of the number of successful / error responses (such as variance) can be used to analyze the completeness / incompleteness of the analysis and / or model information obtained by the Aggregator from multiple other network functions / network elements in multiple rounds of iterations; for example: in multiple rounds of iterations, the more error responses the Aggregator receives from a certain network function / network element, the more incomplete the analysis and / or model information obtained from the network function / network element. During specific implementation, the measured data includes but is not limited to the following:

[0163] 2.1) Measure the number of responses received by the Aggregator when obtaining analysis and / or model information from another network function / network element over multiple iterations: This includes the value of a cumulative counter (CC) associated with the number of analysis / model service responses received by the Aggregator. The Aggregator obtains analysis / model service responses from another network function / network element over multiple iterations. During these multiple iterations, the value of the associated CC increments (e.g., by 1) when the Aggregator receives an analysis / model service response from the network function / network element. The CC value represents the number of analysis / model service responses obtained by the Aggregator from the other network function / network element over multiple iterations.

[0164] 2.2) Measure the number of successful responses received by the Aggregator when obtaining analysis and / or model information from another network function / network element through multiple iterations: This includes the value of a cumulative counter (CC) associated with the number of normal analysis / model service responses received by the Aggregator. The Aggregator obtains analysis / model service responses from another network function / network element through multiple iterations. During these multiple iterations, when the Aggregator receives a normal response from the analysis / model service of the network function / network element, the value of the related CC increments (e.g., by 1). The value of this CC represents the number of normal analysis / model service responses obtained by the Aggregator from the other network function / network element after multiple iterations.

[0165] 2.3) The maximum value of the number of successful responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from other network functions / network elements through multiple iterations: After obtaining the measured values of the number of successful responses received when the Aggregator obtains analytics and / or model information from a certain network function / network element through multiple iterations, the maximum value is taken from the measured values to represent the maximum number of normal analytics / model service responses obtained by the Aggregator from a certain network function / network element after multiple iterations.

[0166] 2.4) The minimum value of the number of successful responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from other network functions / network elements through multiple iterations: After obtaining the measured values of the number of successful responses received when the Aggregator obtains analytics and / or model information from a certain network function / network element through multiple iterations, the minimum value is taken from the measured values to represent the minimum number of normal analytics / model service responses obtained by the Aggregator from a certain network function / network element after multiple iterations.

[0167] 2.5) The average value of the number of successful responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from other network functions / network elements through multiple iterations: After obtaining the measured values of the number of successful responses received when the Aggregator obtains analytics and / or model information from a certain network function / network element through multiple iterations, the arithmetic mean is taken from the measured values to represent the average value of the number of normal analytics / model service responses obtained by the Aggregator from other network functions / network elements after multiple iterations.

[0168] 2.6) Measure the distribution of the number of successful responses received from other network functions / network elements when Aggregator obtains analytics and / or model information from each of the other network functions / network elements through multiple rounds of iterations: when Aggregator obtains analytics and / or model information from multiple network functions / network elements through multiple rounds of iterations, divide the interval between the minimum value of the number of successful responses received from multiple network functions / network elements when Aggregator obtains analytics and / or model information from each of the multiple network functions / network elements through multiple rounds of iterations and the maximum value of the number of successful responses received from multiple network functions / network elements when Aggregator obtains analytics and / or model information from each of the multiple network functions / network elements through multiple rounds of iterations into sub-intervals, and then match the number of successful responses received from a certain network function / network element when Aggregator obtains analytics and / or model information from the certain network function / network element through multiple rounds of iterations to a corresponding sub-interval, thereby obtaining the distribution of the number of successful responses received from other network functions / network elements when Aggregator obtains analytics and / or model information from each of the other network functions / network elements through multiple rounds of iterations.

[0169] 2.7) Measure the number of error responses received from a certain network function / network element when Aggregator obtains analytics and / or model information from the certain network function / network element through multiple rounds of iterations: include the value of a cumulative counter (CC) related to the number of abnormal analytics / model service responses received by Aggregator. Aggregator obtains analytics / model service responses from a certain network function / network element through multiple rounds of iterations, and the value of the related CC is incremented (e.g., the value is incremented by 1) when Aggregator receives an abnormal response from the analytics / model service of the network function / network element in the multiple rounds of iterations; the value of the CC represents the number of abnormal analytics / model service responses received by Aggregator from the certain network function / network element after multiple rounds of iterations.

[0170] 2.8) Measure the maximum value of the number of error responses received from multiple network functions / network elements when Aggregator obtains analytics and / or model information from the multiple network functions / network elements through multiple rounds of iterations: when Aggregator obtains analytics and / or model information from multiple network functions / network elements through multiple rounds of iterations, take the maximum value from the measured values of the number of error responses received from a certain network function / network element when Aggregator obtains analytics and / or model information from the certain network function / network element through multiple rounds of iterations to represent the maximum number of abnormal analytics / model service responses received by Aggregator from the certain network function / network element after multiple rounds of iterations.

[0171] 2.9) The minimum value of the number of error responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from other network functions / network elements through multiple iterations: After obtaining the measurement of the number of error responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from a certain network function / network element through multiple iterations, the minimum value is taken from the measurements to represent the minimum number of abnormal analytics / model service responses obtained by the Aggregator from a certain network function / network element after multiple iterations.

[0172] 2.10) The average value of the number of error responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from other network functions / network elements through multiple iterations: After obtaining the measurement of the number of error responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from a certain network function / network element through multiple iterations, the average value is taken from the measurements to represent the average value of the number of abnormal analytics / model service responses obtained by the Aggregator from other network functions / network elements after multiple iterations.

[0173] 2.11) The distribution of the number of error responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from each network function / network element through multiple iterations: When the Aggregator obtains analytics and / or model information from other network functions / network elements through multiple iterations, the interval between the minimum value of the number of error responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from a certain network function / network element through multiple iterations and the maximum value of the number of error responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from a certain network function / network element through multiple iterations is divided into sub-intervals, and the number of error responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from a certain network function / network element through multiple iterations is matched to the corresponding sub-interval, thereby obtaining the distribution of the number of error responses received from other network functions / network elements when the Aggregator obtains analytics and / or model information from each network function / network element through multiple iterations.

[0174] 3) The PM Service Producer responds to the PM Service Consumer with the measurement results, and the response includes at least one of the following: notification, response.

[0175] 4) Based on the above measurement values, analyze whether the analysis and / or model information obtained by the Aggregator from other multiple network functions / network elements is complete / missing, the completeness / missing degree; for example: the more the number of error responses obtained from a certain network function / network element in multiple iterations, the more incomplete the analysis and / or model information obtained by the Aggregator from the network function / network element.

[0176] 5) Based on the above measurement values and analysis results, optimize the configuration or policy of the NWDAF. For example, evaluate whether the missing degree of the analysis and / or model information obtained by the Aggregator from other multiple network functions / network elements is greater than or equal to a preconfigured threshold, judge whether the configuration of the network function / network element needs to be updated or optimized based on the evaluation result, or judge whether the analysis and / or model information needs to be obtained from a new network function / network element based on the evaluation result.

[0177] Based on the same inventive concept, the performance measurement method applied to the performance measurement service provider device is also provided in the embodiments of the present disclosure. In principle, the method can be executed by any electronic device with computing processing capability.

[0178] Figure 10 A flow chart of a performance measurement method applied to a performance measurement service user device is shown in the embodiments of the present disclosure. As shown in the figure, the method can include the following steps: Figure 10

[0179] S1002, initiate a performance measurement service request to the performance measurement service provider device, wherein the performance measurement service request is used to request to obtain the interaction information when the first network function and / or network element obtains the target data from the second network function and / or network element through multiple iterations;

[0180] S1004, receive the interaction information returned by the performance measurement service provider device according to the performance measurement service request, and determine the performance measurement result of the first network function and / or network element, and / or the second network function and / or network element according to the interaction information when the first network function and / or network element obtains the target data from the second network function and / or network element through multiple iterations.

[0181] In some embodiments, the performance measurement service request is also used to request to obtain the interaction information when the network side device obtains the target data from the user equipment through multiple iterations, so that the performance measurement service provider device determines the performance measurement result of the network side device and / or the user equipment according to the interaction information.

[0182] ​In some embodiments, the performance measurement service request is further used to request to obtain the interaction information of the network side device from the user equipment through multiple iterations to obtain the target data, so that the performance measurement service provider device determines the performance measurement result of the network side device and / or the user equipment according to the interaction information.

[0183] Based on the same inventive concept, the embodiments of the present disclosure further provide a performance measurement method applied to a performance measurement service provider device. In principle, the method can be executed by any electronic device with computing processing capability.

[0184] Figure 11 A flowchart of a performance measurement method applied to a performance measurement service provider device in the embodiments of the present disclosure is shown in FIG. 11. As shown in FIG. 11, the method can include the following steps: Figure 11

[0185] S1102, receiving a performance measurement service request from a performance measurement service provider device, wherein the performance measurement service request is used to request to obtain the interaction information of a first network function and / or network element from a second network function and / or network element through multiple iterations to obtain target data;

[0186] S1104, according to the performance measurement service request, obtaining the interaction information of the first network function and / or network element from the second network function and / or network element through multiple iterations to obtain the target data;

[0187] S1106, according to the interaction information of the first network function and / or network element from the second network function and / or network element through multiple iterations to obtain the target data, determining the performance measurement result of the first network function and / or network element, and / or the second network function and / or network element.

[0188] In some embodiments, the performance measurement service request is further used to request to obtain the interaction information of the network side device from the user equipment through multiple iterations to obtain the target data, so that the performance measurement service provider device determines the performance measurement result of the network side device and / or the user equipment according to the interaction information.

[0189] In some embodiments, the performance measurement service request is further used to request to obtain the interaction information of the network side device from the user equipment through multiple iterations to obtain the target data, so that the performance measurement service provider device determines the performance measurement result of the network side device and / or the user equipment according to the interaction information.

[0190] Based on the same inventive concept, the embodiments of the present disclosure further provide a performance measurement system, as described in the following embodiments. Since the principles of the system embodiments solve problems are similar to the above-mentioned method embodiments, the implementation of the system embodiments can be referred to the implementation of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0191] ​Figure 12 Fig. 1 shows a schematic diagram of a performance measurement system according to an embodiment of the present disclosure. Figure 12 As shown in Fig. 1, the system can include a performance measurement service provider device 1201 and a performance measurement service user device 1202.

[0192] In some embodiments, the performance measurement service user device 1202 is configured to initiate a performance measurement service request to the performance measurement service provider device, wherein the performance measurement service request is configured to request interaction information of the first network function and / or network element when acquiring the target data from the second network function and / or network element through multiple iterations; the performance measurement service provider device 1201 is configured to acquire the interaction information of the first network function and / or network element when acquiring the target data from the second network function and / or network element through multiple iterations according to the performance measurement service request, and feed back to the performance measurement service user device; wherein the performance measurement service user device is further configured to determine the performance measurement result of the first network function and / or network element, and / or the second network function and / or network element according to the interaction information of the first network function and / or network element when acquiring the target data from the second network function and / or network element through multiple iterations.

[0193] In the above embodiments, the interaction process between the performance measurement service provider device and the performance measurement service user device is as shown in Fig. 2. Figure 13

[0194] In some embodiments, the performance measurement service user device is configured to initiate a performance measurement service request to the performance measurement service provider device, wherein the performance measurement service request is configured to request interaction information of the first network function and / or network element when acquiring the target data from the second network function and / or network element through multiple iterations; the performance measurement service provider device is configured to acquire the interaction information of the first network function and / or network element when acquiring the target data from the second network function and / or network element through multiple iterations according to the performance measurement service request, and determine the performance measurement result of the first network function and / or network element, and / or the second network function and / or network element according to the interaction information of the first network function and / or network element when acquiring the target data from the second network function and / or network element through multiple iterations, and feed back the performance measurement result to the performance measurement service user device.

[0195] In the above embodiments, the interaction process between the performance measurement service provider device and the performance measurement service user device is as shown in Fig. 2. Figure 14

[0196] ​​In some embodiments, the performance measurement service request described above is further used to request to obtain interaction information of the network-side device when the target data is obtained by the user equipment through multiple iterations, so that the performance measurement service provider device or the performance measurement service user device determines the performance measurement result of the network-side device and / or the user equipment according to the interaction information.

[0197] In some other embodiments, the performance measurement service request described above is further used to request to obtain interaction information of the user equipment when the target data is obtained by the user equipment from other user equipment through multiple iterations, so that the performance measurement service provider device or the performance measurement service user device determines the performance measurement result of the user equipment according to the interaction information.

[0198] Those skilled in the art can understand that each aspect of the present disclosure can be embodied in the form of a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" herein.

[0199] Based on the same inventive concept, the electronic device in the embodiments of the present disclosure is also provided, which comprises: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the performance measurement method of any one of the above by executing the executable instructions. Since the principle of solving problems of the electronic device embodiment is similar to that of the above method embodiment, the implementation of the electronic device embodiment can be referred to the implementation of the above method embodiment, and the repeated parts will not be described here.

[0200] The electronic device 800 according to this embodiment of the present disclosure will be described below with reference to Figure 15 Figure 15 The display electronic device 1500 is only an example and should not bring any limitation to the function and use range of the embodiments of the present disclosure.

[0201] As shown in Figure 15 , the electronic device 1500 is in the form of a general computing device. The components of the electronic device 1500 can include but are not limited to: the above-mentioned at least one processing unit 1510, the above-mentioned at least one storage unit 1520, and a bus 1530 connecting different system components, including the storage unit 1520 and the processing unit 1510.

[0202] ​The storage unit stores program codes which can be executed by the processing unit 1510, so that the processing unit 1510 performs the steps of the method according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of the present specification. For example, the processing unit 1510 can perform the following steps of the above method embodiments: obtaining interaction information of the first network node when the second network node obtains target data through multiple iterations; and determining the performance measurement result of the first network node and / or the second network node according to the interaction information.

[0203] In some embodiments, the processing unit 1510 can perform the following steps of the above method embodiments: obtaining interaction information of the first network function and / or network element when the second network function and / or network element obtains target data through multiple iterations; and determining the performance measurement result of the first network function and / or network element, and / or the second network function and / or network element according to the interaction information.

[0204] In some other embodiments, the processing unit 1510 can perform the following steps of the above method embodiments: obtaining interaction information of the network side device when the user equipment obtains target data through multiple iterations; and determining the performance measurement result of the network side device and / or the user equipment according to the interaction information.

[0205] In some other embodiments, the processing unit 1510 can perform the following steps of the above method embodiments: obtaining interaction information of the user equipment when the other user equipment obtains target data through multiple iterations; and determining the performance measurement result of the user equipment according to the interaction information.

[0206] The storage unit 1520 can include a readable medium in the form of volatile storage unit, such as random access memory (RAM) 15201 and / or cache memory 15202, and can further include read-only memory (ROM) 15203.

[0207] The storage unit 1520 can further include program / utility 15204 having a set of programs / modules 15205, including without limitation, an operating system, one or more application programs, other programs, and programmatic data, each or any combination thereof, which can include implementation of a network environment.

[0208] The bus 1530 can represent one or more of several types of bus structures, including a storage unit bus or bus controller, a peripheral bus, a graphics acceleration port, a processing unit bus, or a local bus using any of a variety of bus architectures.

[0209] The electronic device 1500 can also communicate with one or more external devices 1540 such as a keyboard or pointing device, a Bluetooth device, or a database and / or 1550 one or more devices that enable a user to interact with the electronic device 1500 and / or one or more devices that enable the electronic device 1500 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface(s) 1550. Still yet, such communication can occur electronically over a network such as a local area network (LAN) and / or a wide area network (WAN) such as the Internet. As an example, the network adapter 1560 can enable the electronic device 1500 to communicate with an Internet or one or more Intranets. In any such implementation, the network adapter 1560 can serve in a role of interfacing the electronic device 1500 to the network. When the electronic device 1500 is in use, the network adapter 1560 can enable multiple devices to transmit information through the network during the same time period.

[0210] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0211] Based on the same inventive concept, the present embodiment also provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the performance measurement method of any one of the above. Since the principle of solving problems of the computer readable storage medium embodiment is similar to that of the above method embodiment, the implementation of the computer readable storage medium embodiment can be referred to the implementation of the above method embodiment, and the repeated parts will not be described here.

[0212] More specific examples of the computer readable storage medium in the present disclosure can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0213] In this disclosure, a computer readable storage medium can include a data signal transporting or broadcasting computer readable program code embodied in the data signal. The data signal can also be transmitted over a network including the Internet. Additionally, the computer readable storage medium can also be any tangible storage medium which can be used to store and / or carry the program codes.

[0214] Optionally, program code embodied on a computer readable storage medium can also be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0215] In an implementation, the program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0216] Based on the same inventive concept, the present disclosure further provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implement the performance measurement method of any one of the above method embodiments. Since the principle of solving problems of this computer program product embodiment is similar to the above method embodiments, the implementation of this computer program product embodiment can be referred to the implementation of the above method embodiments, and the repeated parts will not be described here.

[0217] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such a division is not mandatory. Indeed, according to an embodiment of the present disclosure, the features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functionalities of one module or unit described above can be further divided into several modules or units.

[0218] Moreover, although individual steps of the methods in the present disclosure are described in a particular order in the drawings, this is not required or implied, nor is it necessary to perform all of the steps shown to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into a single step, performed in a different order, broken down into multiple steps, and / or the like.

[0219] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or by hardware coupled with software, as described above. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0220] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known use or custom in the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method of performance measurement, characterized by, The method comprises: obtaining interaction information of a first network function and / or network element from a second network function and / or network element when the first network function and / or network element obtains target data from the second network function and / or network element through multiple iterations; determining a performance measurement result of the first network function and / or network element and / or the second network function and / or network element according to the interaction information.

2. The performance measurement method of claim 1, wherein, The first network function and / or network element is a network function and / or network element supporting a data analysis function.

3. The method of claim 1, wherein, The first network function and / or network element is a network function and / or network element supporting an aggregation capability, the aggregation capability comprising an analysis information aggregation capability and / or a model information aggregation capability.

4. The method of claim 3, wherein, The method of obtaining the interaction information of the first network function and / or network element from the second network function and / or network element when the first network function and / or network element obtains target data from the second network function and / or network element through multiple iterations comprises at least one of: obtaining the interaction information of a first network function and / or network element supporting an analysis information aggregation capability from a second network function and / or network element when the first network function and / or network element obtains analysis information from the second network function and / or network element through multiple iterations; obtaining the interaction information of a first network function and / or network element supporting a model information aggregation capability from a second network function and / or network element when the first network function and / or network element obtains model information from the second network function and / or network element through multiple iterations.

5. The method of claim 1, wherein, The target data comprises analysis information and / or model information.

6. The method of claim 1, wherein, The interaction information comprises a response and / or a notification.

7. The method of claim 1, wherein, The method further comprises at least one of: obtaining interaction information of a network side device from a user equipment when the network side device obtains target data from the user equipment through multiple iterations; obtaining interaction information of a user equipment from other user equipment when the user equipment obtains target data from the other user equipment through multiple iterations.

8. The method of claim 1, wherein, The method further comprises: determining whether the target data obtained by the first network function and / or network element from the second network function and / or network element is complete or missing according to the performance measurement result.

9. The method of claim 1, wherein, The method further comprises: determining a completeness degree or a missing degree of the target data obtained by the first network function and / or network element from the second network function and / or network element according to the performance measurement result.

10. The method of claim 1, wherein, The performance measurement result comprises a response quantity and / or response quantity distribution information.

11. The method of performance measurement of claim 10, wherein, The response quantity comprises at least one of a successful response quantity, a maximum value of the successful response quantity, a minimum value of the successful response quantity, and an average value of the successful response quantity when the first network function and / or network element receives a successful response when obtaining target data from the second network function and / or network element through multiple iterations. The response quantity distribution information comprises a distribution of the successful response quantity when the first network function and / or network element receives a successful response when obtaining target data from the second network function and / or network element through multiple iterations.

12. The method of performance measurement of claim 11, wherein, The method further comprises: determining a maximum value of the successful response quantity and a minimum value of the successful response quantity according to the successful response quantity when the first network function and / or network element receives a successful response when obtaining target data from the second network function and / or network element through multiple iterations; determining a successful response quantity range interval according to the maximum value of the successful response quantity and the minimum value of the successful response quantity. The success response number range interval is divided into a plurality of sub-range intervals, and the success response number of the successful response received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations is matched to a corresponding sub-range interval, to obtain the distribution of the success response number when the first network function and / or network element receives the successful response from the second network function and / or network element when obtaining the target data through multiple iterations.

13. The method of claim 11, wherein, The method further comprises: determining a success response number average according to the success response number of the successful response received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations; calculating a success response number variance according to the success response number of the successful response received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations and the success response number average; determining the distribution of the success response number when the first network function and / or network element receives the successful response from the second network function and / or network element when obtaining the target data through multiple iterations according to the success response number variance.

14. The method of performance measurement according to any of claims 11 to 13, characterized in that, The method further comprises: determining whether the target data obtained by the first network function and / or network element from the second network function and / or network element is complete according to the success response number when the first network function and / or network element obtains the target data from the second network function and / or network element through multiple iterations and the distribution of the success response number.

15. The method of claim 13, wherein, The method further comprises: determining the completeness of the target data obtained by the first network function and / or network element from the second network function and / or network element according to the success response number when the first network function and / or network element obtains the target data from the second network function and / or network element through multiple iterations and the distribution of the success response number.

16. The method of claim 11, wherein, The response number comprises at least one of the following: a non-success response number of the non-success response or the error response not received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations, a non-success response number maximum, a non-success response number minimum, and a non-success response number average; The response number distribution information comprises the distribution of the non-success response number of the non-success response or the error response not received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations.

17. The method of performance measurement of claim 16, wherein, The method further comprises: determining the non-success response number maximum and the non-success response number minimum according to the non-success response number of the non-success response or the error response not received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations; determining a non-success response number range interval according to the non-success response number maximum and the non-success response number minimum; The unsuccessful response number range interval is divided into a plurality of sub-range intervals, and the unsuccessful response number of the unsuccessful response received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations is matched to a corresponding sub-range interval, to obtain the distribution of the number of times of not receiving a successful response or receiving an error response by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations.

18. The method of performance measurement of claim 16, wherein, The method further comprises: determining an average value of the unsuccessful response number according to the unsuccessful response number of the unsuccessful response not received or received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations; calculating a variance of the unsuccessful response number according to the unsuccessful response number of the unsuccessful response not received or received by each second network function body in the multiple iterations and the average value of the unsuccessful response number; determining the distribution of the number of times of not receiving a successful response or receiving an error response by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations according to the variance of the unsuccessful response number.

19. The method of measuring performance according to any one of claims 16 to 18, wherein, The method further comprises: determining whether the target data obtained by the first network function and / or network element from the second network function and / or network element is missing according to the unsuccessful response number of the unsuccessful response not received or received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations and the distribution of the unsuccessful response number.

20. The method of claim 19, wherein, The method further comprises: determining the degree of missing of the target data obtained by the first network function and / or network element from the second network function and / or network element according to the unsuccessful response number of the unsuccessful response not received or received by the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations and the distribution of the unsuccessful response number.

21. The method of claim 1, wherein, The interaction information of the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations is obtained, comprising: receiving a performance measurement service request; obtaining the interaction information of the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations according to the performance measurement service request.

22. The method of performance measurement of claim 1, wherein, The method further comprises: determining whether to update / optimize the configuration information of the first network function and / or network element and / or the second network function and / or network element according to the performance measurement result.

23. The method of performance measurement of claim 1, wherein, The method further comprises: determining whether to reselect the second network function and / or network element for providing the target data for the first network function and / or network element according to the performance measurement result.

24. A method of performance measurement, characterized by, Applied to a performance measurement service user equipment, comprising: initiating a performance measurement service request to a performance measurement service provider equipment, wherein the performance measurement service request is used to request to obtain the interaction information of the first network function and / or network element from the second network function and / or network element when obtaining the target data through multiple iterations; The performance measurement service provider device receives the interaction information returned according to the performance measurement service request, and determines the performance measurement result of the first network function and / or network element and / or the second network function and / or network element according to the interaction information when the first network function and / or network element acquires target data from the second network function and / or network element through multiple iterations.

25. A method of performance measurement, characterized by, The application is applied to a performance measurement service provider device, comprising: receiving a performance measurement service request from a performance measurement service provider device, wherein the performance measurement service request is used to request to acquire interaction information when a first network function and / or network element acquires target data from a second network function and / or network element through multiple iterations; acquiring the interaction information when the first network function and / or network element acquires target data from the second network function and / or network element through multiple iterations according to the performance measurement service request; determining the performance measurement result of the first network function and / or network element and / or the second network function and / or network element according to the interaction information when the first network function and / or network element acquires target data from the second network function and / or network element through multiple iterations.

26. A performance measurement system characterized by, comprising: a performance measurement service provider device and a performance measurement service user device; wherein the performance measurement service user device is used to initiate a performance measurement service request to the performance measurement service provider device, wherein the performance measurement service request is used to request to acquire interaction information when a first network function and / or network element acquires target data from a second network function and / or network element through multiple iterations; the performance measurement service provider device is used to acquire the interaction information when the first network function and / or network element acquires target data from the second network function and / or network element through multiple iterations according to the performance measurement service request, and feed back to the performance measurement service user device; wherein the performance measurement service user device is further used to determine the performance measurement result of the first network function and / or network element and / or the second network function and / or network element according to the interaction information when the first network function and / or network element acquires target data from the second network function and / or network element through multiple iterations.

27. A performance measurement system characterized by, comprising: a performance measurement service provider device and a performance measurement service user device; wherein the performance measurement service user device is used to initiate a performance measurement service request to the performance measurement service provider device, wherein the performance measurement service request is used to request to acquire interaction information when a first network function and / or network element acquires target data from a second network function and / or network element through multiple iterations; The performance measurement service provider device is configured to, according to the performance measurement service request, acquire interaction information of the first network function and / or network element and the second network function and / or network element in the process of the second network function and / or network element acquiring the target data through multiple iterations from the first network function and / or network element, and determine the performance measurement result of the first network function and / or network element and / or the second network function and / or network element according to the interaction information, and feed back the performance measurement result to the performance measurement service user device.

28. An electronic device, comprising: Comprise: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the performance measurement method of any one of claims 1 to 20 via executing the executable instructions.

29. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the performance measurement method of any one of claims 1 to 25.

30. A computer program product, comprising: Computer program or instructions, characterized in that the computer program or instructions are executed by the processor to implement the performance measurement method of any one of claims 1 to 25.

Images