Methods, apparatuses and systems for measuring performance of nwdaf network element services

Abstract

The present disclosure relates to a method, device and system for measuring the performance of a NWDAF network element. The method comprises: obtaining original measurement values of a network data analysis function (NWDAF) network element; generating performance measurement values corresponding to at least one NWDAF network element according to the original measurement values, wherein the performance measurement values are used for statistical analysis related to the performance of the NWDAF network element, evaluation and optimization of the service performance of the NWDAF network element; and reporting the performance measurement values externally. The present disclosure can realize the performance measurement values of the NWDAF network element, and further realize the statistical analysis related to the performance of the NWDAF network element, thereby evaluating and optimizing the service performance of the NWDAF network element.

Description

Technical Field

[0001] This disclosure relates to the field of wireless communication, and in particular to a method, apparatus and system for measuring the service performance of NWDAF network elements. Background Technology

[0002] In the Rel-17 study, TS 23.288 defines the NWDAF (Network Data Analytics Function) network element, which is a network function in 5GC (5G core network). It supports the collection of data from NFs (Network Functions), AFs (Application Functions), and OAM (Operations, Administration, Maintenance), and provides analytical information to NFs, AFs, and OAM. Summary of the Invention

[0003] The inventors discovered through research that NWDAF service consumer devices may have performance requirements for NWDAF services. If the NWDAF network element cannot meet these performance requirements, it will affect the service experience of the NWDAF service consumer device. For example, regarding latency performance requirements, if the NWDAF network element cannot complete the service within a specified time, it will cause the NWDAF service consumer device to wait for too long, thereby affecting the service experience of the NWDAF service consumer device.

[0004] In view of at least one of the above technical problems, this disclosure provides a method, apparatus and system for measuring the service performance of NWDAF network elements, which can realize NWDAF performance measurement values ​​and then realize statistical analysis related to NWDAF performance.

[0005] According to one aspect of this disclosure, a method for measuring the performance of network element services for network data analysis functions is provided, comprising:

[0006] Obtain the raw measurement values ​​of the NWDAF network element for network data analysis;

[0007] Based on the original measurement values, performance measurement values ​​corresponding to at least one NWDAF network element are generated, wherein the performance measurement values ​​are used for statistical analysis related to the performance of the NWDAF network element, to evaluate and optimize the service performance of the NWDAF network element;

[0008] The performance measurements reported externally.

[0009] In some embodiments of this disclosure, the raw measurement value includes at least one of the following: a request or subscription received by the NWDAF network element, a response or notification generated by the NWDAF network element, an error response or notification generated by the NWDAF network element, and a success response or notification generated by the NWDAF network element.

[0010] In some embodiments of this disclosure, the performance measurement values ​​include at least one of the following: the number of requests or subscriptions received by the NWDAF network element, the number of responses or notifications generated by the NWDAF network element, the number of error responses or notifications generated by the NWDAF network element, and the number of successful responses or notifications generated by the NWDAF network element.

[0011] In some embodiments of this disclosure, the number of requests or subscriptions received by the NWDAF network element includes the value of a first cumulative counter, wherein the first cumulative counter is a cumulative counter related to the number of requests or subscriptions received by the NWDAF.

[0012] In some embodiments of this disclosure, generating performance measurement values ​​corresponding to at least one NWDAF network element based on the original measurement values ​​includes:

[0013] When an NWDAF network element receives a request or subscription from an NWDAF performance measurement service consumer device, it increments the value of the first cumulative counter by 1.

[0014] In some embodiments of this disclosure, the number of responses or notifications generated by the NWDAF network element includes the value of a second cumulative counter, wherein the second cumulative counter is a cumulative counter related to the number of responses or notifications generated by the NWDAF.

[0015] In some embodiments of this disclosure, generating performance measurement values ​​corresponding to at least one NWDAF network element based on the original measurement values ​​includes:

[0016] When an NWDAF network element generates a response or notification, the value of the second cumulative counter is incremented by 1.

[0017] In some embodiments of this disclosure, the number of error responses or notifications generated by the NWDAF network element includes the value of a third cumulative counter, wherein the third cumulative counter is a cumulative counter related to the number of error responses or notifications generated by the NWDAF.

[0018] In some embodiments of this disclosure, generating performance measurement values ​​corresponding to at least one NWDAF network element based on the original measurement values ​​includes:

[0019] When an NWDAF network element generates an error response or notification, the value of the third cumulative counter is incremented by 1.

[0020] In some embodiments of this disclosure, the number of successful responses or notifications generated by the NWDAF network element includes the value of a fourth cumulative counter, wherein the fourth cumulative counter is a cumulative counter related to the number of successful responses or notifications generated by the NWDAF.

[0021] In some embodiments of this disclosure, generating performance measurement values ​​corresponding to at least one NWDAF network element based on the original measurement values ​​includes:

[0022] When the NWDAF element generates a successful response or notification, the value of the fourth cumulative counter is incremented by 1.

[0023] In some embodiments of this disclosure, when the performance measurement is associated with specific information, the performance measurement value includes the value of a sub-cumulative counter associated with the specific information, wherein the specific information includes at least one of single network slice selection assistance information, analysis type, request type, subscription type, and analysis identifier.

[0024] In some embodiments of this disclosure, when the specific information is an analysis type, a request type, or a subscription type, the value of the sub-cumulative counter associated with each performance measurement value includes the value of the sub-cumulative counter associated with an analysis service request, the value of the sub-cumulative counter associated with a data management service request, and the value of the sub-cumulative counter associated with a machine learning model service request.

[0025] According to another aspect of this disclosure, a method for measuring the performance of network element services for network data analysis functions is provided, comprising:

[0026] The system receives reported performance measurement values, which are performance measurement values ​​generated by the performance measurement service producer device based on the original measurement values ​​of the network data analysis function NWDAF network element, corresponding to at least one NWDAF network element. These performance measurement values ​​are used for statistical analysis related to the performance of the NWDAF network element, to evaluate and optimize the service performance of the NWDAF network element.

[0027] In some embodiments of this disclosure, the method for measuring the performance of network element services for network data analysis functions further includes:

[0028] Based on the performance measurements, perform statistical analysis related to the performance of NWDAF network elements to evaluate and optimize the service performance of NWDAF network elements.

[0029] According to another aspect of this disclosure, a performance measurement service producer apparatus is provided, comprising:

[0030] The raw measurement value acquisition module is configured to acquire the raw measurement values ​​of the network elements in the Network Data Analysis Function (NWDAF).

[0031] The performance measurement value generation module is configured to generate performance measurement values ​​corresponding to at least one NWDAF network element based on the original measurement values, wherein the performance measurement values ​​are used for statistical analysis related to the performance of the NWDAF network element to evaluate and optimize the service performance of the NWDAF network element.

[0032] The performance measurement reporting module is configured to report the performance measurement values ​​to external parties.

[0033] In some embodiments of this disclosure, the performance measurement service producer apparatus is configured to perform operations implementing the methods described in any of the above embodiments.

[0034] According to another aspect of this disclosure, a performance measurement service consumer device is provided, comprising:

[0035] The performance measurement value receiving module is configured to receive reported performance measurement values, wherein the performance measurement values ​​are generated by the performance measurement service producer device based on the original measurement values ​​of the network data analysis function NWDAF network element, corresponding to at least one NWDAF network element. The performance measurement values ​​are used for statistical analysis related to the performance of the NWDAF network element, to evaluate and optimize the service performance of the NWDAF network element.

[0036] In some embodiments of this disclosure, the performance measurement service consumer device further includes:

[0037] The performance analysis module is configured to perform statistical analysis related to the performance of NWDAF network elements based on the performance measurements, and to evaluate and optimize the service performance of NWDAF network elements.

[0038] According to another aspect of this disclosure, a system for measuring the performance of network element services for network data analysis functions is provided, including a performance measurement service producer device as described in any of the above embodiments and a performance measurement service consumer device as described in any of the above embodiments.

[0039] According to another aspect of this disclosure, a computer apparatus is provided, comprising:

[0040] The memory is configured to store instructions;

[0041] A processor is configured to execute the instructions, causing the computer device to perform operations implementing the methods described in any of the above embodiments.

[0042] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.

[0043] This disclosure enables the measurement of NWDAF network element performance, and then enables statistical analysis related to NWDAF network element performance, thereby evaluating and optimizing the service performance of NWDAF network elements. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 These are schematic diagrams illustrating some embodiments of the method disclosed herein for measuring the performance of network element services for network data analysis functions.

[0046] Figure 2 This is a schematic diagram of some other embodiments of the method disclosed herein for measuring the performance of network element services for network data analysis functions.

[0047] Figure 3 Schematic diagrams of some embodiments of the performance measurement service producer apparatus of this disclosure.

[0048] Figure 4 Schematic diagrams of some embodiments of consumer devices for performance measurement services according to this disclosure.

[0049] Figure 5 These are schematic diagrams of some embodiments of a system disclosed herein for measuring the performance of network element services for network data analysis functions.

[0050] Figure 6 This is a schematic diagram of the structure of some embodiments of the computer device disclosed herein. Detailed Implementation

[0051] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0052] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0053] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0054] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0055] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0056] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0057] Figure 1 This diagram illustrates some embodiments of the method disclosed herein for measuring the performance of network element services for network data analysis functions. Preferably, this embodiment can be executed by the performance measurement service producer device of this disclosure or the system disclosed herein for measuring the performance of network element services for network data analysis functions. The method may include at least one step from steps 11-13, wherein:

[0058] Step 11: Obtain the raw measurement values ​​of the network element in the Network Data Analysis Function (NWDAF).

[0059] In some embodiments of this disclosure, the original measurement value may be at least one of the following: a request or subscription received by the NWDAF network element, a response or notification generated by the NWDAF network element, an error response or notification generated by the NWDAF network element, and a success response or notification generated by the NWDAF network element.

[0060] In some embodiments of this disclosure, the request or subscription received by the NWDAF network element is the request or subscription issued by the NWDAF service consumer device to the NWDAF network element.

[0061] In some embodiments of this disclosure, the NWDAF service consumer device can be NFs, Afs, or OAM.

[0062] In some embodiments of this disclosure, the error response / notification generated by the NWDAF network element refers to the response / notification sent to the service consumer when the NWDAF network element is unable to fulfill the service request of the NWDAF consumer device, such as a timeout error response / notification.

[0063] In some embodiments of this disclosure, the successful response / notification generated by the NWDAF network element refers to the response / notification sent by the NWDAF network element to the service consumer when it is able to satisfy the service request of the NWDAF service consumer device.

[0064] Step 12: Based on the original measurement values, generate performance measurement values ​​corresponding to at least one NWDAF network element, wherein the performance measurement values ​​are used for statistical analysis related to the performance of the NWDAF network element, to evaluate and optimize the service performance of the NWDAF network element.

[0065] In some embodiments of this disclosure, the performance measurement value may include at least one of the following: the number of requests or subscriptions received by the NWDAF network element, the number of responses or notifications generated by the NWDAF network element, the number of error responses or notifications generated by the NWDAF network element, and the number of successful responses or notifications generated by the NWDAF network element.

[0066] In some embodiments of this disclosure, the number of requests or subscriptions received by the NWDAF network element includes the value of a first cumulative counter, wherein the first cumulative counter is a cumulative counter related to the number of requests or subscriptions received by the NWDAF.

[0067] In some embodiments of this disclosure, step 12, which involves generating a performance measurement value corresponding to at least one NWDAF network element based on the original measurement value, may include incrementing the value of a first cumulative counter by 1 when the NWDAF network element receives a request or subscription from an NWDAF service consumer device.

[0068] In some other embodiments of this disclosure, the number of responses or notifications generated by the NWDAF network element includes the value of a second cumulative counter, wherein the second cumulative counter is a cumulative counter related to the number of responses or notifications generated by the NWDAF.

[0069] In some embodiments of this disclosure, step 12, which involves generating a performance measurement value corresponding to at least one NWDAF network element based on the original measurement value, may include incrementing the value of a second cumulative counter by 1 when the NWDAF network element generates a response or notification.

[0070] In some other embodiments of this disclosure, the number of error responses or notifications generated by the NWDAF network element includes the value of a third cumulative counter, wherein the third cumulative counter is a cumulative counter related to the number of error responses or notifications generated by the NWDAF.

[0071] In some embodiments of this disclosure, step 12, which involves generating a performance measurement value corresponding to at least one NWDAF network element based on the original measurement value, may include incrementing the value of a third cumulative counter by 1 in the event that the NWDAF network element generates an error response or notification.

[0072] In some other embodiments of this disclosure, the number of successful responses or notifications generated by the NWDAF network element includes the value of a fourth cumulative counter, wherein the fourth cumulative counter is a cumulative counter related to the number of successful responses or notifications generated by the NWDAF.

[0073] In some embodiments of this disclosure, step 12, which involves generating a performance measurement value corresponding to at least one NWDAF network element based on the original measurement value, may include incrementing the value of a fourth cumulative counter by 1 if the NWDAF network element generates a successful response or notification.

[0074] In some embodiments of this disclosure, when the performance measurement is associated with specific information, the performance measurement value includes the value of a sub-cumulative counter associated with the specific information, wherein the specific information includes at least one of single network slice selection assistance information, analysis type, request type, subscription type, and analysis identifier.

[0075] In some embodiments of this disclosure, if the above performance measurement is related to S-NSSAI (Single Network Slice Selection Assistance Information), the above performance measurement may include the value of a sub-cumulative counter associated with S-NSSAI.

[0076] In some embodiments of this disclosure, if the performance measurement is for an Analytics ID, the performance measurement may include a sub-cumulative counter associated with the Analytics ID.

[0077] In some embodiments of this disclosure, when the specific information is an analysis type, a request type, or a subscription type, the value of the sub-cumulative counter associated with each performance measurement value includes the value of the sub-cumulative counter associated with an analysis service request, the value of the sub-cumulative counter associated with a data management service request, and the value of the sub-cumulative counter associated with a machine learning model service request.

[0078] In some embodiments of this disclosure, the NWDAF network element includes at least one of an analytical logic function (AnLF) and a model training logical function (MTLF), wherein NWDAF(AnLF) means that it only includes AnLF, NWDAF(MTLF) means that it only includes MTLF, and NWDAF means that it includes both AnLF and MTLF.

[0079] In some embodiments of this disclosure, the NWDAF network element can support at least one of the following services: analysis service, data management service, and machine learning model service. Different services correspond to different analysis types or request / subscription types.

[0080] In some embodiments of this disclosure, if the performance measurement is for an analytics type or a request / subscription type, the performance measurement may include the value of a sub-cumulative counter associated with each analytics type or request / subscription type, as follows:

[0081] 1. The value of the sub-cumulative counter related to the number of requests / subscriptions received by each NWDAF network element may include the value of the sub-cumulative counter associated with analysis service requests, the value of the sub-cumulative counter associated with data management service requests, and the value of the sub-cumulative counter associated with machine learning model service requests.

[0082] 2. The value of the sub-cumulative counter related to the number of responses / notifications generated by each NWDAF network element may include the value of the sub-cumulative counter associated with analysis service requests, the value of the sub-cumulative counter associated with data management service requests, and the value of the sub-cumulative counter associated with service requests from machine learning models.

[0083] 3. The value of the sub-cumulative counter related to the number of error responses / notifications generated by each NWDAF network element may include the value of the sub-cumulative counter associated with analysis service requests, the value of the sub-cumulative counter associated with data management service requests, and the value of the sub-cumulative counter associated with service requests from machine learning models.

[0084] 4. The value of the sub-cumulative counter related to the number of successful responses / notifications generated by each NWDAF network element may include the value of the sub-cumulative counter associated with the analysis service request, the value of the sub-cumulative counter associated with the data management service request, and the value of the sub-cumulative counter associated with the service request from the machine learning model.

[0085] Step 13: Report the performance measurements to external parties.

[0086] In some embodiments of this disclosure, step 13 may include: reporting the performance measurement value to the performance measurement service consumer device so that the performance measurement service consumer device can perform statistical analysis related to the NWDAF network element performance based on the performance measurement value, and evaluate and optimize the service performance of the NWDAF network element.

[0087] In some embodiments of this disclosure, the step of evaluating the service performance of an NWDAF network element may include: determining whether the difference between a performance measurement value (e.g., the number of requests or subscriptions received by the NWDAF network element) and a predetermined performance threshold (the maximum number of requests or subscriptions that the NWDAF network element can receive) is greater than a predetermined value; if the difference between the performance measurement value and the predetermined performance threshold is greater than the predetermined value, then the service performance of the NWDAF network element is determined to be excellent; if the difference between the performance measurement value and the predetermined performance threshold is not greater than the predetermined value, then the service performance of the NWDAF network element is determined to be poor.

[0088] The method for measuring the performance of network element services for network data analysis functions provided in the above embodiments of this disclosure is a method for measuring the performance of NWDAF network element interaction.

[0089] In order to detect and prevent technical problems in related technologies, the above embodiments of this disclosure can realize performance measurement values ​​(data). The NWDAF performance measurement values ​​of the above embodiments of this disclosure can be used for statistical analysis related to NWDAF performance, thereby evaluating and optimizing the service performance of NWDAF, detecting potential NWDAF service performance problems in advance, and taking appropriate actions (such as resource allocation, task scheduling, etc.) to prevent or mitigate these problems.

[0090] The purpose of the above embodiments of this disclosure is to propose a method for generating NWDAF interaction performance measurements, monitoring and processing the raw NWDAF measurements, and generating NWDAF performance measurements based on the raw measurements. These performance measurements can be used for statistical analysis related to NWDAF performance, thereby evaluating and optimizing the service performance of NWDAF.

[0091] In some embodiments of this disclosure, the method for measuring the performance of network element services for network data analysis functions may further include: a method for obtaining performance indicators of network element services for network data analysis functions.

[0092] In some embodiments of this disclosure, the method for obtaining network data analysis function (NWDAF) network element service performance indicators may include: obtaining performance measurement values ​​of multiple NWDAF network elements in a network or network slice; generating key performance indicators for the corresponding multiple NWDAF network elements based on the performance measurement values ​​of the multiple NWDAF network elements, wherein the key performance indicators are used to evaluate the overall service status of the NWDAF network elements in the network or network slice and provide the necessary information for resource optimization of the corresponding network or network slice; and reporting the key performance indicators externally.

[0093] In some embodiments of this disclosure, reporting the key performance indicators externally includes reporting the key performance indicators to a performance indicator consumer device.

[0094] In some embodiments of this disclosure, the step of obtaining performance measurement values ​​of multiple Network Data Analysis Function (NWDAF) network elements in a network or network slice may include: obtaining performance measurement values ​​of multiple Network Data Analysis Function (NWDAF) network elements in a network or network slice within a predetermined time interval.

[0095] In some embodiments of this disclosure, the performance measurement values ​​include at least one of the following: the maximum or average number of service requests or subscriptions received by each NWDAF network element associated with the corresponding network or network slice within a predetermined time interval; the maximum or average number of responses or notifications generated by each NWDAF network element; the maximum or average number of error responses or notifications generated by each NWDAF network element; and the maximum or average number of success responses or notifications generated by each NWDAF network element.

[0096] In some embodiments of this disclosure, the key performance indicators include at least one of the following: the maximum or average number of service requests or subscriptions received by multiple NWDAF network elements in the corresponding network or network slice within a predetermined time interval; the maximum or average number of responses or notifications generated by multiple NWDAF network elements; the maximum or average number of error responses or notifications generated by multiple NWDAF network elements; the maximum or average number of successful responses or notifications generated by multiple NWDAF network elements; the error response or notification ratio generated by multiple NWDAF network elements; and the successful response or notification ratio generated by multiple NWDAF network elements.

[0097] In some embodiments of this disclosure, the step of generating key performance indicators for multiple NWDAF network elements in a corresponding network or network slice based on the performance measurement values ​​of multiple NWDAF network elements may include: determining the maximum number of service requests or subscriptions received by multiple NWDAF network elements in a network or network slice based on the sum of the maximum number of service requests or subscriptions received by each NWDAF network element related to the corresponding network or network slice.

[0098] In some embodiments of this disclosure, the step of generating key performance indicators for multiple NWDAF network elements in a corresponding network or network slice based on performance measurement values ​​of multiple NWDAF network elements may include: determining the average number of service requests or subscriptions received by multiple NWDAF network elements in a network or network slice based on the sum of the average number of service requests or subscriptions received by each NWDAF network element related to the corresponding network or network slice.

[0099] In some embodiments of this disclosure, the step of generating key performance indicators for multiple NWDAF network elements in a corresponding network or network slice based on performance measurement values ​​of multiple NWDAF network elements may include: determining the maximum number of responses or notifications generated by multiple NWDAF network elements in a network or network slice based on the sum of the maximum number of responses or notifications generated by each NWDAF network element related to the corresponding network or network slice.

[0100] In some embodiments of this disclosure, the step of generating key performance indicators for multiple NWDAF network elements in a corresponding network or network slice based on performance measurement values ​​of multiple NWDAF network elements may include: determining the average number of responses or notifications generated by multiple NWDAF network elements in a network or network slice based on the sum of the average number of responses or notifications generated by each NWDAF network element related to the corresponding network or network slice.

[0101] In some embodiments of this disclosure, the step of generating key performance indicators for multiple NWDAF network elements in a corresponding network or network slice based on performance measurement values ​​of multiple NWDAF network elements may include: determining the maximum number of error responses or notifications generated by multiple NWDAF network elements in a network or network slice based on the sum of the maximum number of error responses or notifications generated by each NWDAF network element related to the corresponding network or network slice.

[0102] In some embodiments of this disclosure, the step of generating key performance indicators for multiple NWDAF network elements in a corresponding network or network slice based on performance measurement values ​​of multiple NWDAF network elements may include: determining the average number of error responses or notifications generated by multiple NWDAF network elements in a network or network slice based on the sum of the average number of error responses or notifications generated by each NWDAF network element related to the corresponding network or network slice.

[0103] In some embodiments of this disclosure, the step of generating key performance indicators for multiple NWDAF network elements in a corresponding network or network slice based on the performance measurement values ​​of multiple NWDAF network elements may include: determining the maximum number of successful responses or notifications generated by multiple NWDAF network elements in a network or network slice based on the sum of the maximum number of successful responses or notifications generated by each NWDAF network element related to the corresponding network or network slice.

[0104] In some embodiments of this disclosure, the step of generating key performance indicators for multiple NWDAF network elements in a corresponding network or network slice based on performance measurement values ​​of multiple NWDAF network elements may include: determining the average number of successful responses or notifications generated by multiple NWDAF network elements in a network or network slice based on the sum of the average number of successful responses or notifications generated by each NWDAF network element related to the corresponding network or network slice.

[0105] In some embodiments of this disclosure, the step of generating key performance indicators for multiple NWDAF network elements in a network or network slice based on performance measurements of multiple NWDAF network elements may include: determining the ratio of successful responses or notifications generated by multiple NWDAF network elements in a network or network slice to the average number of responses or notifications generated by multiple NWDAF network elements in a network or network slice.

[0106] In some embodiments of this disclosure, the step of generating key performance indicators for multiple NWDAF network elements in a network or network slice based on performance measurements of multiple NWDAF network elements may include: determining the ratio of error responses or notifications generated by multiple NWDAF network elements in a network or network slice based on the ratio of the average number of error responses or notifications generated by multiple NWDAF network elements in a network or network slice to the average number of responses or notifications generated by multiple NWDAF network elements in a network or network slice.

[0107] The acquisition of key performance indicators for NWDAFs in the above embodiments of this disclosure based on multi-NWDAF deployment scenarios is an important research topic. These key performance indicators can be used to reflect and evaluate the overall service status of multiple NWDAFs in a network / network slice, and provide direction for optimizing the service performance of multiple NWDAFs.

[0108] The purpose of the above embodiments of this disclosure is to propose a method for obtaining key performance indicators for NWDAF interaction, which monitors and processes performance measurement values ​​of NWDAF interaction within a predefined interval, calculates the performance measurement values, and generates key performance indicators for multiple NWDAFs in a corresponding network / network slice. These key performance indicators are used to reflect and evaluate the overall service status (e.g., bearer status) of multiple NWDAFs in a corresponding network / network slice, and provide necessary information for network / network slice resource optimization.

[0109] Figure 2This is a schematic diagram of some other embodiments of the method disclosed herein for measuring the performance of network element services for network data analysis functions. Preferably, this embodiment can be executed by a performance measurement service consumer device or a system disclosed herein for measuring the performance of network element services for network data analysis functions. The method may include at least one step in steps 21-22, wherein:

[0110] Step 21: Receive the reported performance measurement values, wherein the performance measurement values ​​are performance measurement values ​​generated by the performance measurement service producer device based on the original measurement values ​​of the network data analysis function NWDAF network element, corresponding to at least one NWDAF network element. The performance measurement values ​​are used for statistical analysis related to the performance of the NWDAF network element, to evaluate and optimize the service performance of the NWDAF network element.

[0111] In some embodiments of this disclosure, step 21 may include: receiving performance measurement values ​​reported by the performance measurement service producer device.

[0112] Step 22: Perform statistical analysis related to the performance of NWDAF network elements based on the performance measurement values, and evaluate and optimize the service performance of NWDAF network elements.

[0113] In some embodiments of this disclosure, step 22, the step of evaluating the service performance of the NWDAF network element, may include: determining whether the difference between a performance measurement value (e.g., the number of requests or subscriptions received by the NWDAF network element) and a predetermined performance threshold (the maximum number of requests or subscriptions that the NWDAF network element can receive) is greater than a predetermined value; if the difference between the performance measurement value and the predetermined performance threshold is greater than the predetermined value, then the service performance of the NWDAF network element is determined to be excellent; if the difference between the performance measurement value and the predetermined performance threshold is not greater than the predetermined value, then the service performance of the NWDAF network element is determined to be poor.

[0114] The embodiments disclosed above can monitor and process the raw measurement values ​​of NWDAF, and generate one or more performance measurement values ​​of NWDAF based on the raw measurement values; the generated performance measurement values ​​include the number of requests / subscriptions received by NWDAF, the number of responses / notifications generated by NWDAF, the number of error responses / notifications generated by NWDAF, the number of success responses / notifications generated by NWDAF, and other performance measurement values.

[0115] The performance measurements generated by the above embodiments of this disclosure can be used for statistical analysis related to NWDAF performance, thereby evaluating and optimizing the service performance of NWDAF, detecting potential NWDAF service performance problems in advance, and taking appropriate actions (such as resource allocation, task scheduling, etc.) to prevent or mitigate these performance problems.

[0116] Figure 3 These are schematic diagrams illustrating some embodiments of the performance measurement service producer apparatus of this disclosure. (See diagram for example.) Figure 3 As shown, the performance measurement service producer apparatus of this disclosure may include a raw measurement value acquisition module 31, a performance measurement value generation module 32, and a performance measurement value reporting module 33, wherein:

[0117] The raw measurement value acquisition module 31 is configured to acquire the raw measurement values ​​of the network data analysis function NWDAF network element.

[0118] In some embodiments of this disclosure, the original measurement value may include at least one of the following: a request or subscription received by the NWDAF network element, a response or notification generated by the NWDAF network element, an error response or notification generated by the NWDAF network element, and a success response or notification generated by the NWDAF network element.

[0119] The performance measurement value generation module 32 is configured to generate performance measurement values ​​corresponding to at least one NWDAF network element based on the original measurement values, wherein the performance measurement values ​​are used for statistical analysis related to the performance of the NWDAF network element to evaluate and optimize the service performance of the NWDAF network element.

[0120] In some embodiments of this disclosure, the performance measurement may include at least one of the following: the number of requests or subscriptions received by the NWDAF network element, the number of responses or notifications generated by the NWDAF network element, the number of error responses or notifications generated by the NWDAF network element, and the number of successful responses or notifications generated by the NWDAF network element.

[0121] In some embodiments of this disclosure, the number of requests or subscriptions received by the NWDAF network element includes the value of a first cumulative counter, wherein the first cumulative counter is a cumulative counter related to the number of requests or subscriptions received by the NWDAF.

[0122] In some embodiments of this disclosure, the performance measurement value generation module 32 may be configured to increment the value of the first cumulative counter by 1 when the NWDAF network element receives a request or subscription from an NWDAF service consumer device.

[0123] In some other embodiments of this disclosure, the number of responses or notifications generated by the NWDAF network element includes the value of a second cumulative counter, wherein the second cumulative counter is a cumulative counter related to the number of responses or notifications generated by the NWDAF.

[0124] In some embodiments of this disclosure, the performance measurement value generation module 32 may be configured to increment the value of the second cumulative counter by 1 when the NWDAF network element generates a response or notification.

[0125] In some other embodiments of this disclosure, the number of error responses or notifications generated by the NWDAF network element includes the value of a third cumulative counter, wherein the third cumulative counter is a cumulative counter related to the number of error responses or notifications generated by the NWDAF.

[0126] In some embodiments of this disclosure, the performance measurement value generation module 32 may be configured to increment the value of the third cumulative counter by 1 when the NWDAF network element generates an error response or notification.

[0127] In some embodiments of this disclosure, the number of successful responses or notifications generated by the NWDAF network element includes the value of a fourth cumulative counter, wherein the fourth cumulative counter is a cumulative counter related to the number of successful responses or notifications generated by the NWDAF.

[0128] In some embodiments of this disclosure, the performance measurement value generation module 32 may be configured to increment the value of the fourth cumulative counter by 1 when the NWDAF network element generates a successful response or notification.

[0129] In some embodiments of this disclosure, when the performance measurement is associated with specific information, the performance measurement value includes the value of a sub-cumulative counter associated with the specific information, wherein the specific information includes at least one of single network slice selection assistance information, analysis type, request type, subscription type, and analysis identifier.

[0130] In some embodiments of this disclosure, when the specific information is an analysis type, a request type, or a subscription type, the value of the sub-cumulative counter associated with each performance measurement value includes the value of the sub-cumulative counter associated with an analysis service request, the value of the sub-cumulative counter associated with a data management service request, and the value of the sub-cumulative counter associated with a machine learning model service request.

[0131] The performance measurement reporting module 33 is configured to report the performance measurement values ​​to external parties.

[0132] In some embodiments of this disclosure, the performance measurement value reporting module 33 can be configured to report the performance measurement value to the performance measurement service consumer device, so that the performance measurement service consumer device can perform statistical analysis related to the performance of the NWDAF network element based on the performance measurement value, and evaluate and optimize the service performance of the NWDAF network element.

[0133] In some embodiments of this disclosure, the performance measurement service producer apparatus can be configured to perform implementations as described in any of the embodiments above (e.g., Figure 1 The operation of the method described in the embodiment)

[0134] The performance measurement service producer apparatus provided in the above embodiments of this disclosure can generate NWDAF performance measurement values ​​based on the original NWDAF measurement values, including monitoring and processing the original NWDAF measurement values, and generating NWDAF interaction performance measurement values ​​based on the original measurement values.

[0135] The performance measurements generated by the above embodiments of this disclosure include the number of requests / subscriptions received by NWDAF, the number of responses / notifications generated by NWDAF, the number of error responses / notifications generated by NWDAF, and the number of success responses / notifications generated by NWDAF, which can be used to evaluate and optimize the service performance of NWDAF.

[0136] The embodiments described above can generate NWDAF interaction performance measurements based on the raw NWDAF measurements.

[0137] The NWDAF performance measurement values ​​generated by the above embodiments of this disclosure can be used for statistical analysis related to NWDAF performance, thereby evaluating and optimizing NWDAF service performance, detecting potential NWDAF service performance problems in advance, and taking appropriate actions to prevent or mitigate these performance problems.

[0138] Figure 4 These are schematic diagrams illustrating some embodiments of consumer devices for performance measurement services according to this disclosure. Figure 4 As shown, the performance measurement service consumer device of this disclosure may include a performance measurement value receiving module 41 and a performance analysis module 42, wherein:

[0139] The performance measurement value receiving module 41 is configured to receive reported performance measurement values, wherein the performance measurement values ​​are generated by the performance measurement service producer device based on the original measurement values ​​of the network data analysis function NWDAF network element, corresponding to at least one NWDAF network element. The performance measurement values ​​are used for statistical analysis related to the performance of the NWDAF network element, to evaluate and optimize the service performance of the NWDAF network element.

[0140] In some embodiments of this disclosure, the performance measurement value receiving module 41 may be configured to receive performance measurement values ​​reported by the performance measurement service producer device.

[0141] The performance analysis module 42 is configured to perform statistical analysis related to the performance of NWDAF network elements based on the performance measurement values, and to evaluate and optimize the service performance of NWDAF network elements.

[0142] In some embodiments of this disclosure, the performance measurement service consumer device can be configured to perform implementations as described in any of the embodiments above (e.g., Figure 2 The operation of the method described in the embodiment)

[0143] There is no method for measuring performance values ​​related to NWDAF interaction in the relevant technologies.

[0144] In order to perform statistical analysis on the service performance of NWDAF network elements, the embodiments of this disclosure introduce a performance measurement scheme for NWDAF network elements. Raw measurements of NWDAF are collected to generate NWDAF network element performance measurement values. These performance measurement values ​​can be used for statistical analysis related to NWDAF network element performance, thereby evaluating and optimizing the service performance of NWDAF network elements.

[0145] The embodiments of this disclosure can generate NWDAF network element performance measurement values ​​based on the original NWDAF measurement values. This mainly includes monitoring and processing the original NWDAF measurements, and generating NWDAF performance measurement values ​​based on the original measurements. The performance measurement values ​​generated by the embodiments of this disclosure include the number of requests / subscriptions received by NWDAF, the number of responses / notifications generated by NWDAF, the number of error responses / notifications generated by NWDAF, and the number of success responses / notifications generated by NWDAF.

[0146] Figure 5 These are schematic diagrams illustrating some embodiments of a system disclosed herein for measuring the performance of network element services used in network data analysis. For example... Figure 5 As shown, the system disclosed herein for measuring the performance of network element services for network data analysis functions may include a performance measurement service producer device 51 and a performance measurement service consumer device 52, wherein:

[0147] The performance measurement service producer device 51 is configured to acquire raw measurement values ​​of at least one network data analysis function (NWDAF) network element 50; generate performance measurement values ​​corresponding to at least one NWDAF network element based on the raw measurement values, wherein the performance measurement values ​​are used for statistical analysis related to the performance of the NWDAF network element, to evaluate and optimize the service performance of the NWDAF network element; and report the performance measurement values ​​to the performance measurement service consumer device 52.

[0148] In some embodiments of this disclosure, the performance measurement service producer device 51 may be as described in any of the embodiments above (e.g., Figure 3 The performance measurement service producer device described in the embodiment)

[0149] In some embodiments of this disclosure, the performance measurement service producer device 51 may be located in other network elements besides the performance measurement service consumer device 52 and the NWDAF network element 50.

[0150] The performance measurement service consumer device 52 is configured to receive performance measurement values ​​reported by the performance measurement service producer device; perform statistical analysis related to the performance of NWDAF network elements based on the performance measurement values; and evaluate and optimize the service performance of NWDAF network elements.

[0151] In some embodiments of this disclosure, the performance measurement service consumer device 52 may be as described in any of the embodiments above (e.g., Figure 4 The performance measurement service consumer device described in the embodiment).

[0152] The embodiments described above can be applied to the 5G field.

[0153] Figure 6 This is a schematic diagram illustrating the structure of some embodiments of the computer device disclosed herein. For example... Figure 6 As shown, the computer device includes a memory 61 and a processor 62.

[0154] Memory 61 is used to store instructions, and processor 62 is coupled to memory 61. Processor 62 is configured to execute instructions stored in memory to implement the above embodiments (e.g., Figure 1 or Figure 2 The method involved in the embodiment).

[0155] In some embodiments of this disclosure, the processor 62 executes Figure 1 In the case of the embodiment method, the computer device of this disclosure can be implemented as a consumer device for performance measurement services of this disclosure.

[0156] In other embodiments of this disclosure, the processor 62 executes Figure 2 In the case of the embodiments, the computer device of this disclosure can be implemented as a producer device for performance measurement services of this disclosure.

[0157] like Figure 6 As shown, the computer device also includes a communication interface 63 for exchanging information with other devices. Additionally, the computer device includes a bus 64, through which the processor 62, communication interface 63, and memory 61 communicate with each other.

[0158] Memory 61 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. Memory 61 may also be a memory array. Memory 61 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0159] Furthermore, processor 62 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0160] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions that, when executed by a processor, implement any of the embodiments described above (e.g., Figure 1 or Figure 2 The method described in the example.

[0161] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] The performance measurement service producer device and performance measurement service consumer device described above can be implemented as a general-purpose processor, programmable logic controller (PLC), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component or any suitable combination thereof for performing the functions described in this application.

[0166] This concludes the detailed description of the present disclosure. To avoid obscuring the concept of the disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0167] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a non-transitory computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0168] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for measuring the service performance of network elements with network data analysis functions, characterized in that, include: Obtain the first measurement value of the Network Data Analysis Function (NWDAF) network element, wherein the first measurement value includes at least one of the following: a request or subscription received by the NWDAF network element, a response or notification generated by the NWDAF network element, an error response or notification generated by the NWDAF network element, and a success response or notification generated by the NWDAF network element. Based on the first measurement value, a performance measurement value corresponding to at least one NWDAF network element is generated, wherein the performance measurement value includes at least one of the following: the number of requests or subscriptions received by the NWDAF network element, the number of responses or notifications generated by the NWDAF network element, the number of error responses or notifications generated by the NWDAF network element, and the number of successful responses or notifications generated by the NWDAF network element. The performance measurements reported externally.

2. The method according to claim 1, characterized in that, The number of requests or subscriptions received by the NWDAF network element includes the value of a first cumulative counter, wherein the first cumulative counter is a cumulative counter related to the number of requests or subscriptions received by the NWDAF. The step of generating performance measurement values ​​corresponding to at least one NWDAF network element based on the first measurement value includes: When an NWDAF network element receives a request or subscription from an NWDAF service consumer device, it increments the value of the first cumulative counter by 1.

3. The method according to claim 1 or 2, characterized in that, The number of responses or notifications generated by the NWDAF network element includes the value of a second cumulative counter, wherein the second cumulative counter is a cumulative counter related to the number of responses or notifications generated by the NWDAF; The step of generating performance measurement values ​​corresponding to at least one NWDAF network element based on the first measurement value includes: When an NWDAF network element generates a response or notification, the value of the second cumulative counter is incremented by 1.

4. The method according to claim 1 or 2, characterized in that, The number of error responses or notifications generated by the NWDAF network element includes the value of a third cumulative counter, wherein the third cumulative counter is a cumulative counter related to the number of error responses or notifications generated by the NWDAF; The step of generating performance measurement values ​​corresponding to at least one NWDAF network element based on the first measurement value includes: When an NWDAF network element generates an error response or notification, the value of the third cumulative counter is incremented by 1.

5. The method according to claim 1 or 2, characterized in that, The number of successful responses or notifications generated by the NWDAF network element includes the value of a fourth cumulative counter, wherein the fourth cumulative counter is a cumulative counter related to the number of successful responses or notifications generated by NWDAF; The step of generating performance measurement values ​​corresponding to at least one NWDAF network element based on the first measurement value includes: When the NWDAF element generates a successful response or notification, the value of the fourth cumulative counter is incremented by 1.

6. The method according to claim 1 or 2, characterized in that, In cases where performance measurements are associated with specific information, the performance measurement value includes the value of a sub-cumulative counter associated with the specific information, wherein the specific information includes at least one of individual network slice selection assistance information, analysis type, request type, subscription type, and analysis identifier.

7. The method according to claim 6, characterized in that, In cases where the specific information is of the analysis type, request type, or subscription type... The values ​​of the sub-cumulative counters associated with each performance metric include the values ​​of the sub-cumulative counters associated with analytics service requests, the values ​​of the sub-cumulative counters associated with data management service requests, and the values ​​of the sub-cumulative counters associated with machine learning model service requests.

8. A method for measuring the service performance of network elements with network data analysis functions, characterized in that, include: The system receives reported performance measurement values, wherein the performance measurement values ​​are performance measurement values ​​generated by the performance measurement service producer device based on the first measurement value of the network data analysis function NWDAF network element, corresponding to at least one NWDAF network element. The first measurement value includes at least one of the following: requests or subscriptions received by the NWDAF network element, responses or notifications generated by the NWDAF network element, error responses or notifications generated by the NWDAF network element, and success responses or notifications generated by the NWDAF network element. The performance measurement values ​​include at least one of the following: the number of requests or subscriptions received by the NWDAF network element, the number of responses or notifications generated by the NWDAF network element, the number of error responses or notifications generated by the NWDAF network element, and the number of success responses or notifications generated by the NWDAF network element.

9. The method according to claim 8, characterized in that, Also includes: Based on the performance measurements, perform statistical analysis related to the performance of NWDAF network elements to evaluate and optimize the service performance of NWDAF network elements.

10. A performance measurement service producer device, characterized in that, include: The first measurement value acquisition module is configured to acquire the first measurement value of the Network Data Analysis Function (NWDAF) network element, wherein the first measurement value includes at least one of the following: a request or subscription received by the NWDAF network element, a response or notification generated by the NWDAF network element, an error response or notification generated by the NWDAF network element, and a success response or notification generated by the NWDAF network element. The performance measurement value generation module is configured to generate performance measurement values ​​corresponding to at least one NWDAF network element based on the first measurement value, wherein the performance measurement value includes at least one of the following: the number of requests or subscriptions received by the NWDAF network element, the number of responses or notifications generated by the NWDAF network element, the number of error responses or notifications generated by the NWDAF network element, and the number of success responses or notifications generated by the NWDAF network element. The performance measurement reporting module is configured to report the performance measurement values ​​to external parties.

11. The performance measurement service producer apparatus according to claim 10, characterized in that, The performance measurement service producer device is configured to perform the method as described in any one of claims 2-7.

12. A performance measurement service consumer device, characterized in that, include: A performance measurement value receiving module is configured to receive reported performance measurement values, wherein the performance measurement values ​​are generated by the performance measurement service producer device based on a first measurement value of the network data analysis function NWDAF network element, corresponding to at least one NWDAF network element. The first measurement value includes at least one of the following: requests or subscriptions received by the NWDAF network element, responses or notifications generated by the NWDAF network element, error responses or notifications generated by the NWDAF network element, and success responses or notifications generated by the NWDAF network element. The performance measurement values ​​include at least one of the following: the number of requests or subscriptions received by the NWDAF network element, the number of responses or notifications generated by the NWDAF network element, the number of error responses or notifications generated by the NWDAF network element, and the number of success responses or notifications generated by the NWDAF network element.

13. The performance measurement service consumer device according to claim 12, characterized in that, Also includes: The performance analysis module is configured to perform statistical analysis related to the performance of NWDAF network elements based on the performance measurements, and to evaluate and optimize the service performance of NWDAF network elements.

14. A system for measuring the service performance of network elements with network data analysis functions, characterized in that, This includes the performance measurement service producer device as described in claim 10 or 11 and the performance measurement service consumer device as described in claim 12 or 13.

15. A computer device, characterized in that, include: The memory is configured to store instructions; A processor is configured to execute the instructions, causing the computer device to perform the method as described in any one of claims 1-9.

16. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-9.

Images