Data collection performance evaluation method, apparatus and system, storage medium

Abstract

The present disclosure relates to a data collection performance evaluation method, device and system, and a storage medium. The data collection performance evaluation method comprises: measuring measurement data of interaction when a functional network element performs data collection, wherein the functional network element is a network element for implementing a network data analysis function; determining a quality value of collected data according to the measurement data; and evaluating data collection performance of the functional network element according to the quality value of the collected data. The present disclosure can measure and evaluate data collection performance of a network element for implementing a network data analysis function, and can quantify and evaluate quality of data collected by the network element for implementing the network data analysis function.

Description

Technical Field

[0001] This disclosure relates to the field of wireless communication, and in particular to a method, apparatus and system for evaluating data collection performance, and a storage medium. Background Technology

[0002] NWDAF (Network Data Analytics Function) is a network element in 5GC (5G core network) that supports the collection of data from NFs (Network Functions), AFs (Application Functions), and OAM (Operations, Administration, Maintenance) for analysis. Summary of the Invention

[0003] The inventors discovered through research that, from the operator's perspective, the NWDAF network element, as a network function that provides network data analysis capabilities, has its analysis performance directly affected by the performance of its data collection. Therefore, how to monitor the performance of the NWDAF network element in data collection is a key issue.

[0004] In view of at least one of the above technical problems, this disclosure provides a data collection performance evaluation method, apparatus and system, and storage medium, which can measure and evaluate the data collection performance of network elements used to implement network data analysis functions, and can quantify and evaluate the quality of data collected by network elements used to implement network data analysis functions.

[0005] According to one aspect of this disclosure, a data collection performance evaluation method is provided, comprising:

[0006] The measurement function network element performs data collection and interactive measurement data, wherein the function network element is a network element used to implement network data analysis functions;

[0007] The quality value of the collected data is determined based on the measurement data;

[0008] The data collection performance of the functional network element is evaluated based on the quality value of the collected data.

[0009] In some embodiments of this disclosure, the measurement data related to the interaction when the functional network element performs data collection during the measurement network data analysis function includes:

[0010] Based on the target service type, target data source type, and target data source, measure the interaction aspects of the functional network element when it collects data from the target data source to perform the target type service. The target service type includes at least one of analysis service, model training service, and data management service.

[0011] In some embodiments of this disclosure, determining the quality value of the collected data based on the measurement data includes:

[0012] Based on the measurement data, the missing data information of the data collected by the functional network element from the target data source for performing the target type service is obtained, wherein the missing data information is used to quantify the quality of the data collected by the functional network element.

[0013] In some embodiments of this disclosure, the measurement data includes the number of requests issued by the functional network element and the number of responses received by the functional network element, wherein the request method includes at least one of request and subscription, and the response method includes at least one of response and notification.

[0014] In some embodiments of this disclosure, the measurement data related to the interaction when the functional network element performs data collection during the measurement network data analysis function includes:

[0015] The number of data collection requests sent by the functional network element to the target data source is measured. When the functional network element triggers a data collection-related request or subscription, the first cumulative counter is incremented by 1, wherein the first cumulative counter is a cumulative counter related to the number of data collection requests or subscriptions triggered by the functional network element.

[0016] The number of data collection responses received by the functional network element from the target data source is measured. When the functional network element receives a data collection-related response or notification, the second cumulative counter is incremented by 1, wherein the second cumulative counter is a cumulative counter related to the number of data collection responses or notifications received by the functional network element.

[0017] In some embodiments of this disclosure, measuring the number of data collection requests sent by the functional network element to the target data source includes:

[0018] When the functional network element triggers a data collection-related request or subscription to perform an analysis task, the first cumulative counter includes a first sub-cumulative counter based on different analysis identifiers. The first sub-cumulative counter is a sub-cumulative counter associated with the analysis identifier and is used to measure the number of data collection requests triggered by the functional network element when performing the analysis indicated by the target analysis identifier.

[0019] In some embodiments of this disclosure, measuring the number of data collection requests sent by the functional network element to the target data source includes:

[0020] When the functional network element triggers a data collection-related request or subscription to perform a model training task, the first cumulative counter includes a second sub-cumulative counter based on different model identifiers or analysis identifiers. The second sub-cumulative counter is a sub-cumulative counter related to the model identifier or analysis identifier. The second sub-cumulative counter is used to measure the number of data collection requests triggered by the functional network element when training the model indicated by the target model identifier.

[0021] In some embodiments of this disclosure, measuring the number of data collection requests sent by the functional network element to the target data source includes:

[0022] When the functional network element triggers a data collection-related request or subscription to perform a data management task, the first cumulative counter includes a fifth sub-cumulative counter based on different data management identifiers. The fifth sub-cumulative counter is a sub-cumulative counter associated with the data management identifier and is used to measure the number of data collection requests triggered by the functional network element when performing the data management task indicated by the target data management identifier.

[0023] In some embodiments of this disclosure, measuring the number of data collection requests sent by the functional network element to the target data source includes:

[0024] Based on different data sources, at least one of the first request count, the second request count, and the third request count is measured, wherein the first request count is the number of data collection requests triggered by the functional network element for each target data source or each type of target data source when performing the analysis indicated by the target analysis identifier; the second request count is the number of data collection requests triggered by the functional network element for each target data source or each type of target data source when training the model indicated by the target model identifier; and the third request count is the number of data collection requests triggered by the functional network element for each target data source or each type of target data source when performing the data management task indicated by the target data management identifier.

[0025] In some embodiments of this disclosure, the measurement of the number of data collection responses received by the functional network element from the target data source includes:

[0026] When the functional network element collects data from the target data source for analysis, the second cumulative counter includes a third sub-cumulative counter based on different analysis identifiers. The third sub-cumulative counter is a sub-cumulative counter associated with the analysis identifier and is used to measure the number of data collection responses received by the functional network element when performing the analysis indicated by the target analysis identifier.

[0027] In some embodiments of this disclosure, the measurement of the number of data collection responses received by the functional network element from the target data source includes:

[0028] When the functional network element collects data from the target data source for model training, based on different model identifiers or analysis identifiers, the second cumulative counter includes a fourth sub-cumulative counter. The fourth sub-cumulative counter is a sub-cumulative counter associated with the model identifier or analysis identifier. The fourth sub-cumulative counter is used to measure the number of data collection responses received by the functional network element when training the model indicated by the target model identifier.

[0029] In some embodiments of this disclosure, the measurement of the number of data collection responses received by the functional network element from the target data source includes:

[0030] When the functional network element collects data from the target data source for data management, the second cumulative counter includes a sixth sub-cumulative counter based on different data management identifiers. The sixth sub-cumulative counter is a sub-cumulative counter associated with the data management identifier and is used to measure the number of data collection responses received by the functional network element when performing the data management task indicated by the target data management identifier.

[0031] In some embodiments of this disclosure, the measurement of the number of data collection responses received by the functional network element from the target data source includes:

[0032] Based on different data sources, at least one of the following is measured: the first response count, the second response count, and the third response count. The first response count is the number of data collection responses received by the functional network element from each target data source or each type of target data source when performing the analysis indicated by the target analysis identifier. The second response count is the number of data collection responses received by the functional network element from each target data source or each type of target data source when training the model indicated by the target model identifier. The third response count is the number of data collection responses received by the functional network element from each target data source or each type of target data source when performing the data management task indicated by the target data management identifier.

[0033] In some embodiments of this disclosure, when the functional network element triggers a data collection-related request or subscription to perform an analysis task, determining the quality value of the collected data based on the measurement data includes: determining a first quality value of the measurement data based on the number of times recorded by a first sub-cumulative counter and the number of times recorded by a third sub-cumulative counter, wherein the first quality value is used to indicate the missing data collected by the functional network element to perform the analysis indicated by the target analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element to perform the analysis.

[0034] In some embodiments of this disclosure, determining the first quality value of the measurement data based on the number of data collection requests triggered by the functional network element and the number of data collection responses received by the functional network element includes:

[0035] Determine the first difference, where the first difference is the difference between the number of times recorded by the first sub-cumulative counter and the number of times recorded by the third sub-cumulative counter;

[0036] The ratio of the first difference to the number of times recorded by the first sub-cumulative counter is used as the first quality value of the measurement data.

[0037] In some embodiments of this disclosure, when measuring the number of first requests and the number of first responses based on different data sources, determining the quality value of the collected data based on the measurement data includes: determining a second quality value of the measurement data based on the number of first requests and the number of first responses, wherein the second quality value is used to indicate the missing data collected by the functional network element from each target data source or each target data source in order to perform the analysis indicated by the target analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each target data source in order to perform the analysis.

[0038] In some embodiments of this disclosure, when the functional network element triggers a data collection service for the model indicated by the training target model identifier or analysis identifier, determining the quality value of the collected data based on the measurement data includes: determining a third quality value of the measurement data based on the number of times recorded by the second sub-cumulative counter and the number of times recorded by the fourth sub-cumulative counter, wherein the third quality value is used to indicate the missing data of the data collected by the functional network element for the model indicated by the training target model identifier or analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for training the model.

[0039] In some embodiments of this disclosure, when measuring the number of second requests and the number of second responses based on different data sources, determining the quality value of the collected data based on the measurement data includes: determining a fourth quality value of the measurement data based on the number of second requests and the number of second responses, wherein the fourth quality value is used to indicate the missing data collected by the functional network element from each target data source or each target data source for training the model indicated by the target model identifier or analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each target data source for training the model.

[0040] In some embodiments of this disclosure, when the functional network element triggers a data collection-related request or subscription to perform a data management task, determining the quality value of the collected data based on the measurement data includes: determining a fifth quality value of the measurement data based on the number of times recorded by the fifth sub-cumulative counter and the number of times recorded by the sixth sub-cumulative counter, wherein the fifth quality value is used to indicate the missing data situation of the data collected by the functional network element to perform the data management task indicated by the data management identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element to perform the data management task.

[0041] In some embodiments of this disclosure, when measuring the number of third requests and the number of third responses based on different data sources, determining the quality value of the collected data based on the measurement data includes: determining a sixth quality value of the measurement data based on the number of third requests and the number of third responses, wherein the sixth quality value is used to indicate the missing data collected by the functional network element from each target data source or each target data source in order to perform the data management task indicated by the data management identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each target data source in order to perform the data management task.

[0042] In some embodiments of this disclosure, the method further includes:

[0043] Receive subscription or request messages, wherein the subscription or request messages include performance evaluation instructions, the performance evaluation instructions include filtering information, and subscription or request behavior;

[0044] According to the performance evaluation instructions, the data collection performance of the functional network element is evaluated in accordance with the method described in any of the above embodiments;

[0045] Send a notification or response message, wherein the notification or response message includes filtering information and a response to a subscription or request operation.

[0046] In some embodiments of this disclosure, the filtering information includes at least one of an analysis identifier list, a model identifier list, and a data management identifier list, wherein the analysis identifier list is used to support measuring the quality of data collected by the functional network element for performing the analysis indicated by the target analysis identifier, the model identifier list is used to support measuring the quality of data collected by the functional network element for training the model indicated by the target model identifier, and the data management identifier list is used to support measuring the quality of data collected by the functional network element for performing the data management task indicated by the target data management identifier.

[0047] In some embodiments of this disclosure, the subscription or request behavior includes at least one of the following operations: measuring the quality of data collected by the functional network element for performing analysis; measuring the quality of data collected by the functional network element for training a model; measuring the quality of data collected by the functional network element from a target data source for performing analysis; measuring the quality of data collected by the functional network element from a target data source for training a model; and measuring the quality of data collected by the functional network element from a target data source for performing data management tasks.

[0048] In some embodiments of this disclosure, the response to a subscription or request operation includes at least one of the following response operations: providing the quality of data collected by the functional network element for performing analysis; providing the quality of data collected by the functional network element for training a model; providing the quality of data collected by the functional network element for performing data management tasks; providing the quality of data collected by the functional network element from a target data source for performing analysis; providing the quality of data collected by the functional network element from a target data source for training a model; and providing the quality of data collected by the functional network element from a target data source for performing data management tasks.

[0049] According to another aspect of this disclosure, a data collection performance evaluation apparatus is provided, comprising:

[0050] The data measurement module is configured to measure the interactive measurement data when the functional network element performs data collection, wherein the functional network element is a network element used to implement network data analysis functions;

[0051] A quality value determination module is configured to determine the quality value of the collected data based on the measurement data.

[0052] The performance evaluation module is configured to evaluate the data collection performance of the functional network element based on the quality value of the collected data.

[0053] In some embodiments of this disclosure, the data collection performance evaluation apparatus is configured to perform operations implementing the methods described in any of the above embodiments.

[0054] According to another aspect of this disclosure, a data collection performance evaluation apparatus is provided, comprising:

[0055] The memory is configured to store instructions;

[0056] The processor is configured to execute the instructions, causing the data collection performance evaluation device to perform operations implementing the method as described in any of the above embodiments.

[0057] According to another aspect of this disclosure, a data collection performance evaluation system is provided, including the data collection performance evaluation apparatus as described in any of the above embodiments.

[0058] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.

[0059] This disclosure enables the measurement and evaluation of the data collection performance of network elements used to implement network data analysis functions, and can quantify and evaluate the quality of data collected by network elements used to implement network data analysis functions. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 These are schematic diagrams illustrating some embodiments of the data collection performance evaluation method disclosed herein.

[0062] Figure 2 This is a schematic diagram illustrating how a network element used to implement network data analysis functions collects data from any 5GC NF in some embodiments of this disclosure.

[0063] Figure 3 This is a schematic diagram of a network element "subscription-notification" service used to implement network data analysis functions in some embodiments of this disclosure.

[0064] Figure 4 This is a schematic diagram of a network element "request-response" service used to implement network data analysis functions in some embodiments of this disclosure.

[0065] Figure 5 These are schematic diagrams of other embodiments of the data collection performance evaluation method disclosed herein.

[0066] Figure 6These are schematic diagrams of some embodiments of the data collection performance evaluation system disclosed herein.

[0067] Figure 7 These are schematic diagrams of some embodiments of the data collection performance evaluation apparatus disclosed herein.

[0068] Figure 8 The diagram shows the structure of some other embodiments of the data collection performance evaluation apparatus disclosed herein. Detailed Implementation

[0069] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0070] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0071] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0072] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0073] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0074] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0075] The inventors discovered through research that, in practice, due to data source or network transmission issues, some data may be missing (lost or not transmitted) when NWDAF network elements request data from other NFs, AFs, and OAMs. The more data is missing, the greater the impact on the analytical performance of the NWDAF network element. Before using this collected data for analytical tasks, the NWDAF network element needs to preprocess this data (data cleaning or data augmentation).

[0076] Therefore, the situation regarding missing data collected in this disclosure is an important indicator for evaluating the quality of data collected by NWDAF network elements. Based on this performance indicator, we can understand the performance of NWDAF network data collection and provide necessary reference for optimizing NWDAF services.

[0077] In view of at least one of the above technical problems, this disclosure provides a method, apparatus and system for measuring the service performance of network elements used to implement network data analysis functions. The disclosure will be described below through specific embodiments.

[0078] Figure 1 This is a schematic diagram of some embodiments of the data collection performance evaluation method of this disclosure. Preferably, this embodiment can be executed by the data collection performance evaluation system or the data collection performance evaluation device of this disclosure. The method includes at least one step from step 100 to step 400, wherein:

[0079] Step 100: Measure the interactive measurement data when the functional network element performs data collection, wherein the functional network element is a network element used to implement network data analysis functions.

[0080] In some embodiments of this disclosure, the functional network element may be an NWDAF network element.

[0081] In some embodiments of this disclosure, step 100 may include: measuring the interaction aspect measurement data corresponding to the functional network element collecting data from the target data source to perform the target type service, based on the target service type, the target data source type, and the target data source, wherein the target service type includes at least one of analysis service, model training service, and data management service.

[0082] In some embodiments of this disclosure, the target data source type includes NF (Network Function Element), AFs (Application Function Element), and OAM (Operation, Maintenance and Management Element), wherein the NF can be an NWDAF element.

[0083] In some embodiments of this disclosure, the measurement data includes the number of requests issued by the functional network element and the number of responses received by the functional network element, wherein the request method includes at least one of request and subscription, and the response method includes at least one of response and notification.

[0084] Figure 2 This diagram illustrates how a network element implementing network data analysis functions collects data from any 5GC NF in some embodiments of this disclosure. The network element implementing network data analysis functions can collect data from any 5GC NF using a "subscribe-notification" service or a "request-response" service. The following describes how... Figure 3 and Figure 4It should be noted that the network element used to implement network data analysis function can be an NWDAF network element.

[0085] Figure 3 This is a schematic diagram illustrating a network element "subscription-notification" service used to implement network data analysis functions in some embodiments of this disclosure. For example... Figure 3 As shown, NF_A is the service consumer (Consumer), and NF_B is the network element used to implement network data analysis functions (e.g., an NWDAF network element). NF_A collects data from NF_B (the service producer). Figure 3 In an embodiment, step 100 may include: measuring the number of subscriptions issued by the network element used to implement network data analysis functions and the number of notifications received by the network element used to implement network data analysis functions.

[0086] Figure 4 This is a schematic diagram illustrating the "request-response" service of a network element used to implement network data analysis functions in some embodiments of this disclosure. For example... Figure 4 As shown, NF_A is the service consumer, NF_B is the network element used to implement network data analysis functions, and NF_A collects data from NF_B (service producer). For Figure 4 In an embodiment, step 100 may include: measuring the number of requests sent by the network element used to implement the network data analysis function and the number of responses received by the network element used to implement the network data analysis function.

[0087] In some embodiments of this disclosure, step 100 may include at least one of steps 110 and 120, wherein:

[0088] Step 110: Measure the number of data collection requests sent by the functional network element to the target data source. If the functional network element triggers a data collection-related request or subscription, increment the first cumulative counter (CC) by 1. The first cumulative counter is a cumulative counter related to the number of data collection requests or subscriptions triggered by the functional network element. The value of the first cumulative counter is the measured value of the number of data collection requests / subscriptions triggered by the functional network element.

[0089] In some embodiments of this disclosure, step 110 may include at least one of steps 111 to 114, wherein:

[0090] Step 111: When the functional network element triggers a data collection-related request or subscription to perform an analysis task, the first cumulative counter includes a first sub-cumulative counter based on different analysis identifiers (IDs). The first sub-cumulative counter is a sub-cumulative counter associated with the analysis identifier. The first sub-cumulative counter is used to measure the number of data collection requests triggered by the functional network element when performing the analysis indicated by the target analysis identifier, i.e., the number of requests for analytics (per Analytics ID).

[0091] Step 112: When the functional network element triggers a data collection-related request or subscription to perform a model training task, the first cumulative counter includes a second sub-cumulative counter based on different model identifiers or analytics identifiers. The second sub-cumulative counter is a sub-cumulative counter related to the model identifier or analytics identifier. The second sub-cumulative counter is used to measure the number of data collection requests triggered by the functional network element when training the model indicated by the target model identifier, i.e., the number of requests for model training (per Analytics / Model ID).

[0092] Step 113: When the functional network element triggers a data collection-related request or subscription to perform a data management task, the first cumulative counter includes a fifth sub-cumulative counter based on different data management identifiers. The fifth sub-cumulative counter is a sub-cumulative counter related to the data management identifier and is used to measure the number of data collection requests triggered by the functional network element when performing the data management task indicated by the target data management identifier.

[0093] Step 114: Based on different data sources, measure at least one of the following: first request count, second request count, and third request count. The first request count is the number of data collection requests to each target data source triggered by the functional network element when performing the analysis indicated by the target analysis identifier, i.e., the number of requests to other dataSource_i for analytics (per Analytics ID), or the first request count is the number of data collection requests to each target data source triggered by the functional network element when performing the analysis indicated by the target analysis identifier. The second request count is the number of data collection requests to each target data source triggered by the functional network element when training the model indicated by the target model identifier, i.e., the number of requests to other dataSource_i for model training (per Analytics / Model ID), or the second request count is the number of data collection requests to each target data source triggered by the functional network element when training the model indicated by the target model identifier. The third request count is the number of data collection requests to each target data source or each type of target data source triggered when performing the data management task indicated by the target data management identifier.

[0094] Step 120: Measure the number of data collection responses received by the functional network element from the target data source. If the functional network element receives a data collection-related response or notification, increment the second cumulative counter by 1. The second cumulative counter is a cumulative counter related to the number of data collection responses or notifications received by the functional network element, and the value of the second cumulative counter is the measured value of the number of data collection responses / notifications received by the functional network element.

[0095] In some embodiments of this disclosure, step 120 may include at least one of steps 121 to 124, wherein:

[0096] Step 121: When the functional network element collects data from the target data source for analysis, the second cumulative counter includes a third sub-cumulative counter based on different analysis identifiers. The third sub-cumulative counter is a sub-cumulative counter associated with the analysis identifier. The third sub-cumulative counter is used to measure the number of data collection responses received by the functional network element when performing the analysis indicated by the target analysis identifier (the number of responses for Analyticsper Analytics ID).

[0097] Step 122: When the functional network element collects data from the target data source for model training, based on different model identifiers or analytics identifiers, the second cumulative counter includes a fourth sub-cumulative counter. The fourth sub-cumulative counter is a sub-cumulative counter related to the model identifier or analytics identifier. The fourth sub-cumulative counter is used to measure the number of data collection responses received by the functional network element when training the model indicated by the target model identifier, i.e., the number of responses for model training (per Analytics / Model ID).

[0098] Step 123: When the functional network element collects data from the target data source for data management, the second cumulative counter includes a sixth sub-cumulative counter based on different data management identifiers. The sixth sub-cumulative counter is a sub-cumulative counter associated with the data management identifier. The sixth sub-cumulative counter is used to measure the number of data collection responses received by the functional network element when performing the data management task indicated by the target data management identifier.

[0099] Step 124: Based on different data sources, measure at least one of the following: first response count, second response count, and third response count. The first response count is the number of data collection responses received by the functional network element from each target data source when performing the analysis indicated by the target analysis identifier, i.e., the number of responses from dataSource_i for analytics (per Analytics ID), or the first response count is the number of data collection responses received by the functional network element from each target data source when performing the analysis indicated by the target analysis identifier. The second response count is the number of data collection responses received by the functional network element from each target data source when training the model indicated by the target model identifier, i.e., the number of responses from other dataSource_i for model training (per Analytics / Model ID), or the second response count is the number of data collection responses received by the functional network element from each target data source when training the model indicated by the target model identifier. The third response count is the number of data collection responses received by the functional network element from each target data source or each target data source when performing the data management task indicated by the target data management identifier.

[0100] Step 200: Determine the quality value of the collected data based on the measurement data.

[0101] In some embodiments of this disclosure, step 200 may include: obtaining, based on the measurement data, the missing data information of the data collected by the functional network element from the target data source for performing the target type service, wherein the missing data information is used to quantify the quality of the data collected by the functional network element.

[0102] In some embodiments of this disclosure, step 200 may include at least one of steps 210 to 260, wherein:

[0103] Step 210: When the functional network element triggers a data collection-related request or subscription to perform an analysis task, a first quality value A of the measurement data is determined based on the number of times recorded by the first sub-cumulative counter (the number of times the functional network element triggers a data collection request) and the number of times recorded by the third sub-cumulative counter (the number of times the functional network element receives a data collection response). The first quality value A is used to indicate the missing data (data quality) collected by the functional network element to perform the analysis indicated by the target analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element to perform the analysis.

[0104] In some embodiments of this disclosure, step 210 may include: when the functional network element triggers a data collection service to perform the analysis indicated by the target analysis ID, obtaining the ratio of "the number of data collection requests triggered by the functional network element - the number of data collection responses received by the functional network element" to "the number of data collection requests triggered by the functional network element" as a first quality value A.

[0105] In some embodiments of this disclosure, step 210 may include at least one of steps 211 to 212, wherein:

[0106] Step 211: Determine the first difference temp, where the first difference is the difference between the number of times recorded by the first sub-cumulative counter and the number of times recorded by the third sub-cumulative counter.

[0107] In some embodiments of this disclosure, step 211 may include: determining a first difference temp according to formula (1).

[0108] temp=the number of requests for analytics(per Analytics ID)-thenumber of responses for Analytics(per Analytics ID) (1)

[0109] Step 212: The ratio of the first difference temp to the number of times recorded by the first sub-cumulative counter is taken as the first quality value A of the measurement data.

[0110] In some embodiments of this disclosure, step 212 may include: determining a first mass value A according to formula (2).

[0111]

[0112] Step 220: Given the number of first requests and the number of first responses measured based on different data sources, determine the second quality value A of the measured data according to the number of first requests (the number of data collection requests triggered by the functional network element for each target data source or each type of target data source) and the number of first responses (the number of data collection responses received by the functional network element from each target data source or each type of target data source). i Wherein, the second mass value A i Used to indicate the missing data (data quality) collected by the functional network element from each target data source or each target data source in order to perform the analysis indicated by the target analysis identifier, and used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each target data source in order to perform the analysis.

[0113] In some embodiments of this disclosure, step 220 may include: based on different data sources, if the functional network element collects data from multiple other data sources to perform the analysis indicated by the target analysis ID, obtaining the ratio of "the number of data collection requests triggered by the functional network element for each target data source or each type of target data source - the number of data collection responses received by the functional network element from each target data source or each type of target data source" to "the number of data collection requests triggered by the functional network element for each target data source or each type of target data source" (represented by A1, A2, A3... respectively).

[0114] In some embodiments of this disclosure, step 220 may include at least one of steps 221 to 222, wherein:

[0115] Step 221, determine the second difference temp i The second difference is the difference between the first number of requests (the number of data collection requests triggered by the functional network element when performing the analysis indicated by the target analysis identifier for each target data source or each type of target data source) and the first number of responses (the number of data collection responses received by the functional network element when performing the analysis indicated by the target analysis identifier from each target data source or each type of target data source).

[0116] In some embodiments of this disclosure, step 221 may include: determining the second difference temp according to formula (3). i .

[0117] temp i =the number of requests to other dataSource i for analytics(perAnalytics ID)-the number of responses from other dataSource i for Analytics(per Analytics ID) (3)

[0118] Step 222, the second difference temp i The ratio of the number of requests to the number of requests is used as the second quality value A of the measurement data. i .

[0119] In some embodiments of this disclosure, step 222 may include: determining a second mass value A according to formula (4). i .

[0120]

[0121] Step 230: When the functional network element triggers a data collection service for the model indicated by the training target model identifier or analysis identifier, a third quality value C of the measurement data is determined based on the number of times recorded by the second sub-cumulative counter and the number of times recorded by the fourth sub-cumulative counter. The third quality value is used to indicate the missing data (data quality) collected by the functional network element for the model indicated by the training target model identifier or analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for training the model.

[0122] In some embodiments of this disclosure, step 230 may include: when the functional network element triggers a data collection service for the model indicated by the training target model ID (or analysis ID), obtaining the ratio (denoted by C) of "the number of data collection requests triggered by the functional network element - the number of data collection responses received by the functional network element" to "the number of data collection requests triggered by the functional network element".

[0123] In some embodiments of this disclosure, step 230 may include at least one of steps 231 to 232, wherein:

[0124] Step 231: Determine the third difference, Temp, where the third difference is the difference between the number of times recorded by the second sub-cumulative counter and the number of times recorded by the fourth sub-cumulative counter.

[0125] In some embodiments of this disclosure, step 231 may include: determining a third difference, Temp, according to formula (5).

[0126] Temp=the number of requests for model training(per Model ID)-thenumber of responses for model training(per Model ID) (5)

[0127] Step 232: The ratio of the third difference Temp to the number of times recorded by the second sub-cumulative counter is taken as the third quality value C of the measurement data.

[0128] In some embodiments of this disclosure, step 232 may include: determining a third mass value C according to formula (6).

[0129]

[0130] Step 240: Based on different data sources, and measuring the number of second requests and the number of second responses, determine a fourth quality value of the measured data according to the number of second requests and the number of second responses. The fourth quality value is used to represent the missing data (data quality) collected by the functional network element from each target data source or each target data source for training the model indicated by the target model identifier or analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each target data source for training the model.

[0131] In some embodiments of this disclosure, step 240 may include: based on different data sources, when the functional network element triggers data collection from multiple other data sources for the model indicated by the training target model ID (or analysis ID), obtaining the ratio of "the number of data collection requests triggered by the functional network element for each target data source or each type of target data source - the number of data collection responses received by the functional network element from each target data source or each type of target data source" to "the number of data collection requests triggered by the functional network element for each target data source or each type of target data source" (represented by T1, T2, T3... respectively).

[0132] In some embodiments of this disclosure, step 240 may include at least one of steps 241 to 242, wherein:

[0133] Step 241, determine the fourth difference, Temp. iThe fourth difference is the difference between the second number of requests (the number of data collection requests triggered for each target data source or each type of target data source when the functional element trains the model indicated by the target model identifier) ​​and the second number of responses (the number of data collection responses received from each target data source or each type of target data source when the functional element trains the model indicated by the target model identifier).

[0134] In some embodiments of this disclosure, step 241 may include: determining the fourth difference Temp according to formula (7). i .

[0135] Temp i =the number of requests to other dataSource i for model training(per Model ID)-the number of responses from other dataSource i for modeltraining(per Model ID) (7)

[0136] Step 242, the fourth difference temp i The ratio of the number of requests to the number of requests in the first instance is used as the fourth quality value T of the measurement data. i .

[0137] In some embodiments of this disclosure, step 242 may include: determining a fourth mass value T according to formula (8). i .

[0138]

[0139] Step 250: When the functional network element triggers a data collection-related request or subscription to perform a data management task, a fifth quality value of the measurement data is determined based on the number of times recorded by the fifth sub-cumulative counter (the number of times the functional network element triggers a data collection request) and the number of times recorded by the sixth sub-cumulative counter (the number of times the functional network element receives a data collection response). The fifth quality value is used to indicate the missing data collected by the functional network element to perform the data management task indicated by the data management identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element to perform the data management task.

[0140] In some embodiments of this disclosure, step 250 may include at least one of steps 251 to 252, wherein:

[0141] Step 251: Determine the fifth difference, where the fifth difference is the difference between the number of times recorded by the fifth sub-cumulative counter and the number of times recorded by the sixth sub-cumulative counter.

[0142] Step 252: The ratio of the fifth difference to the number of times recorded by the fifth sub-cumulative counter is taken as the fifth quality value of the measurement data.

[0143] Step 260: When measuring the number of third requests and the number of third responses based on different data sources, determine a sixth quality value of the measured data according to the number of third requests (the number of data collection requests triggered by the functional network element for each target data source or each type of target data source) and the number of third responses (the number of data collection responses received by the functional network element from each target data source or each type of target data source). The sixth quality value is used to represent the missing data (data quality) collected by the functional network element from each target data source or each type of target data source in order to perform the data management task indicated by the data management identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each type of target data source in order to perform the data management task.

[0144] In some embodiments of this disclosure, step 260 may include at least one of steps 261 to 262, wherein:

[0145] Step 261, determine the sixth difference, wherein the sixth difference is the difference between the third request count (the number of data collection requests triggered by the functional network element when executing the data management task indicated by the target data management identifier for each target data source or each type of target data source) and the third response count (the number of data collection responses received by the functional network element from each target data source or each type of target data source when executing the data management task indicated by the target data management identifier).

[0146] Step 222: The ratio of the sixth difference to the third number of requests is taken as the sixth quality value of the measurement data.

[0147] Step 400: Evaluate the data collection performance of the functional network element based on the quality value of the collected data.

[0148] In order to measure and evaluate the data collection performance of network elements used to implement network data analysis functions, the above embodiments of this disclosure propose a method for quantifying and evaluating the data quality collected by network elements used to implement network data analysis functions. This method can help understand the data collection performance of network elements used to implement network data analysis functions and provides necessary reference for service optimization of network elements used to implement network data analysis functions.

[0149] Figure 5The diagram illustrates some other embodiments of the data collection performance evaluation method of this disclosure. Preferably, this embodiment can be executed by the data collection performance evaluation system or the data collection performance evaluation device of this disclosure. Figure 5 Steps 100, 200, and 400 of the embodiment are the same as Figure 1 Steps 100, 200, and 400 in the embodiment are the same as or similar to those in the example. Figure 1 Compared with the examples, Figure 5 In addition to steps 100, 200, and 400, the method of this embodiment may also include step 300 between steps 200 and 400, wherein:

[0150] Step 100: Measure the interactive measurement data when the functional network element performs data collection, wherein the functional network element is a network element used to implement network data analysis functions.

[0151] In some embodiments of this disclosure, the functional network element may be an NWDAF network element.

[0152] Step 200: Quantify the performance metrics of the functional network data collection.

[0153] In some embodiments of this disclosure, the performance metrics may be Figure 1 The quality values ​​of the measurement data described in the examples.

[0154] Step 300: Enhance the performance evaluation service architecture to support the quantification and evaluation of the performance of the aforementioned network data collection.

[0155] To achieve the aforementioned function of evaluating the quality (performance) of network data collection, it is necessary to enhance the service-oriented architecture (such as...) Figure 6 As shown in the figure, the data collection performance evaluation device can provide management functions or services, and can quantify the quality of data collection of the network element used to realize network data analysis functions and evaluate the data collection performance of the network element used to realize network data analysis functions according to the method provided in this disclosure.

[0156] This disclosure does not limit the deployment location of the data collection performance evaluation device and the performance evaluation service consumer device.

[0157] In some embodiments of this disclosure, the data collection performance evaluation apparatus can be implemented as a performance evaluation service producer apparatus.

[0158] In some embodiments of this disclosure, the data collection performance evaluation device and the performance evaluation service consumer device can be located in the same network element or in different network elements.

[0159] In some embodiments of this disclosure, step 300 may include at least one of steps 310 and 320, wherein:

[0160] Step 310, Enhance the service interface.

[0161] In some embodiments of this disclosure, step 310 may include at least one of steps 311 and 312, wherein:

[0162] Step 311: The data collection performance evaluation device receives a subscription or request message, wherein the subscription or request message includes a performance evaluation instruction, which includes filtering information and subscription or request behavior.

[0163] In some embodiments of this disclosure, step 311 may include: the data collection performance evaluation device receiving a subscription or request message sent by a performance evaluation service consumer device, wherein the subscription or request message includes a performance evaluation instruction, and the performance evaluation instruction includes filtering information and subscription or request behavior.

[0164] In some embodiments of this disclosure, step 311 may include: subscribe / request: when the event (performance evaluation service) consumer device requests the event producer device to evaluate the functional network data collection performance, the performance evaluation service consumer device may also specify an instruction to instruct the data collection performance evaluation device on the desired behavior when the data collection performance evaluation device evaluates the functional network data collection performance.

[0165] In some embodiments of this disclosure, the filter information includes at least one of an Analytics ID list and an ML Model ID list, wherein the Analytics ID list is used to support the data collection performance evaluation device in measuring the quality of the data collected by the functional network element for performing the analysis indicated by the target Analytics ID, and the ML Model ID list is used to support the data collection performance evaluation device in measuring the quality of the data collected by the functional network element for training the model indicated by the target Model ID.

[0166] In some embodiments of this disclosure, the subscription or request action includes at least one of the following operations: measuring the quality of data collected by the functional network element for performing analysis; measuring the quality of data collected by the functional network element for training a model; measuring the quality of data collected by the functional network element from a target data source for performing analysis; and measuring the quality of data collected by the functional network element from a target data source for training a model.

[0167] Step 312, the data collection performance evaluation device sends a notification or response message, wherein the notification or response message includes filtering information and a response to a subscription or request operation.

[0168] In some embodiments of this disclosure, step 312 may include: the data collection performance evaluation device sending a notification or response message to the performance evaluation service consumer device, wherein the notification or response message includes filter information and a response to a subscription or request operation, wherein the filter information includes an Analytics ID list and an MLModel ID list for indicating measurement targets for data collection quality.

[0169] In some embodiments of this disclosure, step 312 may include: notification / response: the event producer evaluates a subscription or request from the event consumer and responds to the event consumer. For example, depending on the content supported by the event producer, the event producer may accept or reject a subscription / request.

[0170] In some embodiments of this disclosure, the response to a subscription or request operation includes at least one of the following response operations: providing the quality of data collected by the functional network element for performing analysis; providing the quality of data collected by the functional network element for training a model; providing the quality of data collected by the functional network element from a target data source for performing analysis; and providing the quality of data collected by the functional network element from a target data source for training a model.

[0171] Figure 3 Schematic diagrams of the performance evaluation service in some embodiments of this disclosure are also provided. For example... Figure 3 As shown, the performance evaluation service consumer device is NF_A, and the data collection performance evaluation device is NF_B. For Figure 3 In this embodiment, step 310 may include: the performance evaluation service consumer device sending a subscribe message to the performance evaluation producer device, requesting the performance evaluation producer device to evaluate the data collection performance of the function; the data collection performance evaluation device evaluating the subscription from the performance evaluation service consumer and returning a notify message to the performance evaluation service consumer.

[0172] Figure 4 Schematic diagrams of the performance evaluation service in other embodiments of this disclosure are also provided. For example... Figure 4 As shown, the performance evaluation service consumer device is NF_A, and the data collection performance evaluation device is NF_B. For Figure 4In this embodiment, step 310 may include: the performance evaluation service consumer device sending a request message to the performance evaluation producer device, requesting the performance evaluation producer device to evaluate the data collection performance of the function; the data collection performance evaluation device evaluating the request from the performance evaluation service consumer and returning a response message to the performance evaluation service consumer.

[0173] Step 320: Enhance the service interface.

[0174] In some embodiments of this disclosure, step 320 may include: evaluating the data collection performance of the network element used to implement network data analysis functions according to the performance evaluation instructions and the method described in any of the above embodiments.

[0175] In some embodiments of this disclosure, step 320 may include: an action for data collection quality analysis: when the event producer receives a subscription / request instruction from the event consumer, the event producer determines the target for data collection quality analysis based on the information contained in the instruction, and correspondingly performs at least one of the following actions, wherein:

[0176] First, based on the measurement method described above in this disclosure, the missing data (data quality) collected by the network element used to implement the network data analysis function for the analysis indicated by the target analysis ID is measured.

[0177] Second, based on the measurement method described above in this disclosure, the missing data (data quality) collected by the network element used to implement the network data analysis function from other target data sources in order to perform the analysis indicated by the target analysis ID is measured.

[0178] Third, based on the measurement method described above, the missing data (data quality) collected by the network element used to implement network data analysis function is measured for the model indicated by the training target model ID (or analysis ID).

[0179] Fourth, based on the measurement method described above in this disclosure, measure the missing data (data quality) collected by the network element used to implement network data analysis function from each other data source for the model indicated by the training target model ID (or analysis ID).

[0180] Step 400: Evaluate the performance of network data collection related to implementing network data analysis functions based on the above performance metrics and service-oriented architecture.

[0181] The embodiments disclosed above can acquire measurement data on network element interactions for implementing network data analysis functions based on service type and data source type, quantify the missing data (quality) collected by the network element for implementing network data analysis functions, and evaluate the data collection performance of the network element for implementing network data analysis functions.

[0182] In order to measure and evaluate the data collection performance of the network element used to implement network data analysis functions, the above embodiments of this disclosure propose a method for quantifying and evaluating the data quality collected by the network element used to implement network data analysis functions. This method can quantify the quality of the data collected by the network element used to implement network data analysis functions, understand the data collection performance of the network element used to implement network data analysis functions, and provide necessary reference for service optimization of the network element used to implement network data analysis functions.

[0183] First, the above embodiments of this disclosure can obtain the interaction aspect data of the network element used to implement the network data analysis function: according to the service type and data source type, measure the interaction aspect measurement data corresponding to the function network element collecting data from the target data source to execute the target type service, including the number of requests / subscriptions, the number of responses / notifications, etc.

[0184] Secondly, the above embodiments of this disclosure can quantify the quality of the data collected by the network element used to implement network data analysis functions: based on the above measurement data, the missing data situation of the data collected by the functional network element from the target data source for executing the target type service is obtained, which is used to quantify the quality of the data collected by the functional network element and evaluate the performance of the functional network element data collection.

[0185] Third, the above embodiments of this disclosure can enhance the service-oriented architecture: enhance the interface and content of the performance evaluation service to support the evaluation of the data collection performance of the network element used to implement the network data analysis function using the above method.

[0186] The data collection performance evaluation method of this disclosure will be described below through specific embodiments.

[0187] Example 1

[0188] Based on different data sources, the missing data (data quality) collected by the NWDAF network element from other target data sources to perform the analysis indicated by the target analysis ID is measured, and the performance of the data collection service triggered by the NWDAF network element to perform the analysis is evaluated:

[0189] The Aggregator NWDAF element collects data from other NWDAF elements (multiple data sources of the same type) to perform the analysis indicated by the target analysis ID.

[0190] Measure the number of data collection requests to other NWDAF elements_i for analytics(per Analytics ID) triggered by the Aggregator NWDAF element when performing the analytics indicated by the target analytics ID.

[0191] Measure the number of data collection responses received by an NWDAF element from each other data source when performing the analysis indicated by the target analytics ID.

[0192] Obtain the ratios of "the number of data collection requests triggered by the Aggregator NWDAF element to each other NWDAF element - the number of data collection responses received by the Aggregator NWDAF element from each other NWDAF element" to "the number of data collection requests triggered by the Aggregator NWDAF element to each other NWDAF element" (represented by A1, A2, A3... respectively).

[0193] temp i =the number of requests to other NWDAF i for analytics(perAnalytics ID)-the number of responses from other NWDAF i for Analytics(perAnalytics ID);

[0194]

[0195] In the above formula, i = 1, 2, 3...

[0196] This disclosure uses A1, A3, A3... to represent the missing data (data quality) collected by the Aggregator NWDAF network element from each other NWDAF network element to perform the analysis indicated by the target analysis ID, and to evaluate the performance of the data collection service triggered by the NWDAF network element for each target NWDAF network element to perform the analysis. Based on A1, A2, A3..., the quality of the data collected by the Aggregator from each data source can be determined, which can provide a reference for improving the service performance of the NWDAF network element, such as selecting appropriate data sources to obtain data.

[0197] Example 2:

[0198] Based on different data sources, the missing data (data quality) collected by the NWDAF network element from other target data sources to perform the analysis indicated by the target analysis ID is measured, and the performance of the data collection service triggered by the NWDAF network element to perform the analysis is evaluated:

[0199] The NWDAF network element performs the analysis indicated by the target analysis ID, collecting data from other NFs, NWDAF network elements, OAM, etc. (multiple different types of data sources).

[0200] Figure 7 These are schematic diagrams illustrating some embodiments of the data collection performance evaluation apparatus disclosed herein. Figure 7 As shown, the data collection performance evaluation device of this disclosure may include a data measurement module 71, a quality value determination module 72, and a performance evaluation module 73, wherein:

[0201] The data measurement module 71 is configured to measure the interactive measurement data when the functional network element performs data collection, wherein the functional network element is a network element used to implement network data analysis functions.

[0202] The quality value determination module 72 is configured to determine the quality value of the collected data based on the measurement data.

[0203] The performance evaluation module 73 is configured to evaluate the data collection performance of the functional network element based on the quality value of the collected data.

[0204] In some embodiments of this disclosure, the data collection performance evaluation apparatus is configured to perform implementations as described in any of the embodiments above (e.g., Figures 1-5 The operation of the method described in any embodiment.

[0205] The embodiments disclosed above can measure interactive data when a network element for implementing network data analysis functions collects data, quantify the quality of data collected by the network element for implementing network data analysis functions from a target data source in order to perform a target type service, evaluate the data collection performance of the network element for implementing network data analysis functions, and provide a reference for service optimization of the network element for implementing network data analysis functions.

[0206] Figure 8 Schematic diagrams of other embodiments of the data collection performance evaluation apparatus of this disclosure. For example... Figure 8 As shown, the data collection performance evaluation device includes a memory 81 and a processor 82.

[0207] Memory 81 is used to store instructions, and processor 82 is coupled to memory 81. Processor 82 is configured to execute instructions stored in memory to implement the above embodiments (e.g., Figures 1-5 The data collection performance evaluation method described in any embodiment.

[0208] like Figure 7 As shown, the data collection performance evaluation device also includes a communication interface 83 for information exchange with other devices. Additionally, the device includes a bus 84, through which the processor 82, communication interface 83, and memory 81 communicate with each other.

[0209] The memory 81 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. The memory 81 may also be a memory array. The memory 81 may also be divided into blocks, and these blocks may be combined into virtual volumes according to certain rules.

[0210] Furthermore, processor 82 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0211] The embodiments of this disclosure can measure the interaction aspects of a network element used to implement network data analysis functions when it collects data from a target data source to perform a target type service, based on the service type and data source type.

[0212] The embodiments disclosed above can, based on the above-mentioned measurement data, obtain the missing data situation of the data collected by the network element used to implement the network data analysis function from the target data source for executing the target type service, so as to quantify the quality of the data collected by the network element used to implement the network data analysis function and evaluate the data collection performance of the network element used to implement the network data analysis function.

[0213] Figure 6 These are schematic diagrams illustrating some embodiments of the data collection performance evaluation system disclosed herein. Figure 6 As shown, the data collection performance evaluation system disclosed herein may include a data collection performance evaluation device 61 and a performance evaluation service consumer device 62, wherein:

[0214] A performance evaluation service consumer device 62 is configured to send a subscription or request message to a data collection performance evaluation device, wherein the subscription or request message includes performance evaluation instructions, which include filtering information and subscription or request actions;

[0215] The data collection performance evaluation device 61 is configured to perform the performance evaluation according to the performance evaluation instructions, as described in any of the above embodiments (e.g., Figures 1-5 The method described in any embodiment evaluates the data collection performance of multiple network elements 60 used to implement network data analysis functions; and sends a notification or response message to a performance evaluation service consumer device, wherein the notification or response message includes filtering information and a response to a subscription or request operation.

[0216] This disclosure does not limit the deployment location of the data collection performance evaluation device and the performance evaluation service consumer device.

[0217] In some embodiments of this disclosure, the data collection performance evaluation apparatus 61 can be implemented as a performance evaluation service producer apparatus.

[0218] In some embodiments of this disclosure, the data collection performance evaluation device 61 and the performance evaluation service consumer device 62 may be located in the same network element or in different network elements.

[0219] In some embodiments of this disclosure, the data collection performance evaluation device 61 may be as described in any of the embodiments above (e.g., Figure 7 or Figure 8 The data collection performance evaluation device described in the embodiment)

[0220] This disclosure enhances the performance evaluation service architecture to support the implementation of the above methods and the evaluation of the data collection performance of the network elements used to implement network data analysis functions.

[0221] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement any of the embodiments described above (e.g., Figures 1-5 The data collection performance evaluation method described in any embodiment.

[0222] In some embodiments of this disclosure, the computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0223] This disclosure provides a method for evaluating the data collection performance of network elements used to implement network data analysis functions, including quantification, service-oriented architecture enhancement, and service content enhancement. This disclosure can quantify the quality of data collected by network elements used to implement network data analysis functions based on service type and data source, understand the data collection performance of the network elements used to implement network data analysis functions, and provide necessary reference for managers to optimize the services of the network elements used to implement network data analysis functions.

[0224] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0225] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0226] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0227] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0228] The data collection performance evaluation device, performance evaluation service consumer device, data measurement module, quality value determination module, and performance evaluation module described above can be implemented as a general-purpose processor, programmable logic controller (PLC), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described in this application.

[0229] This concludes the detailed description of the present disclosure. To avoid obscuring the concept of the disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0230] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a non-transitory computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0231] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for evaluating data collection performance, comprising: The measurement data includes interactive measurement data when a functional network element performs data collection. The functional network element is a network element used to implement network data analysis functions. The measurement data includes the number of requests sent by the functional network element and the number of responses received by the functional network element. The request method includes at least one of request and subscription, and the response method includes at least one of response and notification. The quality value of the collected data is determined based on the measurement data; The data collection performance of the functional network element is evaluated based on the quality value of the collected data; The measurement data collected by the measurement function network element during data collection includes the following interactive measurement data: The number of data collection requests sent by the functional network element to the target data source is measured. When the functional network element triggers a data collection-related request or subscription, the first cumulative counter is incremented by 1, wherein the first cumulative counter is a cumulative counter related to the number of data collection requests or subscriptions triggered by the functional network element.

2. The method according to claim 1, wherein, The measurement data collected by the network element during data collection includes the following interactive measurement data: Based on the target service type, target data source type, and target data source, measure the interaction aspects of the functional network element when it collects data from the target data source to perform the target type service. The target service type includes at least one of analysis service, model training service, and data management service.

3. The method according to claim 2, wherein, The quality values ​​of the collected data are determined based on the measurement data, including: Based on the measurement data, the missing data information of the data collected by the functional network element from the target data source for performing the target type service is obtained, wherein the missing data information is used to quantify the quality of the data collected by the functional network element.

4. The method according to any one of claims 1-3, wherein, The measurement data collected by the measurement function network element during data collection also includes the following interactive measurement data: The number of data collection responses received by the functional network element from the target data source is measured. When the functional network element receives a data collection-related response or notification, the second cumulative counter is incremented by 1, wherein the second cumulative counter is a cumulative counter related to the number of data collection responses or notifications received by the functional network element.

5. The method according to claim 4, wherein, The measurement of the number of data collection requests sent by the functional network element to the target data source includes: When the functional network element triggers a data collection-related request or subscription to perform an analysis task, the first cumulative counter includes a first sub-cumulative counter based on different analysis identifiers. The first sub-cumulative counter is a sub-cumulative counter related to the analysis identifier. The first sub-cumulative counter is used to measure the number of data collection requests triggered by the functional network element when performing the analysis indicated by the target analysis identifier. And / or, When the functional network element triggers a data collection-related request or subscription to perform a model training task, based on different model identifiers or analysis identifiers, the first cumulative counter includes a second sub-cumulative counter. The second sub-cumulative counter is a sub-cumulative counter related to the model identifier or analysis identifier. The second sub-cumulative counter is used to measure the number of data collection requests triggered by the functional network element when it is training the model indicated by the target model identifier. And / or, When the functional network element triggers a data collection-related request or subscription to perform a data management task, the first cumulative counter includes a fifth sub-cumulative counter based on different data management identifiers. The fifth sub-cumulative counter is a sub-cumulative counter related to the data management identifier and is used to measure the number of data collection requests triggered by the functional network element when performing the data management task indicated by the target data management identifier. And / or, Based on different data sources, at least one of the first request count, the second request count, and the third request count is measured, wherein the first request count is the number of data collection requests triggered by the functional network element for each target data source or each type of target data source when performing the analysis indicated by the target analysis identifier; the second request count is the number of data collection requests triggered by the functional network element for each target data source or each type of target data source when training the model indicated by the target model identifier; and the third request count is the number of data collection requests triggered by the functional network element for each target data source or each type of target data source when performing the data management task indicated by the target data management identifier.

6. The method according to claim 5, wherein, The number of times the functional network element receives data collection responses from the target data source is measured includes: When the functional network element collects data from the target data source for analysis, the second cumulative counter includes a third sub-cumulative counter based on different analysis identifiers. The third sub-cumulative counter is a sub-cumulative counter associated with the analysis identifier. The third sub-cumulative counter is used to measure the number of data collection responses received by the functional network element when performing the analysis indicated by the target analysis identifier. And / or, When the functional network element collects data from the target data source for model training, based on different model identifiers or analysis identifiers, the second cumulative counter includes a fourth sub-cumulative counter. The fourth sub-cumulative counter is a sub-cumulative counter related to the model identifier or analysis identifier. The fourth sub-cumulative counter is used to measure the number of data collection responses received by the functional network element when training the model indicated by the target model identifier. And / or, When the functional network element collects data from the target data source for data management, the second cumulative counter includes a sixth sub-cumulative counter based on different data management identifiers. The sixth sub-cumulative counter is a sub-cumulative counter associated with the data management identifier. The sixth sub-cumulative counter is used to measure the number of data collection responses received by the functional network element when executing the data management task indicated by the target data management identifier. And / or, Based on different data sources, at least one of the following is measured: the first response count, the second response count, and the third response count. The first response count is the number of data collection responses received by the functional network element from each target data source or each type of target data source when performing the analysis indicated by the target analysis identifier. The second response count is the number of data collection responses received by the functional network element from each target data source or each type of target data source when training the model indicated by the target model identifier. The third response count is the number of data collection responses received by the functional network element from each target data source or each type of target data source when performing the data management task indicated by the target data management identifier.

7. The method according to claim 6, wherein, When the functional network element triggers a data collection-related request or subscription to perform an analysis task, The step of determining the quality value of the collected data based on the measurement data includes: determining a first quality value of the measurement data based on the number of times recorded by the first sub-cumulative counter and the number of times recorded by the third sub-cumulative counter, wherein the first quality value is used to indicate the missing data of the data collected by the functional network element for performing the analysis indicated by the target analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for performing the analysis.

8. The method according to claim 7, wherein, Determining the first quality value of the measurement data based on the number of data collection requests triggered by the functional network element and the number of data collection responses received by the functional network element includes: Determine the first difference, where the first difference is the difference between the number of times recorded by the first sub-cumulative counter and the number of times recorded by the third sub-cumulative counter; The ratio of the first difference to the number of times recorded by the first sub-cumulative counter is used as the first quality value of the measurement data.

9. The method according to claim 6, wherein, When measuring the number of first requests and the number of first responses based on different data sources, The step of determining the quality value of the collected data based on the measurement data includes: determining a second quality value of the measurement data based on the first number of requests and the first number of responses, wherein the second quality value is used to indicate the missing data collected by the functional network element from each target data source or each target data source in order to perform the analysis indicated by the target analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each target data source in order to perform the analysis.

10. The method according to claim 6, wherein, When the functional network element triggers the data collection service for the model indicated by the training target model identifier or analysis identifier, The step of determining the quality value of the collected data based on the measurement data includes: determining a third quality value of the measurement data based on the number of times recorded by the second sub-cumulative counter and the number of times recorded by the fourth sub-cumulative counter, wherein the third quality value is used to indicate the missing data of the data collected by the functional network element for the model indicated by the training target model identifier or analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for training the model.

11. The method according to claim 6, wherein, In cases where the number of second requests and the number of second responses are measured based on different data sources,... The step of determining the quality value of the collected data based on the measurement data includes: determining a fourth quality value of the measurement data based on the second number of requests and the second number of responses, wherein the fourth quality value is used to indicate the missing data collected by the functional network element from each target data source or each target data source for training the model indicated by the target model identifier or analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each target data source for training the model.

12. The method according to claim 6, wherein, When the functional network element triggers a data collection-related request or subscription to perform a data management task, The step of determining the quality value of the collected data based on the measurement data includes: determining a fifth quality value of the measurement data based on the number of times recorded by the fifth sub-cumulative counter and the number of times recorded by the sixth sub-cumulative counter, wherein the fifth quality value is used to indicate the missing data collected by the functional network element for executing the data management task indicated by the data management identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for executing the data management task.

13. The method according to claim 6, wherein, When measuring the number of third requests and third responses based on different data sources, The step of determining the quality value of the collected data based on the measurement data includes: determining a sixth quality value of the measurement data based on the third request count and the third response count, wherein the sixth quality value is used to indicate the missing data collected by the functional network element from each target data source or each target data source in order to perform the data management task indicated by the data management identifier, and is used to evaluate the performance of the data collection service for each target data source or each target data source triggered by the functional network element in order to perform the data management task.

14. The method according to any one of claims 1-3, further comprising: Receive subscription or request messages, wherein the subscription or request messages include performance evaluation instructions, the performance evaluation instructions include filtering information, and subscription or request behavior; According to the performance evaluation instructions, the data collection performance of the functional network element is evaluated according to the method described in any one of claims 1-13; Send a notification or response message, wherein the notification or response message includes filtering information and a response to a subscription or request operation.

15. The method of claim 14, wherein: The filtering information includes at least one of an analysis identifier list, a model identifier list, and a data management identifier list, wherein the analysis identifier list is used to support measuring the quality of data collected by the functional network element for performing the analysis indicated by the target analysis identifier, the model identifier list is used to support measuring the quality of data collected by the functional network element for training the model indicated by the target model identifier, and the data management identifier list is used to support measuring the quality of data collected by the functional network element for performing the data management task indicated by the target data management identifier; And / or, The subscription or request behavior includes at least one of the following operations: measuring the quality of data collected by the functional network element for performing analysis; measuring the quality of data collected by the functional network element for training a model; measuring the quality of data collected by the functional network element for performing data management tasks; measuring the quality of data collected by the functional network element from a target data source for performing analysis; measuring the quality of data collected by the functional network element from a target data source for training a model; measuring the quality of data collected by the functional network element from a target data source for performing data management tasks. And / or, The response to a subscription or request operation includes at least one of the following response operations: providing the quality of the data collected by the functional network element for performing analysis; providing the quality of the data collected by the functional network element for training a model; providing the quality of the data collected by the functional network element for performing data management tasks; providing the quality of the data collected by the functional network element from a target data source for performing analysis; providing the quality of the data collected by the functional network element from a target data source for training a model; providing the quality of the data collected by the functional network element from a target data source for performing data management tasks.

16. A data collection performance evaluation device, comprising: The data measurement module is configured to measure the interactive measurement data when the functional network element performs data collection. The functional network element is a network element used to implement network data analysis functions. The measurement data includes the number of requests issued by the functional network element and the number of responses received by the functional network element. The request method includes at least one of request and subscription, and the response method includes at least one of response and notification. A quality value determination module is configured to determine the quality value of the collected data based on the measurement data. The performance evaluation module is configured to evaluate the data collection performance of the functional network element based on the quality value of the collected data; The data collection performance evaluation device is configured to measure the number of times the functional network element sends data collection requests to the target data source when measuring the interaction data of the functional network element performing data collection. When the functional network element triggers a data collection-related request or subscription, the first cumulative counter is incremented by 1, wherein the first cumulative counter is a cumulative counter related to the number of data collection requests or subscriptions triggered by the functional network element.

17. The data collection performance evaluation apparatus according to claim 16, wherein, The data collection performance evaluation device is configured to measure the interaction aspects of a functional network element when it performs data collection, based on a target service type, a target data source type, and a target data source. The target service type includes at least one of analysis services, model training services, and data management services.

18. The data collection performance evaluation apparatus according to claim 17, wherein, The data collection performance evaluation device, when determining the quality value of the collected data based on the measurement data, is configured to obtain the missing data of the data collected by the functional network element from the target data source for performing the target type service, based on the measurement data, wherein the missing data is used to quantify the quality of the data collected by the functional network element.

19. The data collection performance evaluation apparatus according to any one of claims 16-18, wherein, In addition to measuring the interaction data when a functional network element performs data collection, the data collection performance evaluation device is also configured to measure the number of data collection responses received by the functional network element from the target data source. When the functional network element receives a data collection-related response or notification, a second cumulative counter is incremented by 1, wherein the second cumulative counter is a cumulative counter related to the number of data collection responses or notifications received by the functional network element.

20. The data collection performance evaluation apparatus according to claim 19, wherein, The data collection performance evaluation device, when measuring the number of data collection requests sent by the functional network element to the target data source, is configured to, when the functional network element triggers data collection-related requests or subscriptions to perform an analysis task, based on different analysis identifiers, include a first sub-cumulative counter, wherein the first sub-cumulative counter is a sub-cumulative counter associated with the analysis identifier, and the first sub-cumulative counter is used to measure the number of data collection requests triggered by the functional network element when performing the analysis indicated by the target analysis identifier; and / or, when the functional network element triggers data collection-related requests or subscriptions to perform a model training task, based on different model identifiers or analysis identifiers, the first cumulative counter includes a second sub-cumulative counter, the second sub-cumulative counter is a sub-cumulative counter associated with the model identifier or analysis identifier, and the second sub-cumulative counter is used to measure the number of data collection requests triggered by the functional network element when training the model indicated by the target model identifier; and / or, when the functional network element performs a data management task... In the event of triggering a data collection-related request or subscription, based on different data management identifiers, the first cumulative counter includes a fifth sub-cumulative counter, which is a sub-cumulative counter associated with the data management identifier. The fifth sub-cumulative counter is used to measure the number of data collection requests triggered by the functional network element when performing the data management task indicated by the target data management identifier; and / or, based on different data sources, measuring at least one of the first request count, the second request count, and the third request count, wherein the first request count is the number of data collection requests triggered by the functional network element for each target data source or each type of target data source when performing the analysis indicated by the target analysis identifier; the second request count is the number of data collection requests triggered by the functional network element for each target data source or each type of target data source when training the model indicated by the target model identifier; and the third request count is the number of data collection requests triggered for each target data source or each type of target data source when performing the data management task indicated by the target data management identifier.

21. The data collection performance evaluation apparatus according to claim 20, wherein, The data collection performance evaluation device, when measuring the number of data collection responses received by the functional network element from the target data source, is configured such that, when the functional network element collects data from the target data source for analysis, based on different analysis identifiers, the second cumulative counter includes a third sub-cumulative counter, wherein the third sub-cumulative counter is a sub-cumulative counter associated with the analysis identifier, and the third sub-cumulative counter is used to measure the number of data collection responses received by the functional network element when performing the analysis indicated by the target analysis identifier; and / or, when the functional network element collects data from the target data source for model training, based on different model identifiers or analysis identifiers, the second cumulative counter includes a fourth sub-cumulative counter, the fourth sub-cumulative counter is a sub-cumulative counter associated with the model identifier or analysis identifier, and the fourth sub-cumulative counter is used to measure the number of data collection responses received by the functional network element when training the model indicated by the target model identifier; and / or, when the functional network element collects data from the target data source for data management... Below, based on different data management identifiers, the second cumulative counter includes a sixth sub-cumulative counter, wherein the sixth sub-cumulative counter is a sub-cumulative counter associated with the data management identifier, and the sixth sub-cumulative counter is used to measure the number of data collection responses received by the functional network element when performing the data management task indicated by the target data management identifier; and / or, based on different data sources, measuring at least one of the first response count, the second response count, and the third response count, wherein the first response count is the number of data collection responses received by the functional network element from each target data source or each type of target data source when performing the analysis indicated by the target analysis identifier, the second response count is the number of data collection responses received by the functional network element from each target data source or each type of target data source when training the model indicated by the target model identifier, and the third response count is the number of data collection responses received by the functional network element from each target data source or each type of target data source when performing the data management task indicated by the target data management identifier.

22. The data collection performance evaluation apparatus according to claim 21, wherein, When the functional network element triggers a data collection-related request or subscription to perform an analysis task, The data collection performance evaluation device, when determining the quality value of the collected data based on the measurement data, is configured to determine a first quality value of the measurement data based on the number of times recorded by a first sub-cumulative counter and the number of times recorded by a third sub-cumulative counter. The first quality value is used to indicate the missing data collected by the functional network element for performing the analysis indicated by the target analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for performing the analysis.

23. The data collection performance evaluation apparatus according to claim 22, wherein, The data collection performance evaluation device is configured to determine a first difference when determining a first quality value of the measurement data based on the number of data collection requests triggered by the functional network element and the number of data collection responses received by the functional network element. The first difference is the difference between the number of times recorded by the first sub-cumulative counter and the number of times recorded by the third sub-cumulative counter. The ratio of the first difference to the number of times recorded by the first sub-cumulative counter is used as the first quality value of the measurement data.

24. The data collection performance evaluation apparatus according to claim 21, wherein, When measuring the number of first requests and the number of first responses based on different data sources, The data collection performance evaluation device, when determining the quality value of the collected data based on the measurement data, is configured to determine a second quality value of the measurement data based on a first number of requests and a first number of responses. The second quality value is used to indicate the missing data collected by the functional network element from each target data source or each target data source in order to perform the analysis indicated by the target analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each target data source in order to perform the analysis.

25. The data collection performance evaluation apparatus according to claim 21, wherein, When the functional network element triggers the data collection service for the model indicated by the training target model identifier or analysis identifier, The data collection performance evaluation device, after determining the quality value of the collected data based on the measurement data, is configured to determine a third quality value of the measurement data based on the number of times recorded by the second sub-cumulative counter and the number of times recorded by the fourth sub-cumulative counter. The third quality value is used to indicate the missing data collected by the functional network element for the model indicated by the training target model identifier or analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for training the model.

26. The data collection performance evaluation apparatus according to claim 21, wherein, In cases where the number of second requests and the number of second responses are measured based on different data sources,... The data collection performance evaluation device, after determining the quality value of the collected data based on the measurement data, is configured to determine a fourth quality value of the measurement data based on the second number of requests and the second number of responses. The fourth quality value is used to indicate the missing data collected by the functional network element from each target data source or each target data source for training the model indicated by the target model identifier or analysis identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each target data source for training the model.

27. The data collection performance evaluation apparatus according to claim 21, wherein, When the functional network element triggers a data collection-related request or subscription to perform a data management task, The data collection performance evaluation device, when determining the quality value of the collected data based on the measurement data, is configured to determine a fifth quality value of the measurement data based on the number of times recorded by the fifth sub-cumulative counter and the number of times recorded by the sixth sub-cumulative counter. The fifth quality value is used to indicate the missing data collected by the functional network element for executing the data management task indicated by the data management identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for executing the data management task.

28. The data collection performance evaluation apparatus according to claim 21, wherein, When measuring the number of third requests and third responses based on different data sources, The data collection performance evaluation device, after determining the quality value of the collected data based on the measurement data, is configured to determine a sixth quality value of the measurement data based on the third number of requests and the third number of responses. The sixth quality value is used to indicate the missing data collected by the functional network element from each target data source or each target data source in order to perform the data management task indicated by the data management identifier, and is used to evaluate the performance of the data collection service triggered by the functional network element for each target data source or each target data source in order to perform the data management task.

29. A data collection performance evaluation device, comprising: The memory is configured to store instructions; The processor is configured to execute the instructions, causing the data collection performance evaluation device to perform operations implementing the method as described in any one of claims 1-15.

30. A data collection performance evaluation system, comprising the data collection performance evaluation apparatus as described in any one of claims 16-29.

31. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-15.

Images