Communication method and apparatus

Abstract

The present application relates to the field of communications. Disclosed are a communication method and apparatus. The method comprises: a first network device sending first information to a consumer node, wherein the first information comprises first identification information and an analysis output; the first network device receiving second information from the consumer node, wherein the second information is used for accuracy evaluation, the second information comprises the first identification information in the first information which is influenced by and / or on which an execution action influencing the accuracy evaluation is based, and the execution action is executed on the basis of the analysis output. By means of the present application, the accuracy of analyzing accuracy evaluation in a subscription scenario is improved.

Description

Communication method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on November 30, 2023, with application number 202311645743.X and application name “Communication Method and Device,” the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of communications, and in particular to communication methods and devices. Background Art

[0003] Currently, consumer nodes use Nnwdaf analysis subscription requests (analytics subscription_subscribe) / Nnwdaf analysis information requests (analytics info_request) to subscribe to analysis from the network data analytics function containing analytics logical function (Anlf). Anlf then provides the analysis content to the consumer node through Nnwdaf analysis subscription notifications (analytics subscription_notify) / Nnwdaf analysis information request responses (analytics info_request response). Subsequently, the consumer node takes pre-configured actions based on the analysis content (also known as analysis output). The execution of the action is fed back to Anlf through feedback information. Anlf then performs accuracy assessment based on the feedback information.

[0004] Currently, Anlf has low accuracy when performing accuracy evaluation based on feedback information.

[0005] Summary of the Invention

[0006] The present application provides a communication method and apparatus for improving the accuracy of accuracy assessment in an analysis subscription scenario.

[0007] To achieve the above objectives, this application adopts the following technical solutions:

[0008] In a first aspect, a communication method is provided, which is applied to a first network device. The method may be performed by the first network device, a component or device (e.g., a processor, chip, or chip system) applied to the first network device, or a logic module or software capable of implementing all or part of the functionality of the first network device. The method includes: sending first information to a consumer node, wherein the first information includes first identification information and an analysis output; and receiving second information from the consumer node, wherein the second information is used for accuracy assessment, and the second information includes first identification information in the first information on which an action affecting the accuracy assessment is based and / or affected, wherein the action is performed based on the analysis output. To facilitate understanding, the second information is further described: the second information is used to perform an accuracy assessment on an analysis subscribed to by the consumer node and / or on an accuracy assessment of a model corresponding to the analysis. In other words, the second information is used to generate accuracy information for the analysis and / or the model corresponding to the analysis. That is, the accuracy assessment may include one or more of: generating accuracy information, performing an accuracy assessment on the analysis subscribed to by the consumer node, and performing an accuracy assessment on the model corresponding to the analysis.

[0009] In another implementation, the second information may further include first information based on and / or influenced by an action performed that may affect the analysis output and / or the true value corresponding to the analysis (also known as the prediction) (optionally, the second information may further include identification information of the corresponding first information). In another implementation, the second information is used to analyze the corresponding accuracy assessment or to determine the accuracy information of the analysis, and the second information includes the first identification information in the first information based on and / or influenced by an action performed that affects the true value corresponding to the analysis output.

[0010] In the first aspect, the first network device can determine, based on the identification information in the second information, the analysis output upon which the consumer node took the execution action. The device can also determine the analysis output upon which the execution action is based and / or affected that affects the accuracy assessment, thereby ensuring the reliability of the data used for the accuracy assessment (for example, by excluding the affected analysis output and corresponding accuracy information from the accuracy assessment, or by correcting the affected analysis output and corresponding accuracy information before using it for the accuracy assessment), thereby improving the accuracy of the accuracy assessment.

[0011] Furthermore, because the identification information in the first and second information can serve as an indicator, the identification information can be used to distinguish the correspondence between the feedback information and the analysis output. This eliminates the need for feedback information to be reported in a one-to-one correspondence with the analysis output. In other words, the consumer node can report multiple pieces of feedback information at once. This facilitates more efficient information statistics, reduces unnecessary messages between the consumer node and the first network device, and thus reduces communication overhead.

[0012] In one possible implementation, the first identification information indicates the first information. Alternatively, the first identification information indicates the analysis output. In other words, the first information includes M analysis outputs and M first identification information, each first identification information indicating an analysis output, where M is a positive integer.

[0013] In this implementation, two designs of the first identification information are provided, which can indicate analysis outputs of different granularities.

[0014] In one possible implementation, the method further includes: determining an analysis output for accuracy assessment based on the second information; and performing accuracy assessment based on the analysis output for accuracy assessment.

[0015] In this implementation, accuracy information with higher reliability for accuracy evaluation is determined based on the second information, thereby improving the accuracy of the accuracy evaluation.

[0016] In one possible implementation, the method further includes: sending third information to the second network device, wherein the third information is used to determine data for accuracy assessment, the third information includes the first information and the second information, or the third information includes an analysis output for accuracy assessment, and the analysis output for accuracy assessment is determined based on the second information; receiving accuracy information for accuracy assessment from the second network device; and performing accuracy assessment based on the data for accuracy assessment.

[0017] In this implementation, the second network device sends data for accuracy assessment to the first network device, and the first network device performs accuracy assessment based on the data used for accuracy assessment, thereby ensuring the reliability of the data used for accuracy assessment (for example, excluding the affected analysis output and corresponding accuracy information and other data from the accuracy assessment, or correcting the affected analysis output and corresponding accuracy information and other data before using them for accuracy assessment), which can improve the accuracy of the accuracy assessment.

[0018] In one possible implementation, the first network device is a network data analysis function network element with an analysis logic function, and the second network device is a network data analysis function network element with a model training logic function.

[0019] In one possible implementation, the method further includes:

[0020] Before sending the first information to the consumer node, an analysis subscription request is received from the consumer node.

[0021] In a second aspect, a communication method is provided, which is applied to a consumer node. The execution subject of the method can be a consumer node, or a component or device applied to a consumer node (such as a processor, chip, or chip system, etc.), or a logic module or software that can realize all or part of the functions of the consumer node. The method includes: receiving first information, wherein the first information includes first identification information and analysis output; determining second information based on the first information, the second information is used for accuracy assessment, and the second information includes the first identification information in the first information on which the execution action affecting the accuracy assessment is based and / or affected, wherein the execution action is performed based on the analysis output; sending the second information to the first network device.

[0022] In the second aspect, the consumer node sends second information to the first network device, which allows the first network device to determine, based on the identification information in the second information, the analysis output based on which the consumer node took the execution action. The first network device also determines the analysis output based on and / or affected by the execution action that affects the accuracy assessment (or the analysis output corresponding to the analysis object affected by the execution action), thereby ensuring the reliability of the data used for the accuracy assessment (for example, excluding the affected analysis output and corresponding accuracy information from the accuracy assessment, or correcting the affected analysis output and corresponding accuracy information before using it for the accuracy assessment), thereby improving the accuracy of the accuracy assessment.

[0023] In a possible implementation, the first identification information indicates the first information, or the first identification information indicates the analysis output.

[0024] In one possible implementation, the method further includes:

[0025] Before receiving the first information, an analysis subscription request is sent.

[0026] In a third aspect, a communication method is provided, which is applied to a second network device. The execution subject of the method can be a second network device, or a component or device (such as a processor, chip, or chip system, etc.) applied to the second network device, or a logic module or software that can realize all or part of the functions of the second network device. The method includes: receiving third information from a first network device, wherein the third information is used to determine data for accuracy assessment, the third information includes first information and second information, or the third information includes an analysis output for accuracy assessment, and the analysis output for accuracy assessment is determined based on the second information; the second information includes first identification information in the first information on which and / or on which an execution action affecting the accuracy assessment is based, the first information includes the first identification information and the analysis output, and the execution action is performed based on the analysis output; determining data for accuracy assessment based on the third information; and sending data for accuracy assessment to the first network device.

[0027] In the third aspect, the second network device determines the accuracy information used for accuracy assessment (the accuracy information is more reliable, for example, data after excluding the accuracy information corresponding to the affected analysis output, or data after correcting the accuracy information corresponding to the affected analysis output) and sends it to the first network device, which can improve the accuracy of the accuracy assessment.

[0028] In a fourth aspect, a communication method is provided, which is applied to a first network device. The execution subject of the method can be the first network device, or a component or device (such as a processor, chip, or chip system, etc.) applied to the first network device, or a logic module or software that can realize all or part of the functions of the first network device. The method includes: sending first information to a consumer node, wherein the first information includes an analysis output and second identification information of an analysis object of the analysis output; receiving second information from the consumer node, wherein the second information is used for accuracy assessment, and the second information includes second identification information of the analysis object affected by the execution action, wherein the execution action is performed based on the analysis output.

[0029] In the fourth aspect, the first network device can determine, based on the identification information in the second information, which analysis output the consumer node took an execution action based on. It can also determine the analysis output corresponding to the analysis object affected by the execution action, thereby ensuring the reliability of the data used for accuracy assessment (for example, excluding the affected analysis output and corresponding accuracy information from the accuracy assessment, or correcting the affected analysis output and corresponding accuracy information before using them in the accuracy assessment), thereby improving the accuracy of the accuracy assessment.

[0030] In a fifth aspect, a communication method is provided, which is applied to a consumer node. The execution subject of the method can be a consumer node, or a component or device applied to the consumer node (such as a processor, chip, or chip system, etc.), or a logic module or software that can realize all or part of the functions of the consumer node. The method includes: receiving first information, wherein the first information includes an analysis output and second identification information of an analysis object of the analysis output; determining second information based on the first information, the second information is used for accuracy assessment, the second information includes second identification information of the analysis object affected by the execution action, wherein the execution action is performed based on the analysis output; and sending the second information to the first network device.

[0031] In the fifth aspect, the consumer node sends the second information to the first network device. Based on the identification information in the second information, the first network device can determine the analysis output based on which the consumer node took the execution action. The first network device also determines the analysis output corresponding to the analysis object affected by the execution action, thereby ensuring the reliability of the data used for accuracy assessment (for example, excluding the affected analysis output and corresponding accuracy information from the accuracy assessment, or correcting the affected analysis output and corresponding accuracy information before using it for accuracy assessment), thereby improving the accuracy of the accuracy assessment.

[0032] In a sixth aspect, a communication device is provided, which is applied to a first network device. The communication device can be the first network device or a chip or system on chip in the first network device. The communication device can implement the functions performed by the first network device in the above-mentioned first aspect or the possible design of the first aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. For example, the communication device includes: a transceiver module for sending first information to a consumer node, wherein the first information includes first identification information and analysis output; the transceiver module is also used to receive second information from the consumer node, wherein the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which the execution action affecting the accuracy evaluation is based and / or affected, wherein the execution action is executed based on the analysis output.

[0033] To facilitate understanding, the second information is further described as follows: the second information is used to perform an accuracy assessment on the analysis subscribed to by the consumer node and / or the accuracy assessment on the model corresponding to the analysis. In other words, the second information is used to generate accuracy information for the analysis and / or the model corresponding to the analysis. In other words, accuracy assessment may include one or more of: generating accuracy information, performing an accuracy assessment on the analysis subscribed to by the consumer node, and performing an accuracy assessment on the model corresponding to the analysis.

[0034] In another implementation, the second information may also include the first information on which and / or which affects the execution action that may affect the analysis output and / or the corresponding real value of the analysis (also known as prediction) (optionally, the second information may also include identification information of the corresponding first information).

[0035] In a possible implementation, the first identification information indicates the first information, or the first identification information indicates the analysis output.

[0036] In a possible implementation, the apparatus further includes a processing module configured to:

[0037] determining an analysis output for accuracy assessment based on the second information;

[0038] The accuracy assessment is performed based on the analysis output for accuracy assessment.

[0039] In one possible implementation, the apparatus further includes a processing module;

[0040] The transceiver module is further configured to send third information to the second network device, wherein the third information is used to determine data for accuracy assessment, and the third information includes the first information and the second information, or the third information includes an analysis output for accuracy assessment, and the analysis output for accuracy assessment is determined based on the second information;

[0041] The transceiver module is further configured to receive accuracy information for accuracy assessment from the second network device;

[0042] A processing module is used for performing accuracy evaluation based on the data used for accuracy evaluation.

[0043] In one possible implementation, the first network device is a network data analysis function network element with an analysis logic function, and the second network device is a network data analysis function network element with a model training logic function.

[0044] In a possible implementation, the transceiver module is further configured to receive an analysis subscription request from the consumer node before sending the first information to the consumer node.

[0045] In the seventh aspect, a communication device is provided, which is applied to a consumer node. The communication device can be a consumer node or a chip or system on chip in the consumer node. The communication device can implement the functions performed by the consumer node in the above-mentioned second aspect or the possible design of the second aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. For example, the communication device includes: a transceiver module for receiving first information, wherein the first information includes first identification information and analysis output; a processing module for determining second information based on the first information, the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which the execution action affecting the accuracy evaluation is based and / or affected, wherein the execution action is executed based on the analysis output; the transceiver module is also used to send the second information to the first network device.

[0046] In a possible implementation, the first identification information indicates the first information, or the first identification information indicates the analysis output.

[0047] In a possible implementation, the transceiver module is further configured to send an analysis subscription request before receiving the first information.

[0048] In an eighth aspect, a communication device is provided, which is applied to a second network device. The communication device can be the second network device or a chip or system-on-chip in the second network device. The communication device can implement the functions performed by the second network device in the third aspect or a possible design of the third aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the communication device includes: a transceiver module for receiving third information from a first network device, wherein the third information is used to determine data for accuracy assessment, the third information includes first information and second information, or the third information includes analysis output for accuracy assessment, the analysis output for accuracy assessment being determined based on the second information; the second information includes first identification information in the first information on which an action affecting the accuracy assessment is based and / or affected, the first information includes the first identification information and the analysis output, the action being performed based on the analysis output; a processing module for determining data for accuracy assessment based on the third information; and the transceiver module is further configured to send the data for accuracy assessment to the first network device.

[0049] In the ninth aspect, a communication device is provided, which is applied to a first network device. The communication device can be the first network device or a chip or system on chip in the first network device. The communication device can implement the function performed by the first network device in the fourth aspect or the possible design of the fourth aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the communication device includes: a transceiver module for sending first information to a consumer node, wherein the first information includes an analysis output and second identification information of an analysis object of the analysis output; the transceiver module is also used to receive second information from the consumer node, wherein the second information is used for accuracy assessment, and the second information includes second identification information of the analysis object affected by the execution action, wherein the execution action is performed based on the analysis output.

[0050] In the tenth aspect, a communication device is provided, which is applied to a consumer node. The communication device can be a consumer node or a chip or system on chip in the consumer node. The communication device can implement the functions performed by the consumer node in the fifth aspect or the possible design of the fifth aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the communication device includes: a transceiver module for receiving first information, wherein the first information includes an analysis output and second identification information of an analysis object of the analysis output; a processing module for determining second information based on the first information, the second information is used for accuracy assessment, and the second information includes second identification information of the analysis object affected by the execution action, wherein the execution action is performed based on the analysis output; the transceiver module is also used to send the second information to the first network device.

[0051] In an eleventh aspect, the present application provides a communication device, comprising a processor and a transceiver, wherein the processor and the transceiver are configured to support the communication device in executing the method of any one of aspects 1 to 5. Furthermore, the communication device may further comprise a memory storing computer instructions, and the processor may execute the computer instructions to execute the method of any one of aspects 1 to 5.

[0052] In a twelfth aspect, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed, the method of any one of the first to fifth aspects is executed.

[0053] In a thirteenth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method of any one of the first to fifth aspects described above.

[0054] In a fourteenth aspect, the present application provides a chip comprising a processor and a transceiver, wherein the processor and the transceiver are used to support a communication device to execute the method of any one of the first to fifth aspects.

[0055] In a fifteenth aspect, the present application provides a communication system, comprising a first network device and a consumer node, wherein the first network device is configured to perform the method of the first aspect or the fourth aspect, and the consumer node is configured to perform the method of the second aspect or the fifth aspect. Furthermore, the communication system may also include a second network device, configured to perform the method of the third aspect.

[0056] Among them, the beneficial effects described in aspects 6 to 15 of this application can refer to the analysis of the beneficial effects of aspects 1 to 5, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] FIG1 is a schematic diagram of a process for subscribing to and issuing accuracy information provided by an embodiment of the present application;

[0058] FIG2 is a schematic diagram of the structure of a communication system provided in an embodiment of the present application;

[0059] FIG3 is a schematic diagram of the structure of another communication system provided in an embodiment of the present application;

[0060] FIG4 is a flow chart of a communication method according to an embodiment of the present application;

[0061] FIG5 is a flow chart of another communication method provided in an embodiment of the present application;

[0062] FIG6 is a schematic diagram of another process of accuracy information subscription and distribution provided in an embodiment of the present application;

[0063] FIG7 is a schematic diagram of another process of accuracy information subscription and issuance provided by an embodiment of the present application;

[0064] FIG8 is a schematic structural diagram of a communication device provided in an embodiment of the present application;

[0065] FIG9 is a schematic structural diagram of another communication device provided in an embodiment of the present application;

[0066] FIG10 is a schematic structural diagram of another communication device provided in an embodiment of the present application;

[0067] FIG11 is a schematic structural diagram of another communication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person skilled in the art will appreciate that, with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0069] It should be noted that the terms "first" and "second" in the specification, claims, and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products, or devices.

[0070] It should be understood that in the embodiments of the present application, "at least one (item)" refers to one or more, "more than one" refers to two or more, "at least two (items)" refers to two or three and more than three, and "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. It should be understood that in the embodiments of the present application, "B corresponding to A" means that B is associated with A. For example, B can be determined based on A. It should also be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information. In addition, the "connection" in the embodiments of the present application refers to various connection methods, such as direct connection or indirect connection, to achieve communication between devices, and the embodiments of the present application do not impose any limitation on this.

[0071] Unless otherwise specified, the "transmission" (transmit / transmission) appearing in the embodiments of the present application refers to bidirectional transmission, including the actions of sending and / or receiving. Specifically, the "transmission" in the embodiments of the present application includes the sending of data, the receiving of data, or the sending of data and the receiving of data. In other words, the data transmission here includes uplink and / or downlink data transmission. Data may include channels and / or signals, uplink data transmission is uplink channel and / or uplink signal transmission, and downlink data transmission is downlink channel and / or downlink signal transmission. The "network" and "system" appearing in the embodiments of the present application express the same concept, and the communication system is the communication network.

[0072] Before introducing the embodiments of the present application, some terms involved in the embodiments of the present application are explained.

[0073] Feedback information: Indicates the actions (also known as execution operations) taken by the consumer node that were influenced by the previously provided analytics content. This execution action may or may not affect the ground truth data (also known as real value data) corresponding to the analytics ID of the analytics content requested when predicting the reference, and therefore affect the accuracy monitoring of the machine learning (ML) model subscribed to based on the following parameter(s):

[0074] Analysis Feedback Information: Indicates that the consumer NF has taken an action affected by a previously provided analysis, which may or may not affect the ground truth data corresponding to the analysis ID requested when predicting the reference, and therefore affect the ML model accuracy monitoring performed by subscribing to the following parameter(s):

[0075] - The corresponding analysis ID(s) that have been used to take the execution action;

[0076] - Indicates whether executing the action affects real-valued data (if any);

[0077] - Timestamp of the execution action taken.

[0078] It should be noted that a consumer node cannot include analytics feedback information in an initial subscription request. Analytics feedback information can be included in a request to modify an existing analytics subscription.

[0079] Analytics Subscription Service Process: The Network Data Analytics Function (NWDAF) provides analytics subscription / unsubscription services for consumer nodes. Any NWDAF service consumer node (e.g., network function element, operations management and maintenance function element) can subscribe / unsubscribe from NWDAF using the Nnwdaf_analytics subscription service to receive notifications of analytics content. NWDAF service users can also use this service to modify or update existing analytics subscriptions. Any entity can use this service.

[0080] For example, in the analysis accuracy information subscription process, the consumer node subscribes to receive the analysis output and analysis accuracy information associated with the analysis ID requested by the consumer node. The process of accuracy information subscription and distribution is shown in Figure 1. Specifically:

[0081] 1. The consumer node selects an appropriate NWDAF with AnLF and subscribes to, modifies, or unsubscribes from analytics accuracy information by calling the analytics subscription request (Nnwdaf_analyticssubscription_subscribe_request). The parameters that can be included in the subscription to trigger accuracy information checking and deployment can be referenced in the prior art and will not be further described.

[0082] 2. When receiving an analytics subscription request, the NWDAF with AnLF verifies the parameters of the analytics accuracy request information (analytics accuracy monitoring and checking).

[0083] Specifically, the NWDAF with AnLF initiates analysis accuracy monitoring and generates analysis accuracy information for the analysis ID specified in the subscription, based on the parameters defined in the analysis accuracy request message. The NWDAF with AnLF calculates the analysis accuracy information. If the NWDAF with AnLF does not have sufficient necessary data, it will perform step 3b to collect ground truth data before calculating the analysis accuracy information.

[0084] The NWDAF with AnLF may have started performing analysis accuracy monitoring and analysis accuracy information generation, which were previously triggered by other consumer nodes. Upon receiving a new request from an NWDAF consumer, the NWDAF with AnLF determines whether new data collection is required to generate analysis accuracy information based on the corresponding analysis subscription.

[0085] In addition to receiving requests from consumer nodes, based on local policies, the NWDAF with AnLF can decide to start analysis accuracy monitoring and analysis accuracy information generation.

[0086] 3a. NWDAF with AnLF performs data collection of the subscribed analysis ID and generates analytics output (analytics output generation).

[0087] 3b. The NWDAF with AnLF performs data collection (e.g., ground truth data collection) to generate accuracy information for the subscribed analysis ID and generates associated analytics accuracy information. If analytics feedback information is included in step 1, the NWDAF with AnLF can consider it and determine through internal logic whether it affects the ground truth data to generate analytics accuracy information.

[0088] 4a. When the subscription in step 1 does not include analysis accuracy request information, the NWDAF with AnLF sends the analysis output to the consumer node through Nnwdaf_analytics subscription_notify(analytics output(s)) according to the parameters defined in the analysis report information included in the subscription request.

[0089] It should be noted that steps 3b and 4a can occur in any order without limitation.

[0090] 4b. The NWDAF with AnLF sends the analysis accuracy information and analysis output of the analysis ID to the consumer node through Nnwdaf_analyticssubscription_notify(accuracy information) according to the parameters defined in the analysis accuracy request information included in the subscription request.

[0091] 4c. The NWDAF with AnLF provides the accuracy information of the analysis for the analysis ID according to the parameters defined in the analysis accuracy request information included in the subscription request. When the period for providing the accuracy information of the analysis indicated in the analysis accuracy request information is different from the period for providing the analysis output indicated in the subscription request, the accuracy information of the analysis will be provided in a separate notification (e.g., Nnwdaf_analyticssubscription_notify).

[0092] 5. When low accuracy or insufficiency of an analysis ID is determined, i.e., the deviation of the analysis output of the trained ML model from the ground truth data (collected from the data producer corresponding to the analysis ID of the time request referenced by the prediction) is greater than the reporting threshold (configured locally or received in the subscription request), the NWDAF with AnLF can notify the consumer node using the stop analytics output consumption indication and the stop analytics output consumption time window. For example, the notification can be: Nnwdaf analytics subscription notify (stop analytics output consumption indication, stop analytics output consumption time window).

[0093] 6. (Optional) The consumer node can decide to stop consuming analytics outputs without unsubscribing from the analytics ID based on its own logic or based on a notification received from NWDAF with an indication to stop consuming analytics outputs. The NWDAF service consumer calls the Nnwdaf_analyticssubscription_subscribe service operation, including the subscription correlation ID, to modify an existing subscription and provides the parameter pause_analytics_consumption flag in the analytics accuracy request message.

[0094] 7. When NWDAF determines that the accuracy of the analytics ID has improved (e.g., it meets the accuracy requirements of the consumer node) or when the time window for pausing analytics consumption has expired, NWDAF notifies the consumer node of the accuracy information of the analytics ID to resume consumption of analytics outputs, thereby reactivating the existing analytics ID subscription that was previously stopped. For example, this can be done through the Nnwdaf analytics subscription notify (resume analytics output consumption indication) notification.

[0095] 8. (Optional) The NWDAF service consumer, based on its own logic, can notify NWDAF to resume providing analytics output, thereby reactivating an existing subscription to an analytics ID that was suspended by the NWDAF service consumer (step 6) or instructed by NWDAF (step 5). The NWDAF service consumer calls the Nnwdaf_analytics subscription_subscribe service operation, including the subscription association ID, to modify the existing subscription and provide the resume analytics subscription parameter in the analytics accuracy request message.

[0096] In the scenario described above, where a consumer node uses an analysis subscription request to subscribe to analysis from NWDAF with AnLF, two scenarios may occur. In the first scenario, the consumer node receives multiple analysis content notifications within a short period of time. For example, if it receives two analysis content notifications, the consumer node may perform an action based on the analysis content of the first analysis content notification, or it may perform an action based on the analysis content of both analysis content notifications. The execution of the action is then reported back to AnLF via feedback information.

[0097] In the second case, the same notification message may contain multiple analysis and prediction contents, such as abnormal behavior prediction, and multiple event identifiers (event ids). If the consumer node takes action based on the analysis output, AnLF will not be able to clearly determine which event id the consumer node has taken action on. For example, AnLF simultaneously provides the prediction of distributed denial of service (DDoS) attack and too frequent service access to the policy control function (PCF) network element, that is, the consumer node is the PCF. In response to the above situations, the PCF may execute the action of requesting the session management function (SMF) network element to release the protocol data unit (PDU) session control. In other words, AnLF will not know which of the above prediction analyses the PCF specifically requests the SMF to release the PDU session.

[0098] In the above two cases, AnLF cannot determine which analysis subscription content notification the consumer node's feedback information corresponds to, resulting in low accuracy when AnLF performs accuracy evaluation based on feedback information.

[0099] For example, in a network element load prediction scenario, AnLF's analysis output is a prediction of the network element load at a certain time period / point in the future. A consumer node subscribes to this analysis. Based on the analysis output (also referred to as the analysis prediction, prediction result, etc.) for the future time, the consumer node determines that the network element load will be high at the future time. The consumer node then takes an action (or behavior) to reduce the load on the network element, thereby reducing the future network element load. If the analysis output is accurate and the action taken by the consumer node is effective, the network element load at the future time will be lower. In other words, the true value data corresponding to the analysis output indicates a lower network element load, which is inconsistent with the analysis output. In this case, if the impact of the consumer node's action on the true value data at the future time is not considered, and the analysis output and the corresponding true value data are not excluded from the accuracy assessment, the accuracy assessment result based on the analysis output and the corresponding true value data will be inaccurate. In fact, AnLF cannot determine which analysis content notification the consumer node's feedback information corresponds to, and therefore cannot exclude such data from the accuracy assessment.

[0100] In order to solve the above technical problems, an embodiment of the present application provides a communication method. The method provided by the embodiment of the present application is described below in conjunction with the drawings in the specification.

[0101] The communication method provided in the embodiments of the present application can be applied to various communication systems, such as: sixth-generation (6G) mobile communication systems, long-term evolution (LTE) systems, fifth-generation (5G) mobile communication systems, wireless fidelity (WiFi) systems, future communication systems, or systems integrating multiple communication systems, etc., and the embodiments of the present application are not limited thereto. 5G can also be referred to as new radio (NR).

[0102] The communication method provided in the embodiment of the present application is described below using the communication system shown in FIG2 and FIG3 as an example.

[0103] Figure 2 shows a schematic diagram of the architecture of a communication system 20 provided in this application. In Figure 2 , the communication system 20 may include a network element 201 and a network element 202 (i.e., a consumer node) in communication with the network element 201. Optionally, the communication system 20 may also include a network element 203 in communication with the network element 201.

[0104] In Figure 2, network element 202 can send an analysis subscription request to network element 201 to subscribe to analysis. After receiving the analysis subscription request, network element 201 provides the analysis content and / or accuracy information to network element 202 through Nnwdaf analysis subscription notifications, etc. Network element 202 then takes preconfigured actions based on the analysis content (also known as analysis output) and provides feedback to network element 201 on the execution of these actions. Network element 201 then performs an accuracy assessment based on one or more of the following: feedback information, real-world data, etc.

[0105] Optionally, when network element 201 does not have the ability to produce or provide accuracy information or is not provided with a model for outputting accuracy information, network element 201 may send a request for generating accuracy information to a network element that has the ability to produce or provide accuracy information or a network element 203 that has a model for outputting accuracy information (the request may carry information for generating accuracy information). For example, network element 203 may be an NWDAF with a model training logic function (MTLF). After receiving the request to generate accuracy information, network element 203 generates accuracy information. Specifically, for example, a model is called to generate accuracy information and the accuracy information is fed back to network element 201.

[0106] In one possible implementation, the communication system 20 shown in FIG2 can be applied to the current 5G network, or other future networks, etc., and the embodiments of the present application do not specifically limit this. This application is described using the communication system 20 shown in FIG2 as an example of applying the network service architecture of the current 5G mobile communication system.

[0107] For example, the communication system 20 shown in FIG2 can be applied to the 5G network shown in FIG3. In FIG3, the 5G network includes a network exposure function (NEF) network element, a network storage function (NF repository function, NRF) network element, an NWDAF network element, an application function (AF) network element, a policy control function (PCF) network element, a unified data repository (UDR) network element, a unified data management (UDM) network element, an access and mobility management function (AMF) network element, a session management function (SMF) network element, a user plane function (UPF) network element, a data network (DN), a terminal, a radio access network (RAN) node, and an operation, administration and maintenance (OAM) network element. Among them, the RAN node can communicate with the AMF network element and the UPF network element, and the terminal can communicate with the AMF network element. The terminal and the RAN node can communicate using air interface technology.

[0108] In Figure 3, N1 is the interface between the terminal and the AMF network element, N2 is the interface between the RAN node and the AMF network element, N3 is the interface between the RAN node and the UPF network element, N4 is the interface between the SMF network element and the UPF network element, and N6 is the interface between the UPF network element and the DN. Namf is the service-based interface provided by the AMF network element, Nsmf is the service-based interface provided by the SMF network element, Nudm is the service-based interface provided by the UDM network element, Nnef is the service-based interface provided by the NEF network element, Nnrf is the service-based interface provided by the NRF network element, Nnwdaf is the service-based interface provided by the NWDAF network element, Naf is the service-based interface provided by the AF network element, Npcf is the service-based interface provided by the PCF network element, and Nudr is the service-based interface provided by the UDR network element.

[0109] It is understandable that if the communication system 20 shown in Figure 2 is applied to the 5G network shown in Figure 3, the network element or entity corresponding to the network element 201 in Figure 2 can be an NWDAF network element with AnLF in the 5G network architecture; the network element or entity corresponding to the network element 202 in Figure 2 can be any network element in the 5G network architecture; and the network element or entity corresponding to the network element 203 in Figure 2 can be an NWDAF network element with MTLF in the 5G network architecture. It should be understood that in specific applications, the network element 201, network element 202, or network element 203 in Figure 2 can also correspond to other network elements in the 5G network without limitation. The main functions of each network element are described in detail below.

[0110] The NWDAF network element can provide data analysis functions for the 5G core network (5G core, 5GC) network function (NF) and / or OAM network element. For example, the 5GC NF or OAM network element requests network data analysis results from the NWDAF network element. After receiving the request, the NWDAF network element collects data from relevant network elements and trains an artificial intelligence (AI) model or a machine learning (ML) model. Finally, the model is used to perform data reasoning and the reasoning results are fed back to the corresponding 5GC NF or OAM network element. It can be understood that, depending on the function, the NWDAF network element can be divided into an NWDAF that supports training (i.e., NWDAF (MTLF)) network element and an NWDAF that supports reasoning (i.e., NWDAF (AnLF)) network element. Optionally, the NWDAF (AnLF) network element can request AI model information from the NWDAF (MTLF) network element for data reasoning. If the NWDAF network element wants to obtain terminal-related data through the OAM network element, it can do so by accessing the minimization of drive tests (MDT) network element. Because MDT network elements belong to OAM network elements, they can directly interact with OAM network elements and indirectly access OAM network elements by accessing MDT network elements. For example, NWDFA network elements can subscribe to relevant terminal information from MDT network elements to collect OAM data reported by the MDT RAN node side.

[0111] NRF network elements are mainly used to provide internal / external addressing functions, etc.

[0112] The NEF network element is the 5G core network's open, standard interface. It exposes 5G network capabilities and events to the outside world and receives relevant external information. For example, the NEF network element has a member selection function that provides a list of candidate members to the AF network element. Based on 3GPP network functions, the 5G core network exposes functions and events to other systems through the NEF network element, providing both openness and system security. The NEF network element standardizes the presentation of 5G core network functions, greatly facilitating third-party access. Furthermore, the NEF network element can shield sensitive network and user information, thereby ensuring the security of the 5G core network.

[0113] The AF network element can interact with other control network elements in the 5G network on behalf of applications, including providing service QoS requirements and routing policy requirements. For example, when the AF needs to obtain information from the 5G core network, it can access the NEF network element. The AF network element can ignore the internal topology of the 5G core network and obtain the required information. This shows that the NEF network element plays a guiding role for the AF network element.

[0114] The PCF network element is mainly used to manage policy rules and user subscription information.

[0115] UDM network elements are mainly used for authentication and credit processing, user identification processing, access authorization, registration / mobility management, subscription management and short message management.

[0116] The AMF network element can support terminals with different mobility management requirements. It performs the following main tasks: non-access stratum (NAS) signaling termination, NAS signaling security, access stratum security control, core network inter-node signaling for mobility between 3GPP access networks, idle mode terminal reachability (including control and execution of paging retransmissions), registration area management, support for intra-system and inter-system mobility, access authentication, access authorization including roaming rights check, mobility management control including subscription and policy, support for network slicing and SMF selection, etc.

[0117] SMF network elements can be used for session management, terminal Internet Protocol (IP) address allocation and management, UPF network element selection and control, configuration of flow control in UPF network elements, routing of traffic to appropriate destinations, policy enforcement and QoS control, and downlink data notification.

[0118] The UPF network element is the anchor point for intra-system and inter-system mobility, the external protocol data unit (PDU) session point connected to the data network, packet routing and forwarding, packet inspection and user plane policy rule execution part, traffic usage reporting, uplink classifier, used to support routing of service flows to the data network, support branch points for multi-host PDU sessions, and QoS processing for the user plane, such as packet filtering, gating, uplink and downlink rate implementation, uplink service verification, downlink packet buffering and downlink data notification triggering.

[0119] Based on the actual needs of operators' network operations, OAM network elements can categorize network management tasks into three main categories: operations, management, and maintenance. Operations primarily involve routine network and service analysis, forecasting, planning, and configuration. Maintenance encompasses daily operational activities such as testing and fault management of the network and its services. Management primarily involves ensuring the normal operation of the network through scheduling. OAM network elements perform fault detection, path discovery, fault location, and performance monitoring, enabling network analysis, forecasting, planning, and configuration. They also perform testing and fault management of the network and its services.

[0120] DN: This can be a carrier service network, internet access, or a third-party service network. The DN can exchange information with the terminal through PDU sessions. PDU sessions can be of various types, such as Internet Protocol version 4 (IPv4) and IPv6.

[0121] A terminal is a device with wireless transceiver capabilities. The terminal can be deployed on land, including indoors, outdoors, handheld, or vehicle-mounted; it can also be deployed on the water (such as ships, etc.); it can also be deployed in the air (such as airplanes, balloons, and satellites, etc.). The terminal can also be called a terminal device, which can be user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc., or a device used to provide voice or data connectivity to users. Among them, UE includes handheld devices with wireless communication capabilities, vehicle-mounted devices (such as cars, bicycles, electric vehicles, airplanes, ships, trains, high-speed railways, etc.), wearable devices (such as smart watches, smart bracelets, pedometers, headphones, etc.) or computing devices. Exemplarily, a UE can be a mobile phone, a tablet computer, a laptop computer, a PDA, a mobile internet device (MID), a satellite terminal, or a computer with wireless transceiver capabilities. A UE may also be a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless modem, a smart point of sale (POS) machine, customer-premises equipment (CPE), an intelligent robot, a robotic arm, workshop equipment, smart home devices (e.g., refrigerators, televisions, air conditioners, electric meters, etc.), a wireless terminal in industrial control, a wireless terminal in unmanned driving, a wireless terminal in telemedicine, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, an in-vehicle terminal, a roadside unit (RSU) with terminal functions, or an aerial device (e.g., an intelligent robot, a hot air balloon, a drone, an airplane), etc. A terminal may also be other devices with terminal functions, for example, a terminal may also be a device that functions as a terminal in device-to-device (D2D) communication.

[0122] As an example and not a limitation, in this application, the terminal may be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. For example, a wearable device is not only a hardware device, but also a device that achieves powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include devices that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as devices that focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0123] In the present application, the terminal may be a terminal in an Internet of Things (IoT) system. IoT is an important component of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection. The terminal in the present application may be a terminal in machine type communication (MTC). The terminal of the present application may be an on-board module, on-board module, on-board component, on-board chip or on-board unit built into a vehicle as one or more components or units. The vehicle may implement the method of the present application through the built-in on-board module, on-board module, on-board component, on-board chip or on-board unit. The terminal of the present application may be a vehicle, such as a car. Therefore, the present application may be applied to Internet of Vehicles, such as vehicle to everything (V2X), long term evolution vehicle (LTE-V), vehicle to vehicle (V2V), etc.

[0124] In this application, the form of the terminal is not limited. The device used to implement the function of the terminal can be a terminal; it can also be a device that can support the terminal to implement the function, such as a chip system. The device can be installed in the terminal or used in conjunction with the terminal.

[0125] A RAN node can be a device with wireless transceiver functions that can help terminals achieve wireless access. The RAN node in this application can also be referred to as a node in the RAN, a RAN node or an access network device, etc. RAN nodes include but are not limited to: an evolved base station (NodeB or eNB or e-NodeB, evolutionary Node B) in LTE, an evolved base station (next generation eNB, ng-eNB) in the next generation LTE, a base station (gNodeB or gNB) in NR, a transmitting point (TP) or a transmission receiving point (TRP), a base station that is subsequently evolved by 3GPP, a next generation base station (next generation NodeB, gNB), a next generation base station in a 6G mobile communication system, a base station in a future mobile communication system, a satellite, an access node in a WiFi system, a wireless relay node, a wireless backhaul node, an integrated access and backhaul (IAB) node, a mobile switching center, and a RAN node in a non-terrestrial network (NTN) communication system, that is, it can be deployed on a high-altitude platform or a satellite, etc. A base station can be a macro base station, a micro base station, a pico base station, a small cell, a relay station, or a balloon base station. Multiple base stations can support networks using the same technology mentioned above, or they can support networks using different technologies mentioned above. A base station can include one or more co-located or non-co-located TRPs. A RAN node can also be a device that functions as a base station in D2D communication, vehicle-to-vehicle communication, drone communication, or machine communication. A RAN node can also be a radio controller in a cloud radio access network (CRAN) scenario. A RAN node can also be a centralized unit (CU), a distributed unit (DU), a CU-control plane (CP), a CU-user plane (UP), a radio unit (RU), a roadside unit (RSU) with base station functionality, a wired access gateway, or a core network element. A RAN node can also be a server, a wearable device, a machine communication device, or an in-vehicle device. For example, a RAN node in V2X technology can be an RSU. The following explanation uses a base station as an example. The multiple RAN nodes may be base stations of the same type or base stations of different types.A base station can communicate with a terminal or through a relay station. A terminal can communicate with multiple base stations using different technologies. For example, a terminal can communicate with a base station supporting an LTE network as well as a base station supporting a 5G network. It can also support dual connectivity with base stations on an LTE network and a base station on a 5G network.

[0126] In this application, the CU and DU can be set separately, or can also be included in the same network element, such as a baseband unit (BBU). The RU can be included in a radio frequency device or radio frequency unit, such as a remote radio unit (RRU), an active antenna unit (AAU), or a remote radio head (RRH). It is understandable that the CU can be divided into a RAN node in the access network, or the CU can be divided into a RAN node in the core network, without limitation here.

[0127] In different systems, CU (or CU-CP and CU-UP), DU or RU may also have different names, but those skilled in the art can understand their meanings. For example, in an open radio access network (ORAN) system, CU may also be called O-CU (open CU), DU may also be called O-DU, CU-CP may also be called O-CU-CP, CU-UP may also be called O-CU-UP, and RU may also be called O-RU. For the convenience of description, this application takes CU, CU-CP, CU-UP, DU and RU as examples for description. Any unit of CU (or CU-CP, CU-UP), DU and RU in this application can be implemented by a software module, a hardware module, or a combination of a software module and a hardware module.

[0128] It is understandable that in some scenarios, the roles of RAN nodes and terminals are relative. For example, a helicopter or drone, which is usually configured as a terminal, can also be configured as a mobile base station, and the device that accesses the RAN node via the helicopter or drone is configured as a terminal.

[0129] In this application, the form of a RAN node is not limited. The device used to implement the functions of a RAN node can be a RAN node; it can also be a device that supports the RAN node to implement the functions, such as a chip system. The device can be installed in a RAN node or used in conjunction with a RAN node.

[0130] FIG4 shows a flow chart of a communication method according to an embodiment of the present application. As shown in FIG4 , the method may include the following steps:

[0131] S410: The consumer node sends an analysis subscription request to the first network device. In response, the first network device receives the analysis subscription request.

[0132] The analysis subscription request is used to request an analysis subscription service. Exemplarily, the analysis subscription request may be an analysis subscription request (analytics subscription_subscribe) or an Nnwdaf analysis information request (analytics info_request). After receiving the analysis subscription request, the first network device may generate analysis output and accuracy information based on the analysis subscription request. The analysis subscription request includes an analysis ID. The analysis ID is used to identify the analysis service for which subscription is requested.

[0133] S420: The first network device sends first information to the consumer node. Correspondingly, the consumer node receives the first information.

[0134] Among them, the first information includes first identification information and analysis output. The first information can be a notify notification message, or the first information can also be other messages, such as a transmission control protocol (TCP) message (in this case, the first identification information can be a TCP message identifier), an application layer message (in this case, the first identification information can be an identifier of an application layer message), etc., without limitation. The first identification information is used to identify the first information, for example, the first identification information is used to identify the analysis definition notification message (Nnwdaf_analyticssubscription_notify). The analysis output is the analysis output or analysis prediction corresponding to the analysis ID. It can be understood that the analysis output is the analysis output corresponding to the analysis ID generated by the first network device. Exemplarily, the analysis output can be the analysis output generated by the first network device based on or using a model corresponding to the model information provided by the second network device. The analysis output can be one or more analysis outputs.

[0135] It will be understood that the first identification information is used to identify the analysis output. For example, the first identification information may indicate the first information; in other words, the first identification information indicates all analysis outputs in the first information. In this case, one first identification information item indicates one first information item, and the first identification information in this case may be called a notify ID. For example, the first information may include: analysis output A, analysis output B, analysis output C, and one first identification information item ID-1, where ID-1 indicates analysis output A, analysis output B, and analysis output C.

[0136] In another example, the first information includes M analysis outputs and M first identification information, each first identification information indicates an analysis output, and M is a positive integer. At this time, one first identification information indicates an analysis output in the first information (usually, one first information will include multiple analysis outputs), which improves the indication granularity. Based on this embodiment, it is possible to more finely distinguish which information is used for accuracy assessment. Exemplarily, at this time, the first information includes: analysis output A, analysis output B, analysis output C, and first identification information ID-1, first identification information ID-2, first identification information ID-3, ID-1 indicates analysis output A, ID-2 indicates analysis output B, and ID-3 indicates analysis output C.

[0137] It should be understood that the two designs of the first information described above can be used in combination, that is, the first information includes both first identification information indicating the first information and first identification information indicating a specific analysis output. For example, the first information may include: analysis output A, analysis output B, analysis output C, as well as first identification information ID-1, first identification information ID-2, first identification information ID-3, and first identification information ID-N, where ID-1 indicates analysis output A, ID-2 indicates analysis output B, ID-3 indicates analysis output C, and ID-N indicates analysis output A, analysis output B, and analysis output C simultaneously.

[0138] In another possible implementation, the first information includes the analysis output and the second identification information of the analysis object of the analysis output. Based on different analysis scenarios, the composition of the analysis object may be diverse. For example, the analysis object may be: UE (for example: user permanent identifier SUPI, general public user identifier GPSI, other identifiers of UE), UE behavior (for example: UE switching behavior, UE registration network and deregistration network behavior, etc.), location area (cell, tracking area identifier TAI,), session (PDU session, analysis subscription, transaction identifier) ​​PDU session, slice, etc. It should be understood that the above analysis object is an example. In specific implementation, the analysis object can also be other objects without limitation.

[0139] After receiving the first information, the consumer node may execute corresponding actions based on the analysis output (the corresponding relationship or judgment logic for which execution actions should be taken based on the analysis output may be pre-configured in the consumer node. The specific configuration process may refer to the existing technology and will not be described in detail here). The consumer node may also determine execution actions that may affect the true value corresponding to the analysis output, such that these execution actions may affect the accuracy assessment of the analysis or the model (used in the analysis), and determine the first identification information in the first information on which the execution action affecting the accuracy assessment is based and / or affected (and / or record the second identification information of the analysis object affected by the execution action).

[0140] After receiving the first information, the consumer node can execute the corresponding action according to the analysis output (the corresponding relationship or judgment logic of which execution actions should be taken based on the analysis output can be pre-configured in the consumer node. The specific configuration process can refer to the existing technology and will not be repeated here). The consumer node determines the execution actions that may and / or may not affect the true value corresponding to the analysis output. It can be understood that these execution actions may affect the accuracy evaluation of the analysis or the model corresponding to the analysis (or the analysis used). And determine the analysis (analysis prediction or output) on which the execution action that affects the accuracy evaluation is based and / or the analysis (analysis prediction or output) affected by the execution action, and the true values ​​corresponding to these analyses (analysis predictions or outputs) are affected by the execution action.

[0141] The consumer node determines the first identification information in the first information (and / or records the second identification information of the analysis object affected by the executed action). Specifically, the consumer node determines the identification information of the above analysis (analysis prediction or output).

[0142] S430: The consumer node sends the second information to the first network device. Correspondingly, the first network device receives the second information from the consumer node.

[0143] The second information includes the first identification information in the first information that the execution action affecting the accuracy assessment is based on and / or affects. The second information can indicate which information should be used when performing the accuracy assessment. The second information can be carried in the notification of feedback information (that is, the feedback information in this case can include: the second information and at least one of the following data: the corresponding analysis ID(s) used to take the execution action, whether the execution action will affect the real value data (if any), the timestamp of the execution action taken, etc.), or it can also be carried in other notifications, without limitation. To facilitate understanding, the second information is further explained: the second information is used to perform accuracy assessment on the analysis subscribed by the consumer node and / or perform accuracy assessment on the model corresponding to the analysis. In other words, the second information is used to generate accuracy information of the analysis and / or the model corresponding to the analysis. That is, accuracy assessment can include: generating accuracy information, accuracy assessment of the analysis subscribed by the consumer node, and accuracy assessment of the model corresponding to the analysis.

[0144] In another implementation, the second information may also include the first information on which and / or which affects the execution action that may affect the analysis output and / or the corresponding real value of the analysis (also known as prediction) (optionally, the second information may also include identification information of the corresponding first information).

[0145] The first identification information in the first information on which the action is performed may refer to the first identification information of the analysis outputs on which the action is performed. The first identification information in the first information affected by the action may refer to the first identification information of the analysis outputs corresponding to the real-value data that may be affected by the action.

[0146] For example, in a scenario where network element load is predicted, the analysis output of AnLF is a prediction of the network element load at a certain time period / point in the future. The consumer node subscribes to the analysis. The consumer node determines that the load of the network element at a future time is high based on the analysis output (also referred to as analysis prediction, prediction result, etc.) at the future time. The consumer node takes an execution action (or behavior) to reduce the load of the network element in order to reduce the future network element load. If the analysis output is accurate and the execution action taken by the consumer node is effective, the network element load at the future time will become lower. In other words, the real value data corresponding to the analysis output is a lower network element load, which is inconsistent with the analysis output. In other words, the execution action taken by the consumer node affects the accuracy assessment of the analysis output (referred to as the load analysis output) that predicts the network element load at a certain time period / point in the future. At this time, the second information may include the load analysis output and / or the first identification information of the load analysis output.

[0147] In a possible implementation, the second information includes the first information on which the execution action affecting the accuracy assessment is based and / or first identification information of the first information.

[0148] In a possible implementation, the second information includes the first information on which the execution action of the impact analysis (also called prediction) or the analysis output is based and / or first identification information of the first information.

[0149] In a possible implementation, the second information includes the first information affected by the execution action and / or first identification information of the first information.

[0150] In yet another possible implementation, the second information includes the second identification information of the analysis object affected by the execution action.

[0151] S440: The first network device performs accuracy evaluation based on the second information.

[0152] Among them, the second information may indicate which information should be used when performing the accuracy assessment. After the first network device receives the second information, it can perform an accuracy assessment based on it. Exemplarily, the first network device evaluates the accuracy of the analysis ID and / or the related model corresponding to the analysis ID based on one or more contents in the second information. It can be understood that the first network device uses the related model to provide the analysis service corresponding to the analysis ID. Specifically, the accuracy assessment may include at least one of the following: an accuracy assessment of the analysis output relative to the corresponding real value data collected (i.e., an accuracy assessment of the analysis), an accuracy assessment of the model that generates the analysis output based on the accuracy information (for example, an assessment of the robustness of the model, etc.), and its specific assessment process can refer to the existing technology and will not be repeated here. It should be understood that the above-mentioned accuracy assessment is an example. In specific implementation, other dimensions of assessment may be performed, such as: model robustness, security, model energy consumption, etc., which are not limited in this specification.

[0153] When conducting an accuracy assessment, data such as the accuracy information corresponding to the first identification information in the first information on which the execution action affecting the accuracy assessment is based and / or affected can be excluded or corrected and then used in the accuracy assessment to achieve the purpose of improving the accuracy of the accuracy assessment, without restriction.

[0154] In one possible implementation, the first information includes first identification information and analysis output, and the second information includes first identification information (referred to as target identification information, which may be one or more) in the first information that is based on and / or affected by the execution action affecting the accuracy assessment. In this implementation, the first network device first determines the analysis output corresponding to the target identification information in the first information (referred to as the target analysis output), then excludes or corrects the target analysis output and corresponding accuracy information before performing the accuracy assessment.

[0155] Specifically, refer to the two possible designs of the first identification information described in S420. When determining the analysis output corresponding to the target identification information in the first information, if the first identification information indicates the first information, the analysis output in the first information corresponding to the target identification information is determined as the target identification information. If the first identification information indicates the analysis output in the first information, the analysis output corresponding to the target identification information is determined as the target identification information.

[0156] In another possible implementation, the first information includes the analysis output and second identification information of an analysis target of the analysis output, and the second information includes the second identification information of the analysis target affected by the execution action (referred to as target identification information). In this implementation, the first network device first determines the analysis target corresponding to the target identification information in the first information (referred to as the target analysis target), then excludes or corrects the analysis output and corresponding accuracy information corresponding to the target analysis target before performing the accuracy assessment.

[0157] In an embodiment of the present application, the first network device can determine, based on the identification information in the second information, the analysis output based on which the consumer node took the execution action. It can also determine the analysis output based on and / or affected by the execution action that affects the accuracy assessment (or the analysis output corresponding to the analysis object affected by the execution action), thereby ensuring the reliability of the data used for the accuracy assessment (for example, excluding the affected analysis output and corresponding accuracy information from the accuracy assessment, or correcting the affected analysis output and corresponding accuracy information before using it for the accuracy assessment), thereby improving the accuracy of the accuracy assessment.

[0158] Furthermore, because the identification information in the first and second information can serve as an indicator, the identification information can be used to distinguish the correspondence between the feedback information and the analysis output. This eliminates the need for feedback information to be reported in a one-to-one correspondence with the analysis output. In other words, the consumer node can report multiple pieces of feedback information at once. This facilitates more efficient information statistics, reduces unnecessary messages between the consumer node and the first network device, and thus reduces communication overhead.

[0159] In one embodiment, as shown in FIG5 , the method may further include:

[0160] S510: The first network device sends third information to the second network device. Correspondingly, the second network device receives the third information.

[0161] Among them, the third information is used to determine the data used for the accuracy assessment. In S410, the first network device can generate accuracy information based on the analysis subscription request. In fact, the first network device may not have the ability to generate accuracy information, and cannot generate accuracy information. It needs to obtain accuracy information from the second network device. Exemplarily, the second network device may be the producer of the model used by the first network device to provide the analysis service, or the second network device provides the model or model information used by the analysis service to the first network device (for example, including one or more of the following: model identification information, model address, model file content, etc.). The first network device may not be provided with a model for outputting accuracy information, and it is necessary to obtain the model for outputting accuracy information from the second network device and deploy it on the second network device. In other words, in this case, the first network device sends the third information required to generate accuracy information to the second network device, and the second network device generates the accuracy information.

[0162] In other words, the third information can be used for model training or model retraining. Exemplarily, the second network device determines that the model needs to be retrained based on the third information.

[0163] The content of the third information can be flexibly set. For example, the third information can include one or more of the following: the first information, the second information, the real value data used to generate the accuracy information (the second network device may also have the ability to collect real value data and other data. If the second network device has cached real value data and other data, then the third information does not need to carry repeated content), etc. In this scenario, the first network device is not sure which analysis outputs (i.e., target analysis outputs) are affected by the execution action of the consumer node, but is determined by the second network device (the determination process can refer to the same principle as the determination process in S440, which will not be repeated here). After the second network device determines the target analysis output, it can exclude / correct the target analysis output and the corresponding real value data, and then generate the accuracy information corresponding to the non-target analysis output and / or the corrected target analysis output (i.e., the accuracy information used for the accuracy evaluation).

[0164] In another possible implementation, the third information may include one or more of the following: analysis outputs used for the accuracy assessment, real value data corresponding to the analysis outputs used for the accuracy assessment, etc. In this scenario, the first network device first determines the analysis outputs used for the accuracy assessment (the determination process can be referred to the description in S440 and will not be repeated here), and uses the real value data required for determining the accuracy information and corresponding to the analysis outputs used for the accuracy assessment as the third information. In this implementation, the second network device no longer needs to determine which are the analysis outputs used for the accuracy assessment, and the third information carries less content and has less communication overhead. After receiving the third information, the second network device generates corresponding accuracy information based on the analysis outputs used for the accuracy assessment in the third information.

[0165] In another possible implementation, the third information may include the first information and / or true value data corresponding to the analysis output of the first information. After receiving the third information, the second network device generates full-scale accuracy information based on the third information and sends the accuracy information and the corresponding identification information in the first information to the first network device. The first network device then determines which data to use for accuracy assessment based on the received full-scale accuracy information, the corresponding identification information, and the second information.

[0166] S520: The second network device sends data for accuracy evaluation to the first network device. Correspondingly, the first network device receives accuracy information for accuracy evaluation.

[0167] S530: The first network device performs accuracy evaluation based on the data used for accuracy evaluation.

[0168] After receiving the accuracy information for accuracy assessment, the first network device may perform accuracy assessment based on the accuracy information. The specific assessment process may refer to the description of S440 and will not be described in detail.

[0169] In an embodiment of the present application, based on the identification information in the second information (which may be the first identification information or the second identification information), it is possible to determine based on which analysis output the consumer node took the execution action, and determine the analysis output based on and / or affected by the execution action that is not affected by the accuracy assessment (or the analysis output corresponding to the analysis object that is not affected by the execution action) as the target analysis output. The second network device sends the accuracy information corresponding to the target analysis output to the first network device. The first network device performs an accuracy assessment based on the accuracy information corresponding to the target analysis output, thereby ensuring the reliability of the data used for the accuracy assessment (for example, excluding the affected analysis output and corresponding accuracy information from the accuracy assessment, or correcting the affected analysis output and corresponding accuracy information before using it for the accuracy assessment), thereby improving the accuracy of the accuracy assessment.

[0170] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the execution logic of each step. To facilitate understanding, the solution provided by the embodiment of the present application is further introduced below with reference to specific examples.

[0171] FIG6 shows the process of accuracy information subscription and issuance provided by an embodiment of the present application. As shown in FIG6 , the process shown in FIG6 modifies the information content carried in the notifications in steps 4a and 6 compared to the current process of accuracy information subscription and issuance, resulting in steps 4a' and 6', and adaptively modulating the subsequent process of accuracy evaluation. It should be understood that, in a specific implementation, the modification of the information content carried in the notifications in steps 4a and 6 can also be migrated to the notifications of other steps to achieve the same purpose without limitation. In addition, to avoid redundancy, the embodiment of the present application no longer describes the steps with the same functions as those in the current process of accuracy information subscription and issuance, but focuses on steps 4a' to 8a, specifically:

[0172] The notification in step 4a' includes the analysis output and identification information. In other words, the notification in step 4a' includes the first information in the above embodiment.

[0173] The identification information may include the first identification information and / or the second identification information. The description of the first identification information and the second identification information can refer to the above embodiments and will not be repeated here.

[0174] Step 5: The consumer node takes relevant execution actions based on the analysis output and internal logic.

[0175] The notification in step 6' includes identification information of the analysis output (or analysis object) affected by the execution action. In other words, the notification in step 6' includes the second information in the above embodiment. The description of the second information can refer to the above embodiment and will not be repeated here.

[0176] In step 7a, the first network device performs an accuracy evaluation based on the second information to obtain an accuracy evaluation result (also referred to as accuracy information).

[0177] In step 8a, the first network device sends the accuracy evaluation result obtained in step 7a to the consumer node.

[0178] In another example, the solution provided by the embodiment of the present application is further introduced in conjunction with the implementation scenario of S520-S530. As shown in Figure 7, Figure 7 shows another process of accuracy information subscription and distribution provided by the embodiment of the present application, including:

[0179] 1. The consumer node sends an analysis subscription request to the first network device.

[0180] 2. The first network device instructs the second network device to call the model through a model subscription request.

[0181] 3. The second network device sends a model subscription notification message to the first network device, indicating that the model subscription notification has been received.

[0182] 4. The first network device sends an ML model monitor registration message (ML model monitor register) to the second network device.

[0183] 5. The second network device sends a model monitoring subscription notification message (ML model monitor subscribe) to the first network device.

[0184] 6. The first network device uses Nnwdaf_analytics subscription notify to send the first information to the consumer node.

[0185] 7. The consumer node performs actions based on the analysis output.

[0186] 8. The consumer node uses Nnwdaf_analytics subscription_subscribe to send feedback information (ie, second information) to the first network device.

[0187] The description of the second information can refer to the aforementioned embodiment and will not be repeated here.

[0188] 9. The first network device determines the data for evaluating the accuracy of the model (i.e., the third information) and sends it to the second network device. Exemplarily, the data can be organized in the form of data tuples, and each data tuple includes: [optional notify ID, analysis output, corresponding analysis input, feedback information corresponding to the notify ID, and optional corresponding real value data].

[0189] Here, referring to the description of S510, the content of the third information can be flexibly designed and will not be described in detail.

[0190] 10. The second network device collects data that generates accuracy information, calculates accuracy information, and generates an accuracy report. The second network device performs additional processing on the data containing the feedback information corresponding to the notify ID based on its internal logic. For example, the second network device may exclude potentially affected analysis outputs from its evaluation.

[0191] 11. The second network device sends an accuracy report to the first network device.

[0192] 12. The first network device performs an accuracy assessment based on the accuracy report to obtain an accuracy assessment result.

[0193] 13. The first network device sends the accuracy evaluation result to the consumer node.

[0194] In combination with the above examples, it is not difficult to see that the embodiment of the present application can determine, based on the identification information in the first information and the second information, based on which analysis output the consumer node took the execution action, and determine the analysis output (or the analysis output corresponding to the analysis object affected by the execution action) on which the execution action that affects the accuracy assessment is based and / or affected, thereby ensuring the reliability of the data used for accuracy assessment (for example, excluding the affected analysis output and corresponding accuracy information and other data from the accuracy assessment, or correcting the affected analysis output and corresponding accuracy information and other data before using them for accuracy assessment), thereby improving the accuracy of the accuracy assessment.

[0195] Furthermore, because the identification information in the first and second information can serve as an indicator, the identification information can be used to distinguish the correspondence between the feedback information and the analysis output. This eliminates the need for feedback information to be reported in a one-to-one correspondence with the analysis output. In other words, the consumer node can report multiple pieces of feedback information at once. This facilitates more efficient information statistics, reduces unnecessary messages between the consumer node and the first network device, and thus reduces communication overhead.

[0196] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the execution logic of each step. It is understandable that each node, such as a consumer node, includes a hardware structure and / or software module corresponding to the execution of each function in order to implement the above functions. Those skilled in the art should easily appreciate that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the method of the embodiment of the present application can be implemented in the form of hardware, software, or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0197] The embodiment of the present application can divide the functional modules of the consumer node according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0198] In a specific implementation, each network element shown in this application, such as a consumer node, may adopt the structure shown in Figure 8 or include the components shown in Figure 8. Figure 8 is a schematic diagram of the structure of a communication device provided in an embodiment of this application. When the communication device has the functions of a consumer node described in an embodiment of this application, the communication device can be a consumer node or a chip or system-on-chip in the consumer node. When the communication device has the functions of a network device described in an embodiment of this application, the communication device can be a network device or a chip or system-on-chip in the network device.

[0199] As shown in Figure 8 , the communication device may include a processor 801, a communication line 802, a transceiver 803, and a memory 804. The processor 801, the memory 804, and the transceiver 803 may be connected via the communication line 802. In one example, the processor 801 may include one or more CPUs, such as CPU0 and CPU1 in Figure 8 .

[0200] As an optional implementation manner, the communication device includes multiple processors. For example, in addition to the processor 801 in FIG. 8 , it may also include a processor 807 .

[0201] The processor 801 may be a central processing unit (CPU), a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 801 may also be other devices with processing capabilities, such as circuits, devices, or software modules.

[0202] The communication line 802 is used to transmit information between the various components included in the communication device.

[0203] Transceiver 803 is used to communicate with other devices or other communication networks. Such other communication networks may be Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc. Transceiver 803 may be an interface circuit, a pin, a radio frequency module, a transceiver, or any other device capable of communication.

[0204] Furthermore, the communication device may further include a memory 804. The memory 804 is configured to store instructions, wherein the instructions may be computer programs.

[0205] The memory 804 may be a read-only memory (ROM) or other types of static storage devices capable of storing static information and / or instructions, a random access memory (RAM) or other types of dynamic storage devices capable of storing information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disk storage, magnetic disk storage media, or other magnetic storage devices. Optical disc storage includes compact discs, laser discs, optical discs, digital versatile discs, or Blu-ray discs, etc.

[0206] It should be noted that the memory 804 can exist independently of the processor 801 or can be integrated with the processor 801. The memory 804 can be used to store instructions, program code, or some data. The memory 804 can be located within the communication device or outside the communication device, without limitation. When the processor 801 executes the instructions stored in the memory 804, the method provided in the embodiment of the present application can be implemented.

[0207] As an optional implementation, the communication apparatus further includes an output device 805 and an input device 806. For example, the input device 806 is a keyboard, a mouse, a microphone, a joystick, or the like, and the output device 805 is a display screen, a speaker, or the like.

[0208] It should be noted that the communication device may be a desktop computer, a portable computer, a network server, a mobile phone, a tablet computer, a wireless terminal, an embedded device, a chip system, or a device having a structure similar to that shown in FIG8 . Furthermore, the component structure shown in FIG8 does not limit the communication device. In addition to the components shown in FIG8 , the communication device may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0209] In the embodiment of the present application, the chip system can be composed of chips, or can include chips and other discrete devices.

[0210] Figure 9 shows a structural diagram of a communication device 900, which is applied to a consumer node. Each module in the device shown in Figure 9 has the function of implementing the corresponding steps in the method embodiment and can achieve its corresponding technical effect. The beneficial effects of the corresponding steps executed by each module can be referred to the description of the corresponding steps and will not be repeated here. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The communication device can be a consumer node or a chip or system on chip in a consumer node. For example, the communication device includes:

[0211] The transceiver module 910 is used to receive first information, wherein the first information includes first identification information and analysis output; the processing module 920 is used to determine second information based on the first information, the second information is used for accuracy assessment, and the second information includes the first identification information in the first information on which the execution action affecting the accuracy assessment is based and / or affected, wherein the execution action is performed based on the analysis output.

[0212] Figure 10 shows a structural diagram of a communication device 100, which is applied to a first network device. Each module in the device shown in Figure 10 has the function of implementing the corresponding steps in the embodiment of the method and can achieve its corresponding technical effect. The beneficial effects corresponding to the steps executed by each module can be referred to the description of the corresponding steps and will not be repeated here. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The communication device can be a first network device or a chip or system on chip in the first network device. For example, the communication device includes:

[0213] The transceiver module 101 is used to send first information to a consumer node, wherein the first information includes first identification information and analysis output; the transceiver module 101 is also used to receive second information from the consumer node, wherein the second information is used for accuracy assessment, and the second information includes first identification information in the first information on which an execution action affecting the accuracy assessment is based and / or affected, wherein the execution action is performed based on the analysis output; the processing module 102 is used to determine the target analysis output that is not affected by the execution action based on the second information; and perform accuracy assessment based on the accuracy information corresponding to the target analysis output.

[0214] Figure 11 shows a structural diagram of a communication device 110, which is applied to the second network device. Each module in the device shown in Figure 11 has the function of implementing the corresponding steps in the method embodiment and can achieve its corresponding technical effect. The beneficial effects of the corresponding steps executed by each module can be referred to the description of the corresponding steps and will not be repeated here. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The communication device can be a second network device or a chip or system on chip in the second network device. For example, the communication device includes:

[0215] The transceiver module 111 is used to receive third information from the first network device, wherein the third information is used to determine the accuracy information of the target analysis output, the target analysis output is the analysis output determined based on the second information and not affected by the execution action, the second information includes at least one of the following: first identification information in the first information on which the execution action affecting the accuracy assessment is based and / or affected, the first information includes the first identification information and the analysis output, the execution action is performed based on the analysis output, and the third information is used to determine the accuracy information of the analysis output; the processing module 112 is used to determine the target analysis output that is not affected by the execution action based on the third information; the processing module 112 is also used to generate the accuracy information of the target analysis output; the transceiver module 111 is also used to send the accuracy information of the target analysis output to the first network device.

[0216] The present application also provides a communication system comprising a first network device and a consumer node. The first network device is configured to execute the steps corresponding to the modules of the communication device 100, and the consumer node is configured to execute the steps corresponding to the modules of the communication device 900. Furthermore, the communication system may also include a second network device configured to execute the steps corresponding to the modules of the communication device 110.

[0217] The embodiments of the present application also provide a computer-readable storage medium. All or part of the processes in the above-mentioned method embodiments can be completed by a computer program to instruct the relevant hardware, and the program can be stored in the above-mentioned computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The computer-readable storage medium can be a terminal device of any of the above-mentioned embodiments, such as: an internal storage unit including a data sending end and / or a data receiving end, such as a hard disk or memory of the terminal device. The above-mentioned computer-readable storage medium can also be an external storage device of the above-mentioned terminal device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the above-mentioned terminal device. Furthermore, the above-mentioned computer-readable storage medium can also include both the internal storage unit of the above-mentioned terminal device and an external storage device. The above-mentioned computer-readable storage medium is used to store the above-mentioned computer program and other programs and data required by the above-mentioned terminal device. The above-mentioned computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0218] The present application also provides a computer instruction. All or part of the process in the above method embodiment can be completed by the computer instruction to instruct the relevant hardware (such as a computer, processor, network device, and terminal, etc.). The program can be stored in the above computer-readable storage medium.

[0219] The present application also provides a chip system. This chip system can be composed of a chip, or can include a chip and other discrete devices, without limitation. The chip system includes a processor and a transceiver. All or part of the processes in the above method embodiments can be completed by the chip system. For example, the chip system can be used to implement the functions performed by the consumer node in the above method embodiments, or to implement the functions performed by the network device in the above method embodiments.

[0220] In one possible design, the above-mentioned chip system also includes a memory, which is used to store program instructions and / or data. When the chip system is running, the processor executes the program instructions stored in the memory to enable the chip system to perform the functions performed by the consumer node in the above-mentioned method embodiment or the functions performed by the network device in the above-mentioned method embodiment.

[0221] In the embodiments of the present application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0222] In an embodiment of the present application, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or a volatile memory (volatile memory), such as a random-access memory (RAM). The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in an embodiment of the present application may also be a circuit or any other device that can implement a storage function, for storing instructions and / or data.

[0223] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0224] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0225] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0226] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device, such as a single-chip microcomputer, a chip, etc., or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0227] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A communication method, characterized in that: The method is applied to a first network device, comprising: Sending first information to a consumer node, wherein the first information includes first identification information and analysis output; Receive second information from a consumer node, wherein the second information is used for accuracy assessment, and the second information includes the first identification information in the first information on which an execution action affecting the accuracy assessment is based and / or affects, wherein the execution action is performed based on the analysis output.

2. The communication method according to claim 1, characterized in that: The first identification information indicates the first information, or the first identification information indicates the analysis output.

3. The communication method according to claim 1 or 2, characterized in that: The method further comprises: determining an analysis output for the accuracy assessment based on the second information; An accuracy assessment is performed based on the analysis output for said accuracy assessment.

4. The communication method according to claim 1 or 2, characterized in that: The method further comprises: Sending third information to the second network device, wherein the third information is used to determine data for the accuracy assessment, the third information includes the first information and / or the second information, or the third information includes an analysis output for the accuracy assessment, and the analysis output for the accuracy assessment is determined according to the second information; receiving the data for the accuracy assessment from the second network device; An accuracy assessment is performed based on the data used for the accuracy assessment.

5. The communication method according to claim 4, characterized in that: The first network device is a network data analysis function network element with analysis logic function, and the second network device is a network data analysis function network element with model training logic function.

6. The communication method according to any one of claims 1 to 5, characterized in that: The method further comprises: Before sending the first information to the consumer node, an analysis subscription request is received from the consumer node.

7. A communication method, characterized in that: The method is applied to a consumer node, comprising: Receiving first information, wherein the first information includes first identification information and analysis output; determining second information according to the first information, the second information being used for accuracy assessment, the second information comprising the first identification information in the first information on which an execution action affecting the accuracy assessment is based and / or affected, wherein the execution action is performed based on the analysis output; The second information is sent to the first network device.

8. The communication method according to claim 7, characterized in that: The first identification information indicates the first information, or the first identification information indicates the analysis output.

9. The communication method according to claim 7 or 8, characterized in that: The method further comprises: Before receiving the first information, an analysis subscription request is sent.

10. A communication method, characterized in that: The method is applied to a second network device, comprising: Receiving third information from a first network device, wherein the third information is used to determine data for accuracy assessment, the third information includes the first information and the second information, or the third information includes an analysis output for the accuracy assessment, the analysis output for the accuracy assessment is determined according to the second information; the second information includes first identification information in the first information on which an execution action affecting the accuracy assessment is based and / or affected, the first information includes the first identification information and the analysis output, and the execution action is performed based on the analysis output; determining the data for the accuracy assessment based on the third information; The data for the accuracy assessment is sent to the first network device.

11. A communication method, characterized in that: The method is applied to a first network device, comprising: Sending first information to a consumer node, wherein the first information includes an analysis output and second identification information of an analysis object of the analysis output; Second information is received from a consumer node, wherein the second information is used for accuracy evaluation, and the second information includes the second identification information of an analysis object affected by an execution action, wherein the execution action is performed based on the analysis output.

12. A communication method, characterized in that: The method is applied to a consumer node, comprising: receiving first information, wherein the first information includes an analysis output and second identification information of an analysis object of the analysis output; determining second information according to the first information, the second information being used for accuracy assessment, the second information comprising the second identification information of the analysis object affected by the execution action, wherein the execution action is executed based on the analysis output; The second information is sent to the first network device.

13. A communication device, characterized in that: The device is applied to a first network device, and includes: A transceiver module, configured to send first information to a consumer node, wherein the first information includes first identification information and analysis output; The transceiver module is also used to receive second information from the consumer node, wherein the second information is used for accuracy assessment, and the second information includes the first identification information in the first information on which the execution action affecting the accuracy assessment is based and / or affects, wherein the execution action is performed based on the analysis output.

14. The communication device according to claim 13, characterized in that: The first identification information indicates the first information, or the first identification information indicates the analysis output.

15. The communication device according to claim 13 or 14, characterized in that: The device also includes a processing module, configured to: determining an analysis output for the accuracy assessment based on the second information; An accuracy assessment is performed based on the analysis output for said accuracy assessment.

16. The communication device according to claim 13 or 14, characterized in that: The device also includes a processing module; The transceiver module is further used to send third information to the second network device, wherein the third information is used to determine data for the accuracy evaluation, and the third information includes the first information and the second information, or the third information includes an analysis output for the accuracy evaluation, and the analysis output for the accuracy evaluation is determined according to the second information; The transceiver module is further used to receive the accuracy information for the accuracy assessment from the second network device; The processing module is used to perform accuracy assessment based on the data used for the accuracy assessment.

17. The communication device according to claim 16, characterized in that: The first network device is a network data analysis function network element with analysis logic function, and the second network device is a network data analysis function network element with model training logic function.

18. The communication device according to any one of claims 13 to 17, characterized in that: The transceiver module is further configured to receive an analysis subscription request from the consumer node before sending the first information to the consumer node.

19. A communication device, characterized in that: The device is applied to a consumer node, and includes: A transceiver module, configured to receive first information, wherein the first information includes first identification information and analysis output; a processing module, configured to determine second information based on the first information, the second information being used for accuracy assessment, the second information comprising the first identification information in the first information on which an execution action affecting the accuracy assessment is based and / or affected, wherein the execution action is performed based on the analysis output; The transceiver module is further used to send the second information to the first network device.

20. The communication device according to claim 19, characterized in that The first identification information indicates the first information, or the first identification information indicates the analysis output.

21. The communication device according to claim 19 or 20, characterized in that: The transceiver module is further configured to send an analysis subscription request before receiving the first information.

22. A communication device, characterized in that: The device is applied to a second network device, comprising: A transceiver module, configured to receive third information from a first network device, wherein the third information is used to determine data for accuracy assessment, the third information includes the first information and the second information, or the third information includes an analysis output for the accuracy assessment, the analysis output for the accuracy assessment is determined based on the second information; the second information includes first identification information in the first information on which an execution action affecting the accuracy assessment is based and / or affected, the first information includes the first identification information and the analysis output, and the execution action is executed based on the analysis output; a processing module, configured to determine the data used for the accuracy assessment according to the third information; The transceiver module is further used to send the data used for the accuracy assessment to the first network device.

23. A communication device, characterized in that: The device is applied to a first network device, and includes: The transceiver module is used to send first information to the consumer node, wherein the first information includes the analysis output and the analysis output. second identification information of the analysis object output by the analysis; The transceiver module is further used to receive second information from the consumer node, wherein the second information is used for accuracy assessment, and the second information includes the second identification information of the analysis object affected by the execution action, wherein the execution action is performed based on the analysis output.

24. A communication device, characterized in that: The device is applied to a consumer node, and includes: A transceiver module, configured to receive first information, wherein the first information includes an analysis output and second identification information of an analysis object of the analysis output; a processing module, configured to determine second information according to the first information, the second information being used for accuracy assessment, the second information comprising the second identification information of the analysis object affected by the execution action, wherein the execution action is executed based on the analysis output; The transceiver module is further used to send the second information to the first network device.

25. A communication device, characterized in that: The communication device comprises a processor and a transceiver, and the processor and the transceiver are used to support the communication device to execute the method according to any one of claims 1-12.

26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed, the method according to any one of claims 1 to 12 is executed.

27. A communication system, characterized in that: The communication system comprises a first network device and a consumer node, wherein the first network device is used to execute the method according to any one of claims 1-6 and 11, and the consumer node is used to execute the method according to any one of claims 7-9 and 12.

Images