Communication method and device

Abstract

The invention discloses a communication method and device, and relates to the field of communication. The method comprises the steps that first network equipment sends first information to a consumer node, and the first information comprises first identification information and analysis output; the first network device receives second information from the consumer node, the second information is used for accuracy evaluation, the second information comprises first identification information in the first information on the basis of an execution action influencing the accuracy evaluation and / or influenced by the execution action, and the execution action is executed based on the analysis output; and the accuracy of accuracy evaluation in the analysis subscription scene is improved.

Description

Technical Field

[0001] This application relates to the field of communications, and in particular, to a communication method and apparatus. Background Art

[0002] Currently, a consumer node uses an Nnwdaf analytics subscription_subscribe / analytics info_request to subscribe to analytics from a network data analytics function containing an analytics logical function (Anlf), and then the Anlf provides analytics content to the consumer node through an Nnwdaf analytics subscription_notify / analytics info_request response. Subsequently, the consumer node takes pre-configured actions based on the analytics content (also known as analytics output), and feeds back the situation of executing the actions to the Anlf through feedback information. The Anlf then performs an accuracy assessment based on the feedback information.

[0003] Currently, the accuracy of the Anlf in performing an accuracy assessment based on feedback information is relatively low. Summary of the Invention

[0004] This application provides a communication method and apparatus for improving the accuracy of accuracy assessment in an analytics subscription scenario.

[0005] To achieve the above object, this application adopts the following technical solutions:

[0006] In a first aspect, a communication method is provided, which is applied to a first network device. The execution subject of this method can be the first network device, or a component or device applied to the first network device (such as a processor, a chip, or a chip system, etc.), or a logical module or software that can implement all or part of the functions of the first network device. The method includes: sending first information to a consumer node, where the first information includes first identification information and an analysis output; receiving second information from the consumer node, where the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which the execution action for accuracy evaluation is based and / or which is affected by the execution action, and the execution action is performed based on the analysis output. For the purpose of helping understanding, a further description of the second information is as follows: the second information is used for accuracy evaluation of the analysis subscribed by the consumer node and / or accuracy evaluation of the model corresponding to the analysis. In other words, the second information is used to generate accuracy information of the analysis and / or the model corresponding to the analysis. That is to say, the accuracy evaluation may include one or more of: generating accuracy information, accuracy evaluation of the analysis subscribed by the consumer node, and accuracy evaluation of the model corresponding to the analysis.

[0007] In another implementation, the second information may further include the first information on which the execution action for the corresponding true value that may affect the analysis output and / or the analysis (also known as prediction) is based and / or which is affected by the execution action (optionally, the second information may further include the identification information of the corresponding first information). In another implementation, the second information is used for accuracy evaluation of the corresponding analysis or for determining the accuracy information of the analysis, and the second information includes the first identification information in the first information on which the execution action for the true value corresponding to the analysis output is based and / or which is affected by the execution action.

[0008] In the first aspect, based on the identification information in the second information, the first network device can determine which analysis output the consumer node has taken the execution action according to. And determine the analysis output on which the execution action for accuracy evaluation is based and / or which is affected by the execution action, thereby ensuring the reliability of the data used for accuracy evaluation (for example, excluding the affected analysis output and corresponding accuracy information and other data from the accuracy evaluation, or correcting the affected analysis output and corresponding accuracy information and other data before using them for accuracy evaluation), and being able to improve the accuracy of the accuracy evaluation.

[0009] In addition, since the identification information in the first information and the second information can play an indicative role, using the identification information can distinguish the correspondence between the feedback information and the analysis output, which enables the feedback information not to be reported one-to-one corresponding to the analysis output. In other words, the consumer node can report multiple feedback information at one time. In this way, it is more conducive to efficient information statistics, reduces unnecessary messages between the consumer node and the first network device, and thus reduces the communication overhead.

[0010] In a possible implementation, the first identification information indicates the first information. Alternatively, the first identification information indicates the analysis output. In other words, at this time, the first information includes M items of analysis output and M items of first identification information, and each item of first identification information indicates one item of analysis output, where M is a positive integer.

[0011] In this implementation, two designs of the first identification information are provided, which can indicate analysis outputs with different granularities.

[0012] In a possible implementation, the method further includes: determining the analysis output for accuracy evaluation according to the second information; performing accuracy evaluation according to the analysis output for accuracy evaluation.

[0013] In this implementation, determining the more reliable accuracy information for accuracy evaluation according to the second information improves the accuracy of the accuracy evaluation.

[0014] In a possible implementation, the method further includes: sending third information to a second network device, where the third information is used to determine the data for accuracy evaluation, the third information includes the first information and the second information, or the third information includes the analysis output for accuracy evaluation, and the analysis output for accuracy evaluation is determined according to the second information; receiving the accuracy information for accuracy evaluation from the second network device; performing accuracy evaluation according to the data for accuracy evaluation.

[0015] In this implementation, the second network device sends the data for accuracy evaluation to the first network device, and the first network device performs accuracy evaluation according to the data for accuracy evaluation, ensuring the reliability of the data for accuracy evaluation (for example, excluding data such as the affected analysis output and the corresponding accuracy information from the accuracy evaluation, or correcting the affected analysis output and the corresponding accuracy information and then using them for accuracy evaluation), which can improve the accuracy of the accuracy evaluation.

[0016] In a possible implementation, the first network device is a network data analysis function network element with analysis logic functions, and the second network device is a network data analysis function network element with model training logic functions.

[0017] In a possible implementation, the method further includes:

[0018] Before sending the first information to the consumer node, receiving an analysis subscription request from the consumer node.

[0019] Second aspect, a communication method is provided, which is applied to a consumer node. The execution subject of this method can be the consumer node, or a component or device applied to the consumer node (such as a processor, a chip, or a chip system, etc.), or a logic module or software that can implement all or part of the functions of the consumer node. The method includes: receiving first information, where the first information includes first identification information and an analysis output; determining second information according to the first information, the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which the execution action for accuracy evaluation is based and / or affected, where the execution action is executed based on the analysis output; sending the second information to a first network device.

[0020] In the second aspect, the consumer node sends the second information to the first network device, which can be used by the first network device to determine, according to the identification information in the second information, which analysis output the consumer node has taken an execution action based on. And determine the analysis output on which the execution action for accuracy evaluation is based and / or affected (or the analysis output corresponding to the analysis object affected by the execution action), thereby ensuring the reliability of the data used for accuracy evaluation (for example, excluding the affected analysis output and corresponding accuracy information and other data from the accuracy evaluation, or correcting the affected analysis output and corresponding accuracy information and other data and then using them for accuracy evaluation), which can improve the accuracy of the accuracy evaluation.

[0021] In a possible implementation, the first identification information indicates the first information, or the first identification information indicates the analysis output.

[0022] In a possible implementation, the method further includes:

[0023] Before receiving the first information, sending an analysis subscription request.

[0024] Third aspect, a communication method is provided, which is applied to a second network device. The execution subject of this method can be the second network device, or a component or device applied to the second network device (such as a processor, a chip, or a chip system, etc.), or a logic module or software that can implement all or part of the functions of the second network device. The method includes: receiving third information from a first network device, where the third information is used to determine data for accuracy evaluation, the third information includes the first information and the second information, or the third information includes an analysis output for accuracy evaluation, and the analysis output for accuracy evaluation is determined according to the second information; the second information includes the first identification information in the first information on which the execution action for accuracy evaluation is based and / or affected, the first information includes the first identification information and an analysis output, and the execution action is executed based on the analysis output; determining data for accuracy evaluation according to the third information; sending the data for accuracy evaluation to the first network device.

[0025] In a third aspect, the second network device determines accuracy information for accuracy evaluation (the reliability of this accuracy information is relatively high. For example, it is data obtained by excluding data such as corresponding accuracy information of the analysis output to be affected, or data obtained by correcting data such as corresponding accuracy information of the analysis output to be affected), and sends it to the first network device, which can improve the accuracy of the accuracy evaluation.

[0026] In a fourth aspect, a communication method is provided, which is applied to the first network device. The execution subject of this method can be the first network device, or a component or device applied to the first network device (such as a processor, a chip, or a chip system, etc.), or a logic module or software that can implement all or part of the functions of the first network device. The method includes: sending first information to a consumer node, where the first information includes an analysis output and second identification information of an analysis object of the analysis output; receiving second information from the consumer node, where the second information is used for accuracy evaluation, and the second information includes second identification information of an analysis object affected by an execution action, and the execution action is executed based on the analysis output.

[0027] In the fourth aspect, based on the identification information in the second information, the first network device can determine which analysis output the consumer node has taken an execution action according to. And determine the analysis output corresponding to the analysis object affected by the execution action, thereby ensuring the reliability of the data used for accuracy evaluation (for example, excluding data such as the affected analysis output and corresponding accuracy information from the accuracy evaluation, or correcting data such as the affected analysis output and corresponding accuracy information and then using it for accuracy evaluation), which can improve the accuracy of the accuracy evaluation.

[0028] In a fifth aspect, a communication method is provided, which is applied to a consumer node. The execution subject of this method can be the consumer node, or a component or device applied to the consumer node (such as a processor, a chip, or a chip system, etc.), or a logic module or software that can implement all or part of the functions of the consumer node. The method includes: receiving first information, where the first information includes an analysis output and second identification information of an analysis object of the analysis output; determining second information according to the first information, where the second information is used for accuracy evaluation, and the second information includes second identification information of an analysis object affected by an execution action, and the execution action is executed based on the analysis output; sending the second information to the first network device.

[0029] In a fifth aspect, a consumer node sends second information to a first network device. Based on the identification information in the second information, the first network device can determine which analysis output the consumer node has taken an execution action according to. And determine the analysis output corresponding to the analysis object affected by the execution action, thereby ensuring the reliability of the data used for accuracy evaluation (for example, excluding the affected analysis output and corresponding accuracy information and other data from the accuracy evaluation, or using the affected analysis output and corresponding accuracy information and other data after correction for accuracy evaluation), which can improve the accuracy of the accuracy evaluation.

[0030] In a sixth aspect, a communication device is provided. The device is applied to the first network device, and the communication device can be the first network device or a chip or system-on-chip in the first network device. The communication device can implement the functions performed by the first network device in the first aspect or possible designs of the first aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the communication device includes: a transceiver module for sending first information to a consumer node, where the first information includes first identification information and an analysis output; the transceiver module is further configured to receive second information from the consumer node, where the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which and / or affected by the execution action that affects the accuracy evaluation, where the execution action is performed based on the analysis output.

[0031] For the sake of understanding, the second information is further described: The second information is used to perform accuracy evaluation on the analysis subscribed by the consumer node and / or perform accuracy evaluation on the model corresponding to the analysis. In other words, the second information is used to generate accuracy information of the analysis and / or the model corresponding to the analysis. That is, the accuracy evaluation may include one or more of: generating accuracy information, performing accuracy evaluation on the analysis subscribed by the consumer node, and performing accuracy evaluation on the model corresponding to the analysis.

[0032] In another implementation, the second information may further include the first information on which and / or affected by the execution action that may affect the analysis output and / or the corresponding true value of the analysis (also known as prediction) (optionally, the second information may further include the identification information of the corresponding first information).

[0033] In a possible implementation, the first identification information indicates the first information, or the first identification information indicates the analysis output.

[0034] In a possible implementation, the device further includes a processing module for:

[0035] Determining the analysis output used for accuracy evaluation according to the second information;

[0036] Accuracy evaluation is performed according to the analysis output for accuracy evaluation.

[0037] In a possible implementation, the apparatus further includes a processing module;

[0038] The transceiver module is further configured to send third information to a second network device, where the third information is used to determine data for accuracy evaluation, the third information includes first information and second information, or the third information includes the analysis output for accuracy evaluation, and the analysis output for accuracy evaluation is determined according to the second information;

[0039] The transceiver module is further configured to receive accuracy information for accuracy evaluation from the second network device;

[0040] The processing module is configured to perform accuracy evaluation according to the data for accuracy evaluation.

[0041] In a possible implementation, the first network device is a network data analysis function network element with an analysis logic function, and the second network device is a network data analysis function network element with a model training logic function.

[0042] In a possible implementation, the transceiver module is further configured to receive an analysis subscription request from a consumer node before sending the first information to the consumer node.

[0043] In a seventh aspect, a communication apparatus is provided. The apparatus is applied to a consumer node, and the communication apparatus may be a consumer node or a chip or a system-on-chip in the consumer node. The communication apparatus can implement the functions performed by the consumer node in the second aspect or the possible designs of the second aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the communication apparatus includes: a transceiver module, configured to receive first information, where the first information includes first identification information and an analysis output; a processing module, configured to determine second information according to the first information, where the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which and / or which affects the execution actions for accuracy evaluation, where the execution actions are performed based on the analysis output; the transceiver module is further configured to send the second information to the first network device.

[0044] In a possible implementation, the first identification information indicates the first information, or the first identification information indicates the analysis output.

[0045] In a possible implementation, the transceiver module is further configured to send an analysis subscription request before receiving the first information.

[0046] In an eighth aspect, a communication device is provided. The device is applied to a second network device, and the communication device may be the second network device, or a chip or system-on-chip in the second network device. The communication device can implement the functions performed by the second network device in the above-mentioned third aspect or possible designs of the third aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the communication device includes: a transceiver module, configured to receive third information from a first network device, where the third information is used to determine data for accuracy evaluation, the third information includes first information and second information, or the third information includes an analysis output for accuracy evaluation, and the analysis output for accuracy evaluation is determined according to the second information; the second information includes first identification information in the first information on which and / or which affects the execution action for accuracy evaluation, the first information includes first identification information and an analysis output, and the execution action is performed based on the analysis output; a processing module, configured to determine data for accuracy evaluation according to the third information; and the transceiver module is further configured to send the data for accuracy evaluation to the first network device.

[0047] In a ninth aspect, a communication device is provided. The device is applied to a first network device, and the communication device may be the first network device, or a chip or system-on-chip in the first network device. The communication device can implement the functions performed by the first network device in the above-mentioned fourth aspect or possible designs of the fourth aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the communication device includes: a transceiver module, configured to send first information to a consumer node, where the first information includes an analysis output and second identification information of an analysis object of the analysis output; and the transceiver module is further configured to receive second information from the consumer node, where the second information is used for accuracy evaluation, and the second information includes second identification information of the analysis object affected by the execution action, and the execution action is performed based on the analysis output.

[0048] In a tenth aspect, a communication device is provided. The device is applied to a consumer node, and the communication device can be the consumer node, or a chip or a system-on-chip in the consumer node. The communication device can implement the functions performed by the consumer node in the fifth aspect or possible designs of the fifth aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the communication device includes: a transceiver module, configured to receive first information, where the first information includes an analysis output and second identification information of an analysis object of the analysis output; a processing module, configured to determine second information according to the first information, where the second information is used for accuracy evaluation, and the second information includes second identification information of the analysis object affected by an execution action, and the execution action is performed based on the analysis output; and the transceiver module is further configured to send the second information to a first network device.

[0049] In an eleventh aspect, the present application provides a communication device. The communication device includes a processor and a transceiver, and the processor and the transceiver are used to support the communication device in executing the method according to any one of the first aspect to the fifth aspect. Further, the communication device may further include a memory, and the memory stores computer instructions, and the processor may run the computer instructions to execute the method according to any one of the first aspect to the fifth aspect.

[0050] In a twelfth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium stores computer instructions, and when the computer instructions run, the method according to any one of the first aspect to the fifth aspect is executed.

[0051] In a thirteenth aspect, the present application provides a computer program product including instructions. When it runs on a computer, it enables the computer to execute the method according to any one of the first aspect to the fifth aspect.

[0052] In a fourteenth aspect, the present application provides a chip. The chip includes a processor and a transceiver, and the processor and the transceiver are used to support the communication device in executing the method according to any one of the first aspect to the fifth aspect.

[0053] In a fifteenth aspect, the present application provides a communication system. The communication system includes a first network device and a consumer node. The first network device is configured to execute the method according to the first aspect or the fourth aspect, and the consumer node is configured to execute the method according to the second aspect or the fifth aspect. Further, the communication system may further include a second network device, and the second network device is configured to execute the method according to the third aspect.

[0054] Among them, for the beneficial effects described in the sixth aspect to the fifteenth aspect of the present application, reference can be made to the analysis of the beneficial effects of the first aspect to the fifth aspect, and details are not described herein again. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic flow chart of subscribing and distributing accuracy information provided by an embodiment of the present application;

[0056] Figure 2 A schematic structural diagram of a communication system provided by an embodiment of the present application;

[0057] Figure 3 A schematic structural diagram of another communication system provided by an embodiment of the present application;

[0058] Figure 4 A schematic flow chart of a communication method provided by an embodiment of the present application;

[0059] Figure 5 A schematic flow chart of another communication method provided by an embodiment of the present application;

[0060] Figure 6 A schematic flow chart of another subscribing and distributing process of accuracy information provided by an embodiment of the present application;

[0061] Figure 7 A schematic flow chart of another subscribing and distributing process of accuracy information provided by an embodiment of the present application;

[0062] Figure 8 A schematic structural diagram of a communication device provided by an embodiment of the present application;

[0063] Figure 9 A schematic structural diagram of another communication device provided by an embodiment of the present application;

[0064] Figure 10 A schematic structural diagram of another communication device provided by an embodiment of the present application;

[0065] Figure 11 A schematic structural diagram of another communication device provided by an embodiment of the present application. Detailed implementation manners

[0066] The network architecture and service scenarios described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those of ordinary skill in the art can know that with the evolution of the network architecture and the emergence of new service scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0067] It should be noted that in the description, claims and drawings of this application, terms such as "first" and "second" are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0068] It should be understood that in the embodiments of this application, "at least one (item)" means one or more, "a plurality" means two or more, "at least two (items)" means two, three or more, and "and / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression means any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A. For example, B can be determined according to A. It should also be understood that determining B according to A does not mean determining B only according to A, but B can also be determined according to A and / or other information. In addition, the "connection" that appears in the embodiments of this application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and this application does not make any limitation on this.

[0069] Unless otherwise specified, the "transmission" (transmit / transmission) that appears in the embodiments of this application refers to two-way transmission, including the actions of sending and / or receiving. Specifically, the "transmission" in the embodiments of this application includes the sending of data, the receiving of data, or the sending and receiving of data. Or rather, the data transmission here includes uplink and / or downlink data transmission. Data can include channels and / or signals. Uplink data transmission is the transmission of uplink channels and / or uplink signals, and downlink data transmission is the transmission of downlink channels and / or downlink signals. The "network" and "system" that appear in the embodiments of this application express the same concept, and a communication system is a communication network.

[0070] Before introducing the embodiments of this application, some terms related to the embodiments of this application are explained.

[0071] Feedback information: Indicates an execution action (also known as an execution operation) taken by a consumer node that is affected by previously provided analytics content. This execution action may or may not affect the ground truth data (also known as true value data) corresponding to the analytics ID requested at the time of prediction reference, and thus affects the monitoring of the accuracy of a machine learning (ML) model subscribed based on the following parameter(s):

[0072] Analytics feedback information: Indicates that the consumer NF has taken an action affected by previously provided analytics. This action may or may not affect the ground truth data corresponding to the analytics ID requested at the time of prediction reference, and thus affects the monitoring of the accuracy of the ML model through subscription based on the following parameter(s):

[0073] - The corresponding analytics ID(s) used to take the execution action;

[0074] - Indicates whether the execution action will affect the ground truth data (if any);

[0075] - The timestamp of the execution action taken.

[0076] It should be noted that the consumer node cannot include analytics feedback information in the initial subscription request. Analytics feedback information can be included in a modification request for an existing analytics subscription.

[0077] Analytics subscription service process: The network data analytics function (NWDAF) provides a service for a consumer node to subscribe to / cancel subscriptions for analytics. Any NWDAF service consumer node (such as a network function network element, an operation management and maintenance function network element, etc.) uses the analytics subscription (Nnwdaf_analytics subscription) service, etc., to subscribe to / cancel subscriptions in the NWDAF to obtain notifications of analytics content. NWDAF service users also use this service to modify or update existing analytics subscriptions. Any entity can use this service.

[0078] Exemplarily, in the analytics accuracy information subscription procedure, the consumer node subscribes to receive analytics output and analytics accuracy information related to the analytics ID requested by the consumer node. The process of accuracy information subscription and distribution is as Figure 1 shown. Specifically:

[0079] 1. The consumer node selects an appropriate NWDAF with AnLF and subscribes to, modifies, or unsubscribes from the accuracy information of the analysis by invoking the analytics subscription request (Nnwdaf_analytics subscription_subscribe_request). The parameters that can be included in the subscription for triggering the accuracy information check and allocation can refer to the prior art and will not be elaborated here.

[0080] 2. When receiving the analytics subscription request, the NWDAF with AnLF validates the parameters of the analytics accuracy request information (analytics accuracy monitoring and checking).

[0081] Specifically, the NWDAF with AnLF starts the analytics accuracy monitoring and generates the accuracy information of the analysis related to the analysis ID indicated in the subscription according to the parameters defined in the analytics accuracy request information. The NWDAF with AnLF will calculate the analytics precision information. If the NWDAF with AnLF does not have sufficient necessary data, it will perform step 3b to collect the ground truth data before calculating the analytics precision information.

[0082] The NWDAF with AnLF may have started to perform the analytics accuracy monitoring and the generation of the accuracy information of the analysis, which were previously triggered by other consumer nodes. After receiving a new request from the NWDAF consumer, the NWDAF with AnLF determines whether new data collection is required to generate the accuracy information of the analysis according to the corresponding analytics subscription.

[0083] In addition to receiving requests from consumer nodes, according to the local policy, the NWDAF with AnLF can decide to start the analytics accuracy monitoring and the generation of the accuracy information of the analysis.

[0084] 3a. The NWDAF with AnLF performs the data collection for the subscribed analysis ID and generates the analytics output (analytics output generation).

[0085] 3b. The NWDAF with AnLF performs the data collection (e.g., ground truth data collection) to generate the accuracy information of the subscribed analysis ID and generates the associated analytics accuracy information (analytics accuracy information generation). If the analytics feedback information is included in step 1, the NWDAF with AnLF can consider it and determine whether it affects the ground truth data through the internal logic to generate the accuracy information of the analysis.

[0086] 4a. When the subscription in step 1 does not contain the analysis accuracy request information, the NWDAF including AnLF sends the analysis output to the consumer node according to the parameters defined in the analysis report information included in the subscription request through Nnwdaf_analyticssubscription_notify(analytics output(s)).

[0087] It should be noted that steps 3b and 4a can occur in any order and are not limited.

[0088] 4b. The NWDAF including AnLF sends the analysis accuracy information and the analysis output of the analysis ID to the consumer node according to the parameters defined in the analysis accuracy request information included in the subscription request through Nnwdaf_analyticssubscription_notify(accuracy information).

[0089] 4c. The NWDAF including AnLF provides the analysis accuracy information of the analysis ID according to the parameters defined in the analysis accuracy request information included in the subscription request. When the period for providing the analysis accuracy information of the analysis indicated in the analysis accuracy request information is different from the period for providing the analysis output indicated in the subscription request, the analysis accuracy information will be provided in a separate notification (for example, Nnwdaf_analyticssubscription_notify).

[0090] 5. When it is determined that the analysis ID has low precision or is insufficient, that is, when the deviation between the analysis output using the trained ML model and the true value data (collected from the data producer corresponding to the analysis ID requested at the time of prediction) is greater than the reporting threshold (configured locally or received in the subscription request), the NWDAF including AnLF can notify the consumer node using the stop analysis output consumption indication and the stop analysis output consumption time window. For example, the notification can be: Nnwdafanalytics subscription notify(stop analyticsoutput consumption indication,stop analytics outputconsumption time window).

[0091] 6. (Optional) The consumer node may decide to stop consuming the analysis output based on its own logic or based on a notification received from the NWDAF with an indication to stop consuming the analysis output, without canceling the subscription to the analysis ID. The NWDAF service consumer invokes the Nnwdaf_analyticssubscription_subscribe service operation, including the subscription association ID, to modify the existing subscription and provides the parameter pause analysis consumption flag in the analysis accuracy request information.

[0092] 7. When the NWDAF determines that the accuracy of the analysis ID has improved (e.g., meets the accuracy requirements of the consumer node) or when the time window for pausing the analysis consumption set by itself ends, the NWDAF notifies the consumer node of the accuracy information of the analysis ID to resume consuming the analysis output. Therefore, the existing subscription to the previously stopped analysis ID is reactivated. For example, it can be notified through Nnwdafanalytics subscription notify(resume analyticsoutput consumption indication).

[0093] 8. (Optional) The NWDAF service consumer may, based on its own logic, notify the NWDAF to resume providing the analysis output, thereby reactivating the existing subscription to the analysis ID that has been paused by the NWDAF service consumer's request (step 6) or NWDAF indication (step 5). The NWDAF service consumer invokes the Nnwdaf_analytics subscription_subscribe service operation, including the subscription association ID, to modify the existing subscription and provides the parameter resume analysis subscription request in the analysis accuracy request information.

[0094] In the scenario where the consumer node subscribes to the analysis from the NWDAF containing the AnLF using the analysis subscription request as introduced above, there may be the following two cases. In the first case, the consumer node receives multiple analysis content notifications within a short period. For example, it receives two analysis content notifications. The consumer node may perform an action based on the analysis content of the first analysis content notification, or the consumer node may also perform an action by synthesizing the analysis content of the two analysis content notifications. Then, the situation of performing this action is fed back to the AnLF through the feedback information.

[0095] In the second case, in the same notification message, multiple analysis and prediction contents may be combined. For example, abnormal behavior prediction may contain multiple event identifiers (event ids). If the consumer node takes an action based on the analysis output. Since the feedback information does not contain the predicted analysis output, AnLF will not be able to determine which event id the consumer node has taken an action on. For example, AnLF provides both distributed denial of service (DDoS) attack and Too frequent Service Access predictions to the Policy Control Function (PCF) network element, that is, the consumer node is the PCF. In response to the above situations, the PCF may perform actions to request the Session Management Function (SMF) to release the Protocol Data Unit (PDU) session control (seesion). In other words, AnLF will not be able to know which of the above prediction analyses the PCF specifically requests the SMF to release the PDU seesion for.

[0096] In the above two cases, AnLF cannot determine which analysis subscription content notification the feedback information of the consumer node corresponds to. This results in a relatively low accuracy when AnLF conducts accuracy evaluation based on the feedback information.

[0097] Exemplarily, in the scenario of predicting network element load, the analysis output of AnLF is to predict the network element load situation at a certain future time period / point. The consumer node subscribes to this analysis. The consumer node determines that the network element load is high at a future time based on the analysis output at the future time (which can also be referred to as analysis prediction, prediction result, etc.). The consumer node takes an execution action (or behavior) to reduce the load of the network element in order to lower the future network element load. If the analysis output is accurate and the execution action taken by the consumer node is effective, the network element load at the future time will become lower. In other words, the true value data corresponding to this analysis output is that the network element load is low, which is inconsistent with the analysis output. At this time, if the impact of the consumer node's execution action on the true value data at the future time is not considered and this analysis output and the corresponding true value data are not excluded from the accuracy evaluation, the result of the accuracy evaluation based on this analysis output and the corresponding true value data will be inaccurate. In fact, AnLF cannot determine which analysis content notification the feedback information of the consumer node corresponds to, so it cannot exclude such data from the accuracy evaluation.

[0098] To solve the above technical problems, an embodiment of the present application provides a communication method. The method provided by the embodiment of the present application will be described below with reference to the accompanying drawings of the specification.

[0099] The communication method provided by the embodiment of the present application can be applied to various communication systems, such as: the 6th generation (6G) mobile communication system, the long term evolution (LTE) system, the 5th generation (5G) mobile communication system, the wireless fidelity (WiFi) system, a future communication system, or a system integrating multiple communication systems, etc. The embodiment of the present application does not make any limitation. Among them, 5G can also be referred to as new radio (NR).

[0100] Next, taking Figure 2 in combination with Figure 3 the communication system shown as an example, the communication method provided by the embodiment of the present application will be described.

[0101] As Figure 2 shown, it is a schematic diagram of the architecture of the communication system 20 provided by the present application. Figure 2 In the communication system 20, it may include a network element 201 and a network element 202 (i.e., a consumer node) communicatively connected to the network element 201. Optionally, the communication system 20 may further include a network element 203 communicatively connected to the network element 201.

[0102] In Figure 2 the network element 202 may send an analysis subscription request to the network element 201 to subscribe to the analysis. After receiving the analysis subscription request, the network element 201 provides analysis content and / or accuracy information to the network element 202 through Nnwdaf analysis subscription notifications, etc. Subsequently, the network element 202 takes a pre-configured action according to the analysis content (also referred to as the analysis output). And feeds back the situation of executing the action to the network element 201 through feedback information. The network element 201 performs accuracy evaluation based on one or more of the following: feedback information, true value data, etc.

[0103] Optionally, when the network element 201 does not have the ability to output or provide accuracy information or does not have a model for outputting accuracy information, the network element 201 may send a request for generating accuracy information to a network element that has the ability to output or provide accuracy information or a network element 203 that has a model for outputting accuracy information (the request may carry information for generating accuracy information). For example, the network element 203 may be an NWDAF with a model training logic function (MTLF). After receiving the request for generating accuracy information, the network element 203 generates accuracy information. The specific method is, for example, to call the model to generate accuracy information and feedback it to the network element 201.

[0104] In a possible implementation, Figure 2 the communication system 20 shown in the figure may be applied to the current 5G network or other future networks, etc. The embodiments of the present application do not make specific limitations in this regard. The present application takes Figure 2 the communication system 20 shown in the figure being applied to the network service architecture of the current 5G mobile communication system as an example for elaboration.

[0105] Exemplarily, Figure 2 the communication system 20 shown in the figure may be applied to Figure 3 the 5G network shown in the figure. In Figure 3Among them, the 5G network includes a network exposure function (NEF) network element, a network repository function (NRF) network element, a NWDAF network element, an application function (AF) network element, a policy control function (PCF) network element, a unified data repository (UDR) network element, a unified data management (UDM) network element, an access and mobility management function (AMF) network element, a session management function (SMF) network element, a user plane function (UPF) network element, a data network (DN), a terminal, a radio access network (RAN) node, and an operation, administration and maintenance (OAM) network element. Among them, the RAN node can communicate with the AMF network element and the UPF network element, and the terminal can communicate with the AMF network element. The terminal and the RAN node can communicate using air interface technology.

[0106] In Figure 3 Among them, N1 is the interface between the terminal and the AMF network element, N2 is the interface between the RAN node and the AMF network element, N3 is the interface between the RAN node and the UPF network element, N4 is the interface between the SMF network element and the UPF network element, and N6 is the interface between the UPF network element and the DN. Namf is the service-based interface provided by the AMF network element, Nsmf is the service-based interface provided by the SMF network element, Nudm is the service-based interface provided by the UDM network element, Nnef is the service-based interface provided by the NEF network element, Nnrf is the service-based interface provided by the NRF network element, Nnwdaf is the service-based interface provided by the NWDAF network element, Naf is the service-based interface provided by the AF network element, Npcf is the service-based interface provided by the PCF network element, and Nudr is the service-based interface provided by the UDR network element.

[0107] It can be understood that if Figure 2 the communication system 20 shown is applied to Figure 3 the 5G network shown, then Figure 2The network element or entity corresponding to the network element 201 in it may be an NWDAF network element with AnLF in the 5G network architecture; Figure 2 The network element or entity corresponding to the network element 202 in it may be any network element in the 5G network architecture; Figure 2 The network element or entity corresponding to the network element 203 in it may be an NWDAF network element with MTLF in the 5G network architecture. It should be understood that in specific applications, Figure 2 The network element 201, network element 202 or network element 203 in it may also correspond to other network elements in the 5G network, without limitation. The main functions of each network element are introduced in detail below.

[0108] The NWDAF network element can provide data analysis functions for 5G core network (5G core, 5GC) network functions (network function, NF) and / or OAM network elements. For example, a 5GC NF or OAM network element can request network data analysis results from the NWDAF network element. After receiving the request, the NWDAF network element collects data from relevant network elements and trains an artificial intelligence (artificial intelligence, AI) model or a machine learning (machine learning, ML) model, and finally uses the model for data inference and feeds the inference results back to the corresponding 5GC NF or OAM network element. It can be understood that according to different functions, the NWDAF network element can be divided into a training-supported NWDAF (i.e., NWDAF(MTLF)) network element and an inference-supported NWDAF (i.e., NWDAF(AnLF)) network element. Optionally, the NWDAF(AnLF) network element can request AI model information from the NWDAF(MTLF) network element for data inference. If the NWDAF network element wants to obtain terminal-related data through the OAM network element, it can be achieved by accessing the minimization of drive tests (MDT) network element. Since the MDT network element belongs to the OAM network element, the MDT network element can directly interact with the OAM network element, and thus indirectly access the OAM network element by accessing the MDT network element. For example, the NWDFA network element can subscribe to information about relevant terminals from the MDT network element to collect OAM data reported by the MDT RAN node side.

[0109] The NRF network element is mainly used to provide internal / external addressing functions, etc.

[0110] The NEF network element is the external interface of the 5G core network that provides standard interfaces, capable of externally providing the capabilities and event openings of the 5G network to receive external relevant information. For example, the NEF network element has a member selection function, which can provide a candidate member list to the AF network element. Based on the 3GPP network function, the 5G core network exposes functions and events to other systems through the NEF network element, providing both openness and ensuring system security. The existence of the NEF network element standardizes the implementation of the 5G core network functions, greatly facilitating third-party access. At the same time, the NEF network element can also shield network and user sensitive information, thus ensuring the security of the 5G core network.

[0111] The AF network element can interact with other control network elements of the 5G network on behalf of the application, including providing service QoS requirements, routing policy requirements, etc. For example, when the AF needs to obtain information through the 5G core network, by accessing the NEF network element, the AF network element can ignore the internal topology of the 5G core network and thus obtain the required information. It can be seen that the NEF network element plays a role in guiding the AF network element.

[0112] The PCF network element is mainly used for managing policy rules, managing user subscription information, etc.

[0113] The UDM network element is mainly used for authentication credit processing, user identification processing, access authorization, registration / mobility management, subscription management, and short message management, etc.

[0114] The AMF network element can support terminals with different mobility management requirements. It performs the following main tasks: non-access stratum (NAS) signaling for terminals, NAS signaling security, access layer security control, core network node signaling for mobility between 3GPP access networks, reachability of idle mode terminals (including control and execution of paging retransmission), registration area management, support for intra-system and inter-system mobility, access authentication, access authorization including roaming privilege checks, mobility management control including subscriptions and policies, support for network slicing and SMF selection, etc.

[0115] The SMF network element can be used for session management, terminal Internet Protocol (IP) address allocation and management, selection and control of the UPF network element, configuring traffic control in the UPF network element, routing traffic to the appropriate destination, policy enforcement, and QoS control sections, and downlink data notification, etc.

[0116] The UPF network element serves as the anchor point for mobility within and between systems, connects to the external protocol data unit (PDU) session point of the data network, performs packet routing and forwarding, packet inspection, and user plane policy rule enforcement, traffic usage reporting, uplink classifier, supports routing service flows to the data network, serves as the branching point for multi-homed PDU sessions, and is used for QoS processing of the user plane, such as packet filtering, gating, uplink and downlink rate enforcement, uplink service verification, downlink packet buffering, and downlink data notification triggering, etc.

[0117] The OAM network element can divide the network management work into three major categories according to the actual needs of the operator network operation: operation, management, and maintenance. Among them, operation mainly completes the analysis, prediction, planning, and configuration of daily networks and services. Maintenance mainly refers to the daily operation activities such as testing and fault management of the network and its services. Management mainly refers to ensuring the normal operation of the network through scheduling. The OAM network element can complete fault detection, path discovery, fault location, and performance monitoring, thereby realizing the analysis, prediction, planning, and configuration of the network, and can also perform testing and fault management on the network and its services.

[0118] DN: It can be an operator service network, Internet access, or a third-party service network, etc. The DN can interact with the terminal through a PDU session. Among them, the PDU session can be divided into multiple types, such as Internet Protocol Version 4 (IPv4), IPv6, etc.

[0119] A terminal is a device with wireless transceiver capabilities. The terminal can be deployed on land, including indoors, outdoors, handheld or vehicle-mounted; it can also be deployed on water (such as ships, etc.); it can also be deployed in the air (such as airplanes, balloons, satellites, etc.). The terminal can also be referred to as a terminal device. The terminal device can be a user equipment (UE), mobile station (MS), mobile terminal (MT), etc., or a device used to provide voice or data connectivity to users. Among them, the UE includes handheld devices with wireless communication capabilities, vehicle-mounted devices (such as cars, bicycles, electric vehicles, airplanes, ships, trains, high-speed trains, etc.), wearable devices (such as smart watches, smart bracelets, pedometers, earphones, etc.) or computing devices. Exemplarily, the UE can be a mobile phone, tablet computer, laptop computer, palmtop computer, mobile internet device (MID), satellite terminal or a computer with wireless transceiver capabilities. The UE can also be a virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless modem, smart point of sale (POS) machine, customer-premises equipment (CPE), intelligent robot, robotic arm, workshop equipment, smart home equipment (such as refrigerators, TVs, air conditioners, electricity meters, etc.), wireless terminals in industrial control, wireless terminals in unmanned driving, wireless terminals in remote medical treatment, wireless terminals in smart grid, wireless terminals in transportation safety, wireless terminals in smart city, wireless terminals in smart home, vehicle-mounted terminals, roadside units (RSUs) with terminal functions, or flying devices (such as intelligent robots, hot air balloons, drones, airplanes), and so on. The terminal can also be other devices with terminal functions. For example, the terminal can also be a device that serves as a terminal function in device-to-device (D2D) communication.

[0120] By way of example and not limitation, in the present application, the terminal may be a wearable device. A wearable device may also be referred to as a wearable intelligent device, which is a general term for devices developed by applying wearable technologies to the intelligent design of daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is either directly worn on the body or integrated into the user's clothes or accessories. For example, a wearable device is not only a hardware device but also a device that realizes powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable intelligent devices include devices with complete functions and large sizes that can realize complete or partial functions without relying on a smartphone, such as smart watches or smart glasses, and also include devices that only focus on a certain type of application function and need to cooperate with other devices such as smartphones, such as various smart bracelets and smart jewelry for physical sign monitoring.

[0121] In the present application, the terminal may be a terminal in an Internet of Things (IoT) system. The IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technologies, thereby realizing an intelligent network of human-machine interconnection and object-object interconnection. The terminal in the present application may be a terminal in machine type communication (MTC). The terminal of the present application may be an in-vehicle module, in-vehicle module group, in-vehicle component, in-vehicle chip, or in-vehicle unit built into a vehicle as one or more components or units. The vehicle can implement the method of the present application through the built-in in-vehicle module, in-vehicle module group, in-vehicle component, in-vehicle chip, or in-vehicle unit. The terminal of the present application may be a means of transportation, such as a vehicle. Therefore, the present application can be applied to vehicle networking, such as vehicle to everything (V2X), long term evolution vehicle (LTE-V), vehicle to vehicle (V2V), etc.

[0122] In the present application, the form of the terminal is not limited. The device for realizing the functions of the terminal may be the terminal; it may also be a device that can support the terminal to realize such functions, such as a chip system. This device may be installed in the terminal or used in matching with the terminal.

[0123] A RAN node can be a device with wireless transceiver functions, which can help a terminal achieve wireless access. The RAN node in this application can also be referred to as a node in the RAN, a RAN node, an access network device, etc. The RAN node includes but is not limited to: the evolved base station in LTE (NodeB or eNB or e-NodeB, evolutional Node B), the evolved base station in the next-generation LTE (next generation eNB, ng-eNB), the base station in NR (gNodeB or gNB), the transmitting point (TP) or the transmission receiving point / transmission reception point (TRP), the base station evolved by 3GPP in the future, the next-generation base station (next generation NodeB, gNB), the next-generation base station in the 6G mobile communication system, the base station in the future mobile communication system, the satellite, the access node in the WiFi system, the wireless relay node, the wireless backhaul node, the integrated access and backhaul (IAB) node, the RAN node in the non-terrestrial network (NTN) communication system of the mobile switching center, that is, it can be deployed on a high-altitude platform or a satellite, etc. The base station can be: a macro base station, a micro base station, a pico base station, a small station, a relay station, or a balloon station, etc. Multiple base stations can support the network of the same technology mentioned above, or can also support the networks of different technologies mentioned above. The base station can include one or more co-site or non-co-site TRPs. The RAN node can also be a device that serves as a base station in D2D communication, vehicle-to-everything communication, unmanned aerial vehicle communication, and machine communication. The RAN node can also be a wireless controller in the cloud radio access network (CRAN) scenario. The RAN node can also be a centralized unit (CU), a distributed unit (DU), a CU-control plane (CP), a CU-user plane (UP), a radio unit (RU), a road side unit (RSU) with base station functions, a wired access gateway, or a core network element, etc. The RAN node can also be a server, a wearable device, a machine communication device, or a vehicle-mounted device, etc. For example, the RAN node in V2X technology can be an RSU. The following takes the RAN node as a base station as an example for illustration. The multiple RAN nodes can be the same type of base station or different types of base stations.The base station can communicate with the terminal, or communicate with the terminal through a relay station. The terminal can communicate with multiple base stations of different technologies. For example, the terminal can communicate with a base station supporting an LTE network, can also communicate with a base station supporting a 5G network, and can also support dual connection with a base station of an LTE network and a base station of a 5G network.

[0124] In this application, the CU and DU can be set separately, or can also be included in the same network element, such as a baseband unit (BBU). The RU can be included in a radio frequency device or a radio frequency unit, such as included in a remote radio unit (RRU), an active antenna unit (AAU), or a remote radio head (RRH). It can be understood that the CU can be divided into a RAN node in the access network, or the CU can be divided into a RAN node in the core network, which is not limited here.

[0125] In different systems, the CU (or CU-CP and CU-UP), DU, or RU may also have different names, but those skilled in the art can understand their meanings. For example, in an open radio access network (ORAN) system, the CU can also be called an O-CU (open CU), the DU can also be called an O-DU, the CU-CP can also be called an O-CU-CP, the CU-UP can also be called an O-CU-UP, and the RU can also be called an O-RU. For the convenience of description, the CU, CU-CP, CU-UP, DU, and RU are used as examples in this application. Any one of the CU (or CU-CP, CU-UP), DU, and RU in this application can be implemented by a software module, a hardware module, or a combination of a software module and a hardware module.

[0126] It can be understood that in some scenarios, the roles of the RAN node and the terminal are relative. For example, a helicopter or a drone that is usually configured as a terminal can also be configured as a mobile base station, and the device accessing the RAN node through the helicopter or the drone is configured as a terminal.

[0127] In this application, the form of the RAN node is not limited. The device for implementing the function of the RAN node can be the RAN node; it can also be a device capable of supporting the RAN node to implement this function, such as a chip system. This device can be installed in the RAN node or used in combination with the RAN node.

[0128] Figure 4 The flowchart of the communication method provided by the embodiment of this application is shown. As Figure 4As shown, the method may include the following steps:

[0129] S410, the consumer node sends an analysis subscription request to the first network device. Correspondingly, the first network device receives the analysis subscription request.

[0130] Among them, the analysis subscription request is used to request an analysis subscription service. Exemplarily, the analysis subscription request may be an analytics subscription_subscribe, or an Nnwdaf analytics information request (analyticsinfo_request), etc. After receiving the analysis subscription request, the first network device may generate an analysis output and accuracy information based on the analysis subscription request. The analysis subscription request includes: an analysis ID. The analysis ID is used to identify the analysis service for which the subscription is requested.

[0131] S420, the first network device sends the first information to the consumer node. Correspondingly, the consumer node receives the first information.

[0132] Among them, the first information includes first identification information and an analysis output. The first information may be a notify notification message, or the first information may also be other messages. For example, a Transmission Control Protocol (TCP) message (in this case, the first identification information may be the TCP message identifier), an application layer message (in this case, the first identification information may be the identifier of the application layer message), etc., without limitation. The first identification information is used to identify the first information. For example, the first identification information is used to identify an analysis definition notification message (Nnwdaf_analytics subscription_notify). The analysis output is the analysis output or analysis prediction corresponding to the analysis ID. It can be understood that the analysis output is the analysis output corresponding to the analysis ID generated by the first network device. Exemplarily, the analysis output may be the analysis output generated by the first network device according to or using the model corresponding to the model information provided by the second network device. The analysis output may be one or more analysis outputs.

[0133] It can be understood that the first identification information is used to identify the analysis output. For example, the first identification information may indicate the first information. In other words, the first identification information indicates all the analysis outputs in the first information. At this time, one first identification information indicates one first information, and the first identification information at this time may be called a notify ID. Exemplarily, at this time, the first information may include: analysis output A, analysis output B, analysis output C, and one first identification information ID-1, and ID-1 indicates analysis output A, analysis output B, and analysis output C.

[0134] In another example, the first information includes M analysis outputs and M pieces of first identification information, where each piece of first identification information indicates one analysis output, and M is a positive integer. At this time, one piece of first identification information indicates one analysis output in the first information (usually, one piece of first information will include multiple analysis outputs), which improves the indication granularity. Based on this implementation, it is possible to more finely distinguish which information is used for accuracy evaluation. Exemplarily, at this time, the first information includes: analysis output A, analysis output B, analysis output C, and first identification information ID-1, first identification information ID-2, and first identification information ID-3. ID-1 indicates analysis output A, ID-2 indicates analysis output B, and ID-3 indicates analysis output C.

[0135] It should be understood that the two designs of the above first information can be combined, that is, the first information includes both the first identification information indicating the first information and the first identification information indicating the specific analysis output. Exemplarily, at this time, the first information includes: analysis output A, analysis output B, analysis output C, and first identification information ID-1, first identification information ID-2, first identification information ID-3, and first identification information ID-N. ID-1 indicates analysis output A, ID-2 indicates analysis output B, ID-3 indicates analysis output C, and ID-N indicates analysis output A, analysis output B, and analysis output C at the same time.

[0136] In another possible implementation, the first information includes the analysis output and the second identification information of the analysis object of the analysis output. Based on different analysis scenarios, the composition of the analysis object may be diverse. For example, the analysis object may be: UE (e.g., user permanent identifier SUPI, general public user identification GPSI, other identifiers of the UE), behavior of the UE (e.g., handover behavior of the UE, behavior of the UE registering and deregistering the network, etc.), location area (cell, tracking area identifier TAI,), session (PDU session, analysis subscription, transaction identifier) PDU session, slice, etc. It should be understood that the above analysis objects are examples, and in specific implementations, the analysis object may also be other objects, which are not limited.

[0137] After receiving the first information, the consumer node can perform corresponding actions according to the analysis output (the corresponding relationship or judgment logic of which execution actions should be taken based on the analysis output can be pre-configured in the consumer node, and the specific configuration process can refer to the prior art and will not be elaborated here). And determine the execution actions that may affect the true value corresponding to the analysis output. These execution actions will affect the accuracy evaluation of the analysis or the model (used in the analysis), and determine the first identification information in the first information on which the execution actions affecting the accuracy evaluation are based and / or affected by (and / or record the second identification information of the analysis object affected by the executed actions).

[0138] After receiving the first information, the consumer node can perform corresponding actions according to the analysis output (the corresponding relationship or judgment logic of which execution actions should be taken based on the analysis output can be pre-configured in the consumer node in advance, and the specific configuration process can refer to the prior art and will not be elaborated here). The consumer node determines the execution actions that may and / or may not affect the true value corresponding to the analysis output. It can be understood that these execution actions may affect the accuracy evaluation of the analysis or the model (or the model used in the analysis). And it determines the analysis (analysis prediction or output) on which the execution action affecting the accuracy evaluation is based and / or the analysis (analysis prediction or output) affected by the execution action, and the true value corresponding to these analyses (analysis prediction or output) is affected by the execution action.

[0139] The consumer node determines the first identification information in the first information (and / or records the second identification information of the analysis object affected by the execution action). Specifically, the consumer node determines the identification information of the above analysis (analysis prediction or output).

[0140] S430, the consumer node sends the second information to the first network device. Correspondingly, the first network device receives the second information from the consumer node.

[0141] Among them, the second information includes the first identification information in the first information on which the execution action affecting the accuracy evaluation is based and / or the first identification information affected by it. The second information can indicate which information should be used for the accuracy evaluation. The second information can be carried in the notification of the feedback information (that is, at this time, the feedback information can include: the second information and at least one of the following data: the corresponding analysis ID(s) that have been used to take the execution action, indicating whether the execution action will affect the true value data (if any), the timestamp of the execution action taken, etc.), or it can also be carried in other notifications without limitation. For the sake of understanding, the second information is further described: The second information is used to evaluate the accuracy of the analysis subscribed by the consumer node and / or to evaluate the accuracy of the model corresponding to the analysis. In other words, the second information is used to generate the accuracy information of the analysis and / or the model corresponding to the analysis. That is, the accuracy evaluation can include one or more of: generating accuracy information, evaluating the accuracy of the analysis subscribed by the consumer node, and evaluating the accuracy of the model corresponding to the analysis.

[0142] In another implementation, the second information can also include the first information on which the execution action that may affect the true value corresponding to the analysis output and / or the analysis (also called prediction) is based and / or the first information affected by it (optionally, the second information can also include the identification information of the corresponding first information).

[0143] The first identification information in the first information based on which the execution action is taken may refer to the first identification information of the analysis outputs based on which the execution action is taken. The first identification information in the first information affected by the execution action may refer to the first identification information of the analysis outputs corresponding to which true value data may be affected by the execution action.

[0144] For example, in the scenario of predicting network element load, the analysis output of AnLF is to predict the network element load situation at a certain future time period / point. The consumer node subscribes to this analysis. The consumer node determines that the network element has a high load at a future time based on the analysis output at the future time (which can also be referred to as analysis prediction, prediction result, etc.). The consumer node takes an execution action (or behavior) to reduce the load of the network element in order to lower the future network element load. If the analysis output is accurate and the execution action taken by the consumer node is effective, the network element load at the future time will become lower. In other words, the true value data corresponding to this analysis output is that the network element load is lower, which is inconsistent with the analysis output. That is to say, the execution action taken by the consumer node affects the accuracy evaluation of the analysis output (abbreviated as load analysis output) of predicting the network element load situation at a certain future time period / point. At this time, the second information may include the load analysis output and / or the first identification information of the load analysis output.

[0145] In a possible implementation manner, the second information includes the first information based on which the execution action affecting the accuracy evaluation is taken and / or the first identification information of the first information.

[0146] In a possible implementation manner, the second information includes the first information based on which the execution action affecting the analysis (also known as prediction) or the analysis output is taken and / or the first identification information of the first information.

[0147] In a possible implementation manner, the second information includes the first information affected by the execution action and / or the first identification information of the first information.

[0148] In yet another possible implementation manner, the second information includes the second identification information of the analysis object affected by the execution action.

[0149] S440, the first network device performs an accuracy evaluation according to the second information.

[0150] Among them, the second piece of information may indicate which information should be used for accuracy evaluation. After receiving the second piece of information, the first network device can perform accuracy evaluation based on it. Exemplarily, the first network device evaluates the accuracy of the analysis ID and / or the relevant model corresponding to the analysis ID based on one or more items in the second piece of information. It can be understood that the first network device uses the relevant model to provide the analysis service corresponding to the analysis ID. Specifically, the accuracy evaluation may include at least one of the following: evaluating the accuracy of the analysis output relative to the corresponding true value data collected (i.e., evaluating the accuracy of the analysis), evaluating the accuracy of the model that generates the analysis output based on the accuracy information (for example, evaluating the robustness of the model, etc.). The specific evaluation process can refer to the prior art and will not be elaborated here. It should be understood that the above accuracy evaluation is for example, and in specific implementation, other dimensions of evaluation can be performed, such as: the robustness, security, and energy consumption of the model, which are not limited in this specification.

[0151] When performing accuracy evaluation, data such as the accuracy information corresponding to the first identification information in the first piece of information on which the execution actions affecting the accuracy evaluation are based and / or that are affected can be excluded or corrected and then used for accuracy evaluation, so as to achieve the purpose of improving the accuracy of the accuracy evaluation, without limitation.

[0152] In a possible implementation manner, the first piece of information includes the first identification information and the analysis output, and the second piece of information includes the first identification information in the first piece of information on which the execution actions affecting the accuracy evaluation are based and / or that are affected (referred to as the target identification information, which can be one or more). In this implementation manner, the first network device first determines the analysis output corresponding to the target identification information in the first piece of information (referred to as the target analysis output), and then excludes or corrects the target analysis output and the corresponding accuracy information, etc., and then performs accuracy evaluation.

[0153] Specifically, refer to the two possible designs of the first identification information introduced in S420. When determining the analysis output corresponding to the target identification information in the first piece of information, if the first identification information indicates the first piece of information, the analysis output in the first piece of information corresponding to the target identification information is determined as the target identification information. If the first identification information indicates the analysis output in the first piece of information, the analysis output corresponding to the target identification information is determined as the target identification information.

[0154] In another possible implementation, the first information includes the analysis output and the second identification information of the analysis object of the analysis output, and the second information includes the second identification information (referred to as the target identification information) of the analysis object affected by the execution action. In this implementation, the first network device first determines the analysis object (referred to as the target analysis object) corresponding to the target identification information in the first information, and then excludes or corrects the analysis output corresponding to the target analysis object and the corresponding accuracy information, etc., and then performs accuracy evaluation.

[0155] In the embodiment of the present application, according to the identification information in the second information, the first network device can determine which analysis output the consumer node has taken an execution action based on. And determine the analysis output on which the execution action affecting the accuracy evaluation is based and / or affected (or the analysis output corresponding to the analysis object affected by the execution action), thereby ensuring the reliability of the data used for accuracy evaluation (for example, excluding the data such as the affected analysis output and the corresponding accuracy information from the accuracy evaluation, or correcting the data such as the affected analysis output and the corresponding accuracy information and then using it for accuracy evaluation), and improving the accuracy of the accuracy evaluation.

[0156] In addition, since the identification information in the first information and the second information can play an indicative role, using the identification information can distinguish the correspondence between the feedback information and the analysis output, which enables the feedback information to be reported without corresponding to the analysis output one by one. In other words, the consumer node can report multiple pieces of feedback information at one time. In this way, it is more conducive to efficient information statistics, reduces unnecessary messages between the consumer node and the first network device, and thus reduces the communication overhead.

[0157] In one embodiment, as Figure 5 shown, the method may further include:

[0158] S510, the first network device sends the third information to the second network device. Correspondingly, the second network device receives the third information.

[0159] Among them, the third information is used to determine the data for the accuracy evaluation. In S410, the first network device may generate accuracy information based on the analysis subscription request. In fact, the first network device may not have the ability to generate accuracy information and thus cannot generate it. Instead, it needs to obtain the accuracy information from the second network device. Exemplarily, the second network device may be the producer of the model used by the first network device to provide analysis services. Or rather, the second network device provides the model or model information (such as, including one or more of the following: model identification information, model address, model file content, etc.) used by the first network device for analysis services. The first network device may not have a model for outputting accuracy information and needs to obtain the model for outputting accuracy information from the second network device, which is deployed on the second network device. In other words, in this case, the first network device sends the third information required to generate accuracy information to the second network device, and the second network device generates the accuracy information.

[0160] In other words, the third information can be used for model training or model retraining. Exemplarily, the second network device determines that the model needs to be retrained according to the third information.

[0161] The content of the third information can be set flexibly. For example, the third information may include one or more of the following: the first information, the second information, the true value data for generating accuracy information (the second network device may also have the ability to collect data such as true value data. If the second network device has cached data such as true value data, there is no need for the third information to carry duplicate content), etc. In this scenario, the first network device does not determine which analysis outputs (i.e., target analysis outputs) are affected by the execution actions of the consumer nodes. Instead, it is determined by the second network device (the determination process can refer to the same principle as the determination process in S440 and will not be elaborated here). After the second network device determines the target analysis outputs, it can exclude / correct the target analysis outputs and the corresponding true value data, etc., and then generate the accuracy information corresponding to the non-target analysis outputs and / or the corrected target analysis outputs respectively (i.e., the accuracy information for the accuracy evaluation).

[0162] In another possible implementation, the third information may include one or more of the following: the analysis output for the accuracy assessment, the true value data corresponding to the analysis output for the accuracy assessment, etc. In this scenario, the first network device first determines the analysis output for the accuracy assessment (the determination process can refer to the description in S440 and will not be elaborated here), and uses the true value data corresponding to the analysis output for the accuracy assessment, which is required to determine the accuracy information, as the third information. In this implementation, the second network device does not need to determine which is the analysis output for the accuracy assessment, and the content carried by the third information is less, resulting in less communication overhead. After receiving the third information, the second network device generates the corresponding accuracy information based on the analysis output for the accuracy assessment in the third information.

[0163] In another possible implementation, the third information may include the first information and / or the true value data corresponding to the analysis output in the first information, etc. After receiving the third information, the second network device generates the full amount of accuracy information based on it, and sends the accuracy information and the identification information in the corresponding first information to the first network device. The first network device determines which data is used for the accuracy assessment based on the received full amount of accuracy information, the corresponding identification information, and the second information.

[0164] S520. The second network device sends the data for the accuracy assessment to the first network device. Correspondingly, the first network device receives the accuracy information for the accuracy assessment.

[0165] S530. The first network device performs the accuracy assessment based on the data for the accuracy assessment.

[0166] Among them, after receiving the accuracy information for the accuracy assessment, the first network device can perform the accuracy assessment based on it. The specific assessment process can refer to the description in S440 and will not be elaborated here.

[0167] In the embodiments of the present application, according to the identification information in the second information (which can be the first identification information or the second identification information), it can be determined which analysis output the consumer node has taken the execution action according to, and the analysis output (or the analysis output corresponding to the analysis object not affected by the execution action) that is the basis for and / or affected by the execution action that has not affected the accuracy assessment is determined as the target analysis output. The second network device sends the accuracy information corresponding to the target analysis output to the first network device. The first network device performs the accuracy assessment based on the accuracy information corresponding to the target analysis output, thereby ensuring the reliability of the data for the accuracy assessment (for example, excluding the data such as the affected analysis output and the corresponding accuracy information from the accuracy assessment, or correcting the data such as the affected analysis output and the corresponding accuracy information and then using it for the accuracy assessment), and improving the accuracy of the accuracy assessment.

[0168] The above mainly introduced the solution provided by the embodiments of the present application from the perspective of the execution logic of each step. For the sake of understanding, the solution provided by the embodiments of the present application will be further introduced below in combination with specific examples.

[0169] Figure 6 The process of subscribing to and distributing accuracy information provided by the embodiments of the present application is shown, as Figure 6 shown, Figure 6 Compared with the current process of subscribing to and distributing accuracy information, the process shown modifies the information content carried in the notifications in steps 4a and 6 to obtain steps 4a' and 6', and adaptively modulates the subsequent process of accuracy evaluation. It should be understood that in specific implementation, the modification of the information content carried in the notifications in steps 4a and 6 can also be migrated to the notifications in other steps to achieve the same purpose, without limitation. In addition, to avoid redundancy, the steps with the same functions as those in the current process of subscribing to and distributing accuracy information will not be described in the embodiments of the present application, but steps 4a' to 8a will be mainly described. Specifically:

[0170] The notification in step 4a' includes analysis output and identification information. In other words, the notification in step 4a' includes the first information in the above embodiments.

[0171] Among them, the identification information may include the above-mentioned first identification information and / or second identification information. For the descriptions of the first identification information and the second identification information, reference can be made to the foregoing embodiments, and details will not be repeated.

[0172] Step 5, the consumer node takes relevant execution actions according to the analysis output and internal logic.

[0173] The notification in step 6' includes the identification information of the analysis output (or analysis object) affected by the execution action. In other words, the notification in step 6' includes the second information in the above embodiments. For the description of the second information, reference can be made to the foregoing embodiments, and details will not be repeated.

[0174] Step 7a, the first network device performs an accuracy evaluation according to the second information to obtain an accuracy evaluation result (which can also be referred to as accuracy information).

[0175] Step 8a, the first network device sends the accuracy evaluation result obtained in step 7a to the consumer node.

[0176] In another example, the solution provided by the embodiments of the present application is further introduced in combination with the implementation scenarios of S520 - S530. As Figure 7 shown, Figure 7 Another process of subscribing to and distributing accuracy information provided by the embodiments of the present application is shown, including:

[0177] 1. The consumer node sends an analysis subscription request to the first network device.

[0178] 2. The first network device instructs the second network device to invoke the model through a model subscription request.

[0179] 3. The second network device sends a model subscription notification message to the first network device, indicating that the model subscription notification has been received.

[0180] 4. The first network device sends an ML model monitor register message to the second network device.

[0181] 5. The second network device sends an ML model monitor subscribe notification message to the first network device.

[0182] 6. The first network device sends the first information to the consumer node using Nnwdaf_analytics subscription notify.

[0183] 7. The consumer node performs an action based on the analysis output.

[0184] 8. The consumer node sends feedback information (i.e., the second information) to the first network device using Nnwdaf_analytics subscription_subscribe.

[0185] Among them, the description of the second information can refer to the foregoing embodiments and will not be elaborated here.

[0186] 9. The first network device determines the data for model accuracy evaluation (i.e., the third information) and sends it to the second network device. Exemplarily, the data can be organized in the form of data tuples, and each data tuple includes: [optional notifyID, analysis output, corresponding analysis input, feedback information corresponding to the notify ID, optional corresponding true value data].

[0187] Among them, referring to the description of S510, the content of the third information can be flexibly designed and will not be elaborated here.

[0188] 10. The second network device collects the data for generating accuracy information, then calculates the accuracy information and generates an accuracy report. The second network device performs additional processing on the data including the feedback information corresponding to the notify ID according to its internal logic. For example, the second network device can exclude the potentially affected analysis output from the evaluation.

[0189] 11. The second network device sends an accuracy report to the first network device.

[0190] 12. The first network device performs an accuracy evaluation based on the accuracy report to obtain an accuracy evaluation result.

[0191] 13. The first network device sends the accuracy evaluation result to the consumer node.

[0192] Combining the above examples, it is not difficult to see that according to the identification information in the first information and the second information in the embodiments of the present application, it is possible to determine which analysis output the consumer node has taken an execution action based on, and determine the analysis output on which the execution action for accuracy evaluation is based and / or the analysis output affected by it (or the analysis output corresponding to the analysis object affected by the execution action). Furthermore, the reliability of the data used for accuracy evaluation is ensured (for example, excluding the affected analysis output and corresponding accuracy information and other data from the accuracy evaluation, or correcting the affected analysis output and corresponding accuracy information and other data before using them for accuracy evaluation), improving the accuracy of the accuracy evaluation.

[0193] In addition, since the identification information in the first information and the second information can play an indicative role, using the identification information can distinguish the correspondence between the feedback information and the analysis output. This enables the feedback information not to be reported one-to-one corresponding to the analysis output. In other words, the consumer node can report multiple pieces of feedback information at one time. In this way, it is more conducive to efficient information statistics, reducing unnecessary messages between the consumer node and the first network device, and thus reducing communication overhead.

[0194] The above mainly introduces the solution provided by the embodiments of the present application from the perspective of the execution logic of each step. It can be understood that each node, such as the consumer node, includes the corresponding hardware structure and / or software module for implementing the above functions. Those skilled in the art should easily realize that, combined with the algorithm steps of each example described in the embodiments disclosed in this article, the method of the embodiments of the present application can be implemented in the form of hardware, software, or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0195] In the embodiments of the present application, the consumer node can be divided into functional modules according to the above method examples. For example, each functional module can be corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical function division. In actual implementation, there may be other division methods.

[0196] In specific implementation, each network element shown in the present application, such as: the consumer node can adopt Figure 8 the shown composition structure or include Figure 8 the shown components. Figure 8 FIG. is a schematic structural diagram of a communication device provided by an embodiment of the present application. When the communication device has the functions of the consumer node described in the embodiments of the present application, the communication device can be a consumer node or a chip or system-on-chip in the consumer node. When the communication device has the functions of the network device described in the embodiments of the present application, the communication device can be a network device or a chip or system-on-chip in the network device.

[0197] Such as Figure 8 shown, the communication device may include a processor 801, a communication line 802, a transceiver 803, and a memory 804. Among them, the processor 801, the memory 804, and the transceiver 803 can be connected through the communication line 802. In one example, the processor 801 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 in

[0198] As an alternative implementation, the communication device includes multiple processors. For example, in addition to Figure 8 the processor 801 in

[0199] the communication device may further include a processor 807.

[0200] The communication line 802 is used to transmit information between the components included in the communication device.

[0201] A transceiver 803 is used to communicate with other devices or other communication networks. The other communication network can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The transceiver 803 can be an interface circuit, a pin, a radio frequency module, a transceiver, or any device capable of implementing communication.

[0202] Further, the communication device may further include a memory 804. The memory 804 is used to store instructions. Among them, the instructions can be computer programs.

[0203] Among them, the memory 804 can be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or can also be a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions. It can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage, magnetic disk storage media, or other magnetic storage devices. Optical disc storage includes compressed optical discs, laser discs, optical discs, digital versatile discs, or Blu-ray discs, etc.

[0204] It should be noted that the memory 804 can exist independently of the processor 801 or can be integrated with the processor 801. The memory 804 can be used to store instructions, program codes, or some data, etc. The memory 804 can be located inside the communication device or outside the communication device, without limitation. When the processor 801 executes the instructions stored in the memory 804, the method provided in the embodiments of the present application can be implemented.

[0205] As an optional implementation manner, the communication device further includes an output device 805 and an input device 806. Exemplarily, the input device 806 is a device such as a keyboard, a mouse, a microphone, or a joystick, and the output device 805 is a device such as a display screen or a speaker.

[0206] It should be noted that the communication device can be a desktop computer, a portable computer, a network server, a mobile phone, a tablet computer, a wireless terminal, an embedded device, a chip system, or a device with a Figure 8 similar structure among them. In addition, Figure 8 the shown composition structure does not constitute a limitation on the communication device. Except Figure 8In addition to the components shown, the communication device may include more or fewer components than those shown, or combine certain components, or have a different component arrangement.

[0207] In an embodiment of the present application, the chip system may be composed of chips, or may include chips and other discrete devices.

[0208] Figure 9 A structural diagram of a communication device 900 is shown. The communication device is applied to a consumer node. Figure 9 In the device shown, each module has the function of implementing the corresponding step in the method embodiment and can achieve its corresponding technical effect. The beneficial effects corresponding to the steps executed by each module can be referred to the description of the corresponding steps and will not be elaborated. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The communication device may be a consumer node or a chip or system-on-chip in the consumer node. For example, the communication device includes:

[0209] A transceiver module 910, configured to receive first information, where the first information includes first identification information and an analysis output; a processing module 920, configured to determine second information according to the first information, where the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which and / or which affects the execution action for accuracy evaluation, where the execution action is executed based on the analysis output.

[0210] Figure 10 A structural diagram of a communication device 100 is shown. The communication device is applied to a first network device. Figure 10 In the device shown, each module has the function of implementing the corresponding step in the method embodiment and can achieve its corresponding technical effect. The beneficial effects corresponding to the steps executed by each module can be referred to the description of the corresponding steps and will not be elaborated. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The communication device may be a first network device or a chip or system-on-chip in the first network device. For example, the communication device includes:

[0211] A transceiver module 101, configured to send first information to a consumer node, where the first information includes first identification information and an analysis output; the transceiver module 101 is further configured to receive second information from the consumer node, where the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which and / or which affects the execution action for accuracy evaluation, where the execution action is executed based on the analysis output; a processing module 102, configured to determine a target analysis output not affected by the execution action according to the second information; and perform accuracy evaluation according to the accuracy information corresponding to the target analysis output.

[0212] Figure 11 The structure diagram of a communication device 110 is shown. This communication device is applied to a second network device. Figure 11 Each module in the shown device has the functions corresponding to the steps in the method embodiment and can achieve the corresponding technical effects. The beneficial effects corresponding to the steps executed by each module can be referred to the description of the corresponding steps and will not be elaborated here. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. This communication device can be a second network device or a chip or system-on-chip in the second network device. For example, this communication device includes:

[0213] A transceiver module 111, configured to receive third information from a first network device, where the third information is used to determine the accuracy information of a target analysis output, the target analysis output is an analysis output determined according to second information and not affected by an execution action, the second information includes at least one of the following: the first identification information in the first information on which and / or which is affected by the execution action affecting the accuracy evaluation, the first information includes the first identification information and the analysis output, the execution action is executed based on the analysis output, and the third information is used to determine the accuracy information of the analysis output; a processing module 112, configured to determine a target analysis output not affected by the execution action according to the third information; the processing module 112 is further configured to generate the accuracy information of the target analysis output; the transceiver module 111 is further configured to send the accuracy information of the target analysis output to the first network device.

[0214] An embodiment of this application further provides a communication system. This communication system includes a first network device and a consumer node. Among them, the first network device is configured to execute the steps corresponding to each module of the communication device 100, and the consumer node is configured to execute the steps corresponding to each module of the communication device 900. Further, this communication system may further include a second network device, and the second network device is configured to execute the steps corresponding to each module of the communication device 110.

[0215] The embodiments of the present application further provide a computer-readable storage medium. All or part of the processes in the above method embodiments can be completed by a computer program instructing relevant hardware. This program can be stored in the above computer-readable storage medium. When this program is executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be the terminal device in any of the foregoing embodiments, such as an internal storage unit including a data sending end and / or a data receiving end, for example, the hard disk or memory of the terminal device. The above computer-readable storage medium can also be an external storage device of the above terminal device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the above terminal device. Further, the above computer-readable storage medium can also include both the internal storage unit of the above terminal device and the external storage device. The above computer-readable storage medium is used to store the above computer program and other programs and data required by the above terminal device. The above computer-readable storage medium can also be used to temporarily store the data that has been output or will be output.

[0216] The embodiments of the present application further provide a computer instruction. All or part of the processes in the above method embodiments can be completed by a computer instruction instructing relevant hardware (such as a computer, a processor, a network device, and a terminal, etc.). This program can be stored in the above computer-readable storage medium.

[0217] The embodiments of the present application further provide a chip system. The chip system can be composed of chips or can include chips and other discrete devices, without limitation. The chip system includes a processor and a transceiver. All or part of the processes in the above method embodiments can be completed by the chip system. For example, the chip system can be used to implement the functions executed by the consumer node in the above method embodiments, or implement the functions executed by the network device in the above method embodiments.

[0218] In a possible design, the above chip system further includes a memory for storing program instructions and / or data. When the chip system runs, the processor executes the program instructions stored in the memory so that the chip system executes the functions executed by the consumer node in the above method embodiments or executes the functions executed by the network device in the above method embodiments.

[0219] In the embodiments of the present application, the processor may be a general-purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, and may implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0220] In the embodiments of the present application, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or may also be a volatile memory, such as a random-access memory (RAM). The memory is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in the embodiments of the present application may also be a circuit or any other device capable of implementing a storage function, for storing instructions and / or data.

[0221] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0222] In several embodiments provided in the present application, it should be understood that the disclosed device and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other may be through some interfaces. The indirect coupling or communication connection of the device or unit may be in an electrical, mechanical or other form.

[0223] The units described as separate components may or may not be physically separated. The components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place, or may also be distributed to multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, may exist physically alone for each unit, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a device, such as a single-chip microcomputer, a chip, etc., or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0225] As described above, the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A communication method, characterized in that, the method is applied to a first network device and includes: sending first information to a consumer node, where the first information includes first identification information and an analysis output; receiving second information from the consumer node, where the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which and / or which affects the execution action for the accuracy evaluation, where the execution action is executed based on the analysis output.

2. The communication method according to claim 1, characterized in that, the first identification information indicates the first information, or the first identification information indicates the analysis output.

3. The communication method according to claim 1 or 2, characterized in that, the method further includes: determining an analysis output for the accuracy evaluation according to the second information; performing an accuracy evaluation according to the analysis output for the accuracy evaluation.

4. The communication method according to claim 1 or 2, characterized in that, the method further includes: sending third information to a second network device, where the third information is used to determine data for the accuracy evaluation, the third information includes the first information and / or the second information, or the third information includes the analysis output for the accuracy evaluation, and the analysis output for the accuracy evaluation is determined according to the second information; receiving the data for the accuracy evaluation from the second network device; performing an accuracy evaluation according to the data for the accuracy evaluation.

5. The communication method according to claim 4, characterized in that, the first network device is a network data analysis function network element with an analysis logic function, and the second network device is a network data analysis function network element with a model training logic function.

6. The communication method according to any one of claims 1-5, characterized in that, the method further includes: receiving an analysis subscription request from the consumer node before sending the first information to the consumer node.

7. A communication method, characterized in that, the method is applied to a consumer node and includes: receiving first information, where the first information includes first identification information and an analysis output; determining second information according to the first information, where the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which and / or which affects the execution action for the accuracy evaluation, where the execution action is executed based on the analysis output; sending the second information to a first network device.

8. The communication method according to claim 7, characterized in that, the first identification information indicates the first information, or the first identification information indicates the analysis output.

9. The communication method according to claim 7 or 8, characterized in that, the method further includes: sending an analysis subscription request before receiving the first information.

10. A communication method, characterized in that, the method is applied to a second network device and includes: Receive third information from a first network device, where the third information is used to determine data for accuracy evaluation. The third information includes first information and second information, or the third information includes an analysis output for the accuracy evaluation, and the analysis output for the accuracy evaluation is determined according to the second information. The second information includes first identification information in the first information on which and / or which affects the execution action for accuracy evaluation. The first information includes the first identification information and an analysis output, and the execution action is performed based on the analysis output. Determine the data for the accuracy evaluation according to the third information. Send the data for the accuracy evaluation to the first network device.

11. A communication method Characterized in that The method is applied to a first network device and includes: Send first information to a consumer node, where the first information includes an analysis output and second identification information of an analysis object of the analysis output. Receive second information from the consumer node, where the second information is used for accuracy evaluation, and the second information includes the second identification information of the analysis object affected by the execution action, where the execution action is performed based on the analysis output.

12. A communication method Characterized in that The method is applied to a consumer node and includes: Receive first information, where the first information includes an analysis output and second identification information of an analysis object of the analysis output. Determine second information according to the first information, where the second information is used for accuracy evaluation, and the second information includes the second identification information of the analysis object affected by the execution action, where the execution action is performed based on the analysis output. Send the second information to the first network device.

13. A communication device Characterized in that The device is applied to a first network device and includes: A transceiver module for sending first information to a consumer node, where the first information includes first identification information and an analysis output. The transceiver module is further configured to receive second information from the consumer node, where the second information is used for accuracy evaluation, and the second information includes the first identification information in the first information on which and / or which affects the execution action for accuracy evaluation, where the execution action is performed based on the analysis output.

14. The communication device according to claim 13 Characterized in that The first identification information indicates the first information, or the first identification information indicates the analysis output.

15. The communication device according to claim 13 or 14 Characterized in that The device further includes a processing module for: Determine an analysis output for the accuracy evaluation according to the second information. Perform accuracy evaluation according to the analysis output for the accuracy evaluation.

16. The communication device according to claim 13 or 14 Characterized in that The device further includes a processing module; The transceiver module is further configured to send third information to a second network device, where the third information is used to determine data for the accuracy evaluation, the third information includes first information and second information, or the third information includes an analysis output for the accuracy evaluation, and the analysis output for the accuracy evaluation is determined according to the second information; The transceiver module is further configured to receive the accuracy information for the accuracy evaluation from the second network device; The processing module is configured to perform an accuracy evaluation according to the data for the accuracy evaluation.

17. The communication device according to claim 16, wherein, The first network device is a network data analysis function network element with an analysis logic function, and the second network device is a network data analysis function network element with a model training logic function.

18. The communication device according to any one of claims 13-17, wherein, The transceiver module is further configured to receive an analysis subscription request from the consumer node before sending the first information to the consumer node.

19. A communication device, wherein, The device is applied to a consumer node and includes: A transceiver module, configured to receive first information, where the first information includes first identification information and an analysis output; A processing module, configured to determine second information according to the first information, the second information is used for accuracy evaluation, the second information includes the first identification information in the first information on which and / or which affects the execution action for the accuracy evaluation, and the execution action is executed based on the analysis output; The transceiver module is further configured to send the second information to a first network device.

20. The communication device according to claim 19, wherein, The first identification information indicates the first information, or the first identification information indicates the analysis output.

21. The communication device according to claim 19 or 20, wherein, The transceiver module is further configured to send an analysis subscription request before receiving the first information.

22. A communication device, wherein, The device is applied to a second network device and includes: A transceiver module, configured to receive third information from a first network device, where the third information is used to determine data for accuracy evaluation, the third information includes first information and second information, or the third information includes an analysis output for the accuracy evaluation, and the analysis output for the accuracy evaluation is determined according to the second information; the second information includes the first identification information in the first information on which and / or which affects the execution action for accuracy evaluation, the first information includes the first identification information and an analysis output, and the execution action is executed based on the analysis output; A processing module, configured to determine the data for the accuracy evaluation according to the third information; The transceiver module is further configured to send the data for the accuracy evaluation to the first network device.

23. A communication device, It is characterized in that The device is applied to a first network device and includes: A transceiver module, configured to send first information to a consumer node, where the first information includes an analysis output and second identification information of an analysis object of the analysis output; The transceiver module is further configured to receive second information from the consumer node, where the second information is used for accuracy evaluation, and the second information includes the second identification information of the analysis object affected by the execution action, and the execution action is executed based on the analysis output.

24. A communication device It is characterized in that The device is applied to a consumer node and includes: A transceiver module, configured to receive first information, where the first information includes an analysis output and second identification information of an analysis object of the analysis output; A processing module, configured to determine second information according to the first information, where the second information is used for accuracy evaluation, and the second information includes the second identification information of the analysis object affected by the execution action, and the execution action is executed based on the analysis output; The transceiver module is further configured to send the second information to the first network device.

25. A communication device It is characterized in that The communication device includes a processor and a transceiver, and the processor and the transceiver are configured to support the communication device to execute the method according to any one of claims 1-12.

26. A computer-readable storage medium It is characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions run, they execute the method according to any one of claims 1-12.

27. A communication system It is characterized in that The communication system includes a first network device and a consumer node, where the first network device is configured to execute the method according to any one of claims 1-6 and 11, and the consumer node is configured to execute the method according to any one of claims 7-9 and 12.

Images