Network management method and device

Abstract

The embodiment of the invention provides a network management method and device, and relates to the technical field of communication, the network management method is applied to a first node for information prediction in a communication network, and comprises the following steps: obtaining reference data matched with a network management scene and current measurement information, the reference data is used for indicating the prediction quality of at least one prediction model matched with the scene; processing the current measurement information through each prediction model in the at least one prediction model to obtain current prediction information corresponding to the prediction model; and sending the current prediction information corresponding to each prediction model and the quality index of the current prediction information to a second node in the communication network, so that the second node performs network management operation aiming at the scene according to the current prediction information of which the quality index meets the quality condition in the current prediction information corresponding to each prediction model. According to the embodiment of the invention, the prediction information can be subjected to quality evaluation, so that the network performance and the user experience are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communication technology, and in particular, to a network management method and device. Background Art

[0002] Artificial intelligence (AI) is a technology that simulates the human brain to perform complex calculations and is widely used for data analysis. For example, the R17 version of the 3rd Generation Partnership Project (3GPP) proposed to apply AI to the new radio (NR, also known as the new wireless, which is the 5G wireless network) to analyze measurement information to obtain prediction information (such as prediction information of resource status, trajectory prediction information of user equipment (UE)), etc. The prediction information can be used to improve network performance and user experience. Among them, the measurement information may include information measured from the communication network that can be used to obtain prediction information, such as load information, user location and other information.

[0003] In the related technology, measurement information for obtaining prediction information is usually collected by nodes in the communication network (such as NR), such as 5G base stations (gNB), and the collected measurement information is processed based on AI technology (such as a pre-trained machine learning model) to obtain prediction information, and then network management operations such as network energy saving, load balancing, and mobility optimization are performed based on the prediction information.

[0004] However, the prediction information obtained through AI technology is likely to be inaccurate, which can easily lead to inaccurate network management operations based on the prediction information, resulting in failure to improve network performance and user experience. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a network management method, in which nodes of a communication network can determine more accurate prediction information by determining quality indicators of different prediction information to perform more accurate network management operations, thereby improving network performance and user experience.

[0006] In a first aspect, an embodiment of the present application provides a network management method, which is applied to a first node performing information prediction in a communication network, the method comprising: obtaining reference data and current measurement information that match a scenario for network management, wherein the reference data is used to indicate the prediction quality of at least one prediction model that matches the scenario; processing the current measurement information through each prediction model of at least one prediction model respectively to obtain current prediction information corresponding to the prediction model; determining quality indicators of the current prediction information corresponding to at least one prediction model respectively based on the reference data; and performing network management operations for the scenario according to the current prediction information whose quality indicators meet quality conditions.

[0007] In an embodiment of the present application, the first node of the communication network can obtain reference data that matches the scenario managed by the network, and the reference data is used to indicate the prediction quality of at least one prediction model that matches the scenario. Then, the first node can determine the quality index of the current prediction information corresponding to the at least one prediction model based on the reference data, that is, evaluate the quality (such as accuracy and / or precision) of the prediction information currently obtained by each prediction model, and send the evaluation result to the second node of the communication network. Based on this, the second node performs network management operations based on the current prediction information whose quality index meets the quality conditions, which can avoid network operations based on inaccurate prediction information, thereby improving network performance and user experience.

[0008] According to the first aspect, the reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

[0009] In the embodiments of the present application, by presetting a test set, a quality indicator of historical prediction information, historical prediction information, and one of prediction auxiliary information, it is possible to evaluate the prediction information before the actual information corresponding to the predicted network information is generated, thereby ensuring network management based on more accurate prediction information and improving network performance and user experience.

[0010] According to the first aspect, or any implementation method of the first aspect above, the reference information includes a preset test set; based on the reference data, the quality indicators of the current prediction information corresponding to at least one prediction model are determined, including: processing the test measurement information through each prediction model respectively to obtain the test prediction information corresponding to the prediction model; for each prediction model, according to the test prediction information corresponding to the prediction model, the test prediction information corresponding to the test measurement information, and the actual information corresponding to the test prediction information, the quality evaluation of the test prediction information corresponding to the prediction model is performed to obtain the quality indicator of the current prediction information corresponding to the prediction model.

[0011] In the embodiment of the present application, the quality indicator of the preset test set is used to approximately represent the quality indicator of the current prediction information, so that the prediction information can be evaluated before the actual information corresponding to the predicted network information is generated, thereby ensuring network management based on more accurate prediction information and improving network performance and user experience.

[0012] According to the first aspect, or any implementation method of the first aspect above, the reference information includes a preset test set; obtaining reference data matching the network management scenario includes one of the following steps: reading the preset test set stored by the first node; receiving the preset test set sent by the third node, wherein the third node is a node in the communication network different from the first node and the second node; receiving the preset test set sent by the second node.

[0013] In an embodiment of the present application, the first node may obtain a preset test set from the third node or the second node, thereby ensuring that the quality of the current prediction information can be subsequently evaluated based on the preset test set.

[0014] According to the first aspect, or any implementation method of the first aspect above, the reference information includes a preset test set; obtaining reference data matching the network management scenario includes: receiving an original test set sent by a third node, wherein the third node is a node in the communication network that is different from the first node and the second node, the original test set includes multiple test sets, each test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; receiving evaluation requirement information sent by the second node; determining the test set in the original test set that meets the evaluation requirement information as the preset test set.

[0015] In an embodiment of the present application, when the third node sends the original test set, the first node can determine a preset test set that meets the evaluation requirements of the second node from the original test set based on the evaluation requirement information sent by the second node, thereby ensuring that the subsequent quality evaluation of the prediction information based on the preset test set better meets the requirements of the second node.

[0016] According to the first aspect, or any implementation method of the first aspect above, the reference information includes quality indicators of historical prediction information; based on the reference data, the quality indicators of current prediction information corresponding to at least one prediction model are determined, including: for each prediction model, the quality indicators of the historical prediction information corresponding to the prediction model are determined as the quality indicators of the current prediction information corresponding to the prediction model.

[0017] In the embodiment of the present application, the quality indicator of the historical prediction information is used to approximate the quality indicator of the current prediction information, so that the prediction information can be evaluated before the actual information corresponding to the predicted network information is generated, thereby ensuring network management based on more accurate prediction information and improving network performance and user experience.

[0018] According to the first aspect, or any implementation method of the first aspect above, the reference information includes quality indicators of historical prediction information; obtaining reference data that matches the scenario managed by the network includes: obtaining the quality indicators of historical prediction information in a fourth node, wherein the fourth node is a node in the communication network that historically sends prediction information for the scenario to the first node; or, when the first node historically predicts information for the scenario, reading the quality indicators of the historical prediction information stored by the first node itself.

[0019] In an embodiment of the present application, when historical prediction information exists in the fourth node, the quality indicator of the historical network information can be obtained from the fourth node. When historical prediction information exists in the first node itself, the quality indicator of the historical network information stored in the first node can be directly read by the first node, thereby ensuring that the quality indicator of the current prediction information can be determined subsequently through the quality indicator of the historical prediction information.

[0020] According to the first aspect, or any implementation method of the first aspect above, the reference information includes historical prediction information; based on the reference data, the quality index of the current prediction information corresponding to at least one prediction model is determined, including: for each prediction model, the historical prediction information corresponding to the prediction model or the current prediction information corresponding to the prediction model is input into a pre-trained evaluation model to obtain the quality index of the current prediction information corresponding to the prediction model; wherein the evaluation model is a model trained using historical network information and the quality index of the historical network information.

[0021] In the embodiment of the present application, the quality index of the current prediction information is obtained through a pre-trained evaluation model, so that the prediction information can be evaluated before the actual information corresponding to the predicted network information is generated, thereby ensuring network management based on more accurate prediction information and improving network performance and user experience.

[0022] According to the first aspect, or any implementation method of the first aspect above, the reference information includes historical prediction information; obtaining reference data that matches the scenario managed by the network includes: obtaining historical prediction information of the neighboring nodes of the first node and the first node that matches the scenario managed by the network, and quality indicators of the historical prediction information.

[0023] In an embodiment of the present application, the first node can obtain historical prediction information for training an evaluation model from itself and neighboring nodes, that is, nodes adjacent to the first node. In this way, the historical prediction information can be obtained relatively more efficiently, thereby improving the evaluation efficiency of the prediction information.

[0024] According to the first aspect, or any implementation of the first aspect above, after determining the quality indicators of the current prediction information corresponding to at least one prediction model based on the reference data, the method also includes: when sending the current prediction information corresponding to each prediction model to the second node in the communication network, sending a retention indication of the current prediction information corresponding to each prediction model to the second node.

[0025] In the embodiment of the present application, by instructing the second node to retain the received prediction information, it can be ensured that the first node can obtain historical prediction information when performing quality evaluation of the prediction information, thereby ensuring that network management operations are performed based on more accurate prediction information, thereby improving network performance and user experience.

[0026] According to the first aspect, or any implementation of the first aspect above, before obtaining reference data that matches the scenario managed by the network, the method also includes: receiving a prediction evaluation indication sent by the second node, the prediction evaluation indication being used to indicate the second node's demand information for quality indicators of the prediction information; and obtaining reference data that matches the scenario managed by the network according to the prediction evaluation indication.

[0027] In the embodiment of the present application, the prediction evaluation indication can indicate the second node's requirement information for the quality index of the prediction information, so that the reference data matching the network management scenario obtained by the first node according to the prediction evaluation indication is more in line with the second node's quality evaluation requirements for the prediction information. For example, whether to perform quality evaluation, accuracy and / or precision conditions of the prediction information, and other quality evaluation requirements.

[0028] According to the first aspect, or any implementation of the first aspect above, the reference data includes prediction auxiliary information; obtaining reference data matching the scenario managed by the network includes: obtaining prediction auxiliary information according to at least one prediction model; determining the quality index of the current prediction information corresponding to at least one prediction model based on the reference data, including: for each prediction model, determining the attribute information and / or operating environment of the prediction model indicated by the prediction auxiliary information as the quality index of the current prediction information corresponding to the prediction model. .

[0029] In the embodiment of the present application, the first node implicitly represents the quality indicator of the current prediction information through the prediction auxiliary information that matches the network management scenario, thereby ensuring that subsequent network management can be performed based on more accurate current prediction information, thereby improving network performance and user experience.

[0030] In a second aspect, an embodiment of the present application provides a network management method, which is applied to a second node that uses prediction information in a communication network, the method comprising: receiving current prediction information and a quality indicator of the current prediction information corresponding to each prediction model in at least one prediction model sent by a first node that performs information prediction in the communication network; performing network management operations for a scenario based on the current prediction information whose quality indicator in the current prediction information corresponding to each prediction model meets a quality condition; wherein the quality indicator of the current prediction information corresponding to each prediction model is determined by the first node based on reference data matching the scenario of network management, the reference data is used to indicate the prediction quality of at least one prediction model matching the scenario, and the prediction model is used to perform information prediction.

[0031] According to the second aspect, the reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

[0032] According to the second aspect, or any implementation of the second aspect above, the reference data includes a preset test set; before receiving the current prediction information and the quality indicator of the current prediction information corresponding to each prediction model in at least one prediction model sent by the first node performing information prediction in the communication network, the method also includes: sending the preset test set to the first node; or, sending evaluation requirement information to the first node.

[0033] According to the second aspect, or any implementation of the second aspect above, the reference information includes a quality indicator of historical prediction information or historical prediction information; the method also includes: in the case of receiving current prediction information corresponding to each prediction model in at least one prediction model sent by a first node for information prediction in a communication network, also receiving a retention indication for the current prediction information corresponding to each prediction model sent by the first node.

[0034] According to the second aspect, or any implementation method of the second aspect above, the reference information includes quality indicators of historical prediction information or historical prediction information; before receiving current prediction information and quality indicators of current prediction information corresponding to each prediction model in at least one prediction model sent by a first node performing information prediction in a communication network, the method also includes: sending a prediction evaluation indication to the first node, the prediction evaluation indication being used to indicate the second node's requirement information for the quality indicator of the prediction information.

[0035] According to the second aspect, or any implementation method of the second aspect above, the reference data includes prediction auxiliary information; based on the current prediction information corresponding to each prediction model, whose quality indicators meet the quality conditions, network management operations for the scenario are performed, including: determining the prediction requirements for the scenario; for each prediction model, determining the current prediction information corresponding to each prediction model, whose quality indicators meet the quality conditions based on the degree of matching between the prediction auxiliary information corresponding to the prediction model and the prediction requirements; performing network management operations for the scenario based on the current prediction information, whose quality indicators meet the quality conditions.

[0036] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0037] In a third aspect, an embodiment of the present application provides a network management method, which is applied to a communication network, wherein the communication network includes a first node for performing information prediction and a second node for using the prediction information, the method including: a first node, for obtaining reference data and current measurement information that match a scenario for network management, wherein the reference data is used to indicate the prediction quality of at least one prediction model that matches the scenario; processing the current measurement information through each prediction model of at least one prediction model respectively to obtain current prediction information corresponding to the prediction model; determining a quality indicator of the current prediction information corresponding to at least one prediction model respectively based on the reference data; sending the current prediction information corresponding to each prediction model and the quality indicator of the current prediction information to a second node in the communication network; a second node, for performing network management operations for the scenario according to the current prediction information whose quality indicator in the current prediction information corresponding to each prediction model meets the quality conditions.

[0038] According to the third aspect, the reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

[0039] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0040] In a fourth aspect, an embodiment of the present application provides a network management device, which is applied to a first node in a communication network for performing information prediction, the device comprising: an information acquisition module, for acquiring reference data and current measurement information matching a scenario of network management, wherein the reference data is used to indicate the prediction quality of at least one prediction model matching the scenario; processing the current measurement information through each prediction model of at least one prediction model respectively to obtain current prediction information corresponding to the prediction model; a quality evaluation module, for determining, based on the reference data, a quality indicator of current prediction information corresponding to each prediction model respectively; an information sending module, for sending the current prediction information corresponding to each prediction model and the quality indicator of the current prediction information to a second node in the communication network, wherein the second node is used to perform network management operations for the scenario according to the current prediction information whose quality indicator in the current prediction information corresponding to each prediction model meets the quality conditions.

[0041] According to the fourth aspect, the reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

[0042] According to the fourth aspect, or any implementation method of the fourth aspect above, the reference information includes a preset test set; a quality evaluation module is specifically used to: process the test measurement information through each prediction model respectively to obtain the test prediction information corresponding to the prediction model; for each prediction model, according to the test prediction information corresponding to the prediction model, the test prediction information corresponding to the test measurement information and the actual information corresponding to the test prediction information, perform quality evaluation on the test prediction information corresponding to the prediction model to obtain the quality indicator of the current prediction information corresponding to the prediction model.

[0043] According to the fourth aspect, or any implementation method of the fourth aspect above, the reference information includes a preset test set; an information acquisition module, specifically used to perform one of the following steps: reading the preset test set stored by the first node; receiving the preset test set sent by the third node, wherein the third node is a node in the communication network different from the first node and the second node; receiving the preset test set sent by the second node.

[0044] According to the fourth aspect, or any implementation method of the fourth aspect above, the reference information includes a preset test set; an information acquisition module is specifically used to: receive an original test set sent by a third node, wherein the third node is a node in the communication network that is different from the first node and the second node, the original test set includes multiple test sets, each test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; receive evaluation requirement information sent by the second node; and determine the test set in the original test set that meets the evaluation requirement information as the preset test set.

[0045] According to the fourth aspect, or any implementation method of the fourth aspect above, the reference information includes quality indicators of historical prediction information; the quality evaluation module is specifically used to: for each prediction model, determine the quality indicators of the quality indicators of the historical prediction information corresponding to the prediction model as the quality indicators of the current prediction information corresponding to the prediction model.

[0046] According to the fourth aspect, or any implementation method of the fourth aspect above, the reference information includes quality indicators of historical prediction information; the information acquisition module is specifically used to: obtain the quality indicators of historical prediction information in a fourth node, wherein the fourth node is a node in the communication network that historically sends prediction information of a scenario by a first node; or, when the first node historically performs information prediction for a scenario, read the quality indicators of the historical prediction information stored by the first node itself.

[0047] According to the fourth aspect, or any implementation method of the fourth aspect above, the reference information includes historical prediction information; the quality evaluation module is specifically used to: for each prediction model, input the historical prediction information corresponding to the prediction model or the current prediction information corresponding to the prediction model into a pre-trained evaluation model to obtain the quality index of the current prediction information corresponding to the prediction model; wherein the evaluation model is a model trained using historical network information and the quality index of the historical network information.

[0048] According to the fourth aspect, or any implementation method of the fourth aspect above, the reference information includes historical prediction information; the information acquisition module is specifically used to: obtain historical prediction information of the neighboring nodes of the first node and the first node that matches the network management scenario, as well as quality indicators of the historical prediction information.

[0049] According to the fourth aspect, or any implementation method of the fourth aspect above, the information sending module is also used to: when sending the current prediction information corresponding to each prediction model to the second node in the communication network, send a retention indication of the current prediction information corresponding to each prediction model to the second node.

[0050] According to the fourth aspect, or any implementation method of the fourth aspect above, the information acquisition module is specifically used to: receive a prediction evaluation indication sent by the second node, the prediction evaluation indication is used to indicate the second node's demand information for the quality indicator of the prediction information; according to the prediction evaluation indication, obtain reference data matching the network management scenario.

[0051] According to the fourth aspect, or any implementation method of the fourth aspect above, the reference data includes prediction auxiliary information; obtaining reference data matching the scenario managed by the network includes: obtaining prediction auxiliary information according to at least one prediction model; a quality evaluation module is specifically used to: for each prediction model, determine the attribute information and / or operating environment of the prediction model indicated by the prediction auxiliary information as the quality indicator of the current prediction information corresponding to the prediction model.

[0052] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0053] In a fifth aspect, an embodiment of the present application provides a network management device, which is applied to a second node that uses prediction information in a communication network, and the device includes: an information receiving module, which is used to receive current prediction information and a quality indicator of the current prediction information corresponding to each prediction model in at least one prediction model sent by a first node that performs information prediction in the communication network; an information application module, which is used to perform network management operations for a scenario based on the current prediction information corresponding to each prediction model whose quality indicator meets the quality conditions; wherein the quality indicator of the current prediction information corresponding to each prediction model is determined by the first node based on reference data matching the scenario of network management, the reference data is used to indicate the prediction quality of at least one prediction model matching the scenario, and the prediction model is used to perform information prediction.

[0054] According to the fifth aspect, the reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

[0055] According to the fifth aspect, or any implementation method of the fifth aspect above, the reference data includes a preset test set; the device also includes: an information sending module, used to: send the preset test set to the first node before the information receiving module receives the current prediction information and the quality indicator of the current prediction information corresponding to each prediction model in at least one prediction model sent by the first node performing information prediction in the communication network; or, send evaluation requirement information to the first node.

[0056] According to the fifth aspect, or any one of the implementation methods of the fifth aspect above, the reference information includes quality indicators of historical prediction information or historical prediction information; the information receiving module is also used to: when receiving current prediction information corresponding to each prediction model in at least one prediction model sent by a first node for information prediction in a communication network, also receive a retention indication of the current prediction information corresponding to each prediction model sent by the first node.

[0057] According to the fifth aspect, or any implementation method of the fifth aspect above, the reference information includes quality indicators of historical prediction information or historical prediction information; the device also includes: an information sending module, which is used to: before the information receiving module receives the current prediction information and the quality indicator of the current prediction information corresponding to each prediction model in at least one prediction model sent by the first node performing information prediction in the communication network, send a prediction evaluation indication to the first node, and the prediction evaluation indication is used to indicate the second node's demand information for the quality indicator of the prediction information.

[0058] According to the fifth aspect, or any implementation method of the fifth aspect above, the reference data includes prediction auxiliary information; an information application module is specifically used to: determine the prediction requirements for the scenario; for each prediction model, determine the current prediction information corresponding to each prediction model whose quality indicators meet the quality conditions based on the matching degree between the prediction auxiliary information corresponding to the prediction model and the prediction requirements; perform network management operations for the scenario based on the current prediction information whose quality indicators meet the quality conditions.

[0059] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0060] In a sixth aspect, the present application provides a communication network, comprising a first node for performing information prediction and a second node for using the prediction information; the first node is used to obtain reference data and current measurement information that match a scenario managed by the network, wherein the reference data is used to indicate the prediction quality of at least one prediction model that matches the scenario; the current measurement information is processed by each prediction model in at least one prediction model to obtain current prediction information corresponding to the prediction model; based on the reference data, a quality indicator of the current prediction information corresponding to at least one prediction model is determined; the current prediction information corresponding to each prediction model and the quality indicator of the current prediction information are sent to the second node in the communication network; the second node is used to perform network management operations for the scenario according to the current prediction information whose quality indicator in the current prediction information corresponding to each prediction model meets the quality conditions.

[0061] According to the sixth aspect, the reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

[0062] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0063] In the seventh aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; the processor and the memory are connected; the memory is used to store one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement a method as in any one of the first to third aspects and any one of the implementation methods of the first to third aspects.

[0064] In an eighth aspect, an embodiment of the present application provides a computer-readable medium for storing a computer program, wherein the computer program includes instructions for executing the method in any possible implementation of the first to third aspects or the first to third aspects.

[0065] In a ninth aspect, an embodiment of the present application provides a computer program comprising instructions for executing the method in any possible implementation of the first to third aspects or the first to third aspects.

[0066] In a tenth aspect, an embodiment of the present application provides a chip, the chip comprising a processing circuit and a transceiver pin. The transceiver pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in any possible implementation of the first aspect to the third aspect or the first aspect to the third aspect to control the receiving pin to receive a signal and control the sending pin to send a signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0068] Figure 1 It is an application diagram of artificial intelligence in the new air interface;

[0069] Figure 2 It is a structural block diagram of a network device provided in an embodiment of the present application;

[0070] Figure 3a This is one of the structural block diagrams of a network management device provided in an embodiment of the present application;

[0071] Figure 3b This is one of the structural block diagrams of a network management device provided in an embodiment of the present application;

[0072] Figure 3c is a structural block diagram of a communication network provided in an embodiment of the present application;

[0073] Figure 4 This is one of the flow diagrams of a network management method provided in an embodiment of the present application;

[0074] Figure 5a This is one of the example diagrams of a data collection process provided by an embodiment of the present application;

[0075] Figure 5b This is one of the example diagrams of a data collection process provided by an embodiment of the present application;

[0076] Figure 5c This is one of the example diagrams of a data collection process provided by an embodiment of the present application;

[0077] Figure 6 This is one of the flow diagrams of a network management method provided in an embodiment of the present application;

[0078] Figure 7 This is one of the flow diagrams of a network management method provided in an embodiment of the present application;

[0079] Figure 8 This is one of the flow diagrams of a network management method provided in an embodiment of the present application;

[0080] Fig. 9 This is one of the flow charts of a network management method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0082] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0083] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0084] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0085] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0086] In order to facilitate understanding of the embodiments of the present application, some background technologies involved in the embodiments of the present application are first introduced:

[0087] For example, Figure 1 This is an application diagram of artificial intelligence in the new air interface. Figure 1As shown, the R17 version of 3GPP passed the SI (Study Item), proposing to apply artificial intelligence (AI) to NR, and improve network performance and user experience through data collection and data analysis based on artificial intelligence. Among them, data collection includes storing data from nodes such as gNB (5G base station), gNB-CU (centralized unit of 5G base station), gNB-DU (distributed unit of 5G base station), UE (user equipment) or other management entities as a database for AI model training and data analysis and reasoning. The model training node (model training host) performs model training through training data (training data), and gives a relatively optimal AI model, that is, an AI model obtained by training (such as a neural network model, a support vector machine (SVM) model, etc.). The model prediction node (model inference host) uses the AI ​​model to analyze and reason the data provided by the data collection, thereby realizing a reasonable prediction based on AI for network operation to make policy adjustments to the network. Relevant policy adjustments can be planned by the execution (actor) node, that is, the execution entity, and sent to multiple network entities for operation, that is, the application of relevant policies to achieve network management. The specific performance of the network applying relevant policies can be input into the database again for storage, that is, feedback. In one example, the model training node can also further perform model training based on the model performance feedback sent by the model prediction node to obtain a better AI model, and deploy / update the model to ensure that the model prediction node can use a better AI model.

[0088] However, the prediction information obtained through AI technology is likely to be inaccurate, which can easily lead to inaccurate network management operations based on the prediction information, resulting in failure to improve network performance and user experience.

[0089] The embodiment of the present application provides a network management method to solve the above problems. In this method, the nodes of the communication network can obtain reference data that matches the scenario of network management, and the reference data is used to indicate the prediction quality of at least one prediction model that matches the scenario. Then, the nodes of the communication network can determine the quality index of the current prediction information corresponding to the at least one prediction model based on the reference data, that is, evaluate the quality (such as accuracy and / or precision) of the prediction information currently obtained by each prediction model. Based on this, the nodes of the communication network perform network management operations according to the current prediction information whose quality index meets the quality conditions, which can avoid network operations based on inaccurate prediction information, thereby improving network performance and user experience.

[0090] Before describing the technical solution of the embodiment of the present application, the operating platform of the network management method of the embodiment of the present application is first described in conjunction with the accompanying drawings. The embodiment of the present application can be applied to nodes of a communication network. Among them, the communication network can be, for example, a 4G system, a 5G system, an NTN system (non-terrestrial network, non-terrestrial network) or any other communication network involving AI application scenarios. In the specific implementation of the embodiment of the present application, the node can be, for example, a terminal device, a network device or other equipment.

[0091] Among them, the terminal device is a device with wireless transceiver function, and the terminal device can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; it can also be deployed on the water surface (such as ships, etc.); it can also be deployed in the air (such as airplanes, balloons and satellites, etc.). The terminal device can be a mobile phone, a tablet computer (pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical, a wireless terminal device in smart grid, a wireless terminal in transportation safety, a wireless terminal device in smart city, a wireless terminal in smart home, and a user equipment (UE), etc.

[0092] A network device is a device in a wireless network, such as a radio access network (RAN) node that connects a terminal device to a wireless network. RAN nodes may be, for example, gNB, transmission reception point (TRP), evolved node B (eNB), radio network controller (RNC), node B (NB), base station controller (BSC), base transceiver station (BTS), home base station (e.g., home evolved node B, or home node B, HNB), base band unit (BBU), or wireless fidelity (Wifi) access point (AP), integrated access and backhaul (IAB), etc.

[0093] For example, Figure 2 is a structural block diagram of a network device provided in an embodiment of the present application. Figure 2 As shown, in a network structure, the network device may be a centralized unit (central unit, CU) or a distributed unit (distributed unit, DU), or the network device may also be a device including a CU and a DU. CU and DU can be understood as units obtained by dividing the base station from a logical functional perspective. CU and DU can be understood as a division of the base station from a logical functional perspective. Among them, CU and DU can be physically separated or deployed together, and the embodiments of the present application do not specifically limit this. In one example, a CU can be connected to a DU, or multiple DUs can share a CU to save costs and facilitate network expansion.

[0094] Among them, the division of CU and DU can be divided according to the protocol stack. One possible way is to deploy the radio resource control (RRC), service data adaptation protocol stack (SDAP) and packet data convergence protocol (PDCP) layers in the CU, and the remaining radio link control (RLC) layer, media access control (MAC) layer and physical layer in the DU. The embodiment of the present application does not limit the above-mentioned protocol stack division method, and any protocol stack division method that can be used for network devices composed of CU and DU can be used in the present application. In one example, the CU and DU can be connected through the F1 interface (an open ground interface). The CU represents the gNB and is connected to the core network through the Ng interface.

[0095] Still see Figure 2 , the network device in the embodiment of the present application may also refer to a centralized unit control plane (CU-CP) node or a centralized unit user plane (CU-UP) node, or the network device may also be a CU-CP and a CU-UP. Among them, CU-CP is responsible for the control plane functions, mainly including RRC and PDCP-C. PDCP-C is mainly responsible for encryption and decryption, integrity protection, data transmission, etc. of the control plane data. CU-UP is responsible for user plane functions, mainly including SDAP and PDCP-U. SDAP is mainly responsible for processing the data of the core network and mapping the flow to the bearer. PDCP-U is mainly responsible for encryption and decryption, integrity protection, header compression, sequence number maintenance, data transmission, etc. of the data plane. Among them, CU-CP and CU-UP are connected through the E1 interface. CU-CP represents the gNB and is connected to the core network through the Ng interface. It is connected to DU through F1-C (control plane). CU-UP is connected to DU through F1-U (user plane). Of course, another possible implementation is that PDCP-C is also in CU-UP.

[0096] The access network device mentioned in the embodiments of the present application may be a device including at least one of a CU and a DU, or a device including a control plane CU node (CU-CP node), a user plane CU node (CU-UP node) and a DU node.

[0097] The wireless access network equipment and terminal equipment can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; they can also be deployed on the water surface; they can also be deployed on aircraft, balloons and satellites in the air. The embodiments of the present application do not limit the specific application scenarios of the wireless access network equipment and terminal equipment.

[0098] The embodiments of the present application can be applied to downlink signal transmission, uplink signal transmission, and device-to-device (D2D) signal transmission. For downlink signal transmission, the sending device is a wireless access network device, and the corresponding receiving device is a terminal device. For uplink signal transmission, the sending device is a terminal device, and the corresponding receiving device is a wireless access network device. For D2D signal transmission, the sending device is a terminal device, and the corresponding receiving device is also a terminal device. The embodiments of the present application do not limit the transmission direction of the signal.

[0099] The wireless access network equipment and the terminal equipment, as well as the terminal equipment and the terminal equipment, can communicate through the licensed spectrum, can also communicate through the unlicensed spectrum, or can also communicate through the licensed spectrum and the unlicensed spectrum at the same time. The wireless access network equipment and the terminal equipment, as well as the terminal equipment and the terminal equipment, can communicate through the spectrum below 6 gigahertz (GHz), that is, 6G, or can communicate through the spectrum above 6G, or can also communicate using the spectrum below 6G and the spectrum above 6G at the same time. The embodiments of the present application do not limit the spectrum resources used between the wireless access network equipment and the terminal equipment.

[0100] It should be understood that Figure 2 The network device shown is only one example of a network device, and the network device may have more or fewer components than shown, may combine two or more components, or may have a different configuration of components. Figure 2 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0101] For example, Figure 3a This is one of the structural block diagrams of a network management device provided in an embodiment of the present application. Figure 3a As shown, the device is applied to a first node in a communication network for information prediction, and may include: a collection module 301 , an evaluation module 302 , and an application module 303 .

[0102] The information acquisition module 301 is used to acquire reference data and current measurement information matching the scenario managed by the network, wherein the reference data is used to indicate the prediction quality of at least one prediction model matching the scenario; the current measurement information is processed by each prediction model of the at least one prediction model to obtain current prediction information corresponding to the prediction model;

[0103] A quality evaluation module 302, for determining, based on reference data, a quality indicator of current prediction information respectively corresponding to at least one prediction model;

[0104] The information sending module 303 is used to send the current prediction information corresponding to each prediction model and the quality indicator of the current prediction information to the second node in the communication network, wherein the second node is used to perform network management operations for the scenario according to the current prediction information corresponding to each prediction model whose quality indicator meets the quality conditions.

[0105] For example, Figure 3b This is one of the structural block diagrams of a network management device provided in an embodiment of the present application. Figure 3b As shown, the network management device is used for a second node using prediction information in a communication network, and the device includes:

[0106] An information receiving module 304 is used to receive current prediction information and a quality indicator of the current prediction information respectively corresponding to each prediction model in at least one prediction model sent by a first node performing information prediction in a communication network;

[0107] An information application module 305 is used to perform a network management operation for a scenario according to the current prediction information corresponding to each prediction model, the quality indicator of which meets the quality condition;

[0108] Among them, the quality indicator of the current prediction information corresponding to each prediction model is determined by the first node based on reference data matching the scenario managed by the network, the reference data is used to indicate the prediction quality of at least one prediction model matching the scenario, and the prediction model is used to perform information prediction.

[0109] It should be understood that Figure 3a and Figure 3b The illustrated device is merely one example of a network management device, and the network management device may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 3a and Figure 3b The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0110] For example, Figure 3c is a structural block diagram of a communication network provided by an embodiment of the present application. Figure 3c As shown, the communication network includes a first node 306 that performs information prediction and a second node 307 that uses the predicted information;

[0111] The first node 306 is configured to obtain reference data and current measurement information matching a scenario managed by the network, wherein the reference data is used to indicate the prediction quality of at least one prediction model matching the scenario; process the current measurement information through each prediction model in the at least one prediction model to obtain current prediction information corresponding to the prediction model; determine a quality indicator of the current prediction information corresponding to the at least one prediction model based on the reference data; and send the current prediction information corresponding to each prediction model and the quality indicator of the current prediction information to the second node in the communication network;

[0112] The second node 307 is used to perform a network management operation for the scenario according to the current prediction information corresponding to each prediction model, the quality indicator of which meets the quality condition.

[0113] In a specific application, the number of the first node 306 and the second node 307 may be one or more. The first node 306 and the second node 307 may be, for example, terminals or network devices, etc., and the embodiment of the present application does not impose any specific limitation on this.

[0114] It should be understood that Figure 3c The network shown is only one example of a communication network, and the communication network may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 3c The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0115] For ease of understanding, the following Figures 4 to 9 The network management method provided in the embodiment of the present application is described in detail.

[0116] For example, Figure 4 This is one of the flow charts of a network management method provided in the embodiment of the present application. Figure 4 As shown, a network management method provided by an embodiment of the application is applied to a node of a communication network, and the method may include the following steps:

[0117] S401, node 1 obtains reference data matching a scenario managed by a network and current measurement information, wherein the reference data is used to indicate a prediction quality of at least one prediction model matching the scenario.

[0118] The first node and the second node may be specifically referred to above. Figure 2The description of the embodiment is not repeated here. In an optional implementation, the reference data may include one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model. Figures 6 to 9 This will be described in detail.

[0119] For the convenience of understanding and description, the application of AI on the RAN side is still used as an example for explanation. AI can have a variety of application scenarios in the communication network, that is, network management scenarios. Taking RAN3 as an example, the scenario can include network energy saving, load balancing, mobility optimization and other scenarios. The current measurement information that matches the network management scenario is the measurement information used to obtain the current prediction information, and the current prediction information is used to perform network management on the scenario. In other words, the measurement information is the input of the prediction model. Among them, the current time is the time for which the quality evaluation is performed in the embodiment of the present application and the second node requests to use it. The following describes the process of network management in these three scenarios.

[0120] Network energy saving: To address the high energy consumption, CO2 emissions and operating expenditure (OPEX) caused by the rapid growth of base stations in communication networks, network energy saving can be performed, which may involve different layers of the network and the mechanisms operate on different time scales. Cell activation / deactivation is an airspace energy saving scheme that uses hierarchical traffic diversion to reduce the energy consumption of the entire radio access network (RAN). When the expected traffic volume is lower than a fixed threshold, the cell may be shut down and the served users may be diverted to a new target cell. Efficient energy consumption can also be achieved in other ways, such as reducing load, modifying coverage or other RAN configuration adjustments. The best energy saving decision depends on many factors, including the load situation of different RAN nodes, RAN node capabilities, KPI / QoS requirements, number of activated users and UE mobility, cell utilization, etc. In order to avoid serious degradation of network performance and / or degradation of energy efficiency caused by wrong cell shutdown, ML (machine learning) technology can be used to optimize energy saving decisions by analyzing data collected in the RAN network. ML algorithms can predict the energy efficiency and load status of the next cycle for better decision-making on cell activation / deactivation to save energy. Based on the predicted load, the system, i.e., the communication network, can dynamically configure energy-saving strategies (e.g., shutdown timing and granularity, offloading actions) to maintain a balance between system performance and energy efficiency and reduce energy consumption.

[0121] Load balancing: The purpose of load balancing is to evenly distribute the load between cells and cell areas, or to transfer part of the traffic from congested cells or cell congested areas, or to divert users from one cell, cell area, carrier or standard to improve network performance. This can be achieved by optimizing cell reselection / handover parameters (handover threshold, etc.) and handover actions. This optimization automation can provide a high-quality user experience while improving system capacity and minimizing manual intervention in network management and optimization tasks. If the load balancing decision depends on the current / past state of the cell load status, it is easy to cause ping-pong handovers between different cells, cell overload and user service quality degradation due to the rapid changes in the network's traffic load and resource status, especially in scenarios with high mobility and a large number of connections. When load balancing, it is difficult to guarantee the overall network and service performance. For load balancing, UEs in congested cells can be diverted to the target cell through handover procedures or handover configuration adaptation. For example, if UEs with time-varying traffic loads are diverted to the target cell, the target cell may be overloaded by the newly arrived large traffic volume, so it is difficult to determine whether the service performance after the offloading action meets the expected target. In order to solve the above problems, solutions based on prediction models can be introduced to improve load balancing performance. Based on the collection of various measurements and feedback from users and network nodes, historical data, etc., solutions based on prediction models and predicted loads can improve load balancing performance to provide a higher quality user experience and increase system capacity.

[0122] Mobility Optimization: Mobility management is a scheme to ensure service continuity during mobility by minimizing dropped calls, radio link failures (RLF), unnecessary handovers, and ping-pong. For future high-frequency networks, as single-node coverage decreases, the frequency of UE handovers between nodes will become high, especially for high-mobility UEs. In addition, for applications with strict QoS requirements such as reliability and latency, QoE (Quality of Experience) is sensitive to handover performance, so mobility management should avoid handover failures and reduce latency during handovers. In addition, the effectiveness of feedback-based adjustments may be weak due to the randomness and uncertainty of the transmission environment. In addition to the baseline case of mobility, optimization areas for mobility include dual connectivity, CHO, and DAPS, which all have additional aspects to deal with in mobility optimization. Mobility aspects of SON that can be enhanced by using AI / ML include: reducing the possibility of unexpected events related to mobility. For example, such unexpected events can include: intra-system handover too late, intra-system handover too early, intra-system handover to the wrong cell, etc.

[0123] Late intra-system handover: After the UE stays in a cell for a long time, a radio link failure (RLF) occurs; the UE attempts to reestablish the radio link connection in a different cell.

[0124] Too early intra-system handover: RLF occurs shortly after a successful handover from a source cell to a target cell, or a handover fails during the handover process; the UE attempts to reestablish the radio link connection in the source cell.

[0125] Intra-system handover to wrong cell: RLF occurs shortly after a successful handover from a source cell to a target cell, or a handover fails during the handover process; the UE attempts to reestablish a radio link connection in a cell other than the source cell and the target cell.

[0126] Successful switch: During a successful switch, there is a potential problem.

[0127] RAN AI can observe multiple HO (Handover) events with associated parameters, train its ML model with this information, and try to identify the parameter sets that lead to successful HO and the parameter sets that lead to unexpected events.

[0128] UE Location / Mobility / Performance Prediction: Predicting the location of the UE is a key part of mobility optimization, as many mobility-related RRM actions (e.g., selecting a handover target cell) can benefit from the predicted UE location / moving path. UE mobility prediction is also a key factor in optimizing early data forwarding, especially for CHO (Conditional Handover). When a UE is served by certain cells, UE performance prediction is a key factor in determining which is the best moving target to maximize efficiency and performance.

[0129] Traffic Steering: By adjusting the handover trigger point, the best combination of Pcell / PSCell / Scell ​​(primary service cell / primary service cell of the primary cell group (dual connection) / primary auxiliary cell / auxiliary cell) is selected to serve the user, which can achieve efficient resource processing. By providing information related to mobility or dual connection to the RAN node, the existing traffic steering can also be improved.

[0130] For example, before initiating a handover, the source gNB can use the UE performance feedback collected for successful handovers that occurred in the past and received from neighboring gNBs. Similarly, for the dual connectivity case, before triggering the addition of a secondary gNB or triggering an SN change, the eNB can use the information (feedback) received from the gNB in ​​the past to successfully complete the SN addition or SN change procedure.

[0131] In both reported examples, the source RAN node of a mobility event or a RAN node acting as a master node (eNB for EN-DC, gNB for NR-DC) can use feedback received from another RAN node as input to an AI / ML function supporting traffic-related decisions (e.g., selection of target cell in the case of mobility, selection of PSCell / Scell(s) in the other case) in order to optimize future decisions.

[0132] It can be understood that the above-mentioned network management scenarios are all examples. The embodiments of the present application do not limit the specific application scenarios. Any scenario that obtains predictive information through artificial intelligence technology for network management can be applied to the present application.

[0133] Figure 5a This is one of the example diagrams of a data collection process provided by the embodiment of the present application. Figure 5a As shown, the 3GPPRAN3 117th meeting introduced a general new process at the XN port (an open interface for interconnection between NG-RAN nodes, and NG-RAN consists of multiple gNBs connected to the 5G core network) for transmitting data used for artificial intelligence. The new process is called the Data Collection Procedure. This process is independent of the use case, data type, and non-UE related. Among them, being independent of the use case means that the data for artificial intelligence (such as AI / ML (machine learning) data) transmitted by the process can be used for all AI use cases, including but not limited to the three use cases (load balancing, mobility optimization, and energy saving) exemplified in S401 above; being independent of the data type means that the AI / ML data transmitted by the process can include input data, output data, feedback data, etc. of the AI ​​model, such as prediction information and measurement information; being non-UE related means that the process is not associated with a specific UE, but can also be used to transmit data for specific UEs, such as per UE performance feedback information.

[0134] The data reporting methods in the above data collection process can include: one-time report and periodic report. This process also supports partial reporting, that is, when the requested node can provide some of the requested items, it does not default to the process startup failure. Figure 5a NG-RAN node 1 initiates the process by sending a data collection request message to NG-RAN node 2 to start reporting AI / ML related information, or to stop reporting AI / ML related information. After receiving the message, NG-RAN node 2 can perform different actions according to the IE (information element, referred to as information element) in the data collection request message, specifically:

[0135] In case the Registration Request IE is set to "start", the requested data collection report shall be started according to the parameters given in the request;

[0136] In the case where the Registration Request IE is set to "Stop", all cell data collection reports shall be stopped and reporting shall be terminated.

[0137] If NG-RAN node 2 is able to provide all requested information, it shall initiate a data collection report based on the request of NG-RAN node 1 and respond with a data collection response message, i.e. a feedback data collection response message, which may contain the collected data.

[0138] If the NG-RAN node 2 is able to provide some but not all of the requested information (i.e. the procedure supports partial reporting), it shall initiate a data collection report for the requested information received and include the node measurement initiation result or each cell measurement initiation result or both in the data collection response message.

[0139] If there is a reporting period IE in the data collection request, it indicates the period of periodic data collection and reporting, that is, periodic reporting. Figure 5b This is one of the example diagrams of a data collection process provided by the embodiment of the present application. Figure 5b As shown, in the scenario of periodic reporting, NG-RAN node 2 can periodically send a data collection update message to NG-RAN node 1 to feedback the data collected in each period. If the reporting period IE does not exist in the data collection request message, NG-RAN node 2 reports only once by default, that is, a one-time report.

[0140] If the Requested Prediction Time IE in the Data Collection Request message is present, it indicates the specific point in time to which the prediction of the requested information applies. The NG-RAN Node 2 shall take it into account when generating the requested prediction information.

[0141] When starting measurements, the Reporting Characteristics IE in the Data Collection Request indicates the type of object on which the NG-RAN Node 2 should perform measurements or predictions. As of RAN3 120th meeting, the data that has been agreed to be transmitted by the process include: predicted resource status information (predicted radio resource status, predicted number of activated UEs, predicted number of RRC connections), UE performance feedback information (average UE throughput DL, average UE throughput UL, average packet delay, average packet loss). The NG-RAN Node 2 shall include in the Data Collection Update message:

[0142] If the first bit "Predicted Radio Resource Status" of the Report Feature IE included in the AI / ML Information Request message is set to "1", the Radio Resource Status IE is predicted.

[0143] If the second bit "Predicted Activated UE Number" of the Reporting Features IE included in the AI / ML Information Request message is set to "1", the Predicted Activated UE Number IE;

[0144] If the third bit "Predicted RRC (Radio Resource Control) Connection" of the Report Feature IE included in the AI / ML Information Request message is set to "1", the RRC Connection IE is predicted.

[0145] If the fourth bit "Average UE Throughput DL" of the Reporting Characteristics IE included in the AI / ML Information Request message is set to "1", then the "Average UE Throughput DL" IE.

[0146] If the fifth bit "Average UE Throughput UL" ​​of the Reporting Feature IE included in the AI / ML Information Request message is set to "1", the UE Average Throughput UL IE.

[0147] If the sixth bit "Average Packet Delay" of the Reporting Characteristics IE included in the AI / ML Information Request message is set to "1", the Average Packet Delay IE.

[0148] If the seventh bit "Average Packet Loss" of the Reporting Characteristics IE contained in the AI / ML Information Request message is set to "1", the Average Packet Loss IE.

[0149] Figure 5c This is one of the example diagrams of a data collection process provided by the embodiment of the present application. Figure 5c As shown, if the requested data collection report cannot be started, the NG-RAN node 2 should send a data collection failure message with an appropriate cause value, that is, a feedback data collection failure message.

[0150] It is understandable that the transmission method of the above current measurement information can be the above Figure 5a to Figure 5c The method in the embodiment.

[0151] S402, node1 processes current measurement information through each prediction model of at least one prediction model to obtain current prediction information corresponding to the prediction model.

[0152] When the node of the communication network obtains the current measurement information, it can process the current measurement information through each prediction model in at least one prediction model to obtain the current prediction information corresponding to the prediction model. In one example, the node is a first node, and the first node itself can execute S402. In one example, the node is a second node, and S402 can specifically include: the second node processes the current measurement information through the above Figure 5a to Figure 5b The process shown sends the current measurement information to the first node and instructs the first node to perform information prediction. Then the first node processes the current measurement information through each prediction model of at least one prediction model to obtain current prediction information corresponding to the prediction model.

[0153] In a specific application, the current prediction information of the same scene can be obtained through at least one prediction model. These prediction models can be different types of neural network models, and / or models obtained by training the same type of neural network model with different samples. The embodiments of the present application are not limited to this. The specific current prediction information of different scenes may be different. Exemplarily, the prediction information involved in the network energy saving, load balancing, and mobility optimization of the scenarios shown in S401 above may include: resource status prediction information (such as predicted wireless resource status, predicted number of activated UEs and / or predicted number of RRC connections), UE trajectory prediction information (predicted cell identity and corresponding residence time), etc.

[0154] S403, node 1 determines, based on the reference data, a quality indicator of current prediction information corresponding to at least one prediction model.

[0155] In an optional embodiment, the above-mentioned reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein, the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

[0156] It is understandable that for different reference data, the communication network nodes may determine the quality indicators in different ways. In order to facilitate understanding and reasonable layout, Figures 6 to 9 The details are described in detail in the embodiments of the present invention.

[0157] S404, node1 sends the current prediction information corresponding to each prediction model and the quality indicator of the current prediction information to node0.

[0158] When the node of the communication network is a first node, such as node1, S404 may include: the first node sends the quality indicator and the obtained current prediction information to the second node, so that the second node performs network management operations for the scenario according to the current prediction information that the quality indicator meets the quality conditions.

[0159] It is understandable that node1 may carry the quality indicator of the current prediction information when sending the current prediction information, or may send the current prediction information and the quality indicator of the current prediction information separately. The embodiment of the present application does not limit the specific sending method of each information.

[0160] S405, node0 performs a network management operation for the scenario according to the current prediction information corresponding to each prediction model and the current prediction information whose quality indicators meet the quality conditions.

[0161] For example, when the quality indicator includes accuracy, the second node selects the current prediction information with an accuracy greater than the accuracy threshold, and performs network management operations for the scenario, such as load balancing. When the node of the communication network is the second node, the second node itself can execute S404. For example, the above prediction information will be transmitted between gNBs, and the gNB will make corresponding AI strategies based on local information after receiving the prediction information. For example, after node0 receives the load prediction information sent by neighboring nodes node1 / 2 / 3, it determines which neighboring nodes are relatively idle, and combines local load prediction information and UE performance information to select which UEs to switch to which cells for load balancing optimization. For another example, node0 predicts the movement trajectory of the UE and sends it to the target cell in advance for mobility enhancement optimization.

[0162] It is understandable that, when there is one prediction model, the current prediction information may be one, and at this time, the node of the communication network may stop performing network management operations for the scenario when the current prediction information does not meet the quality condition. In an example, at this time, the node of the communication network may also feed back response information that the prediction information is unqualified to the first node to obtain prediction information that meets the quality condition, thereby improving network performance and user experience.

[0163] Combine the following Figures 6 to 9 The above-mentioned S402 to S405 are described in detail.

[0164] Figure 6 This is one of the flow charts of a network management method provided in the embodiment of the present application. Figure 6 As shown, the embodiment of the present application can use the precision and / or accuracy of a specific test set (prediction information of known actual values, such as historical prediction information corresponding to actual information) to approximate the evaluation result of the prediction information, that is, the quality indicator. In this embodiment, the specific structure and form of the node are not limited, for example, it can be a node that can predict or need prediction information, such as gNB, UE, 5GC, OAM, etc.

[0165] In an optional example, the network management method provided in the embodiment of the present application may include:

[0166] S601, node c sends an evaluation pre-configuration message to node 1.

[0167] The third-party node node c, i.e., the third node, sends an evaluation preconfiguration message, i.e., a preset test set, to all nodes node 1, i.e., the first node, which can generate prediction information, and preconfigures test sets (such as test data sets and data set indexes) and evaluation indicators and other data for evaluation for these nodes that can generate prediction information. If S601 is present, S602 is optional. If both S601 and S602 exist, the evaluation preconfiguration message sent by node 0, i.e., the second node in S602 does not need to carry a specific test data set, but only needs to carry a test set index to indicate the test data set stored in node1. At this time, it is equivalent to the second node sending evaluation requirement information (including the test set index) to the first node, so that the first node determines the test set that meets the evaluation requirement information in the original test set stored in itself as the preset test set, i.e., the test data set indicated by the test set index. The third-party node node c can be other nodes in communication networks such as 5GC, OAM, NG-RAN, etc., and the embodiments of the present application do not limit this.

[0168] The above-mentioned evaluation preconfiguration message may specifically include the following information: a test set and identification information, wherein the identification information includes an AI model / task / function identification. Among them, the AI ​​model / task / function identification is used to indicate a specific AI model / task / function. The test set includes: a data set index and a test data set, wherein the test data set includes specific test data, such as the input and output of the AI ​​model, that is, the test measurement information and the test prediction information corresponding to the test measurement information, and the actual information corresponding to the test prediction information. In an optional example, the evaluation preconfiguration message may also include an evaluation indication and / or an evaluation index. Among them, the evaluation indication is used to indicate whether it is necessary to calculate the predicted evaluation result, and the implementation form may be a 1-bit indication information. If the evaluation indication is not carried, it is assumed that calculation is required, that is, quality evaluation is performed. The evaluation index is used to indicate how node1 calculates the predicted evaluation based on the test set, that is, how the quality index is calculated.

[0169] For example, the evaluation index can be one or more of the following indicators: mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), accuracy, precision, recall, and comprehensive evaluation index (F1). Among them, MSE is the ratio of the square of the deviation between the predicted value and the actual value to the number of observations. The larger the error, the larger the value; RMSE is the square root of MSE; MAE is the average of the absolute values ​​of the deviation between the predicted value and the actual value; MAPE is the average percentage of the absolute value of the deviation between the predicted value and the actual value divided by the actual value, which is often used as a statistical indicator to measure the accuracy of predictions, such as time series predictions; accuracy is the number of correctly classified samples divided by the total number of samples. Precision is the number of samples predicted as positive and correct divided by the number of samples predicted as positive. Recall is the number of samples predicted as positive and correct divided by the number of samples labeled as positive. The comprehensive evaluation index is the weighted harmonic average of Precision and Recall, that is, F1=2(Precision+Recall) / (Precision*Recall).

[0170] The embodiment of the present application does not limit the association relationship between the above information. Based on the evaluation pre-configuration message, the node can indicate to node1 which AI model / task / function needs to be based on what data set to perform what evaluation index calculation to predict the evaluation result. AI model / task / function refers to a prediction model, prediction task or prediction function based on artificial intelligence technology. Exemplarily, a possible implementation of the evaluation pre-configuration message is as follows: data set 1, AI model ID List 1, evaluation indicators MSE and MAE. The evaluation pre-configuration message indicates that data set 1 is applicable to the AI ​​model identified as ID List 1 of the AI ​​model, and the evaluation index may be one or more of MSE and MAE. Among them, data set 1 may be a test set, that is, it includes a data set index and a test data set, or, in the following S602, data set 1 may only be a data set index. Similarly, another evaluation pre-configuration message may be: data set 2, AI model ID 2, evaluation indicators MSE, MAE. The other evaluation pre-configuration message indicates that data set 2 is applicable to the AI ​​model identified as ID 2 of the AI ​​model, and the evaluation index may be one or more of MSE and MAE.

[0171] It is understandable that different prediction models / tasks / functions can use different or the same data sets, i.e., test sets, and the evaluation indicators in different evaluation pre-configuration messages can be the same or different. The above evaluation pre-configuration messages are only examples, and the embodiments of the present application do not limit the specific content and specific structure of the evaluation pre-configuration messages, which can be set according to specific application scenarios.

[0172] S602, node0 sends an evaluation pre-configuration message to node1.

[0173] The node node0 receiving the prediction information sends prediction information evaluation auxiliary information to the node node1 generating the prediction information, wherein the prediction information evaluation auxiliary information is also the evaluation pre-configuration message in S602.

[0174] If S601 does not exist, that is, there is no preconfigured test set, then the above-mentioned prediction information evaluation auxiliary information may include: may include the following information: test set and identification information, the identification information includes the AI ​​model / task / function identification. Among them, the AI ​​model / task / function identification is used to indicate a specific AI model / task / function. The test set includes: a data set index and a test data set, and the test data set includes the input and output of the AI ​​model, that is, the test measurement information and the test prediction information corresponding to the test measurement information and the actual information corresponding to the test prediction information. In an optional example, the evaluation preconfiguration message may also include an evaluation indication and / or an evaluation index. Among them, the evaluation indication is used to indicate whether it is necessary to calculate the prediction evaluation result, and the implementation form may be 1-bit indication information. If the evaluation indication is not carried, it is assumed that the calculation is required, that is, the quality evaluation is performed. The evaluation index is used to indicate how node1 calculates the prediction evaluation based on the test set, that is, the calculation method of the quality index. The specific description of the evaluation index is shown in S601. The embodiment of the present application does not limit the association relationship between the above information.

[0175] If S601 exists, that is, the test set is preconfigured, there is no need to transmit the complete test data set, and the above-mentioned prediction information evaluation auxiliary information does not need to carry a specific test data set, but only needs to carry a test set index to indicate the test data set stored in node1. Information other than the test set index in the prediction information evaluation auxiliary information can refer to the description in S601 or the description of the prediction information evaluation auxiliary information when S601 does not exist, which will not be repeated here. The embodiment of the present application does not limit the association relationship of the above information.

[0176] In an optional implementation, the process of sending the evaluation auxiliary information of the prediction information may include one of the following (1) and (2):

[0177] (1) In the data collection request message (see above for details) Figure 5a to Figure 5cThe prediction information evaluation auxiliary information is carried in the description of the embodiment, that is, the test set and the evaluation index are specified when the prediction information is requested. Different test sets and the same / different evaluation indexes can be specified when requesting different prediction information, or the same test set and evaluation index can be specified when requesting all prediction information.

[0178] (2) Use a new process to transmit the prediction information evaluation auxiliary information. That is, a new process different from the process of the data collection request message is added to transmit the prediction information evaluation auxiliary information.

[0179] S603, node1 sends prediction information and evaluation results to node0.

[0180] In one example, node1 may carry the evaluation result when sending prediction information to node0.

[0181] Node 1 calculates the prediction information to be sent in S603, that is, the evaluation result of the current prediction information, locally based on the evaluation preconfiguration message, and sends it to node 0. Optionally, at least one of the following data may also be sent in S603: AI model / task / function identifier, data set index, and evaluation indicator identifier.

[0182] In an embodiment of the present application, the node that generates prediction information calculates the evaluation result of the prediction information by obtaining a specific test set. The node that requests the prediction information can obtain the corresponding evaluation index while obtaining the prediction information, and then evaluate the credibility / precision / accuracy of the prediction information, which helps to improve the accuracy of AI decisions related to the prediction information and further improve the accuracy of AI services.

[0183] Figure 7 This is one of the flow charts of a network management method provided in the embodiment of the present application. Figure 7 As shown, the embodiment of the present application can use the precision and / or accuracy of historical prediction information to approximate the evaluation result of the prediction information. In this embodiment, there is no restriction on the node, which can be a node such as gNB, UE, 5GC, OAM, etc. that can predict or need prediction information. Node i in this embodiment refers to a node that can have actual information / evaluation results of historical prediction information.

[0184] In an optional example, if the actual information / evaluation result of the historical prediction information comes from node i, that is, the fourth node, for example, node 1 has previously sent UE trajectory prediction to node i in the past, after the UE switches to node i, node i will obtain the actual trajectory of the UE, or calculate the evaluation result of the UE trajectory prediction, then the following S702-704 needs to be executed.

[0185] In an optional example, if the actual information / evaluation result of the historical prediction information comes from node 1, which is the fourth node. For example, node 1 has previously predicted the local load. When the actual time corresponding to the predicted historical prediction information is reached, node 1 will obtain the local historical actual load, or calculate the evaluation result of the load prediction. That is, node i is node 1, and there is no need to execute the following S703-704.

[0186] Specific:

[0187] In an optional example, the above method may include:

[0188] S701, node0 sends a prediction evaluation indication message to node1 to indicate whether a prediction evaluation result is needed when predicting information is requested. The prediction evaluation indication message can be a 1-bit indication message or a specific evaluation result indicator. Optionally, node0 can also indicate the minimum accuracy of the prediction information. If the requirement is not met, node1 does not need to send the prediction information, but instead indicates to node0 that the prediction information request failed and the reason (for example, the accuracy of the prediction information does not meet the requested minimum accuracy). It can be understood that in one example, S701 may not be executed.

[0189] The method of sending the prediction evaluation indication information is the same as above Figure 6 The method of sending the evaluation auxiliary information of the prediction information in the embodiment is similar, the difference is that the specific information sent is different. For the same part, please refer to the above Figure 6 The method of sending the evaluation auxiliary information of the prediction information in the embodiment will not be described in detail here.

[0190] In an optional example, the above method may include S702, node1 sends prediction information and a prediction information retention indication to node i. Exemplarily, the prediction information retention indication may include: a time period for retaining the prediction information, or a time period for retaining the prediction evaluation result and an evaluation index indication (indicating which evaluation index to select). It may be possible to specify different retention time periods / evaluation indexes when requesting different prediction information, or it may be possible to specify the same retention time period / evaluation index when requesting all prediction information. Any format and / or content of indication information for indicating the retention of the sent prediction information may be used in the present application, and the embodiments of the present application are not limited to this.

[0191] S703, node1 sends a historical prediction evaluation request to node i. The historical prediction evaluation request is equivalent to a historical prediction evaluation request message, which is used by node1 to request the evaluation result of the prediction information previously sent by node i. The historical prediction evaluation request message may include at least one of the following: AI model / task / function identifier, UE identifier list, cell identifier list, and node identifier list. Specifically:

[0192] AI model / task / function identifiers are used to indicate specific AI models / tasks / functions.

[0193] UE identification list is used to indicate which UEs' historical prediction evaluation results are requested.

[0194] The cell identifier list is used to indicate which cells' historical prediction evaluation results are requested.

[0195] The node identifier list is used to indicate which nodes' historical prediction evaluation results are requested.

[0196] Time index, which can be a time point or time period. It is used to indicate when the historical prediction evaluation results are requested.

[0197] The historical prediction information identifier list, such as resource status prediction, UE trajectory prediction, UE performance prediction, etc., is used to indicate which prediction item's historical prediction evaluation results are requested.

[0198] Evaluation indicators are used to indicate the calculation method of the evaluation results of the request history prediction, which can be one or more of the following indicators: mean squared error (MSE), root mean square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), accuracy, precision, recall, and comprehensive evaluation index (F1).

[0199] Node 0 identifier. It is used to indicate that the historical prediction evaluation result requested by node i is sent to node 0. Accordingly, the corresponding reason may also be carried, for example, the requested historical prediction evaluation result is the evaluation result as the latest prediction information.

[0200] The embodiment of the present application does not limit the association relationship of the above information.

[0201] S704, node i feeds back the historical prediction evaluation results to node 1.

[0202] In one example, node i can directly send the historical prediction evaluation result to node 0. The historical prediction evaluation result is the quality indicator of the historical prediction information, which can be the actual value of the historical prediction information or the evaluation result of the historical prediction information obtained by calculation. The calculation method of the evaluation result of the historical prediction information is the same as that of the above Figure 6 The calculation method of the evaluation result of the current prediction information in the embodiment is similar, the difference is that the prediction information is different. For the same part, please refer to the above Figure 6 The description of the embodiment is not repeated here. In an optional example, the historical prediction evaluation result may further include at least one of the following: an AI model identifier, a prediction task identifier, a function identifier, a UE identifier, a time index, and a historical prediction information identifier.

[0203] The embodiment of the present application does not limit the association relationship of the above information.

[0204] S705, node1 sends prediction information and evaluation results to node0, as well as a prediction information retention instruction

[0205] In one example, when node1 sends prediction information to node0, it carries the evaluation result of the above historical prediction information as the evaluation result of the current prediction information. Optionally, when node1 sends prediction information to node0, it can also carry at least one of the following: AI model / task / function identifier, historical prediction related information (UE identifier, time index, node i identifier), evaluation index identifier, and prediction information retention indication. That is, in one example, node1 may not send a prediction information retention indication.

[0206] It is understandable that S702 may be executed before or after S701.

[0207] In the embodiment of the present application, the node that generates prediction information obtains the accuracy of historical prediction information to approximate the evaluation result of the prediction information. The node that requests prediction information can obtain the corresponding evaluation index while obtaining the prediction information, and then evaluate the credibility / precision / accuracy of the prediction information, which helps to improve the accuracy of AI decisions related to the prediction information and further improve the accuracy of AI services.

[0208] Figure 8 This is one of the flow charts of a network management method provided in the embodiment of the present application. Figure 8As shown, the embodiment of the present application uses the AI ​​evaluation model to infer the evaluation result of the prediction information. In this embodiment, there is no restriction on the node, which can be a node that can predict or need prediction information, such as gNB, UE, 5GC, OAM, etc. Node i in this embodiment refers to a node that can have actual information / evaluation results of historical prediction information.

[0209] S801, node0 sends a prediction evaluation instruction to node1.

[0210] S802, node 1 sends prediction information and a prediction information retention indication to node i.

[0211] The above S801 to S802 are the same as S701 to S702, and will not be repeated here. Figure 7 Description of embodiments.

[0212] S803, node 1 sends a historical prediction evaluation request to node i.

[0213] The above S803 is similar to the above S703, except that S803 collects historical prediction information for training to obtain an AI evaluation model, which does not need to be sent to node 0 and does not need to carry the node 0 identifier.

[0214] S804, node i feeds back the historical prediction evaluation results to node 1.

[0215] The above S804 is the same as the above S704, and will not be repeated here. Figure 7 Description of embodiments.

[0216] It can be understood that S802 to S804 can be executed before or after S801.

[0217] S805, node 1 obtains the evaluation result of the prediction information through the evaluation model obtained by pre-training.

[0218] Node1 collects the evaluation results of neighboring nodes and local historical prediction information as AI training data, which is used to train a local AI evaluation model, that is, an evaluation model, to obtain the evaluation results of the prediction information. In one possible implementation, after node1 infers the prediction information through the local AI model, it can process the prediction information through the local AI evaluation model to infer the evaluation results of the prediction information. Among them, a possible implementation method of inferring the evaluation results of the prediction information through the AI ​​evaluation model is as follows:

[0219] The input of the AI ​​evaluation model: output data, i.e. prediction information, and optionally, input data, i.e. current measurement information.

[0220] Output of AI evaluation model: evaluation results.

[0221] That is, in one example, node 1 can input current prediction information and current measurement information into the AI ​​evaluation model to obtain an evaluation result. It can be understood that, at this time, the training data used to train the AI ​​evaluation model also includes historical measurement information.

[0222] S806, node1 sends the prediction information and evaluation results, as well as a prediction information retention indication to node0.

[0223] When node1 sends prediction information to node0, it carries the evaluation result of the above historical prediction information as the evaluation result of the current prediction information. Optionally, when node1 sends prediction information to node0, it may also carry at least one of the following: AI model / task / function identifier, AI evaluation model identifier, evaluation index identifier, prediction information retention indication. That is, in one example, node1 may not send a prediction information retention indication.

[0224] In an embodiment of the present application, the node that generates prediction information infers the evaluation result of the prediction information, and the node that requests the prediction information can obtain the corresponding evaluation index while obtaining the prediction information, and then evaluate the credibility / precision / accuracy of the prediction information, which helps to improve the accuracy of AI decisions related to the prediction information and further improve the accuracy of AI business.

[0225] Fig. 9 This is one of the flow charts of a network management method provided in the embodiment of the present application. Fig. 9 As shown, the embodiment of the present application can use AI auxiliary information, that is, prediction auxiliary information, to implicitly represent the evaluation result of the prediction information. For example, the prediction information from the AI ​​service similar to the AI ​​service in the scene is assumed to be more accurate. In this embodiment, there is no restriction on the node, which can be a node such as gNB, UE, 5GC, OAM, etc. that can predict or need prediction information. Node i in this embodiment refers to a node that can have actual information / evaluation results of historical prediction information.

[0226] The main process is as follows:

[0227] S901, node1 sends prediction information and AI auxiliary information to node0.

[0228] The node node1 that generates prediction information carries the AI ​​auxiliary information when sending the prediction information to the node node0 that receives the prediction information. The AI ​​auxiliary information is used to implicitly indicate the evaluation result of the prediction information.

[0229] The AI ​​auxiliary information may be one or more of the following: AI business identification / usable scenarios (belonging to the operating environment / model attributes), AI model identification (belonging to the model attributes), AI function identification (belonging to the model attributes), local data characteristics (the degree of data non-IID, which is related to the generalization ability of the AI ​​model) (belonging to the operating environment), identification of the equipment vendor to which the model can be applied (belonging to the operating environment), model input and output parameters (belonging to the model attributes), training parameters (belonging to the model attributes) and other model parameters.

[0230] S902, node0 obtains evaluation results of prediction information based on AI auxiliary information.

[0231] After receiving the AI ​​auxiliary information, node0 analyzes the evaluation results of the prediction information based on the prediction requirements of the network management scenario. The above S902, that is, Node0 determines the current prediction information corresponding to each prediction model whose quality indicators meet the quality conditions according to the matching degree between the prediction auxiliary information corresponding to the prediction model and the prediction requirements.

[0232] For example, AI assisted information 1 from node1 includes: mobility optimization AI service; AI assisted information 2 from node2 includes: load balancing AI service. The prediction information comes from node11 / node12, including: UE's QoE prediction. Node0 is doing the UE's mobility optimization service, that is, the predicted demand of the second node is the UE's mobility optimization service, and it is expected to determine the switching strategy of the UE based on the evaluation result of the prediction information. Based on AI assisted information 1 and AI assisted information 12, nde0 determines that AI assisted information 1 is more matched with the predicted demand of node0, and therefore determines that the prediction information of node1 is more accurate, thereby switching the UE to node1.

[0233] It is understandable that node1 and node2 are both first nodes. In one example, the above two types of AI-assisted information may also come from the same first node. The embodiment of the present application does not limit the number of first nodes and whether the AI-assisted information comes from the same node, which depends on the specific application requirements.

[0234] In an embodiment of the present application, a node requesting prediction information can obtain corresponding AI auxiliary information while obtaining the prediction information to approximate the evaluation result, and then evaluate the credibility / precision / accuracy of the prediction information, which helps to improve the accuracy of AI decisions related to the prediction information and further improve the accuracy of AI services.

[0235] It is understandable that, without the need to evaluate the quality of the prediction information based on the current prediction information, the execution order of the above S402 and S403 can be arbitrary, that is, S402 and S403 can be executed sequentially or simultaneously, and the embodiments of the present application do not limit this.

[0236] In an embodiment of the present application, a node of a communication network can obtain reference data that matches a scenario managed by the network, and the reference data is used to indicate the prediction quality of at least one prediction model that matches the scenario. Then, based on the reference data, the node of the communication network can determine the quality index of the current prediction information corresponding to the at least one prediction model, that is, evaluate the quality (such as accuracy and / or precision) of the prediction information currently obtained by each prediction model. Based on this, the node of the communication network performs network management operations based on the current prediction information whose quality index meets the quality conditions, which can avoid network operations based on inaccurate prediction information, thereby improving network performance and user experience.

[0237] In addition, this application Figure 3a to Figure 3c The network management device and communication network shown respectively include hardware and / or software modules corresponding to the execution of each function in order to realize the functions of the network management method in the above-mentioned embodiments of the present application. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0238] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the network management method in the above-mentioned embodiment.

[0239] This embodiment further provides a computer program product. When the computer program product runs on a computer, the computer is enabled to execute the above-mentioned related steps to implement the network management method in the above-mentioned embodiment.

[0240] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0241] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0242] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0243] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A network management method, characterized in that: Applied to a first node performing information prediction in a communication network, the method comprises: Acquire reference data matching a scenario managed by the network and current measurement information, wherein the reference data is used to indicate a prediction quality of at least one prediction model matching the scenario; Processing the current measurement information through each prediction model of the at least one prediction model respectively to obtain current prediction information corresponding to the prediction model; Based on the reference data, determining a quality indicator of current prediction information respectively corresponding to the at least one prediction model; The current prediction information corresponding to each prediction model and the quality indicator of the current prediction information are sent to a second node in the communication network, wherein the second node is used to perform network management operations for the scenario according to the current prediction information corresponding to each prediction model whose quality indicator meets the quality conditions.

2. The method according to claim 1, characterized in that: The reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

3. The method according to claim 2, characterized in that The reference information includes the preset test set; The determining, based on the reference data, a quality indicator of current prediction information respectively corresponding to the at least one prediction model comprises: Processing the test measurement information through each prediction model respectively to obtain test prediction information corresponding to the prediction model; For each prediction model, quality evaluation is performed on the test prediction information corresponding to the prediction model based on the test prediction information corresponding to the prediction model, the test prediction information corresponding to the test measurement information, and the actual information corresponding to the test prediction information to obtain a quality indicator of the current prediction information corresponding to the prediction model.

4. The method according to claim 2 or 3, characterized in that: The reference information includes the preset test set; The step of obtaining reference data matching the network management scenario includes one of the following steps: Reading the preset test set stored in the first node; Receiving the preset test set sent by a third node, wherein the third node is a node in the communication network different from the first node and the second node; Receive the preset test set sent by the second node.

5. The method according to claim 2 or 3, characterized in that: The reference information includes the preset test set; The obtaining of reference data matching the network management scenario includes: receiving an original test set sent by a third node, wherein the third node is a node different from the first node and the second node in the communication network, and the original test set includes a plurality of test sets, each test set including test measurement information, test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; receiving evaluation requirement information sent by the second node; A test set in the original test set that meets the evaluation requirement information is determined as the preset test set.

6. The method according to claim 2, characterized in that The reference information includes a quality indicator of the historical prediction information; The determining, based on the reference data, a quality indicator of current prediction information respectively corresponding to the at least one prediction model comprises: For each prediction model, a quality indicator corresponding to the prediction model among the quality indicators of the historical prediction information is determined as the quality indicator of the current prediction information corresponding to the prediction model.

7. The method according to claim 2 or 6, characterized in that: The reference information includes a quality indicator of the historical prediction information; The obtaining of reference data matching the network management scenario includes: Acquire a quality indicator of the historical prediction information in a fourth node, wherein the fourth node is a node in the communication network to which the first node historically sent the prediction information of the scenario; or, In the case where the first node historically predicts information for the scenario, a quality indicator of the historical prediction information stored by the first node itself is read.

8. The method according to claim 2, characterized in that: The reference information includes the historical prediction information; The determining, based on the reference data, a quality indicator of current prediction information respectively corresponding to the at least one prediction model comprises: For each prediction model, the historical prediction information corresponding to the prediction model or the current prediction information corresponding to the prediction model is input into a pre-trained evaluation model to obtain a quality indicator of the current prediction information corresponding to the prediction model; wherein the evaluation model is a model trained using the historical network information and the quality indicator of the historical network information.

9. The method according to claim 2 or 8, characterized in that: The reference information includes the historical prediction information; The obtaining of reference data matching the network management scenario includes: Obtain neighboring nodes of the first node and historical prediction information in the first node that matches the scenario managed by the network, as well as a quality indicator of the historical prediction information.

10. The method according to any one of claims 6 to 9, characterized in that After determining the quality index of the current prediction information respectively corresponding to the at least one prediction model based on the reference data, the method further includes: In the case of sending the current prediction information corresponding to each prediction model to the second node in the communication network, a retention indication of the current prediction information corresponding to each prediction model is sent to the second node.

11. The method according to any one of claims 6 to 10, characterized in that The obtaining of reference data matching the network management scenario includes: receiving a prediction evaluation indication sent by the second node, where the prediction evaluation indication is used to indicate requirement information of the second node for a quality indicator of prediction information; According to the prediction evaluation indication, reference data matching the scenario of network management is obtained.

12. The method according to claim 2, characterized in that: The reference data includes the prediction auxiliary information; The obtaining of reference data matching the network management scenario includes: Acquiring the prediction auxiliary information according to the at least one prediction model; The determining, based on the reference data, a quality indicator of current prediction information respectively corresponding to the at least one prediction model comprises: For each prediction model, the attribute information and / or the operating environment of the prediction model indicated by the prediction auxiliary information is determined as a quality indicator of the current prediction information corresponding to the prediction model.

13. A network management method, characterized in that: Applied to a second node using prediction information in a communication network, the method comprises: Receiving current prediction information corresponding to each prediction model in at least one prediction model sent by a first node performing information prediction in the communication network and a quality indicator of the current prediction information; Performing a network management operation for the scenario according to the current prediction information corresponding to each prediction model, in which the quality indicator meets the quality condition; Among them, the quality indicator of the current prediction information corresponding to each prediction model is determined by the first node based on reference data matching the scenario managed by the network, and the reference data is used to indicate the prediction quality of at least one prediction model matching the scenario, and the prediction model is used to perform information prediction.

14. The method according to claim 13, characterized in that The reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

15. The method according to claim 14, characterized in that The reference data includes the preset test set; Before receiving the current prediction information and the quality indicator of the current prediction information respectively corresponding to each prediction model in at least one prediction model sent by the first node performing information prediction in the communication network, the method further includes: Sending the preset test set to the first node; or, Send evaluation requirement information to the first node.

16. The method according to claim 14, characterized in that The reference information includes a quality indicator of the historical prediction information or the historical prediction information; The method further comprises: In the case of receiving current prediction information corresponding to each prediction model in at least one prediction model sent by a first node performing information prediction in the communication network, a retention indication for the current prediction information corresponding to each prediction model is also received from the first node.

17. The method according to claim 14 or 16, characterized in that The reference information includes a quality indicator of the historical prediction information or the historical prediction information; Before receiving the current prediction information and the quality indicator of the current prediction information respectively corresponding to each prediction model in at least one prediction model sent by the first node performing information prediction in the communication network, the method further includes: A prediction evaluation indication is sent to the first node, where the prediction evaluation indication is used to indicate the second node's requirement information for a quality indicator of prediction information.

18. The method according to claim 14, characterized in that The reference data includes the prediction auxiliary information; The performing a network management operation for the scenario according to the current prediction information corresponding to each prediction model, in which the quality indicator meets the quality condition, includes: determining the forecasted demand for the scenario; For each prediction model, according to the matching degree between the prediction auxiliary information corresponding to the prediction model and the prediction demand, determine the current prediction information corresponding to each prediction model, in which the quality indicator meets the quality condition; According to the current prediction information that the quality indicator meets the quality condition, a network management operation for the scenario is performed.

19. A network management method, characterized in that: Applied to a communication network, the communication network includes a first node that performs information prediction and a second node that uses the predicted information, the method includes: The first node is used to obtain reference data and current measurement information matching a scenario managed by the network, wherein the reference data is used to indicate the prediction quality of at least one prediction model matching the scenario; process the current measurement information through each prediction model of the at least one prediction model respectively to obtain current prediction information corresponding to the prediction model; determine the quality index of the current prediction information corresponding to each of the at least one prediction model based on the reference data; and send the current prediction information corresponding to each prediction model and the quality index of the current prediction information to the second node in the communication network; The second node is used to perform network management operations for the scenario according to the current prediction information corresponding to each prediction model, in which the quality indicator meets the quality condition.

20. The method according to claim 19, characterized in that The reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

21. A network management device, characterized in that: Applied to a first node for information prediction in a communication network, the device comprises: An information acquisition module, configured to acquire reference data and current measurement information matching a scenario managed by the network, wherein the reference data is used to indicate a prediction quality of at least one prediction model matching the scenario; and to process the current measurement information through each prediction model of the at least one prediction model to obtain current prediction information corresponding to the prediction model; A quality evaluation module, used to determine, based on the reference data, a quality indicator of current prediction information respectively corresponding to the at least one prediction model; An information sending module is used to send the current prediction information corresponding to each prediction model and the quality indicator of the current prediction information to the second node in the communication network, wherein the second node is used to perform network management operations for the scenario according to the current prediction information corresponding to each prediction model whose quality indicator meets the quality conditions.

22. The device according to claim 21, characterized in that The reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

23. The device according to claim 22, characterized in that The reference information includes the preset test set; The quality evaluation module is specifically used for: Processing the test measurement information through each prediction model respectively to obtain test prediction information corresponding to the prediction model; For each prediction model, quality evaluation is performed on the test prediction information corresponding to the prediction model based on the test prediction information corresponding to the prediction model, the test prediction information corresponding to the test measurement information, and the actual information corresponding to the test prediction information to obtain a quality indicator of the current prediction information corresponding to the prediction model.

24. The device according to claim 22 or 23, characterized in that The reference information includes the preset test set; The information acquisition module is specifically used to perform one of the following steps: Reading the preset test set stored in the first node; Receiving the preset test set sent by a third node, wherein the third node is a node in the communication network different from the first node and the second node; Receive the preset test set sent by the second node.

25. The device according to claim 22 or 23, characterized in that The reference information includes the preset test set; The information acquisition module is specifically used for: receiving an original test set sent by a third node, wherein the third node is a node different from the first node and the second node in the communication network, and the original test set includes a plurality of test sets, each test set including test measurement information, test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; receiving evaluation requirement information sent by the second node; A test set in the original test set that meets the evaluation requirement information is determined as the preset test set.

26. The device according to claim 22, characterized in that The reference information includes a quality indicator of the historical prediction information; The quality evaluation module is specifically used for: For each prediction model, a quality indicator corresponding to the prediction model among the quality indicators of the historical prediction information is determined as the quality indicator of the current prediction information corresponding to the prediction model.

27. The device according to claim 22 or 26, characterized in that The reference information includes a quality indicator of the historical prediction information; The information acquisition module is specifically used for: Acquire a quality indicator of the historical prediction information in a fourth node, wherein the fourth node is a node in the communication network to which the first node historically sent the prediction information of the scenario; or, In the case where the first node historically predicts information for the scenario, a quality indicator of the historical prediction information stored by the first node itself is read.

28. The device according to claim 22, characterized in that The reference information includes the historical prediction information; The quality evaluation module is specifically used for: For each prediction model, the historical prediction information corresponding to the prediction model or the current prediction information corresponding to the prediction model is input into a pre-trained evaluation model to obtain a quality indicator of the current prediction information corresponding to the prediction model; wherein the evaluation model is a model trained using the historical network information and the quality indicator of the historical network information.

29. The device according to claim 22 or 28, characterized in that The reference information includes the historical prediction information; The information acquisition module is specifically used for: Obtain neighboring nodes of the first node and historical prediction information in the first node that matches the scenario managed by the network, as well as a quality indicator of the historical prediction information.

30. The device according to any one of claims 26 to 29, characterized in that The information sending module is also used for: In the case of sending the current prediction information corresponding to each prediction model to the second node in the communication network, a retention indication of the current prediction information corresponding to each prediction model is sent to the second node.

31. The device according to any one of claims 26 to 30, characterized in that The information acquisition module is specifically used for: receiving a prediction evaluation indication sent by the second node, where the prediction evaluation indication is used to indicate requirement information of the second node for a quality indicator of prediction information; According to the prediction evaluation indication, reference data matching the scenario of network management is obtained.

32. The device according to claim 22, characterized in that The reference data includes the prediction auxiliary information; The obtaining of reference data matching the network management scenario includes: Acquiring the prediction auxiliary information according to the at least one prediction model; The quality evaluation module is specifically used for: For each prediction model, the attribute information and / or the operating environment of the prediction model indicated by the prediction auxiliary information is determined as a quality indicator of the current prediction information corresponding to the prediction model.

33. A network management device, characterized in that: Applied to a second node using prediction information in a communication network, the device comprises: An information receiving module, configured to receive current prediction information corresponding to each prediction model in at least one prediction model sent by a first node performing information prediction in the communication network and a quality indicator of the current prediction information; An information application module, configured to perform a network management operation for the scenario according to the current prediction information corresponding to each prediction model, in which the quality indicator meets the quality condition; Among them, the quality indicator of the current prediction information corresponding to each prediction model is determined by the first node based on reference data matching the scenario managed by the network, and the reference data is used to indicate the prediction quality of at least one prediction model matching the scenario, and the prediction model is used to perform information prediction.

34. The device according to claim 33, characterized in that The reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

35. The device according to claim 34, characterized in that The reference data includes the preset test set; The device also includes: an information sending module, which is used to: Before the information receiving module receives the current prediction information and the quality indicator of the current prediction information respectively corresponding to each prediction model in at least one prediction model sent by the first node performing information prediction in the communication network, Sending the preset test set to the first node; or, Send evaluation requirement information to the first node.

36. The device according to claim 34, characterized in that The reference information includes a quality indicator of the historical prediction information or the historical prediction information; The information receiving module is further used for: In the case of receiving current prediction information corresponding to each prediction model in at least one prediction model sent by a first node performing information prediction in the communication network, a retention indication for the current prediction information corresponding to each prediction model is also received from the first node.

37. The device according to claim 34 or 36, characterized in that The reference information includes a quality indicator of the historical prediction information or the historical prediction information; The device also includes: an information sending module, which is used to: Before the information receiving module receives the current prediction information and the quality indicator of the current prediction information respectively corresponding to each prediction model in at least one prediction model sent by the first node performing information prediction in the communication network, A prediction evaluation indication is sent to the first node, where the prediction evaluation indication is used to indicate the second node's requirement information for a quality indicator of prediction information.

38. The device according to claim 34, characterized in that The reference data includes the prediction auxiliary information; The information application module is specifically used for: determining the forecasted demand for the scenario; For each prediction model, according to the matching degree between the prediction auxiliary information corresponding to the prediction model and the prediction demand, determine the current prediction information corresponding to each prediction model, in which the quality indicator meets the quality condition; According to the current prediction information that the quality indicator meets the quality condition, a network management operation for the scenario is performed.

39. A communication network, characterized in that: The communication network includes a first node that performs information prediction and a second node that uses the predicted information; The first node is used to obtain reference data and current measurement information matching a scenario managed by the network, wherein the reference data is used to indicate the prediction quality of at least one prediction model matching the scenario; process the current measurement information through each prediction model of the at least one prediction model respectively to obtain current prediction information corresponding to the prediction model; determine the quality index of the current prediction information corresponding to each of the at least one prediction model based on the reference data; and send the current prediction information corresponding to each prediction model and the quality index of the current prediction information to the second node in the communication network; The second node is used to perform network management operations for the scenario according to the current prediction information corresponding to each prediction model, in which the quality indicator meets the quality condition.

40. The communication network according to claim 39, characterized in that The reference data includes one of the following: a preset test set, quality indicators of historical prediction information, historical prediction information, and prediction auxiliary information; wherein the preset test set includes test measurement information and test prediction information corresponding to the test measurement information, and actual information corresponding to the test prediction information; the prediction auxiliary information is used to indicate the attribute information and / or operating environment of each prediction model.

41. An electronic device, characterized in that: include: Processor and memory; The processor is connected to the memory; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 20.

42. A computer-readable storage medium, characterized in that: The method comprises a computer program, wherein when the computer program is run on an electronic device, the electronic device executes the method according to any one of claims 1 to 20.

43. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 20.

44. A computer program product, characterized in that The invention comprises a computer program, which, when executed by an electronic device, enables the electronic device to execute the method according to any one of claims 1 to 20.

Images