Model monitoring processing method, device, network side device and readable storage medium

Abstract

The present application discloses a model monitoring processing method, an apparatus, a network side device, and a readable storage medium, which belong to the technical field of communications. The model monitoring processing method of an embodiment of the present application includes: a step of receiving first information from a second network device by a first network device, the first information including a data analysis result; and a step of transmitting second information from the first network device to the second network device or a third network device, the second information indicating a usage status of the data analysis result by the first network device.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority from Chinese Patent Application No. 202211236921.9, filed in China on October 10, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of communications, and more particularly to a model monitoring processing method, apparatus, network side device, and readable storage medium. [Background technology]

[0003] With the development of communication technology, artificial intelligence models are being introduced into communication systems. Currently, in order to complete a task required by a consumer network element, a network data analytics function (NWDAF) generally needs to use a model to analyze the task, obtain the analysis results, collect data in the network, and compare them with the predicted results to determine model performance information. Because the consumer network element may perform network operations different from normal (network operations determined without the analysis results) based on the analysis results, this may affect the corresponding data used to calculate the model performance information estimate, resulting in a lower accuracy of the model performance information estimate calculation. Summary of the Invention

[0004] The embodiments of the present application provide a model monitoring processing method, apparatus, network side device and readable storage medium that can solve the problem of low accuracy in model performance information estimation calculation.

[0005] In the first aspect, receiving, by the first network device, first information from the second network device, the first information including a data analysis result; A model monitoring processing method is provided, the method including a step of the first network device transmitting second information to the second network device or the third network device, the second information being for indicating the usage status of the data analysis results by the first network device.

[0006] In a second aspect, transmitting first information from the second network device to the first network device, the first information including a data analysis result; and a step in which the second network device receives second information from the first network device, the second information being intended to indicate a usage status of the data analysis results by the first network device.

[0007] In a third aspect, receiving second information from the first network device or the second network device by the third network device, the second information being for indicating a usage status of the data analysis result by the first network device; The third network device performs an estimated calculation of first performance information of a first model based on the second information, wherein the first model is a model obtained by the third network device performing a model training process, and the first model is used by the second network device to generate the data analysis result.

[0008] In a fourth aspect, a first receiving module for receiving first information from a second network device, the first information including a data analysis result; A model monitoring processing device is provided, comprising: a first transmission module for transmitting second information to the second network device or a third network device, the second information being for indicating a usage status of the data analysis result by the first network device.

[0009] In a fifth aspect, a second transmitting module for transmitting first information to the first network device, the first information including a data analysis result; A model monitoring processing device is provided, comprising: a second receiving module for receiving second information from the first network device, the second information being for indicating a usage status of the data analysis results by the first network device.

[0010] In a sixth aspect, a third receiving module for receiving second information from the first network device or the second network device, the second information indicating a usage status of the data analysis result by the first network device; and a second estimation calculation module for performing an estimation calculation of first performance information of a first model based on the second information, wherein the first model is a model obtained by a third network device performing a model training process, and the first model is used by the second network device to generate the data analysis result.

[0011] In a seventh aspect, there is provided a network side device comprising a processor and a memory, wherein the memory stores a program or command executable by the processor, and when the program or command is executed by the processor, the steps of the method described in the first aspect are realized, or the steps of the method described in the second aspect are realized, or the steps of the method described in the third aspect are realized.

[0012] In an eighth aspect, a network side device includes a processor and a communication interface, When the network side device is a first network device, the communication interface is for receiving first information from a second network device, the first information including a data analysis result, and for transmitting second information indicating a usage status of the data analysis result by the first network device to the second network device or a third network device; When the network side device is a second network device, the communication interface is for transmitting first information including a data analysis result to a first network device and receiving second information from the first network device indicating a usage status of the data analysis result by the first network device; When the network side device is a third network device, the communication interface is for receiving second information from a first network device or a second network device, the second information is for indicating the usage status of the data analysis result by the first network device, the processor is for performing an estimated calculation of first performance information of a first model based on the second information, the first model is a model obtained by the third network device performing a model training process, and the first model is for the second network device to generate the data analysis result.

[0013] In a ninth aspect, there is provided a communication system including a first network device capable of executing steps of the model monitoring processing method described in the first aspect, a second network device capable of executing steps of the model monitoring processing method described in the second aspect, and a third network device capable of executing steps of the model monitoring processing method described in the third aspect.

[0014] In a tenth aspect, there is provided a readable storage medium having a program or command stored thereon, the program or command being executed by a processor to achieve the steps of the method according to the first aspect, or to achieve the steps of the method according to the second aspect, or to achieve the steps of the method according to the third aspect.

[0015] In an eleventh aspect, there is provided a chip comprising a processor and a communication interface, the communication interface and the processor being coupled together, and the chip being configured to perform the steps of the method according to the first aspect, or the steps of the method according to the second aspect, or the steps of the method according to the third aspect, by the processor executing a program or command.

[0016] In a twelfth aspect, there is provided a computer program / program product that is stored on a storage medium and that, when executed by at least one processor, causes the steps of the method according to the first aspect to be realized, or the steps of the method according to the second aspect to be realized, or the steps of the method according to the third aspect to be realized. [Effects of the Invention]

[0017] In an embodiment of the present application, the first network device sends the second information to the second network device or the third network device, so that the second network device or the third network device can determine the usage status of the data analysis result by the first network device based on the second information, thereby assisting in the estimated calculation of the first performance information of the first model and further improving the accuracy and reliability of the model performance information calculation. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a schematic diagram of a network configuration to which an embodiment of the present application can be applied; [Figure 2] 1 is a flowchart 1 of a model supervision processing method provided in an embodiment of the present application; [Figure 3] 2 is a flowchart 2 of a model supervision processing method provided in an embodiment of the present application; [Figure 4] 3 is a flowchart 3 of a model supervision processing method provided in an embodiment of the present application; [Figure 5] 4 is a flowchart 4 of a model supervision processing method provided in an embodiment of the present application. [Figure 6]5 is a flowchart 5 of a model supervision processing method provided in an embodiment of the present application. [Figure 7] 6 is a flowchart 6 of a model supervision processing method provided in an embodiment of the present application. [Figure 8] 1 is a configuration diagram of a model monitoring processing device provided in an embodiment of the present application. [Figure 9] FIG. 2 is a configuration diagram 2 of a model monitoring processing device provided in an embodiment of the present application. [Figure 10] 3 is a configuration diagram of a model monitoring processing device provided in an embodiment of the present application. [Figure 11] FIG. 1 is a configuration diagram of a communication device provided in an embodiment of the present application. [Figure 12] FIG. 2 is a configuration diagram of a network side device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0019] The technical solutions in the embodiments of the present application will be clearly explained below with reference to the drawings in the embodiments of the present application. Of course, the described embodiments are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art fall within the scope of protection of the present application.

[0020] The terms "first," "second," and the like used in the specification and claims of this application are not intended to describe a particular order or chronology, but rather to distinguish between similar objects. It should be understood that terms used in this manner may be interchanged where appropriate, so that the embodiments of this application can be practiced in orders other than those illustrated or described herein. It should also be understood that objects distinguished by "first" and "second" generally belong to a single category and do not limit the number of objects; for example, the first object may be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the " / " symbol generally indicates that the related objects before and after are in an "or" relationship.

[0021] It should be noted that the techniques described in the embodiments of this application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-Carrier Frequency Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in the embodiments of this application are often used interchangeably, and the techniques described can be used in other systems and wireless technologies in addition to those mentioned above. While the following description will discuss New Radio (NR) systems for illustrative purposes, and NR terminology is used in much of the following description, these technologies may be used in conjunction with other systems and wireless technologies, such as 6th Generation (6G) networks. th It can also be applied to applications other than NR system applications, such as 6G (Generation 6G) communication systems.

[0022] 1 shows a block diagram of a wireless communication system to which an embodiment of the present application can be applied. The wireless communication system includes a terminal 11 and a network device 12. Here, the terminal 11 may be a mobile phone, a tablet personal computer, a laptop computer (also called a notebook computer), a personal digital assistant (PDA), a personal digital assistant, a netbook, an ultra-mobile personal computer (UMPC), a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, a vehicle user equipment (VUE), a pedestrian user equipment (PUE), a smart home (home devices equipped with wireless communication functions such as a refrigerator, a television, a washing machine, or furniture), a game console, a personal computer (PC), an automated teller machine, a kiosk, or the like. The wearable device includes a smart watch, a smart wristband, a smart earphone, a smart glasses, a smart accessory (such as a smart bracelet, a smart ring, a smart necklace, a smart anklet, or the like), a smart wristlet, a smart wear, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiments of the present application. The network equipment 12 may include an access network equipment or a core network equipment. Here, the access network equipment may be referred to as a radio access network equipment, a radio access network (RAN), a radio access network function, or a radio access network unit.The access network equipment may include a base station, a Wireless Local Area Network (WLAN) access point, or a Wireless Fidelity (WiFi) node, and the base station may be called a Node B, an Evolved Node B (eNB), an access point, a Base Transceiver Station (BTS), a radio base station, a radio transceiver, a Basic Service Set (BSS), an Extended Service Set (ESS), a Home B node, a Home Evolved B node, a Transmitting Receiving Point (TRP), or any other suitable term in the art. As long as the same technical effect can be achieved, the base station is not limited to a specific technical term. It should be noted that the embodiments of this application only take a base station in an NR system as an example, but the specific type of the base station is not limited.

[0023] An embodiment of the present application provides a model supervision processing method, as shown in FIG. Step 201, in which a first network device receives first information from a second network device, the first information including a data analysis result; and step 202, in which the first network device transmits second information to the second network device or the third network device, the second information indicating the usage of the data analysis results by the first network device.

[0024] In an embodiment of the present application, a first network device can first send a task request for a first task to a second network device, the task request requesting acquisition of data analysis results for the first task. The first task can be a data analysis task, and data analysis task information for the data analysis task can include task identifier information, task condition limit information, and task analysis target information. The task identifier information can be a data analysis task ID (analytic ID), and the analytic ID can indicate the purpose of the data analysis task, the data to be acquired, etc. For example, if the analytic ID is associated with a user's mobility trajectory information, it can be known that the data analysis result can be user location information. The task condition limit information and task analysis target information are used to further refine the task request. The second network device can request acquisition of a first model for the first task by sending a model request to a third network device based on the task request. In the model request process, the model request can include task request information and model limit information. Wherein, the task requirement information includes the content of the task requirement for the first task, and the model limitation information further clarifies the necessary task requirements, such as limitations on the accuracy of the model. After receiving the model request, the third network device can select or train a first model that meets the first task requirement and send it to the second network device. The second network device can then perform data analysis on the first task based on the first model to generate a data analysis result for the first task and send first information to the first network device, the first information including the data analysis result. The first network device can then feed back the use status of the data analysis result to the second network device. In this way, the second network device or the third network device can determine a monitoring action for the first model based on the second information.For example, when a first network device uses the data analysis results and the use of the data analysis results by the first network device affects the network or the data in the network, an estimation calculation is not performed for the first performance information of the first model, or relevant data that affects the first performance information estimation calculation is selected and removed during the estimation calculation, thereby increasing the accuracy and reliability of the model performance information estimation calculation.

[0025] Optionally, in some embodiments, when the second network device transmits the first information to the first network device, the second network device may also transmit identifier information, address information, etc. of the third network device. In this way, the first network device can know the information of the third network device and transmit information / messages directly to the third network device. Optionally, prior to this, the first network device may request the second network device to transmit identifier information, address information, etc. of the third network device. Alternatively, the first network device may transmit its own information to the second network device, which then transmits it to the third network device via the second network device. Optionally, after the first network device requests it, the second network device may transmit identifier information, address information, etc. of the third network device in a separate message.

[0026] Alternatively, the third network device may request the second network device to feed back the identifier information, address information, etc. of the first network device, thereby transmitting information / messages to the first device, when or after receiving the model request and selecting or training a first model that satisfies the first task request and transmitting it to the second network device. Alternatively, the third network device may request the second network device to transmit its own information to the first network device, thereby informing the first network device to feed back / transmit the second information, etc. to itself.

[0027] Optionally, the estimation calculation for the first performance information of the first model can be understood as monitoring the first model. Specifically, the first model may be monitored by a second network device, or the first model may be monitored by a third network device. For example, in some embodiments, when the first model is monitored by a third network device, the second information can trigger transfer of the second information from the second network device to the third network device, or the first network device can directly send the second information to the third network device. Here, monitoring the first model can be understood as or replaced with monitoring the first task.

[0028] It should be understood that the first performance information is intended to indicate the model performance of the first model when it is actually used, and the first performance information can be understood as model performance information estimated and calculated based on data collected after data analysis is performed on the first task using the first model after model training is completed.

[0029] Optionally, the data analysis result may be an output value obtained by the second network device performing calculations and inferences using the first model. For example, it may be a prediction of the load of a certain network device. The content of the data analysis result varies depending on the task. For example, in a network device load task, the data analysis result may be the load situation of the network device at a certain time in the future, for example, the load of a User Plane Function (UPF) network element in the morning of the next day.

[0030] It should be noted that the first network device may be understood as a consumer network element (NF), the second network device may be understood as an NWDAF or an analytics logical function (AnLF) therein for performing inference and generating predictive information or generating a summary of historical data, and the third network device may be understood as an NWDAF or a model training logical function (MTLF) therein for generating and training a model.

[0031] In an embodiment of the present application, the first network device sends the second information to the second or third network device, so that the second or third network device can determine how the first network device uses the data analysis result based on the second information, thereby assisting in the estimation calculation of the first performance information of the first model and further improving the accuracy and reliability of the model performance information estimation calculation.

[0032] Optionally, in some embodiments, before the step of the first network device transmitting second information to the second network device or the third network device, the method further comprises: The method further includes the first network device determining the second information based on the first information.

[0033] In an embodiment of the present application, the first network device can determine, based on the first information, whether to perform a first network operation based on the data analysis result (i.e., determine, based on the first information, whether to use the data analysis result), and can further determine second information. Optionally, the usage of the data analysis result is for assisting the second network device or the third network device in estimating and calculating first performance information of a first model, where the first model is a model obtained by the third network device performing a model training process, and the first model is used by the second network device to generate the data analysis result.

[0034] Optionally, in some embodiments, the second information includes first instruction information, which is for instructing whether the first network device has performed a first network operation based on the data analysis result.

[0035] It should be understood that the network operation determined based on the data analysis result and the network operation determined without the data analysis result may be the same or different. Selectively, if they are the same, it can be understood that the first network operation determined based on the data analysis result does not affect the network or the data in the network. If they are different, it can be understood that the network or the data in the network is affected by the use of the data analysis result. Here, when determining a network operation without the data analysis result, the network operation can be determined based on its own previous logic and preset conditions, etc. Since the first network device uses first instruction information to indicate whether it performed the first network operation based on the data analysis result, it can determine a monitoring action for the first model based on the first instruction information. For example, if the first network operation is performed based on the data analysis result and affects the network or the data in the network, it can not perform an estimation calculation for the first performance information of the first model, or select and remove related data that affects the first performance information estimation calculation during the estimation calculation, thereby increasing the accuracy and reliability of the model performance information estimation calculation.

[0036] Optionally, in some embodiments, the second information comprises: type information or description information of the first network operation; first time information for indicating a time range for performing the first network operation; first location information for indicating a location range in which the first network operation is to be performed; Further, at least one of first target information for instructing a target on which the first network operation is to be performed is included; The object includes at least one of a user equipment object, an area object, and a network element object.

[0037] It should be understood that the first time information may be understood to indicate a time range in which the first performance information of the first model is not to be estimated. Data not existing within the time range is not used in the estimation of the first performance information of the first model, and the first location information may be understood to indicate a location range in which the first performance information of the first model is not to be estimated. That is, data not existing within the location range is not used in the estimation of the first performance information of the first model, and the first object information may be understood to indicate an object in which the first performance information of the first model is not to be estimated. That is, data not corresponding to the first object information is not used in the estimation of the first performance information of the first model. Alternatively, in some embodiments, the second information may further include other instruction information for indicating at least one of a time range in which the first performance information of the first model is not to be estimated, a location range in which the first performance information of the first model is not to be estimated, and an object in which the first performance information of the first model is not to be estimated.

[0038] Optionally, the type information of the first network operation may include slice selection, network load balancing, and terminal access control, etc. The description information may be comprehensive information with the type information, adding information such as target, time, and location. For example, when to perform an access control operation on a certain terminal. Here, the location can be understood as an area.

[0039] In an embodiment of the present application, the type information or description information of the first network operation, the first time information, and the first target information are sent to the second network device or the third network device, so that the second network device or the third network device can determine related data that will affect the first performance information estimation calculation based on the type information or description information of the first network operation, the first time information, and the first target information, and further can select and remove or eliminate the related data during the estimation calculation, or collect other related data other than the related data to perform the estimation calculation of the first performance information of the first model.

[0040] Optionally, in some embodiments, the second information comprises: The network device further includes second instruction information, the second instruction information being for indicating whether the first network operation affects the estimated calculation of the first performance information of the first model (by the second network device or the third network device), or for indicating whether the first network operation affects the estimated calculation of the accuracy of the data analysis result.

[0041] In an embodiment of the present application, by further feeding back the second instruction information, when the first network device performs the first network operation based on the data analysis result but does not affect the network or network data, the second network device or the third network device can collect all data (i.e., a full set of data) for estimating and calculating the first performance information, and then perform an estimation calculation of the first performance information based on all the collected data, thereby increasing the amount of data for estimating and calculating the first performance information, and further improving the accuracy of the first model's estimation calculation of the first performance information.

[0042] Optionally, in some embodiments, the second information corresponds to the data analysis results. task identifier information, Task conditional information, The analysis target information of the task further includes at least one piece of data analysis task information.

[0043] The task identifier information, task condition limiting information, and task analysis target information can be used to identify a specific task. Here, the task identifier information indicates the task targeted by the second information, and a specific task can be identified by using all three pieces of information together or the first two pieces of information. The task condition limiting information may also be called analytic filter information and indicates data analysis result filtering information. For example, it includes areas of interest (AOI), single network slice selection assistance information (S-NSSAI), and data network name (DNN). The task analysis target information can be understood as the target of analytic reporting and clearly indicates whether the task analysis target is a certain terminal, multiple terminals, all terminals, a certain network element, multiple network elements, or all network elements.

[0044] Optionally, in some embodiments, the first information corresponds to the data analysis results. reliability information indicating a degree of reliability of the data analysis result of the second network device; Second performance information for the first model, Further comprising at least one of the data types of interest for determining the second indication.

[0045] Optionally, the reliability information may include three reliability levels: high, medium, and low.

[0046] Alternatively, the second performance information may be understood as model performance information when training the first model. That is, it is the result of evaluating the model when the third network device trains the model. For example, the accuracy in training (AiT) of the model is calculated using a training test dataset during training. The representation form of the model performance information is not specifically limited, and may indicate the accuracy or error of the task inference result by the model directly or indirectly. For example, the inference accuracy of the first model may be indicated indirectly by calculating the inference error or inference error rate of the model. Here, the inference error or inference error rate may be calculated in various ways, such as mean absolute error (MAE), mean square error (MSE), etc. Furthermore, it may also be a direct accuracy level such as accuracy rate or precision.

[0047] Optionally, the data type of interest indicates a specific data type of interest, such as load information of a network device, and can help the first network device determine whether its network operation affects the environment. For example, the first network device can determine by itself whether its network operation affects the data type.

[0048] It should be noted that when determining a network operation, the first network device may determine whether the data analysis result is reliable and whether the data analysis result is usable based on the reliability information and second performance information in the first information. For example, when a Session Management Function (SMF) network element selects a UPF based on network load information, it may select the UPF (perform a network operation) if it receives a network load information analysis result indicating that a certain UPF will have a low network load at a certain time in the future. Furthermore, if the SMF learns early that a UPF may be underloaded and selects the UPF, the load of the UPF will no longer be as low as initially analyzed / predicted, and the network operation will affect the network environment (data in the network environment, such as the load level of a particular UPF). Conversely, if the reliability in the first information is low and / or the performance situation during model training is poor, Alternatively, if the internal logic and conditions of some first network devices determine that the first network devices will not use the data analysis results, the network operations performed by the first network devices will not be affected by the data analysis results and will not affect the network environment.

[0049] Optionally, in some embodiments, after the step of the first network device transmitting second information to the second network device or the third network device, the method further comprises: The method further includes the step of the first network device receiving the first performance information from the second network device or the third network device.

[0050] In an embodiment of the present application, the first model may be monitored by a second network device, or the first model may be monitored by a third network device. When the first model is monitored by the third network device, the third network device transmits the first performance information to the second network device, and then transfers the first performance information to the first network device via the second network side, or transmits the first performance information directly from the third network device to the first network device.

[0051] It should be noted that when the first model is monitored by the second network device, the second network device may start a data collection operation after feeding back the first information, or may start a data collection operation after receiving the second information, and the data collection operation can be understood as collecting data that can be used for calculating the first performance information estimate. For example, in some embodiments, the second network device can start a data collection operation after feeding back the first information, and after receiving the second information, can determine related data corresponding to the second information, and then not collect the related data or remove (also referred to as selecting and removing) the related data from the collected full set of data, thereby finally obtaining target data for calculating the first performance information estimate, and finally calculating the first performance information estimate based on the determined target data. When the first model is monitored by the third network device, the second network device forwards the received second information to the third network device, and after receiving the second information, the third network device can determine related data corresponding to the second information, and then not collect the related data or remove (also referred to as selecting and removing) the related data from the collected full set of data, thereby finally obtaining target data for estimating and calculating the first performance information, and finally estimating and calculating the first performance information based on the determined target data.

[0052] Optionally, in some embodiments, the step of transmitting the second information from the first network device to the second network device or the third network device includes: The method includes a step of the first network device transmitting N pieces of second information to the second network device or the third network device, where N is an integer greater than 1, and the N pieces of second information are determined by the first network device based on the N pieces of first information.

[0053] It should be noted that the first information may be fed back immediately after receiving the first information, or may be fed back after the preset condition is met. For example, in some embodiments, the step of the first network device transmitting N pieces of second information to the second network device or the third network device includes: When a predetermined reporting time point or an end time of a predetermined reporting period is reached, the first network device transmits N pieces of second information to the second network device or the third network device; or The method includes a step in which, when the first network device determines that the number of unsent second information has reached N, the first network device transmits N pieces of second information to the second network device or the third network device.

[0054] Optionally, feeding back N pieces of second information when a predetermined reporting time point or the end time of a predetermined reporting period is reached can be understood as triggering feedback of the second information based on a time condition. For example, when a day's worth of network operations is collected, they are all transmitted to the second network device or the third network device. When the first network device determines that the number of untransmitted second information has reached N, feeding back N pieces of second information can be understood as triggering feedback of the second information based on a quantity condition. For example, when 50 network operations are collected (e.g., 50 pieces of first information are received), they are all transmitted to the second network device or the third network device. Optionally, in some embodiments, each piece of first information may correspond to one network operation and one piece of second information.

[0055] Further, in some embodiments, the method further comprises the step of: N is the number of the second information; The method further includes transmitting at least one of the predetermined reporting time point or the predetermined reporting period to the second network device or a third network device.

[0056] In an embodiment of the present application, the number N of the second information and the predetermined reporting time or the predetermined reporting period may be transmitted using the same signaling as the N pieces of second information, or may be transmitted using different signaling, and is not further limited herein.

[0057] For a better understanding of the present application, some examples will now be described in detail.

[0058] In the first embodiment, the first model is monitored by the AnLF. Referring to FIG. 3, the following flow may be specifically included.

[0059] In step 30, a consumer initiates a first task, AnLF initiates a model request, and MTLF selects or trains a first model that meets the first task request, where the first task is similar in concept to the first task in step 202 and also similar in concept to the data analysis task described above.

[0060] In step 31, the MTLF transmits the first model to the AnLF, which then uses the first model to generate data analysis results.

[0061] In step 32, the AnLF sends first information to the consumer, which includes the data analysis result. Specifically, the AnLF can use the first model obtained in step 31 to perform calculations and inferences to obtain an output value. That is, the data analysis result of the first task is, for example, a prediction of the network element load of a certain network element. The AnLF can feed back the data analysis result to the consumer and also notify the consumer of related information about the data analysis result, which can help the consumer decide whether to use the data analysis result.

[0062] wherein the first information includes the data analysis result and the related information, and may further include at least one of identifier information and model identifier information; The identifier information is used to notify the specific task that is the target of the AnLF, and may include, for example, at least one of task identifier information, task condition limiting information, and task analysis target information.

[0063] Here, the related information may include at least one of confidence information, model performance information when training the first model, and label types of interest.

[0064] In step 33, the consumer determines whether to use the data analysis result based on the first information fed back by the AnLF in step 32, determines the network operation to be performed, and further determines whether the consumer's network operation will affect the network environment. After receiving the feedback in step 32, the consumer can determine the network operation to be performed based on the data analysis result in the fed back first information. When determining the network operation, the consumer can determine whether the data analysis result is reliable and usable based on information related to the data analysis result in the fed back first information. For example, when selecting a UPF based on network load information, if an SMF network element receives a network load information analysis result indicating that a certain UPF will have a low network load at a certain time in the future, the SMF may select the UPF (perform an operation). Furthermore, if the SMF learns early that a UPF may have a low load and selects this UPF, the load of the UPF will not be as low as initially analyzed / predicted, and the operation will affect the network environment (data in the network environment, such as the load level of a particular UPF). Conversely, if the reliability of the fed-back first information is low and / or the performance situation during model training is poor, or if some consumers decide not to use the analysis result due to their internal logic and conditions, the operations performed by the consumers will not be affected by the analysis result and will not affect the network environment.

[0065] In step 34, the consumer notifies the AnLF, using the second information, whether or not it has determined and executed a network operation using the analysis results. The consumer can use the second information to provide feedback to the AnLF on whether or not it executed a network operation based on the analysis results, and can also provide feedback on whether or not the operation will affect the network environment and / or the data in the network. Based on this information, the AnLF can determine whether or not to collect relevant data for estimating model performance information. Specifically, if the consumer selects to execute a network operation based on the data analysis results in step 33, for example, by selecting UPF, the consumer can notify the AnLF that it executed the network operation using the analysis results. The consumer can further notify the AnLF that its operation will affect the network environment and the data therein. This signaling (signaling carrying the second information) may include identifier information such as analysis task identifier information, task condition limit information, task analysis target information, and model identifier information. The identifier information is used to instruct the AnLF on the task and / or model corresponding to the value provided by the consumer.

[0066] In step 35, the AnLF determines whether the relevant data corresponding to this feedback can be used in the calculation of the estimated model performance information. Based on the second information fed back by the consumer in step 34, the AnLF determines whether to collect relevant data or whether to consider relevant data corresponding to the operation (the operation determined using the data analysis results in the first information) in the calculation of the estimated model performance information. For example, if the second information fed back by the consumer in step 34 clearly indicates that a network operation was performed based on the data analysis results, the AnLF can determine (the target task and / or model and) corresponding relevant data (which may include model input data, output data, label data, etc.) based on the identifier information and the feedback time, and choose not to collect this data. Alternatively, it can remove this data when calculating the estimated performance information.

[0067] In step 36, the AnLF collects other relevant data and estimates and calculates model performance information of the first model. After the model performance information is estimated and calculated, a subsequent operation is determined, and the subsequent operation may be, for example: Communicating model performance information to consumers; Communicating model performance information and / or related data to the MTLF; and and storing the model performance information and / or related data in an Analytics Data Repository Function (ADRF).

[0068] In Example 2, the first model is monitored by the MTLF. Referring to Figure 4, in this example, when the first information is received, a network operation is performed using the data analysis result, and instead of feeding back the second information every time it is determined, N pieces of temporarily stored second information are fed back to the AnLF after a preset condition is met.

[0069] The pre-set conditions are A time condition for transmitting second information corresponding to the operation within a certain period of time, for example, one day, to the AnLF when network operations are collected; and a quantity condition that when a certain number, for example 50 network operations, are collected, second information corresponding to the 50 network operations is also transmitted to the AnLF.

[0070] In the third embodiment, the first model is monitored by the MTLF. Referring to Fig. 5, this embodiment differs from the first embodiment in that the first model is monitored and model performance information is estimated and calculated by the MTLF. Specifically, after receiving the second information, the AnLF transmits the second information to the MTLF. In other words, the AnLF forwards the second information received from the consumer.

[0071] Furthermore, whether to collect related data corresponding to the network operation (i.e., the second information) or whether to consider the related data in the performance information estimation calculation is determined by the MTLF. Finally, other related data is collected by the AnLF, and the model performance information of the first model is estimated and calculated. After the model performance information is estimated and calculated, a subsequent operation is determined, and the subsequent operation can be, for example: Notifying AnLF of model performance information and forwarding it to consumers via AnLF; Communicating model performance information and / or related data to AnLF; and / or storing the model performance information and / or associated data in the ADRF.

[0072] An embodiment of the present application further provides a model supervision processing method, as shown in Figure 6, which includes: Step 601: transmitting first information from the second network device to the first network device, the first information including a data analysis result; and step 602, wherein the second network device receives second information from the first network device, the second information indicating the first network device's use of the data analysis results.

[0073] Optionally, the usage of the data analysis result is to assist the second network device or the third network device in estimating and calculating first performance information of a first model, the first model being a model obtained by the third network device performing a model training process, and the first model being used by the second network device to generate the data analysis result.

[0074] Optionally, the method further comprises: The method further includes a step in which the second network device performs an estimated calculation of first performance information of a first model based on the second information, wherein the first model is a model obtained by a third network device performing a model training process, and the first model is used by the second network device to generate the data analysis result.

[0075] Optionally, the second information includes first instruction information, and the first instruction information is for instructing whether the first network device has performed a first network operation based on the data analysis result.

[0076] Optionally, the second information comprises: type information or description information of the first network operation; first time information for indicating a time range for performing the first network operation; first location information for indicating a location range in which the first network operation is to be performed; Further, at least one of first target information for instructing a target on which the first network operation is to be performed is included; The object includes at least one of a user equipment object, an area object, and a network element object.

[0077] Optionally, the second information comprises: The network further includes second instruction information, the second instruction information being for indicating whether the first network operation affects the estimated calculation of the first performance information of the first model, or for indicating whether the first network operation affects the estimated calculation of the accuracy of the data analysis result.

[0078] Optionally, the second information corresponds to the data analysis results. task identifier information, Task conditional information, The analysis target information of the task further includes at least one piece of data analysis task information.

[0079] Optionally, the step of the second network device estimating and calculating first performance information of a first model based on the second information includes: The second network device collects target data based on the second information; The second network device estimates and calculates first performance information of the first model based on the target data.

[0080] Optionally, when first instruction information in the second information indicates that the first network device performed a first network operation based on the data analysis result, the target data includes data not related to the first network operation.

[0081] Optionally, the step of collecting target data by the second network device based on the second information includes: The second network device determines, based on the first time information in the second information, data that is not related to the first time information as the target data; The second network device determines, based on the first target information in the second information, data that is not related to the first target information as the target data; and determining, by the second network device, data not associated with the first location information as the target data based on the first location information in the second information.

[0082] In the embodiment of the present application, the fourth network device can be understood as a data source network element.

[0083] Optionally, when the first instruction information in the second information indicates that the first network device did not perform the first network operation based on the data analysis result, the target data includes a full set of data for first performance information estimation calculation obtained by the second network device from the fourth network device based on data analysis task information corresponding to the second information.

[0084] Optionally, after the second network device receiving second information from the first network device, the method further comprises: The method further includes the step of the second network device transmitting the second information to a third network device.

[0085] Optionally, the step of receiving second information from the first network device by the second network device includes: The method includes a step in which the second network device receives N pieces of second information from the first network device, where N is an integer greater than 1, and the N pieces of second information are determined by the first network device based on the N pieces of first information.

[0086] Optionally, the method further comprises: The second network device receives the information transmitted from the first network device. N is the number of the second information; The method further includes receiving at least one of a predetermined reporting time point or a predetermined reporting period.

[0087] Optionally, the first information comprises: reliability information indicating a degree of reliability of the data analysis result of the second network device; Second performance information for the first model, Further comprising at least one of the data types of interest for determining the second indication.

[0088] An embodiment of the present application further provides a model supervision processing method, as shown in Figure 7, which includes: Step 701, in which the third network device receives second information from the first network device or the second network device, the second information being for indicating a usage status of the data analysis result by the first network device; The method includes step 702 in which the third network device estimates and calculates first performance information of a first model based on the second information, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis result.

[0089] Optionally, the data analysis results are used to assist the second network device or the third network device in calculating an estimate of first performance information of a first model.

[0090] Optionally, the second information includes first instruction information, and the first instruction information is for instructing whether the first network device has performed a first network operation based on the data analysis result.

[0091] Optionally, the second information comprises: type information or description information of the first network operation; first time information for indicating a time range for performing the first network operation; first location information for indicating a location range in which the first network operation is to be performed; Further, at least one of first target information for instructing a target on which the first network operation is to be performed is included; The object includes at least one of a user equipment object, an area object, and a network element object.

[0092] Optionally, the second information comprises: The network further includes second instruction information, the second instruction information being for indicating whether the first network operation affects the estimated calculation of the first performance information of the first model, or for indicating whether the first network operation affects the estimated calculation of the accuracy of the data analysis result.

[0093] Optionally, the second information corresponds to the data analysis results. task identifier information, Task conditional information, The analysis target information of the task further includes at least one piece of data analysis task information.

[0094] Optionally, the step of the third network device estimating and calculating first performance information of the first model based on the second information includes: The third network device collects target data based on the second information; The third network device estimates and calculates first performance information of the first model based on the target data.

[0095] Optionally, when first instruction information in the second information indicates that the first network device performed a first network operation based on the data analysis result, the target data includes data not related to the first network operation.

[0096] Optionally, the step of collecting target data by the third network device based on the second information includes: The third network device determines, based on the first time information in the second information, data that is not related to the first time information as the target data; The third network device determines, based on the first target information in the second information, data that is not related to the first target information as the target data; and determining, by the third network device, data not associated with the first location information as the target data based on the first location information in the second information.

[0097] Optionally, the step of collecting target data by the third network device based on the second information includes: The third network device acquires a full set of data for first performance information estimation calculation from a fourth network device based on data analysis task information corresponding to the second information; The method includes a step in which the third network device removes data related to the first time information from the full set data based on first time information in the second information to obtain the target data, and / or the third network device removes data related to the first object information from the full set data based on first object information in the second information to obtain the target data, and / or the third network device removes data related to the first location information from the full set data based on first location information in the second information to obtain the target data.

[0098] Optionally, when the first instruction information in the second information indicates that the first network device did not perform the first network operation based on the data analysis result, the target data includes a full set of data for first performance information estimation calculation obtained by the third network device from the fourth network device based on data analysis task information corresponding to the second information.

[0099] Optionally, the step of receiving second information from the first network device or the second network device by the third network device includes: The method includes a step in which the third network device receives N pieces of second information from the first network device or the second network device, where N is an integer greater than 1, the N pieces of second information are determined by the first network device based on the N pieces of first information, and the first information includes a data analysis result.

[0100] Optionally, the method further comprises: The third network device receives the information transmitted from the first network device or the second network device. N is the number of the second information; The method further includes receiving at least one of a predetermined reporting time point or a predetermined reporting period.

[0101] Optionally, the first information comprises: reliability information indicating a degree of reliability of the data analysis result of the second network device; Second performance information for the first model, Further comprising at least one of the data types of interest for determining the second indication.

[0102] An embodiment of the present application further provides a model monitoring device 800, as shown in FIG. a first receiving module 801 for receiving first information from a second network device, the first information including a data analysis result; and a first transmitting module 802 for transmitting second information to the second network device or a third network device, the second information indicating the usage status of the data analysis results by the first network device.

[0103] Optionally, the usage of the data analysis result is to assist the second network device or the third network device in estimating and calculating first performance information of a first model, the first model being a model obtained by the third network device performing a model training process, and the first model being used by the second network device to generate the data analysis result.

[0104] Optionally, the model monitoring processor 800: The system further comprises a first determination module for determining the second information based on the first information.

[0105] Optionally, the second information includes first instruction information, and the first instruction information is for instructing whether the first network device has performed a first network operation based on the data analysis result.

[0106] Optionally, the second information comprises: type information or description information of the first network operation; first time information for indicating a time range for performing the first network operation; first location information for indicating a location range in which the first network operation is to be performed; Further, at least one of first target information for instructing a target on which the first network operation is to be performed is included; The object includes at least one of a user equipment object, an area object, and a network element object.

[0107] Optionally, the second information comprises: The network further includes second instruction information, the second instruction information being for indicating whether the first network operation affects the estimated calculation of the first performance information of the first model, or for indicating whether the first network operation affects the estimated calculation of the accuracy of the data analysis result.

[0108] Optionally, the second information corresponds to the data analysis results. task identifier information, Task conditional information, The analysis target information of the task further includes at least one piece of data analysis task information.

[0109] Optionally, the first information corresponds to the data analysis results. reliability information indicating a degree of reliability of the data analysis result of the second network device; Second performance information for the first model, Further comprising at least one of the data types of interest for determining the second indication.

[0110] Optionally, the first sending module 802 is specifically for sending N pieces of second information to the second network device or the third network device, where N is an integer greater than 1, and the N pieces of second information are determined by the first network device based on the N pieces of first information.

[0111] Optionally, the first transmission module 802 is specifically for transmitting N pieces of second information to the second network device or the third network device when a predetermined reporting time point or an end time of a predetermined reporting period is reached, or for the first network device to transmit N pieces of second information to the second network device or the third network device when it determines that the number of second information that has not been transmitted has reached N.

[0112] Optionally, the first transmitting module 802 further comprises: N is the number of the second information; and for transmitting at least one of the predetermined reporting time point or the predetermined reporting period to the second network device or a third network device.

[0113] An embodiment of the present application further provides a model supervisory processing device 900, as shown in FIG. a second sending module 901 for sending first information to the first network device, the first information including a data analysis result; and a second receiving module 902 for receiving second information from the first network device, the second information indicating the usage status of the data analysis results by the first network device.

[0114] Optionally, the usage of the data analysis result is to assist the second network device or the third network device in estimating and calculating first performance information of a first model, the first model being a model obtained by the third network device performing a model training process, and the first model being used by the second network device to generate the data analysis result.

[0115] Optionally, the model monitoring processor 900: The network device further includes a first estimation calculation module for estimating and calculating first performance information of a first model based on the second information, the first model being a model obtained by a third network device performing a model training process, and the first model being used by a second network device to generate the data analysis result.

[0116] Optionally, the second information includes first instruction information, which is for instructing the first network device whether to perform a first network operation based on the data analysis result.

[0117] Optionally, the second information comprises: type information or description information of the first network operation; first time information for indicating a time for performing the first network operation; Further, at least one of first target information for instructing a target on which the first network operation is to be performed is included; The object includes at least one of a user equipment object, an area object, and a network element object.

[0118] Optionally, the second information comprises: The network further includes second instruction information, the second instruction information being for indicating whether the first network operation affects the estimated calculation of the first performance information of the first model, or for indicating whether the first network operation affects the estimated calculation of the accuracy of the data analysis result.

[0119] Optionally, the second information corresponds to the data analysis results. task identifier information, Task conditional information, The analysis target information of the task further includes at least one piece of data analysis task information.

[0120] Optionally, the first estimate calculation module: a first collecting unit for collecting target data based on the second information; and a first estimation calculation unit for performing estimation calculation of first performance information of the first model based on the target data.

[0121] Optionally, when first instruction information in the second information indicates that the first network device performed a first network operation based on the data analysis result, the target data includes data not related to the first network operation.

[0122] Optionally, the first collecting unit specifically comprises: determining, based on first time information in the second information, data not related to the first time information as the target data; determining, based on first target information in the second information, data not related to the first target information as the target data; and determining, based on the first location information in the second information, data that is not related to the first location information as the target data.

[0123] Optionally, the first estimation calculation unit specifically comprises: acquiring a full set of data for first performance information estimation calculation from a fourth network device based on data analysis task information corresponding to the second information; The method is for executing the steps of: obtaining the target data by removing data related to the first time information from the full set data based on first time information in the second information; and / or obtaining the target data by removing data related to the first object information from the full set data based on first object information in the second information; and / or obtaining the target data by removing data related to the first position information from the full set data based on first position information in the second information.

[0124] Optionally, when the first instruction information in the second information indicates that the first network device did not perform the first network operation based on the data analysis result, the target data includes a full set of data for first performance information estimation calculation obtained by the second network device from the fourth network device based on data analysis task information corresponding to the second information.

[0125] Optionally, the second sending module 901 is further for sending the second information to a third network device.

[0126] Optionally, the second receiving module 902 is specifically for receiving N pieces of second information from the first network device, where N is an integer greater than 1, and the N pieces of second information are determined by the first network device based on the N pieces of first information.

[0127] Optionally, the second receiving module 902 further receives the following from the first network device: N is the number of the second information; The information is for receiving at least one of a predetermined reporting time point or a predetermined reporting period.

[0128] Optionally, the first information comprises: reliability information indicating a degree of reliability of the data analysis result of the second network device; Second performance information for the first model, Further comprising at least one of the data types of interest for determining the second indication.

[0129] An embodiment of the present application further provides a model monitoring device 1000, as shown in FIG. a third receiving module 1001 for receiving second information from the first network device or the second network device, the second information indicating the usage status of the data analysis result by the first network device; and a second estimation calculation module 1002 for estimating and calculating first performance information of a first model based on the second information, wherein the first model is a model obtained by a third network device performing a model training process, and the first model is used by the second network device to generate the data analysis result.

[0130] Optionally, the data analysis results are used to assist the second network device or the third network device in calculating an estimate of first performance information of a first model.

[0131] Optionally, the second information includes first instruction information, and the first instruction information is for instructing whether the first network device has performed a first network operation based on the data analysis result.

[0132] Optionally, the second information comprises: type information or description information of the first network operation; first time information for indicating a time range for performing the first network operation; first location information for indicating a location range in which the first network operation is to be performed; Further, at least one of first target information for instructing a target on which the first network operation is to be performed is included; The object includes at least one of a user equipment object, an area object, and a network element object.

[0133] Optionally, the second information comprises: The network further includes second instruction information, the second instruction information being for indicating whether the first network operation affects the estimated calculation of the first performance information of the first model, or for indicating whether the first network operation affects the estimated calculation of the accuracy of the data analysis result.

[0134] Optionally, the second information corresponds to the data analysis results. task identifier information, Task conditional information, The analysis target information of the task further includes at least one piece of data analysis task information.

[0135] Optionally, the second estimate calculation module 1002: a second collecting unit for collecting target data based on the second information; and a second estimation calculation unit for performing estimation calculation of first performance information of the first model based on the target data.

[0136] Optionally, when first instruction information in the second information indicates that the first network device performed a first network operation based on the data analysis result, the target data includes data not related to the first network operation.

[0137] Optionally, the second collecting unit specifically comprises: determining, based on first time information in the second information, data not related to the first time information as the target data; determining, based on first target information in the second information, data not related to the first target information as the target data; and determining, based on the first location information in the second information, data that is not related to the first location information as the target data.

[0138] Optionally, the second collecting unit specifically comprises: acquiring a full set of data for first performance information estimation calculation from a fourth network device based on data analysis task information corresponding to the second information; The method is for executing the steps of: obtaining the target data by removing data related to the first time information from the full set data based on first time information in the second information; and / or obtaining the target data by removing data related to the first object information from the full set data based on first object information in the second information; and / or obtaining the target data by removing data related to the first position information from the full set data based on first position information in the second information.

[0139] Optionally, when the first instruction information in the second information indicates that the first network device did not perform the first network operation based on the data analysis result, the target data includes a full set of data for first performance information estimation calculation obtained by the third network device from the fourth network device based on data analysis task information corresponding to the second information.

[0140] Optionally, the third receiving module 1001 specifically includes: The network device is for receiving N pieces of second information from a first network device or a second network device, where N is an integer greater than 1, and the N pieces of second information are determined by the first network device based on the N pieces of first information, and the first information includes a data analysis result.

[0141] Optionally, the third receiving module 1001 specifically receives the following from the first network device or the second network device: N is the number of the second information; The information is for receiving at least one of a predetermined reporting time point or a predetermined reporting period.

[0142] Optionally, the first information comprises: reliability information indicating a degree of reliability of the data analysis result of the second network device; Second performance information for the first model, Further comprising at least one of the data types of interest for determining the second indication.

[0143] The model monitoring processing device in the embodiment of the present application may be an electronic device, for example, an electronic device with an operating system, or a component of an electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal or other device other than a terminal. Exemplarily, the terminal may include, but is not limited to, the types of terminal 11 listed above, and the other device may be a server, a network attached storage (NAS), etc. In the embodiment of the present application, there is no specific limitation.

[0144] The model monitoring processing device provided in the embodiments of the present application can implement each step implemented in the method embodiments of Figures 2 to 7 and achieve the same technical effects, and detailed descriptions thereof will be omitted here to avoid repetition.

[0145] Optionally, as shown in FIG. 11, an embodiment of the present application further provides a communication device 1100, which includes a processor 1101 and a memory 1102, and stores a program or command executable by the processor 1101, and when the program or command is executed by the processor 1101, each step of the embodiment of the model monitoring processing method described above is realized, and the same technical effect can be achieved. In order to avoid repetition, detailed description will be omitted here.

[0146] An embodiment of the present application is a network side device including a processor and a communication interface, When the network side device is a first network device, the communication interface is for executing a step of receiving first information from a second network device, the first information including a data analysis result, and a step of transmitting second information to the second network device or a third network device, the second information indicating a usage status of the data analysis result by the first network device, When the network side device is a second network device, the communication interface is for executing a step of transmitting first information to a first network device, the first information including a data analysis result, and a step of receiving second information from the first network device, the second information indicating a usage status of the data analysis result by the first network device; The present invention further provides a network side device, wherein when the network side device is a third network device, the communication interface is for receiving second information from a first network device or a second network device, the second information is for indicating the usage status of the data analysis result by the first network device, the processor is for performing an estimated calculation of first performance information of a first model based on the second information, the first model is a model obtained by the third network device performing a model training process, and the first model is for the second network device to generate the data analysis result.

[0147] The embodiment of the network-side device corresponds to the embodiment of the above-mentioned network-side method, and each implementation step and realization form of the above-mentioned method embodiment can be applied to the embodiment of the network-side device, and the same technical effects can be achieved.

[0148] Specifically, an embodiment of the present application further provides a network side device. As shown in Figure 12, the network side device 1200 includes an antenna 1201, a radio frequency device 1202, a baseband device 1203, a processor 1204, and a memory 1205. The antenna 1201 is connected to the radio frequency device 1202. In the uplink direction, the radio frequency device 1202 receives information through the antenna 1201 and transmits the received information to the baseband device 1203 for processing. In the downlink direction, the baseband device 1203 processes the information to be transmitted and transmits it to the radio frequency device 1202, which processes the received information before transmitting it via the antenna 1201.

[0149] In the above embodiments, the methods performed by the network side equipment can be implemented in a baseband device 1203, which includes a baseband processor.

[0150] The baseband device 1203 may, for example, include at least one baseband board on which multiple chips are installed, and as shown in FIG. 12, one of the chips is, for example, a baseband processor connected to a memory 1205 via a bus interface and calling a program in the memory 1205 to perform the operations of the network equipment shown in the above method embodiments.

[0151] The network side device may further include a network interface 1206, which may be, for example, a common public radio interface (CPRI).

[0152] Specifically, the network side device 1200 of the embodiment of the present disclosure further includes a command or program stored in the memory 1205 and executable by the processor 1204, and the processor 1204 invokes the command or program in the memory 1205 to execute the method executed by each module shown in Figures 8 to 10, thereby achieving the same technical effect. In order to avoid repetition, detailed description will be omitted here.

[0153] The embodiments of the present application further provide a readable storage medium on which a program or command is stored, and when the program or command is executed by a processor, the steps of the above-mentioned model monitoring processing method embodiment are realized and the same technical effects can be achieved. In order to avoid repetition, detailed descriptions are omitted here.

[0154] Wherein, the processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer readable storage medium such as a computer read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0155] The embodiments of the present application further provide a chip including a processor and a communication interface, the communication interface and the processor being coupled together, and the processor executing a program or command to implement each step of the embodiment of the model supervision processing method, and achieving the same technical effect. In order to avoid repetition, detailed description will be omitted here.

[0156] It should be understood that the chips referred to in the embodiments of this application may also be referred to as system level chips, system chips, chip systems, or system-on-chips, etc.

[0157] The embodiments of the present application further provide a computer program / program product that can be stored in a storage medium and executed by at least one processor to realize each step of the embodiments of the above-mentioned model monitoring processing method and achieve the same technical effects, and detailed descriptions thereof will be omitted here to avoid repetition.

[0158] The embodiments of the present application further provide a communication system that includes a first network device for performing each step of the method embodiments of FIG. 2 and the first network device, a second network device for performing each step of the method embodiments of FIG. 6 and the second network device, and a third network device for performing each step of the method embodiments of FIG. 7 and the third network device, and can achieve the same technical effects. Detailed descriptions will be omitted here to avoid repetition.

[0159] It should be noted that, as used herein, terms such as "comprises," "consists of," or any other variation thereof are intended to include a non-exclusive inclusion, such that a process, method, article, or apparatus comprising a set of elements includes not only those elements but also other elements not expressly specified or inherent in such process, method, article, or apparatus. Unless otherwise specified, an element qualified by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element. It should also be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may include performing functions substantially simultaneously or in the reverse order, depending on such functionality. For example, the described method may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to one example may be combined in other examples.

[0160] From the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be realized in the form of a combination of software and a necessary common hardware platform, and of course, they can also be realized by hardware, but in many cases the former is a more preferred embodiment. Based on this view, the technical solutions of the present application can be substantially embodied in the form of a computer software product, which is stored in a storage medium (e.g., ROM / RAM, magnetic disk, optical disk) and includes a plurality of commands that cause a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0161] Although the examples of the present application have been described above with reference to the drawings, the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative and not limiting. Based on the suggestions of the present application, many forms that a person skilled in the art can make without departing from the spirit of the present application and the scope of protection of the claims are all within the scope of protection of the present application.

Claims

1. receiving, by the first network device, first information from the second network device, the first information including a data analysis result; A model monitoring processing method comprising: a step in which the first network device transmits second information to the second network device or a third network device, the second information being for indicating the usage status of the data analysis results by the first network device.

2. 2. The model monitoring processing method of claim 1, wherein the usage status of the data analysis result is for assisting the second network device or the third network device in estimating calculation of first performance information of a first model, the first model being a model obtained by the third network device performing a model training process, and the first model is for the second network device to generate the data analysis result.

3. 2. The model monitoring processing method of claim 1, wherein the second information includes first instruction information, and the first instruction information is for indicating whether the first network device has performed a first network operation based on the data analysis result.

4. The second information is type information or description information of the first network operation; first time information for indicating a time range for performing the first network operation; first location information for indicating a location range in which the first network operation is to be performed; Further, at least one of first target information for indicating a target on which the first network operation is to be performed is included; The method of claim 2 , wherein the objects include at least one of a user equipment object, an area object, and a network element object.

5. The second information is 5. The model monitoring processing method of claim 3 or claim 4, further comprising second instruction information, the second instruction information being for indicating whether the first network operation affects the estimated calculation of the first performance information of the first model, or for indicating whether the first network operation affects the estimated calculation of the accuracy of the data analysis result.

6. the second information corresponds to the data analysis result. task identifier information, Task conditional information, The model monitoring processing method according to claim 3 , further comprising at least one piece of data analysis task information included in the analysis target information of the task.

7. the first information corresponds to the data analysis result. reliability information indicating a degree of reliability of the data analysis result of the second network device; Second performance information of the first model; 7. A method according to any one of claims 1 to 6, further comprising determining at least one of the data types of interest for determining the second indication.

8. The step of transmitting second information from the first network device to the second network device or the third network device includes:

8. A model monitoring processing method according to claim 1, further comprising a step of the first network device transmitting N pieces of second information to the second network device or the third network device, where N is an integer greater than 1, and the N pieces of second information are determined by the first network device based on the N pieces of first information.

9. The step of transmitting N pieces of second information from the first network device to the second network device or the third network device includes: When a predetermined reporting time point or an end time of a predetermined reporting period is reached, the first network device transmits N pieces of second information to the second network device or the third network device; or 9. The model monitoring processing method of claim 8, further comprising a step of the first network device transmitting N pieces of second information to the second network device or the third network device when the first network device determines that the number of unsent second information has reached N.

10. The first network device, N, which is the number of the second information; The method of claim 9 , further comprising the step of transmitting at least one of the predetermined reporting time point or the predetermined reporting period to the second network device or a third network device.

11. transmitting first information from the second network device to the first network device, the first information including a data analysis result; A model monitoring processing method comprising: a step in which the second network device receives second information from the first network device, the second information being for indicating a usage status of the data analysis results by the first network device.

12. 12. The model monitoring processing method of claim 11, further comprising a step in which the second network device performs an estimated calculation of first performance information of a first model based on the second information, the first model being a model obtained by a third network device performing a model training process, and the first model being used by the second network device to generate the data analysis result.

13. 13. The model monitoring processing method according to claim 11 or 12, wherein the second information includes first instruction information, and the first instruction information is for indicating whether the first network device has performed a first network operation based on the data analysis result.

14. The second information is type information or description information of the first network operation; first time information for indicating a time range for performing the first network operation; first location information for indicating a location range in which the first network operation is to be performed; Further, at least one of first target information for indicating a target on which the first network operation is to be performed is included; The method of claim 13 , wherein the objects include at least one of a user equipment object, an area object, and a network element object.

15. The second information is 15. The model monitoring processing method of claim 13 or 14, further comprising second instruction information, the second instruction information being for indicating whether the first network operation affects the estimated calculation of the first performance information of the first model, or for indicating whether the first network operation affects the estimated calculation of the accuracy of the data analysis result.

16. the second information corresponds to the data analysis result. task identifier information, Task conditional information, The model monitoring processing method according to claim 13 , further comprising at least one piece of data analysis task information included in the analysis target information of the task.

17. The step of the second network device estimating and calculating first performance information of a first model based on the second information includes: the second network device collecting target data based on the second information; The model monitoring processing method according to claim 12 , further comprising: a step in which the second network device performs an estimation calculation of first performance information of the first model based on the target data.

18. 18. The model monitoring processing method of claim 17, wherein when the first instruction information in the second information indicates that the first network device has performed a first network operation based on the data analysis result, the target data includes data not related to the first network operation.

19. The step of collecting target data by the second network device based on the second information includes: The second network device determines, based on the first time information in the second information, data that is not related to the first time information as the target data; The second network device determines, based on the first target information in the second information, data that is not related to the first target information as the target data; and a step in which the second network device determines, based on the first location information in the second information, data not related to the first location information as the target data.

20. The step of collecting target data by the second network device based on the second information includes: The second network device acquires a full set of data for first performance information estimation calculation from a fourth network device based on data analysis task information corresponding to the second information; 19. The model monitoring processing method of claim 18, further comprising the steps of: the second network device, based on first time information in the second information, removing data related to the first time information from the full set data to obtain the target data; and / or the second network device, based on first target information in the second information, removing data related to the first target information from the full set data to obtain the target data; and / or the second network device, based on first location information in the second information, removing data related to the first location information from the full set data to obtain the target data.

21. 18. The model monitoring processing method of claim 17, wherein, when first instruction information in the second information indicates that the first network device did not perform the first network operation based on the data analysis result, the target data includes a full set of data for first performance information estimation calculation obtained by the second network device from a fourth network device based on data analysis task information corresponding to the second information.

22. After the second network device receives the second information from the first network device, The method of claim 11 , further comprising the step of the second network device transmitting the second information to a third network device.

23. The step of receiving second information from the first network device by the second network device includes:

23. A model monitoring processing method according to any one of claims 11 to 22, comprising a step in which the second network device receives N pieces of second information from the first network device, where N is an integer greater than 1, and the N pieces of second information are determined by the first network device based on the N pieces of first information.

24. The second network device receives the information transmitted from the first network device. N, which is the number of the second information; 24. The method of claim 23, further comprising receiving at least one of a predetermined reporting time point or a predetermined reporting period.

25. The first information is reliability information indicating a degree of reliability of the data analysis result of the second network device; Second performance information of the first model; 25. A method according to any one of claims 11 to 24, further comprising determining at least one of the data types of interest for determining the second indication.

26. receiving second information from the first network device or the second network device by the third network device, the second information being for indicating a usage status of the data analysis result by the first network device; A model monitoring processing method comprising: a step in which the third network device performs an estimated calculation of first performance information of a first model based on the second information, wherein the first model is a model obtained by the third network device performing a model training process, and the first model is used by the second network device to generate the data analysis result.

27. 27. The model monitoring processing method of claim 26, wherein the second information includes first instruction information, and the first instruction information is for indicating whether the first network device has performed a first network operation based on the data analysis result.

28. The second information is type information or description information of the first network operation; first time information for indicating a time range for performing the first network operation; first location information for indicating a location range in which the first network operation is to be performed; Further, at least one of first target information for indicating a target on which the first network operation is to be performed is included; 28. The method of claim 27, wherein the objects include at least one of a user equipment object, an area object, and a network element object.

29. The second information is 29. The model monitoring processing method of claim 27 or claim 28, further comprising second instruction information, the second instruction information being for indicating whether the first network operation affects the estimated calculation of the first performance information of the first model, or for indicating whether the first network operation affects the estimated calculation of the accuracy of the data analysis result.

30. The second information corresponds to the data analysis result. task identifier information, Task conditional information, 30. The model monitoring processing method according to claim 27, further comprising at least one piece of data analysis task information among the analysis target information of the task.

31. The step of the third network device estimating and calculating first performance information of a first model based on the second information includes: the third network device collecting target data based on the second information; The model monitoring processing method according to claim 26, further comprising the step of: the third network device performing an estimation calculation of first performance information of the first model based on the target data.

32. 32. The model monitoring processing method of claim 31, wherein when the first instruction information in the second information indicates that the first network device has performed a first network operation based on the data analysis result, the target data includes data not related to the first network operation.

33. The step of collecting target data by the third network device based on the second information includes: The third network device determines, based on the first time information in the second information, data that is not related to the first time information as the target data; The third network device determines, based on the first object information in the second information, data that is not related to the first object information as the target data; and a step in which the third network device determines, based on the first location information in the second information, data not related to the first location information as the target data.

34. The step of collecting target data by the third network device based on the second information includes: The third network device acquires a full set of data for first performance information estimation calculation from a fourth network device based on data analysis task information corresponding to the second information; The model monitoring processing method of claim 32, further comprising the steps of: the third network device, based on first time information in the second information, removing data related to the first time information from the full set data to obtain the target data; and / or the third network device, based on first target information in the second information, removing data related to the first target information from the full set data to obtain the target data; and / or the third network device, based on first location information in the second information, removing data related to the first location information from the full set data to obtain the target data.

35. 32. The model monitoring processing method of claim 31, wherein, when first instruction information in the second information indicates that the first network device did not perform the first network operation based on the data analysis result, the target data includes a full set of data for first performance information estimation calculation obtained by the third network device from a fourth network device based on data analysis task information corresponding to the second information.

36. The step of receiving second information from the first network device or the second network device by the third network device includes:

36. A model monitoring processing method as described in any one of claims 26 to 35, comprising a step in which a third network device receives N pieces of second information from the first network device or the second network device, N being an integer greater than 1, the N pieces of second information being determined by the first network device based on the N pieces of first information, and the first information including data analysis results.

37. The third network device receives the information transmitted from the first network device or the second network device. N, which is the number of the second information; 37. The method of claim 36, further comprising receiving at least one of a predetermined reporting time point or a predetermined reporting period.

38. The first information is reliability information indicating a degree of reliability of the data analysis result of the second network device; Second performance information of the first model; 37. The method of claim 36, further comprising determining at least one of the data types of interest for determining the second indication.

39. a first receiving module for receiving first information from the second network device, the first information including a data analysis result; A model monitoring processing device comprising: a first transmission module for transmitting second information to the second network device or a third network device, the second information being for indicating a usage status of the data analysis results by the first network device.

40. a second transmitting module for transmitting first information to the first network device, the first information including a data analysis result; a second receiving module for receiving second information from the first network device, the second information indicating a usage status of the data analysis results by the first network device;

41. a third receiving module for receiving second information from the first network device or the second network device, the second information indicating a usage status of the data analysis result by the first network device; a second estimation calculation module for performing an estimation calculation of first performance information of a first model based on the second information, wherein the first model is a model obtained by a third network device performing a model training process, and the first model is used by the second network device to generate the data analysis result.

42. A network side device comprising a processor and a memory, wherein the memory stores a program or command executable by the processor, and when the program or command is executed by the processor, the steps of the model monitoring processing method described in any one of claims 1 to 38 are realized.

43. A readable storage medium having stored thereon a program or commands, the program or commands being executed by a processor to implement the steps of the model supervision processing method according to any one of claims 1 to 38.

Images