Model supervision processing method and apparatus, network-side device, and readable storage medium

Abstract

The application discloses a model supervision processing method and device, a network side equipment and a readable storage medium, and belongs to the technical field of communication. The model supervision processing method of the application comprises the following steps: a first network equipment receives first information from a second network equipment, wherein the first information comprises a data analysis result; and the first network equipment sends second information to the second network equipment or a third network equipment, wherein the second information comprises usage range information of the data analysis result of the first network equipment.

Description

Technical Field

[0001] This application belongs to the field of communication technology, specifically relating to a model supervision processing method, apparatus, network-side device, and readable storage medium. Background Technology

[0002] With the development of communication technology, artificial intelligence models have been introduced into communication systems. Currently, the Network Data Analytics Function (NWDAF) typically needs to use model analysis tasks to obtain analysis results and collect data from the network for comparison with the predicted results in order to determine model performance information in order to fulfill the tasks requested by consumer network elements. However, because consumer network elements may make network operations based on these analysis results that differ from typical operations (network operations not determined by the analysis results), this can affect the data used to calculate model performance information, resulting in poor accuracy in the calculation of model performance information. Summary of the Invention

[0003] This application provides a model supervision processing method, apparatus, network-side device, and readable storage medium, which can solve the problem of poor accuracy in model performance information measurement.

[0004] Firstly, a model supervision processing method is provided, including:

[0005] The first network device receives first information from the second network device, the first information including data analysis results;

[0006] The first network device sends second information to the second or third network device, the second information including the scope of use of the data analysis results by the first network device.

[0007] Secondly, a model supervision processing method is provided, including:

[0008] The second network device sends first information to the first network device, the first information including data analysis results;

[0009] The second network device receives second information from the first network device, the second information including the scope of use of the data analysis results by the first network device.

[0010] Thirdly, a model supervision processing method is provided, including:

[0011] The third network device receives second information from the first network device or the second network device, the second information including the scope of use of the data analysis results by the first network device;

[0012] The third network device calculates the first performance information of the first model based on the second information, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results.

[0013] Fourthly, a model supervision processing device is provided, comprising:

[0014] The first receiving module is used to receive first information from the second network device, the first information including data analysis results;

[0015] The first sending module is used to send second information to the second network device or the third network device, the second information including the scope of use of the data analysis results by the first network device.

[0016] Fifthly, a model supervision processing device is provided, comprising:

[0017] The second sending module is used to send first information to the first network device, the first information including data analysis results;

[0018] The second receiving module is used to receive second information from the first network device, the second information including the scope of use of the data analysis results by the first network device.

[0019] Sixthly, a model supervision processing apparatus is provided, comprising:

[0020] The third receiving module is used to receive second information from the first network device or the second network device, the second information including the scope of use of the data analysis results by the first network device;

[0021] The first calculation module is used to calculate the first performance information of the first model based on the second information, wherein the first model is a model obtained by the third network device through the model training process, and the first model is used by the second network device to generate the data analysis results.

[0022] In a seventh aspect, a network-side device is provided, the network-side device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the second aspect.

[0023] Eighthly, a network-side device is provided, including a processor and a communication interface, wherein,

[0024] When the network-side device is a first network device, the communication interface is used to receive first information from a second network device, the first information including data analysis results; and to send second information to the second network device or a third network device, the second information including the scope of use of the data analysis results by the first network device.

[0025] When the network-side device is a second network device, the communication interface is used to send first information to the first network device, the first information including data analysis results; and to receive second information from the first network device, the second information including the scope of use of the data analysis results by the first network device.

[0026] When the network-side device is a third network device, the communication interface is used to receive second information from the first network device or the second network device. The second information includes the scope of use of the data analysis results by the first network device. The processor is used to calculate the first performance information of the first model based on the second information. The first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results.

[0027] A ninth aspect provides a communication system comprising: a first network device, a second network device, and a third network device, wherein the first network device is configured to perform the steps of the model supervision processing method as described in the first aspect, the second network device is configured to perform the steps of the model supervision processing method as described in the second aspect, and the third network device is configured to perform the steps of the model supervision processing method as described in the third aspect.

[0028] In a tenth aspect, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect, or the steps of the method described in the second aspect, or the steps of the method described in the third aspect.

[0029] Eleventhly, a chip is provided, the chip including a processor and a communication interface coupled to the processor, the processor being configured to run a program or instructions to implement the steps of the method described in the first aspect, or the steps of the method described in the second aspect, or the steps of the method described in the third aspect.

[0030] In a twelfth aspect, a computer program / program product is provided, the computer program / program product being stored in a storage medium, the computer program / program product being executed by at least one processor to implement the steps of the method as described in the first aspect, or the steps of the method as described in the second aspect, or the steps of the method as described in the third aspect.

[0031] In this embodiment of the application, by sending second information from the first network device to the second network device or the third network device, the second network device or the third network device can determine the scope of use of the data analysis results by the first network device based on the second information, so as to assist in the calculation of the first performance information of the first model, thereby improving the accuracy and reliability of the calculation of the model performance information. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the network structure applicable to the embodiments of this application.

[0033] Figure 2 This is one of the flowcharts of the model supervision processing method provided in the embodiments of this application;

[0034] Figure 3 This is the second flowchart of the model supervision processing method provided in the embodiments of this application;

[0035] Figure 4 This is the third flowchart of the model supervision processing method provided in the embodiments of this application;

[0036] Figure 5 This is the fourth flowchart of the model supervision processing method provided in the embodiments of this application;

[0037] Figure 6 This is the fifth flowchart of the model supervision processing method provided in the embodiments of this application;

[0038] Figure 7 This is one of the structural diagrams of the model supervision processing device provided in the embodiments of this application;

[0039] Figure 8 This is a second structural diagram of the model supervision processing device provided in the embodiments of this application;

[0040] Figure 9 This is the third structural diagram of the model supervision processing device provided in the embodiments of this application;

[0041] Figure 10 This is a structural diagram of the communication device provided in the embodiments of this application;

[0042] Figure 11 This is a structural diagram of the network-side device provided in the embodiments of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0044] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0045] It is worth noting that the technologies described in this application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in this application are often used interchangeably, and the described technologies can be used with the systems and radio technologies mentioned above, as well as with other systems and radio technologies. The following description describes New Radio (NR) systems for illustrative purposes, and NR terminology is used in most of the following description; however, these technologies can also be applied to applications beyond NR systems, such as 6th generation (6G) radio systems. th Generation 6G communication system.

[0046] Figure 1This diagram illustrates a block diagram of a wireless communication system applicable to embodiments of this application. The wireless communication system includes a terminal 11 and a network-side device 12. Terminal 11 can be a mobile phone, tablet computer, laptop computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, vehicle-mounted device (VUE), pedestrian terminal (PUE), smart home (home devices with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), game console, personal computer (PC), ATM, or self-service machine, etc. Wearable devices include: smartwatches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart chains, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. It should be noted that the specific type of terminal 11 is not limited in this embodiment. Network-side equipment 12 may include access network equipment or core network equipment. Access network equipment may also be referred to as radio access network equipment, radio access network (RAN), radio access network function, or radio access network unit. Access network equipment may include base stations, WLAN access points, or WiFi nodes, etc. Base stations may be referred to as Node B, evolved Node B (eNB), access point, base transceiver station (BTS), radio base station, radio transceiver, Basic Service Set (BSS), Extended Service Set (ESS), home B node, home evolved B node, Transmitting Receiving Point (TRP), or any other suitable term in the field, as long as the same technical effect is achieved. The base station is not limited to specific technical terms. It should be noted that this application embodiment only uses a base station in an NR system as an example for description and does not limit the specific type of base station.

[0047] Reference Figure 2 This application provides a model supervision processing method, such as... Figure 2 As shown, the model supervision processing method includes:

[0048] Step 201: The first network device receives first information from the second network device, the first information including data analysis results;

[0049] Step 202: The first network device sends second information to the second network device or the third network device. The second information includes information on the scope of use of the data analysis results by the first network device.

[0050] In this embodiment, a first network device first sends a task request for a first task to a second network device. This task request requests the data analysis results of the first task. The first task can be a data analysis task, and the data analysis task information includes task identification information, task condition constraints, and task analysis object information. The task identification information can be a data analysis task ID (analytic ID), which indicates the purpose of the data analysis task, the type of data to be obtained, etc. For example, if the analytic ID is associated with a user's mobility trajectory information, then the data analysis result can be the user's location information. The task condition constraints and task analysis object information are used to further refine the task requirements. The second network device can send a model request to a third network device based on the task request to request a first model for the first task. During the model request process, the model request can include task request information and model constraint information. The task request information includes the request content for the first task, and the model constraint information further clarifies the required task requirements, such as limiting the accuracy of the model. After receiving the model request, the third network device can select or train a first model that meets the requirements of the first task and send it to the second network device. Then, the second network device can perform data analysis on the first task based on the first model, generate data analysis results for the first task, and send first information to the first network device. The first information includes the data analysis results. Next, the first network device can provide feedback to the second network device regarding the scope of application of the data analysis results. In this way, the second or third network device can determine its supervisory behavior towards the first model based on this second information. For example, if the first network device uses the data analysis results, and using the data analysis results would affect the network or the data within the network, the first network device may not calculate the first performance information of the first model, or it may filter out relevant data that affects the calculation of the first performance information during the calculation process. This improves the accuracy and reliability of the model performance information calculation.

[0051] Optionally, in some embodiments, when the second network device sends first information to the first network device, it may carry the identification information, address information, etc., of the third network device. This allows the first network device to know the information of the third network device and directly send information / messages to the third network device. Optionally, before this, the first network device may request the second network device to send the identification information, address information, etc., of the third network device, or the first network device may send its own information to the second network device, which will then forward it to the third network device. Optionally, after the first network device makes a request, the second network device may also send the identification information, address information, etc., of the third network device in a separate message.

[0052] Alternatively, after receiving the model request, the third network device may select or train a first model that meets the requirements of the first task and send it to the second network device. Or, it may request the second network device to provide feedback on the first network device's identification information, address information, etc., in order to send information / messages to the first device. Alternatively, it may request the second network device to send its own information to the first network device to inform the first network device to provide feedback / send second information, etc.

[0053] Optionally, calculating the first performance information of the first model can be understood as supervising the first model. Specifically, the first model can be supervised by a second network device or a third network device. For example, in some embodiments, when the first model is supervised by a third network device, the aforementioned second information can also trigger the second network device to forward the second information to the third network device. Here, supervising the first model can be understood or replaced as supervising the first task. The aforementioned second information is determined based on the first information.

[0054] It should be understood that the aforementioned first performance information is used to represent the model performance when the first model is actually used. That is, the aforementioned first performance information can be understood as the model performance information calculated based on the collected data after the first model is used to perform data analysis on the first task after the model training is completed.

[0055] Optionally, the above data analysis results can be the output values ​​obtained by the second network device using the first model for calculation and inference, such as load prediction for a certain network-side device. The content of the data analysis results will vary depending on the task. For example, for a network device load task, the data analysis results could represent the network device load at a future time, such as the high load of the User Plane Function (UPF) on the morning of the following day.

[0056] It should be noted that the first network device mentioned above can be understood as a consumer network element (NF), the second network device mentioned above can be understood as an NWDAF or its analytical logic function (AnLF) used for inference to generate predictive information or to generate summaries of historical data, and the third network device mentioned above can be understood as an NWDAF or its model training logic function (MTLF) used for generating models and training models.

[0057] In this embodiment of the application, by sending second information from the first network device to the second or third network device, the second or third network device can determine the scope of use of the data analysis results by the first network device based on the second information, so as to assist in the calculation of the first performance information of the first model, thereby improving the accuracy and reliability of the calculation of model performance information.

[0058] Optionally, in some embodiments, the scope information is used to assist the second network device or the third network device in calculating the first performance information of the first model, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results.

[0059] In this embodiment, the scope information of the data analysis results can indicate whether the first network device has performed a network operation determined based on the data analysis results, or whether the first network device will perform a network operation determined based on the data analysis results in the future, or whether a network operation will be determined based on the data analysis results in the future. Furthermore, it can also indicate the scope of use and / or prohibition of the first network device.

[0060] Optionally, in some embodiments, the scope of use information includes at least one of the following:

[0061] The first network device allows the use of the range information of the data analysis results;

[0062] The first network device prohibits the use of the range information of the data analysis results.

[0063] In this embodiment of the application, the scope information on which the first network device is allowed to use the data analysis results can be understood as: the scope information on which the first network device is allowed to perform the first network operation based on the data task information; the scope information on which the first network device is prohibited from using the data analysis results can be understood as: the scope information on which the first network device is prohibited from performing the first network operation based on the data task information, or in other words, the scope information on which the first network device is not allowed to perform the first network operation based on the data task information.

[0064] Optionally, in some embodiments, the range information may include at least one of a time range, a location range, and an object range, wherein the object range includes at least one of a user equipment object, a region object, and a network element object.

[0065] The aforementioned time range can be determined by the first network-side device based on the aforementioned first information. This time range may include a first time range within which the first network device prohibits the use of the data analysis results and / or a second time range within which the first network device allows the use of the data analysis results. For example, after determining the first time range, the first network-side device may decide its network operations based on its previous logic and preset conditions rather than using the aforementioned data analysis results within that first time range. Therefore, the network or the data within the network will not be affected by the network operations of the first network device within that first time range. This first time range can be a single time period or two time nodes to identify a period of time.

[0066] Optionally, the aforementioned location range can be determined by the first network-side device based on the aforementioned first information. This time range may include a first location range within which the first network device prohibits the use of the data analysis results and / or a second location range within which the first network device allows the use of the data analysis results. For example, after determining the first location range, the first network-side device may decide its network operations within this first location range without using the aforementioned data analysis results, but rather based on its previous logic and preset conditions. Therefore, within this first location range, the network or data within the network will not be affected by the network operations of the first network device. The first location range may be a tracking area (TA), cell, etc.

[0067] For example, in some embodiments, it is assumed that the first model is supervised by a second network device, in which case the second network device has the following behavior:

[0068] The second network device collects target data based on the second information;

[0069] The second network device calculates the first performance information of the first model based on the target data.

[0070] Optionally, when the scope of use information is scope information that the first network device prohibits the use of the data analysis results, the second network device collects target data based on the second information including at least one of the following:

[0071] The second network device determines data related to the time range as the target data based on the time range in the second information carried by the second information;

[0072] The second network device determines data related to the object range as the target data based on the object range in the second information carried by the second information;

[0073] The second network device determines data related to the location range as the target data based on the location range in the second information carried by the second information.

[0074] Optionally, when the scope of use information is the scope of use that the first network device is allowed to use the data analysis results, the second network device collects target data based on the second information, including at least one of the following:

[0075] The second network device determines data that is not related to the time range as the target data based on the time range in the second information carried by the second information;

[0076] The second network device determines data that is not related to the object range as the target data based on the object range in the second information carried by the second information;

[0077] The second network device determines data that is not related to the location range as the target data based on the location range in the second information carried by the second information.

[0078] Optionally, when the scope of use information is the scope information that the first network device prohibits the use of the data analysis results, the second network device collects target data according to the second information, including:

[0079] The second network device obtains the complete set of data for calculating the first performance information from the fourth network device based on the data analysis task information corresponding to the second information;

[0080] The second network device selects data related to the time range from the complete set of data based on the time range in the second information to obtain the target data; and / or, the second network device selects data related to the object range from the complete set of data based on the object range in the second information to obtain the target data; and / or, the second network device selects data related to the location range from the complete set of data based on the location range in the second information to obtain the target data.

[0081] Optionally, when the scope of use information is the scope of use that the first network device is allowed to use the data analysis results, the second network device collects target data according to the second information, including:

[0082] The second network device obtains the complete set of data for calculating the first performance information from the fourth network device based on the data analysis task information corresponding to the second information;

[0083] The second network device removes data related to the time range from the complete set of data based on the time range in the second information to obtain the target data; and / or, the second network device removes data related to the object range from the complete set of data based on the object range in the second information to obtain the target data; and / or, the second network device removes data related to the location range from the complete set of data based on the location range in the second information to obtain the target data.

[0084] Optionally, in some embodiments, if the first network device does not perform the first network operation based on the data analysis results, the target data includes the complete set of data obtained by the second network device from the fourth network device according to the data analysis task information corresponding to the second information, which is used to calculate the first performance information.

[0085] In this embodiment, the fourth network device can be understood as a data source network element. It should be noted that in the above embodiments, if the first model is supervised by a third network device, the specific process simply involves replacing the second network device in the above embodiments with the third network device, and will not be elaborated further here.

[0086] Optionally, in some embodiments, the second information further includes at least one of the following:

[0087] The first instruction information is used to instruct a request to calculate the first performance information of the first model.

[0088] The information used by the first network device to perform a first network operation based on the data analysis results.

[0089] In this embodiment, by further feeding back the aforementioned second instruction information, when the first network device performs the first network operation based on the data analysis results without affecting the network or its data, the second or third network device can collect all data used to calculate the first performance information (i.e., the complete set of data), and then calculate the first performance information based on all the collected data. This increases the amount of data used to calculate the first performance information, further improving the accuracy of the first performance information calculation for the first model.

[0090] Optionally, if the second information includes information about the first network operation, the second information may further include second indication information, which may indicate whether the first network operation will affect the calculation of the first performance information of the first model by (the second network device or the third network device), or whether the second indication information will affect the calculation of the correctness of the data analysis results.

[0091] Optionally, the information of the first network operation includes:

[0092] The type information or description information of the first network operation.

[0093] Optionally, the instruction to request the calculation of the first performance information of the first model can be an explicit instruction, such as an instruction to request the calculation of the first performance information of the first model displayed by the aforementioned first instruction information, which can be a specific information cell; furthermore, the instruction to perform model supervision on the first model can also be an implicit instruction, for example, the instruction to request the calculation of the first performance information of the first model can be implicitly indicated based on the signaling name carrying the second information.

[0094] Optionally, the type information of the first network operation mentioned above may include slice selection, network load balancing, and terminal access control, etc. The specific network operation description may be a comprehensive information that includes information such as the object, time, and location, as well as the network operation type information; for example, when an access control operation is performed on a particular terminal.

[0095] Optionally, in some embodiments, the second information further includes at least one of the following data analysis task information corresponding to the data analysis result:

[0096] Task identification information;

[0097] Task condition constraints;

[0098] Information about the objects being analyzed in the task.

[0099] The aforementioned task identification information, task condition constraints, and task analysis object information can be used to determine a specific task. The task identification information indicates the task to which the second piece of information pertains; it can also be all three pieces of information together, or just the first two pieces of information, etc., to determine the specific task. The task condition constraints, also known as analytical filter information, indicate the filtering information for the data analysis results. Examples include Areas of Interest (AOI), Single Network Slice Selection Assistance Information (S-NSSAI), and Data Network Name (DNN). The task object information can be understood as the target of the analytical reporting, specifying whether the task analysis object is a single terminal, multiple terminals, all terminals, a single network element, multiple network elements, or all network elements.

[0100] Optionally, in some embodiments, the first information further includes at least one of the following pieces of information corresponding to the data analysis results:

[0101] Confidence information, which is used to indicate the degree of trust that the second network device has in the data analysis results;

[0102] Second performance information of the first model, which is used to represent the performance of the first model during training;

[0103] Data types of interest.

[0104] Optionally, the confidence information mentioned above may include three confidence levels: high, medium, and low.

[0105] Optionally, the aforementioned second performance information can be understood as the model performance information during the training of the first model, i.e., the result of the third network device evaluating the model during training, such as calculating the model's accuracy in training (AiT) using the training and testing datasets during training. The form of the model performance information is not specifically limited; it can indicate the accuracy or error level of the model's inference results for the task from either a positive or negative perspective. For example, the inference accuracy of the first model can be negatively indicated by calculating the model's inference error or inference error rate. The calculation methods for inference error or inference error rate are diverse, such as Mean Absolute Error (MAE) and Mean Square Error (MSE). It can also be a positive indicator of accuracy, such as precision or accuracy rate.

[0106] Optionally, the aforementioned data types of interest are used to indicate the specific data type of interest, such as the load information of a network-side device. This can help the first network device determine whether its network operations will affect the environment. For example, the first network device can determine for itself whether its network operations will affect this data type.

[0107] It should be noted that when determining network operations, the first network device can use the confidence information and second performance information in the first information to determine whether the data analysis result is reliable and thus whether it can be used. For example, when a Session Management Function (SMF) network element selects a UPF based on network load information, if the analysis result of the received network load information indicates that the network load of a certain UPF at a future point in time is low, then the SMF may select that UPF (perform a network operation). Furthermore, since the SMF knows in advance that the UPF may have a low load and selects this UPF, if the load of that UPF is not as low as initially analyzed / predicted, then the network operation affects the network environment (data in the network environment, such as the load level of the UPF). Conversely, if the confidence in the first information is relatively low, and / or the model's performance during training is not good, or some internal logic and conditions of the first network device cause the first network device to decide not to use the data analysis result, then the network operation performed by the first network device will not be affected by the data analysis result, nor will it affect the network environment.

[0108] Optionally, in some embodiments, after the first network device sends the second information to the second network device or the third network device, the method further includes:

[0109] The first network device receives the first performance information from the second network device or the third network device.

[0110] In this embodiment, the first model can be supervised by a second network device or by a third network device. When the first model is supervised by a third network device, the third network device sends first performance information to the second network device, and then the second network device forwards the first performance information to the first network device; or the third network device directly sends the first performance information to the first network device.

[0111] It should be noted that when the second network device supervises the first model, the second network device can begin performing data operations after receiving the first information, or it can begin performing data operations after receiving the second information. This data operation can be understood as collecting a complete set of data that can be used to calculate the first performance information. For example, in some embodiments, the second network device can begin performing data operations after receiving the first information. After receiving the second information, it can determine the relevant data in the complete set of data corresponding to the second information, and then either not collect the relevant data or remove (or filter out) the relevant data from the already collected complete set of data to obtain the target data. Finally, the first performance information is calculated based on the determined target data. When the third network device supervises the first model, the second network device forwards the received second information to the third network device. After receiving the second information, the third network device can determine the relevant data in the complete set of data corresponding to the second information, and then either not collect the relevant data or remove (or filter out) the relevant data from the already collected complete set of data to obtain the target data. Finally, the first performance information is calculated based on the determined target data.

[0112] To better understand this application, the following examples will be used to illustrate it in detail.

[0113] Step 30: The consumer initiates the first task, AnLF initiates a model request, and MTLF selects or trains a first model that meets the requirements of the first task. The concept of the first task is similar to that in step 202, and also similar to the data analysis task concept described earlier.

[0114] Step 31: MTLF sends the first model to AnLF, and AnLF then uses the first model to generate data analysis results.

[0115] Step 32: AnLF sends the first message to the consumer, which includes the data analysis results. Specifically, AnLF can use the first model obtained in step 31 to perform calculations and inferences to obtain the output value, i.e., the data analysis result of the first task, such as the network element load prediction for a certain network element. AnLF feeds back the data analysis result to the consumer, and can also inform the consumer of relevant information about the analysis result. This relevant information can help the consumer decide whether to use the data analysis result.

[0116] In addition to data analysis results and related information, the first piece of information may also include at least one of the following:

[0117] Identification information is used to inform AnLF of the specific task it is targeting. For example, it may include at least one of the following: task identification information, task conditional information, and task object information;

[0118] Model identification information;

[0119] The types of tags you are interested in.

[0120] The aforementioned relevant information may include at least one of confidence information and model performance information during the training of the first model.

[0121] Step 33: Based on the first information fed back by AnLF in step 32, the consumer decides whether to use the data analysis results to determine the network operation to be performed, and sends the second information based on the decision. Specifically, the consumer can determine whether to use the data analysis results based on relevant information from the AnLF analysis results, such as the confidence information of the first model and the model performance information during training. For example, if the confidence information and training performance information of the first model corresponding to the first task are "medium" and "accuracy, 70%", the consumer considers the model analysis results unreliable, chooses not to use the data analysis results of the first model, and sends the second information.

[0122] In step 34, the consumer sends a second message to AnLF indicating its scope of use of the data analysis results. This second message also informs AnLF of the permitted or prohibited timeframes, geographical areas, and target groups for using the data analysis results. AnLF can then collect data based on this second message and calculate the first performance information of the first model based on the collected data. For example, the consumer might include in this second message the timeframe during which no network operations were performed based on the data analysis results; this timeframe could be a period of one week.

[0123] Optionally, the second information may include at least one of the following:

[0124] The instruction message requests model supervision, indicating that AnLF should supervise and compute the model performance. This instruction message can be explicit, such as a specific information cell, or implicit, such as being known based on the signaling name carrying this second information.

[0125] Indication information regarding the use of data analysis results is used to indicate whether the consumer should use the data analysis results sent by AnLF. This indication information can appear explicitly, such as in a specific cell; or it can appear implicitly, such as when the indication information is known based on the signaling name carrying the second information.

[0126] Identification information is used to inform AnLF that it is for a specific task. This identification information may include at least one of the following: task identification information, task conditional information, and task object information.

[0127] The timeframe for determining network operations not based on data analysis results is used to indicate that during this period, the consumer will not use the data analysis results to decide its network operations, but will instead base them on its previous logic and other pre-defined conditions. Therefore, during this period, the consumer's actions will not affect the network or the data within it. This time information can be a single time period or two time points to identify a time interval.

[0128] The location range for network operations not determined based on data analysis results is used to indicate that the consumer will not use the analysis results to determine its network operations within this location range. In other words, when the target UE / network element or other device is within this location range, the consumer's network operations will be determined based on pre-defined conditions such as previous logic, rather than on the data analysis results. This location range can be a specific TA, cell, etc.

[0129] The scope of network operations was not determined based on the data analysis results. This is used to indicate that consumers will not use the analysis results to determine their network operations within this scope.

[0130] Step 35, optionally, AnLF sends a first response message to the consumer indicating whether it intends to begin model supervision of the first model. Specifically, after receiving the second information from step 34, AnLF decides whether to perform model supervision on the first model. Its decision may consider the identification information, time range, location range, etc., from step 34, as well as its own situation (e.g., its own workload). After making the decision, it sends the result back to the consumer. For example, if AnLF's own workload is medium and it has not yet supervised the first model corresponding to the first task, it may supervise the first model according to the time range, location range, and / or object range.

[0131] Step 36: AnLF collects other relevant data, calculates the model performance information of the first model, and after calculating the model performance information, determines the subsequent operations, such as including at least one of the following:

[0132] Inform the consumer about the model's performance information;

[0133] Inform MTLF of the model performance information and / or relevant data;

[0134] Store model performance information and / or related data in the Analytics Data Repository Function (ADRF).

[0135] Example 2: The first model is supervised by MTLF. (Refer to...) Figure 4 The difference between this embodiment and the first embodiment described above is that the MTLF supervises the first model and calculates its performance information. Specifically, after the AnLF receives the second information, the AnLF sends the second information to the MTLF; that is, the AnLF forwards the second information received from the consumer.

[0136] Furthermore, after MTLF determines the model performance information of the first model, at least one of the following can be performed:

[0137] The model performance information is then communicated to AnLF, which in turn forwards it to the consumer.

[0138] Tell AnLF the model performance information and / or relevant data;

[0139] Store model performance information and / or related data in ADRF.

[0140] Reference Figure 5 This application also provides a model supervision processing method, such as... Figure 5 As shown, the model supervision processing method includes:

[0141] Step 501: The second network device sends first information to the first network device, the first information including data analysis results;

[0142] Step 502: The second network device receives second information from the first network device, the second information including the scope of use of the data analysis results by the first network device.

[0143] Optionally, the scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results.

[0144] Optionally, the scope of use information includes at least one of the following:

[0145] The first network device allows the use of the range information of the data analysis results;

[0146] The first network device prohibits the use of the range information of the data analysis results.

[0147] Optionally, the range information includes at least one of time range, location range, and object range, and the objects included in the object range include at least one of user equipment objects, area objects, and network element objects.

[0148] Optionally, the second information further includes at least one of the following:

[0149] The first instruction information is used to instruct a request to calculate the first performance information of the first model.

[0150] The information used by the first network device to perform a first network operation based on the data analysis results.

[0151] Optionally, if the second information includes information about the first network operation, the second information may further include second indication information, which may indicate whether the first network operation will affect the calculation of the first performance information of the first model, or whether the second indication information will affect the calculation of the correctness of the data analysis results.

[0152] Optionally, the information of the first network operation includes:

[0153] The type information or description information of the first network operation.

[0154] Optionally, the second information may also include at least one of the following data analysis task information corresponding to the data analysis results:

[0155] Task identification information;

[0156] Task condition constraints;

[0157] Information about the objects being analyzed in the task.

[0158] Optionally, the first information may also include at least one of the following pieces of information corresponding to the data analysis results:

[0159] Confidence information, which is used to indicate the degree of trust that the second network device has in the data analysis results;

[0160] Second performance information of the first model, which is used to represent the performance of the first model during training.

[0161] Optionally, the method further includes:

[0162] The second network device calculates the first performance information of the first model based on the second information.

[0163] Optionally, the second network device calculates the first performance information of the first model based on the second information, including:

[0164] The second network device collects target data based on the second information;

[0165] The second network device calculates the first performance information of the first model based on the target data.

[0166] Optionally, when the scope of use information is scope information that the first network device prohibits the use of the data analysis results, the second network device collects target data based on the second information including at least one of the following:

[0167] The second network device determines data related to the time range as the target data based on the time range in the second information carried by the second information;

[0168] The second network device determines data related to the object range as the target data based on the object range in the second information carried by the second information;

[0169] The second network device determines data related to the location range as the target data based on the location range in the second information carried by the second information.

[0170] Optionally, when the scope of use information is the scope of use that the first network device is allowed to use the data analysis results, the second network device collects target data based on the second information, including at least one of the following:

[0171] The second network device determines data that is not related to the time range as the target data based on the time range in the second information carried by the second information;

[0172] The second network device determines data that is not related to the object range as the target data based on the object range in the second information carried by the second information;

[0173] The second network device determines data that is not related to the location range as the target data based on the location range in the second information carried by the second information.

[0174] Optionally, when the scope of use information is the scope information that the first network device prohibits the use of the data analysis results, the second network device collects target data according to the second information, including:

[0175] The second network device obtains the complete set of data for calculating the first performance information from the fourth network device based on the data analysis task information corresponding to the second information;

[0176] The second network device selects data related to the time range from the complete set of data based on the time range in the second information to obtain the target data; and / or, the second network device selects data related to the object range from the complete set of data based on the object range in the second information to obtain the target data; and / or, the second network device selects data related to the location range from the complete set of data based on the location range in the second information to obtain the target data.

[0177] Optionally, after the second network device receives the second information from the first network device, the method further includes:

[0178] The second network device sends a third indication message to the first network device, the third indication message being used to indicate whether the second network device should perform the calculation of the first performance information of the first model.

[0179] Optionally, the method further includes:

[0180] The second network device sends the second information to the third network device.

[0181] Reference Figure 6 This application also provides a model supervision processing method, such as... Figure 6 As shown, the model supervision processing method includes:

[0182] Step 601: The third network device receives second information from the first network device or the second network device, the second information including the scope of use of the data analysis results by the first network device;

[0183] Step 602: The third network device calculates the first performance information of the first model based on the second information, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results.

[0184] Optionally, the scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model.

[0185] Optionally, the scope of use information includes at least one of the following:

[0186] The first network device allows the use of the range information of the data analysis results;

[0187] The first network device prohibits the use of the range information of the data analysis results.

[0188] Optionally, the range information includes at least one of time range, location range, and object range, and the objects included in the object range include at least one of user equipment objects, area objects, and network element objects.

[0189] Optionally, the second information further includes at least one of the following:

[0190] The first instruction information is used to instruct a request to calculate the first performance information of the first model.

[0191] The information used by the first network device to perform a first network operation based on the data analysis results.

[0192] Optionally, if the second information includes information about the first network operation, the second information may further include second indication information, which may indicate whether the first network operation will affect the calculation of the first performance information of the first model, or whether the second indication information will affect the calculation of the correctness of the data analysis results.

[0193] Optionally, the information of the first network operation includes:

[0194] The type information or description information of the first network operation.

[0195] Optionally, the second information may also include at least one of the following data analysis task information corresponding to the data analysis results:

[0196] Task identification information;

[0197] Task condition constraints;

[0198] Information about the objects being analyzed in the task.

[0199] Optionally, the third network device, based on the second information, calculates the first performance information of the first model, including:

[0200] The third network device collects target data based on the second information;

[0201] The third network device calculates the first performance information of the first model based on the target data.

[0202] Optionally, the third network device collects target data based on the second information, including at least one of the following:

[0203] The third network device determines data related to the time range as the target data based on the time range in the second information carried by the second information.

[0204] The third network device determines data related to the object range as the target data based on the object range in the second information carried by the second information;

[0205] The third network device determines data related to the location range as the target data based on the location range in the second information carried by the second information.

[0206] Optionally, when the scope of use information is the scope of use permitted by the first network device for the data analysis results, the third network device collects target data based on the second information, including at least one of the following:

[0207] The third network device determines data that is not related to the time range as the target data based on the time range in the second information carried by the second information;

[0208] The third network device determines data that is not related to the object range as the target data based on the object range in the second information carried by the second information;

[0209] The third network device determines data that is not related to the location range as the target data based on the location range in the second information carried by the second information.

[0210] Optionally, when the scope of use information is the scope information that the first network device prohibits the use of the data analysis results, the third network device collects target data according to the second information, including:

[0211] The third network device obtains the complete set of data for calculating the first performance information from the fourth network device based on the data analysis task information corresponding to the second information;

[0212] The third network device selects data related to the time range from the complete set of data according to the time range in the second information to obtain the target data; and / or, the third network device selects data related to the object range from the complete set of data according to the object range in the second information to obtain the target data; and / or, the third network device selects data related to the location range from the complete set of data according to the location range in the second information to obtain the target data.

[0213] The model supervision processing method provided in this application can be executed by a model supervision processing device. This application uses the execution of the model supervision processing method by a model supervision processing device as an example to illustrate the model supervision processing device provided in this application.

[0214] Reference Figure 7 This application also provides a model supervision processing device, such as... Figure 7 As shown, the model supervision processing device 700 includes:

[0215] The first receiving module 701 is used to receive first information from the second network device, the first information including data analysis results;

[0216] The first sending module 702 is used to send second information to the second network device or the third network device, the second information including the scope of use of the data analysis results by the first network device.

[0217] Optionally, the scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results.

[0218] Optionally, the scope of use information includes at least one of the following:

[0219] The first network device allows the use of the range information of the data analysis results;

[0220] The first network device prohibits the use of the range information of the data analysis results.

[0221] Optionally, the range information includes at least one of time range, location range, and object range, and the objects included in the object range include at least one of user equipment objects, area objects, and network element objects.

[0222] Optionally, the second information further includes at least one of the following:

[0223] The first instruction information is used to instruct a request to calculate the first performance information of the first model.

[0224] The information used by the first network device to perform a first network operation based on the data analysis results.

[0225] Optionally, if the second information includes information about the first network operation, the second information may further include second indication information, which may indicate whether the first network operation will affect the calculation of the first performance information of the first model, or whether the second indication information will affect the calculation of the correctness of the data analysis results.

[0226] Optionally, the information of the first network operation includes:

[0227] The type information or description information of the first network operation.

[0228] Optionally, the second information may also include at least one of the following data analysis task information corresponding to the data analysis results:

[0229] Task identification information;

[0230] Task condition constraints;

[0231] Information about the objects being analyzed in the task.

[0232] Optionally, the first information may also include at least one of the following pieces of information corresponding to the data analysis results:

[0233] Confidence information, which is used to indicate the degree of trust that the second network device has in the data analysis results;

[0234] Second performance information of the first model, which is used to represent the performance of the first model during training.

[0235] Reference Figure 8 This application also provides a model supervision processing device, such as... Figure 8 As shown, the model supervision processing device 800 includes:

[0236] The second sending module 801 is used to send first information to the first network device, the first information including data analysis results;

[0237] The second receiving module 802 is used to receive second information from the first network device, the second information including the scope of use of the data analysis results by the first network device.

[0238] Optionally, the scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results.

[0239] Optionally, the scope of use information includes at least one of the following:

[0240] The first network device allows the use of the range information of the data analysis results;

[0241] The first network device prohibits the use of the range information of the data analysis results.

[0242] Optionally, the range information includes at least one of time range, location range, and object range, and the objects included in the object range include at least one of user equipment objects, area objects, and network element objects.

[0243] Optionally, the second information further includes at least one of the following:

[0244] The first instruction information is used to instruct a request to calculate the first performance information of the first model.

[0245] The information used by the first network device to perform a first network operation based on the data analysis results.

[0246] Optionally, if the second information includes information about the first network operation, the second information may further include second indication information, which may indicate whether the first network operation will affect the calculation of the first performance information of the first model, or whether the second indication information will affect the calculation of the correctness of the data analysis results.

[0247] Optionally, the information of the first network operation includes:

[0248] The type information or description information of the first network operation.

[0249] Optionally, the second information may also include at least one of the following data analysis task information corresponding to the data analysis results:

[0250] Task identification information;

[0251] Task condition constraints;

[0252] Information about the objects being analyzed in the task.

[0253] Optionally, the first information may also include at least one of the following pieces of information corresponding to the data analysis results:

[0254] Confidence information, which is used to indicate the degree of trust that the second network device has in the data analysis results;

[0255] Second performance information of the first model, which is used to represent the performance of the first model during training.

[0256] Optionally, the model supervision processing device further includes:

[0257] The second calculation module is used to calculate the first performance information of the first model based on the second information.

[0258] Optionally, the second calculation module includes:

[0259] The second collection unit is used to collect target data based on the second information;

[0260] The second calculation unit is used to calculate the first performance information of the first model based on the target data.

[0261] Optionally, when the scope information is scope information that the first network device prohibits the use of the data analysis results, the second collection unit is specifically used to perform at least one of the following:

[0262] Based on the time range in the second information carried by the second information, data related to the time range is determined as the target data;

[0263] Based on the object range in the second information carried by the second information, determine the data related to the object range as the target data;

[0264] Based on the location range in the second information carried by the second information, data related to the location range is determined as the target data.

[0265] Optionally, when the scope of use information is the scope of use that the first network device is allowed to use the data analysis results, the second collection unit is specifically configured to perform at least one of the following:

[0266] Based on the time range in the second information carried by the second information, data that is not related to the time range is determined as the target data;

[0267] Based on the object range in the second information carried by the second information, data that is not related to the object range is determined as the target data;

[0268] Based on the location range in the second information carried by the second information, data that is not related to the location range is determined as the target data.

[0269] Optionally, when the scope of use information is the scope of use information where the first network device prohibits the use of the data analysis results, the second collection unit is specifically used for:

[0270] Based on the data analysis task information corresponding to the second information, obtain the complete set of data from the fourth network device for calculating the first performance information;

[0271] Based on the time range in the second information, select data related to the time range from the entire dataset to obtain the target data; and / or, based on the object range in the second information, select data related to the object range from the entire dataset to obtain the target data; and / or, based on the location range in the second information, select data related to the location range from the entire dataset to obtain the target data.

[0272] Optionally, the second sending module 801 is further configured to send third indication information to the first network device, the third indication information being used to indicate whether the second network device should perform the calculation of the first performance information of the first model.

[0273] Optionally, the second sending module 801 is further configured to send the second information to a third network device.

[0274] Reference Figure 9 This application also provides a model supervision processing device, such as... Figure 9 As shown, the model supervision processing device 900 includes:

[0275] The third receiving module 901 is used to receive second information from the first network device or the second network device, the second information including the scope of use of the data analysis results by the first network device;

[0276] The first calculation module 902 is used to calculate the first performance information of the first model based on the second information, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results.

[0277] Optionally, the scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model.

[0278] Optionally, the scope of use information includes at least one of the following:

[0279] The first network device allows the use of the range information of the data analysis results;

[0280] The first network device prohibits the use of the range information of the data analysis results.

[0281] Optionally, the range information includes at least one of time range, location range, and object range, and the objects included in the object range include at least one of user equipment objects, area objects, and network element objects.

[0282] Optionally, the second information further includes at least one of the following:

[0283] The first instruction information is used to instruct a request to calculate the first performance information of the first model.

[0284] The information used by the first network device to perform a first network operation based on the data analysis results.

[0285] Optionally, if the second information includes information about the first network operation, the second information may further include second indication information, which may indicate whether the first network operation will affect the calculation of the first performance information of the first model, or whether the second indication information will affect the calculation of the correctness of the data analysis results.

[0286] Optionally, the information of the first network operation includes:

[0287] The type information or description information of the first network operation.

[0288] Optionally, the second information may also include at least one of the following data analysis task information corresponding to the data analysis results:

[0289] Task identification information;

[0290] Task condition constraints;

[0291] Information about the objects being analyzed in the task.

[0292] Optionally, the first calculation module 902 includes:

[0293] The first collection unit is used to collect target data based on the second information;

[0294] The first calculation unit is used to calculate the first performance information of the first model based on the target data.

[0295] Optionally, the first collection unit is specifically configured to perform at least one of the following:

[0296] Based on the time range in the second information carried by the second information, data related to the time range is determined as the target data;

[0297] Based on the object range in the second information carried by the second information, determine the data related to the object range as the target data;

[0298] Based on the location range in the second information carried by the second information, data related to the location range is determined as the target data.

[0299] Optionally, when the scope of use information is the scope of use that the first network device is allowed to use the data analysis results, the first collection unit is specifically configured to perform at least one of the following:

[0300] Based on the time range in the second information carried by the second information, data that is not related to the time range is determined as the target data;

[0301] Based on the object range in the second information carried by the second information, data that is not related to the object range is determined as the target data;

[0302] Based on the location range in the second information carried by the second information, data that is not related to the location range is determined as the target data.

[0303] Optionally, when the scope of use information is the scope information in which the first network device prohibits the use of the data analysis results, the first collection unit is specifically used for:

[0304] Based on the data analysis task information corresponding to the second information, obtain the complete set of data from the fourth network device for calculating the first performance information;

[0305] Based on the time range in the second information, select data related to the time range from the entire dataset to obtain the target data; and / or, based on the object range in the second information, select data related to the object range from the entire dataset to obtain the target data; and / or, based on the location range in the second information, select data related to the location range from the entire dataset to obtain the target data.

[0306] The model supervision processing device in this application embodiment can be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices besides a terminal. For example, the terminal can include, but is not limited to, the type of terminal 10 listed above; other devices can be servers, network attached storage (NAS), etc., and this application embodiment does not specifically limit the type.

[0307] The model supervision processing device provided in this application embodiment can achieve... Figures 2 to 6 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0308] Optional, such as Figure 10 As shown, this application embodiment also provides a communication device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various steps of the above-described model supervision processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0309] This application embodiment also provides a network-side device, including a processor and a communication interface, wherein,

[0310] When the network-side device is a first network device, the communication interface is used to receive first information from a second network device, the first information including data analysis results; and to send second information to the second network device or a third network device, the second information including the scope of use of the data analysis results by the first network device.

[0311] When the network-side device is a second network device, the communication interface is used to send first information to the first network device, the first information including data analysis results; and to receive second information from the first network device, the second information including the scope of use of the data analysis results by the first network device.

[0312] When the network-side device is a third network device, the communication interface is used to receive second information from the first network device or the second network device. The second information includes the scope of use of the data analysis results by the first network device. The processor is used to calculate the first performance information of the first model based on the second information. The first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results.

[0313] This network-side device embodiment corresponds to the above-described network-side device method embodiment. All implementation processes and methods of the above-described method embodiment can be applied to this network-side device embodiment and can achieve the same technical effect.

[0314] Specifically, embodiments of this application also provide a network-side device. For example... Figure 11 As shown, the network-side device 1100 includes: an antenna 1101, a radio frequency (RF) device 1102, a baseband device 1103, a processor 1104, and a memory 1105. The antenna 1101 is connected to the RF device 1102. In the uplink direction, the RF device 1102 receives information through the antenna 1101 and transmits the received information to the baseband device 1103 for processing. In the downlink direction, the baseband device 1103 processes the information to be transmitted and sends it to the RF device 1102. The RF device 1102 processes the received information and transmits it through the antenna 1101.

[0315] The method executed by the network-side device in the above embodiments can be implemented in the baseband device 1103, which includes a baseband processor.

[0316] The baseband device 1103 may include, for example, at least one baseband board on which multiple chips are disposed, such as... Figure 11 As shown, one of the chips is, for example, a baseband processor, which is connected to the memory 1105 via a bus interface to call the program in the memory 1105 and execute the network device operation shown in the above method embodiment.

[0317] The network-side device may also include a network interface 1106, such as a common public radio interface (CPRI).

[0318] Specifically, the network-side device 1100 of this embodiment further includes: instructions or programs stored in memory 1105 and executable on processor 1104, wherein processor 1104 calls the instructions or programs in memory 1105 to execute. Figures 7 to 9 The methods executed by each module shown achieve the same technical effect, and to avoid repetition, they will not be described in detail here.

[0319] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described model supervision processing method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0320] The processor is the processor in the terminal described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0321] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described model supervision processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0322] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0323] This application also provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described model supervision processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0324] This application also provides a communication system, including: a first network-side device, a second network-side device, and a third network-side device, wherein the first network-side device is used to perform, for example... Figure 2 And the various processes of the above-described first network-side device method embodiments, the second network-side device is used to perform such... Figure 5 And the various processes of the above-described second network-side device method embodiments, the third network-side device is used to perform such... Figure 6 The processes of each method embodiment of the aforementioned third network-side device can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0325] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0326] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0327] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model supervision processing method, characterized in that, include: The first network device receives first information from the second network device, the first information including data analysis results; The first network device sends second information to the second network device or the third network device, the second information including the scope of use of the data analysis results by the first network device; The scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results. The scope of use information includes at least one of the following: The first network device allows the use of the range information of the data analysis results; The first network device prohibits the use of the range information of the data analysis results.

2. The method according to claim 1, characterized in that, The range information includes at least one of time range, location range, and object range, and the objects included in the object range include at least one of user equipment objects, area objects, and network element objects.

3. The method according to claim 1 or 2, characterized in that, The second information also includes at least one of the following: The first instruction information is used to instruct a request to calculate the first performance information of the first model. The information used by the first network device to perform a first network operation based on the data analysis results.

4. The method according to claim 3, characterized in that, When the second information includes information about the first network operation, the second information also includes second indication information, which is used to indicate whether the first network operation will affect the calculation of the first performance information of the first model, or whether the first network operation will affect the calculation of the correctness of the data analysis results.

5. The method according to claim 3, characterized in that, The information for the first network operation includes: The type information or description information of the first network operation.

6. The method according to claim 1 or 2, characterized in that, The second information also includes at least one of the following data analysis task information corresponding to the data analysis results: Task identification information; Task condition constraints; Information about the objects being analyzed in the task.

7. The method according to claim 1 or 2, characterized in that, The first information also includes at least one of the following pieces of information corresponding to the data analysis results: Confidence information, which is used to indicate the degree of trust that the second network device has in the data analysis results; Second performance information of the first model, which is used to represent the performance of the first model during training.

8. A model supervision processing method, characterized in that, include: The second network device sends first information to the first network device, the first information including data analysis results; The second network device receives second information from the first network device, the second information including the scope of use of the data analysis results by the first network device; The scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results. The scope of use information includes at least one of the following: The first network device allows the use of the range information of the data analysis results; The first network device prohibits the use of the range information of the data analysis results.

9. The method according to claim 8, characterized in that, The range information includes at least one of time range, location range, and object range, and the objects included in the object range include at least one of user equipment objects, area objects, and network element objects.

10. The method according to claim 8 or 9, characterized in that, The second information also includes at least one of the following: The first instruction information is used to instruct a request to calculate the first performance information of the first model. The information used by the first network device to perform a first network operation based on the data analysis results.

11. The method according to claim 10, characterized in that, When the second information includes information about the first network operation, the second information also includes second indication information, which is used to indicate whether the first network operation will affect the calculation of the first performance information of the first model, or whether the first network operation will affect the calculation of the correctness of the data analysis results.

12. The method according to claim 10, characterized in that, The information for the first network operation includes: The type information or description information of the first network operation.

13. The method according to claim 8 or 9, characterized in that, The second information also includes at least one of the following data analysis task information corresponding to the data analysis results: Task identification information; Task condition constraints; Information about the objects being analyzed in the task.

14. The method according to claim 8 or 9, characterized in that, The first information also includes at least one of the following pieces of information corresponding to the data analysis results: Confidence information, which is used to indicate the degree of trust that the second network device has in the data analysis results; Second performance information of the first model, which is used to represent the performance of the first model during training.

15. The method according to claim 8 or 9, characterized in that, The method further includes: The second network device calculates the first performance information of the first model based on the second information.

16. The method according to claim 15, characterized in that, The second network device calculates the first performance information of the first model based on the second information, including: The second network device collects target data based on the second information; The second network device calculates the first performance information of the first model based on the target data.

17. The method according to claim 16, characterized in that, When the scope of use information is scope information that the first network device prohibits the use of the data analysis results, the second network device collects target data based on the second information, including at least one of the following: The second network device determines data related to the time range as the target data based on the time range in the second information carried by the second information; The second network device determines data related to the object range as the target data based on the object range in the second information carried by the second information; The second network device determines data related to the location range as the target data based on the location range in the second information carried by the second information.

18. The method according to claim 16, characterized in that, When the scope of use information is the scope of use permitted by the first network device for the data analysis results, the second network device collects target data based on the second information, including at least one of the following: The second network device determines data that is not related to the time range as the target data based on the time range in the second information carried by the second information; The second network device determines data that is not related to the object range as the target data based on the object range in the second information carried by the second information; The second network device determines data that is not related to the location range as the target data based on the location range in the second information carried by the second information.

19. The method according to claim 16, characterized in that, When the scope of use information is the scope information that the first network device prohibits the use of the data analysis results, the second network device collects target data according to the second information, including: The second network device obtains the complete set of data for calculating the first performance information from the fourth network device based on the data analysis task information corresponding to the second information; The second network device selects data related to the time range from the complete set of data based on the time range in the second information to obtain the target data; and / or, the second network device selects data related to the object range from the complete set of data based on the object range in the second information to obtain the target data; and / or, the second network device selects data related to the location range from the complete set of data based on the location range in the second information to obtain the target data.

20. The method according to claim 15, characterized in that, After the second network device receives the second information from the first network device, the method further includes: The second network device sends a third indication message to the first network device, the third indication message being used to indicate whether the second network device should perform the calculation of the first performance information of the first model.

21. The method according to claim 8 or 9, characterized in that, The method further includes: The second network device sends the second information to the third network device.

22. A model supervision processing method, characterized in that, include: The third network device receives second information from the first network device or the second network device, the second information including the scope of use of the data analysis results by the first network device; The third network device calculates the first performance information of the first model based on the second information, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results. The scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model; The scope of use information includes at least one of the following: The first network device allows the use of the range information of the data analysis results; The first network device prohibits the use of the range information of the data analysis results.

23. The method according to claim 22, characterized in that, The range information includes at least one of time range, location range, and object range, and the objects included in the object range include at least one of user equipment objects, area objects, and network element objects.

24. The method according to claim 22 or 23, characterized in that, The second information also includes at least one of the following: The first instruction information is used to instruct a request to calculate the first performance information of the first model. The information used by the first network device to perform a first network operation based on the data analysis results.

25. The method according to claim 24, characterized in that, When the second information includes information about the first network operation, the second information also includes second indication information, which is used to indicate whether the first network operation will affect the calculation of the first performance information of the first model, or whether the first network operation will affect the calculation of the correctness of the data analysis results.

26. The method according to claim 24, characterized in that, The information for the first network operation includes: The type information or description information of the first network operation.

27. The method according to claim 22 or 23, characterized in that, The second information also includes at least one of the following data analysis task information corresponding to the data analysis results: Task identification information; Task condition constraints; Information about the objects being analyzed in the task.

28. The method according to claim 22 or 23, characterized in that, The calculation of the first performance information of the first model by the third network device based on the second information includes: The third network device collects target data based on the second information; The third network device calculates the first performance information of the first model based on the target data.

29. The method according to claim 28, characterized in that, The third network device collects target data based on the second information, including at least one of the following: The third network device determines data related to the time range as the target data based on the time range in the second information carried by the second information. The third network device determines data related to the object range as the target data based on the object range in the second information carried by the second information; The third network device determines data related to the location range as the target data based on the location range in the second information carried by the second information.

30. The method according to claim 28, characterized in that, When the scope of use information is the scope of use permitted by the first network device for the data analysis results, the third network device collects target data based on the second information, including at least one of the following: The third network device determines data that is not related to the time range as the target data based on the time range in the second information carried by the second information; The third network device determines data that is not related to the object range as the target data based on the object range in the second information carried by the second information; The third network device determines data that is not related to the location range as the target data based on the location range in the second information carried by the second information.

31. The method according to claim 28, characterized in that, When the scope of use information is the scope information in which the first network device prohibits the use of the data analysis results, the third network device collects target data according to the second information, including: The third network device obtains the complete set of data for calculating the first performance information from the fourth network device based on the data analysis task information corresponding to the second information; The third network device selects data related to the time range from the complete set of data according to the time range in the second information to obtain the target data; and / or, the third network device selects data related to the object range from the complete set of data according to the object range in the second information to obtain the target data; and / or, the third network device selects data related to the location range from the complete set of data according to the location range in the second information to obtain the target data.

32. A model supervision processing device, characterized in that, include: The first receiving module is used to receive first information from the second network device, the first information including data analysis results; The first sending module is used to send second information to the second network device or the third network device, the second information including the scope of use of the data analysis results by the first network device; The scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results. The scope of use information includes at least one of the following: The first network device allows the use of the range information of the data analysis results; The first network device prohibits the use of the range information of the data analysis results.

33. A model supervision processing device, characterized in that, include: The second sending module is used to send first information to the first network device, the first information including data analysis results; The second receiving module is used to receive second information from the first network device, the second information including the scope of use of the data analysis results by the first network device; The scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model, wherein the first model is a model obtained by the third network device through a model training process, and the first model is used by the second network device to generate the data analysis results. The scope of use information includes at least one of the following: The first network device allows the use of the range information of the data analysis results; The first network device prohibits the use of the range information of the data analysis results.

34. A model supervision processing device, characterized in that, include: The third receiving module is used to receive second information from the first network device or the second network device, the second information including the scope of use of the data analysis results by the first network device; The first calculation module is used to calculate the first performance information of the first model based on the second information, wherein the first model is a model obtained by the third network device through the model training process, and the first model is used by the second network device to generate the data analysis results. The scope of use information is used to assist the second network device or the third network device in calculating the first performance information of the first model; The scope of use information includes at least one of the following: The first network device allows the use of the range information of the data analysis results; The first network device prohibits the use of the range information of the data analysis results.

35. A network-side device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the model supervision processing method as described in any one of claims 1 to 31.

36. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the model supervision processing method as described in any one of claims 1 to 31.

Images