Method for determining accuracy of model and network side device

Abstract

The present application discloses a method, apparatus, and network side equipment for determining the accuracy of a model, which belongs to the field of mobile communications. The model accuracy determination method of an embodiment of the present application includes: a first network element infers a task based on a first model; the first network element determines a first accuracy corresponding to the first model, for indicating the accuracy degree of the inference result of the first model for the task; and when the first accuracy reaches a predetermined condition, the first network element sends first information to a second network element, where the first information is used to indicate that the accuracy of the first model does not meet the accuracy demand or is declining, wherein the second network element is a network element providing the first model.

Description

[Technical Field]

[0001] The present application relates to the field of mobile communication technology, and specifically to a method, apparatus and network side equipment for determining the accuracy of a model. [Background technology]

[0002] In communication networks, some network elements are introduced to perform intelligent data analysis, generating data analysis results (also called inference data results) for some tasks. These data analysis results can assist devices inside and outside the network in making policy decisions, with the aim of using artificial intelligence (AI) methods to improve the degree of intelligence of the device's policy decisions.

[0003] The Network Data Analytics Function (NWDAF) trains an AI or machine learning (ML) model based on the training data to obtain a corresponding model to be applied to an AI task. It infers the inference input data for a specific AI task based on the AI / ML model and obtains the inference result data corresponding to the specific AI task. The Policy Control Function (PCF) performs smart policy control and charging (PCC) based on the inference result data, such as formulating a smart user stay policy based on the inference result data of user service behavior to improve the user service experience. The Access and Mobility Management Function (AMF) performs smart mobility management operations based on the inference result data of a certain AI task, such as smartly paging a user based on the inference result data of the user's movement trajectory to improve the paging achievability rate.

[0004] The premise is that devices inside and outside the network can make correct and optimized policy decisions based on AI data analysis results only if they are based on accurate data analysis results. If the accuracy rate of data analysis results is relatively low and they are provided to devices inside and outside the network as error information for reference, they will ultimately make incorrect policy decisions or perform inappropriate operations. Therefore, it is necessary to guarantee the accuracy of data analysis results.

[0005] Although the accuracy in training (AiT) of the model meets the accuracy requirements of the model, it is not possible to determine the accuracy requirements of the inference accuracy (AiU) when the model is actually used. Gaps may exist due to differences in data distribution and insufficient model generalization ability. The accuracy of the inference result data obtained by this model is relatively low, and when it is provided to devices inside and outside the network for reference, it is likely to lead to incorrect policy decisions or inappropriate operations. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide a method, an apparatus, and a network-side device for determining the accuracy of a model, which can solve the problem that the accuracy of inference result data obtained by the model is relatively low. [Means for solving the problem]

[0007] A first aspect provides a method for determining accuracy of a model used in a first network element, the method comprising: a first network element inferring a task based on the first model; The first network element determines a first accuracy corresponding to the first model to indicate a degree of accuracy of an inference result of the first model for the task; When the first accuracy reaches a predetermined condition, the first network element sends first information to a second network element, the first information being used to indicate that the accuracy of the first model does not meet an accuracy demand or is declining; Here, the second network element is a network element that provides the first model.

[0008] A second aspect provides an apparatus for determining model accuracy, the apparatus comprising: an execution module for inferring a task based on the first model; a calculation module for determining a first accuracy corresponding to the first model, the first accuracy indicating a degree of accuracy of an inference result of the first model for the task; a transmission module for transmitting, when the first accuracy reaches a predetermined condition, first information to a second network element to indicate that the accuracy of the first model does not meet an accuracy demand or is declining; Here, the second network element is a network element that provides the first model.

[0009] A third aspect provides a method for determining accuracy of a model used in a second network element, the method comprising: receiving, by a second network element, first information from the first network element, the first information being used to indicate that the accuracy of the first model does not meet the accuracy demand or is deteriorating; The second network element retrains the first model based on the first information.

[0010] A fourth aspect provides an apparatus for determining model accuracy, the apparatus comprising: a transceiver module for receiving first information from the first network element indicating that the accuracy of the first model does not meet the accuracy demand or is deteriorating; and a training module for retraining the first model based on the first information.

[0011] A fifth aspect provides a method for determining accuracy of a model used in a fourth network element, the method comprising: receiving, by a fourth network element, third information from the first network element to instruct the fourth network element to store first data of a task, the task being that the first network element performs inference based on a first model; Here, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0012] A sixth aspect provides an apparatus for determining model accuracy, the apparatus comprising: a communication module for receiving third information from the first network element to instruct the first network element to store first data for the task, the task being for the first network element to perform inference based on a first model; a storage module for storing first data of the task; Here, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0013] A seventh aspect provides a network side device, the network side device including a processor and a memory, the memory storing a program or instructions operable to run on the processor, the program or instructions, when executed by the processor, implementing the steps of the method according to the first aspect, or implementing the steps of the method according to the third aspect, or implementing the steps of the method according to the fifth aspect.

[0014] An eighth aspect provides a model accuracy determination system, the system including a network side device, the network side device including a first network element, a second network element, and a fourth network element, the first network element may be used to perform steps of the model accuracy determination method described in the first aspect, the second network element may be used to perform steps of the model accuracy determination method described in the third aspect, and the fourth network element may be used to perform steps of the model accuracy determination method described in the fifth aspect.

[0015] A ninth aspect provides a readable storage medium, the readable storage medium storing a program or instructions that, when executed by a processor, performs the steps of the method of the first aspect, or the steps of the method of the third aspect, or the steps of the method of the fifth aspect.

[0016] A tenth aspect provides a chip, the chip including a processor and a communication interface, the communication interface coupled to the processor, the processor running a program or instruction to perform steps of the method of the first aspect, or to perform steps of the method of the third aspect, or to perform steps of the method of the fifth aspect.

[0017] An eleventh aspect provides a computer program / program product, the computer program / program product being stored on a storage medium, the computer program / program product being executed by at least one processor to implement the steps of the method according to the first aspect, or to implement the steps of the method according to the third aspect, or to implement the steps of the method according to the fifth aspect. [Effects of the Invention]

[0018] In an embodiment of the present application, a first network element infers a task based on a first model, and determines a first accuracy corresponding to the first model. When the first accuracy reaches a predetermined condition, the first network element sends first information to a second network element to indicate that the accuracy of the first model does not meet the accuracy requirements or has deteriorated, thereby monitoring the accuracy of the model in the actual application process, and taking appropriate measures in a timely manner when the accuracy deteriorates to prevent erroneous policy decisions or inappropriate operations from being performed. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a structural schematic diagram of a wireless communication system to which the embodiments of the present application can be applied; [Figure 2] 1 is a flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Figure 3] 1 is another flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Figure 4] 1 is another flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Figure 5] 1 is another flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Figure 6] 1 is a structural schematic diagram of a model accuracy determination device according to an embodiment of the present application; [Figure 7]1 is another flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Figure 8] FIG. 1 is another structural schematic diagram of a model accuracy determination device according to an embodiment of the present application; [Figure 9] 1 is another flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Figure 10] FIG. 1 is another structural schematic diagram of a model accuracy determination device according to an embodiment of the present application; [Figure 11] 1 is a structural schematic diagram of a communication device according to an embodiment of the present application; [Figure 12] FIG. 2 is a structural schematic diagram of a network-side device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0020] The following clearly describes the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application fall within the scope of protection of the present application.

[0021] The terms "first," "second," etc. in the specification and claims of this application are intended to distinguish between similar objects and are not intended to describe a particular order or sequence. It should be understood that terms used in this manner are interchangeable where appropriate, so that embodiments of this application may be performed in orders other than those illustrated or described herein, and that objects distinguished by "first" and "second" are generally of the same type and do not limit the number of objects; for example, a first object may be one or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the related objects.

[0022] It should be noted that the techniques described in the embodiments of the present application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be applied to other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-Carrier Frequency Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in the embodiments of the present application are always used interchangeably, and the described techniques may be used in the above-mentioned systems and radio technologies as well as other systems and radio technologies. The following description describes a 5th Generation Mobile Communication Technology (5G) system for illustrative purposes, and uses 5G terminology in most of the description below, however, these technologies may also be applied to applications other than 5G system applications, such as 6th Generation (6G) communication systems.

[0023] 1 is a block diagram of a wireless communication system to which the embodiment of the present application can be applied. The wireless communication system includes a terminal 11 and a network side device 12. Here, the terminal 11 may be a terminal-side device such as a mobile phone, a tablet personal computer, a laptop computer (also called a notebook computer), a personal digital assistant (PDA), a palmtop computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, a vehicle-mounted equipment (VUE), a pedestrian-mounted equipment (PUE), a smart home (home equipment with wireless communication capabilities, such as a refrigerator, television, washing machine, or furniture), a game console, a personal computer (PC), a teller machine, or a self-service machine, and wearable devices include a smart watch, a smart wristband, a smart earphone, a smart glasses, a smart accessory (a smart bracelet, a smart hand chain, a smart ring, a smart necklace, a smart ankle bracelet, a smart anklet, etc.), a smart band, a smart garment, etc. It should be noted that the terminal 11 in the embodiment of the present application is not limited to a specific type. The network side equipment 12 may include access network equipment or core network equipment, where the access network equipment 12 may be called radio access network equipment, radio access network (RAN), radio access network function or radio access network unit.The access network equipment 12 may include a base station, a Wireless Local Area Network (WLAN) access point, or a Wireless Fidelity (WiFi) node, etc., and the base station may be called a Node B, an Evolved Node B (eNB), an access point, a Base Transceiver Station (BTS), a radio base station, a radio transceiver, a Basic Service Set (BSS), an Extended Service Set (ESS), a home B node, a home evolved B node, a Transmitting Receiving Point (TRP), or any other suitable term in the art. As long as the same technical effect is achieved, the base station is not limited to a specific technical term. It should be noted that the embodiments of this application only take base stations in a 5G system as examples, and do not limit the specific type of base station.Core network devices include core network nodes, core network functions, mobility management entities (MMEs), access and mobility management functions (AMFs), session management functions (SMFs), user plane functions (UPFs), policy control functions (PCFs), policy and charging rules functions (PCRFs), edge application server discovery functions (EASDFs), unified data management (UDMs), unified data repository (UDRs), home subscriber servers (HSSs), centralized network configuration (CNCs), network repository functions (NRFs), network exposure functions (NEFs), local NEFs (or L-NEFs), binding support functions (BSFs), and application functions (Application Node Functions). It should be noted that the embodiments of the present application only use core network equipment in a 5G system as an example, and do not limit the specific type of core network equipment.

[0024] The following describes in detail the method, apparatus and network side device for determining model accuracy according to the embodiments of the present application through several examples and application scenarios in conjunction with the drawings.

[0025] As shown in Figure 2, an embodiment of the present application provides a method for determining the accuracy of a model, the execution body of this method includes a first network element, and the first network element includes a model inference function network element. In other words, this method can be performed by software or hardware installed in the first network element. The method includes the following steps:

[0026] S210, the first network element infers a task based on the first model.

[0027] In one embodiment, the first network element may be a network element that simultaneously has a model inference function and a model training function, for example, the first network element may be an NWDAF, and the NWDAF may include an Analytics Logical Function network element (AnLF) and a Model Training Logical Function (MTLF).

[0028] In another embodiment, the first network element includes a model inference function network element and the second network element includes a model training function network element, for example, the first network element is an AnLF and the second network element is a MTLF.

[0029] Assuming that the NWDAF is the first network element, the second network element and the first network element in the following embodiments may be the same network element, that is, the MTLF and AnLF are integrated into the NWDAF.

[0030] It should be understood that the first model may be constructed and trained according to actual needs, such as an AI / ML model. The MTLF collects training data and performs model training based on the training data. After training is completed, the MTLF sends information about the trained first model to the AnLF.

[0031] After determining the task to be triggered, the AnLF infers the task based on the first model and obtains inference result data.

[0032] It should be understood that the task is not a single task but a data analysis task for indicating one task type, and after triggering the task, the AnLF can determine a first model corresponding to the task based on the task's identifier information (Analytics ID), etc., and infer the task based on the corresponding first model to obtain inference result data. For example, if the task's Analytics ID=UE mobility is used to predict the movement trajectory of a terminal (also called User Equipment (UE)), the AnLF can infer the task based on the first model corresponding to UE mobility, and the obtained inference result data is predicted terminal location (UE location) information.

[0033] The AnLF may infer a task one or more times based on the first model to obtain a plurality of inference result data or inference result data including a plurality of output result values.

[0034] In one embodiment, the execution of the AnLF inferring a task may be triggered by a third network element sending a task request message, the third network element being the network element that triggers the task, and the third network element may include a consumer network element (consumer network function, consumer NF), which may be a network element of a 5G system, or may be a terminal or a third-party application function (Application Function, AF), etc.

[0035] In another embodiment, the task may be triggered autonomously by the AnLF, for example by setting up a validation test phase, in which the task is simulated and triggered autonomously by the AnLF to test the accuracy of the first model.

[0036] For convenience, in the following embodiments, the AnLF is the first network element, the MTLF is the second network element, and the consumer NF is the third network element.

[0037] S220, the first network element determines a first accuracy corresponding to the first model to indicate a degree of accuracy of an inference result of the first model for the task.

[0038] After completing the inference for the task, the AnLF calculates a first accuracy corresponding to the first model in the inference process, where the first accuracy is AiU. The manner of calculating the first accuracy may be various, and the embodiment of this application only shows one of the embodiments for implementing the invention. The first network element obtains inference result data corresponding to the task based on the first model; the first network element obtaining label data corresponding to the inference result data, the label data may be obtained from the label data source device, and in one embodiment, the label data is data ground truth, i.e., facts and data that actually occurred; The first network element may calculate a first accuracy of the first model based on the inference result data and the label data, specifically, by comparing the inference result data with corresponding label data, determine the number of correct results in the inference result data, and divide the number of correct results by the total number to obtain the first accuracy, which is expressed by the following formula: Primary accuracy = number of correct results / total number Here, the correct result may indicate that the inference result data and the label data match, or that the gap between the inference result data and the label result data is within an allowable range.

[0039] The expression format of the first accuracy may vary and is not limited to a specific percentage value, for example, 90%, but may also be a categorical expression format, for example, high, medium, low, etc., or normalized data, for example, 0.9.

[0040] The first accuracy in the embodiment of the present application may indicate the accuracy of the inference result of the first model for the task from the front or the opposite side, and in one embodiment, the first accuracy is: The accuracy of the inference result of the task (for example, the accuracy of the first model may be directly indicated by calculating the accuracy rate of the inference result of the first model for the task); The degree of error in the inference result of the task (for example, the accuracy of the first model may be indicated from the other side by calculating the error rate or inference error of the inference result for the task of the first model. Here, the method of calculating the inference error may be various, for example, mean absolute error (MAE), mean square error (MSE), etc.) may be used to indicate at least one of the following.

[0041] In one embodiment, the first network element obtaining label data corresponding to the inference result data includes: the first network element determining a source device of label data corresponding to the task; The first network element obtains the label data from the source device.

[0042] Here, the source device of the label data may be determined by AnLF based on type information of the output data of the first model, limiting condition information and target information of the task, etc.

[0043] It should be noted that the trigger conditions of step S220 may vary, may be pre-set by the AnLF, or may be obtained from the MTLF, for example, from a model performance subscription request (Performance Monitoring) sent to the AnLF.

[0044] S230: When the first accuracy reaches a predetermined condition, the first network element sends first information to a second network element to indicate that the accuracy of the first model does not meet the accuracy demand or is declining, where the second network element is a network element that provides the first model.

[0045] The AnLF determines whether the accuracy of the first model satisfies the accuracy demand or has decreased depending on whether the first accuracy has reached a predetermined condition, which may include whether the first accuracy is smaller than a predetermined threshold or has decreased to a certain extent.

[0046] If the AnLF determines that the accuracy of the first model does not meet the accuracy demand or has decreased, it sends first information to the MTLF that provides the first model to inform the MTLF that the accuracy of the first model does not meet the accuracy demand or has decreased.

[0047] In one embodiment, the first network element may send the first information to the second network element based on a trigger event, where the trigger event may include the first network element completing a first accuracy calculation or arriving at a specific time.

[0048] In another embodiment, the first network element may transmit the first information to the second network element based on a preset period.

[0049] It should be explained that when sending first information to the second network element based on a trigger event or based on a preset period, the first information includes: the accuracy of the first model; The accuracy of the first model meets the accuracy requirements; and The accuracy of the first model may be used to indicate at least one of: not meeting accuracy needs or decreasing.

[0050] After receiving the first information, the MTLF may determine a subsequent operation based on the first information, for example, the MTLF may retrain the first model or reselect another model that can be used to infer the task and send it to the AnLF. Here, when selecting the other model, the MTLF may require that the other model satisfy certain conditions, and the certain conditions may include at least one of: a second accuracy of the other model being higher than the second accuracy of the first model; a second accuracy of the other model being higher than the first accuracy of the first model; and a model performance requirement of the other model being higher than a model performance requirement of the first model.

[0051] Before step S230, the method includes: The method further includes the step of the first network element receiving a model performance subscription request from the second network element, the model performance subscription request being used to request the first network element to monitor the accuracy of the first model.

[0052] It should be noted that the first network element may receive the model performance subscription request from the second network element before step S210 or S220.

[0053] In response, the first network element may send a model performance subscription response message (Model Performance Monitoring Notification) to the second network element based on the model performance subscription request, and the model performance subscription response message may carry the first information.

[0054] As can be seen from the technical solutions of the above embodiments, the embodiments of the present application include: a first network element inferring a task based on a first model, determining a first accuracy corresponding to the first model, and when the first accuracy reaches a predetermined condition, the first network element sends first information to a second network element to indicate that the accuracy of the first model does not meet the accuracy requirements or has deteriorated, thereby monitoring the accuracy of the model in the actual application process, and taking relevant measures in a timely manner when the accuracy deteriorates to prevent erroneous policy decisions or inappropriate operations from being performed.

[0055] Based on the above embodiment, further, as shown in FIG. 3, the method for determining the accuracy of the model includes the following steps:

[0056] The MTLF may pre-train the first model, and the training process may include steps A1 to A2.

[0057] Step A1. The MTLF collects training data from training data source devices.

[0058] Step A2. MTLF trains a first model based on the training data.

[0059] Step A5. After completing the training of the first model, the MTLF may send information of the trained first model to the AnLF.

[0060] In one embodiment, the message specifically carrying the information of the first model may be a Nnwdaf_MLModelProvision_Notify or Nnwdaf_MLModelInfo_Response message.

[0061] In one embodiment, before step A5, the method further comprises the following steps:

[0062] Step A4: The AnLF sends a message of the requested model to the MTLF.

[0063] In one embodiment, during the training phase or post-training testing phase of the first model in step A2, the MTLF needs to evaluate the accuracy of the first model and calculate a second accuracy, i.e., AiT, of the first model. The second accuracy may be obtained using the same calculation formula as the first accuracy. Specifically, the MTLF may set a validation data set for evaluating the second accuracy of the first model, where the validation data set includes input data used in the first model and corresponding label data. The MTLF inputs the input data into the trained first model to obtain output data, compares whether the output data matches the label data, and calculates the second accuracy of the first model according to the above formula.

[0064] Accordingly, in one embodiment, when transmitting information about the first model to the AnLF in step A5, the MTLF may simultaneously transmit the second accuracy of the first model, or may transmit the second accuracy of the first model to the AnLF in an independent message.

[0065] In one embodiment, the requested model message may include requirements information for the first model of AnLF, such as model performance requirements, model description format or language requirements, manufacturer requirements, aging requirements, domain requirements, model size requirements, etc. Here, the model performance requirements may be a second accuracy requirement for the first model.

[0066] Accordingly, in one embodiment, when the MTLF transmits information about the first model to the AnLF in step A5, related information corresponding to the first model and demand information, such as the second accuracy, the model description format or language, model aging information, application domain information, model size information, etc., may also be transmitted at the same time.

[0067] In one embodiment, if the task is triggered by a third network element, before step A4, the method further comprises the following steps:

[0068] Step A3. The Consumer NF sends a task request message to the AnLF, and the task request message is used to request inference of the task, thereby triggering the AnLF to execute an inference process for the task based on a first model corresponding to the task.

[0069] The task request message includes description information of the task, which may be various and may include identifier information of the task, analytical filter information of the task, target information of the task, etc. The target and scope of the task may be determined based on the description information of the task.

[0070] The limiting condition information of the task is used to limit the execution range of the task, and may include a time range, an area range, and the like.

[0071] The target information of the task is used to specify an object that is a target of the task, such as a certain terminal identifier (UE ID), a certain terminal group identifier (UE group ID), or any terminal (any UE).

[0072] The AnLF may request a model from the MTLF according to the task request message, and obtain information of a first model and a second accuracy of the first model from the MTLF.

[0073] In one embodiment, steps A1-A2 may be located after step A4, i.e., after receiving a message of the requested model sent by the AnLF, the MTLF trains a first model corresponding to the task and sends a message of the trained first model to the AnLF.

[0074] In one embodiment, as shown in FIG. 3, step S210 includes steps A6 to A8.

[0075] Step A6. AnLF, based on the received task request message, a first model corresponding to the task; type information of the input data of the first model; type information of the output data of the first model; a source device of inference input data corresponding to the task; At least one association information with a source device of label data corresponding to the task is determined.

[0076] Here, the first model corresponding to the task may be determined by the task type indicated by the analytics ID in the task request message, or may be determined by the mapping relationship between the analytics ID and the first model, where the model identifier information (model ID) may represent the first model, for example, model 1.

[0077] The type information of the input data of the first model may be called metadata information of the model, for example, the input data may include a terminal identifier (UE ID), time, and a terminal current service status.

[0078] The type information of the output data of the first model includes a data type, for example, a Tracking Area (TA) or a cell for indicating a UE location.

[0079] Specifically, the AnLF may determine the target and scope of the task based on information such as the analytics filter information and analytics target in the task request message, and further determine a network element that can obtain the inference input data corresponding to the task as the source device of the inference input data corresponding to the task based on the target, scope and metadata information.

[0080] Specifically, the source device of the label data corresponding to the task may be determined by the AnLF determining a network element device type (NF type) capable of providing the output data based on the type information of the output data of the first model, and then determining a specific network element instance corresponding to this network element device type based on the constraint information and target information of the task, and setting this network element instance as the source device of the label data. For example, based on the data type = UE location of the output data of the first model corresponding to the task UE mobility, the AnLF determines that the network element device type AMF type can provide data of the UE location. Based on the task constraint information AOI etc. and the target UE1 of the task, the AnLF queries a unified data management entity (UDM) or a network repository function (NRF) to find that the corresponding AMF instance is AMF1. The AnLF then sets AMF1 as the source device of the label data, and obtains the label data of the UE location from AMF1.

[0081] Step A7. AnLF obtains inference input data corresponding to the task, specifically, AnLF may send a request message for inference input data according to the source device of the inference input data of the task determined in step A6 to collect inference input data corresponding to the task.

[0082] Step A8. AnLF infers inference input data corresponding to the task based on the acquired first model, and obtains inference result data.

[0083] For example, based on a first model corresponding to analytics ID=UE mobility, the AnLF infers numerical values ​​of inference input data corresponding to the task, such as UE ID, time, and the current service status of the UE, and obtains output data whose inference result data is UE location.

[0084] In one embodiment, after obtaining inference result data corresponding to the task based on the first model, the method further includes the following steps:

[0085] Step A9: The first network element sends the inference result data to the third network element, that is, sends the inference result data obtained through AnLF inference to the consumer NF.

[0086] According to the inference result data, the first model corresponding to the analytics ID may be used to inform the consumer NF of the statistics or prediction values ​​obtained by the inference, which may be used to assist the consumer NF in making a corresponding policy decision. For example, the statistics or prediction values ​​corresponding to the UE mobility may be used to assist the AMF in performing user paging optimization.

[0087] In one embodiment, the message specifically carrying the inference result data may be a Nnwdaf_AnalyticsSubscription_Notify or Nnwdaf_AnalyticsInfo_Response message.

[0088] Step A10: AnLF obtains label data corresponding to the inference result data.

[0089] In one embodiment, the message specifically carrying the label data may be an Nnf_EventExposure_Subscribe message.

[0090] Specifically, the AnLF may be used to determine which label data to feed back to the source device of the label data determined in step A6 by sending a label data request message to the source device of the label data, the message including type information of the label data, target information corresponding to the label data, time information (e.g., timestamp, time zone), etc.

[0091] The type information of the label data, the target information corresponding to the label data, the time information, etc. in the label data request message may be determined by the AnLF based on the type information of the output data of the first model, the target information of the task, the limiting condition information of the task, etc. Specifically, the AnLF determines the type information of the label data that needs to be acquired based on the type information of the output data of the first model, the target information of the label data that needs to be acquired based on the target information of the task, and if the AnLF determines based on the limiting condition information of the task that the inference process of the task is a statistical calculation performed for a certain time in the past or a prediction performed for a certain time in the future, the AnLF needs to further acquire label data corresponding to the certain time in the past or the certain time in the future.

[0092] For example, the AnLF sends a label data request message to the AMF or Location Management Function (LMF), where the data type corresponding to the label data carrier is UE location, the target information is UE 1, and the time information is a specific time period, and is used to request the AMF / LMF to feed back the UE location data of UE 1 within a specific time period.

[0093] It should be understood that in step A8, if the AnLF obtains multiple inference result data by executing one or multiple inference processes, the AnLF needs to obtain multiple label data corresponding to the multiple inference result data accordingly.

[0094] In one embodiment, as shown in FIG. 3, step S220 includes steps A11 and A12.

[0095] Step A11. AnLF calculates a first accuracy of the first model based on the inference result data and label data.

[0096] Step A12.AnLF determines whether the first accuracy satisfies a preset condition, and determines whether the first accuracy satisfies a preset condition. Pre-set conditions If the above condition is satisfied, step A13 is executed.

[0097] The predetermined conditions may be set according to actual needs, and in one embodiment, the predetermined conditions are: the first accuracy is less than a first threshold; and the first accuracy is lower than the second accuracy; and The first accuracy is lower than the second accuracy, and the difference between the first accuracy and the second accuracy is greater than a second threshold.

[0098] The method of obtaining the first threshold may be various, and the first threshold may be set by the AnLF, or may be set by the MTLF and transmitted to the AnLF as a judgment condition or trigger condition.

[0099] In one embodiment, the MTLF sets it to the Model Performance Requirement requested when the AnLF requests the first model.

[0100] In another embodiment, when the second network element transmits information of the first model to the first network element, the information may be transmitted together with the information of the first model.

[0101] In another embodiment, the model performance subscription may be carried by a model performance subscription message sent by the second network element to the first network element.

[0102] In one embodiment, as shown in FIG. 3, step S230 includes step A13.

[0103] Step A13: The AnLF sends first information to the MTLF to notify the MTLF that the accuracy of the first model does not meet the accuracy demand or is declining.

[0104] In one embodiment, the first information may be specifically sent by a Nnwdaf_AnalyticsSubscription_Notify message.

[0105] In another embodiment, the AnLF may transmit the first information to the MTLF via a Model Performance Monitoring Notification.

[0106] In another embodiment, the first information may be used to request the MTLF to retrain the first model or to re-request a model, and in this case, the first information may be specifically sent by a Nnwdaf_MLModelProvision_Subscribe or Nnwdaf_MLModelInfo_Request message.

[0107] In one embodiment, the first information comprises: Identifier information of the first model (e.g., Model ID); Identifier information for the task (e.g., Analytics ID); The scope of the first information, including a time range, a region range, and a target range, i.e., the task limiting condition information for indicating the target and scope of the task when the accuracy of the first model does not meet the accuracy demand or is reduced; an indication that the accuracy of the first model does not meet the accuracy requirements or is reduced; the first accuracy; request instruction information to retrain the first model for instructing a MTLF to retrain the first model; model re-request instruction information for requesting acquisition of a model corresponding to the task; and first data for the task for retraining the first model.

[0108] In one embodiment, the first data is inference input data corresponding to the task; output result data corresponding to the task; and label data corresponding to the task.

[0109] In step A14, the MTLF may enter a retraining process for the first model based on the first information. The specific training process is basically the same as that in step A2, with the specific difference being that the training data may include the first data of the task. The retraining process for the first model may involve re-training the initialized first model based on the training data, or training the current first model based on the training data to achieve fine-tuning of the first model, which can achieve faster convergence and save resources.

[0110] In one embodiment, after the MTLF completes retraining of the first model, the method further comprises the steps of:

[0111] Step A15. The first network element receives second information including information of the first model after retraining from the second network element, i.e., the MTLF sends the information of the first model after retraining to the AnLF, allowing the AnLF to recover the inference of the task.

[0112] In one embodiment, the second information comprises: Application condition information of the first model after the retraining (the application condition information may include a time range, a region range, a target range, etc. that are objects of the first model); and a third accuracy of the retrained first model, i.e., AiT of the retrained first model, for indicating the degree of accuracy of the model output results presented by the retrained first model in the training phase or the testing phase.

[0113] In another embodiment, after step A14, the method further comprises: The method further includes a step in which the MTLF transmits information of the retrained first model to a sixth network element that is a network element that needs to perform inference using the first model, where the sixth network element includes a model inference function network element.

[0114] This means that after the MTLF has completed retraining the first model, it may send the retrained first model to other AnLFs that require it, so that the other AnLFs can use the first model.

[0115] As can be seen from the technical solutions described in the above embodiments, in the embodiments of the present application, when the first accuracy reaches a predetermined condition, the first network element sends first information to the second network element indicating that the accuracy of the first model does not meet the accuracy demand or has declined, and causes the second network element to retrain the first model. Thus, when the model accuracy declines, the first model can be retrained in a timely manner, and the accuracy of task inference can be quickly restored, preventing erroneous policy decisions or inappropriate operations from being performed.

[0116] Based on the above embodiment, further, as shown in FIG. 4, the method further includes the following steps:

[0117] Step B14. The first network element sends third information to the fourth network element to instruct the fourth network element to store the first data of the task, and the fourth network element is a network element that receives and stores the first data from the first network element.

[0118] The AnLF may store the first data in a fourth network element, which may include a storage network element and may be an Analytics Data Repository Function (ADRF).

[0119] In one embodiment, the third information is: Identifier information of the task; Limiting condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; label data corresponding to the task; and memory cause information (e.g., the accuracy of the first model does not meet the accuracy demand or is declining).

[0120] Accordingly, the first information sent from the AnLF to the MTLF in step A13 may further include information of a fourth network element, for example, identifier information of the ADRF.

[0121] Step B15. The MTLF acquires the first data from the ADRF based on the first information. Specifically, the MTLF may send data request information for the first data to the ADRF to indicate the data range to be acquired, where the data request information includes: task identifier information; Limiting condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0122] In one embodiment, the data request information may further include a cause for the request, for example, that the first model needs to be retrained, or that the accuracy of the first model does not meet or has deteriorated to meet accuracy demands.

[0123] In one embodiment, the fourth network element may be used to store information of the first model, and after step A14, the method further includes the following steps:

[0124] Step B16: The second network element stores the information of the first model after the retraining in the fourth network element.

[0125] The fourth network element may store application condition information of the first model after retraining and a third accuracy of the first model after retraining.

[0126] The AnLF may obtain information of the first model after retraining from the fourth network element according to actual needs.

[0127] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application have the fourth network element store the first data of the task, so that when the accuracy of the model is reduced, the fourth network element can store the relevant data corresponding to the task in a timely manner for retraining the first model, thereby allowing the first model to be updated in a timely manner, quickly recover the accuracy of inferring the task, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0128] Based on the above embodiment, as shown in FIG. 5, step A14 may further include: obtaining second data for the first model, the second network element being training data for retraining the first model based on the first information; and the second network element retraining the first model based on the second data.

[0129] In one embodiment, the step of the second network element obtaining second data of the first model based on the first information comprises the following steps:

[0130] Step C14. The second network element determines a second data source device of the first model, that is, an inference data source device shown in FIG. 3; Step C15. The second network element obtains the second data from the source device.

[0131] In one embodiment, the step C15 includes a step of the second network element transmitting data request information to the source device to request the source device to provide the second data; Here, the data request information is task identifier information; Limiting condition information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0132] In one embodiment, the second data source device comprises: task identifier information; Limiting condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and the output data type information of the first model.

[0133] In one embodiment, the second data may include first data of a task, ie, inference input data, inference result data, and label data corresponding to the task.

[0134] As can be seen from the technical solutions described in the above embodiments, in the embodiments of the present application, the second network element independently obtains second data of the first model from a second data source device based on the first information to retrain the first model, so that the first model can be updated in a timely manner, quickly restore the accuracy of task inference, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0135] Based on the above embodiment, when the first accuracy reaches a preset condition, the method further comprises: The method further includes the first network element requesting the fifth network element to obtain a second model, which is a model used for the task provided by the fifth network element. For a specific process, see steps A4 to A5. Here, the fifth network element includes a model training function network element, that is, the fifth network element may be another MTLF other than the second network element.

[0136] The first network element infers the task based on the second model to obtain new inference result data for the task, where the inferred task may be a task triggered according to the task request message sent by the consumer NF in step A3, or may be a task triggered by the consumer NF resending the task request message.

[0137] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application obtain a second model from a fifth network element when the first accuracy reaches a preset condition, infer tasks, and obtain new inference result data. When the accuracy of the model is declining, relevant adjustment measures can be taken in a timely manner to quickly restore the accuracy of task inference, and prevent erroneous policy decisions or inappropriate operations from being made.

[0138] Based on the above embodiment, when the first accuracy reaches a preset condition, the method further comprises: The method further includes the first network element sending fourth information to a third network element to indicate that the accuracy of the first model does not meet an accuracy demand or is deteriorating.

[0139] In one embodiment, the fourth information is: All or part of the description information of the task for instructing the task to be inferred using the first model (specifically, it may include an Analytics ID, Analytics filter information, analytics target, etc.); an indication that the accuracy of the first model does not meet the accuracy requirements or is reduced; the first accuracy; Recommended operation information for recommending to the consumer NF an operation to be performed after receiving the first information; and latency information for indicating the time required for the first network element to recover inference for the task (specifically, this may be the latency required for the AnLF to obtain a first model after retraining, perform inference, and obtain inference result data, or the latency required for the AnLF to obtain a second model that can be used to perform the task, perform inference, and obtain inference result data).

[0140] In one embodiment, the recommended operation information includes: Instructing the consumer NF to continue using the inference result data corresponding to the task, i.e., to continue using the already acquired inference result data; Instructing the consumer NF to stop using the inference result data corresponding to the task, i.e., to stop using the inference result data already obtained; and instructing the consumer NF to re-trigger the task to obtain new inference result data, i.e., to re-send the task request message.

[0141] After receiving the first information, the consumer NF may perform step A14, and perform a corresponding operation based on the first information.

[0142] Specifically, the consumer NF may perform at least one of the following operations based on the first information:

[0143] Continue using the inference result data corresponding to the task, which may be performed if the decrease in the first accuracy is relatively small and does not exceed a predetermined width threshold, for example a second threshold, and in one embodiment, continuing to use the inference result data corresponding to the task may appropriately reduce the weight of the inference result data in making policy decisions.

[0144] Stopping use of the inference result data corresponding to the task, which may be performed when the extent of the decrease in the first accuracy is relatively large and has already exceeded a preset extent threshold; retransmitting the task request message to the first network element to re-request the first network element to infer the task; retransmitting the task request message to the seventh network element to request the seventh network element to infer the task, wherein the seventh network element includes a model inference function network element, i.e., the consumer NF may send the task request message to another AnLF other than the first network element.

[0145] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application monitor the accuracy of the model in the actual application process by sending fourth information to a third network element when the first accuracy reaches a predetermined condition, indicating that the accuracy of the first model does not meet the accuracy requirements or has declined, and causing the third network element to perform corresponding operations, thereby preventing erroneous policy decisions or the execution of inappropriate operations.

[0146] In the model accuracy determination method according to the embodiment of the present application, the execution body may be a model accuracy determination device. In the embodiment of the present application, the model accuracy determination device according to the embodiment of the present application is taken as an example to execute the model accuracy determination method.

[0147] As shown in FIG. 6, the model accuracy determination device includes an execution module 601, a calculation module 602, and a transmission module 603.

[0148] The execution module 601 is used to infer a task based on a first model, the calculation module 602 is used to determine a first accuracy corresponding to the first model to indicate the accuracy degree of the inference result of the first model for the task, and the transmission module 603 is used to send first information to a second network element to indicate that the accuracy of the first model does not meet the accuracy demand or has decreased when the first accuracy reaches a predetermined condition, wherein the second network element is a network element that provides the first model.

[0149] Furthermore, the model accuracy determination device includes a model inference function network element.

[0150] Furthermore, the second network element includes a model training function network element.

[0151] Furthermore, the transmission module 603 is further used to receive a model performance subscription request from the second network element, and the model performance subscription request is used to request monitoring of the accuracy of the first model.

[0152] Furthermore, the transmission module 603 is used to obtain inference result data corresponding to the task based on the first model, and obtain label data corresponding to the inference result data.

[0153] The calculation module 602 is used to calculate a first accuracy of the first model based on the inference result data and the label data.

[0154] Furthermore, the first accuracy is The accuracy of the inference result of the task; and the degree of error in the inference result of the task.

[0155] The implementation of the present application can realize the method embodiment shown in FIG. 2 and obtain the same technical effects, and the repeated parts will not be further described here.

[0156] As can be seen from the technical solutions of the above embodiments, the embodiments of the present application infer a task based on a first model, determine a first accuracy corresponding to the first model, and when the first accuracy reaches a predetermined condition, send first information to a second network element to indicate that the accuracy of the first model does not meet the accuracy requirements or has deteriorated, thereby monitoring the accuracy of the model in the actual application process, and taking relevant measures in a timely manner when the accuracy deteriorates, to prevent erroneous policy decisions or inappropriate operations from being made.

[0157] Based on the above embodiment, further, after sending the first information to the second network element, the transmission module is further used to receive second information from the second network element, and the second information includes information of the first model after retraining.

[0158] Furthermore, the first information is Identifier information for the first model; Identifier information of the task; Limiting condition information of the task; an indication that the accuracy of the first model does not meet the accuracy requirements or is reduced; the first accuracy; a request instruction to retrain the first model; Model re-request instruction information for requesting acquisition of a model corresponding to the task (this model may be the first model after retraining, or may be another model that can be used to infer the task); and first data for the task for retraining the first model; and information about a fourth network element that is a network element that receives and stores the first data from the first network element.

[0159] Furthermore, the first data is inference input data corresponding to the task; output result data corresponding to the task; and label data corresponding to the task.

[0160] Furthermore, the second information is application condition information of the first model after the retraining; and a third accuracy of the retrained first model for indicating the degree of accuracy of model output results provided by the retrained first model in a training phase or a testing phase.

[0161] Furthermore, the transmission module determining a source device of label data corresponding to the task; It is used to obtain the label data from the source device.

[0162] Furthermore, after obtaining inference result data corresponding to the task based on the first model, the transmission module further ,before It is used to send the inference result data to a third network element, which is the network element that triggers the task.

[0163] Furthermore, the preset conditions are: the first accuracy is less than a first threshold; and the first accuracy is lower than the second accuracy; and the first accuracy is lower than the second accuracy, and a difference value between the first accuracy and the second accuracy is greater than a second threshold value; Here, the second accuracy is used to indicate the degree of accuracy of the model output results presented by the first model in the training stage or the testing stage.

[0164] Furthermore, the third network element includes a consumer network element.

[0165] The implementation of the present application can realize the method embodiment shown in FIG. 3 and obtain the same technical effects, and the repeated parts will not be further described here.

[0166] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application send first information to a second network element when the first accuracy reaches a predetermined condition, indicating that the accuracy of the first model does not meet the accuracy demand or has declined, and have the second network element retrain the first model, so that when the model accuracy declines, the first model can be retrained in a timely manner, the accuracy of task inference can be quickly restored, and erroneous policy decisions or inappropriate operations can be prevented.

[0167] Based on the above embodiment, further, the transmission module is used for the first network element to send third information to the fourth network element, for instructing the fourth network element to store first data of the task.

[0168] Furthermore, the third information is Identifier information of the task; Limiting condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; label data corresponding to the task; and storage cause information.

[0169] Furthermore, the fourth network element includes a storage network element.

[0170] The implementation of the present application can realize the method embodiment shown in FIG. 4 and obtain the same technical effects, and the repeated parts will not be further described here.

[0171] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application have the fourth network element store the first data of the task, so that when the accuracy of the model is reduced, the fourth network element can store the relevant data corresponding to the task in a timely manner for retraining the first model, thereby allowing the first model to be updated in a timely manner, quickly recover the accuracy of inferring the task, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0172] Based on the above embodiment, further, when the first accuracy reaches a preset condition, the transmitting module is further adapted to request a fifth network element to obtain a second model, which is a model used for the task provided by the fifth network element; The execution module is used to infer the task based on the second model.

[0173] Furthermore, the fifth network element includes a model training function network element.

[0174] The implementation of the present application can realize the above method embodiments and obtain the same technical effects, and the repeated parts will not be further described here.

[0175] As can be seen from the technical solutions described in the above embodiments, in the embodiments of the present application, the second network element independently obtains second data of the first model from a second data source device based on the first information to retrain the first model, so that the first model can be updated in a timely manner, quickly restore the accuracy of task inference, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0176] Based on the above embodiment, when the first accuracy reaches a preset condition, To, The method further includes the first network element sending fourth information to a third network element to indicate that the accuracy of the first model does not meet an accuracy demand or is deteriorating.

[0177] Furthermore, the fourth information is All or part of the description information of the task; an indication that the accuracy of the first model does not meet the accuracy requirements or is reduced; the first accuracy; Recommended operation information and and latency information for indicating the time required for the first network element to recover the inference of the task.

[0178] The implementation of the present application can realize the above method embodiments and obtain the same technical effects, and the repeated parts will not be further described here.

[0179] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application monitor the accuracy of the model in the actual application process by sending fourth information to a third network element when the first accuracy reaches a predetermined condition, indicating that the accuracy of the first model does not meet the accuracy requirements or has declined, and causing the third network element to perform corresponding operations, thereby preventing erroneous policy decisions or the execution of inappropriate operations.

[0180] The model accuracy determination device in the embodiments of the present application may be an electronic device, such as an electronic device having an operating system, or a component of an electronic device, such as an integrated circuit or chip. The electronic device may be a terminal or other device other than a terminal. Exemplarily, the terminal may include, but is not limited to, the types of terminals 11 listed above. The other device may be a server, a network-attached storage (NAS), etc., and the embodiments of the present application are not specifically limited thereto.

[0181] The model accuracy determination device according to the embodiment of the present application can implement each process implemented by the method embodiments of Figures 2 to 5 and achieve the same technical effects, and will not be further described here to avoid repetition.

[0182] As shown in Figure 7, an embodiment of the present application further provides another model accuracy determination method, where the execution body of this method is a second network element, where the second network element includes a model training function network element, in other words, this method may be executed by software or hardware installed in the second network element, and the method includes the following steps:

[0183] S710, the second network element receives first information from the first network element to indicate that the accuracy of the first model does not meet the accuracy demand or is deteriorating; Further, before step S710, the method may further include: The method further includes the second network element sending a model performance subscription request to the first network element to request the first network element to monitor accuracy of the first model.

[0184] S720, the second network element retrains the first model based on the first information.

[0185] Furthermore, the first network element includes a model inference function network element.

[0186] Furthermore, the second network element includes a model training function network element.

[0187] Furthermore, the second network element transmits second information to the first network element, the second information including information of the first model after retraining.

[0188] Furthermore, the first information is Identifier information for the first model; task identifier information, which is a task for the first network element to perform inference based on a first model; Limiting condition information of the task; an indication that the accuracy of the first model does not meet the accuracy requirements or is reduced; the first accuracy indicating the accuracy of an inference result of the first model for the task; a request instruction to retrain the first model; model re-request instruction information for requesting acquisition of a model corresponding to the task; first data for the task for retraining the first model; and information about a fourth network element that is a network element that receives and stores the first data from the first network element.

[0189] Furthermore, step S720 the second network element obtaining second data of the first model based on the first information; and the second network element retraining the first model based on the second data.

[0190] Furthermore, the second information is application condition information of the first model after the retraining; and a third accuracy of the retrained first model for indicating the degree of accuracy of model output results provided by the retrained first model in a training phase or a testing phase.

[0191] Further, after step S720, the method further comprises: The method further includes a step in which the second network element transmits information of the retrained first model to a sixth network element, which is a network element that needs to perform inference using the first model.

[0192] Furthermore, the sixth network element includes a model inference function network element.

[0193] Furthermore, the first accuracy is The accuracy of the inference result of the task; and the degree of error in the inference result of the task.

[0194] The steps S710 to S720 implement the method embodiment shown in FIG. 2 and FIG. 3, and can achieve the same technical effects, and the repeated parts will not be further described here.

[0195] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application receive first information from a first network element to indicate that the accuracy of a first model does not meet the accuracy demand or is declining, and retrain the first model based on the first information, so that when the accuracy of the model declines, the first model can be retrained in a timely manner, the accuracy of task inference can be quickly restored, and erroneous policy decisions or inappropriate operations can be prevented from being made.

[0196] Based on the above embodiment, the second network element obtaining second data of the first model based on the first information includes: determining a source device of second data of the first model by the second network element; The second network element acquires the second data from the source device.

[0197] Furthermore, the second network element obtaining the second data from the source device includes: the second network element transmitting, to the source device, data request information for requesting the source device to provide the second data; Here, the data request information is task identifier information, which is a task for the first network element to perform inference based on a first model; Limiting condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0198] Furthermore, the second data source device task identifier information, which is a task for the first network element to perform inference based on a first model; Limiting condition information of the task; Identifier information for the first model; input data type information of the first model; and the output data type information of the first model.

[0199] The embodiments of the present application can realize the method embodiments shown in FIG. 5 and obtain the same technical effects, and the repeated parts will not be further described here.

[0200] As can be seen from the technical solutions described in the above embodiments, in the embodiments of the present application, the second network element independently obtains second data of the first model from a second data source device based on the first information to retrain the first model, so that the first model can be updated in a timely manner, quickly restore the accuracy of task inference, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0201] Based on the above embodiment, further, the second data includes first data of a task, the task being a task in which the first network element performs inference based on a first model.

[0202] Furthermore, the first data source device includes the fourth network element.

[0203] Furthermore, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0204] Further, after retraining the first model based on the first information, the method further comprises: The second network element may further include storing information of the retrained first model in the fourth network element.

[0205] Furthermore, the fourth network element includes a storage network element.

[0206] The embodiments of the present application can realize the method embodiments shown in FIG. 4 and obtain the same technical effects, and the repeated parts will not be further described here.

[0207] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application have the fourth network element store the first data of the task, so that when the accuracy of the model is reduced, the fourth network element can store the relevant data corresponding to the task in a timely manner for retraining the first model, thereby allowing the first model to be updated in a timely manner, quickly recover the accuracy of inferring the task, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0208] In the model accuracy determination method according to the embodiment of the present application, the execution body may be a model accuracy determination device. In the embodiment of the present application, the model accuracy determination device according to the embodiment of the present application is taken as an example to execute the model accuracy determination method.

[0209] As shown in FIG. 8, the model accuracy determination device includes a transceiver module 801 and a training module 802 .

[0210] The transceiver module 801 is used to receive first information from a first network element indicating that the accuracy of a first model does not meet the accuracy demand or is declining, and the training module 802 is used to retrain the first model based on the first information.

[0211] Furthermore, the transceiver module 801 is further used for sending a model performance subscription request to the first network element to request the first network element to monitor the accuracy of the first model.

[0212] Furthermore, the first network element includes a model inference function network element.

[0213] Furthermore, the second network element includes a model training function network element.

[0214] Moreover, the transceiver module 801 is further used for sending second information, including information of the first model after retraining, to the first network element.

[0215] Furthermore, the first information is Identifier information for the first model; task identifier information, which is a task for the first network element to perform inference based on a first model; Limiting condition information of the task; an indication that the accuracy of the first model does not meet the accuracy requirements or is reduced; the first accuracy indicating the accuracy of an inference result of the first model for the task; a request instruction to retrain the first model; model re-request instruction information for requesting acquisition of a model corresponding to the task; first data for the task for retraining the first model; and information about a fourth network element that is a network element that receives and stores the first data from the first network element.

[0216] Furthermore, the transceiver module 801 is used to obtain second data of the first model based on the first information; The training module 802 is used to retrain the first model based on the second data.

[0217] Furthermore, the second information is application condition information of the first model after the retraining; and a third accuracy of the retrained first model for indicating the degree of accuracy of model output results provided by the retrained first model in a training phase or a testing phase.

[0218] Furthermore, the transceiver module 801 is further used to transmit information of the retrained first model to a sixth network element, which is a network element that needs to be inferred using the first model.

[0219] Furthermore, the sixth network element includes a model inference function network element.

[0220] Furthermore, the first accuracy is The accuracy of the inference result of the task; and the degree of error in the inference result of the task.

[0221] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application receive first information from a first network element to indicate that the accuracy of a first model does not meet the accuracy demand or is declining, and retrain the first model based on the first information, so that when the accuracy of the model declines, the first model can be retrained in a timely manner, the accuracy of task inference can be quickly restored, and erroneous policy decisions or inappropriate operations can be prevented from being made.

[0222] Based on the above embodiment, further, the transceiver module: determining a source device of second data for the first model; It is used to acquire the second data from the source device.

[0223] Furthermore, the transceiver module is used to transmit data request information to the source device to request the source device to provide the second data; Here, the data request information is task identifier information, which is a task for the first network element to perform inference based on a first model; Limiting condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0224] Furthermore, the second data source device task identifier information, which is a task for the first network element to perform inference based on a first model; Limiting condition information of the task; Identifier information for the first model; input data type information of the first model; and the output data type information of the first model.

[0225] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application independently obtain second data of the first model from the second data source device based on the first information to retrain the first model, so that the first model can be updated in a timely manner, quickly restore the accuracy of task inference, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0226] Based on the above embodiment, further, the second data includes first data of a task, the task being a task in which the first network element performs inference based on a first model.

[0227] Furthermore, the source device of the first data includes the fourth network element, and the fourth network element is a network element that receives and stores the first data from the first network element.

[0228] Furthermore, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0229] Furthermore, after retraining the first model based on the first information, the transceiver module is further used for storing the information of the retrained first model in the fourth network element.

[0230] Furthermore, the fourth network element includes a storage network element.

[0231] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application have the fourth network element store the first data of the task, so that when the accuracy of the model is reduced, the fourth network element can store the relevant data corresponding to the task in a timely manner for retraining the first model, thereby allowing the first model to be updated in a timely manner, quickly recover the accuracy of inferring the task, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0232] The model accuracy determination device in the embodiments of the present application may be an electronic device, such as an electronic device having an operating system, or a component of an electronic device, such as an integrated circuit or chip. The electronic device may be a terminal or other device other than a terminal. Exemplarily, the terminal may include, but is not limited to, the types of terminals 11 listed above. The other device may be a server, a network-attached storage (NAS), etc., and the embodiments of the present application are not specifically limited thereto.

[0233] The model accuracy determination device according to the embodiment of the present application can implement each process implemented by the method embodiment of Figure 7 and achieve the same technical effect, and will not be further described here to avoid repetition.

[0234] As shown in Figure 9, an embodiment of the present application further provides a method for determining the accuracy of another model, where the execution body of this method is a fourth network element, where the fourth network element includes a storage network element, in other words, this method may be performed by software or hardware installed in the fourth network element, and the method includes the following steps:

[0235] S910, a fourth network element receives third information from the first network element to instruct the fourth network element to store first data of a task, the task being that the first network element performs inference based on a first model; Here, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0236] Furthermore, the third information is Identifier information of the task; Limiting condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; label data corresponding to the task; and storage cause information.

[0237] Further, after step S910, the method further comprises: receiving, by the fourth network element, data request information from a second network element that is a network element providing the first model; The fourth network element further includes transmitting first data of the task to the second network element.

[0238] Furthermore, the data request information task identifier information, which is a task for the first network element to perform inference based on a first model; Limiting condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0239] Furthermore, after transmitting the first data of the task to the second network element, the method includes: The method further includes the fourth network element receiving information of the retrained first model from the second network element.

[0240] Further, after receiving information of the retrained first model from the second network element, the method includes: The method further includes the fourth network element sending information of the retrained first model to the first network element.

[0241] Furthermore, the first network element includes a model inference function network element.

[0242] Furthermore, the second network element includes a model training function network element.

[0243] Furthermore, the fourth network element includes a storage network element.

[0244] The above step S910 realizes the embodiment of the method shown in FIG. 4, and can achieve the same technical effect, and the repeated parts will not be further described here.

[0245] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application have the fourth network element store the first data of the task, so that when the accuracy of the model is reduced, the fourth network element can store the relevant data corresponding to the task in a timely manner for retraining the first model, thereby allowing the first model to be updated in a timely manner, quickly recover the accuracy of inferring the task, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0246] In the model accuracy determination method according to the embodiment of the present application, the execution body may be a model accuracy determination device. In the embodiment of the present application, the model accuracy determination device according to the embodiment of the present application is taken as an example to execute the model accuracy determination method.

[0247] As shown in FIG. 10, the model accuracy determination device includes a communication module 1001 and a storage module 1002.

[0248] The communication module 1001 is used to receive third information from a first network element to instruct the first network element to store first data of a task, the first data being a task in which the first network element performs inference based on a first model; and the storage module 1002 is used to store the first data of the task; Here, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0249] Furthermore, the third information is Identifier information of the task; Limiting condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; label data corresponding to the task; and storage cause information.

[0250] Furthermore, the communication module 1001 further receives data request information from a second network element, which is a network element that provides the first model; The fourth network element is used to transmit first data of the task to the second network element.

[0251] Furthermore, the data request information task identifier information, which is a task for the first network element to perform inference based on a first model; Limiting condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0252] Moreover, the communication module 1001 is further used for receiving information of the retrained first model from the second network element.

[0253] Moreover, the communication module 1001 is further used for sending information of the retrained first model to the first network element.

[0254] Furthermore, the first network element includes a model inference function network element.

[0255] Furthermore, the second network element includes a model training function network element.

[0256] Additionally, the model accuracy determination device includes a storage network element.

[0257] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application store the first data of the task, so that when the accuracy of the model is reduced, the relevant data corresponding to the task can be stored in a timely manner for retraining the first model, so that the first model can be updated in a timely manner, quickly restore the accuracy of inferring the task, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0258] The model accuracy determination device in the embodiments of the present application may be an electronic device, such as an electronic device having an operating system, or a component of an electronic device, such as an integrated circuit or chip. The electronic device may be a terminal or other device other than a terminal. Exemplarily, the terminal may include, but is not limited to, the types of terminals 11 listed above. The other device may be a server, a network-attached storage (NAS), etc., and the embodiments of the present application are not specifically limited thereto.

[0259] The model accuracy determination device according to the embodiment of the present application can implement each process implemented by the method embodiment of Figure 9 and achieve the same technical effect, and will not be further described here to avoid repetition.

[0260] Optionally, as shown in Fig. 11, an embodiment of the present application further provides a communication device 1100, which includes a processor 1101 and a memory 1102, and the memory 1102 stores a program or instruction that can run on the processor 1101. For example, if the communication device 1100 is a terminal, when the program or instruction is executed by the processor 1101, it can realize each step of the embodiment of the method for determining the accuracy of the model and achieve the same technical effect. If the communication device 1100 is a network-side device, when the program or instruction is executed by the processor 1101, it can realize each step of the embodiment of the method for determining the accuracy of the model and achieve the same technical effect, and in order to avoid repetition, it will not be described further here.

[0261] Specifically, an embodiment of the present application further provides a network side device. As shown in Fig. 12, the network side device 1200 includes a processor 1201, a network interface 1202, and a memory 1203. Here, the network interface 1202 is, for example, a common public radio interface (CPRI).

[0262] Specifically, the network side device 1200 of the embodiment of the present invention further includes instructions or programs stored in the memory 1203 and operable on the processor 1201, and the processor 1201 calls the instructions or programs in the memory 1203 to execute the methods performed by the modules shown in Figures 6, 8 and 10, and achieves the same technical effects, which will not be described further here to avoid repetition.

[0263] The embodiments of the present application further provide a readable storage medium, which stores a program or instruction, and when the program or instruction is executed by a processor, it can realize each process of the embodiment of the method for determining the accuracy of the model and achieve the same technical effect, and in order to avoid repetition, it will not be further described here.

[0264] Wherein, the processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0265] An embodiment of the present application further provides a chip, the chip including a processor and a communication interface, the communication interface coupled to the processor, the processor running a program or instruction to realize each process of the embodiment of the method for determining the accuracy of the above model, and can achieve the same technical effect, and in order to avoid repetition of description, no further description will be given here.

[0266] It should be understood that the chips referred to in the embodiments of this application may be referred to as system level chips, system chips, chip systems, or system-on-chips.

[0267] The embodiments of the present application further provide a computer program / program product, which is stored in a storage medium and can be executed by at least one processor to realize each process of the embodiments of the model accuracy determination method and achieve the same technical effects, and will not be further described here to avoid repetition.

[0268] An embodiment of the present application further provides a model accuracy determination system, the system including a network side device, the network side device including a first network element, a second network element, and a fourth network element, the first network element may be used to perform steps of the model accuracy determination method described above, the second network element may be used to perform steps of the model accuracy determination method described above, and the fourth network element may be used to perform steps of the model accuracy determination method described above.

[0269] It should be noted that, in this specification, the terms "comprise," "include," "includes," or any other variations thereof are intended to cover the non-exclusive "comprise," whereby a process, method, article, or apparatus comprising a set of elements not only includes those elements, but also other elements not expressly listed or inherent in such process, method, article, or apparatus. Absent further limitations, an element defined by the phrase "comprises one of" does not preclude the presence of other identical elements in the process, method, article, or apparatus comprising that element. It should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may include performing functions in an essentially simultaneous manner or in the reverse order based on the functions involved. For example, the described method may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to some examples may be combined in other examples.

[0270] As will be apparent to those skilled in the art from the above description of the embodiments, the methods of the above embodiments can be realized in the form of software and a necessary general-purpose hardware platform. Of course, they can also be realized in hardware, but in many cases, the former is a more preferred embodiment. Based on this understanding, the technical proposal of the present application, in substance or in part contributing to the prior art, may be embodied in the form of a computer software product, which is stored in a storage medium (e.g., a read-only memory (ROM) / random-access memory (RAM), a magnetic disk, or an optical disk) and includes some instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, a network device, etc.) to execute the methods described in the embodiments of the present application.

[0271] Although the embodiments of the present application have been described above in conjunction with the drawings, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not limiting. Those skilled in the art can take the teachings of the present application into account and implement many forms without departing from the spirit and scope of the claims, all of which fall within the scope of protection of the present application.

[0272] CROSS-REFERENCE TO RELATED APPLICATIONS This invention claims priority to a Chinese patent application submitted to the Patent Office of the People's Republic of China on March 7, 2022, bearing application number 202210224481.9 and entitled "Method, apparatus and network-side equipment for determining the accuracy of a model," and to a Chinese patent application submitted to the Patent Office of the People's Republic of China on August 9, 2022, bearing application number 202210958867.2 and entitled "Method, apparatus and network-side equipment for determining the accuracy of a model," the entire contents of which are incorporated herein by reference.

Claims

1. 1. A method for determining model accuracy, the method being performed by a first network element, comprising: the first network element inferring a task based on a first model; The first network element determines a first accuracy corresponding to the first model to indicate a degree of accuracy of an inference result of the first model for the task; When the first accuracy reaches a predetermined condition, the first network element sends first information to a second network element, the first information being used to indicate that the accuracy of the first model does not meet an accuracy demand or is declining; wherein the second network element is a network element that provides the first model; The first network element determining a first accuracy corresponding to the first model includes: The first network element obtains inference result data corresponding to the task based on the first model; and The first network element obtains label data corresponding to the inference result data; the first network element calculating a first accuracy of the first model based on the inference result data and the label data; The first network element obtaining label data corresponding to the inference result data includes: the first network element determining a source device of label data corresponding to the task; and the first network element obtaining the label data from the source device.

2. Before the first network element transmits the first information to the second network element, the method for determining the accuracy of the model includes:

2. The method of claim 1, further comprising: the first network element receiving a model performance subscription request from the second network element, the model performance subscription request being used to request the first network element to monitor the accuracy of the first model.

3. After transmitting the first information to the second network element, the method for determining the accuracy of the model includes:

2. The method of claim 1, further comprising the first network element receiving second information from the second network element, the second information including information of the first model after retraining.

4. The first accuracy is The accuracy of the inference result of the task; and the degree of error in the inference result of the task.

5. The first information is Identifier information for the first model; Identifier information of the task; Limiting condition information of the task; an indication that the accuracy of the first model does not meet the accuracy requirements or is reduced; the first accuracy; a request instruction to retrain the first model; model re-request instruction information for requesting acquisition of a model corresponding to the task; first data for the task for retraining the first model; and information of a fourth network element that receives and stores the first data from the first network element.

6. The method for determining the accuracy of the model comprises: The first network element further includes sending third information to the fourth network element to instruct the fourth network element to store first data of the task; The third information is Identifier information of the task; Limiting condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; label data corresponding to the task; 2. The method of claim 1, further comprising at least one of:

7. The first data is inference input data corresponding to the task; output result data corresponding to the task; and label data corresponding to the task.

8. The second information is application condition information of the first model after the retraining; and a third accuracy of the first model after retraining, for indicating the degree of accuracy of model output results presented by the first model after retraining in a training phase or a testing phase.

9. After obtaining inference result data corresponding to the task based on the first model, the method for determining the accuracy of the model includes: The first network element further includes transmitting the inference result data to a third network element, the third network element being a network element that triggers the task; the third network element comprises a consumer network element; The method of claim 1 .

10. The predetermined conditions are: the first accuracy is less than a first threshold; and the first accuracy is lower than the second accuracy; and the first accuracy is lower than the second accuracy, and a difference value between the first accuracy and the second accuracy is greater than a second threshold value; 2. The method for determining the accuracy of a model according to claim 1, wherein the second accuracy is used to indicate the degree of accuracy of the model output results presented by the first model in a training phase or a testing phase.

11. When the first accuracy reaches a predetermined condition, the method for determining accuracy of the model includes: The first network element requests a fifth network element to obtain a second model, the second model being a model used for the task provided by the fifth network element; the first network element inferring the task based on the second model; The fifth network element includes a model training function network element. The method of claim 1 .

12. When the first accuracy reaches a predetermined condition, the method for determining accuracy of the model includes: The first network element may further include transmitting fourth information to a third network element to indicate that the accuracy of the first model does not meet an accuracy demand or is deteriorating; The fourth information is All or part of the description information of the task; an indication that the accuracy of the first model does not meet the accuracy requirements or is reduced; the first accuracy; Recommended operation information and and latency information for indicating the time required for the first network element to recover the inference of the task.

13. the first network element includes a model inference function network element; Or, The method of claim 1 , wherein the second network element comprises a model training function network element.

14. 14. A network side device comprising a processor and a memory, the memory storing a program or instructions operable on the processor, the program or instructions, when executed by the processor, realizing the method for determining the accuracy of a model according to any one of claims 1 to 13.

Images