Method for determining accuracy of model and network side device

Abstract

The present application discloses a method, apparatus, and network side equipment for determining the accuracy of a model, which belongs to the field of mobile communications. The model accuracy determination method of an embodiment of the present application includes: a first network element infers a task based on a first model; the first network element determines a first accuracy corresponding to the first model, for indicating the accuracy degree of the inference result of the first model for the task; and when the first accuracy reaches a predetermined condition, the first network element sends first information to a second network element, where the first information is used to indicate that the accuracy of the first model does not meet the accuracy demand or is declining, wherein the second network element is a network element providing the first model.

Description

[Technical field]

[0001] The present application relates to the field of mobile communication technology, and in particular to a method, apparatus and network side equipment for determining the accuracy of a model. [Background technology]

[0002] In communication networks, some network elements have been introduced to perform smart data analysis and generate data analysis results (also called inference data results) for some tasks, which can assist devices inside and outside the network to make policy decisions, with the aim of using artificial intelligence (AI) methods to enhance the degree of smartness of the devices' policy decisions.

[0003] The Network Data Analytics Function (NWDAF) performs AI or Machine Learning (ML) model training based on the training data to obtain the corresponding model to be applied to the AI ​​task. Based on the AI / ML model, it infers the inference input data of a certain AI task and obtains the inference result data corresponding to a specific AI task. The Policy Control Function (PCF) performs smart Policy Control and Charging (PCC) based on the inference result data, such as formulating a smart user stay policy based on the inference result data of the user service behavior to improve the user's service experience; or the Access and Mobility Management Function (AMF) performs smart mobility management operations based on the inference result data of a certain AI task, such as smartly paging the user based on the inference result data of the user's movement trajectory to improve the paging achievability rate.

[0004] The premise is that devices inside and outside the network must be based on correct data analysis results to make correct and optimized policy decisions based on AI data analysis results. If the accuracy rate of the data analysis results is relatively low and they are provided as error information for reference to devices inside and outside the network, they will ultimately make incorrect policy decisions or perform inappropriate operations. Therefore, it is necessary to guarantee the accuracy of the data analysis results.

[0005] Although the accuracy in training (AiT) of the model meets the accuracy demands of the model, it is not possible to determine the accuracy demands that the inference accuracy (AiU) of the model can achieve when it is actually used. There may be gaps due to differences in data distribution and insufficient model generalization ability, and the accuracy of the inference result data obtained by the model is relatively low. When the data is provided to devices inside and outside the network for reference, it is likely to lead to erroneous policy decisions or inappropriate operations. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide a method, an apparatus, and a network side device for determining the accuracy of a model, which can solve the problem that the accuracy of inference result data obtained by a model is relatively low. [Means for solving the problem]

[0007] A first aspect provides a method of determining accuracy of a model used in a first network element, the method comprising: A first network element infers a task based on a first model; determining a first accuracy corresponding to the first model, the first network element indicating a degree of accuracy of an inference result of the first model for the task; When the first accuracy reaches a preset condition, the first network element sends first information to a second network element, the first information being used to indicate that the accuracy of the first model does not meet an accuracy demand or is deteriorating; Here, the second network element is a network element that provides the first model.

[0008] A second aspect provides an apparatus for determining accuracy of a model, the apparatus comprising: an execution module for inferring a task based on the first model; a computation module for determining a first accuracy corresponding to the first model, the first accuracy indicating a degree of accuracy of an inference result of the first model for the task; a transmission module for transmitting first information to a second network element to indicate that the accuracy of the first model does not meet an accuracy demand or is degraded when the first accuracy reaches a preset condition; Here, the second network element is a network element that provides the first model.

[0009] A third aspect provides a method of determining accuracy of a model used in a second network element, the method comprising: receiving, by a second network element, first information from the first network element, the first information being used to indicate that an accuracy of the first model does not meet an accuracy need or is deteriorating; The second network element retrains the first model based on the first information.

[0010] A fourth aspect provides an apparatus for determining accuracy of a model, the apparatus comprising: a transceiver module for receiving first information from a first network element to indicate that an accuracy of the first model does not meet an accuracy need or is deteriorating; and a training module for retraining the first model based on the first information.

[0011] A fifth aspect provides a method of determining accuracy of a model used in a fourth network element, the method comprising: receiving, by a fourth network element, third information from the first network element to instruct the fourth network element to store first data for a task, the task being that the first network element performs inference based on a first model; Here, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0012] A sixth aspect provides an apparatus for determining accuracy of a model, the apparatus comprising: a communication module for receiving third information from the first network element for instructing the first network element to store first data for the task, the task being for the first network element to perform inference based on a first model; a storage module for storing first data of the task; Here, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0013] A seventh aspect provides a network side device, the network side device including a processor and a memory, the memory storing a program or instructions operable to run on the processor, the program or instructions, when executed by the processor, performing steps of the method according to the first aspect, or performing steps of the method according to the third aspect, or performing steps of the method according to the fifth aspect.

[0014] An eighth aspect provides a model accuracy determination system, the system including a network side device, the network side device including a first network element, a second network element, and a fourth network element, the first network element may be used to perform steps of the model accuracy determination method described in the first aspect, the second network element may be used to perform steps of the model accuracy determination method described in the third aspect, and the fourth network element may be used to perform steps of the model accuracy determination method described in the fifth aspect.

[0015] A ninth aspect provides a readable storage medium, the readable storage medium storing a program or instructions which, when executed by a processor, perform the steps of the method of the first aspect, or perform the steps of the method of the third aspect, or perform the steps of the method of the fifth aspect.

[0016] A tenth aspect provides a chip, the chip comprising a processor and a communication interface, the communication interface coupled to the processor, the processor running a program or instructions for performing steps of the method of the first aspect, or for performing steps of the method of the third aspect, or for performing steps of the method of the fifth aspect.

[0017] An eleventh aspect provides a computer program / program product, the computer program / program product being stored on a storage medium, the computer program / program product being executed by at least one processor to perform the steps of the method of the first aspect, or to perform the steps of the method of the third aspect, or to perform the steps of the method of the fifth aspect. Effect of the Invention

[0018] In an embodiment of the present application, a first network element infers a task based on a first model, and determines a first accuracy corresponding to the first model. When the first accuracy reaches a preset condition, the first network element sends first information to a second network element, indicating that the accuracy of the first model does not meet the accuracy requirement or is declining, thereby monitoring the accuracy of the model in the actual application process, and taking corresponding measures in a timely manner when the accuracy is declining, to prevent erroneous policy decisions or inappropriate operations from being executed. [Brief description of the drawings]

[0019] [Figure 1] 1 is a structural schematic diagram of a wireless communication system to which the embodiments of the present application can be applied; [Diagram 2] 1 is a flow chart of a method for determining the accuracy of a model according to an embodiment of the present application. [Diagram 3] 4 is another flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Figure 4] 4 is another flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Diagram 5] 4 is another flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Figure 6] FIG. 2 is a structural schematic diagram of a model accuracy determination device according to an embodiment of the present application; [Figure 7]4 is another flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Figure 8] FIG. 2 is another structural schematic diagram of a model accuracy determining apparatus according to an embodiment of the present application; [Figure 9] 4 is another flowchart of a method for determining the accuracy of a model according to an embodiment of the present application. [Figure 10] FIG. 2 is another structural schematic diagram of a model accuracy determining apparatus according to an embodiment of the present application; [Figure 11] 1 is a structural schematic diagram of a communication device according to an embodiment of the present application; [Figure 12] FIG. 2 is a structural schematic diagram of a network side device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The following clearly describes the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application are all within the scope of protection of the present application.

[0021] The terms "first," "second," etc. in the specification and claims of the present application are intended to distinguish between similar objects and are not intended to describe a particular order or sequence. It is to be understood that the terms so used are interchangeable where appropriate, such that the embodiments of the present application may be performed in an order other than that shown or described herein, and that the objects distinguished by "first" and "second" are generally of the same type and do not limit the number of objects, e.g., the first object may be one or more. It is to be noted that "and / or" in the specification and claims represents at least one of the objects connected, and the character " / " generally represents an "or" relationship between the related objects.

[0022] It should be noted that the techniques described in the embodiments of the present application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be applied to other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in the embodiments of the present application are always used interchangeably, and the techniques described may be used in the above-mentioned systems and radio technologies, or in other systems and radio technologies. The following description describes a 5th Generation Mobile Communication Technology (5G) system for illustrative purposes, and uses 5G terminology in most of the description below, however, these technologies may also be applied to applications other than 5G system applications, such as 6th Generation (6G) communication systems.

[0023] 1 is a block diagram of a wireless communication system to which the embodiment of the present application can be applied. The wireless communication system includes a terminal 11 and a network side device 12. Here, the terminal 11 may be a terminal device such as a mobile phone, a tablet personal computer, a laptop computer (also called a notebook computer), a personal digital assistant (PDA), a palmtop computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, a vehicle-mounted equipment (VUE), a pedestrian terminal (PUE), a smart home (home equipment having a wireless communication function, such as a refrigerator, a television, a washing machine, or furniture), a game console, a personal computer (PC), a deposit machine, or a self-service machine, and the wearable device includes a smart watch, a smart wristband, a smart earphone, a smart glasses, a smart accessory (a smart bracelet, a smart hand chain, a smart ring, a smart necklace, a smart ankle bracelet, a smart anklet, etc.), a smart band, a smart clothing, etc. It should be noted that the terminal 11 in the embodiment of the present application is not limited to a specific type. The network side equipment 12 may include an access network equipment or a core network equipment, where the access network equipment 12 may be called a radio access network equipment, a radio access network (Radio Access Network, RAN), a radio access network function or a radio access network unit.The access network equipment 12 may include a base station, a Wireless Local Area Network (WLAN) access point, or a Wireless Fidelity (WiFi) node, etc., and the base station may be called a Node B, an Evolved Node B (eNB), an access point, a Base Transceiver Station (BTS), a radio base station, a radio transceiver, a Basic Service Set (BSS), an Extended Service Set (ESS), a home B node, a home evolved B node, a Transmitting Receiving Point (TRP), or any other suitable term in the art. As long as the same technical effect is achieved, the base station is not limited to a specific technical term, and it should be explained that in the embodiments of the present application, only base stations in a 5G system are introduced as examples, and the specific type of the base station is not limited.The core network devices are: Core Network Node, Core Network Function, Mobility Management Entity (MME), Access and Mobility Management Function (AMF), Session Management Function (SMF), User Plane Function (UPF), Policy Control Function (PCF), Policy and Charging Rules Function (PCRF), Edge Application Server Discovery Function (EASDF), Unified Data Management (UDM), Unified Data Repository (UDR), Home Subscriber Server (HSS), Centralized network configuration (CNC), Network Repository Function (NRF), Network Exposure Function (NEF), Local NEF (Local NEF or L-NEF), Binding Support Function (BSF), Application Function (Application Node Function (ADF)), and Application Node Function (ADN). Function, AF), etc., but are not limited thereto. It should be noted that in the embodiments of the present application, only the core network equipment in the 5G system is introduced as an example, and the specific type of the core network equipment is not limited.

[0024] The following describes in detail the method, apparatus and network side device for determining model accuracy according to the embodiments of the present application through several embodiments and application scenarios thereof in conjunction with the drawings.

[0025] As shown in Figure 2, an embodiment of the present application provides a model accuracy determination method, the execution body of this method includes a first network element, and the first network element includes a model inference function network element, in other words, the method can be executed by software or hardware installed in the first network element. The method includes the following steps:

[0026] S210, a first network element infers a task based on a first model.

[0027] In one embodiment, the first network element may be a network element having both a model inference function and a model training function, for example, the first network element may be an NWDAF, and the NWDAF may include an Analytics Logical Function network element (AnLF) and a Model Training Logical Function (MTLF).

[0028] In another embodiment, the first network element includes a model inference function network element and the second network element includes a model training function network element, for example, the first network element is an AnLF and the second network element is an MTLF.

[0029] Taking the NWDAF as the first network element, the second network element and the first network element in the following embodiments may be the same network element, that is, the MTLF and the AnLF are integrated into the NWDAF.

[0030] It should be understood that the first model may be constructed and trained according to actual needs, such as an AI / ML model. The MTLF collects training data and performs model training based on the training data. After training is completed, the MTLF sends the information of the first model after training to the AnLF.

[0031] After determining a task to be triggered, the AnLF infers the task based on the first model and obtains inference result data.

[0032] It should be understood that the task is not a single task, but a data analysis task for indicating one task type, and after triggering the task, the AnLF can determine a first model corresponding to the task based on the identifier information (Analytics ID) of the task, etc., and infer the task based on the corresponding first model to obtain inference result data. For example, if the Analytics ID=UE mobility of the task is used to predict the movement trajectory of a terminal (also called User Equipment (UE)), the AnLF can infer the task based on the first model corresponding to UE mobility, and the obtained inference result data is predicted terminal location (UE location) information.

[0033] The AnLF may infer a task one or more times based on the first model to obtain inference result data including a plurality of inference result data, or a plurality of output result values.

[0034] In one embodiment, the execution of the AnLF inferring a task may be triggered by a third network element sending a task request message, the third network element being the network element that triggers the task, and the third network element may include a consumer network element (consumer Network Function, consumer NF), which may be a network element of a 5G system, a terminal or a third party application function (Application Function, AF), etc.

[0035] In another embodiment, the task may be triggered autonomously by the AnLF, for example by setting up a validation test phase in which the task is simulated and triggered autonomously by the AnLF to test the accuracy of the first model.

[0036] For simplicity, in the following embodiments, an AnLF is the first network element, an MTLF is the second network element, and a consumer NF is the third network element.

[0037] S220, the first network element determines a first accuracy corresponding to the first model to indicate a degree of accuracy of an inference result of the first model for the task.

[0038] After completing the inference for the task, the AnLF calculates a first accuracy corresponding to the first model in the inference process, and the first accuracy is AiU. The manner of calculating the first accuracy may be various, and the embodiment of the present application shows only one of the embodiments for implementing the invention. The step S220 is: The first network element obtains inference result data corresponding to the task based on the first model; the first network element obtaining label data corresponding to the inference result data, the label data may be obtained from the label data source device, and in one embodiment, the label data is data ground truth, i.e., facts and data that actually occurred; The first network element may calculate a first accuracy of the first model based on the inference result data and the label data, specifically, the inference result data may be compared with the corresponding label data to determine the number of correct results in the inference result data, and the first accuracy is calculated by dividing the number of correct results by the total number, which is represented by the following formula: Primary accuracy = number of correct results / total number Here, the correct result may represent that the inference result data and the label result data match, or that the gap between the inference result data and the label result data is within an acceptable range.

[0039] The expression format of the first accuracy may be various and is not limited to a specific percentage value, for example, 90%, but may be a categorical expression format, for example, high, medium, low, etc., or normalized data, for example, 0.9.

[0040] The first accuracy in the embodiment of the present application may indicate the accuracy of the inference result of the first model for the task from the front or the opposite side, and in one embodiment, the first accuracy is: The accuracy of the inference result of the task (for example, the accuracy of the first model may be directly indicated by calculating the accuracy rate of the inference result of the first model for the task); and the degree of error of the inference result of the task (e.g., the accuracy of the first model may be indicated from the opposite side by calculating an error rate or inference error of the inference result for the task of the first model. Here, the method of calculating the inference error may be various, for example, Mean Absolute Error (MAE), Mean Square Error (MSE), etc.).

[0041] In one embodiment, the first network element obtaining label data corresponding to the inference result data includes: the first network element determining a source device of label data corresponding to the task; The first network element obtains the label data from the source device.

[0042] Here, the source device of the label data may be determined by AnLF based on type information of the output data of the first model, limiting condition information and target information of the task, etc.

[0043] It should be noted that the trigger conditions of step S220 may vary and may be pre-set by the AnLF or may be obtained from the MTLF, for example from a model performance subscription request (Performance Monitoring) sent to the AnLF.

[0044] S230, when the first accuracy reaches a preset condition, the first network element sends first information to a second network element to indicate that the accuracy of the first model does not meet an accuracy demand or is declining, where the second network element is a network element that provides the first model.

[0045] The AnLF determines whether the accuracy of the first model satisfies the accuracy demand or is degraded according to whether the first accuracy reaches a preset condition. The preset condition may include that the first accuracy is smaller than a preset threshold or the degraded width reaches a certain degree.

[0046] When the AnLF determines that the accuracy of the first model does not meet the accuracy demand or is deteriorating, it transmits first information to the MTLF providing the first model to inform the MTLF that the accuracy of the first model does not meet the accuracy demand or is deteriorating.

[0047] In one embodiment, the first network element may send the first information to the second network element based on a triggering event, where the triggering event may include the first network element completing a first accuracy calculation or arriving at a particular time.

[0048] In another embodiment, the first network element may transmit the first information to the second network element based on a preset periodicity.

[0049] It should be explained that, when sending the first information to the second network element based on a trigger event or based on a preset period, the first information includes: the accuracy of the first model; and The accuracy of the first model meets the accuracy requirements; and This may be used to indicate at least one of that the accuracy of the first model does not meet accuracy needs or is deteriorating.

[0050] After receiving the first information, the MTLF may determine a subsequent operation based on the first information, for example, the MTLF may retrain the first model, or reselect another model that can be used to infer the task and send it to the AnLF, where the MTLF may require the other model to satisfy a certain condition when selecting the other model, and the certain condition may include at least one of: a second accuracy of the other model is higher than a second accuracy of the first model, a second accuracy of the other model is higher than a first accuracy of the first model, and a model performance requirement of the other model is higher than a model performance requirement of the first model.

[0051] Prior to step S230, the method further comprises: The method further includes the step of the first network element receiving a model performance subscription request from the second network element, the model performance subscription request being used to request the first network element to monitor accuracy of the first model.

[0052] It should be noted that the first network element may receive the model performance subscription request from the second network element before step S210 or S220.

[0053] In response, the first network element may send a Model Performance Monitoring Notification to the second network element based on the model performance subscription request, and the model performance subscription response message may carry the first information.

[0054] As can be seen from the technical solutions of the above embodiments, the embodiments of the present application include: a first network element infers a task based on a first model, determines a first accuracy corresponding to the first model, and when the first accuracy reaches a predetermined condition, the first network element sends first information to a second network element, indicating that the accuracy of the first model does not meet the accuracy demand or has deteriorated, thereby monitoring the accuracy of the model in the actual application process, and taking corresponding measures in a timely manner when the accuracy is deteriorated, to prevent erroneous policy decisions or inappropriate operations from being executed.

[0055] Based on the above embodiment, further as shown in FIG. 3, the method for determining the accuracy of the model includes the following steps:

[0056] The MTLF may pre-train the first model, and the training process may include steps A1 to A2.

[0057] Step A1. The MTLF collects training data from training data source devices.

[0058] Step A2. MTLF trains a first model based on the training data.

[0059] Step A5. After completing the training of the first model, the MTLF may send information of the trained first model to the AnLF.

[0060] In one embodiment, the message specifically carrying the information of the first model may be a Nnwdaf_MLModelProvision_Notify or Nnwdaf_MLModelInfo_Response message.

[0061] In one embodiment, before step A5, the method further comprises the steps of:

[0062] Step A4. The AnLF sends a message of the requested model to the MTLF.

[0063] In one embodiment, in the training phase or the post-training test phase of the first model in step A2, the MTLF needs to evaluate the accuracy of the first model and calculate a second accuracy of the first model, i.e., AiT. The second accuracy may be obtained using the same calculation formula as the first accuracy. Specifically, the MTLF may set a validation data set for evaluating the second accuracy of the first model, where the validation data set includes input data used in the first model and corresponding label data, and the MTLF inputs the input data into the first model after training to obtain output data, compares whether the output data and the label data match, and further calculates the second accuracy of the first model according to the above formula.

[0064] Accordingly, in one embodiment, when sending information of the first model to the AnLF in step A5, the MTLF may simultaneously send a second accuracy of the first model, or may send the second accuracy of the first model to the AnLF by an independent message.

[0065] In one embodiment, the requested model message may include requirements information for the first model of the AnLF, such as model performance requirements, model description format or language requirements, manufacturer requirements, aging requirements, domain requirements, model size requirements, etc., where the model performance requirements may be a second accuracy requirement for the first model.

[0066] Accordingly, in one embodiment, when the MTLF transmits information of the first model to the AnLF in step A5, related information corresponding to the first model and demand information, such as the second accuracy, the description format or language of the model, model aging information, application area information, model size information, etc., may be transmitted at the same time.

[0067] In one embodiment, if the task is triggered by a third network element, before step A4, the method further comprises the following steps:

[0068] Step A3. The Consumer NF sends a task request message to the AnLF, and the task request message is used to trigger the AnLF to execute an inference process for the task based on a first model corresponding to the task by requesting inference of the task.

[0069] The task request message includes description information of the task, which may be various and may include identifier information of the task, limiting condition information of the task (Analytics Filter Information), target information of the task (Analytics Target), etc. The target and scope of the task may be determined based on the description information of the task.

[0070] The limiting condition information of the task is used to limit the execution range of the task, and may include a time range, an area range, and the like.

[0071] The target information of the task is used to specify an object that is a target of the task, such as a certain terminal identifier (UE ID), a certain terminal group identifier (UE group ID), or any terminal (any UE).

[0072] The AnLF may request a model from a MTLF according to the task request message, and obtain information of a first model and a second accuracy of the first model from the MTLF.

[0073] In one embodiment, steps A1-A2 may be located after step A4, i.e., after receiving a message of a requested model sent by the AnLF, the MTLF trains a first model corresponding to the task, and sends a message of the trained first model to the AnLF.

[0074] In one embodiment, as shown in FIG. 3, step S210 includes steps A6 to A8.

[0075] Step A6. AnLF, based on the received task request message, a first model corresponding to the task; type information of input data of the first model; type information of the output data of the first model; a source device of inference input data corresponding to the task; At least one association information with a source device of the label data corresponding to the task is determined.

[0076] Here, the first model corresponding to the task may be determined by determining the first model that the task needs to use based on the task type indicated by the analytics ID in the task request message, or may be determined based on a mapping relationship between the analytics ID and the first model, where the first model, for example, model 1, may be represented by model identifier information (model ID).

[0077] The type information of the input data of the first model may be called metadata information of the model, for example, the input data may include a terminal identifier (UE ID), time, and a terminal current service state, etc.

[0078] The type information of the output data of the first model includes a data type, for example, a Tracking Area (TA) or a cell for indicating a UE location.

[0079] Specifically, the AnLF may determine the target and scope of the task based on information such as the analytics filter information and the analytics target in the task request message, and further determine a network element that can obtain the inference input data corresponding to the task as the source device of the inference input data corresponding to the task based on the target, scope and metadata information.

[0080] Specifically, the source device of the label data corresponding to the task may be determined by the AnLF determining a network element device type (NF type) capable of providing the output data based on the type information of the output data of the first model, and further determining a specific network element instance corresponding to the network element device type based on the constraint condition information and target information of the task, and setting the network element instance as the source device of the label data. For example, based on the data type=UE location of the output data of the first model corresponding to the task UE mobility, the AnLF determines that the data of the UE location can be provided by the network element device type AMF type, and based on the constraint condition information AOI of the task and the target UE1 of the task, the AnLF queries a unified data management entity (UDM) or a network repository function (NRF) that the corresponding AMF instance is AMF1, and then the AnLF sets AMF1 as the source device of the label data, and obtains the label data of the UE location from AMF1.

[0081] Step A7. The AnLF obtains inference input data corresponding to the task, specifically, the AnLF may send a request message for inference input data according to the source device of the inference input data of the task determined in step A6 to collect the inference input data corresponding to the task.

[0082] Step A8. AnLF infers inference input data corresponding to the task based on the acquired first model, and obtains inference result data.

[0083] For example, the AnLF infers inference input data corresponding to the task, such as numerical values ​​of UE ID, time, current service status of the UE, etc., based on a first model corresponding to analytics ID=UE mobility, and obtains output data whose inference result data is UE location.

[0084] In one embodiment, after obtaining inference result data corresponding to the task based on the first model, the method further includes the steps of:

[0085] Step A9. The first network element sends the inference result data to a third network element, that is, sends the inference result data obtained through AnLF inference to a consumer NF.

[0086] According to the inference result data, the first model corresponding to the analytics ID may be used to inform the consumer NF of the statistics or prediction values ​​obtained by the inference, and used to assist the consumer NF in implementing a corresponding policy decision. For example, the statistics or prediction values ​​corresponding to the UE mobility may be used to assist the AMF in performing user paging optimization.

[0087] In one embodiment, the message specifically carrying the inference result data may be a Nnwdaf_AnalyticsSubscription_Notify or Nnwdaf_AnalyticsInfo_Response message.

[0088] In step A10, AnLF obtains label data corresponding to the inference result data.

[0089] In one embodiment, the message specifically carrying the label data may be a Nnf_EventExposure_Subscribe message.

[0090] Specifically, the AnLF may be used to determine which label data to feed back to the source device of the label data determined in step A6, by sending a label data request message including type information of the label data, object information corresponding to the label data, time information (e.g., timestamp, time zone), etc.

[0091] The type information of the label data, the target information corresponding to the label data, the time information, etc. in the request message for label data may be determined by the AnLF based on the type information of the output data of the first model, the target information of the task, and the limiting condition information of the task, etc. Specifically, the AnLF determines the type information of the label data that needs to be acquired based on the type information of the output data of the first model, the AnLF determines the target information of the label data that needs to be acquired based on the target information of the task, and when the AnLF determines based on the limiting condition information of the task that the inference process of the task is a statistical calculation performed for a certain time in the past or a prediction performed for a certain time in the future, the AnLF needs to further acquire the label data corresponding to the certain time in the past or the certain time in the future.

[0092] For example, the AnLF sends a label data request message to the AMF or a Location Management Function (LMF), where the data type corresponding to the label data carrier is UE location, the target information is UE 1, and the time information is a specific time period, and is used to request the AMF / LMF to feed back the UE location data of UE 1 within a specific time period.

[0093] It should be understood that in step A8, if the AnLF obtains multiple inference result data by executing one or multiple inference processes, the AnLF needs to obtain multiple label data corresponding to the multiple inference result data accordingly.

[0094] In one embodiment, as shown in FIG. 3, the step S220 includes steps A11 to A12.

[0095] Step A11.AnLF calculates a first accuracy of the first model based on the inference result data and label data.

[0096] Step A12. AnLF determines whether the first accuracy satisfies a preset condition, and executes step A13 if the first accuracy satisfies AnLF.

[0097] The preset conditions may be set according to actual needs. In one embodiment, the preset conditions include: the first accuracy being less than a first threshold; and the first accuracy is less than the second accuracy; and The first accuracy is lower than the second accuracy, and a difference value between the first accuracy and the second accuracy is greater than a second threshold.

[0098] The method of obtaining the first threshold may be various, and the first threshold may be set by the AnLF, or may be set by the MTLF and transmitted to the AnLF as a judgment condition or a trigger condition.

[0099] In one embodiment, the MTLF sets it to the Model Performance Requirement requested when the AnLF requests the first model.

[0100] In another embodiment, when the second network element transmits the information of the first model to the first network element, the information of the first model may be transmitted together with the information of the first model.

[0101] In another embodiment, the model performance subscription may be carried by a model performance subscription message sent by the second network element to the first network element.

[0102] In one embodiment, as shown in FIG. 3, step S230 includes step A13.

[0103] Step A13. The AnLF sends first information to the MTLF to inform the MTLF that the accuracy of the first model does not meet the accuracy demand or is degraded.

[0104] In one embodiment, the first information may be specifically sent by a Nnwdaf_AnalyticsSubscription_Notify message.

[0105] In another embodiment, the AnLF may transmit the first information to the MTLF via a Model Performance Monitoring Notification.

[0106] In another embodiment, the first information may be used to request the MTLF to retrain the first model or to re-request a model, and in this case, the first information may be specifically sent by a Nnwdaf_MLModelProvision_Subscribe or Nnwdaf_MLModelInfo_Request message.

[0107] In one embodiment, the first information comprises: Identifier information of the first model (e.g., Model ID); Identifier information for the task (e.g., Analytics ID); The scope of the first information, including a time scope, an area scope, and a target scope, is specified by limiting condition information of the task, i.e., the scope and target scope of the task when the accuracy of the first model does not meet the accuracy demand or is decreased; an indication that the accuracy of the first model does not meet or is degraded by an accuracy requirement; The first accuracy; and request instruction information to retrain the first model, for instructing a MTLF to retrain the first model; model re-request instruction information for requesting acquisition of a model corresponding to the task; and first data for the task for retraining the first model.

[0108] In one embodiment, the first data comprises: inference input data corresponding to the task; Output result data corresponding to the task; and label data corresponding to the task.

[0109] Step A14. The MTLF may enter a retraining process of the first model based on the first information, and the specific training process is basically the same as that in step A2, with the specific difference being that the training data may include the first data of the task. The retraining process of the first model may be to re-perform model training on the initialized first model based on the training data, or to train the current first model based on the training data to realize fine-tuning of the first model, which can achieve faster convergence and save resources.

[0110] In one embodiment, after the MTLF completes retraining the first model, the method further comprises the steps of:

[0111] Step A15. The first network element receives second information including information of the first model after retraining from the second network element, i.e., the MTLF sends the information of the first model after retraining to the AnLF, allowing the AnLF to recover the inference of the task.

[0112] In one embodiment, the second information comprises: Application condition information of the first model after the retraining (the application condition information may include a time range, a region range, a target range, etc. that are objects of the first model); and a third accuracy of the first model after retraining, i.e., the AiT of the first model after retraining, for indicating the degree of accuracy of the model output result presented by the first model after retraining in the training phase or the testing phase.

[0113] In another embodiment, after step A14, the method further comprises: The method further includes a step of the MTLF transmitting information of the retrained first model to a sixth network element that is a network element that needs to infer using the first model, where the sixth network element includes a model inference function network element.

[0114] This corresponds to the fact that after the MTLF completes retraining the first model, it may send the retrained first model to other AnLFs as required, so that the other AnLFs can use the first model.

[0115] As can be seen from the technical solutions described in the above embodiments, in the embodiments of the present application, when the first accuracy reaches a preset condition, the first network element sends first information to the second network element indicating that the accuracy of the first model does not meet the accuracy demand or has deteriorated, and causes the second network element to retrain the first model, so that when the accuracy of the model is deteriorated, the first model can be retrained in a timely manner, the accuracy of task inference can be quickly restored, and erroneous policy decisions or inappropriate operations can be prevented.

[0116] Based on the above embodiment, further as shown in FIG. 4, the method further includes the following steps:

[0117] Step B14. The first network element sends third information to the fourth network element to instruct the fourth network element to store the first data of the task, and the fourth network element is a network element that receives and stores the first data from the first network element.

[0118] The AnLF may store the first data in a fourth network element, which may include a storage network element and may be an Analytics Data Repository Function (ADRF).

[0119] In one embodiment, the third information comprises: Identifier information of the task; Limitation condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; Label data corresponding to the task; and memory cause information (eg, the accuracy of the first model does not meet the accuracy demand or is declining).

[0120] Accordingly, the first information sent from the AnLF to the MTLF in step A13 may further include information of a fourth network element, for example identifier information of the ADRF.

[0121] Step B15. The MTLF acquires the first data from the ADRF based on the first information, and specifically, the MTLF may send data request information for the first data to the ADRF to indicate a data range to be acquired, where the data request information includes: Identifier information of a task; Limitation condition information of the task; Target information of the task; Identifier information of the first model; input data type information of the first model; and output data type information of the first model.

[0122] In one embodiment, the data request information may further include a request cause, for example, that the first model needs to be retrained or that the accuracy of the first model does not meet or has deteriorated an accuracy demand.

[0123] In one embodiment, the fourth network element may be used to store information of the first model, and after step A14, the method further comprises the following steps:

[0124] Step B16: The second network element stores the information of the first model after the retraining in the fourth network element.

[0125] The fourth network element may store application condition information of the first model after retraining and a third accuracy of the first model after retraining.

[0126] The AnLF may obtain information of the first model after retraining from the fourth network element according to actual demand.

[0127] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application provide that the fourth network element stores the first data of the task, so that when the accuracy of the model is reduced, the fourth network element can timely store relevant data corresponding to the task for retraining the first model, so that the first model can be updated timely and quickly restore the accuracy of inferring the task, and prevent erroneous policy decisions or the execution of inappropriate operations.

[0128] Based on the above embodiment, as shown in FIG. 5, the step A14 is obtaining second data for the first model, the second network element being training data for retraining the first model based on the first information; and the second network element retraining the first model based on the second data.

[0129] In one embodiment, the step of the second network element obtaining second data of the first model based on the first information comprises the steps of:

[0130] Step C14. The second network element determines a second data source device of the first model, that is, an inference data source device shown in FIG. 3; Step C15. The second network element obtains the second data from the source device.

[0131] In one embodiment, the step C15 includes a step of the second network element transmitting data request information to the source device for requesting the source device to provide the second data; Here, the data request information is Identifier information of a task; Limitation condition information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0132] In one embodiment, the second data source device comprises: Identifier information of a task; Limitation condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0133] In one embodiment, the second data may include first data of a task, ie, inference input data, inference result data, and label data corresponding to the task.

[0134] As can be seen from the technical solutions described in the above embodiments, in the embodiments of the present application, the second network element self-obtains second data of the first model from a second data source device based on the first information, so as to retrain the first model, so that the first model can be updated in a timely manner, quickly restore the accuracy of task inference, and prevent erroneous policy decisions or the execution of inappropriate operations.

[0135] Based on the above embodiment, when the first accuracy reaches a preset condition, the method further comprises: The first network element further includes: requesting the fifth network element to obtain a second model, which is a model used for the task provided by the fifth network element; the specific process can refer to steps A4 to A5. Here, the fifth network element includes a model training function network element, that is, the fifth network element can be another MTLF other than the second network element.

[0136] The first network element infers the task based on the second model to obtain new inference result data of the task, where the inferred task may be a task triggered according to the task request message sent by the consumer NF in step A3, or may be a task triggered by the consumer NF re-sending the task request message.

[0137] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application obtain a second model from a fifth network element when the first accuracy reaches a preset condition, infer a task, and obtain new inference result data, so that when the accuracy of the model is declining, corresponding adjustment measures can be taken in a timely manner, so as to quickly restore the accuracy of inferring the task, and prevent erroneous policy decisions or the execution of inappropriate operations.

[0138] Based on the above embodiment, when the first accuracy reaches a preset condition, the method further comprises: The method further includes the first network element sending fourth information to a third network element to indicate that an accuracy of the first model does not meet an accuracy demand or is deteriorating.

[0139] In one embodiment, the fourth information is: All or part of the description information of the task for instructing the task to be inferred using the first model (specifically, it may include an Analytics ID, Analytics filter information, analytics target, etc.); an indication that the accuracy of the first model does not meet or is degraded by an accuracy requirement; The first accuracy; and Recommended operation information for recommending to a consumer NF an operation to be performed after receiving the first information; and latency information for indicating the time required for the first network element to recover inference for the task (specifically, the latency required for the AnLF to obtain a first model after retraining, perform inference and obtain inference result data, or the latency required for the AnLF to obtain a second model that can be used to perform the task, perform inference and obtain inference result data).

[0140] In one embodiment, the recommended operation information comprises: Instructing the consumer NF to continue using the inference result data corresponding to the task, i.e., to continue using the already obtained inference result data; Instructing the consumer NF to stop using the inference result data corresponding to the task, i.e., to stop using the inference result data already obtained; and instructing the consumer NF to re-trigger the task to obtain new inference result data, i.e., to re-send the task request message.

[0141] After receiving the first information, the consumer NF may execute step A14, and perform a corresponding operation according to the first information.

[0142] Specifically, the consumer NF may perform at least one of the following operations based on the first information.

[0143] Continue to use the inference result data corresponding to the task, which may be performed if the decrease in the first accuracy is relatively small and does not exceed a predetermined width threshold, for example a second threshold, and in one embodiment, continuing to use the inference result data corresponding to the task may appropriately reduce the weight of the inference result data in making policy decisions.

[0144] Stopping the use of inference result data corresponding to the task, which may be performed when the deterioration width of the first accuracy is relatively large and has already exceeded a preset width threshold; retransmitting the task request message to the first network element to re-request the first network element to infer the task; retransmitting the task request message to a seventh network element to request the seventh network element to infer the task, the seventh network element including a model inference function network element, i.e., the consumer NF may send the task request message to another AnLF other than the first network element.

[0145] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application monitor the accuracy of the model in the actual application process by sending fourth information to a third network element when the first accuracy reaches a preset condition, indicating that the accuracy of the first model does not meet the accuracy demand or is declining, and causing the third network element to perform a corresponding operation, and promptly notifying the third network element to take corresponding measures when the accuracy is declining, thereby preventing erroneous policy decisions or the execution of inappropriate operations.

[0146] In the model accuracy determination method according to the embodiment of the present application, the execution body may be a model accuracy determination device. In the embodiment of the present application, the model accuracy determination device is taken as an example to execute the model accuracy determination method to describe the model accuracy determination device according to the embodiment of the present application.

[0147] As shown in FIG. 6, the model accuracy determining device includes an execution module 601, a calculation module 602 and a transmission module 603.

[0148] The execution module 601 is used to infer a task based on a first model, the calculation module 602 is used to determine a first accuracy corresponding to the first model to indicate the accuracy degree of the inference result of the first model for the task, and the transmission module 603 is used to send first information to a second network element to indicate that the accuracy of the first model does not meet the accuracy demand or has decreased when the first accuracy reaches a preset condition, where the second network element is a network element that provides the first model.

[0149] Additionally, the model accuracy determination device includes a model inference function network element.

[0150] Furthermore, the second network element includes a model training function network element.

[0151] Furthermore, the transmission module 603 is further used for receiving a model performance subscription request from the second network element, where the model performance subscription request is used for requesting monitoring of the accuracy of the first model.

[0152] Further, the transmission module 603 is used for obtaining inference result data corresponding to the task based on the first model, and obtaining label data corresponding to the inference result data.

[0153] The calculation module 602 is used to calculate a first accuracy of the first model based on the inference result data and the label data.

[0154] Furthermore, the first accuracy is The accuracy of the inference result of the task; and and the degree of error in the inference result of the task.

[0155] The implementation of the present application can realize the embodiment of the method shown in FIG. 2 and obtain the same technical effects, and the repeated parts will not be further described here.

[0156] As can be seen from the technical solutions of the above embodiments, the embodiments of the present application infer a task based on a first model, determine a first accuracy corresponding to the first model, and when the first accuracy reaches a preset condition, send first information to a second network element to indicate that the accuracy of the first model does not meet the accuracy demand or is declining, thereby monitoring the accuracy of the model in the actual application process, and taking corresponding measures in a timely manner when the accuracy is declining, to prevent erroneous policy decisions or the execution of inappropriate operations.

[0157] Based on the above embodiment, further, after sending the first information to the second network element, the transmission module is further used to receive second information from the second network element, where the second information includes information of the first model after retraining.

[0158] Furthermore, the first information is Identifier information of the first model; Identifier information of the task; Limitation condition information of the task; an indication that the accuracy of the first model does not meet or is degraded by an accuracy requirement; The first accuracy; and A request instruction to retrain the first model; and model re-request instruction information for requesting acquisition of a model corresponding to the task (this model may be the first model after retraining, or may be another model that can be used to infer the task); and first data for the task for retraining the first model; and information of a fourth network element that is a network element that receives and stores the first data from the first network element.

[0159] Furthermore, the first data is inference input data corresponding to the task; Output result data corresponding to the task; and label data corresponding to the task.

[0160] Furthermore, the second information is Application condition information of the first model after the retraining; and a third accuracy of the first model after retraining for indicating a degree of accuracy of model output results provided by the first model after retraining in a training phase or a testing phase.

[0161] Furthermore, the transmission module determining a source device of label data corresponding to the task; It is used to obtain the label data from the source device.

[0162] Furthermore, after obtaining inference result data corresponding to the task based on the first model, the transmission module is further used for the first network element to transmit the inference result data to a third network element, which is a network element that triggers the task.

[0163] Furthermore, the preset condition is the first accuracy being less than a first threshold; and the first accuracy is less than the second accuracy; and the first accuracy is lower than the second accuracy, and a difference value between the first accuracy and the second accuracy is greater than a second threshold value; Here, the second accuracy is used to indicate the degree of accuracy of the model output result presented by the first model in the training stage or the testing stage.

[0164] Furthermore, the third network element includes a consumer network element.

[0165] The implementation of the present application can realize the embodiment of the method shown in FIG. 3 and obtain the same technical effects, and the repeated parts will not be further described here.

[0166] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application send first information to a second network element when the first accuracy reaches a preset condition, indicating that the accuracy of the first model does not meet the accuracy demand or has declined, and have the second network element retrain the first model, so as to timely retrain the first model when the accuracy of the model declines, and quickly restore the accuracy of task inference, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0167] Based on the above embodiment, further, the transmission module is used for the first network element to send third information to a fourth network element, for instructing the fourth network element to store the first data of the task.

[0168] Furthermore, the third information is Identifier information of the task; Limitation condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; Label data corresponding to the task; and storage cause information.

[0169] Furthermore, the fourth network element includes a storage network element.

[0170] The implementation of the present application can realize the embodiment of the method shown in FIG. 4 and obtain the same technical effects, and the repeated parts will not be further described here.

[0171] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application provide that the fourth network element stores the first data of the task, so that when the accuracy of the model is reduced, the fourth network element can timely store relevant data corresponding to the task for retraining the first model, so that the first model can be updated timely and quickly restore the accuracy of inferring the task, and prevent erroneous policy decisions or the execution of inappropriate operations.

[0172] Based on the above embodiment, further, when the first accuracy reaches a preset condition, the sending module is further adapted to request a fifth network element to obtain a second model, which is a model used for the task provided by the fifth network element; The execution module is used to infer the task based on the second model.

[0173] Furthermore, the fifth network element includes a model training function network element.

[0174] The implementation of the present application can realize the above method embodiments and obtain the same technical effects, and the repeated parts will not be further described here.

[0175] As can be seen from the technical solutions described in the above embodiments, in the embodiments of the present application, the second network element self-obtains second data of the first model from a second data source device based on the first information, so as to retrain the first model, so that the first model can be updated in a timely manner, quickly restore the accuracy of task inference, and prevent erroneous policy decisions or the execution of inappropriate operations.

[0176] Based on the above embodiment, when the first accuracy reaches a preset condition, the method further comprises: The method further includes the first network element sending fourth information to a third network element to indicate that an accuracy of the first model does not meet an accuracy demand or is deteriorating.

[0177] Furthermore, the fourth information is All or a part of the description information of the task; an indication that the accuracy of the first model does not meet or is degraded by an accuracy requirement; The first accuracy; and Recommended operation information, and latency information to indicate the time required for the first network element to recover inference of the task.

[0178] The implementation of the present application can realize the above method embodiments and obtain the same technical effects, and the repeated parts will not be further described here.

[0179] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application monitor the accuracy of the model in the actual application process by sending fourth information to a third network element when the first accuracy reaches a preset condition, indicating that the accuracy of the first model does not meet the accuracy demand or is declining, and causing the third network element to perform a corresponding operation, and promptly notifying the third network element to take corresponding measures when the accuracy is declining, thereby preventing erroneous policy decisions or the execution of inappropriate operations.

[0180] The model accuracy determination device in the embodiment of the present application may be an electronic device, such as an electronic device having an operating system, or may be a component in an electronic device, such as an integrated circuit or chip. The electronic device may be a terminal or other devices other than a terminal. Exemplarily, the terminal may include, but is not limited to, the types of terminal 11 listed above, and the other devices may be servers, network attached storage (NAS), etc., and the embodiment of the present application is not specifically limited.

[0181] The model accuracy determination apparatus according to the embodiments of the present application can realize each process realized by the method embodiments of Figures 2 to 5 and achieve the same technical effects, and will not be further described here to avoid repetition.

[0182] As shown in Figure 7, the embodiment of the present application further provides another model accuracy determination method, and the execution body of this method is a second network element, where the second network element includes a model training function network element, in other words, this method can be executed by the software or hardware installed in the second network element. The method includes the following steps:

[0183] S710, the second network element receives first information from the first network element to indicate that the accuracy of the first model does not meet the accuracy demand or is deteriorating; Further, before step S710, the method further comprises: The method further includes the step of the second network element sending a model performance subscription request to the first network element to request the first network element to monitor accuracy of the first model.

[0184] S720, the second network element retrains the first model based on the first information.

[0185] Furthermore, the first network element includes a model inference function network element.

[0186] Furthermore, the second network element includes a model training function network element.

[0187] Furthermore, the second network element transmits second information, including information of the first model after retraining, to the first network element.

[0188] Furthermore, the first information is Identifier information for the first model; Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; an indication that the accuracy of the first model does not meet or is degraded by an accuracy requirement; the first accuracy level indicating a degree of accuracy of an inference result of the first model for the task; A request instruction to retrain the first model; and model re-request instruction information for requesting acquisition of a model corresponding to the task; first data for the task for retraining the first model; and information of a fourth network element that is a network element that receives and stores the first data from the first network element.

[0189] Furthermore, step S720 obtaining second data of the first model based on the first information by the second network element; and the second network element retraining the first model based on the second data.

[0190] Furthermore, the second information is Application condition information of the first model after the retraining; and a third accuracy of the first model after retraining for indicating a degree of accuracy of model output results provided by the first model after retraining in a training phase or a testing phase.

[0191] Further, after step S720, the method further comprises: The method further includes a step in which the second network element transmits information of the retrained first model to a sixth network element, which is a network element that needs to perform inference using the first model.

[0192] Furthermore, the sixth network element includes a model inference function network element.

[0193] Furthermore, the first accuracy is The accuracy of the inference result of the task; and and the degree of error in the inference result of the task.

[0194] The above steps S710 to S720 realize the embodiment of the method shown in FIG. 2 and FIG. 3, and can obtain the same technical effects, and the repeated parts will not be further described here.

[0195] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application receive first information from a first network element to indicate that the accuracy of a first model does not meet the accuracy demand or is declining, and retrain the first model based on the first information, so that when the accuracy of the model is declining, the first model can be retrained in a timely manner, the accuracy of task inference can be quickly restored, and erroneous policy decisions or inappropriate operations can be prevented from being executed.

[0196] Based on the above embodiment, the second network element obtaining second data of the first model based on the first information includes: determining a source device of second data of the first model by the second network element; The second network element acquires the second data from the source device.

[0197] Furthermore, the second network element acquiring the second data from the source device, the second network element transmitting to the source device data request information for requesting the source device to provide the second data; Here, the data request information is Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; Target information of the task; Identifier information of the first model; input data type information of the first model; and output data type information of the first model.

[0198] Furthermore, the second data source device Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0199] The embodiments of the present application can realize the method embodiments shown in FIG. 5 and obtain the same technical effects, and the repeated parts will not be further described here.

[0200] As can be seen from the technical solutions described in the above embodiments, in the embodiments of the present application, the second network element self-obtains second data of the first model from a second data source device based on the first information, so as to retrain the first model, so that the first model can be updated in a timely manner, quickly restore the accuracy of task inference, and prevent erroneous policy decisions or the execution of inappropriate operations.

[0201] Based on the above embodiment, further, the second data includes first data of a task, the task being a task of the first network element making an inference based on a first model.

[0202] Furthermore, the first data source device includes the fourth network element.

[0203] Furthermore, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0204] Further, after retraining the first model based on the first information, the method further comprises: The second network element further includes storing information of the retrained first model in the fourth network element.

[0205] Furthermore, the fourth network element includes a storage network element.

[0206] The embodiments of the present application can realize the method embodiments shown in FIG. 4 and obtain the same technical effects, and the repeated parts will not be further described here.

[0207] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application provide that the fourth network element stores the first data of the task, so that when the accuracy of the model is reduced, the fourth network element can timely store relevant data corresponding to the task for retraining the first model, so that the first model can be updated timely and quickly restore the accuracy of inferring the task, and prevent erroneous policy decisions or the execution of inappropriate operations.

[0208] In the model accuracy determination method according to the embodiment of the present application, the execution body may be a model accuracy determination device. In the embodiment of the present application, the model accuracy determination device according to the embodiment of the present application is taken as an example to execute the model accuracy determination method to describe the model accuracy determination device according to the embodiment of the present application.

[0209] As shown in FIG. 8, the model accuracy determining device includes a transceiving module 801 and a training module 802 .

[0210] The transceiver module 801 is used for receiving first information from a first network element to indicate that the accuracy of a first model does not meet an accuracy demand or is declining, and the training module 802 is used for retraining the first model based on the first information.

[0211] Furthermore, the transceiver module 801 is further used for sending a model performance subscription request to the first network element to request the first network element to monitor the accuracy of the first model.

[0212] Furthermore, the first network element includes a model inference function network element.

[0213] Furthermore, the second network element includes a model training function network element.

[0214] Moreover, the transceiver module 801 is further used for sending second information, including information of the first model after retraining, to the first network element.

[0215] Furthermore, the first information is Identifier information for the first model; Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; an indication that the accuracy of the first model does not meet or is degraded by an accuracy requirement; the first accuracy level indicating a degree of accuracy of an inference result of the first model for the task; A request instruction to retrain the first model; and model re-request instruction information for requesting acquisition of a model corresponding to the task; first data for the task for retraining the first model; and information of a fourth network element that is a network element that receives and stores the first data from the first network element.

[0216] Furthermore, the transceiver module 801 is used to obtain second data of the first model based on the first information; The training module 802 is used to retrain the first model based on the second data.

[0217] Furthermore, the second information is Application condition information of the first model after the retraining; and a third accuracy of the first model after retraining for indicating a degree of accuracy of model output results provided by the first model after retraining in a training phase or a testing phase.

[0218] Furthermore, the transceiver module 801 is further used for transmitting information of the first model after retraining to a sixth network element, which is a network element that needs to be inferred using the first model.

[0219] Furthermore, the sixth network element includes a model inference function network element.

[0220] Furthermore, the first accuracy is The accuracy of the inference result of the task; and and the degree of error in the inference result of the task.

[0221] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application receive first information from a first network element to indicate that the accuracy of a first model does not meet the accuracy demand or is declining, and retrain the first model based on the first information, so that when the accuracy of the model is declining, the first model can be retrained in a timely manner, the accuracy of task inference can be quickly restored, and erroneous policy decisions or inappropriate operations can be prevented from being executed.

[0222] Based on the above embodiment, further, the transceiver module: determining a source device of second data for the first model; It is used to acquire the second data from the source device.

[0223] Furthermore, the transceiver module is used to transmit data request information to the source device for requesting the source device to provide the second data; Here, the data request information is Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0224] Furthermore, the second data source device Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0225] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application independently obtain second data of the first model from a second data source device based on the first information to retrain the first model, so that the first model can be updated in a timely manner, quickly restore the accuracy of task inference, and prevent erroneous policy decisions or inappropriate operations from being performed.

[0226] Based on the above embodiment, further, the second data includes first data of a task, the task being a task of the first network element making an inference based on a first model.

[0227] Furthermore, a source device of the first data includes the fourth network element, the fourth network element being a network element that receives and stores the first data from the first network element.

[0228] Furthermore, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0229] Furthermore, after retraining the first model based on the first information, the transceiver module is further used for storing the information of the retrained first model in the fourth network element.

[0230] Furthermore, the fourth network element includes a storage network element.

[0231] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application provide that the fourth network element stores the first data of the task, so that when the accuracy of the model is reduced, the fourth network element can timely store relevant data corresponding to the task for retraining the first model, so that the first model can be updated timely and quickly restore the accuracy of inferring the task, and prevent erroneous policy decisions or the execution of inappropriate operations.

[0232] The model accuracy determination device in the embodiment of the present application may be an electronic device, such as an electronic device having an operating system, or may be a component in an electronic device, such as an integrated circuit or chip. The electronic device may be a terminal or other devices other than a terminal. Exemplarily, the terminal may include, but is not limited to, the types of terminal 11 listed above, and the other devices may be servers, network attached storage (NAS), etc., and the embodiment of the present application is not specifically limited.

[0233] The model accuracy determination apparatus according to the embodiment of the present application can realize each process realized by the embodiment of the method of Figure 7 and achieve the same technical effect, and will not be described further here in order to avoid repetition of description.

[0234] As shown in Figure 9, the embodiment of the present application further provides another model accuracy determination method, and the execution body of this method is a fourth network element, where the fourth network element includes a storage network element, in other words, this method can be executed by the software or hardware installed in the fourth network element. The method includes the following steps:

[0235] S910, a fourth network element receives third information from the first network element to instruct the fourth network element to store first data of a task, the task being that the first network element performs inference based on a first model; Here, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0236] Furthermore, the third information is Identifier information of the task; Limitation condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; Label data corresponding to the task; and storage cause information.

[0237] Further, after step S910, the method further comprises: receiving, by the fourth network element, data request information from a second network element that is a network element providing the first model; and transmitting first data of the task by the fourth network element to the second network element.

[0238] Furthermore, the data request information includes: Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; Target information of the task; Identifier information of the first model; input data type information of the first model; and output data type information of the first model.

[0239] Further, after transmitting the first data of the task to the second network element, the method further comprises: The fourth network element further includes receiving information of the retrained first model from the second network element.

[0240] Further, after receiving information of the retrained first model from the second network element, the method further comprises: The method further includes the fourth network element sending information of the retrained first model to the first network element.

[0241] Furthermore, the first network element includes a model inference function network element.

[0242] Furthermore, the second network element includes a model training function network element.

[0243] Furthermore, the fourth network element includes a storage network element.

[0244] The above step S910 realizes the embodiment of the method shown in FIG. 4, and the same technical effects can be obtained, and the repeated parts will not be further described here.

[0245] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application provide that the fourth network element stores the first data of the task, so that when the accuracy of the model is reduced, the fourth network element can timely store relevant data corresponding to the task for retraining the first model, so that the first model can be updated timely and quickly restore the accuracy of inferring the task, and prevent erroneous policy decisions or the execution of inappropriate operations.

[0246] In the model accuracy determination method according to the embodiment of the present application, the execution body may be a model accuracy determination device. In the embodiment of the present application, the model accuracy determination device is taken as an example to execute the model accuracy determination method to describe the model accuracy determination device according to the embodiment of the present application.

[0247] As shown in FIG. 10, the model accuracy determination device includes a communication module 1001 and a storage module 1002.

[0248] The communication module 1001 is used to receive third information from a first network element to instruct the first network element to store first data of a task, the first network element performing inference based on a first model, and the storage module 1002 is used to store the first data of the task; Here, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

[0249] Furthermore, the third information is Identifier information of the task; Limitation condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; Label data corresponding to the task; and storage cause information.

[0250] Furthermore, the communication module 1001 further receives data request information from a second network element, the second network element being a network element providing the first model; The fourth network element is used to transmit first data of the task to the second network element.

[0251] Furthermore, the data request information includes: Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

[0252] Moreover, the communication module 1001 is further used for receiving information of the first model after retraining from the second network element.

[0253] Moreover, the communication module 1001 is further used for transmitting information of the retrained first model to the first network element.

[0254] Furthermore, the first network element includes a model inference function network element.

[0255] Furthermore, the second network element includes a model training function network element.

[0256] Additionally, the model accuracy determination device includes a storage network element.

[0257] As can be seen from the technical solutions described in the above embodiments, the embodiments of the present application store the first data of the task, so that when the accuracy of the model is reduced, the relevant data corresponding to the task can be stored in a timely manner for retraining the first model, so that the first model can be updated in a timely manner and quickly restore the accuracy of inferring the task, and prevent erroneous policy decisions or the execution of inappropriate operations.

[0258] The model accuracy determination device in the embodiment of the present application may be an electronic device, such as an electronic device having an operating system, or may be a component in an electronic device, such as an integrated circuit or chip. The electronic device may be a terminal or other devices other than a terminal. Exemplarily, the terminal may include, but is not limited to, the types of terminal 11 listed above, and the other devices may be servers, network attached storage (NAS), etc., and the embodiment of the present application is not specifically limited.

[0259] The model accuracy determination apparatus according to the embodiment of the present application can realize each process realized by the embodiment of the method of Figure 9 and achieve the same technical effect, and will not be described further here in order to avoid repetition of description.

[0260] Optionally, as shown in Fig. 11, an embodiment of the present application further provides a communication device 1100, which includes a processor 1101 and a memory 1102, and the memory 1102 stores a program or instruction that can run on the processor 1101, for example, when the communication device 1100 is a terminal, the program or instruction can be executed by the processor 1101 to realize each step of the embodiment of the method for determining the accuracy of the model, and the same technical effect can be achieved. When the communication device 1100 is a network side device, the program or instruction can be executed by the processor 1101 to realize each step of the embodiment of the method for determining the accuracy of the model, and the same technical effect can be achieved, and in order to avoid repetition, no further description will be given here.

[0261] Specifically, an embodiment of the present application further provides a network side device. As shown in Fig. 12, the network side device 1200 includes a processor 1201, a network interface 1202 and a memory 1203. Here, the network interface 1202 is, for example, a common public radio interface (CPRI).

[0262] Specifically, the network side device 1200 of the embodiment of the present invention further includes instructions or programs stored in the memory 1203 and capable of running on the processor 1201, and the processor 1201 calls the instructions or programs in the memory 1203 to execute the methods performed by each module shown in Figures 6, 8 and 10, and achieves the same technical effect, and in order to avoid repetition of description, will not be described further here.

[0263] An embodiment of the present application further provides a readable storage medium, in which a program or instruction is stored, which, when executed by a processor, can realize each process of the embodiment of the method for determining the accuracy of the above model, and achieve the same technical effect, and will not be described further here in order to avoid repetition.

[0264] Wherein, the processor is the processor in the terminal according to the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read only memory ROM, a random access memory RAM, a magnetic disk, or an optical disk.

[0265] An embodiment of the present application further provides a chip, the chip including a processor and a communication interface, the communication interface is coupled to the processor, the processor is used to run a program or instruction to realize each process of the embodiment of the above model accuracy determination method, and can achieve the same technical effect, and in order to avoid repetition of description, it will not be described further here.

[0266] It should be understood that the chips referred to in the embodiments of the present application may be referred to as system level chips, system chips, chip systems, or systems on chips.

[0267] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium, and which can be executed by at least one processor to realize each process of the embodiment of the above-mentioned model accuracy determination method and achieve the same technical effects, which will not be described further here in order to avoid repetition.

[0268] An embodiment of the present application further provides a model accuracy determination system, the system including a network side equipment, the network side equipment including a first network element, a second network element, and a fourth network element, the first network element may be used to perform steps of the model accuracy determination method described above, the second network element may be used to perform steps of the model accuracy determination method described above, and the fourth network element may be used to perform steps of the model accuracy determination method described above.

[0269] It should be explained that in this specification, the terms "comprise", "include", or any other variants thereof are intended to cover the non-exclusive "comprise", whereby a process, method, article, or apparatus that includes a set of elements includes not only those elements, but also other elements not expressly listed or inherent to such process, method, article, or apparatus. In the absence of further limitations, an element limited by the phrase "comprises a" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes this element. It should be pointed out that the scope of the method and apparatus in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may include performing functions in an essentially simultaneous manner or in reverse order based on the functions involved, for example, the described method can be performed in a different order than described, and various steps can be added, omitted, or combined. Also, features described with reference to some examples can be combined in other examples.

[0270] From the above description of the embodiments, it is clear to those skilled in the art that the methods of the above embodiments can be realized in the form of software and a necessary general-purpose hardware platform. Of course, they may also be realized in hardware, but in many cases the former is a more preferred embodiment. In light of this understanding, the technical proposal of the present application may be substantially or the contributions made to the prior art may be embodied in the form of a computer software product, which is stored in a storage medium (e.g., a Read Only Memory (ROM) / Random-Access Memory (RAM), a magnetic disk, an optical disk) and includes some instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the embodiments of the present application.

[0271] The above describes the embodiments of the present application in conjunction with the drawings, but the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not limiting. Those skilled in the art can take the suggestions of this application and make many forms without departing from the spirit and scope of the claims of this application, all of which belong to the protection scope of this application.

[0272] CROSS-REFERENCE TO RELATED APPLICATIONS The present invention claims priority to a Chinese patent application submitted to the China Patent Office on March 7, 2022, bearing application number 202210224481.9 and titled "METHING, APPARATUS AND NETWORK SIDE EQUIPMENT FOR DETERMINING MODEL ACCURACY", and a Chinese patent application submitted to the China Patent Office on August 9, 2022, bearing application number 202210958867.2 and titled "METHING, APPARATUS AND NETWORK SIDE EQUIPMENT FOR DETERMINING MODEL ACCURACY", the entire contents of which are incorporated herein by reference.

Claims

1. 1. A method for determining the accuracy of a model, comprising: A first network element infers a task based on a first model; determining a first accuracy corresponding to the first model, the first network element indicating a degree of accuracy of an inference result of the first model for the task; When the first accuracy reaches a preset condition, the first network element sends first information to a second network element, the first information being used to indicate that the accuracy of the first model does not meet an accuracy demand or is deteriorating; wherein the second network element is a network element that provides the first model.

2. Before the first network element transmits the first information to the second network element, the method further comprises:

2. The method of claim 1, further comprising: the first network element receiving a model performance subscription request from the second network element, the model performance subscription request being used to request the first network element to monitor accuracy of the first model.

3. After transmitting the first information to the second network element, the method further comprises:

10. The method of claim 1, further comprising the first network element receiving second information from the second network element, the second information including information of the first model after retraining.

4. The first network element determining a first accuracy corresponding to the first model includes: The first network element obtains inference result data corresponding to the task based on the first model; and the first network element obtaining label data corresponding to the inference result data; The method of claim 1 , further comprising the first network element calculating a first accuracy of the first model based on the inference result data and the label data.

5. The first accuracy is The accuracy of the inference result of the task; and and an error degree of the inference result of the task.

6. The first information is Identifier information for the first model; Identifier information of the task; Limitation condition information of the task; an indication that the accuracy of the first model does not meet or is degraded by an accuracy requirement; The first accuracy; and A request instruction to retrain the first model; and model re-request instruction information for requesting acquisition of a model corresponding to the task; first data for the task for retraining the first model; and information of a fourth network element that is a network element that receives and stores the first data from the first network element.

7. The method comprises:

2. The method of claim 1, further comprising: the first network element sending third information to a fourth network element to instruct the fourth network element to store the first data of the task.

8. The third information is Identifier information of the task; Limitation condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; Label data corresponding to the task; 8. The method of claim 7, further comprising at least one of:

9. The first data is inference input data corresponding to the task; Output result data corresponding to the task; and label data corresponding to the task.

10. The second information is Application condition information of the first model after the retraining; and a third accuracy of the first model after retraining for indicating a degree of accuracy of model output results provided by the first model after retraining in a training phase or a testing phase.

11. The first network element obtaining label data corresponding to the inference result data includes: the first network element determining a source device of label data corresponding to the task; The method of claim 4 , further comprising the first network element obtaining the label data from the source device.

12. After obtaining inference result data corresponding to the task based on the first model, the method includes: The method of claim 4 , further comprising the first network element transmitting the inference result data to a third network element, the third network element being the network element that triggers the task.

13. The preset condition is: the first accuracy being less than a first threshold; and the first accuracy is less than the second accuracy; and the first accuracy is lower than the second accuracy, and a difference value between the first accuracy and the second accuracy is greater than a second threshold value; The method of claim 1 , wherein the second accuracy is used to indicate a degree of accuracy of a model output result provided by the first model during a training phase or a testing phase.

14. When the first accuracy reaches a preset condition, the method further comprises: the first network element requests a fifth network element to obtain a second model, the second model being a model used for the task provided by the fifth network element; The method of claim 1 , further comprising: the first network element inferring the task based on the second model.

15. When the first accuracy reaches a preset condition, the method further comprises:

2. The method of claim 1, further comprising: the first network element sending fourth information to a third network element to indicate that an accuracy of the first model does not meet an accuracy demand or is deteriorating.

16. The fourth information is All or a part of the description information of the task; an indication that the accuracy of the first model does not meet or is degraded by an accuracy requirement; The first accuracy; and Recommended operation information, and latency information to indicate the time required for the first network element to recover inference of the task.

17. The method of claim 1 , wherein the first network element comprises a model inference function network element.

18. The method of claim 1 , wherein the second network element comprises a model training function network element.

19. The method of claim 12 , wherein the third network element comprises a consumer network element.

20. The method of claim 6 , wherein the fourth network element comprises a storage network element.

21. 15. The method of claim 14, wherein the fifth network element comprises a model training function network element.

22. 1. An apparatus for determining the accuracy of a model, comprising: an execution module for inferring a task based on the first model; a computation module for determining a first accuracy corresponding to the first model, the first accuracy indicating a degree of accuracy of an inference result of the first model for the task; a transmission module for transmitting first information to a second network element to indicate that the accuracy of the first model does not meet an accuracy demand or is degraded when the first accuracy reaches a preset condition; wherein the second network element is a network element that provides the first model.

23. 1. A method for determining the accuracy of a model, comprising: receiving, by a second network element, first information from the first network element, the first information being used to indicate that an accuracy of the first model does not meet an accuracy need or is deteriorating; and the second network element retraining the first model based on the first information.

24. Before the second network element receives first information from the first network element, the method further comprises:

24. The method of claim 23, further comprising the second network element sending a model performance subscription request to the first network element to request the first network element to monitor accuracy of the first model.

25. 24. The method of claim 23, further comprising the second network element sending second information to the first network element, the second information including information of the first model after retraining.

26. The first information is Identifier information for the first model; Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; an indication that the accuracy of the first model does not meet or is degraded by an accuracy requirement; a first accuracy level indicating a degree of accuracy of an inference result of the first model for the task; A request instruction to retrain the first model; and model re-request instruction information for requesting acquisition of a model corresponding to the task; first data for the task for retraining the first model; and information of a fourth network element that is a network element that receives and stores the first data from the first network element.

27. The first accuracy is The accuracy of the inference result of the task; and and an error degree of the inference result of the task.

28. retraining the first model based on the first information includes: obtaining second data for the first model based on the first information by the second network element; and the second network element retraining the first model based on the second data.

29. The second network element obtaining second data of the first model based on the first information, determining a source device of second data of the first model by the second network element; and the second network element obtaining the second data from the source device.

30. The second network element acquiring the second data from the source device, the second network element transmitting to the source device data request information for requesting the source device to provide the second data; Here, the data request information is Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and an output data type information of the first model.

31. The second data source device is Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; Identifier information for the first model; input data type information of the first model; and output data type information of the first model.

32. 31. The method of claim 28, wherein the second data includes first data of a task, the task being a task of the first network element making an inference based on a first model.

33. 33. The method of claim 32, wherein a source device of the first data includes a fourth network element, the fourth network element being a network element that receives and stores the first data from the first network element.

34. The first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

35. The second information is Application condition information of the first model after the retraining; and a third accuracy of the first model after retraining to indicate a degree of accuracy of model output results that the first model after retraining provides in a training phase or a testing phase.

36. After retraining the first model based on the first information, the method further comprises:

24. The method of claim 23, further comprising the second network element storing the retrained first model information in a fourth network element.

37. After retraining the first model based on the first information, the method further comprises:

24. The method of claim 23, further comprising the second network element transmitting information of the retrained first model to a sixth network element that is a network element that needs to infer using the first model.

38. 24. The method of claim 23, wherein the first network element comprises a model inference function network element.

39. 24. The method of claim 23, wherein the second network element comprises a model training function network element.

40. 27. The method of claim 26, wherein the fourth network element comprises a storage network element.

41. 40. The method of claim 37, wherein the sixth network element comprises a model inference function network element.

42. 1. An apparatus for determining the accuracy of a model, comprising: a transceiver module for receiving first information from a first network element to indicate that an accuracy of the first model does not meet an accuracy need or is deteriorating; a training module for retraining the first model based on the first information.

43. 1. A method for determining the accuracy of a model, comprising: receiving, by a fourth network element, third information from the first network element to instruct the fourth network element to store first data for a task, the task being that the first network element performs inference based on a first model; Here, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

44. The third information is Identifier information of the task; Limitation condition information of the task; Target information of the task; inference input data corresponding to the task; inference result data corresponding to the task; Label data corresponding to the task; 44. The method of claim 43, further comprising at least one of:

45. After receiving the third information from the first network element, the method includes: receiving, by the fourth network element, data request information from a second network element that is a network element providing the first model; 44. The method of claim 43, further comprising: the fourth network element transmitting first data of the task to the second network element.

46. The data request information is Identifier information of a task in which the first network element performs inference based on a first model; Limitation condition information of the task; Target information of the task; Identifier information for the first model; input data type information of the first model; and an output data type information of the first model.

47. After transmitting the first data of the task to the second network element, the method further comprises:

46. ​​The method of claim 45, further comprising the fourth network element receiving retrained first model information from the second network element.

48. After receiving information of the retrained first model from the second network element, the method includes:

48. The method of claim 47, further comprising the fourth network element sending information of the retrained first model to the first network element.

49. 44. The method of claim 43, wherein the first network element comprises a model inference capability network element.

50. 46. ​​The method of claim 45, wherein the second network element comprises a model training function network element.

51. 44. The method of claim 43, wherein the fourth network element comprises a storage network element.

52. 1. An apparatus for determining the accuracy of a model, comprising: a communication module for receiving third information from the first network element for instructing the first network element to store first data for the task, the task being for the first network element to perform inference based on a first model; a storage module for storing first data of the task; Here, the first data is inference input data corresponding to the task; inference result data corresponding to the task; and label data corresponding to the task.

53. A network side device comprising a processor and a memory, the memory storing a program or instructions operable on the processor, the program or instructions, when executed by the processor, realizing the method for determining the accuracy of a model according to any one of claims 1 to 21, or the method for determining the accuracy of a model according to any one of claims 23 to 41, or the steps of the method for determining the accuracy of a model according to any one of claims 43 to 51.

54. A readable storage medium storing a program or instructions which, when executed by a processor, implements the method for determining the accuracy of a model according to any one of claims 1 to 21, or the method for determining the accuracy of a model according to any one of claims 23 to 41, or the steps of the method for determining the accuracy of a model according to any one of claims 43 to 51.

Images