Information acquisition method, network side device and storage medium

Abstract

The present application discloses an information acquisition method, an apparatus, a network side device, and a storage medium, which belong to the technical field of communications. The information acquisition method of an embodiment of the present application includes: a step of a first network side device sending first request information to a second network side device, the first request information being used to request the second network side device to determine model performance information of a target model; and a step of the first network side device receiving the model performance information of the target model sent from the second network side device, the first request information including target data and / or attribute information of the target data, the target data being used to determine the model performance information of the target model, and the attribute information of the target data being used to acquire the target data.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application filed with the China Patent Office on September 20, 2022, bearing application number 202211146190.9 and entitled "Information Acquisition Method, Apparatus, Network-Side Equipment, and Storage Medium," the entire contents of which are incorporated herein by reference.

[0002] The present application belongs to the technical field of communications, and specifically relates to an information acquisition method, an apparatus, a network side device, and a storage medium. [Background technology]

[0003] With the rapid development of computer and communication technologies, the application of artificial intelligence (AI) technology to communication networks is gradually increasing. Some network-side devices, such as network data analytics functions (NWDAFs), can perform intelligent data analysis for specific tasks, such as AI inference tasks, to generate data analysis results, which can then support policy decision-making by devices inside and outside the communication network. That is, artificial intelligence technology can enhance the intelligence of policy decision-making. For example, the network data analytics function can train an artificial intelligence / machine learning (ML) model based on training data to obtain a model suitable for a specific task. In actual use, data analysis can be performed based on the model and corresponding input data to obtain data analysis results corresponding to the specific task.

[0004] Accurate data analysis results are a prerequisite for devices inside and outside the communication network to make correct policy decisions. If the accuracy of the data analysis results is low and they are provided to devices inside and outside the communication network as a reference for policy decisions, the devices inside and outside the communication network are likely to make incorrect policy decisions or perform inappropriate operations.

[0005] The accuracy of data analysis results can be determined by the model performance during actual use of the artificial intelligence / machine learning model. Therefore, how to obtain model performance information during actual use in order to determine the accuracy of data analysis results and ensure that devices inside and outside the communication network make correct policy decisions is a technical problem that those skilled in the art must solve as soon as possible. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide an information acquisition method, device, network-side device, and storage medium for acquiring model performance information during actual use to ensure that devices inside and outside the communication network make correct policy decisions. [Means for solving the problem]

[0007] In the first aspect, a step of a first network side device sending first request information to a second network side device, the first request information being used to request the second network side device to determine model performance information of a target model; receiving, by the first network side device, model performance information of the target model transmitted from the second network side device; The present invention provides an information acquisition method, wherein the first request information includes target data and / or attribute information of the target data, the target data being used to determine model performance information of the target model, and the attribute information of the target data being used to acquire the target data.

[0008] In a second aspect, a first sending module used to send first request information to a second network side device, the first request information being used to request the second network side device to determine model performance information of a target model; a first receiving module used to receive model performance information of the target model transmitted from the second network side device; An information acquisition device is provided, wherein the first request information includes target data and / or attribute information of the target data, the target data being used to determine model performance information of the target model, and the attribute information of the target data being used to acquire the target data.

[0009] In a third aspect, a step of receiving first request information transmitted from the first network side device by the second network side device, the first request information including target data and / or attribute information of the target data, and the attribute information of the target data being used to acquire the target data; The second network side device determines model performance information of a target model based on the target data; the second network side device sending the model performance information to the first network side device.

[0010] In a fourth aspect, a second receiving module used to receive first request information transmitted from a first network side device, the first request information including target data and / or attribute information of the target data; a first determination module used to determine model performance information of a target model based on the target data; and a second sending module used to send the model performance information to the first network side device.

[0011] In a fifth aspect, receiving, by the third network side device, the second instruction information and the third instruction information transmitted from the first network side device; the third network side device storing the first data and / or information related to the first data according to the second instruction information; and collecting and storing second data and / or information related to the second data by the third network side device according to the third instruction information; The present invention provides an information acquisition method, wherein the first data and the second data are each part of target data, the target data is used to determine model performance information of a target model, and related information of the first data and related information of the second data are used to acquire the target data.

[0012] In a sixth aspect, a third receiving module used to receive the second instruction information and the third instruction information transmitted from the first network side device; a storage module used to store the first data and / or information related to the first data in accordance with the second instruction information, and to collect and store the second data and / or information related to the second data in accordance with the third instruction information; The present invention provides an information acquisition device, wherein the first data and the second data are each part of target data, the target data is used to determine model performance information of a target model, and related information of the first data and related information of the second data are used to acquire the target data.

[0013] In a seventh aspect, there is provided a network side device comprising a processor and a memory, wherein a program or command executable by the processor is stored in the memory, and when the program or command is executed by the processor, the steps of the information acquisition method described in the first aspect are realized, or the steps of the information acquisition method described in the third aspect are realized, or the steps of the information acquisition method described in the fifth aspect are realized.

[0014] In an eighth aspect, there is provided a readable storage medium having a program or command stored therein, the program or command being executed by a processor to realize the steps of the information acquisition method described in the first aspect, or the steps of the information acquisition method described in the third aspect, or the steps of the information acquisition method described in the fifth aspect.

[0015] In a ninth aspect, there is provided a computer program / program product stored on a storage medium, which, when executed by at least one processor, realizes the steps of the information acquisition method described in the first aspect, or the steps of the information acquisition method described in the third aspect, or the steps of the information acquisition method described in the fifth aspect. [Effects of the Invention]

[0016] In an embodiment of the present application, the first network side device sends first request information to the second network side device, and the first request information is used to request the second network side device to determine model performance information of the target model. The second network side device determines the model performance information of the target model based on the target data, and then sends the model performance information of the target model to the first network side device. In this way, the first network side device obtains the model performance information of the target model, and can further use the model performance information to determine the accuracy of the data analysis result in the actual use stage of the target model. Thus, when the data analysis result is highly accurate, it can be provided to devices inside and outside the communication network to assist their policy decision-making, and ensure that devices inside and outside the communication network make correct policy decisions. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram of a wireless communication system to which an embodiment of the present application can be applied. [Figure 2] 1 is a flowchart illustrating an information acquisition method according to an embodiment of the present application. [Figure 3] 1 is a flowchart of a specific example of information acquisition in an embodiment of the present application. [Figure 4] 10 is a flowchart of another specific example of information acquisition in an embodiment of the present application. [Figure 5] 3 is a schematic diagram of the configuration of an information acquisition device corresponding to FIG. 2 in an embodiment of the present application. [Figure 6] 1 is a flowchart illustrating another information acquisition method according to an embodiment of the present application; [Figure 7] 7 is a schematic diagram of the configuration of an information acquisition device corresponding to FIG. 6 in an embodiment of the present application. [Figure 8] 1 is a flowchart illustrating another information acquisition method according to an embodiment of the present application; [Figure 9] 9 is a schematic diagram of the configuration of an information acquisition device corresponding to FIG. 8 in an embodiment of the present application. [Figure 10] FIG. 2 is a schematic diagram illustrating the configuration of a network-side device in an embodiment of the present application. [Figure 11] FIG. 10 is a schematic diagram of another configuration of a network-side device in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, the technical solutions in the embodiments of the present application will be clearly explained with reference to the drawings in the embodiments of the present application, and it should be understood that the described embodiments are only a part of the embodiments of the present application, not all of the embodiments, and all other embodiments obtained by those skilled in the art based on the embodiments in the present application fall within the scope of protection of the present application.

[0019] The terms "first," "second," and the like used in the specification and claims of this application are not intended to describe a particular order or chronology, but rather to distinguish between similar objects. It should be understood that terms used in this manner may be interchanged where appropriate so that the embodiments of this application can be practiced in orders other than those illustrated or described herein. It should also be understood that the objects distinguished by "first" and "second" generally belong to a single category and do not limit the number of objects; for example, the first object may be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the " / " symbol generally indicates that the related objects before and after are in an "or" relationship.

[0020] It should be noted that the techniques described in the embodiments of this application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-Carrier Frequency Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in the embodiments of this application are often used interchangeably, and the techniques described can be used in other systems and wireless technologies in addition to those mentioned above. While the following description will discuss New Radio (NR) systems for illustrative purposes, and NR terminology is used in much of the following description, these technologies may be used in conjunction with other systems and wireless technologies, such as 6th Generation (6G) networks. th It can also be applied to applications other than NR system applications, such as 6G (Generation 6G) communication systems.

[0021] 1 is a block diagram of a wireless communication system to which the embodiment of the present application can be applied. The wireless communication system includes a terminal 11 and a network side device 12.

[0022] Here, the terminal 11 may be a terminal-side device such as a mobile phone, a tablet personal computer, a laptop computer (also called a notebook computer), a personal digital assistant (PDA), a personal digital assistant (PDA), a netbook, an ultra-mobile personal computer (UMPC), a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, a vehicle-mounted equipment (VUE), a pedestrian-mounted equipment (PUE), a smart home (a home device equipped with a wireless communication function such as a refrigerator, a television, a washing machine, or furniture), a game console, a personal computer (PC), an automated teller machine (ATM), or a kiosk. Wearable devices include a smart watch, a smart wristband, a smart earphone, a smart glasses, a smart accessory (a smart bracelet, a smart bracelet, a smart ring, a smart necklace, a smart anklet, a smart wristlet, a smart wear, etc.). It should be noted that the specific type of the terminal 11 is not limited in the embodiments of the present application.

[0023] The network side equipment 12 may include access network equipment or core network equipment.

[0024] Here, the access network equipment may be referred to as a radio access network equipment, a radio access network (RAN), a radio access network function, or a radio access network unit. The access network equipment may include a base station, a WLAN access point, a WiFi node, etc. The base station may be referred to as a Node B, an evolved Node B (eNB), an access point, a base transceiver station (BTS), a radio base station, a radio transceiver, a basic service set (BSS), an extended service set (ESS), a home B node, a home evolved B node, a transmitting receiving point (TRP), or any other appropriate term in the art. As long as the same technical effect can be achieved, the base station is not limited to a specific technical term. It should be noted that although the embodiments of this application only take a base station in an NR system as an example, the specific type of the base station is not limited.

[0025] Core network devices include core network nodes, core network functions, mobility management entities (MMEs), access and mobility management functions (AMFs), session management functions (SMFs), user plane functions (UPFs), policy control functions (PCFs), policy and charging rules functions (PCRFs), edge application server discovery functions (EASDFs), unified data management (UDMs), unified data repository (UDRs), home subscriber servers (HSSs), centralized network configuration (CNCs), network repository functions (NRFs), network exposure functions (NEFs), local NEFs (Local NEFs or L-NEFs), binding support functions (BSFs), and application functions (Application Node Functions). The present application may include, but is not limited to, at least one of the following: a core network device in an NR system, a QoS control function (QoS control function), ...

[0026] For ease of understanding, we first describe the use cases of the embodiments of this application, and the related technologies and concepts involved.

[0027] Embodiments of the present application can be applied to performing a target task using an artificial intelligence / machine learning model, such as a target model. The target model is a selected or trained model that can meet the needs of the target task. During actual use, corresponding input data can be input into the target model to obtain corresponding output data, i.e., a data analysis result corresponding to the target task can be obtained. The data analysis result may include predictive information for a future time period or time point, or may include summary information for historical data from a past time period or time point. The accuracy of the data analysis result can be determined based on model performance information of the target model during actual use.

[0028] As explained in the background art, the network data analysis function can perform intelligent data analysis for specific tasks to generate data analysis results, which can support policy decision-making by devices inside and outside the communication network. The network data analysis function can be divided into two network elements: the Analytics Logical Function (AnLF) and the Model Training Logical Function (MTLF).

[0029] Here, the analytical logic function can make inferences to generate predictive information or generate summaries of historical data, for example, to provide inferential analytical services based on consumer requests.

[0030] The model training logic function can generate models and train models, for example, to provide artificial intelligence / machine learning model training services based on consumer requests.

[0031] In a specific example, the interaction process between the analysis logic function (AnLF) and the model training logic function (MTLF) is as follows: The AnLF can request information about one or a set of ML models associated with one or a set of task identifiers from the MTLF by calling an ML model information request, e.g., Nnwdaf_MLModelInfo_Request, and the AnLF uses the ML models to perform data analysis. When the MTLF receives an ML model information request sent from the AnLF, it can determine whether an existing ML model can be used for the request or whether the existing ML model needs to be further trained. If further training is needed, the MTLF initiates a data collection operation to collect data from network-side devices such as a Network Function (NF), an Application Function (AF), or an Operation, Administration, and Maintenance (OAM) for training the ML model. The network function may be an AMF, a Data Collection Coordination Function (DCCF), an Analytics Data Repository Function (ADRF), etc. The MTLF can provide ML model information to the AnLF by invoking an ML model information request response, such as an Nnwdaf_MLModelInfo_Request response, where the ML model information includes information such as the file address of one or a set of ML models.

[0032] In another embodiment, the interaction process between the Network Data Analysis Function (NWDAF) and the Analysis Data Repository Function (ADRF) is as follows: The NWDAF can send data and / or analysis results to the ADRF by invoking a data management storage request, such as Nadrf_DataManagement_StorageRequest. The ADRF stores the data and / or analytical results transmitted from the NWDAF. The ADRF can determine, based on the data and / or analytical results transmitted from the NWDAF, whether it has already stored or is storing the same data and / or analytical results, and can determine not to store the data and / or analytical results transmitted from the NWDAF if the ADRF has already stored or is storing the same data and / or analytical results. The ADRF sends data and / or analysis result storage information to the NWDAF by invoking a data management storage request response, such as an Nadrf_DataManagement_StorageRequest response.

[0033] The above interaction process between the NWDAF and ADRF can also be applied to the interaction process between the DCCF, Messaging Framework Adaptor Function (MFAF), and ADRF.

[0034] As can be seen from the above description, MTLF is mainly used to generate AI / ML models, train the models, and provide the trained models to AnLF, while AnLF is used to actually use the models, perform inference to obtain predictive information, and generate summary information from historical data, while ADRF is mainly used for data storage. For any AI / ML model, the achievable accuracy of the model may differ at different stages, i.e., the accuracy the model can achieve at the training stage may differ from the accuracy the model can achieve at the actual use stage. The achievable accuracy of the model at different stages can differ due to factors such as different data distributions at different stages and insufficient model generalization ability. Generally, the achievable accuracy of the model at the actual use stage is lower than the achievable accuracy at the training stage.

[0035] Therefore, AnLF needs to know the model performance information at the stage of actual use of the model, and in this way, it can use the model performance information to determine the accuracy of the data analysis results at the stage of actual use of the model, and thereby provide the data analysis results to devices inside and outside the communication network when they are highly accurate to assist their policy decisions, ensuring that devices inside and outside the communication network make correct policy decisions.

[0036] The use cases of the embodiments of the present application and the related technologies and concepts involved have been described above. Below, we will explain in detail the information acquisition methods provided in the embodiments of the present application by several embodiments and their use cases with reference to the drawings.

[0037] As shown in FIG. 2, which is an implementation flowchart of the information acquisition method provided in the embodiment of the present application, the method may include the following steps S210 and S220.

[0038] S210: The first network side device sends first request information to the second network side device, the first request information is used to request the second network side device to determine model performance information of the target model, the first request information includes target data and / or attribute information of the target data, the target data is used to determine the model performance information of the target model, and the attribute information of the target data is used to obtain the target data.

[0039] In the embodiments of the present application, the first network-side device may be any network-side device that has obtained a target model and needs to use the target model to perform actual data analysis, such as AnLF, NWDAF, etc. The second network-side device may be any network-side device that can determine model performance information of the target model, such as MTLF, NWDAF, etc.

[0040] For example, a consumer network element such as an SMF, an AMF, an UPF, or a PCF can send a task request to a first network side device according to actual needs, where the task request is used to request the execution of a target task and obtain a data analysis result for the target task. After receiving the task request, the first network side device can obtain a target model, for example, can send a model request to a second network side device, where the model request is used to request the acquisition of a target model for executing the target task. The second network side device selects or trains a target model that meets the target task execution needs and then transmits it to the first network side device. After obtaining the target model, the first network side device can actually use the target model, and during actual use, can obtain a data analysis result corresponding to the target task based on the target model and corresponding input data.

[0041] The first network side device can send first request information to the second network side device when the set model performance information determination trigger condition is achieved during the actual use phase of the target model, and the first request information is used to request the second network side device to detect the model performance status of the target model and determine the model performance information of the target model.

[0042] The model performance information determination trigger condition can be set according to actual circumstances, for example, the model performance information determination trigger condition is considered to be achieved when a set time interval is reached, that is, the model performance status is detected periodically, or the model performance information determination trigger condition is considered to be achieved when a detection instruction of a consumer network element is received. When the model performance information determination trigger condition is achieved, it can trigger the first network side device to send first request information to the second network side device, and thus the second network side device detects the model performance status of the target model and determines the model performance information of the target model.

[0043] The first request information may include target data, or may include attribute information of the target data, or may include target data and attribute information of the target data, where the target data is used to determine model performance information of the target model, and the attribute information of the target data is used to obtain the target data. When the first request information includes the target data and attribute information of the target data, the attribute information of the target data is supplementary and descriptive to the target data.

[0044] After receiving the first request information sent from the first network side device, the second network side device can determine model performance information of the target model based on the target data, and then send the model performance information to the first network side device. If the first request information includes the target data, the second network side device can directly determine the model performance information of the target model based on the target data. If the first request information includes attribute information of the target data, the second network side device can obtain the target data according to the attribute information of the target data before determining the model performance information of the target model based on the target data.

[0045] The target data and / or attribute information of the target data included in the first request information may specifically be related information of a first partial data of the target data and a second partial data of the target data, where the first partial data and the second partial data are each part of the target data, and the related information of the first partial data is used to acquire the first partial data. The combination of the first partial data and the second partial data can constitute the target data. That is, the first network side device transmits the related information of the first partial data and the second partial data to the second network side device. The second network side device can acquire the first partial data using the related information of the first partial data. For example, the second network side device can determine a third network side device that stores the first partial data using the related information of the first partial data, acquire the first partial data from the third network side device, and combine the acquired first partial data with the second partial data received from the first network side device to acquire the target data.

[0046] In the embodiments of the present application, the target data is relevant data corresponding to the target model in the actual use stage, and may include input data corresponding to the target model, output data corresponding to the target model, label data corresponding to the target model, etc. The relevant data corresponding to the target model when training the model before actually using it is training data. Of course, when the target model needs to be retrained after actually using it, the training data corresponding to the retraining stage may include target data corresponding to the previous actual use stage.

[0047] S220: The first network side device receives model performance information of the target model transmitted from the second network side device.

[0048] The first network side device can send first request information to the second network side device to request the second network side device to detect the model performance status of the target model and determine model performance information, and then the second network side device can send the model performance information of the target model to the first network side device. After receiving the model performance information of the target model sent from the second network side device, the first network side device can know the model performance status of the target model in the actual use stage.

[0049] After obtaining the model performance information of the target model, the first network side device can determine the accuracy of the target model during actual use based on the model performance information of the target model. If the accuracy of the target model during actual use is low, the accuracy of the data analysis results obtained using the target model is considered to be low, and the first network side device can stop feeding back the data analysis results to the consumer network element to avoid the consumer network element making an incorrect policy decision or performing an inappropriate operation based on the data analysis results with low accuracy. If the accuracy of the target model during actual use is high, the accuracy of the data analysis results obtained using the target model is considered to be high, and the first network side device can continue feeding back the data analysis results to the consumer network element to ensure the consumer network element making a correct policy decision or performing an appropriate operation based on the data analysis results with high accuracy.

[0050] Alternatively, when the accuracy of the target model is low during actual use, the first network side device can send a model retraining request or a model reselection request to the second network side device to request the second network side device to retrain the target model or perform model reselection to improve the accuracy of the model.

[0051] According to the method provided in the embodiments of the present application, the first network side device sends first request information to the second network side device, and the first request information is used to request the second network side device to detect the model performance status of the target model. The second network side device detects the model performance status of the target model based on the target data, determines model performance information, and then sends the model performance information of the target model to the first network side device. In this way, the first network side device obtains the model performance information of the target model, and can further use the model performance information to determine the accuracy of the data analysis result in the actual use stage of the target model. Therefore, when the data analysis result is highly accurate, it can be provided to devices inside and outside the communication network to assist their policy decision-making, and ensure that the devices inside and outside the communication network make correct policy decisions.

[0052] In one embodiment of the present application, the first request information comprises: 1st instruction information, Model performance desired information, Model monitoring time requirement information, The model may further include at least one of the following: The first instruction information is used to instruct the second network side device to determine model performance information of the target model; The desired model performance information is used to represent model performance information of a target model desired by the first network side device; The model monitoring time request information is used to instruct the second network side device on the time range for determining the model performance information of the target model.

[0053] In an embodiment of the present application, the first request information may further include at least one of the following (1) to (4): (1) First instruction information. The first instruction information is used to instruct the second network side device to determine model performance information of the target model. After receiving the first request information sent from the first network side device, the second network side device can know that the first network side device requests to determine model performance information of the target model from the first instruction information included in the first request information, and can further perform corresponding operations. The first indication information may be explicit, for example, carried by a specific cell, or the first indication information may be implicit, for example, expressed by the name of a message carrying the first request information. (2) Model performance preference information. The model performance preference information is used to represent the model performance information of the target model desired by the first network side device. The model performance preference may be a threshold, a preset condition, etc. The first network side device can inform the second network side device of the model performance information of the target model that it desires and can accept through the model performance preference information. After the second network side device determines the model performance information of the target model, if the model performance information does not meet the model performance preference corresponding to the model performance preference information, it can trigger subsequent operations such as retraining the model, reselecting the model, or notifying the first network side device that the model performance has deteriorated and does not meet expectations. Of course, the desired model performance information may be included in the model acquisition request sent from the first network side device to the second network side device. (3) Model monitoring time request information. The model monitoring time request information is used to instruct the second network side device on the time range for determining the model performance information of the target model. The first network side device can use the model monitoring time request information to inform the second network side device of the time range within which the model performance status of the target model should be monitored. For example, if the model monitoring time request information specifies one month, the second network side device needs to periodically acquire relevant data to determine the model performance information within the next month. Furthermore, for example, if the first request information does not include the model monitoring time request information or specifies "immediately," the second network side device can acquire target data after receiving the first request information and determine the model performance information based on the target data. This determination is made at once, and the second network side device does not need to collect data to monitor the model over a long period of time. (4) Model monitoring area request information. The model monitoring area request information is used to instruct the second network side device on the area range in which to determine the model performance information of the target model. The first network side device can use the model monitoring area request information to inform the second network side device of the area range within which to monitor the model performance status of the target model. For example, when a cell is specified in the model monitoring area request information, the second network side device can obtain related data within the cell range and determine the model performance information.

[0054] The first request information includes at least one of the above information, and the above information can constrain the second network side device to determine the model performance information of the target model, so as to better meet the demand of the first network side device.

[0055] In one embodiment of the present application, the attribute information of the target data is: task information corresponding to the goal data; Information about the model that is the target of the target data; data description information corresponding to the target data; It may include at least one of the stored information corresponding to the target data.

[0056] That is, the attribute information of the target data may include one or more of task information corresponding to the target data, information on the model that is the subject of the target data, data description information corresponding to the target data, and storage information corresponding to the target data, and the target data can be obtained by the attribute information of the target data.

[0057] Here, the task information corresponding to the goal data may include at least one of the following (1) to (3). (1) Identifier information of the target task corresponding to the target data, for example, Analytics ID. The identifier information of the target task corresponding to the target data can indicate which specific task the target data corresponds to. (2) Target task condition restriction information, such as analytic filter information. The target task condition restriction information is used to express the filtering conditions for the data analysis results corresponding to the target task. For example, the data analysis results are filtered based on the Area of ​​Interest (AOI), Single Network Slice Selection Assistance information (S-NSSAI), Data Network Name (DNN), etc. (3) Target information for a target task. For example, Target of analytic reporting. The target information for a target task can indicate whether the target task is to be executed and data analysis is to be performed on one device, multiple devices, or all devices.

[0058] The information on the model that is the target of the target data may include at least one of the following (1) and (2). (1) Identifier information of the model that is the target of the target data. For example, Model ID. The identifier information of the model that is the target of the target data can indicate which model the target data is for. In the embodiment of the present application, the model that is the target of the target data is the target model. (2) Filtering information of the model that is the target of the target data. The filtering information of the model that is the target of the target data can specify the filtering conditions that the model that is the target of the target data should satisfy, such as AOI, S-NSSAI, DNN, etc.

[0059] The data description information corresponding to the target data may include at least one of the following (1) to (3). (1) Data Category Information of Target Data The data category information of the target data can indicate which data category of target data to acquire, for example, acquire target data of the terminal location information category. (2) Time range information corresponding to target data. The time range information corresponding to target data is used to indicate the time range in which the time information corresponding to the target data exists. The time range information corresponding to the target data can indicate within which time range the data should be acquired. The time range information may include a start time point and an end time point. For example, the time information corresponding to the data is the generation time, and the time range information can indicate that data to be generated between the start time point and the end time point should be acquired. (3) Location range information corresponding to target data. The location range information corresponding to target data is used to indicate the location range in which the location information corresponding to the target data exists. The location range information corresponding to target data can indicate within which location range the data should be acquired. For example, the location information corresponding to data is the location that exists when the data is generated, and the location range information can indicate that data located within the location range at the time of generation should be acquired. For example, data located within a certain tracking area (TA) or cell at the time of generation can be acquired.

[0060] The stored information corresponding to the target data may include at least one of the following (1) to (4). (1) Identifier information of the third network side device that stores the target data. For example, the ID of the third network side device. The identifier information of the third network side device that stores the target data can indicate in which device the target data is stored. (2) Address information of the device on the third network side. For example, the IP address or fully qualified domain name (FQDN) of the device on the third network side. The address information of the device on the third network side can instruct how to establish a connection with the device on the third network side. (3) Storage identifier information of the target data. For example, a Storage Transaction Identifier. The storage identifier information of the target data can indicate how to search for the target data. Specifically, the storage identifier information may be information such as an index. Different storage methods may have different corresponding storage identifier information. (4) Identifier information of the first network side device, for example, the ID of the first network side device.

[0061] To obtain the target data more accurately, the identifier information of the first network side device and the stored identifier information of the target data can be used in combination.

[0062] The first request information sent by the first network side device to the second network side device includes attribute information of the target data, and the attribute information of the target data includes one or more of the above information. In this way, the second network side device can accurately obtain the target data according to the attribute information of the target data, and determine the model performance information of the target model based on the target data, thereby preferably meeting the needs of the first network side device to detect the model performance status.

[0063] It should be noted that model performance information of a target model can be used to indicate the degree of accuracy and / or error of an inference result during actual inference. The model performance information may be expressed in a variety of ways, such as a specific percentage value such as 90%, a categorical expression such as high, medium, or low, or normalized data such as 0.9. The embodiments of the present application do not specifically limit the expression form of the model performance information. The model performance information of a target model may directly or indirectly indicate the degree of accuracy or error of the target task inference result by the target model. For example, the inference accuracy of the target model can be indirectly indicated by calculating the inference error or inference error rate of the target model. Here, the inference error or inference error rate may be calculated in a variety of ways, such as mean absolute error (MAE), mean square error (MSE), etc.

[0064] It should also be noted that when the first request information includes related information of the first partial data and the second partial data, the related information of the first partial data can refer to attribute information of the target data, and for example, the related information of the first partial data may include at least one of task information corresponding to the first partial data, information on the model that is the target of the first partial data, data description information corresponding to the first partial data, and storage information corresponding to the first partial data, and in this way the second network side device can accurately obtain the first partial data using the related information of the first partial data.

[0065] In one embodiment of the present application, when the first request information includes attribute information of the target data, before the first network side device sends the first request information to the second network side device, the method includes: Step 1: a step in which a first network side device acquires first data, the first data being a part or all of target data; The method may further include step 2 of the first network side device sending second instruction information to the third network side device, wherein the second instruction information includes the first data and / or information related to the first data and is used to instruct the third network side device to store the first data and / or information related to the first data.

[0066] For ease of explanation, the above two steps will be combined.

[0067] In an embodiment of the present application, the first request information may include attribute information of the target data, and the first network side device may first obtain the first data before sending the first request information to the second network side device, for example, by determining a data source network element based on task information and obtaining the first data through the data source network element. The first data is part or all of the target data. After obtaining the first data, the first network side device may send second instruction information to the third network side device, the second instruction information including the first data and / or information related to the first data.

[0068] The second indication may be explicit, for example carried by a specific cell, or the second indication may be implicit, for example expressed by the name of the message carrying the information.

[0069] The third network side device may be any network side device capable of storing data, for example, an ADRF. The second instruction information sent by the first network side device to the third network side device is used to instruct the third network side device to store the first data and / or information related to the first data. The third network side device can store the first data and / or information related to the first data using the second instruction information.

[0070] When the first data is the entire target data, it means that the entire target data is acquired by the first network side device, and then stored in the third network side device, and when the second network side device needs to acquire the target data, it can be acquired by interacting with the third network side device.

[0071] When the first data is a part of the target data, it means that a part of the target data is acquired by the first network side device, and this part of the data is instructed to be stored in the third network side device, and the other part of the target data is acquired by other means, for example, by collecting and acquiring it by the third network side device, and the third network side device collects and stores the other part of the target data, and thus the target data is stored in the third network side device.When the second network side device needs to acquire the target data, it can acquire it by interacting with the third network side device.Furthermore, for example, it can be acquired by collecting and acquiring it by the second network side device.

[0072] In one embodiment of the present application, the method comprises: The method may further include the step of: the first network side device receiving first storage feedback information sent from the third network side device; The first storage feedback information includes storage identifier information corresponding to the first data.

[0073] In an embodiment of the present application, the first network side device obtains the first data and sends second indication information to the third network side device. The third network side device can store the first data and / or information related to the first data, and after storing, return first storage feedback information to the first network side device, where the first storage feedback information may include storage identifier information corresponding to the first data. Of course, if the third network side device determines that the first data and / or information related to the first data has already been stored after receiving the second indication information, it can directly return the first storage feedback information to the first network side device.

[0074] After receiving the first storage feedback information sent from the third network side device, the first network side device can know that the third network side device has already completed storing the first data and / or information related to the first data, and can obtain corresponding storage identifier information, thereby including the storage identifier information in the attribute information of the target data included in the first request information sent to the second network side device, thereby facilitating the second network side device to obtain the target data.

[0075] In one embodiment of the present application, the first data comprises: input data corresponding to the target model; output data corresponding to the target model, may include at least one of label data corresponding to the target model; The label data corresponding to the target model may include actual generated data associated with the input data and / or output data corresponding to the target model.

[0076] In an embodiment of the present application, the first data may include at least one of input data corresponding to the target model, output data corresponding to the target model, and label data corresponding to the target model. The input data corresponding to the target model is input data corresponding to the target model in an actual use stage, the output data corresponding to the target model is output data obtained by the target model performing data analysis on the input data in an actual use stage, and the label data corresponding to the target model is actually generated data associated with the input data and / or output data corresponding to the target model, or data labeled based on the actually generated data.

[0077] For example, the input data corresponding to the target model is terminal location data of a certain cell in a certain time period, and the target task corresponding to the target model is to predict the number of terminals in the cell within one week in the future. When the input data is input into the target model, corresponding output data can be obtained, which is the predicted information of the number of terminals in the cell within one week in the future. After one week, the actual number of terminals in the cell can be obtained, and this actual number of terminals can be used as label data corresponding to the target model, and the label data has a correlation with the corresponding input data and output data.

[0078] In one embodiment of the present application, the related information of the first data is: task information corresponding to the first data; Information about the model that is the subject of the first data; The first data may include at least one piece of time information corresponding to the first data.

[0079] Here, the task information corresponding to the first data may include at least one of the following (1) to (3). (1) Identifier information of a target task corresponding to the first data: The identifier information of a target task corresponding to the first data can indicate which specific task the first data corresponds to. (2) Condition restriction information of the target task: The condition restriction information of the target task is used to represent filtering conditions for the data analysis results corresponding to the target task. (3) Target information for the target task. The target information for the target task can indicate whether the target for executing the target task and performing data analysis is a certain terminal, multiple terminals, or all terminals.

[0080] The information on the model that is the target of the first data may include at least one of the following (1) and (2). (1) Identifier information of the model that is the target of the first data. The identifier information of the model that is the target of the first data can indicate which model the first data is for. In the embodiment of the present application, the model that is the target of the first data is the target model. (2) Filtering information of the model that is the target of the first data: The filtering information of the model that is the target of the first data can specify the filtering conditions that the model that is the target of the first data should satisfy.

[0081] The time information corresponding to the first data may include at least one of the following (1) to (3). (1) Time information corresponding to the input data. Specifically, the time information corresponding to the input data may include at least one of time point information when the input data is generated, time point information when the input data is acquired by the first network side device, and time point information when the input data is input to the target model. (2) Time information corresponding to the output data. Specifically, the time information corresponding to the output data may include at least one of time point information at the time of generating the output data, target time point information for the output data, and time information corresponding to input data associated with the output data. (3) Time information corresponding to the label data. Specifically, the time information corresponding to the label data may include at least one of time point information when the label data is generated and time point information when the label data is acquired by the first network side device.

[0082] Here, the time point when the output data is generated may be the same as the time point when the input data is input to the target model, or may be the time point when the input data is input to the target model and the target model calculates and then generates the output data.

[0083] The target time point for the output data may be a time point corresponding to the forecast information included in the output data. For example, if the forecast information included in the output data is the number of terminals in a certain cell one week from now, for example, on X day, the target time point is X day one week from now. Corresponding label data is generated at the target time point. The output data and the label data can be associated based on the target time point for the output data and the time point when the label data was generated.

[0084] The time information corresponding to the input data associated with the output data may include at least one of time point information when the input data associated with the output data is generated, time point information when the input data associated with the output data is acquired by the first network side device, and time point information when the input data associated with the output data is input to the target model. The output data and the input data can be associated with each other by the time information corresponding to the input data associated with the output data.

[0085] The time information corresponding to the input data, output data, and label data can assist in determining subsequent model performance information. For example, by comparing the time information corresponding to the output data with the time information corresponding to the label data, the output data corresponding to the target model and the label data corresponding to the output data can be associated / bound / mapped to compare / calculate / generate model performance information. Furthermore, for example, the time information corresponding to the input data corresponding to the target model can be used to determine the time range information, i.e., a predetermined time range, covered by these data. For example, if the time information corresponding to the input data indicates that the earliest time is Monday of a certain week and the latest time is Friday of that week, this means that the time range covered by these data is from Monday to Friday of that week. The time range can be used to record and ensure the timeliness of the model performance information.

[0086] In one embodiment of the present application, when the first request information includes attribute information of the target data, before the step of the first network side device sending the first request information to the second network side device, the method includes: The method may further include a step in which the first network side device transmits to the third network side device third instruction information used to instruct the third network side device to collect and store second data that is part or all of the target data and / or information related to the second data.

[0087] In an embodiment of the present application, when the first request information includes attribute information of the target data, before the first network side device transmits the first request information to the second network side device, the first network side device can transmit third instruction information to the third network side device, the third instruction information being used to instruct the third network side device to collect and store the second data and / or information related to the second data. The second data may be a part or all of the target data.

[0088] When the second data is the whole of the target data, the third network side device can collect and store the whole of the target data according to the third instruction information, and can store information related to the second data, and can refer to the attribute information of the target data for the information related to the second data. The second network side device can obtain the whole of the target data through the third network side device.

[0089] When the second data is a part of the target data, the third network side device can collect and store a part of the target data and store information related to the second data according to the third instruction information. The second network side device can obtain a part of the target data through the third network side device, and the other part of the target data can be collected and obtained by the second network side device.

[0090] The first data and the second data are each part of the target data, and when the first data and the second data are different, the combination of the first data and the second data can constitute the target data. In this way, after the first network side device obtains the first data, it can send second instruction information to the third network side device to instruct the third network side device to store the first data and / or information related to the first data, and further send third instruction information to the third network side device to instruct the third network side device to collect and store the second data and / or information related to the second data. The third instruction information and the second instruction information can be transmitted in the same message or in different messages.

[0091] The first network side device and the third network side device exchange part of the target data, and the other part of the data is collected and obtained by the third network side device, thereby contributing to reducing the amount of data exchanged and saving network resources.

[0092] In one embodiment of the present application, the method comprises: The method may further include the first network side device receiving second storage feedback information sent from the third network side device; The second storage feedback information includes storage identifier information corresponding to the second data.

[0093] The third network side device can collect and store the second data and / or information related to the second data, and then transmit second storage feedback information to the first network side device. After receiving the second storage feedback information transmitted from the third network side device, the first network side device can know the storage identifier information corresponding to the second data, and thereby include the storage identifier information in the attribute information of the target data contained in the first request information transmitted to the second network side device, thereby facilitating the second network side device to acquire the target data.

[0094] The first storage feedback information and the second storage feedback information may be carried in the same message or may be transmitted in different messages.

[0095] In one embodiment of the present application, the related information of the second data is: task information corresponding to the second data; Information about the model that is the subject of the second data; The data may include at least one of the data description information corresponding to the second data.

[0096] The related information of the second data is used by the third network side device to collect the second data, and the related information of the second data can refer to the attribute information of the target data or the related information of the first data, and detailed explanations are omitted here.

[0097] In one embodiment of the present application, the third instruction information may include information on the device where the second data exists and / or collection request information corresponding to the second data. The information on the device where the second data exists can instruct the third network device where to collect the second data, and the collection request information corresponding to the second data can instruct the third network device on a specific request for collecting the second data.

[0098] Here, the information about the device in which the second data exists may include at least one of the following (1) and (2). (1) Identifier information of the device in which the second data exists, such as the ID of the device in which the second data exists. (2) Address information of the device where the second data exists, such as the IP address or FQDN of the device where the second data exists.

[0099] The collection request information corresponding to the second data may include at least one of the following (1) to (4). (1) Data category information of the second data: The data category information of the second data can indicate which data category of the second data to collect, for example, collecting second data of the terminal location information category. (2) Time range information corresponding to the second data. The time range information corresponding to the second data is used to indicate the time range in which the time information corresponding to the second data exists. The time range information corresponding to the second data can indicate within which time range the data is to be collected. The time range information can include a start time point and an end time point. For example, the time information corresponding to the data is a generation time, and the time range information can indicate that data generated between the start time point and the end time point is to be collected. (3) Location range information corresponding to the second data. The location range information corresponding to the second data is used to indicate the location range in which the location information corresponding to the second data exists. The location range information corresponding to the second data can indicate within which location range the data should be collected. For example, the location information of the data is the location at the time the data is generated, and the location range information can indicate that data located within the location range at the time of generation should be collected. For example, data located within a certain tracking area or cell at the time of generation can be collected. (4) Collection time information of second data. The collection time information of second data is used to indicate a time request for collecting the second data. The collection time information of second data can indicate within what time range the second data is to be collected. For example, if the collection time is one month in the future, the third network side device needs to periodically collect the second data within one month in the future. Furthermore, for example, if the collection time is one month in the past, the third network side device needs to collect the second data within the previous month.

[0100] The third instruction information includes information on the device where the second data exists and / or collection request information corresponding to the second data so that the third network side device can collect and store the second data and / or information related to the second data in a timely manner.

[0101] In one embodiment of the present application, the first data may include input data corresponding to the target model and / or output data corresponding to the target model, and the second data may include label data corresponding to the target model. That is, the first network side device can collect input data corresponding to the target model and obtain corresponding output data based on the target model and the input data, and then send second instruction information to a third network side device to instruct the third network side device to store the first data and / or information related to the first data, including the input data and output data, and simultaneously send third instruction information to the third network side device to instruct the third network side device to collect and store second data and / or information related to the second data, including label data. After obtaining the first data and the second data from the third network side device, the second network side device can detect model performance status of the target model by using the output data included in the first data and the label data included in the second data to determine model performance information, or can obtain corresponding output data by using the input data included in the first data, and then detect model performance status of the target model by using the output data and label data to determine model performance information.

[0102] Alternatively, the first data may include input data corresponding to the target model, and the second data may include label data corresponding to the target model. That is, the first network side device can collect input data corresponding to the target model, and then send second instruction information to the third network side device to instruct the third network side device to store the first data including the input data and / or information related to the first data, and simultaneously send third instruction information to the third network side device to instruct the third network side device to collect and store the second data and / or information related to the second data including label data. The second network side device can obtain the first data and the second data from the third network side device, and then obtain output data corresponding to the target model using the input data included in the first data, and then detect the model performance status of the target model using the output data and the label data.

[0103] Alternatively, the first data may include output data corresponding to the target model and label data corresponding to the target model, and the second data may include input data corresponding to the target model.

[0104] Alternatively, the first data may include label data corresponding to the target model, and the second data may include input data corresponding to the target model.

[0105] In either case, it is sufficient that the second network side device can finally obtain label data associated with the output data corresponding to the target model. In this way, the second network side device can detect the model performance status of the target model based on the output data and the corresponding label data and determine model performance information.

[0106] In one embodiment of the present application, the step of the second network side device acquiring the target data according to the attribute information of the target data includes: a first step in which a second network side device determines a third network side device that stores the target data based on attribute information of the target data; a second step in which the second network side device transmits second request information to the third network side device, the second request information being used to request acquisition of target data; The method may further include a third step of receiving, by the second network side device, the target data transmitted from the third network side device.

[0107] To make it easier to explain, we will combine the above three steps.

[0108] In an embodiment of the present application, the first network side device can store target data and / or attribute information of the target data in the third network side device by communicating with the third network side device. When the first network side device needs to detect the model performance status of the target model, it can send first request information to the second network side device. The second network side device can first determine the third network side device that stores the target data based on the attribute information of the target data included in the first request information, and then request the third network side device to obtain the target data by sending second request information. The third network side device can return corresponding target data based on the second request information. After receiving the target data sent from the third network side device, the second network side device can detect the model performance status of the target model based on the target data and determine model performance information.

[0109] The second request information may include at least one of the following (1) to (5). (1) Identifier information of the first network side device, for example, the ID of the first network side device. (2) Memory identifier information of target data. When the target data is all data transmitted from the first network side device to the third network side device and stored therein, there may be one memory identifier information of the target data. When the target data includes data transmitted from the first network side device to the third network side device and stored therein, and data that the first network side device instructs the third network side device to collect and store therein, there may be multiple memory identifier information of the target data, each of which is used to represent data stored at a different time. (3) Task information corresponding to the goal data. (4) Information about the model that is the target of the target data. (5) Data description information corresponding to the target data.

[0110] According to the above information, the second network side device can clarify the target data it wants to obtain, and thus the third network side device can provide target data that meets the request.

[0111] In one embodiment of the present application, after the step of the second network side device determining model performance information of the target model based on the target data, the method includes: The method may further include a step of determining, by the second network side device, whether the model performance information satisfies the model performance preference corresponding to the model performance preference information; The step of transmitting model performance information from the second network side device to the first network side device includes: The method may include a step of the second network side device transmitting the model performance information to the first network side device when the model performance information meets or does not meet the model performance expectation.

[0112] In an embodiment of the present application, the first network side device sends first request information to the second network side device, and the second network side device detects the model performance status of the target model based on the target data to determine model performance information, which may include information such as model accuracy rate, error rate, etc.

[0113] The first request information may include model performance preference information, and the second network side device can determine the model performance information and then further determine whether the model performance information satisfies the model performance preference corresponding to the model performance preference information. For example, when the model performance preference is a model accuracy rate higher than a first threshold, if the model accuracy rate included in the model performance information is higher than the first threshold, it means that the model performance information satisfies the model performance preference corresponding to the model performance preference information; conversely, it means that the model performance information does not satisfy the model performance preference corresponding to the model performance preference information.

[0114] If the model performance information meets the desired model performance, the second network side device can send the model performance information to the first network side device to inform the first network side device that the current model performance of the target model is good and the target model can continue to be used.

[0115] If the model performance information does not meet the desired model performance, the second network side device can further transmit the model performance information to the first network side device to inform the first network side device that the model performance of the current target model has deteriorated and continued use may result in inaccurate data analysis results. The first network side device can determine whether to continue using the target model according to the actual situation.

[0116] In one embodiment of the present application, the method comprises: The method may further include the step of the second network side device retraining the target model or performing model reselection when the model performance information does not meet the desired model performance.

[0117] In an embodiment of the present application, after determining the model performance information, the second network side device can further determine whether the model performance information meets the model performance expectations corresponding to the model performance expectations information. If the model performance information does not meet the model performance expectations, the second network side device can retrain the target model and provide the retrained target model to the first network side device for actual use. Alternatively, the second network side device can reselect a model and provide the reselected model to the first network side device for actual use. This contributes to improving the accuracy rate of the model when it is actually used.

[0118] In one embodiment of the present application, the method comprises: Step 1: the first network side device sends fourth instruction information to the second network side device according to the model performance information, the fourth instruction information is used to instruct the second network side device to retrain the target model or perform model reselection; The method may further include step 2, in which the first network side device receives the retrained model or the reselected model sent from the second network side device.

[0119] For ease of explanation, the above two steps will be combined.

[0120] After the second network side device determines the model performance information and sends the model performance information to the first network side device, the first network side device can send fourth instruction information to the second network side device according to the model performance information. For example, when it determines that it is necessary to retrain the target model or perform model reselection according to the actual situation, it sends the fourth instruction information to the second network side device. The fourth instruction information is used to instruct the second network side device to retrain the target model or perform model reselection.

[0121] The second network side device can retrain the target model or reselect the model according to the fourth instruction information, and after retraining or reselecting, send the retrained model or reselected model to the first network side device.

[0122] The first network side device can actually use the retrained model or reselected model after receiving it from the second network side device.

[0123] Retraining the target model or performing model reselection can improve the accuracy of the model during actual use.

[0124] In one embodiment of the present application, the target data may include label data corresponding to the target model, and input data and / or output data corresponding to the target model, and the step of the second network side device determining model performance information of the target model based on the target data may include: Step 1: a second network side device compares each output data with associated label data to obtain a comparison result; The method may further include a step 2 in which the second network side device determines model performance information of the target model according to the comparison result.

[0125] For ease of explanation, the above two steps will be combined.

[0126] In embodiments of the present application, the target data may include label data corresponding to the target model, and input data and / or output data corresponding to the target model, that is, the target data may include label data corresponding to the target model and input data corresponding to the target model, or the target data may include label data corresponding to the target model and output data corresponding to the target model, or the target data may include label data corresponding to the target model, input data corresponding to the target model, and output data corresponding to the target model.

[0127] After the second network side device acquires the target data, if the target data includes label data corresponding to the target model and input data corresponding to the target model, the second network side device can obtain output data corresponding to the target model based on the target model and the input data corresponding to the target model. If the target data includes label data corresponding to the target model and output data corresponding to the target model, or if the target data includes label data corresponding to the target model, input data corresponding to the target model, and output data corresponding to the target model, after the second network side device acquires the target data, that is, output data corresponding to the target model is obtained.

[0128] The correlation between the output data and the label data corresponding to the target model can be determined based on the time information of the output data and the label data corresponding to the target model.

[0129] The second network side device can compare each output data with the associated label data to obtain a comparison result. The comparison result can determine the accuracy rate of each output data compared with the associated label data, thereby determining model performance information of the target model. The second network side device feeds back the model performance information to the first network side device, and the first network side device can determine further operations based on the model performance information.

[0130] For ease of understanding, the following will again describe the embodiments of the present application using the specific example shown in Figure 3, taking as an example the first network side device is an AnLF, the second network side device is an MTLF, and the third network side device is an ADRF.

[0131] Step 0: This is the flow related to task initiation, model selection, and model transmission. Specifically, the consumer network element (NF) initiates a task request to the AnLF, requesting to obtain the data analysis results of the target task. The AnLF then sends a model request to the MTLF, requesting to obtain a target model for executing the target task. The MTLF then selects or trains a target model that meets the target task execution needs and transmits it to the AnLF.

[0132] Step 1~Step 5: The AnLF sends first request information to the MTLF to request the MTLF to assist in detecting the model performance status. A specific scenario is when, after receiving the target model feedback from the MTLF, the AnLF needs to know more about the model performance status in a real-time environment, such as when using real-time data or in actual use, to determine whether to continue using the target model for data analysis. Before sending the first request information to the MTLF, the AnLF can first collect and store the required target data in the ADRF and send the address of the ADRF to the MTLF. The MTLF can then obtain the required target data from the designated ADRF.

[0133] The target data may include input data, output data, and label / label data when the target model is actually used. Input data is input data to the target model during the target model usage / inference process and used for inference of the target model. Output data may be a prediction, inference data, or inference result data, i.e., output data / prediction value / inference data, etc. obtained after inputting a set of input data into the target model and performing calculation / prediction / inference. Label data may be understood as actual measurements in a real environment and may include a ground truth, i.e., a correct labeling value. Generally, a set of data may include a set of input data, output data corresponding to the input data, and label data corresponding to the input data.

[0134] Step 1: The AnLF collects relevant data from the data source network element. The AnLF can determine the target data source network element according to the task information. The task information can be included in the task request sent by the consumer network element in step 0. Next, the AnLF obtains relevant data from the target data source network element. Here, the relevant data obtained in this step may include input data and / or label data. After collecting the input data, the AnLF can input the input data into the target model, and obtain corresponding output data through the target model's calculation.

[0135] Step 2: The AnLF sends second instruction information to the ADRF to store the first data in the ADRF. Specifically, the AnLF sends the first data (the above-mentioned input data, output data, and label data) and related information of the first data, such as task information corresponding to the first data, model information corresponding to the first data, and time information corresponding to the first data, to the ADRF using the second instruction information, and stores the first data and / or related information of the first data in the ADRF.

[0136] The time information corresponding to the first data can assist in detecting subsequent model performance conditions. For example, by comparing the time information corresponding to the output data with the time information corresponding to the label data, the output data of the target model can be associated / bound / mapped to the label data corresponding to the output data, thereby comparing / calculating / generating model performance information. Furthermore, for example, the time information corresponding to the input data can provide information on the time range covered by these data (referring to a predetermined time range; for example, if the time information corresponding to the input data indicates that the earliest time is Monday of a certain week and the latest time is Friday of that week, this means that the time range covered by these data is Monday to Friday of a certain week), which can be used to record and ensure the timeliness of the model performance information.

[0137] In one implementation, the second indication information can be sent by Nadrf_DataManagement_StorageRequest.

[0138] Specifically, the second instruction information sent by the AnLF to the ADRF may include at least one of the following: Input data; Output data; Label data; Time information corresponding to the input data, which may include time point information when the input data is generated, time point information when the input data is acquired by the AnLF, and time point information when the input data is input to the target model. Time information corresponding to output data. The time information corresponding to the output data may include time point information when the output data is generated, and the time point information may be obtained from the time point when the input data is input to the target model, and may also be obtained from the time point when the output data is generated after the input data is input to the target model and calculated by the target model. The time information corresponding to the output data may further include target time point information for the output data, and the target time point information may be obtained from the target time point corresponding to the task to be predicted by the output data. The time information corresponding to the output data may further include an initial time point corresponding to the output data, and the initial time point may be obtained from time information corresponding to the input data associated with the output data. Time information corresponding to the label data. This may include time point information when the label data was generated and time point information when the label data was acquired by AnLF. Target task identifier information, such as an Analytics ID, is used to indicate that the first data is for a specific task. Conditional information for the target task, such as analytical filter information, is used to indicate filtering information for data analysis results, including AOI, S-NSSAI, DNN, etc. Target information for the target task. For example, Target of analytic reporting, which is used to indicate that data analysis is targeted at a device, multiple devices, or all devices. Model identifier information, such as a Model ID, is used to indicate that the first data is for a certain model. Model filtering information, which is used to indicate the conditions that the model that is the target of the first data must satisfy, such as AOI, S-NSSAI, or DNN.

[0139] Step 3: The ADRF stores the received first data and / or information related to the first data.

[0140] Step 4: After the ADRF completes the storage, it returns storage completion indication information, such as first storage feedback information, and feeds back the storage identifier information of this storage, such as Transaction Reference ID, to the AnLF to inform it that the storage is successful.

[0141] Step 5: The AnLF sends first request information to the MTLF, requesting the MTLF to assist in detecting the model performance status of the target model and determining model performance information. In this process, the first request information may specifically include at least one of the following: First instruction information, which is used to instruct the MTLF to determine model performance information, may be explicit, e.g., carried by a specific cell, or implicit, e.g., represented by the name of a message carrying the first request information. Model performance preference information, i.e., preference for model performance information, such as a threshold, a preset condition, etc., is used to inform the MTLF of the model performance information it can accept. If the model performance information calculated by the MTLF does not meet the model performance preference, it will trigger subsequent operations such as retraining the model, reselecting the model, or notifying the AnLF that the model performance has deteriorated and does not meet expectations. The model performance preference information may be notified to the MTLF when the AnLF sends a model request to the MTLF in step 0. Identifier information of the ADRF. Specifically, it may be identifier information of a network element, such as a network element ID. Address information of the ADRF. Specifically, it may be address information of the network element, such as the IP address or FQDN of the network element. Identifier information of the AnLF. Specifically, it may be identifier information of a network element, such as a network element ID. Memory identifier information, such as a Transaction Reference ID, which is the memory identifier information returned by the ADRF in step 4. Model monitoring time requirement information, i.e., time information for model monitoring, can be used to instruct the MTLF on the time range within which the model performance status should be monitored. For example, if the time information is one month, the MTLF needs to periodically collect model performance-related information within the next month to calculate the model performance status. This information may not be required, or the time information may be a very short time, such as immediate collection, which means that the current calculation is performed once and there is no need to collect and monitor data over a long period of time. Model monitoring area request information. Identifier information for the target task, such as an Analytics ID, which is used to indicate that the target data to be obtained is for a specific task. Conditional information for the target task, such as analytic filter information, is used to specify the filtering conditions for the data analysis results, including AOI, S-NSSAI, DNN, etc. Target information for the target task. For example, Target of analytic reporting, which is used to indicate that data analysis is targeted at a device, multiple devices, or all devices. Model identifier information, such as Model ID, is used to indicate that the target data to be acquired is for a certain model. Model filtering information, which specifies the conditions that the model that is the target of the target data to be acquired must satisfy, such as AOI, S-NSSAI, or DNN. Data category information: Used to indicate the data category of the target data to be acquired, such as the location information of the terminal. Time range information corresponding to the target data. This is used to indicate within which time range the target data is to be acquired. For example, this is determined based on the time when the data was generated, and data generated within this time range is acquired. Location range information corresponding to target data. Used to indicate the location range of target data to acquire. For example, it is determined based on the location at the time of data generation, and data generated within a certain tracking area or cell is acquired.

[0142] Steps 6-7: The MTLF sends second request information to the ADRF, requesting it to acquire the target data. Specifically, the MTLF determines the identifier information and address information of the target ADRF based on the information in step 5, and sends second request information to the target ADRF to acquire the target data. After receiving the second request information, the ADRF feeds back target data that meets the demand to the MTLF. For example, based on the AnLF identifier information and memory identifier information, it can search for data corresponding to the memory identifier information and feed it back to the MTLF. For example, the ADRF can determine specific data based on a combination of attribute information of the target data, such as task identifier information and model identifier information, and feed it back to the MTLF.

[0143] The second request information is AnLF identifier information, Target data storage identifier information; Identifier information of a target task corresponding to the target data; Condition-specific information for the target task, target information for the target task; Identifier information for the model that is the target of the target data; Data category information for goal data, time range information corresponding to the target data; The target data may include at least one of the location range information corresponding thereto.

[0144] Steps 8-10: After the MTLF acquires the target data of the target model in the actual environment, it calculates the model performance status to obtain model performance information and determines subsequent operations based on the model performance information. The MTLF can calculate the model performance information by comparing the label data with the output data. Furthermore, by comparing the model performance information with the desired model performance information in step 5, it determines whether model retraining or new model selection is necessary. For example, when the model performance information does not meet the desired model performance corresponding to the desired model performance information, model retraining or new model selection can be performed. Comparison of these two pieces of information can also determine whether to feed back the model performance information to the AnLF. For example, the model performance information can be fed back to the AnLF when the model performance information does not meet the desired model performance corresponding to the desired model performance information, or the model performance information can be notified to the AnLF immediately after calculation, or periodically.

[0145] The first data in the above example may be the entire target data.

[0146] Hereinafter, the embodiment of the present application will be described again by another example shown in Figure 4, in which the first data is part of the target data. The flow of this example is almost the same as the previous example, but steps 1, 2, 4, 5 and 6 are different.

[0147] Specifically, in step 1 of this example, the AnLF may not be responsible for collecting all relevant data, but may have collected only some of the data, e.g., only input data and not label data. Thus, in step 2, the AnLF not only needs to inform the ADRF to store the first data it has collected, e.g., input data (step 2a), but also to inform the ADRF to collect and store second data, e.g., label data, from designated data source network elements (step 2b).

[0148] When the AnLF informs the ADRF of only some of the target data, the third instruction information sent by the AnLF to the ADRF in step 2b is used to instruct the ADRF to collect and store the second data and / or information related to the second data from the target data source network element, and may specifically include at least one of the following: Information about the device where the secondary data resides. This information allows the ADRF to know where to collect the secondary data. Identifier information of the device where the second data exists, such as a network element ID. Address information of the device where the second data exists, such as the IP address or FQDN of the network element. Collection request information corresponding to the second data, based on which the ADRF can determine the second data to be collected. Data category information of the second data. Used to specify which data to collect, for example, data in the terminal location information category. Time range information corresponding to the second data, which is used to indicate within which time range the data is to be collected. Location range information corresponding to the second data, which is used to indicate within which location range data is to be collected. Second data collection time information, which is used to instruct the ADRF within what time range the second data will be collected. For example, if the collection time information is one month, the ADRF needs to periodically collect the second data within one month in the future. It may also be a time period in the past.

[0149] In one implementation, the task of step 2b can be accomplished by a Nadrf_DataManagement_StorageSubscriptionRequest.

[0150] Since step 2b is added, second memory feedback information including corresponding memory identifier information can be further sent in step 4, and the first request information in step 5 also needs to include the memory identifier information, so that the MTLF can request corresponding data from the ADRF in step 6.

[0151] The embodiments of the present application provide a method for enabling AnLF to obtain model performance information, and assist AnLF in obtaining model performance information to make more rational choices. For example, when AnLF finds that the model performance situation is poor, it will not use the data analysis results of the model, avoiding adverse effects on network conditions and user experience.

[0152] Meanwhile, by enhancing the data and content that can be stored in the ADRF, the ADRF can store relevant data for models. The data stored in the ADRF can also be collected and used by other network elements. For example, the MTLF can obtain the relevant data from the ADRF to calculate model performance status or use this information to train other models.

[0153] The information acquisition method provided in the embodiments of the present application may be executed by an information acquisition device. In the embodiments of the present application, the information acquisition device provided in the embodiments of the present application will be described as an example in which the information acquisition device executes the information acquisition method.

[0154] As shown in FIG. 5, the information acquisition device 500 a first sending module 510 used to send first request information to a second network side device, the first request information being used to request the second network side device to determine model performance information of the target model; and a first receiving module 520, which is used to receive model performance information of the target model sent from the second network side device; The first request information includes target data and / or attribute information of the target data, where the target data is used to detect the model performance status of the target model, and the attribute information of the target data is used to acquire the target data.

[0155] According to the device provided in the embodiment of the present application, first request information is sent to a second network side device, and the first request information is used to request the second network side device to determine model performance information of a target model. After the second network side device receives the model performance information of the target model determined based on the target data, it can further use the model performance information to determine the accuracy of the data analysis result in the actual use stage of the target model. Thus, when the data analysis result is highly accurate, it can be provided to devices inside and outside the communication network to assist their policy decision-making, and ensure that the devices inside and outside the communication network make correct policy decisions.

[0156] In a specific embodiment of the present application, the first request information includes: 1st instruction information, Model performance desired information, Model monitoring time requirement information, model monitoring area request information; The first instruction information is used to instruct the second network side device to determine model performance information of the target model; The desired model performance information is used to represent model performance information of a target model desired by the first network side device; The model monitoring time request information is used to instruct the second network side device on a time range for determining the model performance information of the target model; The model monitoring area request information is used to instruct the second network side device on the area range for determining the model performance information of the target model.

[0157] In a specific embodiment of the present application, the attribute information of the target data is: task information corresponding to the goal data; Information about the model that is the target of the target data; data description information corresponding to the target data; At least one of the stored information corresponding to the target data is included.

[0158] In a specific embodiment of the present application, task information corresponding to goal data includes: Identifier information of a target task corresponding to the target data; Condition-specific information for the target task, including at least one of target information that is the target of the target task; The condition restriction information of the target task is used to represent filtering conditions for the data analysis results corresponding to the target task.

[0159] In a specific embodiment of the present application, information of a model that is the target of the target data is: The target data includes at least one of identifier information of the model that is the target of the target data and filtering information of the model that is the target of the target data.

[0160] In a specific embodiment of the present application, the data description information corresponding to the target data includes: The target data includes at least one of data category information of the target data, time range information corresponding to the target data, and location range information corresponding to the target data; The time range information corresponding to the target data is used to indicate a time range in which the time information corresponding to the target data exists; The location range information corresponding to the target data is used to indicate the location range in which the location information corresponding to the target data exists.

[0161] In a specific embodiment of the present application, the stored information corresponding to the target data includes: The information includes at least one of identifier information of the third network side device that stores the target data, address information of the third network side device, storage identifier information of the target data, and identifier information of the first network side device.

[0162] In a specific embodiment of the present application, when the first request information includes attribute information of the target data, the information acquisition device 500: The device further includes a first acquisition module used to acquire first data, which is a part or all of the target data, before transmitting the first request information to the second network side device; The first sending module 510 is further used for sending second instruction information to the third network side device, where the second instruction information includes the first data and / or related information of the first data, and is used to instruct the third network side device to store the first data and / or related information of the first data.

[0163] In a specific embodiment of the present application, the first receiving module 520 further comprises: used for receiving first stored feedback information transmitted from a third network side device; The first storage feedback information includes storage identifier information corresponding to the first data.

[0164] In a specific embodiment of the present application, the first data comprises: The data includes at least one of input data corresponding to the target model, output data corresponding to the target model, and label data corresponding to the target model.

[0165] In one specific embodiment of the present application, the label data corresponding to the target model includes actual generated data associated with the input data and / or output data corresponding to the target model.

[0166] In a specific embodiment of the present application, the related information of the first data is: The information includes at least one of task information corresponding to the first data, information on a model that is a target of the first data, and time information corresponding to the first data.

[0167] In a specific embodiment of the present application, the task information corresponding to the first data includes: The first data includes at least one of identifier information of a target task corresponding to the first data, condition limiting information of the target task, and target information of a target of the target task.

[0168] In a specific embodiment of the present application, the information of the model that is the subject of the first data is: The information includes at least one of identifier information of the model that is the target of the first data and filtering information of the model that is the target of the first data.

[0169] In a specific embodiment of the present application, the time information corresponding to the first data is: The time information includes at least one of time information corresponding to the input data, time information corresponding to the output data, and time information corresponding to the label data.

[0170] In a specific embodiment of the present application, the time information corresponding to the input data is: Time point information when the input data was generated, time point information when the input data is acquired by the first network side device; The input data includes at least one of the time point information when the input data is input to the target model.

[0171] In a specific embodiment of the present application, the time information corresponding to the output data is: Time point information when output data is generated, Target time point information for output data; The output data includes at least one piece of time information corresponding to the input data that is associated with the output data.

[0172] In a specific embodiment of the present application, the time information corresponding to the label data is: Time point information when label data was generated, The label data includes at least one of the time point information when the label data is acquired by the first network side device.

[0173] In a specific embodiment of the present application, the second transmitting module further comprises: When the first request information includes attribute information of the target data, the third instruction information is used to send third instruction information to the third network side device before sending the first request information to the second network side device, and the third instruction information is used to instruct the third network side device to collect and store the second data and / or information related to the second data, and the second data is part or all of the target data.

[0174] In a specific embodiment of the present application, the first receiving module 520 further comprises: used for receiving second stored feedback information transmitted from a third network side device; The second storage feedback information includes storage identifier information corresponding to the second data.

[0175] In a specific embodiment of the present application, the related information of the second data is: The information includes at least one of task information corresponding to the second data, information on a model that is the subject of the second data, and data description information corresponding to the second data.

[0176] In a specific embodiment of the present application, the third instruction information includes information on the device where the second data exists and / or collection request information corresponding to the second data.

[0177] In a specific embodiment of the present application, the information on the device where the second data exists is: The information includes at least one of identifier information of a device in which the second data exists and address information of a device in which the second data exists.

[0178] In a specific embodiment of the present application, the collection request information corresponding to the second data is: The information includes at least one of data category information of the second data, time range information corresponding to the second data, location range information corresponding to the second data, and collection time information of the second data; the time range information corresponding to the second data is used to indicate a time range in which the time information corresponding to the second data exists; the location range information corresponding to the second data is used to indicate a location range in which the location information corresponding to the second data exists; The collection time information of the second data is used to represent a time requirement for collecting the second data.

[0179] In a specific embodiment of the present application, the first data includes input data corresponding to the target model and / or output data corresponding to the target model, and the second data includes label data corresponding to the target model.

[0180] In a specific embodiment of the present application, the first sending module 510 is further used to send fourth instruction information to the second network side device according to the model performance information, and the fourth instruction information is used to instruct the second network side device to retrain the target model or perform model reselection; The first receiving module 520 is further used for receiving the re-trained model or the re-selected model sent from the second network side device.

[0181] The information acquisition device 500 provided in the embodiment of the present application can implement each step implemented in the method embodiment shown in FIG. 2 and achieve the same technical effect, and detailed description thereof will be omitted here to avoid repetition.

[0182] For the method embodiment shown in FIG. 2, the present application further provides an information obtaining method, as shown in FIG. 6, the method includes: a step S610 in which the second network side device receives first request information transmitted from the first network side device, the first request information including target data and / or attribute information of the target data used to acquire the target data; A step S620 in which the second network side device determines model performance information of the target model based on the target data; The method may include a step S630 in which the second network side device transmits the model performance information to the first network side device.

[0183] According to the method provided in the embodiments of the present application, the second network side device receives first request information sent from the first network side device, detects the model performance status of the target model based on the target data, determines model performance information, and then sends the model performance information of the target model to the first network side device. In this way, the first network side device obtains the model performance information of the target model, and can further use the model performance information to determine the accuracy of the data analysis result in the actual use stage of the target model. Thus, when the data analysis result is highly accurate, it can be provided to devices inside and outside the communication network to assist their policy decision-making, and ensure that devices inside and outside the communication network make correct policy decisions.

[0184] In a specific embodiment of the present application, the first request information includes: Further including at least one of first instruction information, model performance desired information, model monitoring time request information, and model monitoring area request information; The first instruction information is used to instruct the second network side device to determine model performance information of the target model; The desired model performance information is used to represent model performance information of a target model desired by the first network side device; The model monitoring time request information is used to instruct the second network side device on a time range for determining the model performance information of the target model; The model monitoring area request information is used to instruct the second network side device on the area range for determining the model performance information of the target model.

[0185] In a specific embodiment of the present application, the attribute information of the target data is: The information includes at least one of task information corresponding to the target data, information on a model that is the target of the target data, data description information corresponding to the target data, and storage information corresponding to the target data.

[0186] In a specific embodiment of the present application, task information corresponding to goal data includes: The target task includes at least one of identifier information of the target task corresponding to the target data, condition limiting information of the target task, and target information of the target task; The condition restriction information of the target task is used to represent filtering conditions for the data analysis results corresponding to the target task.

[0187] In a specific embodiment of the present application, information of a model that is the target of the target data is: The target data includes at least one of identifier information of the model that is the target of the target data and filtering information of the model that is the target of the target data.

[0188] In a specific embodiment of the present application, the data description information corresponding to the target data includes: The target data includes at least one of data category information of the target data, time range information corresponding to the target data, and location range information corresponding to the target data; The time range information corresponding to the target data is used to indicate a time range in which the time information corresponding to the target data exists; The location range information corresponding to the target data is used to indicate the location range in which the location information corresponding to the target data exists.

[0189] In a specific embodiment of the present application, the stored information corresponding to the target data includes: The information includes at least one of identifier information of the third network side device that stores the target data, address information of the third network side device, storage identifier information of the target data, and identifier information of the first network side device.

[0190] In a specific embodiment of the present application, when the first request information includes attribute information of the target data, before the step of the second network side device determining model performance information of the target model based on the target data: The method further includes a step of the second network side device acquiring the target data according to the attribute information of the target data.

[0191] In a specific embodiment of the present application, the step of the second network side device acquiring the target data according to the attribute information of the target data includes: The second network side device determines a third network side device that stores the target data according to attribute information of the target data; a step of the second network side device sending second request information to the third network side device, the second request information being used to request to acquire target data; The second network side device receives the target data transmitted from the third network side device.

[0192] In a specific embodiment of the present application, the second request information includes: The information includes at least one of identifier information of the first network side device, storage identifier information of the target data, task information corresponding to the target data, information of a model that is the target of the target data, and data description information corresponding to the target data.

[0193] In a specific embodiment of the present application, after the second network side device determines model performance information of the target model based on the target data: The second network side device further includes determining whether the model performance information satisfies the model performance preference corresponding to the model performance preference information; The step of transmitting model performance information from the second network side device to the first network side device includes: The method includes a step of the second network side device sending the model performance information to the first network side device when the model performance information meets or does not meet the model performance expectation.

[0194] In one specific embodiment of the present application, The method further includes the step of the second network side device retraining the target model or performing model reselection when the model performance information does not meet the desired model performance.

[0195] In a specific embodiment of the present application, after the second network side device sends the model performance information to the first network side device: receiving, by the second network side device, fourth instruction information transmitted from the first network side device; the second network side device retraining the target model or reselecting the model according to the fourth instruction information; The method further includes a step of the second network side device sending the retrained model or the reselected model to the first network side device.

[0196] In a specific embodiment of the present application, the target data includes label data corresponding to the target model, and input data and / or output data corresponding to the target model, and the step of the second network side device determining model performance information of the target model based on the target data includes: a step of the second network side device comparing each output data with the associated label data to obtain a comparison result; The second network side device determines model performance information of the target model according to the comparison result.

[0197] The implementation steps of the method embodiment shown in FIG. 6 can refer to the implementation steps of the method embodiment shown in FIG. 2, and the same technical effects are achieved, so detailed descriptions are omitted here to avoid repetition.

[0198] The information acquisition method provided in the embodiments of the present application may be executed by an information acquisition device. In the embodiments of the present application, the information acquisition device provided in the embodiments of the present application will be described as an example in which the information acquisition device executes the information acquisition method.

[0199] As shown in FIG. 7, the information acquisition device 700 a second receiving module 710 used to receive first request information transmitted from a first network side device, the first request information including target data and / or attribute information of the target data; a first determination module 720 used to determine model performance information of the target model based on the target data; and a second sending module 730 used to send the model performance information to the first network side device.

[0200] According to the device provided in the embodiment of the present application, first request information sent from a first network side device is received, the model performance status of the target model is detected based on the target data, model performance information is determined, and then the model performance information of the target model is sent to the first network side device. In this way, the first network side device obtains the model performance information of the target model, and can further use the model performance information to determine the accuracy of the data analysis result in the actual use stage of the target model. Thus, when the data analysis result is highly accurate, it can be provided to devices inside and outside the communication network to assist their policy decision-making, and ensure that devices inside and outside the communication network make correct policy decisions.

[0201] In a specific embodiment of the present application, the first request information includes: Further including at least one of first instruction information, model performance desired information, model monitoring time request information, and model monitoring area request information; The first instruction information is used to instruct the second network side device to determine model performance information of the target model; The desired model performance information is used to represent model performance information of a target model desired by the first network side device; The model monitoring time request information is used to instruct the second network side device on a time range for determining the model performance information of the target model; The model monitoring area request information is used to instruct the second network side device on the area range for determining the model performance information of the target model.

[0202] In a specific embodiment of the present application, the attribute information of the target data is: The information includes at least one of task information corresponding to the target data, information on a model that is the target of the target data, data description information corresponding to the target data, and storage information corresponding to the target data.

[0203] In a specific embodiment of the present application, task information corresponding to goal data includes: The target task includes at least one of identifier information of the target task corresponding to the target data, condition limiting information of the target task, and target information of the target task; The condition restriction information of the target task is used to represent filtering conditions for the data analysis results corresponding to the target task.

[0204] In a specific embodiment of the present application, information of a model that is the target of the target data is: The target data includes at least one of identifier information of the model that is the target of the target data and filtering information of the model that is the target of the target data.

[0205] In a specific embodiment of the present application, the data description information corresponding to the target data includes: The target data includes at least one of data category information of the target data, time range information corresponding to the target data, and location range information corresponding to the target data; The time range information corresponding to the target data is used to indicate a time range in which the time information corresponding to the target data exists; The location range information corresponding to the target data is used to indicate the location range in which the location information corresponding to the target data exists.

[0206] In a specific embodiment of the present application, the stored information corresponding to the target data includes: The information includes at least one of identifier information of the third network side device that stores the target data, address information of the third network side device, storage identifier information of the target data, and identifier information of the first network side device.

[0207] In a specific embodiment of the present application, the information acquisition device 700 includes: The system further includes a second acquisition module, which is used to acquire target data according to the attribute information of the target data before determining model performance information of the target model based on the target data when the first request information includes attribute information of the target data.

[0208] In a specific embodiment of the present application, the second acquisition module: determining a third network side device that stores the target data based on the attribute information of the target data; Sending second request information to the third network side device to request acquisition of the target data; It is used to receive target data sent from the third network side device.

[0209] In a specific embodiment of the present application, the second request information includes: The information includes at least one of identifier information of the first network side device, storage identifier information of the target data, task information corresponding to the target data, model information corresponding to the target data, and data description information corresponding to the target data.

[0210] In a specific embodiment of the present application, the information acquisition device 700 includes: Further, a second determination module is used to determine whether the model performance information of the target model satisfies the model performance preference corresponding to the model performance preference information after determining the model performance information of the target model based on the target data; The second sending module 730 is further used for sending the model performance information to the first network side device when the model performance information meets or does not meet the model performance expectation.

[0211] In a specific embodiment of the present application, the information acquisition device 700 includes: The system further comprises a first execution module that is used to retrain the target model or perform model reselection when the model performance information does not meet the desired model performance.

[0212] In a specific embodiment of the present application, the information acquisition device 700 includes: After transmitting the model performance information to the first network side device, receive fourth instruction information transmitted from the first network side device; The fourth instruction information is used to retrain the target model or to reselect the model; The device further comprises a second execution module used for sending the retrained model or the reselected model to the first network side device.

[0213] In a specific embodiment of the present application, the target data includes label data corresponding to the target model, and input data and / or output data corresponding to the target model, and the first determination module 720: comparing each output data with the associated label data to obtain a comparison result; The results of the comparison are used to determine model performance information for the target model.

[0214] The information acquisition device 700 provided in the embodiment of the present application can implement each step implemented in the method embodiment shown in FIG. 6 and achieve the same technical effect, and detailed description thereof will be omitted here to avoid repetition.

[0215] In addition to the method embodiments shown in FIG. 2 and FIG. 6, an embodiment of the present application further provides an information acquisition method, as shown in FIG. 8, the method includes: Step S810 in which the third network side device receives the second instruction information and the third instruction information transmitted from the first network side device; Step S820: the third network side device stores the first data and / or information related to the first data according to the second instruction information; and step S830, in which the third network side device collects and stores the second data and / or information related to the second data according to the third instruction information; The first data and the second data are each part of the target data, the target data is used to determine model performance information of the target model, and the related information of the first data and the related information of the second data are used to obtain the target data.

[0216] According to the method provided in the embodiments of the present application, the third network side device stores the first data and / or information related to the first data, and collects and stores the second data and / or information related to the second data according to the instruction information sent from the first network side device, thereby reducing the amount of data exchanged with the first network side device, saving network resources, and accurately providing the target data when a request is made to obtain the target data.

[0217] In a specific embodiment of the present application, the method comprises: a step of receiving, by a third network side device, second request information sent from the second network side device, the second request information being used to acquire target data; The method may further include a step of the third network side device transmitting the target data to the second network side device.

[0218] For the implementation steps of the method embodiment shown in FIG. 8, reference can be made to the implementation steps of the method embodiments shown in FIG. 2 and FIG. 6, and detailed descriptions thereof will be omitted here to avoid repetition and achieve the same technical effects.

[0219] The information acquisition method provided in the embodiments of the present application may be executed by an information acquisition device. In the embodiments of the present application, the information acquisition device provided in the embodiments of the present application will be described as an example in which the information acquisition device executes the information acquisition method.

[0220] As shown in FIG. 9, the information acquisition device 900 a third receiving module 910 used to receive the second instruction information and the third instruction information sent from the first network side device; a storage module 920 used to store the first data and / or information related to the first data according to the second instruction information, and to collect and store the second data and / or information related to the second data according to the third instruction information; The first data and the second data are each part of the target data, the target data is used to determine model performance information of the target model, and the related information of the first data and the related information of the second data are used to obtain the target data.

[0221] According to the device provided in the embodiments of the present application, the third network side device stores the first data and / or information related to the first data, and collects and stores the second data and / or information related to the second data according to the instruction information sent from the first network side device, thereby reducing the amount of data exchanged with the first network side device, saving network resources, and accurately providing target data when a request is made to obtain the target data.

[0222] In a specific embodiment of the present application, the information acquisition device 900 includes: a fourth receiving module used to receive second request information sent from the second network side device, the second request information being used to acquire target data; The device further includes a third sending module used to send the target data to the second network side device.

[0223] The information acquisition device 900 provided in the embodiment of the present application can implement each step implemented in the method embodiment shown in FIG. 8 and achieve the same technical effect, and detailed description thereof will be omitted here to avoid repetition.

[0224] In contrast to the above method and device embodiments, as shown in FIG. 10, an embodiment of the present application further provides a network side device 1000, which includes a processor 1001 and a memory 1002, and stores programs or commands executable by the processor 1001. When the programs or commands are executed by the processor 1001, each step of the above information acquisition method embodiment is realized, and the same technical effect can be achieved. Detailed description thereof will be omitted here to avoid repetition.

[0225] Specifically, an embodiment of the present application further provides a network side device. As shown in Fig. 11, the network side device 1100 includes a processor 1101, a network interface 1102, and a memory 1103. Here, the network interface 1102 is, for example, a common public radio interface (CPRI).

[0226] Specifically, the network side device 1100 of the embodiment of the present invention further includes a command or program stored in the memory 1103 and executable by the processor 1101, and the processor 1101 calls the command or program in the memory 1103 to execute the method performed by each module shown in Figure 5, Figure 7 or Figure 9, thereby achieving the same technical effect, and detailed description will be omitted here to avoid repetition.

[0227] An embodiment of the present application further provides a readable storage medium storing a program or command, which, when executed by a processor, realizes each step of the method embodiment shown in FIG. 2, the method embodiment shown in FIG. 6, or the method embodiment shown in FIG. 8, and can achieve the same technical effect. Detailed description thereof will be omitted here to avoid repetition.

[0228] Wherein, the processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer readable storage medium such as a computer read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0229] An embodiment of the present application further provides a computer program / program product stored in a storage medium, which, when executed by at least one processor, can realize each step of the method embodiment shown in FIG. 2, the method embodiment shown in FIG. 6, or the method embodiment shown in FIG. 8, and achieve the same technical effects. Detailed descriptions thereof will be omitted here to avoid repetition.

[0230] It should be noted that, as used herein, terms such as "comprises," "consists of," or any other variation thereof are intended to include a non-exclusive inclusion, such that a process, method, article, or apparatus comprising a set of elements includes not only those elements but also other elements not expressly specified or inherent in such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element. It should also be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may include performing functions substantially simultaneously or in the reverse order, depending on such functionality. For example, the described method may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to one example may be combined in other examples.

[0231] From the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be realized in the form of a combination of software and a necessary common hardware platform, and of course, they can also be realized by hardware, but in many cases the former is a more preferred embodiment. Based on this view, the technical solutions of the present application can be substantially embodied in the form of a computer software product, which is stored in a storage medium (e.g., ROM / RAM, magnetic disk, optical disk) and includes a plurality of commands that cause a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0232] Although the examples of the present application have been described above with reference to the drawings, the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative and not limiting. Based on the suggestions of the present application, many forms that a person skilled in the art can make without departing from the spirit of the present application and the scope of protection of the claims are all within the scope of protection of the present application.

Claims

1. a step of a first network side device transmitting first request information to a second network side device, the first request information being used to request the second network side device to determine model performance information of a target model; receiving, by the first network side device, model performance information of the target model transmitted from the second network side device; An information acquisition method, wherein the first request information includes target data and / or attribute information of the target data, the target data being used to determine model performance information of the target model, and the attribute information of the target data being used to acquire the target data.

2. The first request information is first instruction information; Model performance desired information, Model monitoring time requirement information, model monitoring area request information; the first instruction information is used to instruct the second network side device to determine model performance information of the target model; The desired model performance information is used to represent model performance information of the target model desired by the first network side device, the model monitoring time request information is used to instruct the second network side device on a time range for determining model performance information of the target model; 2. The information acquisition method according to claim 1, wherein the model monitoring area request information is used to instruct the second network side device on an area range for determining model performance information of the target model.

3. The attribute information of the target data is task information corresponding to the goal data; Information on the model that is the target of the target data; data description information corresponding to the target data; The information acquisition method according to claim 1 , further comprising the step of: acquiring at least one of the stored information corresponding to the target data;

4. Task information corresponding to the goal data is Identifier information of a target task corresponding to the target data; condition limiting information of the target task; The target information includes at least one of target information that is a target of the target task; The information acquisition method according to claim 3 , wherein the condition limiting information of the target task is used to represent a filtering condition for a data analysis result corresponding to the target task.

5. The information on the model that is the target of the target data is: Identifier information of a model that is the target of the target data; The information acquisition method according to claim 3 , further comprising at least one of filtering information of a model that is a target of the target data.

6. The data description information corresponding to the target data is data category information of the target data; time range information corresponding to the target data; at least one of location range information corresponding to the target data; the time range information corresponding to the target data is used to indicate a time range in which the time information corresponding to the target data exists; The information acquisition method according to claim 3 , wherein the location range information corresponding to the target data is used to indicate a location range in which the location information corresponding to the target data exists.

7. The stored information corresponding to the target data is Identifier information of a third network side device that stores the target data; Address information of the third network side device; storage identifier information of the target data; The information acquisition method according to claim 3 , wherein the information includes at least one of identifier information of the first network side device.

8. When the first request information includes attribute information of the target data, before the step of the first network side device transmitting the first request information to the second network side device, a step of the first network side device acquiring first data, the first data being a part or all of the target data; 8. The information acquisition method according to claim 1, further comprising: a step of transmitting second instruction information from the first network side device to a third network side device, the second instruction information including the first data and / or information related to the first data, and being used to instruct the third network side device to store the first data and / or information related to the first data.

9. The method further includes a step of receiving, by the first network side device, first storage feedback information transmitted from the third network side device; The information acquisition method according to claim 8 , wherein the first storage feedback information includes storage identifier information corresponding to the first data.

10. The first data is input data corresponding to the target model; output data corresponding to the target model; The information acquisition method of claim 8 , further comprising at least one of label data corresponding to the target model.

11. The information acquisition method of claim 10 , wherein the label data corresponding to the target model includes actual generated data associated with the input data and / or the output data corresponding to the target model.

12. The related information of the first data is task information corresponding to the first data; Information on a model that is the subject of the first data; The information acquisition method according to claim 10 , further comprising at least one piece of time information corresponding to the first data.

13. Task information corresponding to the first data: Identifier information of a target task corresponding to the first data; condition limiting information of the target task; The information acquisition method according to claim 12 , wherein the information acquisition method includes at least one of target information that is a target of the target task.

14. Information on a model that is the target of the first data, Identifier information of a model that is the subject of the first data; The information acquisition method according to claim 12 , further comprising at least one of filtering information of a model that is a target of the first data.

15. When the first request information includes attribute information of the target data, before the step of the first network side device transmitting the first request information to the second network side device, 8. The information acquisition method according to claim 1, further comprising a step in which the first network side device transmits third instruction information to the third network side device, the third instruction information being used to instruct the third network side device to collect and store second data and / or information related to the second data, and the second data being part or all of the target data.

16. The method further includes a step of receiving second storage feedback information sent by the first network side device from the third network side device; the second storage feedback information includes storage identifier information corresponding to the second data; The information acquisition method according to claim 15.

17. The related information of the second data is task information corresponding to the second data; Information on a model that is the subject of the second data; The information acquisition method according to claim 15 , further comprising at least one of data description information corresponding to the second data.

18. The information acquisition method according to claim 15 , wherein the third instruction information includes information on a device in which the second data exists and / or collection request information corresponding to the second data.

19. The information on the device in which the second data exists is Identifier information of a device in which the second data exists; The information acquisition method according to claim 18 , wherein the information includes at least one of address information of a device where the second data exists.

20. Collection request information corresponding to the second data: data category information of the second data; time range information corresponding to the second data; location range information corresponding to the second data; at least one of collection time information of the second data; the time range information corresponding to the second data is used to indicate a time range in which the time information corresponding to the second data exists; the location range information corresponding to the second data is used to indicate a location range in which the location information corresponding to the second data exists; The information acquisition method according to claim 18 , wherein the collection time information of the second data is used to represent a time requirement for collecting the second data.

21. 21. The method of claim 1, wherein the model performance information includes at least one of a model accuracy rate and a model error rate.

22. a first sending module used to send first request information to a second network side device, the first request information being used to request the second network side device to determine model performance information of a target model; a first receiving module used to receive model performance information of the target model transmitted from the second network side device; An information acquisition device wherein the first request information includes target data and / or attribute information of the target data, the target data being used to determine model performance information of the target model, and the attribute information of the target data being used to acquire the target data.

23. a step of receiving first request information transmitted from the first network side device by the second network side device, the first request information including target data and / or attribute information of the target data, and the attribute information of the target data being used to acquire the target data; The second network side device determines model performance information of a target model based on the target data; the second network side device transmitting the model performance information to the first network side device.

24. The first request information is first instruction information; Model performance desired information, Model monitoring time requirement information, model monitoring area request information; the first instruction information is used to instruct the second network side device to determine model performance information of the target model; The desired model performance information is used to represent model performance information of the target model desired by the first network side device, the model monitoring time request information is used to instruct the second network side device on a time range for determining model performance information of the target model; 24. The information obtaining method according to claim 23, wherein the model monitoring area request information is used to instruct the second network side device on an area range for determining model performance information of the target model.

25. The attribute information of the target data is task information corresponding to the goal data; Information on the model that is the target of the target data; data description information corresponding to the target data; 24. The information acquisition method of claim 23, including at least one of the stored information corresponding to the target data.

26. When the first request information includes attribute information of the target data, before the step of the second network side device determining model performance information of the target model based on the target data, The information acquisition method according to claim 23, further comprising the step of the second network side device acquiring the target data according to attribute information of the target data.

27. The step of the second network side device acquiring the target data based on attribute information of the target data includes: The second network side device determines a third network side device that stores the target data according to attribute information of the target data; a step of the second network side device transmitting second request information to the third network side device, wherein the second request information is used to request acquisition of the target data; The information acquisition method according to claim 26, further comprising: receiving, by the second network side device, the target data transmitted from the third network side device.

28. The second request information is Identifier information of the first network side device; storage identifier information of the target data; task information corresponding to the goal data; Information on the model that is the target of the target data; 28. The information acquisition method of claim 27, further comprising at least one of data description information corresponding to the target data.

29. After the step of the second network side device determining model performance information of the target model based on the target data, The information acquisition method of claim 24, further comprising the step of: when the model performance information does not satisfy the desired model performance, the second network side device retraining the target model or performing model reselection.

30. The target data includes label data corresponding to the target model, and input data and / or output data corresponding to the target model, and the step of the second network side device determining model performance information of the target model based on the target data includes: the second network side device comparing each of the output data with the associated label data to obtain a comparison result; The information acquisition method according to any one of claims 23 to 29, further comprising: determining model performance information of the target model by the second network side device according to the comparison result.

31. The step of the second network side device transmitting the model performance information to the first network side device includes:

31. The information acquisition method according to claim 23, further comprising the step of: when the model performance information satisfies or does not satisfy a model performance request, the second network side device transmitting the model performance information to the first network side device.

32. 32. A method of obtaining information according to any one of claims 23 to 31, wherein the step of determining model performance information for a target model comprises the step of monitoring the target model.

33. 33. The information acquisition method according to claim 1, wherein the first network side device includes an analysis logic function AnLF, and the second network side device includes a model training logic function MTLF.

34. a second receiving module used to receive first request information transmitted from a first network side device, the first request information including target data and / or attribute information of the target data, the attribute information of the target data being used to acquire the target data; a first determination module used to determine model performance information of a target model based on the target data; a second sending module used to send the model performance information to the first network side device.

35. receiving, by the third network side device, the second instruction information and the third instruction information transmitted from the first network side device; the third network side device storing the first data and / or information related to the first data according to the second instruction information; and collecting and storing second data and / or information related to the second data by the third network side device according to the third instruction information; An information acquisition method, wherein the first data and the second data are each part of target data, the target data is used to determine model performance information of a target model, and related information of the first data and related information of the second data are used to acquire the target data.

36. a step of receiving second request information transmitted from the second network side device by the third network side device, the second request information being used to acquire the target data; The third network side device further includes a step of transmitting the target data to the second network side device.

36. The information acquisition method according to claim 35.

37. a third receiving module used to receive the second instruction information and the third instruction information transmitted from the first network side device; a storage module used to store the first data and / or information related to the first data in accordance with the second instruction information, and to collect and store the second data and / or information related to the second data in accordance with the third instruction information; the first data and the second data are each part of target data, the target data is used to determine model performance information of a target model, and the related information of the first data and the related information of the second data are used to obtain the target data; Information acquisition device.

38. A network side device comprising a processor and a memory, wherein a program or command executable by the processor is stored in the memory, and when the program or command is executed by the processor, the steps of the information acquisition method described in any one of claims 1 to 21 are realized, or the steps of the information acquisition method described in any one of claims 23 to 33 are realized, or the steps of the information acquisition method described in claim 35 or 36 are realized.

39. A readable storage medium on which a program or command is stored, and when the program or command is executed by a processor, the steps of the information acquisition method described in any one of claims 1 to 21 are realized, or the steps of the information acquisition method described in any one of claims 23 to 33 are realized, or the steps of the information acquisition method described in claim 35 or 36 are realized.

Images