Data acquisition methods, devices, systems and equipment

Abstract

The present application discloses a data acquisition method, apparatus, system and device, which belong to the data processing field. In an embodiment of the data acquisition method of the present application, the method includes: a first network side device sending a first data acquisition request to a second network side device, the first data acquisition request being used to request target data; the target data being used to determine performance information of the target model; the first network side device receiving storage information from the second network side device and acquiring the target data based on the storage information; and the first network side device analyzing the target data to obtain performance information of the target model.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application bearing application number 202211146189.6 and entitled "Data Acquisition Method, Apparatus, System and Device" filed with the China Patent Office on September 20, 2022, the entire contents of which are incorporated herein by reference.

[0002] This application is in the field of data processing, and more particularly relates to data acquisition methods, devices, systems and equipment. [Background technology]

[0003] In conventional network and protocol standards, devices can obtain a target model corresponding to an analysis task by performing model training on training data, and then perform model inference on the input data of the analysis task based on the target model obtained by training, and the output data of the target model can be used as the model inference result of the analysis task. The model inference result can be used to assist devices inside and outside the network in making policy decisions and improve the degree of intelligence of the device's policy decisions.

[0004] As can be understood, when the accuracy rate of the model inference results is high, devices inside and outside the network can refer to the model inference results to make policy decisions, obtain appropriate policy decisions, or perform appropriate operations; conversely, when the accuracy rate of the model inference results is low, devices inside and outside the network can refer to the model inference results to make policy decisions, and may make incorrect policy decisions or perform inappropriate operations. Therefore, in order to obtain accurate policy decisions, it is necessary to ensure the accuracy of the model inference results.

[0005] However, in the above process, due to reasons such as different distributions of training data and input data or insufficient generalization ability of the target model, the accuracy achieved in the training stage of the target model is generally lower than the accuracy that can be achieved in the actual model inference process, and furthermore, the accuracy rate of policy decisions made by devices inside and outside the network based on the model inference results of the model is low. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide a data acquisition method, device, system and equipment that can solve the problem that the accuracy achieved by a target model during the training phase is generally lower than the accuracy achieved when the target model is input into the actual model inference process, and that the accuracy of policy decisions made by devices inside and outside the network based on the model inference results of the target model is low. [Means for solving the problem]

[0007] According to a first aspect, an embodiment of the present application provides a data transmission method, the method comprising: a first network-side device sending a first data acquisition request to a second network-side device, the first data acquisition request being used to request target data, and the target data being used to determine performance information of the target model; The first network side device receives storage information from the second network side device and acquires the target data based on the storage information; The first network-side device analyzes the target data to obtain performance information of the target model.

[0008] According to a second aspect, embodiments of the present application provide another data acquisition method, the method comprising: a second network side device receiving a first data acquisition request from a first network side device, the first data acquisition request being used to request target data, and the target data being used to determine performance information of the target model; The second network side device transmits stored information to the first network side device, and the stored information is used to instruct the first network side device to acquire the target data.

[0009] According to a third aspect, an embodiment of the present application provides a data acquisition device, the data acquisition device comprising: a request module, the request module being used to send a first data acquisition request to a second network-side device, the first data acquisition request being used to request target data, the target data being used to determine performance information of the target model; an acquisition module for receiving stored information from the second network side device and acquiring the target data based on the stored information; and an analysis module for analyzing the target data to obtain performance information of the target model.

[0010] According to a fourth aspect, an embodiment of the present application provides another data acquisition device, the data acquisition device comprising: a receiving module, the receiving module being used to receive a first data acquisition request from a first network-side device, the first data acquisition request being used to request target data, the target data being used to determine performance information of the target model; and a transmission module used to transmit stored information to the first network side device, the stored information being used to instruct the first network side device to acquire the target data.

[0011] According to a fifth aspect, an embodiment of the present application provides a data acquisition system including a first network side device and a second network side device, wherein: the first network side device sends a first data acquisition request to a second network side device, the first data acquisition request is used to request target data, and the target data is used to determine performance information of the target model; the second network side device receives the first data acquisition request and transmits storage information to the first network side device, the storage information being used to instruct the first network side device to acquire the target data; the first network side device receives the stored information and acquires the target data based on the stored information; The first network-side device analyzes the target data to obtain performance information of the target model.

[0012] According to a sixth aspect, an embodiment of the present application provides another data acquisition system, including a first network side device, a second network side device, and a third network side device, wherein: the first network side device sends a first data acquisition request to a second network side device, the first data acquisition request is used to request target data, and the target data is used to determine performance information of the target model; the second network side device receives the first data acquisition request and transmits stored information to the first network side device; the first network side device receives the stored information, and acquires the target data from the third network side device based on the stored information; The first network-side device analyzes the target data to obtain performance information of the target model.

[0013] According to a seventh aspect, there is provided an apparatus including a processor and a memory, said memory storing a program or instructions operable to run on said processor, said program or instructions, when executed by said processor, to implement the steps of the method according to the first and / or second aspects.

[0014] According to an eighth aspect, there is provided a communications device including a processor and a memory, the memory storing a program or instructions operable to run on the processor, the program or instructions performing the steps of the method according to the first aspect and / or the second aspect when executed by the processor, wherein the communications device is a network side device or a terminal device.

[0015] According to a ninth aspect, there is provided a readable storage medium having stored thereon a program or instructions which, when executed by a processor, perform the steps of the method according to the first and / or second aspects.

[0016] According to a tenth aspect, there is provided a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor running a program or instructions and adapted to implement the method according to the first aspect and / or the second aspect.

[0017] An eleventh aspect provides a computer program / program product, the computer program / program product being stored on a storage medium, the computer program / program product being executed by at least one processor to implement the steps of the methods of the first and / or second aspects. [Effects of the Invention]

[0018] In an embodiment of the present application, a first network side device sends a first data acquisition request to a second network side device, and the first data acquisition request is used to request target data. The second network side device sends storage information to the first network side device. The first network side device acquires the target data based on the storage information and analyzes the target data to obtain performance information of the target model.

[0019] In this way, the first network side device can obtain the storage information of the target data from the second network side device, and then obtain the target data. By analyzing the target data, it can obtain performance information of the target model, and further estimate the accuracy of the target model in the current actual inference process, which can be used to adjust subsequent policy decisions based on the target model, thereby improving the accuracy rate of policy decisions made by devices inside and outside the network based on the model inference results of the target model. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a block diagram of a wireless communication system to which an embodiment of the present application can be applied. [Figure 2] 1 is a flowchart illustrating an implementation of a data acquisition method in an embodiment of the present application. [Figure 3] 1 is a flowchart of a specific example of data acquisition in an embodiment of the present application. [Figure 4] 10 is a flowchart of another specific example of data acquisition in an embodiment of the present application. [Figure 5] 1 is a flowchart illustrating another data acquisition method according to an embodiment of the present application. [Figure 6] 3 is a structural schematic diagram of a data acquisition device corresponding to FIG. 2 in an embodiment of the present application; [Figure 7] 6 is a structural schematic diagram of a data acquisition device corresponding to FIG. 5 in an embodiment of the present application. [Figure 8]1 is a schematic diagram of a data acquisition system in an embodiment of the present application. [Figure 9] FIG. 2 is a schematic diagram of another data acquisition system in accordance with an embodiment of the present application. [Figure 10] FIG. 2 is a structural schematic diagram of a network-side device in an embodiment of the present application; [Figure 11] FIG. 2 is another structural schematic diagram of a network-side device in an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0021] The following clearly and completely describes the technical solutions in the embodiments of the present application, in conjunction with the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application fall within the scope of protection of the present application.

[0022] The terms "first," "second," etc. in the specification and claims of this application are intended to distinguish between similar objects and are not intended to describe a particular order or sequence. It should be understood that terms used in this manner are interchangeable where appropriate, so that embodiments of this application may be performed in orders other than those illustrated or described herein, and that objects distinguished by "first" and "second" are generally of the same type and do not limit the number of objects; for example, a first object may be one or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the related objects.

[0023] It should be noted that the techniques described in the embodiments of the present application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be applied to other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-Carrier Frequency Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in the embodiments of the present application are always used interchangeably, and the described techniques may be used in the above-mentioned systems and radio technologies, as well as other systems and radio technologies. The following description describes a New Radio (NR) system for illustrative purposes, and NR terminology is used in most of the following description. However, these techniques may also be used in applications other than NR system applications, such as sixth generation (6G) networks. th This may be applied to 6G (6th Generation) communication systems.

[0024] 1 is a block diagram of a wireless communication system to which the embodiment of the present application can be applied. The wireless communication system includes a terminal 11 and a network side device 12. Here, the terminal 11 may be a terminal-side device such as a mobile phone, a tablet personal computer, a laptop computer (also called a notebook computer), a personal digital assistant (PDA), a palmtop computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, a vehicle-mounted equipment (VUE), a pedestrian-mounted equipment (PUE), a smart home (home devices with wireless communication capabilities, such as a refrigerator, television, washing machine, or furniture), a game console, a personal computer (PC), a teller machine, or a self-service machine, and the wearable device includes a smart watch, a smart band, a smart earphone, a smart glasses, a smart accessory (smart bracelet, smart bracelet, smart ring, smart necklace, smart anklet, smart wristband, smart wristband, smart clothing, etc.). It should be noted that the embodiments of the present application do not limit the specific type of the terminal 11. The network side equipment 12 may include access network equipment or core network equipment, where the access network equipment 12 may also be called radio access network equipment, radio access network (RAN), radio access network function or radio access network unit.The access network equipment 12 may include a base station, a WLAN access point, a WiFi node, or the like. The base station may be called a Node B, an evolved Node B (eNB), an access point, a base transceiver station (BTS), a radio base station, a radio transceiver, a basic service set (BSS), an extended service set (ESS), a home B node, a home evolved B node, a transmitting and receiving point (TRP), or any other appropriate term in the field. As long as the same technical effect is achieved, the base station is not limited to a specific technical term. For illustrative purposes, in the embodiments of this application, only base stations in an NR system are taken as examples, and the specific type of base station is not limited.Core network devices include core network nodes, core network functions, mobility management entities (MMEs), access and mobility management functions (AMFs), session management functions (SMFs), user plane functions (UPFs), policy control functions (PCFs), policy and charging rules functions (PCRFs), edge application server discovery functions (EASDFs), unified data management (UDMs), unified data repository (UDRs), home subscriber servers (HSSs), centralized network configuration (CNCs), network repository functions (NRFs), network exposure functions (NEFs), local NEFs (or L-NEFs), binding support functions (BSFs), and application functions (Application Node Functions). The core network device may include, but is not limited to, at least one of a QoS function (e.g., QoS Function, QoS Function, QoS Control ...

[0025] For ease of understanding, application scenarios of embodiments of the present application, related technologies and concepts will be described first.

[0026] The embodiments of the present application can be applied to a scenario in which an analytical task is performed using an artificial intelligence / machine learning model, such as a target model. The target model is a model that can meet the requirements of a selected or trained analytical task. During actual use, corresponding input data can be input into the target model to obtain corresponding output data, i.e., a data analysis result corresponding to the analytical task can be obtained. The data analysis result may include predictive information for a future time period or time node, or summary information of historical data for a previous time period or time node. The accuracy of the data analysis result can be determined based on performance information of the target model during actual use.

[0027] As described in the background art, the network data analysis function can perform intelligent data analysis for specific tasks and generate data analysis results, which can assist internal and external devices in the communication network to make policy decisions. The network data analysis function can be divided into two network elements: the Analytics Logical Function (AnLF) and the Model Training Logical Function (MTLF).

[0028] wherein the analytical logic function can perform inferences to generate predictive information or generate summary information of historical data, for example, to provide inferential analytical services based on consumer requests; The model training logic function can generate models and perform model training, for example, to provide artificial intelligence / machine learning model training services based on consumer requests.

[0029] In one specific example, the interaction process between the analysis logic function (AnLF) and the model training logic function (MTLF) is as follows:

[0030] The AnLF can request information of one or a set of ML models associated with one or a set of task identifiers from the MTLF by calling an ML model information request, e.g., Nnwdaf_MLModelInfo_Request, and the AnLF performs data analysis using the ML model information; When the MTLF receives the ML model information request sent by the AnLF, it can determine whether a conventional ML model can be used for this request or whether the conventional ML model needs to be further trained. If further training is required, the MTLF initiates a data collection operation to collect data from network side devices such as a Network Function (NF), an Application Function (AF), or an Operation, Administration, and Maintenance (OAM) and use it for ML model training. The network function may be an AMF, a Data Collection Coordination Function (DCCF), an Analytics Data Repository Function (ADRF), etc. The MTLF can request a response by calling ML model information, for example, by calling Nnwdaf_MLModelInfo_Request response to provide ML model information to the AnLF, where the ML model information includes information such as the file address of one or a set of ML models.

[0031] In another specific example, the interaction process between the Network Data Analysis Function (NWDAF) and the Data Analysis Repository Function (ADRF) is as follows:

[0032] The NWDAF can send data and / or analysis results to the ADRF by invoking a data management storage request, such as Nadrf_DataManagement_StorageRequest.

[0033] The ADRF stores the data and / or analysis results transmitted by the NWDAF. The ADRF may determine whether the same data and / or analysis results have already been stored or are stored based on the data and / or analysis results transmitted by the NWDAF, and if the same data and / or analysis results have already been stored or are stored, the ADRF may determine not to store the data and / or analysis results transmitted by the NWDAF.

[0034] The ADRF sends data and / or analysis result storage information to the NWDAF by invoking a data management storage request response, such as an ADRF_DataManagement_StorageRequest response.

[0035] The above interaction process between the NWDAF and ADRF can also be applied to the interaction process between the DCCF, Messaging Framework Adaptor Function (MFAF), and ADRF.

[0036] As can be seen from the above description, the MTLF is mainly used to generate an AI / ML model, perform model training, and provide the trained model to the AnLF, the AnLF is used to put the model into actual use and perform inference to obtain predictive information or generate summary information of historical data, and the ADRF is mainly used for data storage. For any one AI / ML model, the accuracy achievable at different stages of the model may be different, i.e., the accuracy achievable at the training stage of the model may be different from the accuracy achievable at the actual use stage of the model. There may be differences in the accuracy achievable at different stages of the model due to factors such as different data distributions at different stages and insufficient generalization ability of the model. Generally, the accuracy achievable at the actual use stage of the model is lower than the accuracy achievable at the training stage of the model.

[0037] Therefore, AnLF needs to know the performance information of the model during the actual use phase, and then determine the accuracy of the data analysis results of the model during the actual use phase based on the performance information. If the accuracy rate of the data analysis results is high, the results can be provided to internal and external devices of the communication network to assist in their policy decisions, ensuring that the internal and external devices of the communication network make accurate policy decisions.

[0038] The above describes the application scenarios of the embodiments of the present application, as well as the related technologies and concepts. Below, we will explain in detail the data acquisition method according to the embodiments of the present application through several embodiments and their application scenarios, in conjunction with the drawings.

[0039] The following describes in detail the data acquisition method according to the embodiments of the present application through several examples and its application scenarios in conjunction with the drawings.

[0040] Referring to FIG. 2, a step flowchart of the data acquisition method of the present application is shown, which may specifically include the following steps:

[0041] In step S11, the first network side device transmits a first data acquisition request to the second network side device.

[0042] In an embodiment of the present application, the first network side device may be any one of network side devices that can detect performance information of the target model, and in one embodiment, the first network side device includes a model training logic function MTLF, and the second network side device may be any one of network side devices that needs to obtain the target model and perform actual data analysis using the target model, and in one embodiment, the second network side device includes an analysis logic function AnLF. The first data acquisition request is used to request target data.

[0043] In this step, the first network side device sends a first data acquisition request to the second network side device, notifies the second network side device that it intends to analyze the performance information when the target model is actually used, and needs to analyze the required target data for the performance information of the target model, thereby requesting the second network side device to assist it in preparing the target data.

[0044] Here, the target data is used to determine performance information of the target model, and may include any one or more of input data, output data, and label data when the target model is actually used. Here, the input data is input data that is input to the target model in a usage process or inference process of the target model to perform inference. The output data is output data / predicted value / inference data, etc. obtained after the target model performs calculation / prediction / inference using a set of input data, and may be predicted value, inference data, or inference result data. The label data includes actually generated data related to the input data and / or output data corresponding to the target model, and may be ground truth, i.e., accurate label values, or may be understood as actual measured values ​​in a real environment.

[0045] For example, the target data may include label data corresponding to the target model, and input data and / or output data corresponding to the target model, i.e., the target data may include label data corresponding to the target model and input data corresponding to the target model, or the target data may include label data corresponding to the target model and output data corresponding to the target model, or the target data may include label data corresponding to the target model, input data corresponding to the target model and output data corresponding to the target model.

[0046] For example, the input data corresponding to the target model is terminal location data of a certain cell within a set time period, and the analysis task corresponding to the target model is to predict the number of terminals in this cell within one week in the future. The input data can be input into the target model to obtain corresponding output data, which is the predicted information of the number of terminals in this cell within one week in the future. The actual number of terminals in this cell after one week can be obtained, and this actual number of terminals can be used as label data corresponding to the target model, and this label data has an association with the corresponding input data and output data.

[0047] The first network-side device may send a first data acquisition request to the second network-side device in different cases. For example, in one implementation, the first network-side device may send a first data acquisition request to the second network-side device after receiving a model acquisition request for a target model of one of the devices. That is, when another device requests the target model from the first network-side device, the first network-side device may send a first data acquisition request to the second network-side device, and further analyze the target data to determine whether the current target model can be sent to the other device for use.

[0048] Alternatively, in one implementation, the first network side device may send a first data acquisition request to the second network side device a predetermined time after sending the target model to the second network side device, that is, the first network side device may send a first data acquisition request to the second network side device after the target model has been used for too long, and further analyze the target data to determine whether the performance information of the current target model is suitable for continued use, or whether the current target model needs to be updated, thereby improving the accuracy of the target model.

[0049] In addition, in one implementation method, the first network side device may send a first data acquisition request to the second network side device according to a preset period, that is, the first network side device may periodically trigger the second network side device to send a first data acquisition request, and further analyze the target data to periodically detect performance information of the target model, thereby updating the target model in a timely manner, thereby improving the accuracy of the target model.

[0050] In the present application, the first data acquisition request carries at least one of the following: (1) Identifier information for the analysis task corresponding to the target model, such as an Analytics ID, which is used to specify the analysis task for the target data.

[0051] (2) Analytic task condition restriction information, such as analytic filter information, is used to specify restriction conditions for the target data of an analytical task, such as filtering the target data of an analytical task based on the Area of ​​Interest (AOI), Single Network Slice Selection Assistance information (S-NSSAI), Data Network Name (DNN), etc.

[0052] (3) Target information for the analytical task, such as Target of analytic reporting, is used to indicate the target of the analytical task, and may be, for example, a certain device, multiple devices, or all devices.

[0053] (4) Model identifier information corresponding to the target model, such as Model ID, is used to indicate that the target data is for a certain model, and in the embodiment of the present application, the model for the target data is the target model.

[0054] (5) Data type information of target data, which is used to instruct which data type of target data to acquire, for example, to instruct to acquire target data of terminal location information type.

[0055] (6) Time information of the target data: Used to indicate the time period for which target data is to be acquired, this time information may include a start time node and an end time node for acquiring the target data, and may indicate that data between the start time node and the end time node of the generation time is to be acquired as the target data, depending on the generation time of the data, for example.

[0056] (7) Location information of target data: This is used to indicate within which location range target data should be acquired. For example, depending on the location at the time of data generation, it may be possible to instruct to acquire target data within this location range, such as target data within a certain tracking area (TA) or cell.

[0057] (8) First instruction information, which is used by the first network side device to instruct the collection source of the desired target data, i.e., the device from which the first network side device should acquire the target data, where the collection source of the target data includes any one or more of the second network side device, the third network side device, and the fourth network side device, where the third network side device is used to store the target data and may be any one storage network element for storing the target data, such as an ADRF, and the fourth network side device is a data source of the target data and may be a data source network element or a database of the target model, but is not specifically limited.

[0058] In addition, the first data acquisition request may carry second indication information for indicating that the first network side device requests acquisition of target data, and the second indication information may be explicit, for example, carried by a specific cell, or implicit, for example, expressed by the name of the first data acquisition request. After receiving the first data acquisition request sent from the first network side device, the second network side device can know, based on the second indication information included in the first data acquisition request, that the first network side device requests to acquire target data for analyzing performance information of the target model, and can further perform corresponding operations.

[0059] In step S12, the first network side device receives the stored information from the second network side device, and acquires the target data based on the stored information.

[0060] In an embodiment of the present application, after receiving a first data acquisition request sent from the first network side device, the second network side device may send stored information to the first network side device, where the stored information may be for a target model, for a target task corresponding to the target model, or for target data, and is used to instruct the first network side device to acquire the target data. Here, the target data may be stored in the second network side device itself, or may be stored in another device, such as a fourth network side device corresponding to the target model or a third network side device that stores the target data. In this way, the first network side device can acquire target data from different devices in different cases.

[0061] For example, in one implementation method, the step of the second network side device sending stored information to the first network side device may include the second network side device obtaining target data and the second network side device sending stored information to the first network side device, wherein the stored information includes the target data.

[0062] For example, if the first data acquisition request carries first instruction information and the first instruction information indicates that the collection source of the target data includes a second network side device, the above-mentioned implementation method may be adopted, or the second network side device may decide on its own whether to adopt the above-mentioned implementation method. Here, the step of the second network side device acquiring the target data may include the second network side device determining a fourth network side device corresponding to the target model based on task information sent from the user equipment, where the task information is used to instruct the first network side device to train to obtain the target model, and then the second network side device acquires the target data from the fourth network side device. For example, the task information may include condition restriction information, identifier information, and target information of the analysis task, etc. In this way, the second network side device can determine the fourth network side device based on the task information, acquire the target data from the fourth network side device, and directly transmit the target data to the first network side device.

[0063] Here, the user equipment, i.e., a consumer network element, such as an SMF, AMF, UPF, or PCF, may send a task request to a second network-side device according to actual needs. The task request includes task information and is used to request the execution of one analysis task and obtain a data analysis result for the analysis task. After receiving the task request, the second network-side device may obtain a target model for executing the analysis task, for example, send a model request to the first network-side device, which is used to request the acquisition of the target model. The first network-side device selects or trains a target model that meets the requirements for executing the analysis task, and then sends it to the second network-side device. After obtaining the target model, the second network-side device may put the target model into actual use, and during actual use, output data corresponding to the input data of the analysis task can be obtained based on the target model and corresponding input data.

[0064] That is, after receiving the first data acquisition request, the second network side device determines the fourth network side device corresponding to the target model based on the task information, and then acquires target data, such as input data and / or label data corresponding to the target model, from the fourth network side device, and then sends the target data to the first network side device based on the storage information, or the first network side device may acquire the target data from the second network side device based on the storage information.

[0065] In another implementation manner, the step of the second network side device sending the stored information to the first network side device may include the second network side device obtaining the target data, and the second network side device storing the target data in a third network side device corresponding to the target data, and sending the stored information to the first network side device, wherein the stored information includes instruction information for the third network side device.

[0066] For example, if the first data acquisition request carries first instruction information and the first instruction information indicates that the collection source of the target data includes a third network side device that stores the target data, the above-mentioned implementation method may be adopted, or the second network side device may decide by itself whether to adopt the above-mentioned implementation method. In this way, the first network side device can determine the third network side device based on the stored information, and further acquire the target data from the third network side device.

[0067] In other words, after acquiring the target data, the second network side device stores the target data in a third network side device for storing target data, and then sends instruction information of the third network side device to the first network side device, so that the first network side device can acquire the target data from the third network side device.

[0068] Here, the step of the second network side device storing the target data in the third network side device may include the second network side device sending data storage instruction information, in which the target data is carried, to the third network side device, and the second network side device receiving storage identifier information returned from the third network side device, wherein the storage identifier information is used to indicate that the target data has been successfully stored and to retrieve the target data.

[0069] That is, the second network side device stores the target data in the third network side device according to the data storage instruction information, and after the third network side device successfully stores the target data, it returns storage identifier information to the second network side device to notify the second network side device that it has successfully stored the target data in the third network side device, and at the same time, the second network side device may further obtain the target data based on the storage identifier information, for example, the storage identifier information may include an index of the target data, etc.

[0070] In an embodiment of the present application, the data storage instruction information carries at least one of the following: (1) Input data corresponding to the target model.

[0071] (2) Output data corresponding to the target model.

[0072] (3) Label data corresponding to the target model.

[0073] (4) Time information corresponding to the input data. For example, the time information corresponding to the input data may include at least one of time node information when the input data is generated, time node information when the input data is acquired by the second network side device, and time node information when the input data is input to the target model.

[0074] (5) Time information corresponding to the output data. For example, the time information for the output data may include at least one of time node information at the time of generating the output data, target time node information for the output data, and time information corresponding to input data related to the output data.

[0075] (6) Time information corresponding to label data. For example, the time information corresponding to label data may include at least one of time node information when the label data is generated and time node information when the label data is acquired by the second network side device.

[0076] Here, the time node at the time of output data generation may be the same as the time node at the time when input data is input to the target model, or may be the time node at which output data is generated after input data is input to the target model and calculations of the target model are completed. The target time node for the output data may be a time node corresponding to the prediction information included in the output data. For example, the prediction information included in the output data is the number of terminals in a certain cell one week from now, and the target time node is one week from now. The target time node corresponds to label data generation. The output data and label data can be associated based on the target time node for the output data and the time node at the time of label data generation.

[0077] The time information corresponding to the input data related to the output data may include at least one of time node information when the input data related to the output data was generated, time node information when the input data related to the output data was acquired by the second network side device, and time node information when the input data related to the output data was input to the target model. The output data and the input data can be associated based on the time information corresponding to the output data related to the output data.

[0078] (7) Identifier information of the analysis task corresponding to the target model.

[0079] (8) Conditional information for the analysis task.

[0080] (9) Target information for the analysis task.

[0081] (10) Model identifier information corresponding to the target model.

[0082] (11) Model filtering information corresponding to the target model, which is used to indicate the filtering conditions that the target model needs to meet, such as AOI, S-NSSAI, DNN, etc.

[0083] In addition, the step of the second network side device storing the target data in the third network side device corresponding to the target model may further include the second network side device sending data acquisition instruction information to the third network side device to instruct the third network side device to request the target data from the fourth network side device, and the second network side device receiving acquisition identifier information returned from the third network side device, where the acquisition identifier information is used to indicate that the third network side device has already acquired and stored the target data from the fourth network side device.

[0084] In other words, the second network side device notifies the third network side device of the fourth network side device that stores the target data using data acquisition instruction information, and then the third network side device acquires the target data from the fourth network side device and, after successfully storing the target data, returns acquisition identifier information to the second network side device to notify the second network side device that the target data has been successfully stored in the third network side device.

[0085] In an embodiment of the present application, the data acquisition instruction information carries at least one of the following: (1) Identification information of the fourth network side device, for example, the ID of the fourth network side device, which is used to indicate the data source of the target data.

[0086] (2) Address information of the fourth network device, such as the IP address or fully qualified domain name (FQDN) of the fourth network device, which is used to instruct how to establish a connection with the fourth network device.

[0087] (3) Data type information of the target data.

[0088] (4) Time information of the target data.

[0089] (5) Location information of target data.

[0090] (6) Target data collection time information.

[0091] That is, the second network side device may be responsible for collecting all the target data and then directly store all the target data in the third network side device, or the second network side device may not be responsible for collecting the target data, but may send instruction information for the fourth network side device to store the target data to the third network side device, and then the third network side device may obtain the target data from the fourth network side device.

[0092] In addition, in one implementation method, the above two methods may be performed simultaneously, that is, the second network side device is responsible for collecting some of the target data, and then sends the collected target data and instruction information for the fourth network side device to a third network side device corresponding to the target model, and simultaneously sends the collected target data and instruction information for the fourth network side device to the third network side device, so that the third network side device obtains other part of the target data from the fourth network side device.

[0093] In both of the above two implementation methods, the second network side device may acquire target data from the fourth network side device, and the second network side device may further process the acquired target data. For example, if the target data includes output data of the target model, the second network side device's acquisition of the target data from the fourth network side device includes the second network side device acquiring input data of the target model from the fourth network side device, and the second network side device inputting the input data into the target model and processing it to obtain corresponding output data.

[0094] In another implementation manner, the step of the second network side device sending the stored information to the first network side device may include the second network side device sending the stored information including instruction information of the fourth network side device to the first network side device, or the second network side device sending the stored information including task information sent from the user equipment to the first network side device, where the task information is used to instruct the first network side device to train to obtain the target model.

[0095] For example, if the first data acquisition request carries first instruction information and the first instruction information indicates that the collection source of the target data includes a fourth network side device corresponding to the target model, the above implementation method may be adopted, or the second network side device may decide whether to adopt the above implementation method by itself. Here, the second network side device may acquire instruction information for the fourth network side device based on task information sent from the user device. In this way, the first network side device may determine the fourth network side device based on stored information, and further acquire the target data from the fourth network side device.

[0096] That is, the first network side device may determine the fourth network side device based on the stored information, and then obtain the target data from the fourth network side device. For example, the second network side device may directly send instruction information of the fourth network side device to the first network side device, or the second network side device may send task information to the first network side device, and the first network side device may obtain and determine information of the fourth network side device from the task information. There is no specific limitation.

[0097] In an embodiment of the present application, the stored information carries at least one of the following: (1) Identification information of the third network side device, such as the ID of the third network side device, which is used to indicate in which third network side device the target data is stored; (2) Address information of the third network device, such as the IP address or fully qualified domain name (FQDN) of the third network device, which is used to instruct the first network device how to establish a connection with the third network device; (3) Identification information of the second network side device, for example, the ID of the second network side device, (4) Storage identifier information of the target data, such as a Storage Transaction Identifier, which is returned by the third network side device to the second network side device to indicate that the target data has been successfully stored in the third network side device. It may be used in combination with the identifier information of the third network side device to more accurately obtain the target data, and may specifically be information such as an index.

[0098] In one implementation, the first network side device may further receive related information from the second network side device, and can more accurately obtain the target data based on the stored information and the related information, where the related information includes: (1) Identifier information of an analysis task corresponding to a target model; (2) analysis task condition limitation information that indicates the limitation conditions for the target data of the analysis task; (3) Target information for the analysis task; (4) model identifier information corresponding to the target model; (5) model filtering information corresponding to the target model; (6) data type information of the target data; (7) time information of the target data; (8) Location information of the target data.

[0099] For example, the second network side device, the third network side device, or the fourth network side device may store a large amount of other data in addition to the target data, and the first network side device needs to obtain the target data from the large amount of data, so the information carried in the storage information needs to have an indication effect on the target data, and after obtaining the storage information, the first network side device can accurately obtain the target data from the large amount of data based on the information carried in the storage information, such as the data type information, time information, and location information of the target data. These data may all be data related to the target model, for example, output data within a relatively long time period of the target model, and the first network side device may therefore determine the target data from these output data based on the time information of the target data carried in the storage information, or these data may include data unrelated to the target model, and the first network side device may therefore determine the target data from these data based on the time information, location information, and data type information of the target data carried in the storage information, and there are no specific limitations.

[0100] In one implementation, the first network side device may further receive requirement information from the second network side device and determine subsequent steps to be performed based on the requirement information. Here, the requirement information is used to indicate minimum requirements for the performance information of the target model, for example, the requirement information may be information such as a threshold value, a condition, etc. The second network side device may use the requirement information to notify the first network side device under what circumstances the performance information of the target model is acceptable. After the first network side device analyzes the target data and obtains the performance information of the target model, if the performance information does not satisfy the requirement information, it may trigger some subsequent operations, such as retraining the target model, reselecting the target model, or notifying the second network side device that the performance of the target model has deteriorated or does not meet expectations. Conversely, if the performance information of the target model can satisfy the requirement information, it may trigger other operations, such as notifying the second network side device that the performance of the target model meets expectations, or no subsequent operations may be performed, for example, no target model needs to be updated or no target model needs to be reselected. In one implementation, the request information may be carried in a model acquisition request sent by the second network side device to the first network side device.

[0101] In one implementation, the storage information carries identifier information and / or address information of the third network side device, and therefore the step of the first network side device acquiring the target data based on the storage information may include the first network side device determining the third network side device based on the storage information, the first network side device sending a second data acquisition request for the target data to the third network side device, and the first network side device receiving the target data returned from the third network side device.

[0102] That is, the first network side device can determine a third network side device based on the stored information, and then send a second data acquisition request to the third network side device that stores the target data to acquire the target data. The target data may also be stored in a fourth network side device, and the process by which the first network side device acquires the target data from the fourth network side device based on the stored information is similar to the process of acquiring the target data from the third network side device, and will not be further described here.

[0103] Here, the second data acquisition request includes: (1) Identifier information of the second network side device, for example, the ID of the second network side device, for indicating which device the target data is stored in the third network side device; (2) memory identifier information of the target data; (3) Identifier information of the analysis task corresponding to the target model; (4) analysis task condition limitation information that indicates the limitation conditions for the target data of the analysis task; (5) Target information for the analysis task; (6) model identifier information corresponding to the target model; (7) data type information of the target data; (8) time information of the target data; (9) At least one of the following is carried: location information of the target data;

[0104] In another implementation, the target data may be carried in the storage information, and the step of the first network side device acquiring the target data based on the storage information may include the first network side device acquiring the target data from the storage information. In other words, the second network side device may store information and directly transmit the target data to the first network side device.

[0105] Here, what information is carried in the stored information, or which device the first network side device acquires the target data from, may be determined based on first instruction information carried in the first data acquisition request sent from the first network side device, or may be determined by the second network side device itself. Specifically, if the first instruction information indicates that the collection source of the target data desired by the first network side device includes the fourth network side device and / or the third network side device, information of the fourth network side device and / or the third network side device, such as identifier information and / or address information, may be carried in the stored information returned from the second network side device, so that the first network side device can acquire the target data from the fourth network side device and / or the third network side device. If the first instruction information indicates that the collection source of the target data desired by the first network side device includes the second network side device, the target data stored in the second network side device may be carried in the stored information returned from the second network side device, so that the second network side device directly transmits the target data to the first network side device.

[0106] In one implementation, the stored information further carries time zone information, which is used to indicate the acquisition time and / or acquisition time range of the target data, and the step of the first network side device acquiring the target data based on the stored information includes the first network side device acquiring the target data at a time corresponding to the time zone information based on the stored information. For example, the target data is for a certain period in the future, and only performance information for that period is required, so the time information may indicate that the first network side device should collect the target data once after this time. As another example, if data for a certain period needs to be used to calculate model performance multiple times, a time range for a certain period may be specified, and this information may be determined by the second network side device itself or indicated in a request from the first network side device received by the second network side device.

[0107] In step S13, the first network side device analyzes the target data and obtains performance information of the target model.

[0108] After acquiring the target data, the first network side device can analyze the target data, determine performance information of the target model within a time period corresponding to the target data, and determine subsequent operations based on the performance information. For example, the first network side device can calculate performance information of the target model by comparing the label data with the output data, and further determine whether the target model needs to be retrained or a new target model needs to be selected based on the performance information. Alternatively, the first network side device can compare the label data with the output data to determine whether to feed back the performance information to the second network side device. Alternatively, after calculating the performance information of the target model, the first network side device can notify the performance information to the second network side device, or periodically feed back the performance information of the target model to the second network side device.

[0109] For example, after the first network side device analyzes target data and obtains performance information of the target model, if the performance information satisfies a preset condition, the first network side device may transmit the target model to any one of the devices that requests the target model. Conversely, if the performance information does not satisfy the preset condition, the first network side device may retrain the target model and provide the retrained target model to other devices that request the target model for actual use. Alternatively, the first network side device may select a target model again and provide the reselected target model to other devices for actual use.

[0110] Here, the performance information satisfying the predetermined condition may refer to the performance information of the target model conforming to the requirement information for the performance information of the target model carried in the storage information. For example, the performance information of the target model may refer to the difference between the output data and label data of the target model. Therefore, if this difference is smaller than the predetermined difference in the requirement information, the performance information may be considered to satisfy the predetermined condition. Alternatively, the performance information may refer to the variance between the output data of the target model. Therefore, if the variance is smaller than the predetermined variance in the requirement information, it indicates that the output data is stable and the performance information may be considered to satisfy the predetermined condition. Alternatively, the predetermined condition may be determined by the first network side device regardless of the requirement information carried in the storage information, and is not specifically limited.

[0111] In addition, after the first network side device analyzes the target data and obtains performance information of the target model, the first network side device can update the model parameters of the target model based on the performance information, thereby improving the accuracy of the model in the actual use stage.

[0112] In an embodiment of the present application, the time information corresponding to the target data can assist the first network side device in analyzing the target data. For example, by comparing the time information corresponding to the output data with the time information corresponding to the label data, the output data of the model can be bound or mapped to the label data corresponding to the output data to obtain performance information of the target model. For example, the time information corresponding to the input data can be used to know the time range information covered by these data. The time range information refers to a certain time range. For example, if the time information corresponding to the input data shows that the earliest time is Monday of a certain week and the latest time is Friday of the same week, it indicates that the time range covered by these data is from Monday to Friday of a certain week. This time range can be used to record and ensure the timeliness of model performance.

[0113] In one embodiment of the present application, the target data may include label data corresponding to the target model, and input data and / or output data corresponding to the target model, and the first network side device analyzing the target data and obtaining performance information of the target model may include the first network side device comparing each output data with associated label data respectively and obtaining a comparison result, and the first network side device determining the performance information of the target model based on the comparison result.

[0114] In one implementation manner, after the first network side device obtains the target data, if the target data includes label data corresponding to the target model and input data corresponding to the target model, the first network side device can obtain output data corresponding to the target model based on the target model and the input data corresponding to the target model. If the target data includes label data corresponding to the target model and output data corresponding to the target model, or if the target data includes label data corresponding to the target model, input data corresponding to the target model, and output data corresponding to the target model, the first network side device obtains the output data corresponding to the target model after obtaining the target data.

[0115] The association relationship between the output data and the label data can be determined based on the time information of the output data corresponding to the target model and the label data corresponding to the target model. The first network side device can compare each output data with the associated label data to obtain a comparison result. Based on the comparison result, the accuracy rate of each output data compared with the associated label data can be known, thereby determining performance information of the target model. The first network side device feeds back the performance information of the target model to the second network side device, and the second network side device can determine further operations based on the performance information.

[0116] In one implementation, after the first network side device analyzes the target data and obtains performance information of the target model, the method includes: The method further includes, when the performance information does not reach the model performance requirement corresponding to the model performance requirement information, the first network side device feeding back the performance information to the second network side device.

[0117] In one implementation, the performance information of the target model includes an accuracy rate. Specifically, the model performance information of the target model may be used to indicate the accuracy and / or error rate of the inference result when the target model actually performs inference. The performance format of the model performance information may be various, for example, a specific percentage value such as 90%, a classification expression format such as high, medium, low, etc., or normalized data such as 0.9. The embodiments of the present application do not specifically limit the performance format of the model performance information. The model performance information of the target model may indicate the accuracy or error rate of the inference result of the target model for the target task from either the front or back side. For example, the inference accuracy of the target model may be indicated from the back side by calculating the inference error or inference error rate of the target model. Here, the inference error or inference error rate may be calculated in various ways, for example, the mean absolute error (MAE), the mean square error (MSE), etc.

[0118] For ease of understanding, the following will again describe the embodiments of the present application using the specific example shown in Figure 3, taking as an example the second network side device is an AnLF, the first network side device is an MTLF, the third network side device is an ADRF, and the fourth network side device is a data source network element.

[0119] Step 0: The related flow of task initiation, model selection, and model distribution. Specifically, the consumer network element (NF) initiates a task request to the AnLF to request obtaining the task result of the analysis task. The AnLF sends a model request to the MTLF to request obtaining a target model for performing the analysis task. The MTLF selects or trains a target model that meets the requirements for performing the analysis task and sends it to the AnLF.

[0120] Step 1: The MTLF sends a first data acquisition request to the AnLF, informing the AnLF that it intends to calculate the performance information of the target model in actual use and requires related target data, thereby requesting the AnLF to assist it in preparing the target data.

[0121] The target data may include input data, output data, and label data when the target model is actually used. The input data is input data that is input to the target model during the target model usage process / inference process to allow the target model to perform inference. The output data may be a prediction, inference data, or inference result data, i.e., output data / prediction value / inference data obtained after inputting a set of input data into the target model and performing calculation / prediction / inference. The label data may be understood as actual measurements in a real environment and may include ground truth, i.e., accurate label values. In general, a set of data may include a set of input data, output data corresponding to the input data, and label data corresponding to the input data.

[0122] The circumstances that may trigger the MTLF to send a first data acquisition request to the AnLF are as follows: After a certain period of time, for example, when the target model has been used for a certain period of time and another AnLF requests the target model from the MTLF, the MTLF may first request target data of the target model from the AnLF and calculate performance information, and then further determine operations such as whether to transmit the target model based on the performance information of the target model; If the target model is used for too long, new test data needs to be collected to update the performance information and model during model training, for example, updating target models with different granularities based on different domains; Alternatively, it may be periodically triggered, and the MTLF compares target data with the target model, which it is responsible for actively requesting.

[0123] Here, the first data acquisition request includes: Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; data type information of the target data; time information of the target data; location information of the target data; At least one of first instruction information is carried, the first instruction information being used to indicate the source of collection of the target data desired by the first network side device, and the source of collection of the data to be processed being first instruction information including any one or more of the second network side device, a fourth network side device corresponding to the target model, and a third network side device that stores the target model data.

[0124] Step 2: The method by which the AnLF determines to collect target data may be specifically determined based on the data type information, time information, location information and / or first instruction information of the target data in the first data acquisition request in step 1. For example, if first instruction information is carried in the first data acquisition request, the collection source of the target data indicated by the first instruction information may be used.

[0125] Step 3: The AnLF collects relevant data from the fourth network side device. The AnLF can determine the target fourth network side device based on the task information. This task information may be carried in the task request sent by the consumer network element in step 0. Then, the AnLF acquires relevant data from the target fourth network side device. Here, the relevant data acquired in this step may include input data and / or label data. After collecting the input data, the AnLF can input the input data into the target model, and obtain corresponding output data through the operation of the target model.

[0126] Step 4: The AnLF sends data storage instruction information to the ADRF to store the target data in the ADRF. Specifically, the AnLF sends the target data, including input data, output data, and label data, and related information of the target data, such as task information corresponding to the target data, model information corresponding to the target data, and data information corresponding to the target data, to the ADRF according to the data storage instruction information, so as to store the target data and / or related information of the target data in the ADRF.

[0127] The time information corresponding to the target data can assist in the detection of subsequent performance information. For example, by comparing the time information corresponding to the output data with the time information corresponding to the label data, the output data of the target model and the label data corresponding to the output data can be associated / bound / mapped, thereby comparing / calculating / generating performance information. Furthermore, for example, the time information corresponding to the input data can provide information on the time range covered by these data. The time range information indicates a certain time range. For example, if the earliest time and the latest time of the time information corresponding to the input data are Monday and Friday of a certain week, respectively, this indicates that the time range covered by these data is from Monday to Friday of a certain week. This time range can be used to record and ensure the timeliness of the performance information.

[0128] In one of the implementation methods, the data storage instruction information may be sent by Nadrf_DataManagement_StorageRequest.

[0129] Specifically, the data storage instruction information sent by the AnLF to the ADRF is as follows: The input data, The output data, Label data, Time information corresponding to the input data, which may include time node information at the time of generating the input data, time node information at which the input data is acquired, and time node information at which the input data is input to the target model; time information corresponding to the output data, where the time information corresponding to the output data may include time node information at the time of generating the output data, and the time node information may be obtained by the time node when the input data is input to the target model, and may further be obtained by the time node when the input data is input to the target model and generates the output data after the calculation of the target model is completed; the time information corresponding to the output data may further include target time node information for the output data, and the target time node information may be obtained by the target time node corresponding to the task predicted by the output data; and the time information corresponding to the output data may further include an initial time node corresponding to the output data, and the initial time node may be obtained by the time information corresponding to the input data related to the output data; time information corresponding to the label data, which may include time node information at the time of generating the label data and time node information at the time of acquiring the label data; Identifier information of an analysis task corresponding to the target model, such as an Analytics ID, and time information corresponding to label data used to indicate that the target data is for a specific task; Conditional limitation information of an analysis task, such as analytic filter information, is used to indicate filtering information for target data, and includes, for example, AOI, S-NSSAI, DNN, etc.; Target information for the analytical task, e.g., Target of analytic reporting, used to indicate that the target of data analysis is a device, multiple devices, or all devices; and Model identifier information corresponding to the target model, for example, a Model ID, used to indicate that the target data is for a certain model; and and model filtering information corresponding to the target model, which is used to indicate conditions that the target model must satisfy, e.g., AOI, S-NSSAI, DNN.

[0130] Step 5: The ADRF stores the received target data.

[0131] Step 6: After the storage is completed, the ADRF may return information indicating the completion of storage, such as the storage identifier information stored this time, and may also feed back, for example, the Transaction Reference ID to the AnLF to notify that the storage was successful.

[0132] Step 7: The AnLF sends memory information to the MTLF to request the MTLF to assist in detecting the performance information of the target model. In this process, the memory information specifically includes: Identifier information of the ADRF, which may be specifically identifier information of a network element, such as a network element ID; and Address information of the ADRF, which may be specifically address information of a network element, such as an IP address, FQDN, etc. of the network element; Identifier information of the AnLF, which may be specifically identifier information of a network element, such as a network element ID; Storage identifier information, for example, storing a Transaction Reference ID, i.e., the storage identifier information returned from the ADRF in step 4; Data type information for indicating the data type of the target data to be acquired, such as the location information of the terminal; Time zone information of the target data, which is used to indicate the time of the target data to be acquired, and may be the time of generation of the data, such as a start time node, an end time node, etc.; The location range information corresponding to the target data is used to indicate the location range of the target data to be acquired, and may be determined based on the location at the time of data generation, for example, to acquire data within a certain TA, cell, or may include at least one of the following: location range information corresponding to the target data.

[0133] In addition, the AnLF can further send related information to the MTLF, so that the MTLF can more accurately obtain target data, where the related information includes but is not limited to: Requirement information for the analysis result of the target model, i.e., requirement information for performance information, such as thresholds, preset conditions, etc., is used to inform the MTLF of acceptable performance information. If the performance information calculated by the MTLF does not satisfy the model performance requirement information, some subsequent operations, such as model retraining, model reselection, degradation of AnLF model performance, or notification that the expectation is not met, may be triggered. This model performance requirement information may be notified to the MTLF when the AnLF sends a model request to the MTLF in step 0. Identifier information for the analysis task, such as an Analytics ID, which is used to indicate that the target data to be obtained is for a specific task; Analytical task condition limiting information, such as analytic filter information, is used to indicate filtering information for data analysis results, including AOI, S-NSSAI, DNN, etc. Target information for the analytical task, e.g., Target of analytic reporting, used to indicate that the target of data analysis is a device, multiple devices, or all devices; and Model identifier information, e.g., Model ID, used to indicate that the target data to be retrieved is for a certain model; Model filtering information: Specifies the conditions that the model must meet for the target data to be acquired, such as AOI, S-NSSAI, and DNN.

[0134] Steps 8-9: The MTLF sends a second data acquisition request to the ADRF to request acquisition of the target data. Specifically, the MTLF determines the identifier information and address information of the target ADRF based on the information in step 7, and sends a second data acquisition request to the target ADRF to acquire the target data. After receiving the second data acquisition request, the ADRF feeds back target data that meets the requirements to the MTLF. For example, the ADRF may find data corresponding to the memory identifier information based on the AnLF identifier information and memory identifier information, and feed this back to the MTLF. For example, the ADRF may determine specific data based on a combination of information related to the target data, such as task identifier information and model identifier information, and feed this back to the MTLF.

[0135] The second data acquisition request is AnLF identifier information, Memory identifier information for the target data; Identifier information of an analysis task corresponding to the target data; task condition limiting information for the analysis task; Target information for the analysis task; model identifier information corresponding to the target model; Data type information of the target data; time information of the target data; and location information of the target data.

[0136] Step 10: After acquiring target data in the actual environment of the target model, the MTLF calculates performance information to obtain the performance information, and determines subsequent operations based on the performance information. The MTLF can calculate an analysis result by comparing the label data with the output data. Furthermore, the MTLF can determine whether model retraining or new model selection is necessary based on the analysis result. For example, if the performance information does not meet the model performance requirements corresponding to the model performance requirement information, model retraining or new model selection can be performed. A comparison of these two pieces of information can determine whether to feed back the performance information to the AnLF. In one embodiment, if the performance information of the target model does not meet the model performance requirements corresponding to the model performance requirement information, the first network side device feeds back the performance information to the second network side device. For example, the performance information can be fed back to the AnLF when the performance information does not meet the model performance requirements corresponding to the model performance requirement information, or the performance information can be calculated and then notified to the AnLF, or periodic feedback can be performed.

[0137] The following describes the embodiment of the present application again with another example shown in Figure 4, in which the first data is part of the target data. This example has the same general flow as the previous example, with the differences being in steps 3, 4, 5 and steps 8, 9.

[0138] Specifically, in step 3 of this example, the AnLF is not responsible for collecting all relevant data, but may only collect input data, for example, without collecting label data. In step 4, the AnLF not only notifies the ADRF to store the target data (e.g., input data) it has collected (step 4a), but also needs to notify the ADRF to collect and store other target data (e.g., label data) in a designated fourth network-side device (steps 4b and 5).

[0139] If the AnLF only informs the ADRF of some of the target data, the AnLF may also send data acquisition instruction information to the ADRF, which is also used to instruct the ADRF to collect and store another part of the target data in the fourth network side device of the target data in step 4b, specifically: Identifier information of a fourth network side device, such as a network element ID; Address information of the fourth network side device, such as the IP address and FQDN of the network element, Data type information of the target data to indicate what data to collect, such as device location information; Time information of the target data, which is used to indicate the time when the data is to be collected, and may be the time when the data is generated, such as a start time node, an end time node, etc.; Location information of the target data, which is used to indicate the range of data to be collected and can be determined based on the location at the time of data generation, for example, a certain TA, cell, etc.; and collection time information of the target data, which is used to instruct the ADRF within what time range to collect the second data, and if the collection time information is one month, the ADRF needs to periodically collect the second data within one month in the future, which may be a period in the past.

[0140] In one implementation, the task of step 4b may be completed by Nadrf_DataManagement_StorageSubscriptionRequest.

[0141] Because step 4b has been added, acquisition identifier information must also be fed back in step 7, and the second data acquisition request in step 8 must also include this information in order for the MTLF to request the corresponding data from the ADRF.

[0142] As can be seen from the above, the technical solution of the present application allows a first network-side device to obtain storage information of target data from a second network-side device, then obtain target data, and analyze the target data to obtain performance information of the target model, which can then estimate the accuracy of the target model in the current actual inference process, which can be used to adjust subsequent policy decisions based on the target model, thereby improving the accuracy of policy decisions made by devices inside and outside the network based on the model inference results of the target model.

[0143] As shown in FIG. 5 , an embodiment of the present application further provides a data acquisition method, which includes the following steps: In step S21, the second network side device receives a first data acquisition request from the first network side device, the first data acquisition request is used to request target data, and the target data is used to determine performance information of the target model; In step S22, the second network side device transmits storage information to the first network side device, and the storage information is used to instruct the first network side device to acquire the target data.

[0144] As can be seen from the above, the technical solution of the present application allows a first network-side device to obtain storage information of target data from a second network-side device, then obtain target data, and analyze the target data to obtain performance information of the target model, which can then estimate the accuracy of the target model in the current actual inference process, which can be used to adjust subsequent policy decisions based on the target model, thereby improving the accuracy of policy decisions made by devices inside and outside the network based on the model inference results of the target model. Optionally, the first data acquisition request includes: Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; data type information of the target data; time information of the target data; location information of the target data; and at least one of: first instruction information used to indicate a source of collection of the target data desired by the first network side device, the source of collection of the target data including any one or more of the second network side device, the third network side device, and the fourth network side device, wherein the third network side device is used to store the target data, and the fourth network side device is a data source of the target data.

[0145] Optionally, the second network side device transmitting the stored information to the first network side device includes: The second network side device acquires the target data; The second network side device transmits storage information to the first network side device, the storage information including the target data.

[0146] Optionally, the second network side device transmitting the stored information to the first network side device includes: The second network side device acquires the target data; The second network side device stores the target data in a third network side device and stores information in the first network side device, and the stored information includes instruction information for the third network side device.

[0147] Optionally, when the target data includes output data of the target model, the second network side device acquires the target data: The second network side device acquires input data of the target model; The second network-side device inputs the input data into the target model for processing, and obtains corresponding output data.

[0148] Optionally, the second network side device storing the target data in a third network side device: The second network side device transmits to a third network side device data storage instruction information in which the target data is to be delivered; and receiving, by the second network side device, storage identifier information returned from the third network side device, for indicating that the target data has been successfully stored.

[0149] Optionally, the data storage instruction information includes: input data corresponding to the target model; output data corresponding to the target model; label data corresponding to the target model; time information corresponding to the input data; time information corresponding to the output data; time information corresponding to the label data; Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; and model filtering information corresponding to the target model.

[0150] Optionally, the second network side device storing the target data in a third network side device: The second network side device transmits data acquisition instruction information to a third network side device, and the data acquisition instruction information is used by the third network side device to instruct a fourth network side device to request the target data; The second network side device receives acquisition identifier information returned from the third network side device, and the identifier information is used to indicate that the third network side device has already acquired and stored the target data from the fourth network side device.

[0151] Optionally, the data acquisition instruction information includes: Identifier information of the third network side device; Address information of the third network side device; data type information of the target data; time information of the target data; location information of the target data; and collection time information of the target data.

[0152] Optionally, the second network side device transmitting the stored information to the first network side device includes: The second network side device transmits stored information including instruction information of the fourth network side device to the first network side device; or The second network side device transmits, to the first network side device, stored information including task information transmitted from the user equipment; Here, the task information is used to instruct the first network side device to train and obtain the target model.

[0153] Optionally, the stored information includes: Identifier information of a third network side device; Address information of a third network side device; Identifier information of the second network side device; and storage identifier information of the target data.

[0154] Optionally, the second network side device transmitting the stored information to the first network side device includes: The second network side device transmits the stored information and related information to the first network side device; Here, the related information is task identifier information of an analysis task corresponding to the target model; task condition limiting information for the analysis task; target information for the analysis task; model identifier information corresponding to the target model; model filtering information corresponding to the target model; data type information of the target data; time information of the target data; and location information of the target data.

[0155] Optionally, the second network side device transmitting the stored information to the first network side device includes: The second network side device transmits storage information and request information to the first network side device, and the request information is used to indicate a minimum requirement for performance information of the target model.

[0156] Optionally, the target data includes any one or more of input data, output data, and label data of a target model.

[0157] Optionally, the label data corresponding to the target model includes actual generated data related to the input data and / or the output data corresponding to the target model.

[0158] The implementation process of the method embodiment shown in FIG. 5 can refer to the respective implementation processes of the method embodiment shown in FIG. 2, and achieves the same technical effect, and will not be further described here to avoid repetition of description.

[0159] The data acquisition method according to the embodiment of the present application may be executed by a data acquisition device. In the embodiment of the present application, the data acquisition device according to the embodiment of the present application will be described taking the data acquisition method executed by the data acquisition device as an example.

[0160] As shown in Figure 6, it is a structural diagram of a data acquisition device of the present application. The present application provides a data acquisition device, and the data acquisition device includes: a request module 301, which is used to send a first data acquisition request to a second network-side device, the first data acquisition request being used to request target data, and the target data being used to determine performance information of the target model; an acquiring module 302 for receiving stored information from the second network side device and acquiring the target data based on the stored information; an analysis module 303 for analyzing the target data to obtain performance information of the target model.

[0161] As can be seen from the above, the technical solution of the present application is such that the first network-side device obtains the storage information of the target data from the second network-side device, and obtains the target data based on the storage data, and further analyzes the target data to obtain an analysis result that can embody the performance information of the target model, and it can be understood that the accuracy of the current actual inference process of the target model can be estimated based on the analysis result, which is used to adjust the subsequent policy decision based on the target model, thereby improving the accuracy rate of the policy decision made by the devices inside and outside the network based on the model inference result of the target model. Optionally, the first data acquisition request includes: Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; data type information of the target data; time information of the target data; location information of the target data; At least one of the following is carried: first instruction information, which is used to indicate a source of collection of the target data desired by the first network side device, and the source of collection of the data to be processed includes any one or more of the second network side device, the third network side device, and the fourth network side device, wherein the third network side device is used to store the target data, and the fourth network side device is a data source of the target data.

[0162] Optionally, the stored information includes: Identifier information of a third network side device; Address information of a third network side device; Identifier information of the second network side device; and storage identifier information of the target data.

[0163] Optionally, the target data includes any one or more of input data, output data, and label data of a target model.

[0164] Optionally, the label data corresponding to the target model includes actual generated data related to the input data and / or the output data corresponding to the target model.

[0165] Optionally, the acquiring module 302 is specifically used to receive related information from the second network side device, and acquire the target data according to the stored information and the related information; Here, the related information is task identifier information of an analysis task corresponding to the target model; task condition limiting information for the analysis task; target information for the analysis task; model identifier information corresponding to the target model; model filtering information corresponding to the target model; data type information of the target data; time information of the target data; and location information of the target data.

[0166] Optionally, the analysis module 303 is specifically used to receive requirement information from the second network side device, and the requirement information is used to indicate a minimum requirement for performance information of the target model, and based on the requirement information, analyze the target data to obtain performance information of the target model.

[0167] Optionally, the storage information carries at least one of identifier information of the third network side device, address information of the third network side device, identifier information of the fourth network side device, and address information of the fourth network side device; The acquisition module 302 specifically includes: The first network side device determines a third network side device based on the stored information; The first network side device transmits a second data acquisition request for the target data to the third network side device; The first network side device receives the target data returned from the third network side device.

[0168] Optionally, the second data acquisition request includes: Identifier information of the second network side device; storage identifier information of the target data; Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; data type information of the target data; time information of the target data; and location information of the target data.

[0169] Optionally, the target data is carried in the storage information, and the acquisition module 302 specifically: The first network-side device is used to acquire the target data from the stored information.

[0170] Optionally, the stored information further carries time zone information, the time zone information being used to indicate an acquisition time and / or an acquisition time range of the target data, and the acquisition module 302 further comprises: The first network-side device is used to acquire the target data at a time corresponding to the time zone information based on the stored information.

[0171] Optionally, the request module 301 specifically: After a predetermined time has elapsed since the first network side device transmitted the target model to the second network side device, the first network side device transmits a first data acquisition request for the target model to the second network side device; or The first network side device is used to transmit a first data acquisition request to a second network side device in accordance with a preset cycle.

[0172] Optionally, the device comprises: The network device further includes an execution module for transmitting the target model to any one of devices that requests the target model when the performance information satisfies a predetermined condition.

[0173] Optionally, the device comprises: The first network-side device further includes an execution module for updating model parameters of the target model based on the performance information.

[0174] Optionally, the request module 301 specifically: The first network side device is used to transmit a first data acquisition request to a second network side device in accordance with a preset cycle.

[0175] The data acquisition device according to the embodiment of FIG. 6 of the present application can realize each process realized by the embodiment of the method of FIG. 2 and achieve the same technical effect, and will not be further described here to avoid repetition of description.

[0176] As shown in Figure 7, the structure diagram of the data acquisition device of the present application, the present application provides a data acquisition device, the data acquisition device includes: a receiving module 401, which is used to receive a first data acquisition request from a first network-side device, the first data acquisition request being used to request target data, and the target data being used to determine performance information of the target model; a sending module 402 used to send stored information to the first network side device, the stored information including a sending module 402 for instructing the first network side device to acquire the target data;

[0177] As can be seen from the above, the technical solution of the present application allows a first network-side device to obtain storage information of target data from a second network-side device, and obtain target data based on the storage data, and then analyze the target data to obtain an analysis result that can embody the performance information of the target model. As can be seen, the accuracy of the target model in the current actual inference process can be estimated based on the analysis result, which is used to adjust subsequent policy decisions based on the target model, thereby improving the accuracy of policy decisions made by devices inside and outside the network based on the model inference results of the target model.

[0178] Optionally, the first data acquisition request includes: Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; data type information of the target data; time information of the target data; location information of the target data; and at least one of: first instruction information used to indicate a source of collection of the target data desired by the first network side device, the source of collection of the target data including any one or more of the second network side device, the third network side device, and the fourth network side device, wherein the third network side device is used to store the target data, and the fourth network side device is a data source of the target data.

[0179] Optionally, the sending module 402 specifically: The second network side device acquires the target data; The second network side device transmits stored information to the first network side device, and the stored information includes the target data.

[0180] Optionally, the sending module 402 specifically: The second network side device acquires the target data; The second network side device is used to store the target data in a third network side device and transmit stored information to the first network side device, and the stored information includes instruction information for the third network side device.

[0181] Optionally, if the target data includes the output data of the target model, the sending module 402 specifically: The second network side device acquires input data of the target model; The second network-side device is used to input the input data into the target model for processing, and obtain corresponding output data.

[0182] Optionally, the sending module 402 specifically: The second network side device transmits to a third network side device data storage instruction information in which the target data is to be delivered; The second network side device receives storage identifier information sent back from the third network side device, the storage identifier information indicating that the target data has been successfully stored.

[0183] Optionally, the data storage instruction information includes: input data corresponding to the target model; output data corresponding to the target model; label data corresponding to the target model; time information corresponding to the input data; time information corresponding to the output data; time information corresponding to the label data; Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; and model filtering information corresponding to the target model.

[0184] Optionally, the sending module 402 specifically: The second network side device transmits data acquisition instruction information to a third network side device, and the data acquisition instruction information is used by the third network side device to instruct a fourth network side device to request the target data; The second network side device receives acquisition identifier information returned from the third network side device, and the identifier information is used to indicate that the third network side device has already acquired and stored the target data from the fourth network side device.

[0185] Optionally, the data acquisition instruction information includes: Identifier information of the third network side device; Address information of the third network side device; data type information of the target data; time information of the target data; location information of the target data; and collection time information of the target data.

[0186] Optionally, when the first instruction information is carried in the first data acquisition request, and the first instruction information indicates that the collection source of the target data includes a fourth network-side device corresponding to the target model, the sending module 402 specifically: The second network side device is used to determine a fourth network side device corresponding to the target model based on task information sent from the user equipment, and send storage information to the first network side device, wherein the storage information includes instruction information for the fourth network side device; or The second network side device transmits, to the first network side device, stored information including task information transmitted from the user equipment; Here, the task information is used to instruct the first network side device to train and obtain the target model.

[0187] Optionally, the stored information includes: Identifier information of a third network side device; Address information of a third network side device; Identifier information of the second network side device; and storage identifier information of the target data.

[0188] Optionally, the sending module 402 specifically: used to transmit stored information and related information to the first network side device; Here, the related information is Identifier information for the analysis task corresponding to the model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; model filtering information corresponding to the target model; data type information of the target data; time information of the target data; and location information of the target data.

[0189] Optionally, the sending module 402 is specifically used to send storage information and request information to the first network side device, and the request information is used to indicate the minimum requirements for performance information of the target model.

[0190] Optionally, the target data includes any one or more of input data, output data, and label data of a target model.

[0191] Optionally, the label data corresponding to the target model includes actual generated data related to the input data and / or the output data corresponding to the target model.

[0192] The data acquisition device according to the embodiment of FIG. 7 of the present application can realize each process realized by the embodiment of the method of FIG. 5 and achieve the same technical effect, and will not be further described here to avoid repetition of description.

[0193] The data acquisition device in the embodiments of the present application may be a terminal device, for example, a terminal device having an operating system, or a component of the terminal device, for example, an integrated circuit or a chip. The terminal device may be a terminal or other device other than a terminal. Exemplarily, the terminal may include, but is not limited to, the types of terminals 11 listed above. The other device may be a server, a network-attached storage (NAS), etc., and the embodiments of the present application are not specifically limited thereto.

[0194] The data acquisition device according to the embodiment of the present application can implement each process implemented by the method embodiment of FIG. 2 or FIG. 5, and will not be further described here to avoid repetition of description.

[0195] As shown in FIG. 8, it is a schematic diagram of a data acquisition system of the present application. The present application provides a data acquisition system, including a first network side device and a second network side device, where: the first network side device sends a first data acquisition request to a second network side device, the first data acquisition request is used to request target data, and the target data is used to determine performance information of the target model; the second network side device receives the first data acquisition request and transmits storage information to the first network side device, the storage information being used to instruct the first network side device to acquire the target data; the first network side device receives the stored information and acquires the target data based on the stored information; The first network side device analyzes the target data and obtains performance information of the target model.

[0196] As can be seen from the above, the technical solution of the present application allows a first network-side device to obtain storage information of target data from a second network-side device, and obtain target data based on the storage data, and then analyze the target data to obtain an analysis result that can embody the performance information of the target model. As can be seen, the accuracy of the target model in the current actual inference process can be estimated based on the analysis result, which is used to adjust subsequent policy decisions based on the target model, thereby improving the accuracy of policy decisions made by devices inside and outside the network based on the model inference results of the target model.

[0197] As shown in FIG. 9, it is a schematic diagram of a data acquisition system of the present application, and the present application provides a data acquisition system including a first network side device, a second network side device and a third network side device, wherein: the first network side device sends a first data acquisition request to a second network side device, the first data acquisition request is used to request target data, and the target data is used to determine performance information of the target model; the second network side device receives the first data acquisition request and transmits stored information to the first network side device; the first network side device receives the stored information, and acquires the target data from the third network side device based on the stored information; The first network side device analyzes the target data and obtains performance information of the target model.

[0198] As can be seen from the above, the technical solution of the present application allows a first network-side device to obtain storage information of target data from a second network-side device, and obtain target data based on the storage data, and then analyze the target data to obtain an analysis result that can embody the performance information of the target model. As can be seen, the accuracy of the target model in the current actual inference process can be estimated based on the analysis result, which is used to adjust subsequent policy decisions based on the target model, thereby improving the accuracy of policy decisions made by devices inside and outside the network based on the model inference results of the target model.

[0199] Corresponding to the above method embodiments and apparatus embodiments, as shown in Figure 10, an embodiment of the present application further provides a network side device 800, which includes a processor 801 and a memory 802, and the memory 802 stores programs or instructions that can be executed on the processor 801, and when the programs or instructions are executed by the processor 801, each step of the above data acquisition method embodiments can be realized and similar technical effects can be achieved. In order to avoid repetition, no further description will be given here.

[0200] Specifically, an embodiment of the present application further provides a network side device 900. As shown in Fig. 11, the network side device 900 includes a processor 901, a network interface 902, and a memory 903. Here, the network interface 902 is, for example, a common public radio interface (CPRI).

[0201] Specifically, the network side device 900 of the embodiment of the present invention further includes instructions or programs stored in the memory 903 and capable of running on the processor 901, and the processor 901 can call the instructions or programs in the memory 903 to execute the methods performed by each module shown in FIG. 5 or FIG. 7, and achieve the same technical effects, which will not be further described here to avoid repetition.

[0202] The embodiments of the present application further provide a readable storage medium, on which a program or instruction is stored, which, when executed by a processor, can realize each process of the method embodiment shown in Figure 2 or the method embodiment shown in Figure 5, and achieve the same technical effects. In order to avoid repetition, no further description will be given here.

[0203] Here, the processor is the processor in the terminal in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0204] The embodiments of the present application further provide a computer program / program product, which is stored in a storage medium and can be executed by at least one processor to implement each process of the method embodiment shown in Figure 2 or the method embodiment shown in Figure 5, and achieve the same technical effects. In order to avoid repetition, no further description will be given here.

[0205] It should be noted that, in this specification, the terms "comprises," "includes," or any other variations thereof are intended to cover the non-exclusive "comprises," whereby a process, method, article, or apparatus comprising a set of elements not only includes those elements, but also other elements not expressly listed or inherent in such process, method, article, or apparatus. Absent further limitations, an element defined by the phrase "comprises one of" does not preclude the presence of other identical elements in the process, method, article, or apparatus comprising that element. It should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may include performing functions in an essentially simultaneous manner or in the reverse order based on the functions involved. For example, a described method may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to some examples may be combined in other examples.

[0206] From the above description of the embodiments, it is clear to those skilled in the art that the methods of the above embodiments may be realized in the form of software and a necessary general-purpose hardware platform, or of course, by hardware, and that in many cases the former is a more preferable embodiment. From this understanding, the substantial part of the technical solution of the present application or the part that contributes to the prior art may be embodied in the form of a software product, and this computer software product may be stored in a storage medium (e.g., ROM / RAM, magnetic disk, optical disk) and include some instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, network device, etc.) to execute the method of each embodiment of the present application.

[0207] Although the above describes the embodiments of the present application in conjunction with the drawings, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not limiting. Those skilled in the art can implement many forms under the guidance of the present application without departing from the spirit and scope of protection of the claims, and all forms fall within the scope of protection of the present application.

[0208] The embodiments of the present application further provide a chip, the chip including a processor and a communication interface, the communication interface is coupled to the processor, the processor runs a program or instruction, and is used to realize each process of the embodiments of the data acquisition method described above, and can achieve the same technical effect. In order to avoid repetition, no further description will be given here.

[0209] It should be understood that the chips referred to in the embodiments of this application may be referred to as system level chips, system chips, chip systems, or system-on-chips.

[0210] The embodiments of the present application further provide a computer program / program product, which is stored in a storage medium and can be executed by at least one processor to realize each process of the above data acquisition method embodiments and achieve the same technical effects. In order to avoid repetition, no further description will be given here.

[0211] An embodiment of the present application further provides a data acquisition system including a terminal and a network side device, wherein the terminal may be used to perform steps of the data acquisition method as described above, and the network side device may be used to perform steps of the data acquisition method as described above.

[0212] It should be noted that, in this specification, the terms "comprises," "includes," or any other variations thereof are intended to cover the non-exclusive "comprises," whereby a process, method, article, or apparatus comprising a set of elements not only includes those elements, but also other elements not expressly listed or inherent in such process, method, article, or apparatus. Absent further limitations, an element defined by the phrase "comprises one of" does not preclude the presence of other identical elements in the process, method, article, or apparatus comprising that element. It should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may include performing functions in an essentially simultaneous manner or in the reverse order based on the functions involved. For example, a described method may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to some examples may be combined in other examples.

[0213] From the above description of the embodiments, it will be apparent to those skilled in the art that the methods of the above embodiments can be realized in the form of software and a required general-purpose hardware platform. Of course, they can also be realized in hardware, but in many cases the former is a more preferred embodiment. From this understanding, the substantial or prior art contributions of the technical solution of the present application may be embodied in the form of a software product, and this computer software product may be stored in a storage medium (e.g., ROM / RAM, magnetic disk, optical disk) and include some instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, network device, etc.) to execute the methods of the embodiments of the present application.

[0214] Although the above describes the embodiments of the present application in conjunction with the drawings, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not limiting. Those skilled in the art can implement many forms under the guidance of the present application without departing from the spirit and scope of protection of the claims, and all forms fall within the scope of protection of the present application.

Claims

1. 1. A data acquisition method, comprising: a first network-side device sending a first data acquisition request to a second network-side device, the first data acquisition request being used to request target data, and the target data being used to determine performance information of the target model; The first network side device receives storage information from the second network side device and acquires the target data based on the storage information; The data acquisition method includes the first network side device analyzing the target data to obtain performance information of the target model.

2. The first data acquisition request includes: Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; data type information of the target data; time information of the target data; location information of the target data; 2. The method of claim 1, wherein at least one of: first instruction information is carried, the first instruction information being used to indicate a desired collection source of the target data by the first network side device, the collection source including any one or more of the second network side device, a third network side device, and a fourth network side device, wherein the third network side device is used to store the target data, and the fourth network side device is a data source of the target data.

3. The stored information includes: Identifier information of a third network side device; Address information of a third network side device; Identifier information of the second network side device; The method of claim 1 , wherein at least one of: a storage identifier information of the target data;

4. The method of claim 1 , wherein the target data includes any one or more of input data, output data, and label data of a target model.

5. The method of claim 4 , wherein the label data corresponding to the target model comprises actual generated data associated with the input data and / or the output data corresponding to the target model.

6. Before the first network side device acquires the target data based on the stored information, The first network side device receives related information from the second network side device; The first network side device further includes acquiring the target data based on the stored information; the first network side device acquiring the target data based on the stored information includes the first network side device acquiring the target data based on the stored information and the related information; Here, the related information is task identifier information of an analysis task corresponding to the target model; task condition limiting information for the analysis task; target information for the analysis task; model identifier information corresponding to the target model; model filtering information corresponding to the target model; data type information of the target data; time information of the target data; and location information of the target data.

7. Before the first network side device analyzes the target data and obtains performance information of the target model, the method includes:

2. The method of claim 1, further comprising the first network side device receiving requirement information from the second network side device, the requirement information being used to indicate minimum requirements for performance information of the target model.

8. The storage information carries identifier information and / or address information of the third network side device, and the first network side device acquires the target data based on the storage information, The first network side device determines the third network side device based on the stored information; The first network side device transmits a second data acquisition request for the target data to the third network side device; The method according to claim 1 , further comprising: receiving, by the first network side device, the target data returned from the third network side device.

9. The second data acquisition request includes: Identifier information of the second network side device; storage identifier information of the target data; Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; data type information of the target data; time information of the target data; The method of claim 8 , wherein at least one of the target data and location information is conveyed.

10. The storage information further carries time zone information, the time zone information being used to indicate an acquisition time and / or an acquisition time range of the target data, and the first network side device acquiring the target data based on the storage information, The method according to claim 1 , further comprising: the first network side device acquiring the target data at a time corresponding to the time zone information based on the stored information.

11. The first network side device transmits a first data acquisition request to a second network side device, After a predetermined time has elapsed since the first network side device transmitted the target model to the second network side device, transmitting a first data acquisition request for the target model to the second network side device; or The method according to claim 1 , further comprising: transmitting a first data acquisition request from a first network side device to a second network side device in accordance with a preset cycle.

12. After the first network side device analyzes the target data and obtains performance information of the target model, the method includes:

8. The method according to claim 1, further comprising: when the performance information satisfies a predetermined condition, the first network side device transmits the target model to any one device that requests the target model.

13. After the first network side device analyzes the target data and obtains performance information of the target model, the method includes: The method of claim 1 , further comprising: the first network-side device updating model parameters of the target model based on the performance information.

14. After the first network side device analyzes the target data and obtains performance information of the target model, the method includes:

8. The method according to claim 1, further comprising: when the performance information does not reach a model performance requirement corresponding to model performance requirement information, the first network side device feeding back the performance information to the second network side device.

15. The first network side device analyzes the target data and obtains performance information of the target model, The method of claim 1 , further comprising: the first network-side device calculating target data and obtaining performance information of a target model.

16. The method of claim 1 , wherein the target model performance information comprises an accuracy rate.

17. The method according to claim 1 , wherein the first network side equipment comprises a model training logic function MTLF and the second network side equipment comprises an analysis logic function AnLF.

18. 1. A data acquisition method, comprising: a second network side device receiving a first data acquisition request from a first network side device, the first data acquisition request being used to request target data, and the target data being used to determine performance information of the target model; A data acquisition method comprising: the second network side device transmitting stored information to the first network side device, the stored information being used to instruct the first network side device to acquire the target data.

19. The first data acquisition request includes: Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; data type information of the target data; time information of the target data; location information of the target data; 19. The method of claim 18, wherein at least one of the following is carried: first instruction information, which is used to indicate a desired collection source of the target data by the first network side device, the collection source of the target data including any one or more of the second network side device, a third network side device, and a fourth network side device, wherein the third network side device is used to store the target data, and the fourth network side device is a data source of the target data.

20. The second network side device transmitting the stored information to the first network side device, The second network side device acquires the target data; The method of claim 18, further comprising: the second network side device storing the target data in a third network side device and transmitting storage information to the first network side device, wherein the storage information includes instruction information for the third network side device.

21. When the target data includes output data of the target model, the second network side device acquires the target data, The second network side device acquires input data of the target model; The method of claim 20, further comprising: the second network-side device inputting the input data into the target model for processing to obtain corresponding output data.

22. The second network side device storing the target data in a third network side device, The second network side device transmits data storage instruction information to a third network side device, the data storage instruction information carrying the target data; The method of claim 20, further comprising: receiving, by the second network side device, storage identifier information returned from the third network side device, wherein the storage identifier information is used to indicate successful storage of the target data and is used to retrieve the target data.

23. The data storage instruction information includes: input data corresponding to the target model; output data corresponding to the target model; label data corresponding to the target model; time information corresponding to the input data; time information corresponding to the output data; time information corresponding to the label data; Identifier information of an analysis task corresponding to the target model; condition limitation information of the analysis task that indicates a limitation condition for the target data of the analysis task; target information for the analysis task; model identifier information corresponding to the target model; and model filtering information corresponding to the target model.

24. The second network side device storing the target data in a third network side device, The second network side device transmits data acquisition instruction information to a third network side device, and the data acquisition instruction information is used by the third network side device to instruct a fourth network side device to request the target data; 21. The method of claim 20, comprising: the second network side device receiving acquisition identifier information returned from the third network side device, the acquisition identifier information being used to indicate that the third network side device has already acquired and stored the target data from the fourth network side device.

25. The data acquisition instruction information includes: Identifier information of the third network side device; Address information of the third network side device; data type information of the target data; time information of the target data; location information of the target data; and collection time information of the target data.

26. The stored information includes: Identifier information of a third network side device; Address information of a third network side device; Identifier information of the second network side device; 20. The method of claim 18, wherein at least one of: a storage identifier information of the target data;

27. The second network side device transmitting the stored information to the first network side device, The second network side device transmits the stored information and related information to the first network side device; Here, the related information is task identifier information of an analysis task corresponding to the target model; task condition limiting information for the analysis task; target information for the analysis task; model identifier information corresponding to the target model; model filtering information corresponding to the target model; data type information of the target data; time information of the target data; and location information of the target data.

28. The second network side device transmitting the stored information to the first network side device, 20. The method of claim 18, further comprising: the second network side device transmitting stored information and request information to the first network side device, the request information being used to indicate minimum requirements for performance information of the target model.

29. The method of claim 18 , wherein the target data includes any one or more of input data, output data, and label data for a target model.

30. 30. The method of claim 29, wherein the label data corresponding to the target model comprises actual generated data associated with the input data and / or the output data corresponding to the target model.

31. The method comprises: The method of claim 18, further comprising: the second network side device receiving the performance information fed back from the first network side device.

32. 32. The method of any one of claims 18 to 31, wherein the target model performance information comprises an accuracy rate.

33. 33. The method according to claim 18, wherein the first network side equipment comprises a model training logic function MTLF, and the second network side equipment comprises an analysis logic function AnLF.

34. A data acquisition device, a request module, the request module being used to send a first data acquisition request to a second network-side device, the first data acquisition request being used to request target data, the target data being used to determine performance information of the target model; an acquisition module for receiving stored information from the second network side device and acquiring the target data based on the stored information; an analysis module for analyzing the target data to obtain performance information for the target model.

35. 1. A data acquisition device, comprising: a receiving module, the receiving module being used to receive a first data acquisition request from a first network-side device, the first data acquisition request being used to request target data, the target data being used to determine performance information of the target model; a transmission module used to transmit stored information to the first network side device, the stored information being used to instruct the first network side device to acquire the target data.

36. 1. A data acquisition system, comprising: a first network side device and a second network side device, wherein: the first network side device sends a first data acquisition request to a second network side device, the first data acquisition request is used to request target data, and the target data is used to determine performance information of the target model; the second network side device receives the first data acquisition request and transmits storage information to the first network side device, the storage information being used to instruct the first network side device to acquire the target data; the first network side device receives the stored information and acquires the target data based on the stored information; The first network-side device analyzes the target data to obtain performance information of the target model.

37. A data acquisition system, comprising: a first network side device, a second network side device, and a third network side device, wherein: the first network side device sends a first data acquisition request to a second network side device, the first data acquisition request is used to request target data, and the target data is used to determine performance information of the target model; the second network side device receives the first data acquisition request and transmits stored information to the first network side device; the first network side device receives the stored information, and acquires the target data from the third network side device based on the stored information; The first network-side device analyzes the target data to obtain performance information of the target model.

38. An apparatus comprising a processor and a memory, the memory storing a program or instructions operable on the processor, the program or instructions implementing the steps of the data acquisition method of any one of claims 1 to 17 or 18 to 33 when executed by the processor.

39. A readable storage medium having stored thereon a program or instructions which, when executed by a processor, implements the steps of the data acquisition method of any one of claims 1 to 17 or 18 to 33.

Images