Information acquisition method and apparatus, network-side device, and storage medium

Abstract

The application discloses an information acquisition method and device, a network side equipment and a storage medium, and belongs to the technical field of communication. The information acquisition method provided by the application comprises the following steps: a first network side equipment sends first request information to a second network side equipment, the first request information is used for requesting the second network side equipment to determine model performance information of a target model; and the first network side equipment receives the model performance information of the target model sent by the second network side equipment. The first request information comprises target data and / or attribute information of the target data, the target data is used for determining the model performance information of the target model, and the attribute information of the target data is used for acquiring the target data.

Description

Technical Field

[0001] This application belongs to the field of communication technology, specifically relating to an information acquisition method, apparatus, network-side device, and storage medium. Background Technology

[0002] With the rapid development of computer and communication technologies, the application of artificial intelligence (AI) technology in communication networks is gradually increasing. Some network-side devices, such as Network Data Analytics Functions (NWDAFs), can perform intelligent data analysis for specific tasks, such as AI inference tasks, and generate data analysis results. These results can assist internal and external devices in making strategic decisions. In other words, AI technology can improve the intelligence of strategic decision-making. For example, the network data analytics function can train an AI / machine learning (ML) model based on training data to obtain a model suitable for a specific task. In practical use, data analysis can be performed based on this model and corresponding input data to obtain the data analysis results corresponding to that specific task.

[0003] Accurate data analysis results are a prerequisite for internal and external devices in communication networks to make correct strategic decisions. If the accuracy of the data analysis results is low, providing them to internal and external devices in communication networks as a reference for strategic decisions may easily lead to incorrect strategic decisions or inappropriate operations.

[0004] The accuracy of data analysis results can be judged by the performance of artificial intelligence / machine learning models in actual use. Therefore, how to obtain model performance information in actual use to judge the accuracy of data analysis results and provide a guarantee for making correct strategic decisions for internal and external devices of communication networks is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This application provides an information acquisition method, apparatus, network-side device, and storage medium to acquire model performance information during actual use, thereby ensuring that internal and external devices of the communication network make correct strategic decisions.

[0006] Firstly, an information acquisition method is provided, including:

[0007] The first network-side device sends a first request message to the second network-side device, the first request message being used to request the second network-side device to determine the model performance information of the target model;

[0008] The first network-side device receives the model performance information of the target model sent by the second network-side device;

[0009] The first request information includes target data and / or attribute information of the target data. The target data is used to determine the model performance information of the target model, and the attribute information of the target data is used to acquire the target data.

[0010] Secondly, an information acquisition device is provided, comprising:

[0011] The first sending module is used to send a first request message to the second network-side device, wherein the first request message is used to request the second network-side device to determine the model performance information of the target model;

[0012] The first receiving module is used to receive the model performance information of the target model sent by the second network-side device;

[0013] The first request information includes target data and / or attribute information of the target data. The target data is used to determine the model performance information of the target model, and the attribute information of the target data is used to acquire the target data.

[0014] Thirdly, an information acquisition method is provided, including:

[0015] The second network-side device receives a first request information sent by the first network-side device, wherein the first request information includes target data and / or attribute information of the target data, and the attribute information of the target data is used to acquire the target data;

[0016] The second network-side device determines the model performance information of the target model based on the target data;

[0017] The second network-side device sends the model performance information to the first network-side device.

[0018] Fourthly, an information acquisition device is provided, comprising:

[0019] The second receiving module is used to receive a first request information sent by the first network-side device, wherein the first request information includes target data and / or attribute information of the target data;

[0020] The first determining module is used to determine the model performance information of the target model based on the target data;

[0021] The second sending module is used to send the model performance information to the first network-side device.

[0022] Fifthly, an information acquisition method is provided, including:

[0023] The third network-side device receives the second and third indication information sent by the first network-side device;

[0024] The third network-side device stores the first data and / or related information of the first data according to the second instruction information;

[0025] The third network-side device collects and stores the second data and / or related information of the second data according to the third instruction information;

[0026] The first data and the second data are partial data of the target data, which is used to determine the model performance information of the target model. The relevant information of the first data and the relevant information of the second data are used to acquire the target data.

[0027] Sixthly, an information acquisition device is provided, comprising:

[0028] The third receiving module is used to receive the second indication information and the third indication information sent by the first network-side device;

[0029] The storage module is configured to store first data and / or related information of the first data according to the second instruction information; and to collect and store second data and / or related information of the second data according to the third instruction information.

[0030] The first data and the second data are partial data of the target data, which is used to determine the model performance information of the target model. The relevant information of the first data and the relevant information of the second data are used to acquire the target data.

[0031] In a seventh aspect, a network-side device is provided, including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, wherein when the program or instructions are executed by the processor, they implement the information acquisition method as described in the first aspect, or the information acquisition method as described in the third aspect, or the steps of the information acquisition method as described in the fifth aspect.

[0032] Eighthly, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the information acquisition method as described in the first aspect, or the information acquisition method as described in the third aspect, or the information acquisition method as described in the fifth aspect.

[0033] In a ninth aspect, a computer program / program product is provided, the computer program / program product being stored in a storage medium, the computer program / program product being executed by at least one processor to implement the information acquisition method as described in the first aspect, or the information acquisition method as described in the third aspect, or the steps of the information acquisition method as described in the fifth aspect.

[0034] In this embodiment, the first network-side device sends a first request message to the second network-side device. The first request message is used to request the second network-side device to determine the model performance information of the target model. After the second network-side device determines the model performance information of the target model based on the target data, it sends the model performance information of the target model to the first network-side device. The first network-side device then obtains the model performance information of the target model. Based on the model performance information, it can determine the accuracy of the data analysis results of the target model in the actual use stage. When the accuracy of the data analysis results is high, it can provide the internal and external devices of the communication network with assistance in making strategic decisions, thus ensuring that the internal and external devices of the communication network make correct strategic decisions. Attached Figure Description

[0035] Figure 1 This is a block diagram of a wireless communication system applicable to embodiments of this application;

[0036] Figure 2 This is a flowchart illustrating an implementation of an information acquisition method in this application.

[0037] Figure 3 This is a flowchart illustrating a specific example of information acquisition in an embodiment of this application;

[0038] Figure 4 A flowchart illustrating another specific example of information acquisition in this application embodiment;

[0039] Figure 5 In the embodiments of this application, and Figure 2 A schematic diagram of the corresponding information acquisition device;

[0040] Figure 6 This is a flowchart illustrating another information acquisition method implemented in this application.

[0041] Figure 7 In the embodiments of this application, and Figure 6 A schematic diagram of the corresponding information acquisition device;

[0042] Figure 8 This is a flowchart illustrating another information acquisition method implemented in this application.

[0043] Figure 9 In the embodiments of this application, and Figure 8A schematic diagram of the corresponding information acquisition device;

[0044] Figure 10 This is a schematic diagram of the structure of a network-side device in an embodiment of this application;

[0045] Figure 11 This is a schematic diagram of another structure of the network-side device in the embodiments of this application; Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0047] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0048] It is worth noting that the technologies described in this application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in this application are often used interchangeably, and the described technologies can be used with the systems and radio technologies mentioned above, as well as with other systems and radio technologies. The following description describes New Radio (NR) systems for illustrative purposes, and NR terminology is used in most of the following description; however, these technologies can also be applied to applications beyond NR systems, such as 6th generation (6G) radio systems. th Generation 6G communication system.

[0049] Figure 1 This diagram illustrates a block diagram of a wireless communication system applicable to embodiments of this application. The wireless communication system includes a terminal 11 and a network-side device 12.

[0050] Terminal 11 can be a mobile phone, tablet computer, laptop computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, vehicle-mounted device (VUE), pedestrian terminal (PUE), smart home (home devices with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), game console, personal computer (PC), ATM, or self-service machine, etc. Wearable devices include: smartwatches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart chains, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. It should be noted that the specific type of terminal 11 is not limited in this embodiment.

[0051] Network-side equipment 12 may include access network equipment or core network equipment.

[0052] In this context, access network equipment can also be referred to as radio access network equipment, radio access network (RAN), radio access network function, or radio access network unit. Access network equipment may include base stations, WLAN access points, or WiFi nodes, etc. Base stations may be referred to as Node B, evolved Node B (eNB), access point, base transceiver station (BTS), radio base station, radio transceiver, Basic Service Set (BSS), Extended Service Set (ESS), home B node, home evolved B node, Transmitting Receiving Point (TRP), or any other suitable term in the field, as long as the same technical effect is achieved. The term "base station" is not limited to specific technical terms. It should be noted that this application embodiment only uses a base station in an NR system as an example for description and does not limit the specific type of base station.

[0053] Core network equipment may include, but is not limited to, at least one of the following: core network node, core network function, Mobility Management Entity (MME), Access and Mobility Management Function (AMF), Session Management Function (SMF), User Plane Function (UPF), Policy Control Function (PCF), Policy and Charging Rules Function (PCRF), Edge Application Server Discovery Function (EASDF), Unified Data Management (UDM), Unified Data Repository (UDR), Home Subscriber Server (HSS), Centralized network configuration (CNC), Network Repository Function (NRF), Network Exposure Function (NEF), Local NEF (or L-NEF), and Binding Support. Core network functions (BSF), application functions (AF), etc. It should be noted that this application embodiment only uses core network equipment in the NR system as an example for description, and does not limit the specific type of core network equipment.

[0054] To facilitate understanding, the application scenarios, related technologies, and concepts involved in the embodiments of this application will be explained first.

[0055] This application's embodiments can be applied to scenarios where artificial intelligence / machine learning models, such as target models, are used to perform target tasks. The target model is a selected or trained model that meets the requirements of the target task. In practical use, the corresponding input data is input into the target model to obtain corresponding output data, i.e., the data analysis results corresponding to the target task. These data analysis results may include prediction information for a future time period or time node, or a summary of historical data from a previous time period or time node. The accuracy of the data analysis results can be judged by the model performance information of the target model in the practical use phase.

[0056] As described in the background section, network data analysis functions can perform intelligent data analysis for specific tasks and generate data analysis results. These results can assist internal and external devices in communication networks in making strategic decisions. The network data analysis function can be divided into two network elements: the Analytics Logical Function (AnLF) and the Model Training Logical Function (MTLF).

[0057] Among them, the analysis logic function can perform reasoning to generate predictive information or generate summary information of historical data, such as providing reasoning analysis services according to consumer requests;

[0058] The model training logic function can generate and train models, such as providing artificial intelligence / machine learning model training services based on consumer requests.

[0059] In a specific example, the interaction between the Analysis Logic Function (AnLF) and the Model Training Logic Function (MTLF) is as follows:

[0060] AnLF can request information about one or more ML models associated with one or more task identifiers from MTLF by calling ML Model Information Request, such as Nnwdaf_MLModelInfo_Request. AnLF uses ML models for data analysis.

[0061] When the MTLF receives an ML model information request from the AnLF, it can determine whether the existing ML model can be used for the request or whether further training of the existing ML model is needed. If further training is needed, the MTLF initiates a data collection operation, collecting data from network-side devices such as Network Functions (NF), Application Functions (AF), or Operation Administration and Maintenance (OAM) for ML model training. Network functions can be AMF, Data Collection Coordination Function (DCCF), Analytics Data Repository Function (ADRF), etc.

[0062] MTLF can provide ML model information to AnLF by calling ML model information request response, such as Nnwdaf_MLModelInfo_Requestresponse. ML model information includes information such as the file address of one or a group of ML models.

[0063] In another specific example, the interaction between the Network Data Analysis Function (NWDAF) and the Data Analysis Repository Function (ADRF) is as follows:

[0064] NWDAF can send data and / or analysis results to ADRF by calling data management storage requests, such as Nadrf_DataManagement_StorageRequest.

[0065] ADRF stores the data and / or analysis results sent by NWDAF. ADRF can determine whether the same data and / or analysis results have been or are being stored based on the data and / or analysis results sent by NWDAF. If ADRF has already stored or is storing the same data and / or analysis results, ADRF can decide not to store the data and / or analysis results sent by NWDAF.

[0066] ADRF sends storage information for data and / or analysis results to NWDAF by invoking data management storage request responses, such as the Nadrf_DataManagement_StorageRequest response.

[0067] The interaction process between NWDAF and ADRF described above also applies to the interaction process between DCCF, the MessagingFramework Adaptor Function (MFAF), and ADRF.

[0068] As described above, MTLF is primarily used to generate and train AI / ML models. The trained models are then provided to AnLF, which deploys them in real-world applications for inference to obtain predictive information or to generate summaries of historical data. ADRF is mainly used for data storage. For any given AI / ML model, the accuracy it can achieve at different stages may vary; that is, the accuracy it achieves during training may differ from the accuracy it achieves in real-world use. Factors such as different data distributions at different stages and insufficient model generalization ability can all lead to discrepancies in the accuracy achievable at different stages. Generally, the accuracy achieved by a model in real-world use will be lower than the accuracy achieved during training.

[0069] Therefore, AnLF needs to know the model's performance information during actual use in order to judge the accuracy of the data analysis results during actual use based on the model's performance information. This is so that when the accuracy of the data analysis results is high, it can be provided to the internal and external devices of the communication network to assist them in making strategic decisions, thus ensuring that the internal and external devices of the communication network make correct strategic decisions.

[0070] The application scenarios, related technologies and concepts involved in the embodiments of this application have been described above. The information acquisition method provided by the embodiments of this application will be described in detail below with reference to the accompanying drawings through some embodiments and their application scenarios.

[0071] See Figure 2 The diagram shown is an implementation flowchart of an information acquisition method provided in this application embodiment. The method may include the following steps:

[0072] S210: The first network-side device sends a first request message to the second network-side device. The first request message is used to request the second network-side device to determine the model performance information of the target model. The first request message includes target data and / or attribute information of the target data. The target data is used to determine the model performance information of the target model, and the attribute information of the target data is used to obtain the target data.

[0073] In this embodiment, the first network-side device can be any network-side device that acquires the target model and has a need to perform actual data analysis using the target model, such as AnLF or NWDAF. The second network-side device can be any network-side device capable of determining the model performance information of the target model, such as MTLF or NWDAF.

[0074] Consumer network elements, such as SMF, AMF, UPF, and PCF, can send task requests to the first network-side device according to actual needs. These task requests request the execution of a target task to obtain data analysis results. Upon receiving the task request, the first network-side device can acquire the target model, for example, by sending a model request to the second network-side device. This model request requests the acquisition of a target model for executing the target task. The second network-side device selects or trains a target model that meets the execution requirements of the target task and sends it to the first network-side device. After acquiring the target model, the first network-side device can put it into practical use. During practical use, it can obtain the data analysis results corresponding to the target task based on the target model and the corresponding input data.

[0075] During the actual use phase of the target model, if the set triggering conditions for determining model performance information are met, the first network-side device can send a first request message to the second network-side device. This first request message is used to request the second network-side device to detect the model performance of the target model and determine the model performance information of the target model.

[0076] The trigger condition for determining model performance information can be set according to actual conditions. For example, reaching a set time interval can be considered as meeting the trigger condition, i.e., periodically detecting the model performance. Alternatively, receiving a detection instruction from a consumer network element can also be considered as meeting the trigger condition. Once the trigger condition is met, the first network-side device will send a first request message to the second network-side device, enabling the second network-side device to detect the model performance of the target model and determine its performance information.

[0077] The first request information may include target data, or target data attribute information, or both target data and target data attribute information. The target data is used to determine the model representation information of the target model, and the target data attribute information is used to acquire the target data. When the first request information includes both target data and target data attribute information, the target data attribute information serves as a supplement and description of the target data.

[0078] After receiving the first request information sent by the first network-side device, the second network-side device can determine the model performance information of the target model based on the target data, and then send the model performance information to the first network-side device. If the first request information includes target data, the second network-side device can directly determine the model performance information of the target model based on the target data. If the first request information includes attribute information of the target data, before determining the model performance information of the target model based on the target data, the second network-side device can obtain the target data based on the attribute information of the target data.

[0079] The first request information includes target data and / or attribute information of the target data. Specifically, it can be related information of a first part of the target data and a second part of the target data. The first and second parts of the data are each a portion of the target data, and the related information of the first part of the data is used to obtain the first part of the data. The combination of the first and second parts of the data can constitute the target data. That is, the first network-side device sends the related information of the first part of the data and the second part of the data to the second network-side device. The second network-side device can obtain the first part of the data based on the related information of the first part of the data. For example, it can determine the third network-side device storing the first part of the data based on the related information of the first part of the data, obtain the first part of the data from the third network-side device, and combine the obtained first part of the data with the second part of the data received from the first network-side device to obtain the target data.

[0080] In this embodiment, the target data refers to the relevant data corresponding to the target model during the actual use stage. It may include the input data, output data, and label data of the target model. The relevant data corresponding to the model training before the target model is put into actual use is the training data. Of course, if the target model needs to be retrained after it is put into actual use, the training data corresponding to the retraining stage may include the target data corresponding to the previous actual use stage.

[0081] S220: The first network-side device receives the model performance information of the target model sent by the second network-side device.

[0082] The first network-side device sends a first request message to the second network-side device, requesting the second network-side device to detect the model performance of the target model. After determining the model performance information, the second network-side device can send the target model performance information back to the first network-side device. Upon receiving the target model performance information sent by the second network-side device, the first network-side device can then know the model performance of the target model in the actual use phase.

[0083] After obtaining the performance information of the target model, the first network-side device can determine the accuracy of the target model in actual use. If the accuracy of the target model is low, the data analysis results obtained using the target model are considered to have low accuracy. The first network-side device can then stop feeding back the data analysis results to the consumer network element to prevent the consumer network element from making incorrect policy decisions or performing inappropriate operations based on the low-accuracy data analysis results. If the accuracy of the target model is high, the data analysis results obtained using the target model are considered to have high accuracy. The first network-side device can continue feeding back the data analysis results to the consumer network element to ensure that the consumer network element makes correct policy decisions or performs appropriate operations based on the high-accuracy data analysis results.

[0084] Alternatively, if the accuracy of the target model is low in actual use, the first network-side device can send a model retraining request or a model reselection request to the second network-side device to request the second network-side device to retrain the target model or reselect the model to improve the accuracy of the model.

[0085] Using the method provided in the embodiments of this application, a first network-side device sends a first request message to a second network-side device. The first request message requests the second network-side device to detect the model performance of the target model. The second network-side device detects the model performance of the target model based on the target data. After determining the model performance information, the second network-side device sends the model performance information of the target model to the first network-side device. The first network-side device then obtains the model performance information of the target model. Based on the model performance information, it can determine the accuracy of the data analysis results of the target model in the actual use stage. When the accuracy of the data analysis results is high, it can provide the internal and external devices of the communication network with assistance in making strategic decisions, thus ensuring that the internal and external devices of the communication network make correct strategic decisions.

[0086] In one embodiment of this application, the first request information may further include at least one of the following:

[0087] First instruction message;

[0088] The model represents the expected information;

[0089] Information on model supervision time requirements;

[0090] Information required for model supervision region;

[0091] The first instruction information is used to instruct the second network-side device to determine the model performance information of the target model;

[0092] Model performance expectation information is used to represent the model performance information of the target model expected by the first network-side device;

[0093] The model supervision time requirement information is used to indicate the time range within which the second network-side device determines the model performance information of the target model.

[0094] In this embodiment of the application, the first request information may further include at least one of the following:

[0095] (1) First instruction information. The first instruction information is used to instruct the second network-side device to determine the model performance information of the target model. After receiving the first request information sent by the first network-side device, the second network-side device can know that the first network-side device has a request to determine the model performance information of the target model according to the first instruction information included in the first request information, and then can perform corresponding operations.

[0096] The first instruction information can be explicit, such as being carried by a specific cell; the first instruction information can also be implicit, such as being expressed by the name of the message carrying the first request information.

[0097] (2) Model Performance Expectation Information. Model performance expectation information represents the expected model performance information of the target model by the first network-side device. The model performance expectation can be a threshold, preset conditions, etc. The first network-side device can use the model performance expectation information to inform the second network-side device of the expected and acceptable model performance information of the target model. If, after the second network-side device determines the model performance information of the target model, the model performance information in this table does not meet the model performance expectation corresponding to the model performance expectation information, some subsequent operations can be triggered, such as model retraining, model reselection, informing the first network-side device of the model performance decline, failure to meet expectations, etc.

[0098] Of course, the expected model performance information can also be carried in the model acquisition request sent by the first network-side device to the second network-side device.

[0099] (3) Model Supervision Time Requirement Information. Model supervision time requirement information indicates the time range within which the second network-side device determines the model performance information of the target model. The first network-side device can use this information to inform the second network-side device of the timeframe within which it should supervise the target model's performance. For example, if the model supervision time requirement information indicates one month, the second network-side device needs to periodically acquire relevant data to determine the model performance information within the next month. Alternatively, if the first request information does not include model supervision time requirement information or indicates immediate supervision, the second network-side device can acquire the target data immediately upon receiving the first request information and determine the model performance information based on the target data. This determination is a one-time event, and the second network-side device does not need to collect data for model supervision over a long period.

[0100] (4) Model Supervision Region Requirement Information. The model supervision region requirement information is used to indicate the area within which the second network-side device should determine the model performance information of the target model. The first network-side device can use the model supervision region requirement information to inform the second network-side device within which area the model performance of the target model should be supervised. For example, if the model supervision region requirement information indicates a certain cell, the second network-side device can obtain relevant data within that cell to determine the model performance information.

[0101] The first request information includes at least one of the above information. This information can constrain the determination of the model performance information of the target model by the second network-side device, so as to better meet the needs of the first network-side device.

[0102] In one embodiment of this application, the attribute information of the target data may include at least one of the following:

[0103] Task information corresponding to the target data;

[0104] The target data refers to the information of the model;

[0105] Data description information corresponding to the target data;

[0106] Storage information corresponding to the target data.

[0107] The attribute information of the target data can include one or more of the following: the task information corresponding to the target data, the model information to which the target data is applied, the data description information corresponding to the target data, and the storage information corresponding to the target data. The target data can be obtained based on the attribute information of the target data.

[0108] The task information corresponding to the target data may include at least one of the following:

[0109] (1) Identification information of the target task corresponding to the target data. Such as Analytics ID. The identification information of the target task corresponding to the target data can indicate which specific task the target data is for;

[0110] (2) Target task constraint information. Such as analytical filter information. The target task constraint information is used to represent the filtering conditions of the data analysis results corresponding to the target task. For example, filtering of data analysis results based on Area of ​​Interest (AOI), Single Network Slice Selection Assistance information (S-NSSAI), Data Network Name (DNN), etc.

[0111] (3) Target information of the objective task. For example, Target of analytic reporting. The target information of the objective task indicates whether the data analysis is performed on a single terminal, multiple terminals, or all terminals.

[0112] The information that the target data is directed to the model may include at least one of the following:

[0113] (1) Identification information of the model to which the target data is targeted. Such as Model ID. The identification information of the model to which the target data is targeted indicates that the target data is targeted at a certain model. In the embodiments of this application, the model to which the target data is targeted is the target model;

[0114] (2) Filtering information for the model targeted by the target data. The filtering information for the model targeted by the target data indicates the filtering conditions that the model targeted by the target data needs to meet, such as AOI, S-NSSAI, DNN, etc.

[0115] The data description information corresponding to the target data may include at least one of the following:

[0116] (1) Data category information of the target data. The data category information of the target data indicates which type of data to obtain, such as obtaining target data of the terminal location information category;

[0117] (2) Time range information corresponding to the target data. The time range information corresponding to the target data is used to indicate the time range within which the time information corresponding to the target data is located. The time range information corresponding to the target data can indicate which time range of data to obtain. This time range information can include the start time node and the end time node. For example, if the time information corresponding to the data is the generation time, this time range information can indicate that the data generated between the start time node and the end time node should be obtained;

[0118] (3) Location range information corresponding to the target data. Location range information corresponding to the target data is used to indicate the location range of the target data. The location range information corresponding to the target data can be used to indicate which location range of data to acquire. For example, if the location information corresponding to the data is the location where the data was generated, this location range information can be used to indicate the acquisition of data whose location was within this location range at the time of generation, such as data whose location was within a certain tracking area (TA) or cell at the time of generation.

[0119] The storage information corresponding to the target data may include at least one of the following:

[0120] (1) Identification information of the third network-side device storing the target data. For example, the ID of the third network-side device. The identification information of the third network-side device storing the target data can indicate which device the target data is stored in;

[0121] (2) Address information of the third-party network device. For example, the IP address and fully qualified domain name (FQDN) of the third-party network device. The address information of the third-party network device indicates how to establish a connection with it.

[0122] (3) Storage identification information of the target data. For example, Storage Transaction Identifier. The storage identification information of the target data indicates how to find the target data. The storage identification information can specifically include index information, etc. Different storage methods can have different storage identification information.

[0123] (4) Identification information of the first network-side device. Such as the ID of the first network-side device.

[0124] The identification information of the first network-side device and the storage identification information of the target data can be used in combination to obtain the target data more accurately.

[0125] The first request information sent by the first network-side device to the second network-side device includes the attribute information of the target data. The attribute information of the target data includes one or more of the above-mentioned information, so that the second network-side device can accurately obtain the target data based on the attribute information of the target data, and determine the model performance information of the target model based on the target data, which better meets the detection needs of the first network-side device for the model performance.

[0126] It should be noted that the model performance information of the target model can be used to indicate the correctness and / or error level of the reasoning results during actual inference. The model performance information can be presented in various forms, such as specific percentage values ​​(e.g., 90%), categorical expressions (e.g., high, medium, low), or normalized data (e.g., 0.9). This application does not specifically limit the form of the model performance information. The model performance information of the target model can indicate the accuracy or error level of the target model's reasoning results for the target task, either positively or negatively. For example, the inference accuracy of the target model can be indicated negatively by calculating the inference error or inference error rate. The calculation method for the inference error or inference error rate can be varied, such as Mean Absolute Error (MAE), Mean Square Error (MSE), etc.

[0127] Additionally, it should be noted that when the first request information includes relevant information about the first part of the data and the second part of the data, the relevant information about the first part of the data can refer to the attribute information of the target data. For example, the relevant information about the first part of the data may include at least one of the following: the task information corresponding to the first part of the data, the information of the model to which the first part of the data is applied, the data description information corresponding to the first part of the data, and the storage information corresponding to the first part of the data, so that the second network-side device can accurately obtain the first part of the data through the relevant information about the first part of the data.

[0128] In one embodiment of this application, when the first request information includes attribute information of the target data, the method may further include the following steps before the first network-side device sends the first request information to the second network-side device:

[0129] Step 1: The first network-side device acquires the first data, which may be part or all of the target data;

[0130] Step 2: The first network-side device sends a second instruction message to the third network-side device. The second instruction message includes the first data and / or related information of the first data. The second instruction message is used to instruct the third network-side device to store the first data and / or related information of the first data.

[0131] For ease of description, the two steps above will be explained together.

[0132] In this embodiment, the first request information may include attribute information of the target data. Before the first network-side device sends the first request information to the second network-side device, the first network-side device may first obtain the first data, such as by determining the data source network element through task information and obtaining the first data through the data source network element. The first data is part or all of the target data. After obtaining the first data, the first network-side device may send second indication information to the third network-side device. The second indication information includes the first data and / or related information of the first data.

[0133] The second indication information can be explicit, such as being carried by a specific cell; the second indication information can also be implicit, such as being expressed through the name of the message carrying the information.

[0134] The third network-side device can be any network-side device capable of storing data, such as ADRF. The second indication information sent by the first network-side device to the third network-side device is used to instruct the third network-side device to store the first data and / or related information of the first data. The third network-side device can store the first data and / or related information of the first data according to the second indication information.

[0135] If the first data is all the target data, it means that the target data is obtained by the first network-side device and then stored in the third network-side device. If the second network-side device needs to obtain the target data, it can obtain it through interaction with the third network-side device.

[0136] If the first data is a portion of the target data, it means that part of the target data was obtained by the first network-side device. This portion of data is instructed to be stored in the third network-side device. The remaining portion of the target data can be obtained in other ways, such as being collected by the third network-side device. After the third network-side device collects the remaining portion of the target data, it stores it, thus storing the target data in the third network-side device. If the second network-side device needs to obtain the target data, it can do so through interaction with the third network-side device. Alternatively, it can be collected by the second network-side device.

[0137] In one embodiment of this application, the method may further include the following steps:

[0138] The first network-side device receives the first storage feedback information sent by the third network-side device;

[0139] The first storage feedback information includes the storage identifier information corresponding to the first data.

[0140] In this embodiment, the first network-side device acquires the first data and sends a second instruction to the third network-side device. The third network-side device stores the first data and / or related information of the first data. After storage, it can return first storage feedback information to the first network-side device. The first storage feedback information may include storage identifier information corresponding to the first data. Of course, if the third network-side device determines that it has already stored the first data and / or related information of the first data after receiving the second instruction, it can directly return the first storage feedback information to the first network-side device.

[0141] After receiving the first storage feedback information sent by the third network side device, the first network side device learns that the third network side device has completed the storage of the first data and / or related information of the first data, and can obtain the corresponding storage identification information. This allows the storage identification information to be included in the attribute information of the target data in the first request information sent to the second network side device, so as to facilitate the second network side device in obtaining the target data.

[0142] In one embodiment of this application, the first data may include at least one of the following:

[0143] Input data corresponding to the target model;

[0144] Output data corresponding to the target model;

[0145] Label data corresponding to the target model;

[0146] The label data corresponding to the target model may include the actual generated data associated with the input data and / or output data corresponding to the target model.

[0147] In this embodiment, the first data may include at least one of the following: input data corresponding to the target model, output data corresponding to the target model, and label data corresponding to the target model. The input data corresponding to the target model refers to the input data corresponding to the target model during actual use; the output data corresponding to the target model refers to the output data obtained by analyzing the input data during actual use; and the label data corresponding to the target model refers to the actually generated data associated with the input data and / or output data corresponding to the target model, or data labeled based on the actually generated data.

[0148] For example, the input data for the target model consists of terminal location data for a specific cell within a given time period. The target task of the target model is to predict the number of terminals in that cell within the next week. Inputting the input data into the target model yields the corresponding output data, which is the predicted number of terminals in the cell for the next week. After one week, the actual number of terminals in the cell is obtained. This actual number of terminals can be used as the label data for the target model, and this label data is correlated with the corresponding input and output data.

[0149] In one embodiment of this application, the relevant information of the first data may include at least one of the following:

[0150] The task information corresponding to the first data;

[0151] The first data refers to the information of the model;

[0152] The first data corresponds to the time information.

[0153] The task information corresponding to the first data may include at least one of the following:

[0154] (1) Identification information of the target task corresponding to the first data. The identification information of the target task corresponding to the first data indicates which specific task the first data is for;

[0155] (2) Condition constraints for the target task. The condition constraints for the target task are used to represent the filtering conditions for the data analysis results corresponding to the target task;

[0156] (3) Target object information. The target object information indicates whether the target task is to perform data analysis on a single terminal, multiple terminals, or all terminals.

[0157] The information for the model in the first data set may include at least one of the following:

[0158] (1) Identification information of the model to which the first data is targeted. The identification information of the model to which the first data is targeted indicates that the first data is targeted at a specific model. In the embodiments of this application, the model to which the first data is targeted is the target model;

[0159] (2) Filtering information for the model targeted by the first data. The filtering information for the model targeted by the first data indicates the filtering conditions that the model targeted by the first data needs to meet.

[0160] The time information corresponding to the first data may include at least one of the following:

[0161] (1) Time information corresponding to the input data. Specifically, the time information corresponding to the input data may include at least one of the following: the time node information when the input data is generated, the time node information when the input data is acquired by the first network-side device, and the time node information when the input data is input into the target model;

[0162] (2) Time information corresponding to the output data. Specifically, the time information corresponding to the output data may include at least one of the following: the time node information when the output data was generated, the target time node information to which the output data is targeted, and the time information corresponding to the input data associated with the output data;

[0163] (3) Time information corresponding to the tag data. Specifically, the time information corresponding to the tag data may include at least one of the following: the time node information when the tag data is generated, and the time node information when the tag data is acquired by the first network-side device.

[0164] The time point at which the output data is generated can be the same as the time point at which the input data is input into the target model, or it can be the time point at which the output data is generated after the input data is input into the target model and the target model has completed its calculation.

[0165] The target time point for the output data can be the time point corresponding to the predicted information included in the output data. For example, if the predicted information included in the output data is the number of terminals in a certain community on day X one week later, then the target time point is day X one week later. Tag data is generated corresponding to the target time point. Based on the target time point of the output data and the time point when the tag data is generated, the output data and tag data can be correlated.

[0166] The time information corresponding to the input data associated with the output data may include at least one of the following: the time node information when the input data associated with the output data was generated, the time node information when the input data associated with the output data was acquired by the first network-side device, and the time node information when the input data associated with the output data was input into the target model. The output data and input data can be associated based on the time information corresponding to the output data associated with it.

[0167] Utilizing the time information corresponding to input, output, and label data can aid in determining subsequent model performance information. For instance, by comparing the time information corresponding to the output data and the label data, the output data of the target model and its corresponding label data can be associated / bound / mapped to compare / calculate / generate model performance information. Furthermore, the time information corresponding to the input data of the target model reveals the time range covered by this data. For example, if the earliest time information for the input data is Monday of a certain week and the latest time is Friday of that week, then the time range covered by this data is Monday to Friday of that week. This time range can be used to record and ensure the timeliness of model performance information.

[0168] In one embodiment of this application, when the first request information includes attribute information of the target data, the method may further include the following steps before the first network-side device sends the first request information to the second network-side device:

[0169] The first network-side device sends a third instruction message to the third network-side device. The third instruction message is used to instruct the third network-side device to collect and store the second data and / or related information of the second data. The second data is part or all of the target data.

[0170] In this embodiment of the application, when the first request information includes attribute information of the target data, before the first network-side device sends the first request information to the second network-side device, the first network-side device may send third indication information to the third network-side device. The third indication information is used to instruct the third network-side device to collect and store the second data and / or related information of the second data. The second data may be part or all of the target data.

[0171] If the second data comprises all the target data, the third network-side device can collect and store all the target data based on the third indication information, and can also store relevant information about the second data, which can be found in the attribute information of the target data. The second network-side device can obtain all the target data through the third network-side device.

[0172] If the second data is a portion of the target data, the third network-side device can collect and store the portion of the target data based on the third indication information, and can also store relevant information about the second data. The second network-side device can obtain the portion of the target data through the third network-side device, and the other portions of the target data can be collected by the second network-side device.

[0173] If the first data and the second data are each partial data of the target data, and the first data and the second data are different, but the combination of the first data and the second data can constitute the target data, then after the first network-side device obtains the first data, it sends a second instruction message to the third network-side device, instructing the third network-side device to store the first data and / or related information of the first data. It can also send a third instruction message to the third network-side device, instructing the third network-side device to collect and store the second data and / or related information of the second data. The third instruction message and the second instruction message can be carried in the same message or transmitted through different messages.

[0174] The first network-side device exchanges part of the target data with the third network-side device, while the other part of the data is collected by the third network-side device. This helps to reduce the amount of data exchange and save network resources.

[0175] In one embodiment of this application, the method may further include the following steps:

[0176] The first network-side device receives the second storage feedback information sent by the third network-side device;

[0177] The second storage feedback information includes the storage identifier information corresponding to the second data.

[0178] After the third network-side device collects and stores the second data and / or related information of the second data, it can send second storage feedback information to the first network-side device. After receiving the second storage feedback information sent by the third network-side device, the first network-side device can obtain the storage identification information corresponding to the second data, so as to include the storage identification information in the attribute information of the target data included in the first request information sent to the second network-side device, thereby facilitating the second network-side device to obtain the target data.

[0179] The first and second stored feedback information can be carried in the same message, or they can be sent through different messages.

[0180] In one embodiment of this application, the relevant information of the second data may include at least one of the following:

[0181] The task information corresponding to the second data;

[0182] The second set of data pertains to information about the model;

[0183] The second data corresponds to the data description information.

[0184] The relevant information of the second data is used by the third network-side device to collect the second data. The relevant information of the second data can refer to the attribute information of the target data or the relevant information of the first data, which will not be elaborated here.

[0185] In one embodiment of this application, the third instruction information may include information about the device where the second data is located and / or collection request information corresponding to the second data. The information about the device where the second data is located can instruct the third network-side device where to collect the second data, and the collection request information corresponding to the second data can instruct the third network-side device on the specific requirements for collecting the second data.

[0186] The information of the device where the second data is located may include at least one of the following:

[0187] (1) Identification information of the device where the second data is located, such as the ID of the device where the second data is located;

[0188] (2) Address information of the device where the second data is located, such as the IP address and FQDN of the device where the second data is located.

[0189] The collection requirements for the second data may include at least one of the following:

[0190] (1) Data category information of the second data. The data category information of the second data indicates which category of second data to collect, such as collecting terminal location information;

[0191] (2) Time range information corresponding to the second data. The time range information corresponding to the second data is used to indicate the time range within which the time information corresponding to the second data is located. The time range information corresponding to the second data can indicate which time range of data to collect. This time range information can include the start time node and the end time node. For example, if the time information corresponding to the data is the generation time, this time range information can indicate that data generated between the start time node and the end time node should be collected;

[0192] (3) Location range information corresponding to the second data. The location range information corresponding to the second data is used to indicate the location range of the location information corresponding to the second data. The location range information corresponding to the second data can be used to indicate which location range of data to collect. For example, the location information of the data is the location where the data was generated. The location range information can be used to indicate the collection of data whose location was within that location range when it was generated, such as data whose location was within a certain tracking area or cell when it was generated.

[0193] (4) Second data collection time information. The second data collection time information is used to indicate the time requirement for collecting the second data. This information indicates the time frame within which the second data should be collected. For example, if the collection period is for the next month, then the third network-side device needs to collect the second data regularly within the next month. Similarly, if the collection period is for the previous month, then the third network-side device needs to collect the second data from the previous month.

[0194] The third instruction information includes information about the device where the second data is located and / or the collection requirement information corresponding to the second data, so as to facilitate the third network-side device to collect and store the second data and / or related information of the second data in a timely manner.

[0195] In one embodiment of this application, the first data may include input data and / or output data corresponding to the target model, and the second data includes label data corresponding to the target model. That is, the first network-side device collects the input data corresponding to the target model, and can simultaneously obtain corresponding output data based on the target model and the input data. Then, it sends a second instruction to the third network-side device, instructing the third network-side device to store the first data and / or related information of the first data, including the input data and output data. Simultaneously, it can send a third instruction to the third network-side device, instructing the third network-side device to collect and store the second data and / or related information of the second data, including the label data. After obtaining the first and second data from the third network-side device, the second network-side device can detect the model performance of the target model based on the output data included in the first data and the label data included in the second data, and determine the model performance information. Alternatively, it can obtain the corresponding output data based on the input data included in the first data, and then detect the model performance of the target model based on the output data and the label data, and determine the model performance information.

[0196] Alternatively, the first data may include the input data corresponding to the target model, and the second data may include the label data corresponding to the target model. That is, the first network-side device collects the input data corresponding to the target model, and then sends a second instruction to the third network-side device, instructing the third network-side device to store the first data including the input data and / or related information of the first data. Simultaneously, it may send a third instruction to the third network-side device, instructing it to collect and store the second data including the label data and / or related information of the second data. After obtaining the first and second data from the third network-side device, the second network-side device can obtain the output data corresponding to the target model based on the input data included in the first data, and then detect the model performance of the target model based on the output data and label data.

[0197] Alternatively, the first data may include the output data and label data corresponding to the target model, and the second data may include the input data corresponding to the target model.

[0198] Alternatively, the first data may include the label data corresponding to the target model, and the second data may include the input data corresponding to the target model.

[0199] In either case, as long as the second network-side device can ultimately obtain the output data and associated label data corresponding to the target model, the second network-side device can detect the model performance of the target model based on the output data and the corresponding label data, and determine the model performance information.

[0200] In one embodiment of this application, the second network-side device obtains target data based on the attribute information of the target data, which may include the following steps:

[0201] The first step: The second network-side device determines the third network-side device to store the target data based on the attribute information of the target data;

[0202] The second step: The second network-side device sends a second request message to the third network-side device. The second request message is used to request the acquisition of the target data.

[0203] The third step: The second network-side device receives the target data sent by the third network-side device.

[0204] For ease of description, the three steps above will be explained together.

[0205] In this embodiment, the first network-side device, through interaction with the third network-side device, can store target data and / or attribute information of the target data in the third network-side device. When the first network-side device needs to detect the model performance of the target model, it can send a first request message to the second network-side device. Based on the attribute information of the target data included in the first request message, the second network-side device can first determine the third network-side device storing the target data, and then send a second request message to the third network-side device to request the acquisition of the target data. The third network-side device can return the corresponding target data according to the second request message. After receiving the target data sent by the third network-side device, the second network-side device can detect the model performance of the target model based on the target data and determine the model performance information.

[0206] The second request information may include at least one of the following:

[0207] (1) Identification information of the first network-side device, such as the ID of the first network-side device;

[0208] (2) Storage identification information of target data. If the target data is all the data sent by the first network-side device to the third network-side device for storage, there can be one storage identification information for the target data. If the target data includes the data sent by the first network-side device to the third network-side device for storage and the data that the first network-side device instructs the third network-side device to collect and store, there can be multiple storage identification information for the target data, which are used to represent the data stored at different times.

[0209] (3) Task information corresponding to the target data;

[0210] (4) Information about the model to which the target data is applied;

[0211] (5) Data description information corresponding to the target data.

[0212] Based on the above information, the second network-side device can clearly identify the target data it wants to obtain, making it easier for the third network-side device to provide the target data that meets the requirements.

[0213] In one embodiment of this application, after the second network-side device determines the model performance information of the target model based on the target data, the method may further include the following steps:

[0214] The second network-side device determines whether the model performance information meets the model performance expectation corresponding to the model performance expectation information.

[0215] The process of a second network-side device sending model performance information to a first network-side device may include the following steps:

[0216] The second network-side device sends the model performance information to the first network-side device, depending on whether the model performance information meets or does not meet the model performance expectations.

[0217] In this embodiment, the first network-side device sends a first request message to the second network-side device. The second network-side device then detects the model performance of the target model based on the target data to determine model performance information. This model performance information may include model accuracy, error rate, and other similar data.

[0218] The first request information may include model performance expectation information. After determining the model performance information, the second network-side device can further determine whether the model performance information meets the model performance expectation corresponding to the model performance expectation information. For example, the model performance expectation is that the model accuracy is higher than a first threshold. If the model accuracy included in the model performance information is higher than the first threshold, it indicates that the model performance information meets the model performance expectation corresponding to the model performance expectation information; otherwise, it indicates that the model performance information does not meet the model performance expectation corresponding to the model performance expectation information.

[0219] If the model performance information meets the model performance expectations, the second network-side device can send the model performance information to the first network-side device to inform the first network-side device that the current target model is performing well and can continue to be used.

[0220] The second network-side device can also send model performance information to the first network-side device if the model performance does not meet expectations. This informs the first network-side device that the current target model's performance has declined, and continued use may lead to inaccurate data analysis results. The first network-side device can then determine whether to continue using the target model based on the actual situation.

[0221] In one embodiment of this application, the method may further include the following steps:

[0222] If the model performance information does not meet the expected model performance, the second network-side device will retrain the target model or reselect the model.

[0223] In this embodiment, after determining the model performance information, the second network-side device can further determine whether the model performance information meets the model performance expectation corresponding to the model performance expectation information. If the model performance information does not meet the model performance expectation, the second network-side device can retrain the target model and provide the retrained target model to the first network-side device for practical use. Alternatively, the second network-side device can reselect the model and provide the reselected model to the first network-side device for practical use. This helps improve the accuracy of the model in the practical use stage.

[0224] In one embodiment of this application, the method may further include the following steps:

[0225] Step 1: The first network-side device sends a fourth instruction to the second network-side device based on the model performance information. The fourth instruction is used to instruct the second network-side device to retrain the target model or reselect the model.

[0226] Step 2: The first network-side device receives the retrained model or the reselected model sent by the second network-side device.

[0227] For ease of description, the two steps above will be explained together.

[0228] After the second network-side device determines the model performance information and sends it to the first network-side device, the first network-side device can send a fourth instruction to the second network-side device based on the model performance information. For example, if it is determined that the target model needs to be retrained or a new model selection needs to be performed based on the actual situation, the fourth instruction is sent to the second network-side device. The fourth instruction is used to instruct the second network-side device to retrain the target model or reselect the new model.

[0229] The second network-side device can retrain the target model or reselect the model according to the fourth instruction information, and send the retrained model or the reselected model to the first network-side device after retraining or reselection.

[0230] After receiving the retrained model or the reselected model sent by the second network-side device, the first network-side device can put it into practical use.

[0231] Retraining the target model or reselecting a model can improve the model's accuracy in practical applications.

[0232] In one embodiment of this application, the target data may include label data corresponding to the target model, as well as input data and / or output data corresponding to the target model. The second network-side device determines the model performance information of the target model based on the target data, which may include the following steps:

[0233] Step 1: The second network-side device compares each output data with its associated tag data to obtain the comparison result;

[0234] Step 2: The second network-side device determines the model performance information of the target model based on the comparison results.

[0235] For ease of description, the two steps above will be explained together.

[0236] In this embodiment, the target data may include label data corresponding to the target model, as well as input data and / or output data corresponding to the target model. That is, the target data may include label data and input data corresponding to the target model, or the target data may include label data and output data corresponding to the target model, or the target data may include label data, input data, and output data corresponding to the target model.

[0237] After the second network-side device acquires the target data, if the target data includes the label data and input data corresponding to the target model, then the second network-side device can obtain the output data corresponding to the target model based on the target model and its corresponding input data. If the target data includes the label data and output data corresponding to the target model, or if the target data includes the label data, input data, and output data corresponding to the target model, then the second network-side device obtains the output data corresponding to the target model upon acquiring the target data.

[0238] The correlation between the output data and the label data can be determined based on the time information of the output data and the label data corresponding to the target model.

[0239] The second network-side device can compare each output data point with its associated label data to obtain a comparison result. Based on the comparison result, the accuracy of each output data point relative to its associated label data can be determined, thereby identifying the model performance information of the target model. The second network-side device feeds back the model performance information to the first network-side device, which can then determine further actions based on the model performance information.

[0240] For ease of understanding, the following example uses AnLF as the first network-side device, MTLF as the second network-side device, and ADRF as the third network-side device. Figure 3 The specific examples shown further illustrate the embodiments of this application:

[0241] Step 0 involves a process of task initiation, model selection, and model delivery. Specifically, the consumer network element (Consumer NF) sends a task request to AnLF, requesting the data analysis results for the target task. AnLF then sends a model request to MTLF, requesting the target model to execute the target task. MTLF selects or trains a target model that meets the execution requirements of the target task and sends it to AnLF.

[0242] Steps 1-5:

[0243] AnLF sends an initial request to MTLF, requesting MTLF's assistance in assessing model performance. In one specific scenario, after receiving feedback from MTLF regarding the target model, AnLF might want to know how the model performs in a real-world environment, such as using real, real-time data, or during actual application, to determine whether to continue using the target model for data analysis. Before sending the initial request to MTLF, AnLF can collect the necessary target data and store it in ADRF, then send the ADRF address to MTLF. MTLF can then retrieve the required target data from the specified ADRF.

[0244] Target data can include input data, output data, and label data used in the actual operation of the target model. Input data refers to the data fed into the target model during its use / inference process to enable inference. Output data can be predictions, inference data, or inference results—that is, the output / predicted / inference data obtained after processing / predicting / inference using a set of input data. Label data can be understood as actual measurements in the real environment and can include ground truth values, i.e., correct labeled values. Generally, a set of data can include a set of input data, the corresponding output data, and the corresponding label data.

[0245] Step 1: AnLF collects relevant data from the data source network element. AnLF can determine the target data source network element based on task information. This task information can be carried in the task request sent by the consumer network element in Step 0. Then, AnLF obtains the relevant data from the target data source network element. The relevant data obtained in this step may include input data and / or label data. After collecting the input data, AnLF can input the input data into the target model, and the target model can then perform calculations to obtain the corresponding output data.

[0246] Step 2: AnLF sends a second instruction to ADRF to store the first data in ADRF. Specifically, AnLF sends the first data (the aforementioned input data, output data, and label data) and related information such as the task information, model information, and time information corresponding to the first data to ADRF via the second instruction, so that ADRF can store the first data and / or related information of the first data.

[0247] The time information corresponding to the initial data can help in subsequent model performance evaluation. For example, by comparing the time information corresponding to the output data and the time information corresponding to the label data, the output data of the target model and its corresponding label data can be associated / bound / mapped to compare / calculate / generate model performance information. Furthermore, the time information corresponding to the input data reveals the time range it covers (a specific time frame; for example, if the earliest time information for the input data is Monday of a certain week and the latest time is Friday of that week, then the time range covered by the data is Monday to Friday of that week). This time range can be used to record and ensure the timeliness of model performance information.

[0248] In one implementation method, a second instruction message can be sent via Nadrf_DataManagement_StorageRequest.

[0249] Specifically, the second indication information sent by AnLF to ADRF may include at least one of the following:

[0250] Input data;

[0251] Output data;

[0252] Tag data;

[0253] The time information corresponding to the input data can include the time node information when the input data is generated, the time node information when the input data is acquired by AnLF, and the time node information when the input data is input into the target model.

[0254] The time information corresponding to the output data can include the time node information when the output data is generated. This time node information can be obtained from the time node when the input data is input into the target model, or from the time node when the input data is input into the target model and the target model generates the output data after calculation. The time information corresponding to the output data can also include the target time node information to which the output data is targeted. This target time node information can be obtained from the target time node corresponding to the task predicted by the output data. The time information corresponding to the output data can also include the initial time node corresponding to the output data. This initial time node can be obtained from the time information corresponding to the input data associated with the output data.

[0255] The time information corresponding to the tag data can include the time node information when the tag data was generated and the time node information when the tag data was acquired by AnLF.

[0256] Target task identification information, such as Analytics ID, is used to indicate that the first data is for a specific task;

[0257] The conditions and constraints of the target task, such as Analytic filter information, are used to indicate the filtering information of the data analysis results, including AOI, S-NSSAI, DNN, etc.

[0258] The target information of the task, such as Target of analytic reporting, indicates whether the data analysis targets a single terminal, multiple terminals, or all terminals.

[0259] Model identification information, such as Model ID, is used to indicate that the first data is for a specific model;

[0260] Model filtering information is used to indicate the conditions that the model targeted by the first data needs to meet, such as AOI, S-NSSAI, and DNN.

[0261] Step 3: ADRF stores the received first data and / or related information of the first data.

[0262] Step 4: After storage is completed, ADRF will return storage completion indication information, such as the first storage feedback information. It can also return the storage identification information of this storage, such as the Transaction Reference ID, to AnLF to inform it that the storage was successful.

[0263] Step 5: AnLF sends a first request message to MTLF, requesting MTLF's assistance in detecting the target model's performance and determining model performance information. In this process, the first request message may specifically include at least one of the following:

[0264] The first instruction information is used to instruct the MTLF to determine model representation information. This first instruction information can be explicit, such as being carried by a specific information cell; or it can be implicit, such as being expressed through the name of the message carrying the first request information.

[0265] Model performance expectation information refers to the expected performance of the model. This includes thresholds, preset conditions, etc. This expectation information informs the MTLF of the acceptable model performance. If the model performance calculated by the MTLF does not meet the expectation, subsequent operations are triggered, such as model retraining, model reselection, informing the AnLF of a performance decline, or failure to meet expectations. This expectation information can also be provided to the MTLF by the AnLF when sending a model request in step 0.

[0266] The identification information of ADRF can specifically be the identification information of network elements, such as network element ID, etc.

[0267] The address information of ADRF can specifically be the address information of network elements, such as the IP address and FQDN of the network element;

[0268] The identification information of AnLF can specifically be the identification information of network elements, such as network element ID, etc.

[0269] Storage identification information, such as Transaction Reference ID, is the storage identification information returned by ADRF in step 4;

[0270] Model supervision time requirement information, i.e., the time frame for model supervision, indicates the timeframe within which MTLF should supervise model performance. For example, if the time frame is one month, then MTLF should periodically collect data related to model performance to calculate model performance within the next month. Alternatively, this information may be absent, or the time frame may be very short, such as immediately, indicating that the calculation is a one-off event and does not require long-term data collection for supervision.

[0271] Information required for model supervision region;

[0272] The target task identification information, such as the Analytics ID, is used to indicate that the target data to be obtained is for a specific task;

[0273] The target task's conditional constraints, such as Analytic filter information, are used to indicate the filtering conditions for the data analysis results, including AOI, S-NSSAI, DNN, etc.

[0274] The target information of the task, such as Target of analytic reporting, indicates whether the data analysis targets a single terminal, multiple terminals, or all terminals.

[0275] Model identification information, such as Model ID, is used to indicate that the target data to be obtained is for a specific model;

[0276] Model filtering information: indicates the conditions that the model must meet for the target data to be acquired, such as AOI, S-NSSAI, DNN;

[0277] Data category information is used to indicate the data category of the target data to be acquired, such as the location information of the terminal.

[0278] The time range information corresponding to the target data is used to indicate which time range of target data to obtain. For example, based on the data's generation time, data generated within that time range can be obtained.

[0279] The location range information corresponding to the target data is used to indicate which location range of the target data to obtain. For example, it can be determined according to the location where the data was generated, and the data generated in a certain tracking area or cell can be obtained.

[0280] Steps 6-7: The MTLF sends a second request to the ADRF, requesting to obtain the target data. Specifically, the MTLF determines the identification and address information of the target ADRF using the information from step 5, and sends the second request to the target ADRF to obtain the target data. After receiving the second request, the ADRF feeds back the target data that meets the MTLF's requirements. For example, it can find the data corresponding to the storage identification information based on the AnLF's identification information plus the storage identification information, and feed it back to the MTLF. Alternatively, the ADRF can determine the specific data based on a combination of attribute information of the target data, such as task identification information, model identification information, etc., and feed it back to the MTLF.

[0281] The second request information may include at least one of the following:

[0282] AnLF's identification information;

[0283] Storage identification information of target data;

[0284] The identification information of the target task corresponding to the target data;

[0285] Information regarding the conditions and constraints of the target task;

[0286] Information about the target object of the task.

[0287] The identification information of the model to which the target data is directed;

[0288] Data category information of the target data;

[0289] The time range information corresponding to the target data;

[0290] Location range information corresponding to the target data.

[0291] Steps 8-10: After obtaining the target model's target data in the real-world environment, MTLF calculates the model's performance to obtain model performance information and determines subsequent actions based on this information. MTLF can calculate model performance information by comparing the label data and output data. This model performance information is then compared with the expected model performance information from Step 5 to determine whether model retraining or selecting a new model is necessary. For example, if the model performance information does not meet the expected model performance, retraining or selecting a new model may be necessary. Alternatively, the comparison of these two pieces of information can determine whether to provide feedback on model performance information to AnLF. For instance, feedback can be provided to AnLF when the model performance information does not meet the expected model performance, or AnLF can be notified immediately after the model performance information is calculated, or feedback can be provided periodically.

[0292] In the example above, the first data can be all the data in the target data.

[0293] The following is through Figure 4 Another example is shown to further illustrate the embodiments of this application. In this example, the first data is a portion of the target data. This example has a general flow similar to the previous example, with the difference being in steps 1, 2, 4, 5, and 6.

[0294] Specifically, in step 1 of this example, AnLF is not responsible for collecting all relevant data; it may only collect partial data, such as only collecting input data and not tag data. Therefore, in step 2, AnLF must not only inform ADRF to store the first data it has collected (step 2a), such as the first data including input data, but also inform ADRF to collect and store the second data from the specified data source element, such as tag data (step 2b).

[0295] When AnLF only provides ADRF with partial target data, the third instruction information sent by AnLF to ADRF in step 2b is used to instruct ADRF to collect and store the second data and / or related information of the second data from the target data source network element. Specifically, it may include at least one of the following:

[0296] Information about the device where the second data is located is used by ADRF to determine where to collect the second data.

[0297] The second data contains the identification information of the device, such as the network element ID;

[0298] The second data contains the address information of the device, such as the IP address and FQDN of the network element;

[0299] The collection requirements information corresponding to the second data point can be used by ADRF to determine the second data point to be collected.

[0300] The second data category information is used to indicate what data to collect, such as terminal location information and other data.

[0301] The second data corresponds to the time range information, which is used to indicate which time range of data to collect.

[0302] The location range information corresponding to the second data is used to indicate which location range of data to collect.

[0303] The second data collection time information is used to indicate the time frame within which ADRF should collect the second data. For example, if the collection time frame is one month, then ADRF should collect the second data regularly within the next month. It can also be a period of time in the past.

[0304] In one implementation, step 2b can be accomplished using Nadrf_DataManagement_StorageSubscriptionRequest.

[0305] Because of the addition of step 2b, a second storage feedback message can also be sent in step 4, which includes the corresponding storage identification information. The first request message in step 5 also needs to include the storage identification information so that MTLF can request the corresponding data from ADRF in step 6.

[0306] This application provides a method that enables AnLF to obtain model performance information, helping AnLF to make more reasonable choices. For example, when AnLF finds that the model is performing poorly, it will no longer use the model's data analysis results, thereby avoiding adverse effects on network status and user experience.

[0307] On the other hand, the data and content that ADRF can store have been enhanced, enabling ADRF to store model-related data. The data stored in ADRF can also be collected and used by other network elements. For example, MTLF can obtain this relevant data from ADRF to calculate model performance or train other models based on this information.

[0308] The information acquisition method provided in this application can be executed by an information acquisition device. This application uses an information acquisition device executing the information acquisition method as an example to illustrate the information acquisition device provided in this application.

[0309] like Figure 5 As shown, the information acquisition device 500 may include the following modules:

[0310] The first sending module 510 is used to send a first request message to the second network-side device. The first request message is used to request the second network-side device to determine the model performance information of the target model.

[0311] The first receiving module 520 is used to receive the model performance information of the target model sent by the second network-side device;

[0312] The first request information includes target data and / or attribute information of the target data. The target data is used to detect the model performance of the target model, and the attribute information of the target data is used to acquire the target data.

[0313] Using the apparatus provided in this application embodiment, a first request message is sent to a second network-side device. The first request message is used to request the second network-side device to determine the model performance information of the target model. After receiving the model performance information of the target model determined by the second network-side device based on the target data, the accuracy of the data analysis results of the target model in the actual use stage can be judged based on the model performance information. This allows the data analysis results to be provided to the internal and external devices of the communication network to assist them in making strategic decisions when the accuracy is high, thus ensuring that the internal and external devices of the communication network make correct strategic decisions.

[0314] In one specific embodiment of this application, the first request information further includes at least one of the following:

[0315] First instruction message;

[0316] The model represents the expected information;

[0317] Information on model supervision time requirements;

[0318] Information required for model supervision region;

[0319] The first instruction information is used to instruct the second network-side device to determine the model performance information of the target model;

[0320] Model performance expectation information is used to represent the model performance information of the target model expected by the first network-side device;

[0321] The model supervision time requirement information is used to indicate the time range within which the second network-side device determines the model performance information of the target model;

[0322] The model supervision region requirement information is used to instruct the second network-side device to determine the region range of the target model's model performance information.

[0323] In one specific embodiment of this application, the attribute information of the target data includes at least one of the following:

[0324] Task information corresponding to the target data;

[0325] The target data refers to the information of the model;

[0326] Data description information corresponding to the target data;

[0327] Storage information corresponding to the target data.

[0328] In one specific embodiment of this application, the task information corresponding to the target data includes at least one of the following:

[0329] The identification information of the target task corresponding to the target data;

[0330] Information regarding the conditions and constraints of the target task;

[0331] The target task is aimed at providing information about the target object.

[0332] Among them, the condition constraint information of the target task is used to represent the filtering conditions of the data analysis results corresponding to the target task.

[0333] In one specific embodiment of this application, the information of the model targeted by the target data includes at least one of the following:

[0334] The identification information of the model to which the target data is directed;

[0335] The filtering information for the model targeted by the target data.

[0336] In one specific embodiment of this application, the data description information corresponding to the target data includes at least one of the following:

[0337] Data category information of the target data;

[0338] The time range information corresponding to the target data;

[0339] Location range information corresponding to the target data;

[0340] Among them, the time range information corresponding to the target data is used to indicate the time range in which the time information corresponding to the target data is located;

[0341] The location range information corresponding to the target data is used to indicate the location range of the target data.

[0342] In one specific embodiment of this application, the storage information corresponding to the target data includes at least one of the following:

[0343] Identification information of the third network-side device storing the target data;

[0344] Address information of the third network-side device;

[0345] Storage identification information of target data;

[0346] Identification information of the first network-side device.

[0347] In one specific embodiment of this application, when the first request information includes attribute information of the target data, the information acquisition device 500 further includes:

[0348] The first acquisition module is used to acquire first data before sending the first request information to the second network side device. The first data is part or all of the target data.

[0349] The first sending module 510 is further configured to send second instruction information to the third network-side device. The second instruction information includes first data and / or related information of the first data, and is used to instruct the third network-side device to store the first data and / or related information of the first data.

[0350] In one specific embodiment of this application, the first receiving module 520 is further configured to:

[0351] Receive the first storage feedback information sent by the third network-side device;

[0352] The first storage feedback information includes the storage identifier information corresponding to the first data.

[0353] In one specific embodiment of this application, the first data includes at least one of the following:

[0354] Input data corresponding to the target model;

[0355] Output data corresponding to the target model;

[0356] Label data corresponding to the target model.

[0357] In one specific embodiment of this application, the label data corresponding to the target model includes the actual generated data associated with the input data and / or output data corresponding to the target model.

[0358] In one specific embodiment of this application, the relevant information of the first data includes at least one of the following:

[0359] The task information corresponding to the first data;

[0360] The first data refers to the information of the model;

[0361] The first data corresponds to the time information.

[0362] In one specific embodiment of this application, the task information corresponding to the first data includes at least one of the following:

[0363] The first data corresponds to the identification information of the target task;

[0364] Information regarding the conditions and constraints of the target task;

[0365] Information about the target object of the task.

[0366] In one specific embodiment of this application, the information of the model to which the first data pertains includes at least one of the following:

[0367] The first data refers to the identification information of the model;

[0368] The first data refers to the filtering information of the model.

[0369] In one specific embodiment of this application, the time information corresponding to the first data includes at least one of the following:

[0370] The time information corresponding to the input data;

[0371] Output the time information corresponding to the data;

[0372] The time information corresponding to the tag data.

[0373] In one specific embodiment of this application, the time information corresponding to the input data includes at least one of the following:

[0374] The time point information when the input data was generated;

[0375] The time point information when the input data is acquired by the first network-side device;

[0376] Information on the time point when the input data is input into the target model.

[0377] In one specific embodiment of this application, the time information corresponding to the output data includes at least one of the following:

[0378] Output the time point information when the data was generated;

[0379] The target time point information for the output data;

[0380] The time information corresponding to the input data associated with the output data.

[0381] In one specific embodiment of this application, the time information corresponding to the tag data includes at least one of the following:

[0382] The time point information when the tag data was generated;

[0383] The time point information when the tag data is acquired by the first network-side device.

[0384] In one specific embodiment of this application, the second sending module is further configured to:

[0385] If the first request information includes attribute information of the target data, before sending the first request information to the second network-side device, a third instruction information is sent to the third network-side device. The third instruction information is used to instruct the third network-side device to collect and store the second data and / or related information of the second data. The second data is part or all of the target data.

[0386] In one specific embodiment of this application, the first receiving module 520 is further configured to:

[0387] Receive the second storage feedback information sent by the third network-side device;

[0388] The second storage feedback information includes the storage identifier information corresponding to the second data.

[0389] In one specific embodiment of this application, the relevant information of the second data includes at least one of the following:

[0390] The task information corresponding to the second data;

[0391] The second set of data pertains to information about the model;

[0392] The second data corresponds to the data description information.

[0393] In one specific embodiment of this application, the third instruction information includes information about the device where the second data is located and / or collection request information corresponding to the second data.

[0394] In one specific embodiment of this application, the information of the device where the second data is located includes at least one of the following:

[0395] The identification information of the device where the second data is located;

[0396] The address information of the device where the second data is located.

[0397] In one specific embodiment of this application, the collection requirement information corresponding to the second data includes at least one of the following:

[0398] The second data category information;

[0399] The time range information corresponding to the second data;

[0400] The location range information corresponding to the second data;

[0401] Second data collection time information;

[0402] Among them, the time range information corresponding to the second data is used to indicate the time range in which the time information corresponding to the second data is located;

[0403] The location range information corresponding to the second data is used to indicate the location range of the location information corresponding to the second data;

[0404] The second data collection time information is used to indicate the time requirement for collecting the second data.

[0405] In one specific embodiment of this application, the first data includes input data and / or output data corresponding to the target model, and the second data includes label data corresponding to the target model.

[0406] In one specific embodiment of this application, the first sending module 510 is further configured to send fourth instruction information to the second network-side device based on the model performance information. The fourth instruction information is used to instruct the second network-side device to retrain the target model or reselect the model.

[0407] The first receiving module 520 is also used to receive the retrained model or the reselected model sent by the second network-side device.

[0408] The information acquisition device 500 provided in this application embodiment can achieve... Figure 2 The various processes implemented in the method embodiment shown achieve the same technical effect, and will not be described again here to avoid repetition.

[0409] Corresponding to Figure 2 In addition to the method embodiments shown, this application also provides an information acquisition method, see [link to relevant documentation]. Figure 6 As shown, the method may include the following steps:

[0410] S610: The second network-side device receives the first request information sent by the first network-side device, wherein the first request information includes target data and / or attribute information of the target data, and the attribute information of the target data is used to obtain the target data;

[0411] S620: The second network-side device determines the model performance information of the target model based on the target data;

[0412] S630: The second network-side device sends model performance information to the first network-side device.

[0413] Applying the method provided in the embodiments of this application, the second network-side device receives the first request information sent by the first network-side device, detects the model performance of the target model based on the target data, determines the model performance information, and then sends the model performance information of the target model to the first network-side device. The first network-side device then obtains the model performance information of the target model, and can determine the accuracy of the data analysis results of the target model in the actual use stage based on the model performance information. This allows the device to provide the internal and external devices of the communication network with assistance in making strategic decisions when the accuracy of the data analysis results is high, thus ensuring that the internal and external devices of the communication network make correct strategic decisions.

[0414] In one specific embodiment of this application, the first request information further includes at least one of the following:

[0415] First instruction message;

[0416] The model represents the expected information;

[0417] Information on model supervision time requirements;

[0418] Information required for model supervision region;

[0419] The first instruction information is used to instruct the second network-side device to determine the model performance information of the target model;

[0420] Model performance expectation information is used to represent the model performance information of the target model expected by the first network-side device;

[0421] The model supervision time requirement information is used to indicate the time range within which the second network-side device determines the model performance information of the target model;

[0422] The model supervision region requirement information is used to instruct the second network-side device to determine the region range of the target model's model performance information.

[0423] In one specific embodiment of this application, the attribute information of the target data includes at least one of the following:

[0424] Task information corresponding to the target data;

[0425] The target data refers to the information of the model;

[0426] Data description information corresponding to the target data;

[0427] Storage information corresponding to the target data.

[0428] In one specific embodiment of this application, the task information corresponding to the target data includes at least one of the following:

[0429] The identification information of the target task corresponding to the target data;

[0430] Information regarding the conditions and constraints of the target task;

[0431] The target task is aimed at providing information about the target object.

[0432] Among them, the condition constraint information of the target task is used to represent the filtering conditions of the data analysis results corresponding to the target task.

[0433] In one specific embodiment of this application, the information of the model targeted by the target data includes at least one of the following:

[0434] The identification information of the model to which the target data is applied;

[0435] The filtering information for the model targeted by the target data.

[0436] In one specific embodiment of this application, the data description information corresponding to the target data includes at least one of the following:

[0437] Data category information of the target data;

[0438] The time range information corresponding to the target data;

[0439] Location range information corresponding to the target data;

[0440] Among them, the time range information corresponding to the target data is used to indicate the time range in which the time information corresponding to the target data is located;

[0441] The location range information corresponding to the target data is used to indicate the location range of the target data.

[0442] In one specific embodiment of this application, the storage information corresponding to the target data includes at least one of the following:

[0443] Identification information of the third network-side device storing the target data;

[0444] Address information of the third network-side device;

[0445] Storage identification information of target data;

[0446] Identification information of the first network-side device.

[0447] In one specific embodiment of this application, when the first request information includes attribute information of the target data, before the second network-side device determines the model representation information of the target model based on the target data, the method further includes:

[0448] The second network-side device obtains the target data based on the attribute information of the target data.

[0449] In one specific embodiment of this application, the second network-side device obtains the target data based on the attribute information of the target data, including:

[0450] The second network-side device determines the third network-side device that stores the target data based on the attribute information of the target data.

[0451] The second network-side device sends a second request message to the third network-side device. The second request message is used to request the acquisition of target data.

[0452] The second network-side device receives the target data sent by the third network-side device.

[0453] In one specific embodiment of this application, the second request information includes at least one of the following:

[0454] Identification information of the first network-side device;

[0455] Storage identification information of target data;

[0456] Task information corresponding to the target data;

[0457] The target data refers to the information of the model;

[0458] Data description information corresponding to the target data.

[0459] In one specific embodiment of this application, after the second network-side device determines the model performance information of the target model based on the target data, it further includes:

[0460] The second network-side device determines whether the model performance information meets the model performance expectation corresponding to the model performance expectation information.

[0461] The second network-side device sends model performance information to the first network-side device, including:

[0462] The second network-side device sends the model performance information to the first network-side device, depending on whether the model performance information meets or does not meet the model performance expectations.

[0463] In one specific embodiment of this application, it further includes:

[0464] If the model performance information does not meet the expected model performance, the second network-side device will retrain the target model or reselect the model.

[0465] In one specific embodiment of this application, after the second network-side device sends model representation information to the first network-side device, it further includes:

[0466] The second network-side device receives the fourth indication information sent by the first network-side device;

[0467] The second network-side device retrains the target model or reselects the model based on the fourth instruction information;

[0468] The second network-side device sends the retrained model or the reselected model to the first network-side device.

[0469] In one specific embodiment of this application, the target data includes label data corresponding to the target model, as well as input data and / or output data corresponding to the target model. The second network-side device determines the model performance information of the target model based on the target data, including:

[0470] The second network-side device compares each output data with its associated tag data to obtain the comparison result;

[0471] The second network-side device determines the model performance information of the target model based on the comparison results.

[0472] Figure 6 The implementation process of the method embodiment shown can be found in [reference needed]. Figure 2 The implementation processes of the methods shown in the embodiments achieve the same technical effects, and will not be described again here to avoid repetition.

[0473] The information acquisition method provided in this application can be executed by an information acquisition device. This application uses an information acquisition device executing the information acquisition method as an example to illustrate the information acquisition device provided in this application.

[0474] like Figure 7 As shown, the information acquisition device 700 may include the following modules:

[0475] The second receiving module 710 is used to receive first request information sent by the first network side device, the first request information including target data and / or attribute information of the target data;

[0476] The first determining module 720 is used to determine the model performance information of the target model based on the target data;

[0477] The second sending module 730 is used to send model performance information to the first network-side device.

[0478] The apparatus provided in this application receives a first request message sent by a first network-side device, detects the model performance of the target model based on the target data, determines the model performance information, and then sends the model performance information of the target model to the first network-side device. The first network-side device then obtains the model performance information of the target model and can determine the accuracy of the data analysis results of the target model in the actual use stage based on the model performance information. This allows the apparatus to provide the internal and external devices of the communication network with assistance in making strategic decisions when the accuracy of the data analysis results is high, thus ensuring that the internal and external devices of the communication network make correct strategic decisions.

[0479] In one specific embodiment of this application, the first request information further includes at least one of the following:

[0480] First instruction message;

[0481] The model represents the expected information;

[0482] Information on model supervision time requirements;

[0483] Information required for model supervision region;

[0484] The first instruction information is used to instruct the second network-side device to determine the model performance information of the target model;

[0485] Model performance expectation information is used to represent the model performance information of the target model expected by the first network-side device;

[0486] The model supervision time requirement information is used to indicate the time range within which the second network-side device determines the model performance information of the target model;

[0487] The model supervision region requirement information is used to instruct the second network-side device to determine the region range of the target model's model performance information.

[0488] In one specific embodiment of this application, the attribute information of the target data includes at least one of the following:

[0489] Task information corresponding to the target data;

[0490] The target data refers to the information of the model;

[0491] Data description information corresponding to the target data;

[0492] Storage information corresponding to the target data.

[0493] In one specific embodiment of this application, the task information corresponding to the target data includes at least one of the following:

[0494] The identification information of the target task corresponding to the target data;

[0495] Information regarding the conditions and constraints of the target task;

[0496] The target information of the task;

[0497] Among them, the condition constraint information of the target task is used to represent the filtering conditions of the data analysis results corresponding to the target task.

[0498] In one specific embodiment of this application, the information of the model targeted by the target data includes at least one of the following:

[0499] The identification information of the model to which the target data is applied;

[0500] The filtering information for the model targeted by the target data.

[0501] In one specific embodiment of this application, the data description information corresponding to the target data includes at least one of the following:

[0502] Data category information of the target data;

[0503] The time range information corresponding to the target data;

[0504] Location range information corresponding to the target data;

[0505] Among them, the time range information corresponding to the target data is used to indicate the time range in which the time information corresponding to the target data is located;

[0506] The location range information corresponding to the target data is used to indicate the location range of the target data.

[0507] In one specific embodiment of this application, the storage information corresponding to the target data includes at least one of the following:

[0508] Identification information of the third network-side device storing the target data;

[0509] Address information of the third network-side device;

[0510] Storage identification information of target data;

[0511] Identification information of the first network-side device.

[0512] In one specific embodiment of this application, the information acquisition device 700 further includes a second acquisition module, used for:

[0513] If the first request information includes the attribute information of the target data, the target data is obtained based on the attribute information of the target data before determining the model performance information of the target model based on the target data.

[0514] In one specific embodiment of this application, the second acquisition module is used for:

[0515] Based on the attribute information of the target data, determine the third network-side device that stores the target data;

[0516] Send a second request message to the third network-side device. The second request message is used to request the acquisition of target data.

[0517] Receive target data sent by a third network-side device.

[0518] In one specific embodiment of this application, the second request information includes at least one of the following:

[0519] Identification information of the first network-side device;

[0520] Storage identification information of target data;

[0521] Task information corresponding to the target data;

[0522] Model information corresponding to the target data;

[0523] Data description information corresponding to the target data.

[0524] In one specific embodiment of this application, the information acquisition device 700 further includes:

[0525] The second determining module is used to determine whether the model performance information meets the model performance expectation corresponding to the model performance expectation information after determining the model performance information of the target model based on the target data.

[0526] The second sending module 730 is also used to send the model performance information to the first network-side device when the model performance information meets or does not meet the model performance expectations.

[0527] In one specific embodiment of this application, the information acquisition device 700 further includes a first execution module, used for:

[0528] If the model performance information does not meet the expected model performance, the target model should be retrained or a new model should be selected.

[0529] In one specific embodiment of this application, the information acquisition device 700 further includes a second execution module, used for:

[0530] After sending the model performance information to the first network-side device, receive the fourth instruction information sent by the first network-side device;

[0531] Based on the fourth instruction, the target model is retrained or a new model is selected.

[0532] The retrained model or the reselected model is sent to the first network-side device.

[0533] In one specific embodiment of this application, the target data includes label data corresponding to the target model, and input data and / or output data corresponding to the target model. The first determining module 720 is used for:

[0534] Each output data point is compared with its associated label data to obtain the comparison results;

[0535] Based on the comparison results, determine the model performance information of the target model.

[0536] The information acquisition device 700 provided in this application embodiment can achieve... Figure 6 The various processes implemented in the method embodiment shown achieve the same technical effect, and will not be described again here to avoid repetition.

[0537] Corresponding to Figure 2 , Figure 6 The method embodiment shown in this application also provides an information acquisition method, such as... Figure 8 As shown, the method may include the following steps:

[0538] S810: The third network-side device receives the second and third indication information sent by the first network-side device;

[0539] S820: The third network-side device stores the first data and / or related information of the first data according to the second instruction information;

[0540] S830: The third network-side device collects and stores the second data and / or related information of the second data according to the third instruction information;

[0541] The first data and the second data are partial data of the target data. The target data is used to determine the model performance information of the target model, and the relevant information of the first data and the second data is used to obtain the target data.

[0542] By applying the method provided in the embodiments of this application, the third network-side device stores first data and / or related information of the first data according to the instruction information sent by the first network-side device, and collects and stores second data and / or related information of the second data, thereby reducing the amount of data exchanged with the first network-side device, saving network resources, and facilitating the accurate provision of target data when there is a request to obtain target data.

[0543] In one specific embodiment of this application, the method may further include the following steps:

[0544] The third network-side device receives a second request message sent by the second network-side device. The second request message is used to obtain target data.

[0545] The third network-side device sends the target data to the second network-side device.

[0546] Figure 8 The implementation process of the method embodiment shown can be found in [reference needed]. Figure 2 , Figure 6 The implementation processes of the methods shown in the embodiments achieve the same technical effects, and will not be described again here to avoid repetition.

[0547] The information acquisition method provided in this application can be executed by an information acquisition device. This application uses an information acquisition device executing the information acquisition method as an example to illustrate the information acquisition device provided in this application.

[0548] like Figure 9 As shown, the information acquisition device 900 may include the following modules:

[0549] The third receiving module 910 is used to receive the second indication information and the third indication information sent by the first network-side device;

[0550] Storage module 920 is used to store first data and / or related information of the first data according to the second instruction information; and to collect and store second data and / or related information of the second data according to the third instruction information.

[0551] The first data and the second data are partial data of the target data. The target data is used to determine the model performance information of the target model, and the relevant information of the first data and the second data is used to obtain the target data.

[0552] Using the apparatus provided in the embodiments of this application, the third network-side device stores first data and / or related information of the first data according to the instruction information sent by the first network-side device, and collects and stores second data and / or related information of the second data, thereby reducing the amount of data exchanged with the first network-side device, saving network resources, and facilitating the accurate provision of target data when there is a request to obtain target data.

[0553] In one specific embodiment of this application, the information acquisition device 900 further includes:

[0554] The fourth receiving module is used to receive the second request information sent by the second network-side device, and the second request information is used to obtain target data;

[0555] The third sending module is used to send the target data to the second network-side device.

[0556] The information acquisition device 900 provided in this application embodiment can achieve... Figure 8 The various processes implemented in the method embodiments shown achieve the same technical effect, and will not be described again here to avoid repetition.

[0557] Corresponding to the above method embodiments and apparatus embodiments, such as Figure 10 As shown, this application embodiment also provides a network-side device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various steps of the above-described information acquisition method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0558] Specifically, embodiments of this application also provide a network-side device. For example... Figure 11 As shown, the network-side device 1100 includes a processor 1101, a network interface 1102, and a memory 1103. The network interface 1102 is, for example, a common public radio interface (CPRI).

[0559] Specifically, the network-side device 1100 of this embodiment further includes: instructions or programs stored in memory 1103 and executable on processor 1101, wherein processor 1101 calls the instructions or programs in memory 1103 to execute. Figure 5 or Figure 7 or Figure 9 The methods executed by each module shown achieve the same technical effect, and to avoid repetition, they will not be described in detail here.

[0560] This application embodiment also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the above-described functionality. Figure 2 The method embodiment shown or Figure 6 The method embodiment shown or Figure 8 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0561] The processor is the processor in the terminal described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0562] This application embodiment also provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the above. Figure 2The method embodiment shown or Figure 6 The method embodiment shown or Figure 8 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0563] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0564] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0565] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An information acquisition method, characterized in that, include: The first network-side device sends a first request message to the second network-side device. The first request message is used to request the second network-side device to determine the model performance information of the target model. The model performance information of the target model is used to indicate the correctness and / or error of the reasoning result during actual reasoning. The first network-side device receives the model performance information of the target model sent by the second network-side device; Wherein, the first request information includes target data and / or attribute information of the target data, the target data is used to determine the model performance information of the target model, the attribute information of the target data is used to obtain the target data, and the target data includes label data corresponding to the target model, as well as input data and / or output data corresponding to the target model; The first request information also includes at least one of the following: First instruction message; The model represents expected information; Wherein, the first indication information is used to instruct the second network-side device to determine the model performance information of the target model; The model performance expectation information is used to represent the model performance information of the target model expected by the first network-side device.

2. The method according to claim 1, characterized in that, The first request information also includes at least one of the following: Information on model supervision time requirements; Information required for model supervision region; The model supervision time requirement information is used to instruct the second network-side device on the time range within which it determines the model performance information of the target model; The model supervision region requirement information is used to instruct the second network-side device to determine the region range of the target model's model performance information.

3. The method according to claim 1, characterized in that, The attribute information of the target data includes at least one of the following: The task information corresponding to the target data; The target data refers to the information of the model; The data description information corresponding to the target data; The storage information corresponding to the target data.

4. The method according to claim 3, characterized in that, The task information corresponding to the target data includes at least one of the following: The target data corresponds to the identification information of the target task; The conditional constraints of the target task; The target task targets information about the object; The condition constraint information of the target task is used to represent the filtering conditions of the data analysis results corresponding to the target task.

5. The method according to claim 3, characterized in that, The target data for the model includes at least one of the following: The target data refers to the identification information of the model; The target data refers to the filtering information of the model.

6. The method according to claim 3, characterized in that, The data description information corresponding to the target data includes at least one of the following: The data category information of the target data; The time range information corresponding to the target data; The location range information corresponding to the target data; Wherein, the time range information corresponding to the target data is used to indicate the time range in which the time information corresponding to the target data is located; The location range information corresponding to the target data is used to represent the location range of the location information corresponding to the target data.

7. The method according to claim 3, characterized in that, The storage information corresponding to the target data includes at least one of the following: Identification information of the third network-side device storing the target data; The address information of the third network-side device; The storage identification information of the target data; The identification information of the first network-side device.

8. The method according to any one of claims 1 to 7, characterized in that, If the first request information includes attribute information of the target data, the method further includes the following before the first network-side device sends the first request information to the second network-side device: The first network-side device acquires first data, which is part or all of the target data. The first network-side device sends a second indication message to the third network-side device. The second indication message includes the first data and / or related information of the first data. The second indication message is used to instruct the third network-side device to store the first data and / or related information of the first data.

9. The method according to claim 8, characterized in that, Also includes: The first network-side device receives the first storage feedback information sent by the third network-side device; The first storage feedback information includes the storage identifier information corresponding to the first data.

10. The method according to claim 8, characterized in that, The first data includes at least one of the following: The input data corresponding to the target model; The output data corresponding to the target model; The label data corresponding to the target model.

11. The method according to claim 10, characterized in that, The label data corresponding to the target model includes the actual generated data associated with the input data and / or the output data corresponding to the target model.

12. The method according to claim 10, characterized in that, The relevant information of the first data includes at least one of the following: Task information corresponding to the first data; The information of the model targeted by the first data; The time information corresponding to the first data.

13. The method according to claim 12, characterized in that, The task information corresponding to the first data includes at least one of the following: The identification information of the target task corresponding to the first data; The conditional constraints of the target task; The target task refers to the object information.

14. The method according to claim 12, characterized in that, The information of the model targeted by the first data includes at least one of the following: The identification information of the model to which the first data is applied; The first data refers to the filtering information of the model.

15. The method according to any one of claims 1 to 7, characterized in that, If the first request information includes attribute information of the target data, the method further includes the following before the first network-side device sends the first request information to the second network-side device: The first network-side device sends a third instruction message to the third network-side device. The third instruction message is used to instruct the third network-side device to collect and store the second data and / or related information of the second data. The second data is part or all of the target data.

16. The method according to claim 15, characterized in that, Also includes: The first network-side device receives the second storage feedback information sent by the third network-side device; The second storage feedback information includes the storage identifier information corresponding to the second data.

17. The method according to claim 15, characterized in that, The relevant information for the second data includes at least one of the following: The task information corresponding to the second data; The second data pertains to information about the model; The data description information corresponding to the second data.

18. The method according to claim 15, characterized in that, The third indication information includes information about the device where the second data is located and / or the collection requirement information corresponding to the second data.

19. The method according to claim 18, characterized in that, The information of the device where the second data is located includes at least one of the following: The identification information of the device where the second data is located; The address information of the device where the second data is located.

20. The method according to claim 18, characterized in that, The collection requirements for the second data include at least one of the following: The second data contains data category information; The second data corresponds to the time range information; The location range information corresponding to the second data; The second data collection time information; Wherein, the time range information corresponding to the second data is used to indicate the time range in which the time information corresponding to the second data is located; The location range information corresponding to the second data is used to indicate the location range in which the location information corresponding to the second data is located; The collection time information of the second data is used to indicate the time requirement for collecting the second data.

21. An information acquisition device, characterized in that, include: The first sending module is used to send a first request message to the second network-side device. The first request message is used to request the second network-side device to determine the model performance information of the target model. The model performance information of the target model is used to indicate the correctness and / or error of the reasoning result during actual reasoning. The first receiving module is used to receive the model performance information of the target model sent by the second network-side device; Wherein, the first request information includes target data and / or attribute information of the target data, the target data is used to determine the model performance information of the target model, the attribute information of the target data is used to obtain the target data, and the target data includes label data corresponding to the target model, as well as input data and / or output data corresponding to the target model; The first request information also includes at least one of the following: First instruction message; The model represents expected information; Wherein, the first indication information is used to instruct the second network-side device to determine the model performance information of the target model; The model performance expectation information is used to represent the model performance information of the target model expected by the first network-side device.

22. An information acquisition method, characterized in that, include: The second network-side device receives a first request information sent by the first network-side device, wherein the first request information includes target data and / or attribute information of the target data, the attribute information of the target data is used to obtain the target data, and the target data includes label data corresponding to the target model, as well as input data and / or output data corresponding to the target model; The second network-side device determines the model performance information of the target model based on the target data. The model performance information of the target model is used to indicate the correctness and / or error of the reasoning results during actual reasoning. The second network-side device sends the model performance information to the first network-side device; The first request information also includes at least one of the following: First instruction message; The model represents expected information; Wherein, the first indication information is used to instruct the second network-side device to determine the model performance information of the target model; The model performance expectation information is used to represent the model performance information of the target model expected by the first network-side device.

23. The method according to claim 22, characterized in that, The first request information also includes at least one of the following: Information on model supervision time requirements; Information required for model supervision region; The model supervision time requirement information is used to instruct the second network-side device on the time range within which it determines the model performance information of the target model; The model supervision region requirement information is used to instruct the second network-side device to determine the region range of the target model's model performance information.

24. The method according to claim 22, characterized in that, The attribute information of the target data includes at least one of the following: The task information corresponding to the target data; The target data refers to the information of the model; The data description information corresponding to the target data; The storage information corresponding to the target data.

25. The method according to claim 22, characterized in that, If the first request information includes attribute information of the target data, before the second network-side device determines the model representation information of the target model based on the target data, the method further includes: The second network-side device obtains the target data based on the attribute information of the target data.

26. The method according to claim 25, characterized in that, The second network-side device obtains the target data based on the attribute information of the target data, including: The second network-side device determines the third network-side device that stores the target data based on the attribute information of the target data; The second network-side device sends a second request message to the third network-side device, the second request message being used to request the acquisition of the target data; The second network-side device receives the target data sent by the third network-side device.

27. The method according to claim 26, characterized in that, The second request information includes at least one of the following: Identification information of the first network-side device; The storage identification information of the target data; The task information corresponding to the target data; The target data refers to the information of the model; The data description information corresponding to the target data.

28. The method according to claim 23, characterized in that, After the second network-side device determines the model performance information of the target model based on the target data, the method further includes: If the model performance information does not meet the expected model performance, the second network-side device may retrain the target model or reselect the model.

29. The method according to any one of claims 22 to 28, characterized in that, The target data includes label data corresponding to the target model, as well as input data and / or output data corresponding to the target model. Based on the target data, the second network-side device determines the model performance information of the target model, including: The second network-side device compares each of the output data with the associated tag data to obtain a comparison result; The second network-side device determines the model performance information of the target model based on the comparison results.

30. An information acquisition device, characterized in that, include: The second receiving module is used to receive a first request information sent by the first network-side device. The first request information includes target data and / or attribute information of the target data. The attribute information of the target data is used to acquire the target data. The target data includes label data corresponding to the target model, as well as input data and / or output data corresponding to the target model. The first determining module is used to determine the model performance information of the target model based on the target data. The model performance information of the target model is used to indicate the correctness and / or error of the reasoning result during actual reasoning. The second sending module is used to send the model performance information to the first network-side device; The first request information also includes at least one of the following: First instruction message; The model represents expected information; Wherein, the first indication information is used to instruct the second network-side device to determine the model performance information of the target model; The model performance expectation information is used to represent the model performance information of the target model expected by the first network-side device.

31. An information acquisition method, characterized in that, include: The third network-side device receives the second and third indication information sent by the first network-side device; The third network-side device stores the first data and / or related information of the first data according to the second instruction information; The third network-side device collects and stores the second data and / or related information of the second data according to the third instruction information; The first data and the second data are partial data of the target data. The target data is used to determine the model performance information of the target model. The relevant information of the first data and the relevant information of the second data are used to acquire the target data. The target data includes the label data corresponding to the target model, as well as the input data and / or output data corresponding to the target model. The model performance information of the target model is used to indicate the correctness and / or error of the reasoning result during actual reasoning.

32. The method according to claim 31, characterized in that, Also includes: The third network-side device receives a second request information sent by the second network-side device, the second request information being used to obtain the target data; The third network-side device sends the target data to the second network-side device.

33. An information acquisition device, characterized in that, include: The third receiving module is used to receive the second indication information and the third indication information sent by the first network-side device; The storage module is configured to store the first data and / or related information of the first data according to the second instruction information; Collect and store the second data and / or related information of the second data in accordance with the third instruction information; The first data and the second data are partial data of the target data. The target data is used to determine the model performance information of the target model. The relevant information of the first data and the relevant information of the second data are used to acquire the target data. The target data includes the label data corresponding to the target model, as well as the input data and / or output data corresponding to the target model. The model performance information of the target model is used to indicate the correctness and / or error of the reasoning result during actual reasoning.

34. A network-side device, characterized in that, The method includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the information acquisition method as described in any one of claims 1 to 20, or to implement the information acquisition method as described in any one of claims 22 to 29, or to implement the steps of the information acquisition method as described in claim 31 or 32.

35. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the information acquisition method as described in any one of claims 1 to 20, or the information acquisition method as described in any one of claims 22 to 29, or the steps of the information acquisition method as described in claim 31 or 32.

Images