Model selection method, network side device and readable storage medium

Abstract

The present application discloses a model selection method, an apparatus and a network side device, which belong to the technical field of communication. The model selection method according to the embodiment of the present application includes: a first network element records model information of N candidate models, and the N candidate models are applied to a data analysis task corresponding to a first analysis identifier; the first network element receives a first model request message from a second network element; and if the model information of M candidate models among the N candidate models matches the model information corresponding to the first model request message, the first network element sends model information of P candidate models among the M candidate models to the second network element, where N, M and P are all positive integers, and N≧M≧P.

Description

[Technical field]

[0001] The present application relates to the technical field of communications, and in particular to a model selection method, apparatus and network side device. [Background technology]

[0002] In a communication network, some network elements (such as a Network Data Analytics Function (NWDAF)) can perform intelligent data analysis to generate data analysis results for some tasks. The data analysis results can assist the policy decisions made by devices inside and outside the network, thereby improving the degree of intelligence of the policy decisions made by the devices.

[0003] The Model Training Logical Function (MTLF) in the NWDAF can obtain a model to be applied to a certain AI data analysis task by training an artificial intelligence (AI) / machine language (ML) model based on the training data. The Analytics Logical Function (AnLF) in the NWDAF can obtain a data analysis result (also called an inference result (analytics)) of a certain specific AI data analysis task by performing model inference based on the AI / ML model and inference input data. This allows the Policy Control Function (PCF) in the network to intelligently execute the corresponding policy based on the data analysis result. For example, an intelligent user stay policy can be formulated based on the analysis result of the user's service behavior to improve the user's service experience. Alternatively, the Access and Mobility Management Function (AMF) can intelligently execute mobility management operations based on the data analysis result, for example, intelligently paging the user based on the analysis result of the user's movement trajectory to improve the paging reachability.

[0004] However, in the above process, whether devices inside and outside the network can make correct policy decisions based on data analysis results is based on the premise that the data analysis results are correct. If the accuracy of the data analysis results is low, incorrect information may be provided to devices inside and outside the network for reference, which may lead to incorrect policy decisions or inappropriate operations by devices inside and outside the network. Therefore, how to ensure the data analysis performance (e.g., accuracy) of AI / ML models is an urgent issue. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application provide a model selection method, apparatus, and network side device that can solve the problem of poor data analysis performance of conventional AI / ML models. [Means for solving the problem]

[0006] In a first aspect, a model selection method is provided, the method including: a first network element recording model information of N candidate models, wherein the N candidate models are applied to a data analysis task corresponding to a first analysis identifier; the first network element receiving a first model request message from a second network element; and if model information of M candidate models among the N candidate models matches model information corresponding to the first model request message, the first network element sending model information of P candidate models among the M candidate models to the second network element, where N, M and P are all positive integers, and N≧M≧P.

[0007] In a second aspect, a model selection device is provided, the model selection device including: a recording module used to record model information of N candidate models, where the N candidate models are applied to a data analysis task corresponding to a first analysis identifier; a first receiving module used to receive a first model request message from a second network element; and a first transmitting module used to transmit model information of P candidate models of the M candidate models to the second network element if the model information of M candidate models of the N candidate models matches the model information corresponding to the first model request message, where N, M and P are all positive integers, and N≧M≧P.

[0008] In a third aspect, there is provided a network side device including a processor and a memory, the memory storing programs or commands executable by the processor, and when the programs or commands are executed by the processor, the steps of the method according to the first aspect are realized.

[0009] In a fourth aspect, there is provided a network side device including a processor and a communication interface, wherein the processor is used for recording model information of N candidate models, where the N candidate models are applied to a data analysis task corresponding to a first analysis identifier, and the communication interface is used for receiving a first model request message from a second network element and, if the model information of M candidate models among the N candidate models matches the model information corresponding to the first model request message, transmitting model information of P candidate models among the M candidate models to the second network element, where N, M and P are all positive integers, and N≧M≧P.

[0010] In a fifth aspect, there is provided a readable storage medium having stored thereon a program or commands which, when executed by a processor, cause the steps of the model selection method according to the first aspect to be implemented.

[0011] In a sixth aspect, there is provided a chip including a coupled processor and a communication interface, the processor being adapted to execute programs or commands to implement the model selection method described in the first aspect.

[0012] In a seventh aspect, there is provided a computer program / program product stored on a storage medium and which, when executed by at least one processor, implements the steps of the model selection method according to the first aspect. Effect of the Invention

[0013] In an embodiment of the present application, the first network element records model information of N candidate models applied to a data analysis task corresponding to a first analysis identifier, receives a first model request message from a second network element, and can send model information of P candidate models among the M candidate models to the second network element if the model information of M candidate models among the N candidate models matches the model information corresponding to the first model request message, where N, M and P are all positive integers, and N≧M≧P. With this method, the first network element can record model information of the candidate models applied to the data analysis task, so that after the first network element receives the first model request message, it can determine whether the model information of the recorded candidate models matches the model information corresponding to the first model request message, and thus can determine an optimal model that meets the first model request message or a model that meets the performance requirements. In this way, the performance of the model in data analysis can be ensured. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram of a wireless communication system provided by an embodiment of the present application. [Diagram 2] 1 is a schematic flow chart of a model selection method provided by an embodiment of the present application; [Diagram 3] FIG. 1 is a schematic diagram of the application flow of the model selection method provided by the embodiment of the present application. [Figure 4] FIG. 2 is a second schematic diagram of the application flow of the model selection method provided by the embodiment of the present application. [Diagram 5] FIG. 2 is a schematic diagram of the structure of a model selection device provided by an embodiment of the present application; [Figure 6] FIG. 2 is a schematic diagram of the structure of a network side device provided by an embodiment of the present application; [Figure 7] FIG. 2 is a hardware schematic diagram of a network side device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In the following, the technical solutions in the embodiments of the present application will be clearly described with reference to the drawings in the embodiments of the present application, and it is to be understood that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application are within the scope of protection of the present application.

[0016] The terms "first", "second", etc. in the specification and claims of this application are not intended to describe a particular order or sequence, but are intended to distinguish between similar objects. It should be understood that the terms used in this way may be interchangeable in some cases so that the embodiments of this application can be implemented in an order other than that shown or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first model may be one or more. In the specification and claims, "and / or" indicates at least one of the objects connected, and the sign " / " generally indicates that the related objects before and after are in an "or" relationship.

[0017] It should be noted that the technology described in the embodiments of the present application is not limited to be used in Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but may also be used in other wireless communication systems such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), and other systems. The technical terms "system" and "network" in the embodiments of the present application are often used interchangeably, and the technology described may be used in the above systems and wireless technologies as well as other systems and wireless technologies. In the following description, for illustrative purposes, a 5G system is described and 5G terminology is used in much of the following description, however, these techniques may be applied outside of 5G system applications, such as in 6th Generation (6G) communication systems.

[0018] 1 shows a block diagram of a wireless communication system to which the embodiment of the present application can be applied. The wireless communication system includes a terminal 11 and a network side device 12. The terminal is also called User Equipment (UE), and the terminal 11 may be a terminal side device such as a mobile phone, a tablet personal computer, a laptop computer (also called a notebook computer), a personal digital assistant (PDA), a palmtop computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, a vehicle-mounted device (VUE), a pedestrian terminal (PUE), a smart home (a home device with a wireless communication function such as a refrigerator, a television, a washing machine, or furniture), a game machine, a personal computer, an automated teller machine, or a kiosk. The wearable device includes a smart watch, a smart bracelet, a smart earphone, a smart glass, a smart accessory (such as a smart bangle, a smart chain bracelet, a smart ring, a smart necklace, a smart anklet, a smart wristband, a smart wear, etc.) It should be noted that the embodiment of the present application does not limit the specific type of the terminal 11. The network side device 12 may include an access network device or a core network device, and the access network device 12 is also called a radio access network device, a radio access network (Radio Access Network, RAN), a radio access network function, or a radio access network unit.The access network device 12 may include a base station, a WLAN access point, or a WiFi node, etc., and the base station may also be called a Node B, an evolved Node B (eNB), an access point, a base transceiver station (BTS), a radio base station, a radio transceiver, a basic service set (BSS), an extended service set (ESS), a home node B, a home evolved node B, a transmitting receiving point (TRP), or any other suitable technical term in the field, and the base station is not limited to a specific technical term as long as the same technical effect is achieved. It should be noted that in the embodiments of the present application, only a base station in an NR system is described as an example, and the specific type of the base station is not limited.The core network device includes the following: Core Network Node, Core Network Function, Network Data Analytics Function (NWDAF), analytics data storage function (ADRF), Mobility Management Entity (MME), Access and Mobility Management Function (AMF), Session Management Function (SMF), User Plane Function (UPF), Policy Control Function (PCF), Policy and Charging Rules Function (PCRF), Edge Application Server Discovery Function (EASDF), Unified Data Management (UDM), Unified Data Repository (UDR), Home Subscriber Server (HSS), Centralized network configuration (CNC), Network Repository Function (NRF), Network Exposure Function (NEF), Local NEF (Local NEF), and Local NEF (Local NEF). The core network function may include, but is not limited to, at least one of a Binding Support Function (NEF or L-NEF), a Binding Support Function (BSF), an Application Function (AF), etc. It should be noted that in the embodiments of the present application, only the core network device in the 5G system is described as an example, and the specific type of the core network device is not limited.

[0019] In the embodiments of the present application, the MTLF and the AnLF may be independent network elements or logical function modules within the NWDAF. The MTLF and the AnLF may be located in the same NWDAF, may be located in different NWDAFs, or may be located outside the NWDAF.

[0020] In the following, the model selection method provided by the embodiments of the present application will be described in detail with reference to the drawings through several embodiments and their application scenarios.

[0021] As shown in FIG. 2, an embodiment of the present application provides a model selection method, which includes the following steps 201 to 203.

[0022] Step 201: A first network element records model information of N candidate models.

[0023] The N candidate models may be applied to a data analysis task corresponding to the first analytic ID. The data analysis task may also be referred to as an inference task.

[0024] It should be noted that the analytic ID may be used to identify a data analysis task and may itself be a type, i.e., the analytic ID may be used to indicate the type of the data analysis task. It can be understood that the data analysis task corresponding to the above first analytic ID may be the data analysis task identified by the first analytic ID.

[0025] For example, if the analytic ID is UE mobility, the data analysis task identified by the analytic ID is prediction of the UE's movement trajectory. If the analytic ID is UE communication, the data analysis task identified by the analytic ID is prediction of the UE's communication performance and behavior.

[0026] Optionally, in the embodiment of the present application, the first network element may include a MTLF or an analytics data storage function (ADRF).

[0027] In some embodiments, the model information for the N candidate models recorded by the MTLF may be model information determined by the MTLF itself, and the model information for the N candidate models recorded by the ADRF may be model information obtained from the MTLF.

[0028] Optionally, in the embodiment of the present application, the above N candidate models may be multiple different models, or may be different models obtained by training or retraining one model multiple times.Specifically, it can be determined according to actual usage needs, and is not limited in the embodiment of the present application.

[0029] Optionally, in an embodiment of the present application, the model information of the candidate model may include at least one of the following a. to f.:

[0030] A model identifier (model ID) of the candidate model, which is used to uniquely identify the candidate model.

[0031] b. Model file information of the candidate model, in which any information regarding the candidate model, such as the network structure, weight parameters, input / output data, etc. of the candidate model, may be stored.

[0032] c. Download address information of the candidate model, which is used to indicate the storage address of the model file of the candidate model, i.e., the download address information of the candidate model, which is used to indicate the download address of the model file of the candidate model. For example, if the candidate model is stored in the ADRF, the download address information may be the address information of the ADRF.

[0033] d. An analysis identifier for the candidate model that is used to identify the data analysis task to which the candidate model is to be applied, e.g., a first analysis identifier.

[0034] e. Usage scope information of the candidate model, which may indicate at least one of a usage area range, a usage time range, and a usage target range.

[0035] The use object may include one UE, multiple UEs, or any UE. Specifically, it can be determined according to actual use needs, and is not limited in the embodiment of the present application. The UE can be identified by UE ID, UE address information, UE group ID, etc.

[0036] f. Model performance information of the candidate models.

[0037] In embodiments of the present application, model performance information may be used to characterize the degree of agreement or matching between the model output (i.e., predicted or statistical value) and the true value, where matching refers to the deviation between the predicted value (or statistical value) and the true value within a predefined range.

[0038] In one embodiment, the degree of matching can be obtained by the result of one or more comparisons, for example, by comparing the predicted value with the true value multiple times, or in another embodiment, the degree of matching can be a value of the ratio of the number of times the predicted value and the true value coincide or match with each other to the total number of predictions.

[0039] Optionally, in the embodiment of the present application, the performance indicated by the model performance information includes at least one of the following: model accuracy, model mean absolute error (MAE). Of course, in practical implementation, the model performance information may further include other information that can reflect the performance of the model, and is specifically determined according to practical use needs.

[0040] It should be noted that the performance indicated by the model performance information may be expressed in any possible form, such as a numerical value, a level, a percentage, etc. For example, when the performance indicated by the model performance indication information includes a model accuracy, the model accuracy may be expressed as an exact numerical value.

[0041] In the embodiment of the present application, the model information of the candidate model can reflect the performance performance and the corresponding usage scope in the actual use process of the candidate model, so that when another network element (e.g., the second network element in the embodiment of the present application) requests a model, the first network element can determine an optimal model that matches the model request message (e.g., the first model request message in the embodiment of the present application) sent by the other network element through these information. For example, if the model information includes usage scope information, the first network element can provide different models for different usage scopes, thereby improving the performance of the model in data analysis in the network. If the model information includes model performance information, the first network element can select a model that meets the conditions according to the performance requirements of the requested model, thereby improving the performance of the model in data analysis in the network.

[0042] It should be explained that the above actual use process may include the process of actual use and data inference (or data analysis) in the network.

[0043] Step 202: A first network element receives a first model request message from a second network element.

[0044] Optionally, in the embodiments of the present application, the second network element may include an AnLF or an MTLF.

[0045] In an embodiment of the present application, the first network element is a MTLF and the second network element is an AnLF, or the first network element is an ADRF and the second network element is an AnLF or an MTLF, or the first network element is an MTLF 1 and the second network element is an MTLF 2. MTLF 1 and MTLF 2 may be different MTLFs.

[0046] It should be noted that in the embodiment of the present application, the first network element is illustratively described as an MTLF and the second network element is illustratively described as an AnLF.

[0047] Optionally, in the embodiment of the present application, the above first model request message may include at least one of the following A. to E.:

[0048] A. A second analysis identifier that may be used to identify the data analysis task to which the requested model is applied. For example, the second analysis identifier may be UE mobility, in which case the data analysis task to which the requested model is applied is prediction of a user's movement trajectory.

[0049] B. Model filtering information that may be used to indicate conditions that the requested model must meet, such as area of ​​interest (AOI), single network slice selection assistance information (S-NSSAI), data network name (DNN), etc.

[0050] C. Model object information that may be used to indicate a training object of the requested model, which may be one UE, multiple UEs, or any UE.

[0051] D. Model reporting information may include at least one of the following: the reporting manner of the model required, application time, or reporting time. The reporting manner of the model may include periodic reporting or conditional reporting. Of course, in practical implementation, the model reporting information may further include any other possible information, and is not specifically limited in the embodiment of the present application.

[0052] E. Model performance requirements information that may be used to dictate the performance that a required model must meet, e.g., minimum accuracy, maximum MAE that a model must reach, etc.

[0053] Step 203: If the model information of M candidate models among the N candidate models matches the model information corresponding to the first model request message, the first network element sends the model information of P candidate models among the M candidate models to the second network element.

[0054] Here, N, M, and P are all positive integers, and N≧M≧P.

[0055] It can be understood that the above P candidate models may be M candidate models, or may be a part of the M candidate models.

[0056] In an embodiment of the present application, after the first network element receives the first model request message, the first network element can determine, from the N candidate models, a candidate model whose model information matches the model information corresponding to the first model request message, for example, the M candidate models. After the M candidate models are determined, the first network element can send model information of a part or all of the M candidate models (i.e., the P candidate models) to the second network element, so that the second network element can perform a data analysis task corresponding to the first analysis identifier based on the P candidate models, thereby obtaining a corresponding data analysis result.

[0057] Optionally, in the embodiment of the present application, the model information of the above M candidate models matching with the model information corresponding to the first model request message may include at least one of the following:

[0058] (1) The first analysis identifier and the second analysis identifier are the same. When the first analysis identifier and the second analysis identifier are the same, it can be understood that the data analysis task corresponding to the first analysis identifier and the data analysis task corresponding to the second analysis identifier are the same.

[0059] (2) The usage area range included in the model usage range information of the M candidate models matches the model filtering information included in the above-mentioned first model request message. For example, the S-NSSAI, the region of interest, and the DNN included in the model usage range information of the M candidate models are the same as the S-NSSAI, the region of interest, or the DNN in the above-mentioned model filtering information, or the S-NSSAI, the region of interest, or the DNN included in the model usage range information of the M candidate models includes the S-NSSAI, the region of interest, or the DNN in the above-mentioned model filtering information. The model usage range information of the M candidate models may include multiple DNNs, that is, the model usage range information may correspond to one DNN list, and in this case, the DNN in the model filtering information may be the DNN in the DNN list.

[0060] (3) The usage target range included in the model usage range information of the M candidate models matches the model target information included in the first model request message. For example, the usage target range included in the model usage range information of the M candidate models may include multiple targets (e.g., multiple UEs), that is, the usage target range may correspond to one target list, and in this case, the target indicated by the model target information may be a target in the target list.

[0061] (4) The usage time range included in the model usage range information of the M candidate models matches the model report information included in the first model request message, for example, the usage time range included in the model usage range information of the M candidate models is the same as the application time of the model corresponding to the model report information, or the application time of the model corresponding to the model report information is included in the usage time range.

[0062] (5) The model performance information of the M candidate models satisfies the performance indicated by the model performance requirement information included in the first model request message, for example, the performance of the models corresponding to the model performance information of the M candidate models is higher than the performance indicated by the model performance requirement information.

[0063] Optionally, in an embodiment of the present application, when the first network element is an MTLF and the second network element is an AnLF, the AnLF and the MTLF request and obtain model information by Nnwdaf_MLModelProvision_Subscribe and Nnwdaf_MLModelProvision_Notify, or the AnLF and the MTLF request and obtain model information by Nnwdaf_MLModelInfo_Request message and Nnwdaf_MLModelInfo_Response message.

[0064] Optionally, when the first network element is an ADRF and the second network element is an AnLF, the AnLF and the ADRF request and obtain the model information by Nadrf_DataManagement_RetrievalRequest message and Nadrf_DataManagement_RetrievalResponse message, or the AnLF and the ADRF request and obtain the model information by Nadrf_DataManagement_RetrievalSubscribe and Nadrf_DataManagement_RetrievalNotify.

[0065] In the embodiment of the present application, the first network element can record model information of the candidate model applied to the data analysis task, so that after the first network element receives the first model request message, it can determine whether the recorded model information of the candidate model matches the model information corresponding to the first model request message sent by the second network element, thereby determining an optimal model that meets the first model request message or a model that meets the performance requirements. In this way, the performance of the model in data analysis can be ensured.

[0066] Optionally, before the above step 201, the model selection method provided by the embodiment of the present application may further include the following step 204.

[0067] Step 204: The first network element determines N candidate models that meet a predetermined condition from the K models according to the model performance information of the K models.

[0068] The K models may be applied to the data analysis task corresponding to the first analysis identifier, where K is a positive integer. The performance indicated by the model performance information is the highest among the K models. the performance indicated by the model performance information is higher than the first default performance.

[0069] It should be noted that the performance indicated by the model performance information can be expressed in any possible form, such as a numerical value, a level, a percentage, etc. For example, when the performance indicated by the model performance indication information includes a model accuracy, the model accuracy can be expressed in an exact numerical value. It can be understood that the above first predetermined performance can also be a numerical value, a level, or a percentage.

[0070] Optionally, in embodiments of the present application, the K models may include a plurality of candidate models to be applied to the data analysis task corresponding to the first analysis identifier, or the K models may include a plurality of models obtained by performing training multiple times on the data analysis task corresponding to the first analysis identifier.

[0071] Optionally, in an embodiment of the present application, if the above K models include multiple models obtained by training multiple times using a data analysis task corresponding to the first analysis identifier, the model selection method provided by the embodiment of the present application may further include steps 205 and 206 below.

[0072] Step 205: If the performance of the first model sent by the first network element to the fourth network element is lower than the second default performance, the first network element obtains the second model by retraining the first model or reselecting a model according to the usage range information of the first model.

[0073] The above second model is a model among K models. The above fourth network element may be an AnLF, for example, the fourth network element may be AnLF 1.

[0074] Optionally, in the embodiment of the present application, the above fourth network element may be the same as the second network element, that is, the model request message for obtaining the first model and the above first model request message may be sent by the same network element, for example, both may be sent by ANLF 1. For example, after ANLF 1 loses the optimal model due to an abnormality, it can re-obtain the optimal model by sending a model request message to the MTLF or ADRF.

[0075] Optionally, in the embodiment of the present application, the above first model usage range information may be obtained from a fourth network element.

[0076] In the embodiment of the present application, the above-mentioned first network element retrains the first model according to the use range information of the first model, refers to retraining the first model for the use range of the first model. Specifically, taking the first network element as an example of MTLF and the fourth network element as AnLF 1, the MTLF collects label data corresponding to the use range of the first model as a training data set, and retrains the model based on the training data set. The training data in the training data set may include label data obtained from a data provider and label data obtained from AnLF 1 by the MTLF. It can be understood that the training data used in the MTLF includes label data.

[0077] Based on the above retraining of the first model, the first network element can update and repeat the first model according to the performance performance of the first model in the current network, record the performance performance of the model each time, and search and record the optimal model or the most suitable model corresponding to the usage scope information of the first model. Then, when other network elements (such as the above fourth network element or the fifth network element) request to obtain a model, the first network element can provide the optimal model or the most suitable model according to the usage scope of the requested model.

[0078] The above-mentioned model reselection refers to the following: after a first network element transmits model information of a first model to a fourth network element, when there are multiple selectable candidate models (including the first model already transmitted) in response to a model request message (e.g., a second model request message in an embodiment of the present application) of the fourth network element, if the performance of the first model is lower than the second default performance, the first network element can reselect another candidate model (e.g., the second model) from the multiple candidate models and transmit it to the fourth network element.

[0079] Based on the above model reselection, the first network element can obtain and record the performance performances in the actual use process of the above multiple models and their corresponding use scopes. Then, when another network element (such as the fourth network element or the fifth network element) requests to obtain a model, the first network element can provide the optimal model or the most appropriate model according to the use scope of the requested model.

[0080] Step 206: The first network element sends model information of the second model to the fourth network element or the fifth network element.

[0081] Optionally, in the embodiment of the present application, the above-mentioned fifth network element may be an AnLF. It can be understood that the fifth network element and the above-mentioned fourth network element are different AnLFs, for example, the fourth network element is AnLF 1 and the fifth network element is AnLF 2.

[0082] Optionally, in the embodiment of the present application, the above-mentioned first network element sending model information of the second model to the fourth network element may include two possible embodiments. In one form, after the first network element obtains the second model, the first network element actively pushes the model information of the second model to the fourth network element. In another form, after the first network element obtains the second model, the first network element can send the model information of the second model to the fourth network element when receiving a model request message (e.g., a third model request message) retransmitted by the fourth network element. Optionally, the usage scope of the requested model corresponding to the third model request message is the same as the usage scope of the second model.

[0083] In the embodiment of the present application, when the second model is a model obtained by retraining the first model, and the second model is an updated replacement of the first model, the model identifier of the second model is the same as the model identifier of the first model (e.g., model ID1). When the second model is a model obtained by reselecting a model, or a new model obtained by retraining the first model (not replacing the first model), the model identifier of the second model corresponds to a new model identifier (e.g., model ID2), which may be different from the model identifier of the first model.

[0084] Optionally, the model selection method provided by the embodiment of the present application may further include the following step 207:

[0085] Step 207: The first network element sends the usage range information of the second model to the fourth network element or the fifth network element.

[0086] The range of use indicated by the range of use information of the second model is the same as the range of use indicated by the range of use information of the first model, or the range of use indicated by the range of use information of the second model is determined by the range of use indicated by the range of use information of the first model.

[0087] It should be explained that the above-mentioned use range information of the second model is determined by the use range information of the first model, which can be understood as the use range indicated by the use range information of the second model includes the use range indicated by the use range information of the first model, or the use range indicated by the use range information of the second model intersects with the use range indicated by the use range information of the first model. The use range can include at least one of a use time range, a use object range, and a use area range.

[0088] In the embodiment of the present application, when the first network element sends the above-mentioned second model to the fourth network element, the first network element can send the usage range information of the second model to the fourth network element. When the first network element sends the second model to the fifth network element, the first network element can send the usage range information of the second model to the fifth network element.

[0089] In the embodiment of the present application, after the first network element obtains the model performance information of the first model, the first model can determine whether the performance of the first model is lower than the above-mentioned second default performance according to the model performance information of the first model, and thus can determine whether to retrain the first model. Based on this, the model selection method provided by the embodiment of the present application may further include any one of the following steps 208, 209, or 210.

[0090] Step 208: The first network element calculates model performance information of the first model.

[0091] The model performance information of the first model is used to indicate the performance of the first model.

[0092] For example, taking the first network element as an MTLF, the MTLF obtains a corresponding validation dataset (including label data, ground truth, etc.) according to the usage scope information of the first model, and calculates the output result performance of the first model, for example, the output result accuracy of the first model, based on the validation dataset and the data analysis output result of the first model itself, thereby obtaining model performance information of the first model.

[0093] Optionally, in the embodiment of the present application, based on the above step 208, the first network element can record the usage scope information of the first model, which indicates the scope of the data task analysis or model performance measurement by the first model, including the usage area scope of the first model, the usage time scope of the first model, the usage target scope of the first model, etc.

[0094] Step 209: The first network element receives model performance information of the first model from the fourth network element.

[0095] For example, taking the fourth network element as an AnLF, the AnLF obtains a corresponding validation dataset (including label data, ground truth, etc.) according to the usage scope information of the first model, and calculates the output result performance of the first model, for example, the output result accuracy of the first model, based on the validation dataset and the data analysis output result set of the first model, to obtain model performance information of the first model, and send the model performance information of the first model to the first network element, for example, the MTLF.

[0096] Optionally, in the embodiment of the present application, based on the above step 209, the fourth network element can send the usage range information of the first model to the first network element, so that the first network element can receive the usage range information of the first model from the fourth network element. For details of the usage range information of the first model, refer to the relevant description in the above embodiment, and the description will be omitted here to avoid duplication.

[0097] Optionally, in the present embodiment, the fourth network element may include a plurality of network elements, based on which the plurality of network elements can jointly use the same first model and use the first model in the same or intersecting usage scope, and the plurality of network elements can each report model performance information of the first model, for example, the output result accuracy (i.e., model accuracy) of the first model.

[0098] Step 210: A first network element receives model performance information for a first model from a data analysis consumer.

[0099] Optionally, in the embodiment of the present application, the above data analysis consumer may be any other possible data analysis consumer, such as PCF, AMF, etc. The specifics can be determined according to actual usage needs, and are not limited in the embodiment of the present application.

[0100] In an embodiment of the present application, the data analysis consumer can obtain a corresponding validation data set (including label data, ground truth, etc.) according to the data analysis result of the first model obtained from the fourth network element (e.g., AnLF) and the extent to which the data analysis result of the model is used by the data analysis consumer, and obtain model performance information of the first model by calculating the output result performance of the first model, for example, the output result accuracy of the first model, based on the validation data set and the data analysis result of the first model, and send the model performance information of the first model to the first network element, for example, MTLF. The data analysis consumer can directly send the model performance information of the first model to the first network element, or can send it to the first network element via the fourth network element.

[0101] Optionally, in the embodiment of the present application, the above data analysis consumers may be multiple, i.e., multiple data analysis consumers may use the above data analysis results, and these multiple data analysis consumers may jointly calculate and report model performance information of the above first model, such as the output result accuracy of the first model (i.e., model accuracy).

[0102] Optionally, in the embodiment of the present application, based on the above step 210, the data analysis consumer can send the usage range information of the first model to the first network element, so that the first network element can receive the usage range information of the first model from the data analysis consumer. The data analysis consumer may send the usage range information of the first model directly to the first network element, or may send it to the first network element via a fourth network element. For details of the usage range information of the first model, please refer to the relevant description in the above embodiment, and the description will be omitted here to avoid duplication.

[0103] It should be noted that in an embodiment of the present application, when the above K models include multiple candidate models to be applied to the data analysis task corresponding to the above first analysis identifier, the first network element can obtain model performance information of the multiple candidate models through the above steps 208, 209 or 210.

[0104] Optionally, in an embodiment of the present application, before the above step 205, the model selection method provided by an embodiment of the present application may further include the following steps 211 and 212.

[0105] Step 211: The first network element receives a second model request message from the fourth network element.

[0106] Step 212: The first network element sends model information of the first model to the fourth network element through a second model request message.

[0107] The above first model may include one model or multiple models. For details of the second model request message, refer to the detailed description of the first model request message in the above embodiment, and the description is omitted here.

[0108] Optionally, in an embodiment of the present application, if the above first model includes multiple models, before the above step 212, the model selection method provided by the embodiment of the present application may further include the following step 213 or step 214.

[0109] Step 213: The first network element determines whether the fourth network element supports the acquisition of multiple models.

[0110] Step 214: The first network element determines the model usage capability of the fourth network element.

[0111] In an embodiment of the present application, if there are multiple models matching the model information corresponding to the second model request message, the first network element can determine whether the fourth network element supports the acquisition of multiple models or whether the fourth network element supports the use of multiple models by determining the model use capability of the fourth network element, and if the fourth network element can support the use of multiple models, the first network element sends all model information of the multiple models to the fourth network element; otherwise, the first network element sends only one of the multiple models to the fourth network element, that is, the first network element does not send model information of the multiple models to the fourth network element simultaneously.

[0112] Optionally, in an embodiment of the present application, when the above first network element is an ADRF, the model selection method provided by the embodiment of the present application may further include the following step 215:

[0113] Step 215: The first network element obtains model information of the N candidate models from the third network element.

[0114] In an embodiment of the present application, the third network element may be an MTLF. Thus, before the ADRF records the model information of the N candidate models, the ADRF may first obtain the model information of the N candidate models from the MTLF.

[0115] It can be understood that when the first network element is a MTLF, the MTLF may obtain the model information of the above N candidate models by itself, and then record the model information of the N candidate models.

[0116] Optionally, in an embodiment of the present application, before the above step 201 (the first network element records model information of the N candidate models), the model selection method provided by the embodiment of the present application may further include the following step 216.

[0117] Step 216: The first network element obtains usage range information and / or model performance information of each candidate model among the N candidate models.

[0118] In an embodiment of the present application, before the first network element records the model information of the above-mentioned N candidate models, the first network element can first obtain the usage range information and / or model performance information of each candidate model among the N candidate models, and then record the model information of the N candidate models in correspondence with each other.

[0119] In this embodiment, it can be understood that the first network element does not perform screening on the N candidate models, but only records information corresponding to the N candidate models (including usage range information, model performance information, etc.), and then, after receiving a model request message sent by another network element, can select an appropriate model according to the information and send it to the network element.

[0120] Optionally, in the embodiment of the present application, the above step 216 may be specifically realized by the following step 216a, step 216b or step 216c.

[0121] Step 216a: The first network element determines usage range information of each candidate model among the N candidate models and / or calculates model performance information of each candidate model among the N candidate models.

[0122] Step 216b: The first network element receives usage range information and / or model performance information of each candidate model among the N candidate models from the seventh network element.

[0123] The seventh network element may include one network element or multiple network elements. The seventh network element may be an AnLF.

[0124] Optionally, the seventh network element may be the same as or different from the second network element in the above embodiment.Specifically, it can be determined according to actual usage needs, and is not limited in the embodiment of the present application.

[0125] Step 216c: The first network element receives from the data analysis consumer usage scope information and / or model performance information for each candidate model among the N candidate models.

[0126] It should be noted that for details of the first network element acquiring the usage range information and / or model performance information of each candidate model among the N candidate models, reference can be made to the detailed description of the first network element acquiring the usage range information and / or model performance information of the first model in the above embodiment, and the description will be omitted here to avoid duplication.

[0127] Optionally, in an embodiment of the present application, if the above first network element is an MTLF, after the above step 201 (the first network element records model information of the N candidate models), the model selection method provided by the embodiment of the present application may further include the following step 217.

[0128] Step 217: The first network element stores the model information of the N candidate models in the sixth network element.

[0129] The sixth network element may include an ADRF or a unified data repository (UDR).

[0130] Optionally, in the embodiment of the present application, based on the above step 217, the above step 203 may be specifically realized by the following step 203a.

[0131] Step 203a: A first network element sends first information to a second network element, where the first information is used to indicate that model information of the P candidate models is stored in a sixth network element.

[0132] The above-mentioned first information may include at least one of the following: identifier information, fully qualified domain name (FQDN), and address information of the sixth network element. Of course, the first information may further include other information corresponding to the sixth network element, which can be specifically determined according to actual usage needs, and is not limited in the embodiment of the present application.

[0133] It can be understood that at least one of the identifier information, FQDN, or address information of the sixth network element can be download address information of the P candidate models, thereby allowing the second network element to download model information of the P candidate models through at least one of the identifier information, FQDN, or address information of the sixth network element.

[0134] In an embodiment of the present application, after the MTLF stores the model information of the N candidate models in the sixth network element, when the second network element sends the first model request message to the first network element, if the model information of M candidate models among the N candidate models matches the model information corresponding to the first model request message, the first network element can send the first information to the second network element, so that after the second network element receives the first information, the second network element can download the model information of the P candidate models from the sixth network element.

[0135] Optionally, in an embodiment of the present application, after the first network element stores the model information of the above N candidate models in the sixth network element, the eighth network element can obtain the model information of the models from the sixth network element by directly sending a model request message to the sixth network element. The eighth network element can be configured to obtain the model information of the models from the sixth network element, and the eighth network element can be an AnLF, for example, an AnLF 3.

[0136] Hereinafter, a model selection method provided by an embodiment of the present application will be described by way of example with reference to the drawings.

[0137] Example 1 As shown in FIG. 3, in step 1a, AnLF 1 may send a model request message (e.g., request 1) to the MTLF, which may include an analytic ID, model filter information 1, etc.

[0138] In step 1b, the MTLF may provide model information of an ML model (e.g., a model to be applied to AI task 1) to the AnLF 1, which may be an initial model with a model identifier model ID1.

[0139] In step 2, the MTLF / AnLF 1 / Data Analytics Consumer performs analytics performance evaluation on the ML model from step 1b. It records the model identifier of the model, the data analytics task to which the model is applied, and the scope of model use. Once the AnLF 1 / Data Analytics Consumer performs analytics performance evaluation on the model, it reports the model performance information and the scope of use information of the model to the MTLF.

[0140] In step 3, if the performance of the ML model is lower than the default performance, the MTLF reselects or retrains a new ML model according to the model usage scope, which may be a different model from the initial model or may be an updated model of the initial model.

[0141] In step 4a (optionally), AnLF 1 sends a model request message (eg, request 2) to the MTLF.

[0142] In step 4b, the MTLF sends model information of the new ML model to the AnLF 1. The model information may include a model identifier and a model usage scope. If the new ML model is a model different from the initial model, the model identifier may be model ID2, and if the new ML model is a model that updates the initial model, the model identifier may be model ID1.

[0143] In step 5, the MTLF / AnLF 1 / Data Analytics Consumer performs performance measurement / evaluation on the new ML model in step 4b. For details, please refer to the explanation in step 2 above.

[0144] In step 6, the MTLF determines a target model according to the model performance information corresponding to the multiple models obtained in steps 2 to 5 above. The target model may be the one with the highest / best performance among the multiple models, or the target model may be the one with performance higher / better than a default performance among the multiple models. There may be one target model or multiple target models.

[0145] In step 7, AnLF 2 sends a model request message (e.g., request 3) to the MTLF to request a model corresponding to the analytic ID, which may be the same as the analytic ID in step 1a. The model request message includes model filter information (e.g., model filter information 2), etc.

[0146] In step 8, the MTLF determines whether the model information corresponding to the model request message in step 7 matches the model information of the target model, and if so, executes step 9 below.

[0147] In step 9, the MTLF sends the model information of the target model to AnLF 2.

[0148] Example 2 Still referring to Fig. 3, this embodiment differs from embodiment 1 in that in step 1b, for the model request message sent by AnLF 1, if there are multiple candidate models in the MTLF that meet the requirements, the MTLF can simultaneously publish the multiple candidate models to AnLF 1. Optionally, before publishing the models, the MTLF must first determine whether the AnLF supports obtaining or has the ability to use multiple models, and if not, the MTLF will not send multiple candidate models to the AnLF simultaneously.

[0149] In step 2, performance measurements can be performed on the multiple candidate models in a manner similar to step 2 in the first embodiment.

[0150] Example 3 As shown in FIG. 4, this embodiment differs from embodiments 1 and 2 in that in step 6b, the MTLF stores the model information of the acquired target model in a database or an integrated data platform (e.g., ADRF, UDR, etc.).

[0151] In step 7, AnLF 2 sends a model acquisition request message to a database or an integrated data platform.

[0152] In step 8, the database or integrated data platform determines whether the model information corresponding to the model request message in step 7 matches the model information of the target model, and if so, performs step 9 below.

[0153] The matching method in step 8 above may be similar to the matching method in step 8 in the first embodiment.

[0154] In step 9, the database or integrated data platform sends the model information of the target model to AnLF 2.

[0155] It should be noted that for steps not described in Example 2 and Example 3, reference can be made to the relevant descriptions in Example 1, and the descriptions will be omitted here to avoid duplication.

[0156] Example 4 Steps 1 to 6 are the same as steps 1 to 6 in the first embodiment above.

[0157] In step 6a, the MTLF stores model information of the target model (ie, the candidate model in the embodiment of the present application) in the ADRF, and records the corresponding ADRF information (eg, the identifier information of the ADRF, the address information of the ADRF, etc.).

[0158] Steps 7 and 8 are the same as steps 7 and 8 in the first embodiment above.

[0159] In step 9, the MTLF sends the ADRF information stored with the matched target model to the AnLF 2, so that the AnLF 2 can obtain the model information of the target model from the ADRF.

[0160] The model selection method provided by the embodiment of the present application may be executed by a model selection device. In the embodiment of the present application, the model selection device provided by the embodiment of the present application will be described by taking the model selection method executed by the model selection device as an example.

[0161] As shown in FIG. 5, an embodiment of the present application provides a model selection apparatus 300, which includes a recording module 301, a first receiving module 302, and a first sending module 303. The recording module 301 is used to record model information of N candidate models, where the N candidate models are applied to a data analysis task corresponding to a first analysis identifier. The first receiving module 302 is used to receive a first model request message from a second network element. The first sending module 303 is used to send model information of P candidate models of the M candidate models to the second network element when the model information of M candidate models of the N candidate models matches the model information corresponding to the first model request message, where N, M, and P are all positive integers, and N≧M≧P.

[0162] Optionally, in an embodiment of the present application, the model selection device further includes a determination module used to determine N candidate models that satisfy a predefined condition from the K models according to model performance information of the K models, where the K models are applied to a data analysis task corresponding to the first analysis identifier, where K is a positive integer. The predefined condition includes any one of: the performance indicated by the model performance information is the highest among the K models; and the performance indicated by the model performance information is higher than a first predefined performance.

[0163] Optionally, in an embodiment of the present application, the model selection device further includes a first acquiring module used for acquiring model information of the N candidate models from a third network element.

[0164] Optionally, in an embodiment of the present application, the model information of the candidate model comprises: A model identifier for the candidate model; Model file information of the candidate models, download address information of the candidate model, which is used to indicate the storage address of the model file of the candidate model; an analysis identifier for the candidate model, which is used to identify the data analysis task to which the candidate model is to be applied; Use range information of the candidate model, Model performance information of candidate models Includes at least one of the following:

[0165] Optionally, in an embodiment of the present application, the usage range information may include: Area of ​​use, Usage time range, Scope of use Indicate at least one of the following:

[0166] Optionally, in an embodiment of the present application, the performance indicated by the model performance information includes at least one of: model accuracy, model mean absolute error.

[0167] Optionally, in an embodiment of the present application, the model selection apparatus further includes a second acquisition module used to acquire usage range information and / or model performance information of each candidate model among the N candidate models.

[0168] Optionally, in an embodiment of the present application, the second acquisition module is specifically used to determine usage range information of each candidate model and / or calculate model performance information of each candidate model, or the second acquisition module is specifically used to receive usage range information and / or model performance information of each candidate model from the seventh network element, or the second acquisition module is specifically used to receive usage range information and / or model performance information of each candidate model from a data analysis consumer.

[0169] Optionally, in an embodiment of the present application, the first model request message comprises: a second analysis identifier used to identify the data analysis task to which the requested model is to be applied; model filtering information, which is used to indicate conditions that a requested model must satisfy; model target information used to indicate training targets for the required model; Model reporting information, including at least one of the following: the required model reporting method, application time, or reporting time; Model performance requirement information used to dictate the performance that a required model must meet Includes at least one of the following:

[0170] Optionally, in the embodiment of the present application, the model information of the M candidate models matches the model information corresponding to the first model request message by: the first analysis identifier and the second analysis identifier are the same; The usage area ranges included in the model usage range information of the M candidate models match the model filtering information; The usage target ranges included in the model usage range information of the M candidate models match the model target information; The usage time ranges included in the model usage range information of the M candidate models match the model report information; The model performance information of the M candidate models satisfies the performance indicated by the model performance requirement information. Includes at least one of the following:

[0171] Optionally, in an embodiment of the present application, the K models include a plurality of candidate models to be applied to the data analysis task corresponding to the first analysis identifier, or the K models include a plurality of models obtained by performing training multiple times on the data analysis task corresponding to the first analysis identifier.

[0172] Optionally, in an embodiment of the present application, when the K models include multiple models obtained by performing training multiple times by a data analysis task corresponding to the first analysis identifier, the model selection device further includes: an execution module used to obtain a second model, which is a model among the K models, by retraining the first model or reselecting a model based on usage range information of the first model when the performance of the first model transmitted by the transmission module to the fourth network element is lower than a second default performance; and a second transmission module used to transmit model information of the second model to the fourth network element or the fifth network element.

[0173] Optionally, in an embodiment of the present application, the model selection device further includes a third sending module used to send usage range information of the second model to the fourth network element or the fifth network element, and the usage range indicated by the usage range information of the second model is the same as the usage range indicated by the usage range information of the first model, or the usage range indicated by the usage range information of the second model is determined by the usage range indicated by the usage range information of the first model.

[0174] Optionally, in an embodiment of the present application, the model selection device further includes a calculation module used to calculate model performance information of the first model, or a second receiving module also used to receive model performance information of the first model from the fourth network element, or a third receiving module also used to receive model performance information of the first model from the data analysis consumer, wherein the model performance information of the first model is used to indicate the performance of the first model.

[0175] Optionally, in an embodiment of the present application, the model selection device further includes a storage module used for storing model information of the N candidate models in the sixth network element.

[0176] Optionally, in an embodiment of the present application, the first sending module is specifically used for sending first information to the second network element, and the first information is used for indicating that the model information of the P candidate models is stored in the sixth network element, and the first information includes at least one of identifier information, FQDN, and address information of the sixth network element.

[0177] Optionally, in an embodiment of the present application, the sixth network element includes an ADRF or a UDR.

[0178] Optionally, in an embodiment of the present application, the second network element includes an AnLF or an MTLF.

[0179] For the model selection device provided by the embodiment of the present application, the model selection device can record model information of the candidate model applied to the data analysis task, so that after the model selection device receives the first model request message, it can determine whether the recorded model information of the candidate model matches the model information corresponding to the first model request message, thereby determining the optimal model that meets the first model request message or the model that meets the performance requirements. In this way, the performance of the model in data analysis can be ensured.

[0180] The model selection device in the embodiment of the present application may be a network side device, for example, a network side device with an operating system, or may be a component in the network side device, for example, an integrated circuit or chip. The network side device may be an NWDAF, a MTLF, an AnLF, an ARDF, etc., and is not specifically limited in the embodiment of the present application.

[0181] The model selection apparatus provided by the embodiments of the present application can realize each process realized by the embodiments of the model selection method described above and achieve the same technical effects, and in order to avoid duplication, the description will be omitted here.

[0182] Optionally, as shown in FIG. 6 , the embodiment of the present application further provides a network side device 400, including a processor 401 and a memory 402, in which a program or command executable by the processor 401 is stored. For example, when the network side device 400 is a network side device, when the program or command is executed by the processor 401, the steps of the above-mentioned model selection method embodiment can be realized and the same technical effects can be achieved. In order to avoid duplication, the description will be omitted here.

[0183] Specifically, an embodiment of the present application further provides a network side device. As shown in Fig. 7, the network side device 500 includes a processor 501, a network interface 502 and a memory 503. The network interface 502 is, for example, a common public radio interface (CPRI).

[0184] Specifically, the network side device 500 according to the embodiment of the present invention further includes commands or programs stored in the memory 503 and executable by the processor 501, and the processor 501 calls the commands or programs in the memory 503 to execute the methods performed by each module shown in FIG. 6, and achieves the same technical effects, and in order to avoid duplication, the description will be omitted here.

[0185] The embodiments of the present application further provide a readable storage medium having a program or command stored thereon, which, when executed by a processor, realizes each process of the embodiments of the above model selection method and achieves the same technical effects, and in order to avoid duplication, the description will be omitted here.

[0186] The processor is the processor in the terminal in the above embodiment. The readable storage medium includes a computer readable storage medium such as a computer read only memory ROM, a random access memory RAM, a magnetic disk, or an optical disk.

[0187] The embodiments of the present application also provide a chip, which includes a coupled processor and a communication interface, and the processor is for realizing each process of the embodiments of the above model selection method by executing a program or command, and the same technical effect can be achieved, and in order to avoid duplication, the description will be omitted here.

[0188] It should be understood that the chip described in the embodiments of the present application may also be referred to as a system on a chip, a system chip, a chip system, or an SoC.

[0189] The embodiments of the present application also provide a computer program / program product, which is stored in a storage medium and executed by at least one processor to realize each process of the embodiments of the above model selection method and achieve the same technical effects, and thus will not be described again here to avoid duplication.

[0190] It should be noted that, in this specification, the terms "comprise", "comprises" or any other variants thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus including a set of elements includes not only those elements, but also other elements not expressly stated or inherent to such process, method, article, or apparatus. Unless otherwise specified, an element limited by a statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus including the element. It should also be noted that the scope of the method and apparatus in the embodiments of this application is not limited to performing the functions in the order shown or discussed herein, and may further include performing the functions substantially simultaneously or in the opposite order, depending on the functions involved. For example, the methods described above may be performed in a different order than described, and further, steps may be added, omitted, or combined. Also, features described with reference to some examples may be combined with other examples.

[0191] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be realized in the form of a combination of software and a required general-purpose hardware platform, and of course, they can also be realized by hardware, but in many cases the former is a more preferred embodiment. Based on this view, the technical solution of the present application can be substantially or the part that contributes to the prior art can be embodied as a computer software product, which is stored in a storage medium (e.g., ROM / RAM, magnetic disk, optical disk) and includes some commands for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network side device, etc.) to execute the methods according to the embodiments of the present application.

[0192] Although the examples of the present application have been described above with reference to the drawings, the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative and not limiting. Based on the suggestions of the present application, many forms that a person skilled in the art can make without departing from the spirit of the present application and the scope of protection of the claims are all within the scope of protection of the present application.

[0193] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to a Chinese patent application bearing application number 202210483837.0, filed in China on May 5, 2022, the entire contents of which are incorporated herein by reference.

Claims

1. a first network element recording model information of N candidate models, the N candidate models being applied to a data analysis task corresponding to a first analysis identifier; receiving a first model request message from a second network element by the first network element; if model information of M candidate models among the N candidate models matches model information corresponding to the first model request message, the first network element sends model information of P candidate models among the M candidate models to the second network element; Here, N, M, and P are all positive integers, and N≧M≧P. Model selection method.

2. Prior to the step of the first network element recording model information of the N candidate models, The method further includes the step of: determining, by the first network element, the N candidate models that satisfy a predetermined condition from the K models according to model performance information of the K models, where the K models are applied to a data analysis task corresponding to the first analysis identifier, where K is a positive integer; The predetermined condition is: The performance indicated by the model performance information is the highest among the K models; and The performance indicated by the model performance information is higher than a first predetermined performance; and The method of claim 1 , comprising any one of:

3. The method further includes the step of: the first network element obtaining model information of the N candidate models from a third network element. The method according to claim 1 or 2.

4. Model information for the candidate models is as follows: A model identifier for the candidate model; Model file information of the candidate models, download address information of the candidate model, which is used to indicate the storage address of the model file of the candidate model; an analysis identifier for the candidate model, which is used to identify the data analysis task to which the candidate model is to be applied; Use range information of the candidate model, Model performance information of candidate models The method according to any one of claims 1 to 3, comprising at least one of:

5. The use range information is Area of ​​use, Usage time range, Scope of use The method of claim 4 , further comprising indicating at least one of:

6. The method of claim 2 or 4, wherein the performance indicated by the model performance information includes at least one of the following: model accuracy, model mean absolute error.

7. Prior to the step of the first network element recording model information of the N candidate models, The first network element further includes obtaining usage range information and / or model performance information of each candidate model among the N candidate models. The method according to any one of claims 1 to 6.

8. The step of the first network element obtaining usage range information and / or model performance information of each candidate model among the N candidate models, the first network element determining usage range information for each of the candidate models and / or calculating model performance information for each of the candidate models; receiving, by the first network element, from a seventh network element, usage range information and / or model performance information for each of the candidate models; and receiving, by the first network element, from a data analysis consumer, usage range information and / or model performance information for each of the candidate models. The method of claim 7.

9. The first model request message includes: a second analysis identifier used to identify the data analysis task to which the requested model is to be applied; model filtering information, which is used to indicate conditions that a requested model must satisfy; model target information used to indicate training targets for the required model; model reporting information, including at least one of the following: a required model reporting format, application time, or reporting time; Model performance requirement information used to dictate the performance that a required model must meet The method according to any one of claims 1 to 8, comprising at least one of the following:

10. The model information of the M candidate models matches the model information corresponding to the first model request message, the first analysis identifier and the second analysis identifier are the same; The usage area range included in the model usage range information of the M candidate models matches the model filtering information; the use target ranges included in the model use range information of the M candidate models match the model target information; The usage time range included in the model usage range information of the M candidate models matches the model report information; The model performance information of the M candidate models satisfies the performance indicated by the model performance requirement information. The method of claim 9 , comprising at least one of:

11. 11. The method of claim 2, wherein the K models include a plurality of candidate models to be applied to the data analysis task corresponding to the first analysis identifier, or the K models include a plurality of models obtained by performing training multiple times on the data analysis task corresponding to the first analysis identifier.

12. When the K models include a plurality of models obtained by performing training multiple times by a data analysis task corresponding to the first analysis identifier, When the performance of the first model sent by the first network element to the fourth network element is lower than a second default performance, the first network element retrains the first model or reselects a model according to the usage range information of the first model to obtain a second model, which is a model among the K models; the first network element sending model information of the second model to the fourth network element or the fifth network element; The method of claim 11 further comprising:

13. The method further includes the step of transmitting usage range information of the second model from the first network element to the fourth network element or the fifth network element; The usage range indicated by the usage range information of the second model is the same as the usage range indicated by the usage range information of the first model, or the usage range indicated by the usage range information of the second model is determined by the usage range indicated by the usage range information of the first model. The method of claim 12.

14. the first network element calculating model performance information for the first model; receiving model performance information of the first model from the fourth network element by the first network element; the first network element receiving model performance information of the first model from a data analytics consumer; the model performance information of the first model is used to indicate performance of the first model. The method of claim 12.

15. After the step of the first network element recording model information of the N candidate models, The method further includes the step of the first network element storing model information of the N candidate models in a sixth network element. The method according to any one of claims 1 to 14.

16. The step of transmitting model information of P candidate models among the M candidate models from the first network element to the second network element comprises: transmitting first information from the first network element to the second network element, the first information being used to indicate that model information of the P candidate models is stored in the sixth network element; The method of claim 15 , wherein the first information includes at least one of an identifier information, a fully qualified domain name FQDN, and address information of the sixth network element.

17. 17. The method according to claim 15 or 16, wherein the sixth network element comprises an analytical data storage function ADRF or a unified data repository UDR.

18. The method according to any one of claims 1 to 16, wherein the first network element comprises a model training logic function MTLF or an analytical data storage function ADRF.

19. The method of claim 1 , wherein the second network element comprises an analysis logic function AnLF or MTLF.

20. a recording module used for recording model information of N candidate models, the N candidate models being applied to a data analysis task corresponding to a first analysis identifier; a first receiving module used to receive a first model request message from a second network element; a first sending module used to send model information of P candidate models among the M candidate models to the second network element when model information of M candidate models among the N candidate models matches model information corresponding to the first model request message; Here, N, M, and P are all positive integers, and N≧M≧P. Model selection device.

21. A determination module is used to determine the N candidate models that satisfy a predetermined condition from the K models according to model performance information of the K models, the K models being applied to a data analysis task corresponding to the first analysis identifier, where K is a positive integer; The predetermined condition is: The performance indicated by the model performance information is the highest among the K models; and The performance indicated by the model performance information is higher than a first predetermined performance; and 21. The apparatus of claim 20, comprising any one of the following:

22. a first acquiring module used to acquire model information of the N candidate models from a third network element; 22. Apparatus according to claim 20 or 21.

23. Model information for the candidate models is as follows: A model identifier for the candidate model; Model file information of the candidate models, download address information of the candidate model, which is used to indicate the storage address of the model file of the candidate model; an analysis identifier for the candidate model, which is used to identify the data analysis task to which the candidate model is to be applied; Use range information of the candidate model, Model performance information of candidate models The apparatus according to any one of claims 20 to 22, comprising at least one of:

24. The use range information is Area of ​​use, Usage time range, Scope of use 24. The apparatus of claim 23, further comprising:

25. 24. The apparatus of claim 21 or 23, wherein the performance indicated by the model performance information includes at least one of: model accuracy, model mean absolute error.

26. a second acquisition module used to acquire usage range information and / or model performance information of each of the N candidate models; Apparatus according to any one of claims 20 to 25.

27. The second acquisition module is specifically used to determine the usage range information of each of the candidate models and / or calculate the model performance information of each of the candidate models; or The second acquisition module is specifically used for receiving the usage range information and / or model performance information of each of the candidate models from a seventh network element; or The apparatus of claim 26, wherein the second acquisition module is specifically used for receiving usage scope information and / or model performance information of each of the candidate models from a data analytics consumer.

28. The first model request message includes: a second analysis identifier used to identify the data analysis task to which the requested model is to be applied; model filtering information, which is used to indicate conditions that a requested model must satisfy; model target information used to indicate training targets for the required model; model reporting information, including at least one of the following: a required model reporting format, application time, or reporting time; Model performance requirement information used to dictate the performance that a required model must meet The apparatus according to any one of claims 20 to 27, comprising at least one of:

29. The model information of the M candidate models matches the model information corresponding to the first model request message, the first analysis identifier and the second analysis identifier are the same; The usage area range included in the model usage range information of the M candidate models matches the model filtering information; the use target ranges included in the model use range information of the M candidate models match the model target information; The usage time range included in the model usage range information of the M candidate models matches the model report information; The model performance information of the M candidate models satisfies the performance indicated by the model performance requirement information.

30. The apparatus of claim 28, comprising at least one of:

30. The apparatus of any one of claims 21 to 29, wherein the K models include a plurality of candidate models to be applied to a data analysis task corresponding to the first analysis identifier, or the K models include a plurality of models obtained by performing training multiple times on a data analysis task corresponding to the first analysis identifier.

31. When the K models include a plurality of models obtained by performing training multiple times by a data analysis task corresponding to the first analysis identifier, an execution module, which is used to obtain a second model, which is a model among the K models, by retraining the first model or reselecting a model according to the usage range information of the first model when the performance of the first model sent by the sending module to the fourth network element is lower than a second default performance; a second sending module used to send model information of the second model to the fourth network element or the fifth network element; 31. The apparatus of claim 30, further comprising:

32. a third sending module used to send usage range information of the second model to the fourth network element or the fifth network element; 32. The apparatus of claim 31 , wherein the range of use indicated by the usage range information of the second model is the same as the range of use indicated by the usage range information of the first model, or the range of use indicated by the usage range information of the second model is determined by the range of use indicated by the usage range information of the first model.

33. a calculation module used to calculate model performance information of the first model; or a second receiving module used to receive model performance information of the first model from the fourth network element; or a third receiving module used to receive model performance information of the first model from a data analysis consumer; the model performance information of the first model is used to indicate performance of the first model.

32. The apparatus of claim 31.

34. and a storage module used for storing model information of the N candidate models in a sixth network element. Apparatus according to any one of claims 20 to 33.

35. The first sending module is specifically used for sending first information to the second network element, and the first information is used for indicating that model information of the P candidate models is stored in the sixth network element; 35. The apparatus of claim 34, wherein the first information includes at least one of identifier information, a fully qualified domain name FQDN, and address information of the sixth network element.

36. 36. Apparatus according to claim 34 or 35, wherein the sixth network element comprises an analytical data storage function ADRF or a unified data repository UDR.

37. The apparatus of claim 20, wherein the second network element comprises an analysis logic function AnLF or a model training logic function MTLF.

38. A network side device including a processor and a memory, the memory storing a program or command executable by the processor, and when the program or command is executed by the processor, the steps of the model selection method described in any one of claims 1 to 19 are realized.

39. A readable storage medium having stored thereon a program or command which, when executed by a processor, causes the steps of the model selection method according to any one of claims 1 to 19 to be implemented.

40. A computer program product which, when executed by at least one processor, implements the model selection method according to any one of claims 1 to 19.

Images