VFL performance monitoring method and system, MTLF, electronic equipment and product

Abstract

The invention relates to a VFL performance monitoring method and system, an MTLF, electronic equipment and a product, and relates to the technical field of communication. The performance monitoring method of the vertical federated learning VFL is applied to a model training logic function MTLF and comprises the steps that data used for performance monitoring of a VFL model is acquired from one or more devices in a network, and the data used for performance monitoring of the VFL model corresponds to a reasoning result of the VFL model; determining a performance monitoring result of the VFL model according to the data for performance monitoring of the VFL model; and according to a performance monitoring result, determining whether to re-train the VFL model.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a VFL performance monitoring method, system, MTLF, electronic equipment and product. Background Art

[0002] With the widespread application of AI (Artificial Intelligence) technology in the field of communications, the 3GPP (3rd Generation Partnership Project) standard has introduced the network element NWDAF (Network Data Analytics Function) for AI / ML (Machine Learning) model reasoning and training in the core network since R16. The introduction of NWDAF marks an important step in network intelligence. It is responsible for data collection, model training, reasoning judgment and intelligent prediction, and outputs analysis results for network, network management and application execution policy decisions.

[0003] The distributed intelligent architecture based on federated learning is an important breakthrough in the combination of intelligence, federated learning and communication networks. Federated learning mainly includes horizontal federated learning (HFL) and vertical federated learning (VFL). Vertical federated learning (VFL) is a dataset where participants have different feature spaces but the same sample space. They jointly train their own models while protecting the data from leaving the local environment. 3GPP began to introduce related research on HFL in R18, while VFL is still under research and the related processes have not yet been standardized. Summary of the invention

[0004] According to some embodiments of the present disclosure, a performance monitoring method for vertical federated learning (VFL) is provided, which is applied to a model training logic function (MTLF), including: acquiring performance monitoring data for a VFL model from one or more devices in a network, wherein the performance monitoring data for the VFL model corresponds to an inference result of the VFL model; determining a performance monitoring result of the VFL model based on the performance monitoring data for the VFL model; and determining whether to retrain the VFL model based on the performance monitoring result.

[0005] In some embodiments, the data used for performance monitoring of the VFL model includes accuracy evaluation data of the VFL model, and obtaining the data used for performance monitoring of the VFL model from one or more devices in the network includes: sending a first subscription request to a data analysis logical network element AnLF, wherein the first subscription request includes an analysis ID and an identifier of the VFL model; and receiving accuracy evaluation data sent by AnLF based on the analysis ID and the identifier of the VFL model.

[0006] In some embodiments, data for performance monitoring of the VFL model includes feedback information, and obtaining data for performance monitoring of the VFL model from one or more devices in the network includes: sending a second subscription request to the consumer end through the data analysis logical network element AnLF, wherein the second subscription request includes the analysis ID and the identifier of the VFL model; receiving feedback information sent by the consumer end based on the analysis ID and the identifier of the VFL model through AnLF, wherein the feedback information includes feedback information from the consumer end on the accuracy of the VFL model.

[0007] In some embodiments, the data used for performance monitoring of the VFL model includes actual measurement data corresponding to the inference results of the VFL model, and obtaining the data used for performance monitoring of the VFL model from one or more devices in the network includes: obtaining the actual measurement data corresponding to the inference results of the VFL model from at least one device including user equipment, wireless access network elements, and core network elements.

[0008] In some embodiments, the performance monitoring results include at least one of the accuracy and stability of the VFL model. Based on the performance monitoring results, determining whether to retrain the VFL model includes: when at least one of the accuracy and stability of the VFL model does not meet a preset condition, determining to retrain the VFL model.

[0009] In some embodiments, the method further includes: sending a performance report to the data analysis logic network element AnLF, wherein the performance report includes a performance monitoring result, an analysis ID, and an identifier of the VFL model.

[0010] In some embodiments, the method further includes: in the case of determining to retrain the VFL model, sending a notification message to the data analysis logical network element AnLF, wherein the notification information includes information indicating to retrain the VFL model, the analysis ID, and an identifier of the VFL model.

[0011] In some embodiments, the method further includes: when MTLF is the server and it is determined that the VFL model is to be retrained, one or more clients participating in the retraining are determined according to the identifier of the VFL model; and retraining indication information is sent to the one or more clients participating in the retraining, wherein the retraining indication information includes an analysis ID, an identifier of the VFL model, filter information, and indication information of training samples used for retraining.

[0012] In some embodiments, the method also includes: receiving intermediate result information sent by one or more clients participating in the retraining, wherein the intermediate result information sent by each client participating in the retraining includes an analysis ID, an identifier of the VFL model, and an intermediate result of the retraining; determining the gradient parameters of each client participating in the retraining based on the intermediate result information; and sending gradient indication information to each client participating in the retraining, wherein the gradient indication information corresponding to each client participating in the retraining includes the analysis ID, the identifier of the VFL model, and the gradient parameters of each client participating in the retraining.

[0013] In some embodiments, the method further includes: sending retraining training information to the data analysis logical network element AnLF, wherein the retraining training information includes at least one of a retraining training progress and a training result.

[0014] In some embodiments, the method further includes: when the MTLF is a client and determines to retrain the VFL model, sending a retraining request to the server, wherein the retraining request includes an analysis ID, an identifier of the VFL model, and an indication of a request for retraining; receiving a retraining response sent by the server, wherein the retraining response includes the analysis ID, the identifier of the VFL model, filter information, and indication information of training samples used for retraining; and retraining the VFL model according to the retraining response.

[0015] In some embodiments, based on the retraining response, retraining the VFL model includes: inputting the training samples used for retraining into the VFL model to obtain the intermediate results of retraining; sending the intermediate result information to the server, wherein the intermediate result information sent by each client participating in the retraining includes the analysis ID, the identifier of the VFL model and the intermediate result of retraining; receiving the gradient indication information sent by the server, wherein the gradient indication information includes the analysis ID, the identifier of the VFL model, and the gradient parameters; and adjusting the parameters of the VFL model according to the gradient parameters.

[0016] In some embodiments, MTLF is the server, and the performance monitoring method also includes: receiving an analysis subscription request sent by a data analysis logical network element AnLF, wherein the analysis subscription request includes an analysis ID, filter information, information on an analysis report target, and an identifier of a VFL model; selecting one or more clients participating in reasoning from clients participating in training according to the VFL association ID and the filter information; and sending reasoning indication information to one or more clients participating in reasoning, wherein the reasoning indication information includes the analysis ID and the identifier of the VFL model.

[0017] In some embodiments, the method further includes: receiving an analysis response message sent by each of one or more clients participating in the reasoning, wherein the analysis response message includes the intermediate reasoning result of each client participating in the reasoning, the analysis ID and the identifier of the VFL model; and generating the reasoning result of the VFL model according to the intermediate reasoning result of each client participating in the reasoning.

[0018] In some embodiments, the method further includes: sending an analysis result message to AnLF, wherein the analysis result message includes an inference result, an analysis ID, and an identifier of the VFL model.

[0019] In some embodiments, the MTLF is a client, and the performance monitoring method further includes: receiving reasoning indication information sent by the server, wherein the reasoning indication information includes analysis ID, filter information, information of the analysis report target, and an identifier of the VFL model; performing reasoning using a local model corresponding to the VFL model according to the reasoning indication information to obtain an intermediate reasoning result; and sending an analysis response message to the server so that the server generates a reasoning result of the VFL model according to the analysis response message, wherein the analysis response message includes the intermediate reasoning result of each client participating in the reasoning, the analysis ID, and the identifier of the VFL model.

[0020] In some embodiments, the method further includes: sending a registration request to a network repository function NRF, wherein the registration request includes at least one of network function NF profile information, analysis ID, address information, service area, and capability type; receiving a registration response sent by the NRF, wherein the registration response includes information indicating whether the registration is successful.

[0021] In some embodiments, MTLF is a server, and the performance monitoring method further includes: receiving an analysis request sent by a data analysis logical network element AnLF, wherein the analysis request includes an analysis ID, information about an analysis report target, filter information, and an analysis target period; determining whether to support becoming a server according to the capability type of MTLF; in response to supporting becoming a server, sending a discovery request to a network repository function NRF, wherein the discovery request includes an analysis ID and filter information; receiving a discovery response sent by NRF based on the discovery request, wherein the discovery response includes identifications of one or more clients participating in the training; sending a training preparation message to one or more clients participating in the training, wherein the training preparation message includes an analysis ID, information about an analysis report target, filter information, an analysis target period, and a VFL association ID; receiving a training preparation response sent by one or more clients participating in the training, wherein the training preparation response includes indication information on whether to participate in the training.

[0022] In some embodiments, the method further includes: sending a training indication message to one or more clients participating in the training, wherein the training indication message includes an analysis ID, an identifier of a VFL model, filter information, and indication information of training samples; performing feature alignment of training samples with one or more clients participating in the training; receiving intermediate result information sent by each of the one or more clients participating in the training, wherein the intermediate result information sent by each client participating in the training includes an analysis ID, an identifier of a VFL model, and an intermediate result of the training; determining a gradient parameter of each client participating in the training based on the intermediate result information; and sending gradient indication information to each client participating in the training, wherein the gradient indication information corresponding to each client participating in the training includes an analysis ID, an identifier of a VFL model, and a gradient parameter of each client participating in the training.

[0023] In some embodiments, the MTLF is a client, and the performance monitoring method further includes: receiving a training preparation message sent by a server, wherein the training preparation message includes an analysis ID, information of an analysis report target, filter information, an analysis target cycle, and a VFL association ID; and sending a training preparation response to the server, wherein the training preparation response includes indication information on whether to participate in the training.

[0024] In some embodiments, the method further includes: receiving a training indication message sent by a server, wherein the training indication message includes an analysis ID, an identifier of a VFL model, filter information, and indication information of a training sample; obtaining a training sample according to the indication information of the training sample; aligning features of the training sample with one or more clients participating in the training; inputting the training sample into a local model corresponding to the VFL model to obtain an intermediate result; sending intermediate result information to the server, wherein the intermediate result information includes an analysis ID, an identifier of a VFL model, and an intermediate result of the training; and receiving gradient indication information sent by the server, wherein the gradient indication information includes an analysis ID, an identifier of a VFL model, and gradient parameters.

[0025] According to other embodiments of the present disclosure, there is provided an acquisition module for acquiring data for performance monitoring of a VFL model from one or more devices in a network, wherein the data for performance monitoring of the VFL model corresponds to an inference result of the VFL model; a first determination module for determining a performance monitoring result of the VFL model based on the data for performance monitoring of the VFL model; and a second determination module for determining whether to retrain the VFL model based on the performance monitoring result.

[0026] According to some further embodiments of the present disclosure, an electronic device is provided, including: a processor; and a memory coupled to the processor, for storing instructions, which, when executed by the processor, causes the processor to execute the VFL performance monitoring method of any embodiment of the present disclosure.

[0027] According to some further embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, the VFL performance monitoring method of any embodiment of the present disclosure is implemented.

[0028] According to still other embodiments of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor, cause the processor to perform the VFL performance monitoring method according to any embodiment of the present disclosure.

[0029] According to some further embodiments of the present disclosure, a VFL performance monitoring system is provided, comprising: the MTLF of any of the aforementioned embodiments; and one or more devices in a network, wherein the one or more devices are configured to send data for performance monitoring of a VFL model to the MTLF, wherein the data for performance monitoring of the VFL model corresponds to an inference result of the VFL model.

[0030] The present disclosure proposes a VFL performance monitoring method, which can use MTLF to implement performance monitoring of the VFL model. The MTLF interacts with one or more devices in the network to obtain data for performance monitoring of the VFL model, and the performance monitoring result of the VFL model is determined according to the data for performance monitoring of the VFL model, and then it is determined whether to retrain the VFL model according to the performance monitoring result. Through the VFL performance monitoring method disclosed in the present disclosure, it can be found in time whether the performance of the VFL meets the requirements during the application of the VFL model, and then it is determined whether to retrain the VFL model to ensure the accuracy of the VFL model and improve the accuracy of VFL reasoning.

[0031] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 A schematic flow chart showing a method for monitoring the performance of a VFL according to some embodiments of the present disclosure.

[0034] Figure 2 A schematic flow chart showing a method for monitoring the performance of a VFL according to other embodiments of the present disclosure.

[0035] Figure 3 A schematic diagram showing the structure of the MTLF of some embodiments of the present disclosure.

[0036] Figure 4 A schematic diagram showing the structure of a VFL performance monitoring system according to some embodiments of the present disclosure.

[0037] Figure 5 A schematic diagram showing the structure of an electronic device according to some embodiments of the present disclosure.

[0038] Figure 6 A schematic diagram showing the structure of an electronic device according to some other embodiments of the present disclosure. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0040] The present disclosure proposes a VFL performance monitoring method, which can realize a method for monitoring the performance of a VFL model during the application of the VFL model. Figure 1~2 , some embodiments of the VFL performance monitoring method disclosed in the present invention are described.

[0041] Figure 1 Flowcharts of some embodiments of the VFL performance monitoring method disclosed in the present invention. Figure 1 As shown, the method of this embodiment includes: steps S102 to S106. NWDAF (Network Data Analytics Function) can be divided into AnLF (Analytics logical function) and MTLF (Model Training logical function). The method of this embodiment can be executed by MTLF.

[0042] In step S102, data for performance monitoring of the VFL model is obtained from one or more devices in the network.

[0043] The data used for performance monitoring of the VFL model corresponds to the inference result of the VFL model. The data used for performance monitoring of the VFL model may include at least one of accuracy evaluation information, feedback information and actual measurement data for evaluating whether the inference result is accurate. The one or more devices include at least one of AnLF, consumer end, user equipment (UE), radio access network (RAN) network element and core network (CN) network element. AnLF can evaluate the accuracy of the VLF model and determine the accuracy evaluation information. The consumer end, i.e., the user of the inference result of the VFL model, can send feedback information to MTLF. UE, RAN network element and CN network element can send actual measurement data to MTLF. For example, the VFL model is used to locate the UE, AnLF can provide accuracy evaluation information on whether the positioning is accurate, the consumer end can provide feedback information on whether the positioning is accurate, and the RAN network element can provide accurate positioning data of the UE. For example, the RAN network element is a base station, etc., and the CN network element is an AMF (Access and Mobility management Function).

[0044] The data for performance monitoring of the VFL model may be acquired periodically or after each inference. For example, one or more devices may be determined based on the inference result or analysis type of the VFL model.

[0045] In some embodiments, the data used for performance monitoring of the VFL model includes accuracy evaluation data of the VFL model. The MTLF sends a first subscription request to the AnLF, wherein the first subscription request includes an analysis ID and an identifier of the VFL model; and receives the accuracy evaluation data sent by the AnLF based on the analysis ID and the identifier of the VFL model.

[0046] The analytics ID is used to identify the analysis and / or analysis type corresponding to the VFL model. For example, the analytics identifier may indicate a request for an analytics service experience. The VFL model and the corresponding analysis (or reasoning) may be identified by the analytics ID and the identifier of the VFL model. Based on the analytics ID and the identifier of the VFL model, AnLF determines the VFL and the corresponding reasoning result, and may also obtain the authenticity data corresponding to the reasoning result, perform accuracy analysis on the reasoning result of the VFL model, and determine the accuracy evaluation data. For example, the accuracy evaluation data may include accuracy, average error, etc. The accuracy may be multiple accuracy rates for a certain period of time or multiple periods of time, or accuracy rates for different objects, etc. The first subscription request may also include a VFL correlation ID. The VFL correlation ID may be used in the process of associating the VFL, so that AnLF may know which VFL task it is. The VFL correlation ID may also be used to associate some client information, such as the type and weight of the associated client. The types of clients include MTLF and AF (Application Function). The weight of the client may be used to aggregate the intermediate results of each client during training or reasoning.

[0047] In some embodiments, the data used for performance monitoring of the VFL model includes feedback information. The MTLF sends a second subscription request to the consumer end through the data AnLF, wherein the second subscription request includes the analysis ID and the identifier of the VFL model; and the feedback information sent by the consumer end based on the analysis ID and the identifier of the VFL model is received through AnLF, wherein the feedback information includes feedback information from the consumer end on the accuracy of the VFL model or the accuracy of the VFL model for different objects.

[0048] Based on the analysis ID and the identifier of the VFL model, the consumer can determine the VFL model and the corresponding inference result, thereby generating feedback information. The feedback information may include indication information of whether the inference result of the VFL model is accurate.

[0049] In some embodiments, the data used for performance monitoring of the VFL model includes actual measurement data corresponding to the reasoning results of the VFL model, and the MTLF obtains the actual measurement data corresponding to the reasoning results of the VFL model from at least one device including user equipment, wireless access network elements, and core network elements.

[0050] For example, the actual measurement data is positioning data, service quality data, etc., which is determined by the analysis task corresponding to the VFL model. For example, the MTLF sends a third subscription request to at least one device of the user equipment, the wireless access network element, and the core network element, wherein the third subscription request includes the analysis ID and the identifier of the VFL model; and receives the actual measurement data sent by at least one device of the user equipment, the wireless access network element, and the core network element based on the analysis ID and the identifier of the VFL model.

[0051] In step S104, a performance monitoring result of the VFL model is determined according to the data used for performance monitoring of the VFL model.

[0052] In the case where the data used for performance monitoring of the VFL model include accuracy evaluation data and feedback information of the VFL model, the accuracy evaluation data or feedback information may include specific indicators such as the accuracy of the VFL model, which may be directly used as the performance monitoring result. In the case where the data used for performance monitoring of the VFL model include actual measurement data, specific indicators such as the accuracy of the VFL model may be determined based on the actual measurement data and the inference result as the performance monitoring result.

[0053] In some embodiments, the performance monitoring results include at least one of the accuracy and stability of the VFL model. The stability can be determined by the change in accuracy over multiple time periods, or the stability can be determined by the change in accuracy corresponding to multiple objects.

[0054] In step S106, it is determined whether to retrain the VFL model according to the performance monitoring result.

[0055] In some embodiments, when at least one of the accuracy and stability of the VFL model does not meet a preset condition, it is determined to retrain the VFL model.

[0056] For example, when the accuracy of the VFL model is lower than the accuracy threshold, or the stability evaluation value is lower than the corresponding threshold, it is determined to retrain the VFL model.

[0057] The method of the above embodiment can use MTLF to implement performance monitoring of the VFL model. The data for performance monitoring of the VFL model is obtained by interacting with one or more devices in the network through MTLF, and the performance monitoring result of the VFL model is determined according to the performance monitoring data for the VFL model, and then according to the performance monitoring result, it is determined whether to retrain the VFL model. Through the performance monitoring method of VFL in the above embodiment, it can be found in time whether the performance of VFL meets the requirements during the application of the VFL model, and then it is determined whether to retrain the VFL model to ensure the accuracy of the VFL model and improve the accuracy of VFL reasoning.

[0058] In some embodiments, the MTLF sends a performance report to the AnLF, wherein the performance report includes a performance monitoring result, an analysis ID, and an identifier of the VFL model.

[0059] After each performance monitoring result is determined, the MTLF may send the performance monitoring result to the AnLF, and the AnLF may be responsible for unified management of the performance monitoring results of the VFL model. The performance report may also include the VFL association ID.

[0060] In some embodiments, when the MTLF determines to retrain the VFL model, the MTLF sends a notification message to the AnLF, wherein the notification information includes information indicating to retrain the VFL model, the analysis ID, and the identifier of the VFL model.

[0061] The notification message may also include the VFL association ID. The MTLF may notify the AnLF of the retraining information. The AnLF is a subscriber of the VFL model and needs to report the performance and training status of the model to it. The AnLF may feed back the retraining information to the consumer or change the subscription to the VFL model.

[0062] The VFL model can correspond to one or more clients and servers. Each client corresponds to a local model of the VFL model, and the server can also correspond to a local model of the VFL model. MTLF can be a server or a client. The following describes the different situations in which MTLF is a server or a client.

[0063] In some embodiments, when MTLF is the server and determines to retrain the VFL model, MTLF determines one or more clients participating in the retraining based on the identifier of the VFL model; and sends retraining indication information to the one or more clients participating in the retraining, wherein the retraining indication information includes the analysis ID, the identifier of the VFL model, the filter information, and indication information of the training samples used for retraining.

[0064] MTLF can store the corresponding relationship between the identifier of the VFL model and one or more clients participating in the training. The clients participating in the training can be used as clients participating in the retraining. If one or more local models that need to be retrained can be determined, one or more clients corresponding to the one or more local models can be used as clients participating in the retraining.

[0065] For example, the filter information includes at least one of single network slice selection assistance information (S-NSSAI), application ID, area of ​​interest (AOI), and data network access identifier (DN Access Identifier, DNAI). The filter information can be used to indicate the range of data used and can be used to determine whether the client matches the filter information.

[0066] The indication information of the training samples used for retraining may include the identifier of the training samples, the UE ID corresponding to the training samples, etc. The training samples used for retraining may be the same as the previous training samples, or different training samples may be used. In addition, the local model of the client may also be replaced. If the local model of the client is replaced, the retraining indication information may also include indication information of replacing the local model of the client.

[0067] The client participating in the retraining can be MELF or AF. In the case where the client participating in the retraining is AF, if the indication information of the training samples used for the retraining includes the internal UE ID, it is necessary to convert the internal UE ID into an external UE ID through NRF (Network Repository Function) or NEF (Network Exposure Function) and then send it to the client. This can improve security.

[0068] The client participating in the retraining may retrain the local model according to the retraining instruction information. For example, the client participating in the retraining obtains the training samples (may recollect the training samples) according to the instruction information of the training samples used for the retraining, and inputs them into the local model to obtain the intermediate results of the retraining.

[0069] In some embodiments, MELF receives intermediate result information sent by one or more clients participating in retraining, wherein the intermediate result information sent by each client participating in retraining includes an analysis ID, an identifier of a VFL model, and an intermediate result of retraining; determines a gradient parameter of each client participating in retraining based on the intermediate result information; and sends gradient indication information to each client participating in retraining, wherein the gradient indication information corresponding to each client participating in retraining includes an analysis ID, an identifier of a VFL model, and a gradient parameter of each client participating in retraining.

[0070] Since the training samples of the clients participating in the retraining are not labeled and the gradient cannot be calculated, the clients participating in the retraining transmit the intermediate results to the server with labeled data. The server aggregates the intermediate training results of the clients participating in the retraining, calculates the gradient of the local model of each client participating in the retraining based on the labeled data, and returns the gradient parameters to the clients participating in the retraining. After receiving the gradient parameters, the clients participating in the retraining update the local model and proceed to the next round of training. The server can determine the weight of each client participating in the retraining based on the VFL association ID, and determine the gradient parameters of each client participating in the retraining based on the weight of each client participating in the retraining.

[0071] In some embodiments, the MTLF sends retraining training information to the AnLF, wherein the retraining training information includes at least one of a retraining training progress and a training result.

[0072] MTLF can send retraining training information to AnLF after each round of training, so that AnLF can know the progress of retraining and the training results when the retraining is finished.

[0073] In some embodiments, when MTLF is a client and determines to retrain the VFL model, a retraining request is sent to the server, wherein the retraining request includes the analysis ID, the identifier of the VFL model, and the retraining request indication information; a retraining response is received from the server, wherein the retraining response includes the analysis ID, the identifier of the VFL model, filter information, and indication information of the training samples used for retraining; and the VFL model is retrained according to the retraining response.

[0074] MTLF can be a client. If the client determines that the VFL model needs to be retrained, it can request the server to retrain it. All clients participating in the training can retrain together, or only MTLF can act as a client to retrain the local model.

[0075] In some embodiments, the training samples used for retraining are input into the VFL model to obtain the intermediate results of retraining; the intermediate result information is sent to the server, wherein the intermediate result information sent by each client participating in the retraining includes the analysis ID, the identifier of the VFL model and the intermediate result of retraining; the gradient indication information sent by the server is received, wherein the gradient indication information includes the analysis ID, the identifier of the VFL model, and the gradient parameters; and the parameters of the VFL model are adjusted according to the gradient parameters.

[0076] The process of retraining the MTLF as a client can refer to the above embodiment, which will not be described in detail here. The AF as a client can also perform the retraining process, which will not be described in detail here.

[0077] Before the performance of the VFL model is monitored, the process of using the VFL model for reasoning and training is described below in conjunction with some embodiments.

[0078] In some embodiments, MTLF is a server, which receives an analysis subscription request sent by AnLF, wherein the analysis subscription request includes an analysis ID, filter information, information of an analysis report target, and an identifier of a VFL model; selects one or more clients participating in reasoning from clients participating in training according to the VFL association ID and filter information; and sends reasoning indication information to one or more clients participating in reasoning, wherein the reasoning indication information includes the analysis ID and the identifier of the VFL model.

[0079] After the training of the VFL model is completed, AnLF can send an analysis subscription request to the VFL server NWDAF through Nnwdaf_AnalyticsSubscription_Subscribe. The analysis subscription request is used to request reasoning (analysis) using the VFL model. The information of the analysis report target may include the UE ID.

[0080] The MTLF server decides to start the distributed VFL reasoning program to generate analysis results together with the client. The server selects the client participating in the reasoning from the clients participating in the VFL model training based on the VFL association ID and filter information, and notifies the client to participate in the distributed reasoning. The VFL model-related ID contains the type and weight of each client. The MTLF server can try to select clients with large weights and belonging to MTLF during reasoning. If there is AF in the client and the AF weight is not large, do not select it. The reasoning indication information can also include filter information and analysis report target information. The client can determine whether it matches the filter information based on the filter information, and can determine the analysis report target corresponding to the data used for reasoning based on the information of the analysis report target.

[0081] In some embodiments, MTLF acts as a server and receives an analysis response message sent by each of one or more clients participating in reasoning, wherein the analysis response message includes the intermediate reasoning result of each client participating in reasoning, the analysis ID, and the identifier of the VFL model; and generates the reasoning result of the VFL model based on the intermediate reasoning result of each client participating in reasoning.

[0082] If the client participating in the reasoning does not have the required reasoning data, it will first collect data, then use local data and local models to perform reasoning, and send the intermediate reasoning results to the server after reasoning. The server collects the intermediate reasoning results of each client participating in the reasoning, aggregates the results based on the VFL model association ID, and generates the final reasoning result.

[0083] In some embodiments, an analysis result message is sent to AnLF, wherein the analysis result message includes the inference result, the analysis ID, and the identifier of the VFL model.

[0084] For example, the server sends the inference result to AnLF through Nnwdaf_AnalyticsSubscription_Notify (NWDAF analysis subscription notification) or Naf_EventExposure_Notify (Naf interface event exposure notification). Furthermore, NWDAF with AnLF returns the inference result to the consumer through Nnwdaf_AnalyticsInfo_Request (NWDAF analysis information response) or Nnwdaf_AnalyticsSubscription_Notify (NWDAF analysis subscription notification) service.

[0085] In some embodiments, MTLF is a client, which receives reasoning indication information sent by a server, wherein the reasoning indication information includes analysis ID, filter information, information of analysis report target and identification of VFL model; according to the reasoning indication information, reasoning is performed using a local model corresponding to the VFL model to obtain an intermediate reasoning result; an analysis response message is sent to the server, so that the server generates a reasoning result of the VFL model according to the analysis response message, wherein the analysis response message includes the intermediate reasoning result of each client participating in the reasoning, the analysis ID and the identification of the VFL model.

[0086] When MTLF is used as the client, distributed reasoning is performed. For details, please refer to the above embodiment, which will not be repeated here. When AF is used as the client, the reasoning process can also be performed, which will not be repeated here. The server can also use the local model for reasoning, which will not be repeated here.

[0087] The method of the above embodiment realizes distributed reasoning using the VFL model through interaction between the server and the client. The accuracy of the interaction between the server and the client is ensured through the involvement of the above interaction process, thereby improving the accuracy of reasoning.

[0088] The following describes how to train the VFL model in conjunction with some embodiments.

[0089] In some embodiments, the MTLF sends a registration request to the NRF (or NEF), wherein the registration request includes at least one of NF (Network Function) profile information, analysis ID, address information, service area, and capability type; and receives a registration response sent by the NRF, wherein the registration response includes information indicating whether the registration is successful.

[0090] AF can also register with NRF as a client. NF profile information includes NF ID, NF type, NF capability, PLMN (Public Land Mobile Network) ID, etc. Capability types include server and client.

[0091] In some embodiments, MTLF is a server, receives an analysis request sent by AnLF, wherein the analysis request includes an analysis ID, information of an analysis report target, filter information, and an analysis target period; determines whether to support becoming a server according to the capability type of MTLF; in response to supporting becoming a server, sends a discovery request to NRF, wherein the discovery request includes an analysis ID and filter information; receives a discovery response sent by NRF based on the discovery request, wherein the discovery response includes the identifiers of one or more clients participating in the training; sends a training preparation message to one or more clients participating in the training, wherein the training preparation message includes an analysis ID, information of an analysis report target, filter information, an analysis target period, and a VFL association ID; receives a training preparation response sent by one or more clients participating in the training, wherein the training preparation response includes indication information on whether to participate in the training.

[0092] The analysis target period can be used to indicate the target period corresponding to the data used for analysis. The consumer can send an analysis request or subscription message to AnLF through the Nnwdaf_AnalyticsSubscription_Subscribe service. The request includes the analysis ID, filter information, analysis target period, and analysis report target information. AnLF can subscribe to the ML model from MTLF through the Nnwdaf_MLModelProvison_Subscribe service. MTLF determines that the model needs to be trained using the VFL method. If the MTLF does not support becoming a server, it will find another MTLF that can become a server. The VFL server discovers and selects one or more participating training clients (MTLF or AF) from NRF (or NEF). The screening conditions include analysis ID, external UE ID, filtering information, client capability identification, etc. The server notifies the selected client to prepare for VFL model training. The server can also notify AnLF that the VFL method will be used for training.

[0093] In some embodiments, MTLF is a server, which sends a training indication message to one or more clients participating in the training, wherein the training indication message includes an analysis ID, an identifier of a VFL model, filter information, and indication information of training samples; performs feature alignment of training samples with one or more clients participating in the training; receives intermediate result information sent by each of the one or more clients participating in the training, wherein the intermediate result information sent by each client participating in the training includes an analysis ID, an identifier of a VFL model, and an intermediate result of the training; determines a gradient parameter of each client participating in the training based on the intermediate result information; sends gradient indication information to each client participating in the training, wherein the gradient indication information corresponding to each client participating in the training includes an analysis ID, an identifier of a VFL model, and a gradient parameter of each client participating in the training.

[0094] The indication information of the training sample may include information such as the analysis report target, the analysis target period, and other information used to indicate the training sample. After the client receives the training indication message, if there is no training sample locally, it can collect data for training. For example, the network element from which the client collects information depends on the training task and model. For example, for a positioning task, it can be collected from the UE, AMF, RAN and / or LMF (Location Management Function). When collecting information, the analysis ID, data report address (if stored in ADRF (Analytics Data Repository Function)), and UE ID can be provided. If it is collected from the UE, it is first necessary to query the UDM (User Data Management) whether the UE authorized the client to collect data when signing the contract.

[0095] The server and client perform data sampling (mapping the external UE ID and the internal UE ID) and feature alignment (so that different training entities have different feature spaces). If the VFL client contains AF, the data alignment is performed with the assistance of NEF. For example, the server matches the internal and external UE IDs one by one to ensure that the sample order between clients is consistent. The client trains the local model with local data and obtains intermediate results. The client's local data is unlabeled and the gradient cannot be calculated. The client transmits the intermediate results to the server with labeled data. The server aggregates the client's intermediate training results, calculates the gradient of each client's local model based on the labeled data, and returns the gradient parameters to the client; after receiving the gradient parameters, the client updates the local model and performs the next round of training.

[0096] In some embodiments, MTLF is a client, which receives a training preparation message sent by a server, wherein the training preparation message includes an analysis ID, information of an analysis report target, filter information, an analysis target cycle, and a VFL association ID; and sends a training preparation response to the server, wherein the training preparation response includes indication information on whether to participate in the training.

[0097] In some embodiments, MTLF is a client, which receives a training indication message sent by a server, wherein the training indication message includes an analysis ID, an identifier of a VFL model, filter information, and indication information of a training sample; obtains a training sample according to the indication information of the training sample; performs feature alignment of the training sample with one or more clients participating in the training; inputs the training sample into a local model corresponding to the VFL model to obtain an intermediate result; sends intermediate result information to the server, wherein the intermediate result information includes an analysis ID, an identifier of a VFL model, and an intermediate result of the training; and receives gradient indication information sent by the server, wherein the gradient indication information includes an analysis ID, an identifier of a VFL model, and gradient parameters.

[0098] The server can feed back the training progress and results to AnLF.

[0099] The above embodiment provides a method for interaction between the server, the client and the AnLF during the VFL model training process, which can realize the distributed training of the VFL model, ensure the accuracy of the interaction information, and thus improve the accuracy of the VFL model.

[0100] Combine the following Figure 2 Some embodiments of the VFL performance monitoring method of the present disclosure are described.

[0101] Figure 2 Flowcharts of some embodiments of the VFL performance monitoring method disclosed in the present invention. Figure 2 As shown, the method of this embodiment includes: steps S201 to S225.

[0102] In step S201 , the server (NWDAF including MTLF) and the client (NWDAF or AF) register with the NRF.

[0103] In step S202, the consumer sends an analysis request to AnLF (NWDAF including AnLF).

[0104] In step S203, AnLF sends an analysis request to MTLF.

[0105] In step S204, the MTLF determines that VFL training is required. If the MTLF supports being a server, it discovers and selects a client (MTLF or AF) to participate in the training from the NRF (NEF), and notifies the selected client to prepare for VFL training.

[0106] In step S205, the server notifies AnLF, which has subscribed to the VFL model, that the model training will adopt the VFL method.

[0107] In step S206 , the client collects data for training.

[0108] In step S207, the server and the client perform data sample and feature alignment. If the client includes AF, the data alignment is performed with the assistance of NEF.

[0109] In step S208, the client trains the local model using the local data and obtains an intermediate result.

[0110] In step S209, the client sends the intermediate result to the server.

[0111] In step S210, the server aggregates the intermediate training results of the client.

[0112] In step S211, the server calculates the gradient parameters of each client local model and returns the gradient parameters to the client.

[0113] In step S212, after receiving the gradient parameters, the client updates the local model and performs the next round of training.

[0114] In step S213, the server feeds back the training progress and results to AnLF that subscribes to the VFL model.

[0115] In step S214, after the VFL model training is completed, AnLF sends an analysis subscription request to the server.

[0116] In step S215, the server decides to start the distributed VFL reasoning program to generate analysis results together with the client. The server selects the client participating in reasoning from the clients participating in training.

[0117] In step S216, the server notifies the client to participate in distributed reasoning.

[0118] In step S217 , the client uses local data and local models to perform reasoning. If the required reasoning data is not available, data collection is performed first.

[0119] In step S218, the client sends the intermediate inference result to the server.

[0120] In step S219, the server collects the intermediate reasoning results of the clients, and aggregates the intermediate reasoning results of each client to generate a final reasoning result.

[0121] In step S220, the server sends the inference result to AnLF, and AnLF sends the inference result to the consumer.

[0122] In step S221, MTLF subscribes to the accuracy evaluation data of the VFL model provided by AnLF or the feedback information from the consumer end to monitor the real-time performance of the VFL model.

[0123] In step S222, the MTLF collects data for monitoring model performance from at least one device of a user equipment, a radio access network element, and a core network element, and monitors the performance of the VFL model.

[0124] In step S223, the MTLF monitors the performance of the VFL model and evaluates the accuracy and stability of the model.

[0125] In step S224 , the MTLF determines whether to retrain the VFL model.

[0126] In step S225, MTLF feeds back the VFL performance report to AnLF, and notifies AnLF if retraining is required.

[0127] The various steps in the above embodiment may refer to the description in the aforementioned embodiment, and the subsequent retraining process may also refer to the aforementioned embodiment, which will not be described in detail here.

[0128] The VFL performance monitoring method disclosed in the present invention monitors the performance of the VFL model in real time, evaluates the accuracy and efficiency of the model, and optimizes it according to the monitoring results, which can improve the accuracy of the reasoning results of the VFL model. Through VFL technology, efficient data utilization and intelligent analysis can be achieved between different network entities without sharing original data or models, making intelligent collaboration safer. VFL training and reasoning only need to exchange intermediate results and do not involve model transmission, so it is compatible with the participation of network elements from different manufacturers. The VFL training, reasoning and performance monitoring method disclosed in the present invention supports distributed collaboration, so that VFL clients with different local models can jointly participate in training and reasoning, improving analysis and prediction performance.

[0129] The present disclosure also provides a model training logic function. Figure 3 Give a description.

[0130] Figure 3 FIG. 1 is a block diagram of some embodiments of the model training logic function (MTLF) disclosed in the present invention. Figure 3As shown, the MTLF 30 of this embodiment includes: an acquisition module 310 , a first determination module 320 , and a second determination module 330 .

[0131] The acquisition module 310 is used to acquire data for performance monitoring of the VFL model from one or more devices in the network, wherein the data for performance monitoring of the VFL model corresponds to the inference result of the VFL model.

[0132] The first determination module 320 is used to determine the performance monitoring result of the VFL model according to the data used for performance monitoring of the VFL model.

[0133] The second determination module 330 is used to determine whether to retrain the VFL model according to the performance monitoring result.

[0134] In some embodiments, the data used for performance monitoring of the VFL model includes accuracy evaluation data of the VFL model. The acquisition module 310 is used to send a first subscription request to the data analysis logical network element AnLF, wherein the first subscription request includes an analysis ID and an identifier of the VFL model; and receive the accuracy evaluation data sent by AnLF based on the analysis ID and the identifier of the VFL model.

[0135] In some embodiments, the data used for performance monitoring of the VFL model includes feedback information, and the acquisition module 310 is used to send a second subscription request to the consumer end through the data analysis logical network element AnLF, wherein the second subscription request includes the analysis ID and the identifier of the VFL model; and receive feedback information sent by the consumer end based on the analysis ID and the identifier of the VFL model through AnLF, wherein the feedback information includes feedback information from the consumer end on the accuracy of the VFL model.

[0136] In some embodiments, the data used for performance monitoring of the VFL model includes actual measurement data corresponding to the inference results of the VFL model, and the acquisition module 310 is used to obtain the actual measurement data corresponding to the inference results of the VFL model from at least one device including user equipment, wireless access network elements and core network elements.

[0137] In some embodiments, the performance monitoring result includes at least one of the accuracy and stability of the VFL model, and the second determination module 330 is used to determine to retrain the VFL model when at least one of the accuracy and stability of the VFL model does not meet a preset condition.

[0138] In some embodiments, the MTLF further includes a sending module 340, which is used to send a performance report to the data analysis logical network element AnLF, wherein the performance report includes a performance monitoring result, an analysis ID, and an identifier of a VFL model.

[0139] In some embodiments, the sending module 340 is used to send a notification message to the data analysis logical network element AnLF when it is determined that the VFL model needs to be retrained, wherein the notification information includes information indicating that the VFL model needs to be retrained, the analysis ID, and the identifier of the VFL model.

[0140] In some embodiments, the sending module 340 is further used to send retraining training information to the data analysis logical network element AnLF, wherein the retraining training information includes at least one of the retraining training progress and training results.

[0141] In some embodiments, the sending module 340 is also used to send a registration request to the network repository function NRF, wherein the registration request includes at least one of the network function NF profile information, analysis ID, address information, service area, and capability type; the MTLF also includes a receiving module 350, which is used to receive a registration response sent by the NRF, wherein the registration response includes information indicating whether the registration is successful.

[0142] When MTLF is the server, the specific functions of each module in MTLF are as follows.

[0143] In some embodiments, when it is determined that the VFL model is to be retrained, a sending module 340 is used to determine one or more clients participating in the retraining based on the identifier of the VFL model; and send retraining indication information to the one or more clients participating in the retraining, wherein the retraining indication information includes an analysis ID, an identifier of the VFL model, filter information, and indication information of training samples used for retraining.

[0144] In some embodiments, the MTLF also includes a third determination module, and the receiving module 350 is used to receive intermediate result information sent by one or more clients participating in the retraining, wherein the intermediate result information sent by each client participating in the retraining includes an analysis ID, an identifier of the VFL model, and the intermediate result of the retraining; the third determination module is used to determine the gradient parameters of each client participating in the retraining based on the intermediate result information; the sending module 340 is also used to send gradient indication information to each client participating in the retraining, wherein the gradient indication information corresponding to each client participating in the retraining includes the analysis ID, the identifier of the VFL model, and the gradient parameters of each client participating in the retraining.

[0145] In some embodiments, the receiving module 350 is also used to receive an analysis subscription request sent by the data analysis logical network element AnLF, wherein the analysis subscription request includes an analysis ID, filter information, information on the analysis report target, and an identifier of the VFL model; the MTLF also includes a selection module for selecting one or more clients participating in reasoning from clients participating in training based on the VFL association ID and filter information; the sending module 340 is also used to send reasoning indication information to one or more clients participating in reasoning, wherein the reasoning indication information includes the analysis ID and the identifier of the VFL model.

[0146] In some embodiments, the receiving module 350 is also used to receive an analysis response message sent by each of the one or more clients participating in the reasoning, wherein the analysis response message includes the intermediate reasoning result of each client participating in the reasoning, the analysis ID and the identifier of the VFL model; the third determination module is used to generate the reasoning result of the VFL model based on the intermediate reasoning result of each client participating in the reasoning.

[0147] In some embodiments, the sending module 340 is further used to send an analysis result message to AnLF, wherein the analysis result message includes an inference result, an analysis ID, and an identifier of the VFL model.

[0148] In some embodiments, the receiving module 350 is also used to receive an analysis request sent by the data analysis logical network element AnLF, wherein the analysis request includes an analysis ID, information about the analysis report target, filter information, and an analysis target period; receive a discovery response sent by the NRF based on the discovery request, wherein the discovery response includes the identifiers of one or more clients participating in the training; receive a training preparation response sent by one or more clients participating in the training, wherein the training preparation response includes indication information on whether to participate in the training; the sending module 340 is also used to determine whether to support becoming a server according to the capability type of the MTLF; in response to supporting becoming a server, send a discovery request to the network repository function NRF, wherein the discovery request includes an analysis ID, filter information; send a training preparation message to one or more clients participating in the training, wherein the training preparation message includes the analysis ID, information about the analysis report target, filter information, analysis target period, and VFL association ID.

[0149] In some embodiments, the MTLF also includes a sample processing module, and the sending module 340 is also used to send a training indication message to one or more clients participating in the training, wherein the training indication message includes an analysis ID, an identifier of a VFL model, filter information, and indication information of the training sample; the sample processing module is used to perform feature alignment of the training sample with one or more clients participating in the training; the receiving module 350 is also used to receive intermediate result information sent by each of the one or more clients participating in the training, wherein the intermediate result information sent by each client participating in the training includes an analysis ID, an identifier of a VFL model, and an intermediate result of the training; the third determination module is used to determine the gradient parameters of each client participating in the training based on the intermediate result information; the sending module 340 is also used to send gradient indication information to each client participating in the training, wherein the gradient indication information corresponding to each client participating in the training includes the analysis ID, the identifier of the VFL model, and the gradient parameters of each client participating in the training.

[0150] When MTLF is the client, the specific functions of each module in MTLF are as follows.

[0151] In some embodiments, when it is determined that the VFL model is to be retrained, the sending module 340 is used to send a retraining request to the server, wherein the retraining request includes the analysis ID, the identifier of the VFL model, and the retraining request indication information; the receiving module 350 is used to receive the retraining response sent by the server, wherein the retraining response includes the analysis ID, the identifier of the VFL model, the filter information, and the indication information of the training samples used for retraining. The MTLF also includes a training module for retraining the VFL model according to the retraining response.

[0152] In some embodiments, the training module is used to input the training samples used for retraining into the VFL model to obtain the intermediate results of retraining. The sending module 340 is used to send the intermediate result information to the server, wherein the intermediate result information sent by each client participating in the retraining includes the analysis ID, the identifier of the VFL model and the intermediate result of retraining; the receiving module 350 is used to receive the gradient indication information sent by the server, wherein the gradient indication information includes the analysis ID, the identifier of the VFL model, and the gradient parameters; the training module is used to adjust the parameters of the VFL model according to the gradient parameters.

[0153] In some embodiments, the MTLF also includes an inference module, and the receiving module 350 is used to receive the inference indication information sent by the server, wherein the inference indication information includes the analysis ID, filter information, information of the analysis report target and the identifier of the VFL model; the inference module is used to perform inference using the local model corresponding to the VFL model according to the inference indication information to obtain an intermediate inference result; the sending module 340 is used to send an analysis response message to the server, so that the server generates the inference result of the VFL model according to the analysis response message, wherein the analysis response message includes the intermediate inference result of each client participating in the inference, the analysis ID and the identifier of the VFL model.

[0154] In some embodiments, the receiving module 350 is used to receive a training preparation message sent by the server, wherein the training preparation message includes an analysis ID, information on the analysis report target, filter information, an analysis target cycle, and a VFL association ID; the sending module 340 is used to send a training preparation response to the server, wherein the training preparation response includes indication information on whether to participate in the training.

[0155] In some embodiments, the MTLF also includes a sample processing module, a receiving module 350 for receiving a training indication message sent by a server, wherein the training indication message includes an analysis ID, an identifier of a VFL model, filter information, and indication information of a training sample; a sample processing module for obtaining a training sample according to the indication information of the training sample; and performing feature alignment of the training sample with one or more clients participating in the training; a training module for inputting the training sample into a local model corresponding to the VFL model to obtain an intermediate result; a sending module 340 for sending intermediate result information to the server, wherein the intermediate result information includes an analysis ID, an identifier of a VFL model, and an intermediate result of the training; and a receiving module 350 for receiving gradient indication information sent by the server, wherein the gradient indication information includes an analysis ID, an identifier of a VFL model, and gradient parameters.

[0156] The present disclosure also provides a VFL performance monitoring system. Figure 4 Give a description.

[0157] Figure 4 FIG. 4 is a structural diagram of some embodiments of the VFL performance monitoring system disclosed in the present invention. Figure 4 As shown, the system 4 of this embodiment includes: the MTLF 30 of any of the aforementioned embodiments, and one or more devices 42 in the network, wherein the one or more devices 42 are used to send data for performance monitoring of the VFL model to the MTLF, wherein the data for performance monitoring of the VFL model corresponds to the inference result of the VFL model.

[0158] The one or more devices 42 may be at least one of AnLF, consumer end, UE, RAN network element, and CN network element. MTLF may be a server and / or a client. The devices included in the system 4 and the functions of each device may refer to the aforementioned performance monitoring method, which will not be described in detail here.

[0159] The present disclosure also provides an electronic device, Figure 5 as well as Figure 6 Give a description.

[0160] Figure 5 FIG. 1 is a structural diagram of some embodiments of the electronic device disclosed in the present invention. Figure 5 As shown, the electronic device 50 of this embodiment includes: a memory 510 and a processor 520 coupled to the memory 510 , and the processor 520 is configured to execute the VFL performance monitoring method in any of some embodiments of the present disclosure based on instructions stored in the memory 510 .

[0161] The memory 510 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0162] Figure 6 FIG. 1 is a structural diagram of some other embodiments of the electronic device disclosed in the present invention. Figure 6 As shown, the electronic device 60 of this embodiment includes: a memory 610 and a processor 620, which are similar to the memory 510 and the processor 520 respectively. It can also include an input and output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650 and the memory 610 and the processor 620 can be connected, for example, through a bus 660. Among them, the input and output interface 630 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 640 provides a connection interface for various networked devices, for example, it can be connected to a database server or a cloud storage server. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0163] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0167] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A performance monitoring method for vertical federated learning VFL, applied to model training logic function MTLF, including: Acquire data for performance monitoring of the VFL model from one or more devices in the network, wherein the data for performance monitoring of the VFL model corresponds to the inference result of the VFL model; Determining a performance monitoring result of the VFL model according to the data used for performance monitoring of the VFL model; According to the performance monitoring result, it is determined whether to retrain the VFL model.

2. The performance monitoring method according to claim 1, wherein: The data for performance monitoring of the VFL model includes accuracy evaluation data of the VFL model, and the data for performance monitoring of the VFL model obtained from one or more devices in the network includes: Sending a first subscription request to the data analysis logic network element AnLF, wherein the first subscription request includes an analysis ID and an identifier of the VFL model; The accuracy evaluation data is received, which is sent by the AnLF based on the analysis ID and the identifier of the VFL model.

3. The performance monitoring method according to claim 1, wherein: The data for performance monitoring of the VFL model includes feedback information, and the acquiring of the data for performance monitoring of the VFL model from one or more devices in the network includes: Sending a second subscription request to the consumer end through the data analysis logic network element AnLF, wherein the second subscription request includes the analysis ID and the identifier of the VFL model; The feedback information sent by the consumer end based on the analysis ID and the identifier of the VFL model is received through the AnLF, wherein the feedback information includes feedback information of the consumer end on the accuracy of the VFL model.

4. The performance monitoring method according to claim 1, wherein: The data for performance monitoring of the VFL model includes actual measurement data corresponding to the inference result of the VFL model, and the data for performance monitoring of the VFL model obtained from one or more devices in the network includes: Acquire actual measurement data corresponding to the inference result of the VFL model from at least one device among user equipment, wireless access network element and core network element.

5. The performance monitoring method according to claim 1, wherein: The performance monitoring result includes at least one of the accuracy and stability of the VFL model, and determining whether to retrain the VFL model according to the performance monitoring result includes: When at least one of the accuracy and stability of the VFL model does not meet a preset condition, it is determined to retrain the VFL model.

6. The performance monitoring method according to claim 1, further comprising: A performance report is sent to the data analysis logic network element AnLF, wherein the performance report includes the performance monitoring result, the analysis ID and the identifier of the VFL model.

7. The performance monitoring method according to claim 1, further comprising: In the case of determining to retrain the VFL model, a notification message is sent to the data analysis logic network element AnLF, wherein the notification information includes information indicating to retrain the VFL model, an analysis ID, and an identifier of the VFL model.

8. The performance monitoring method according to claim 1, further comprising: In a case where the MTLF is a server and it is determined to retrain the VFL model, determining one or more clients participating in the retraining according to an identifier of the VFL model; Sending retraining indication information to the one or more clients participating in the retraining, wherein the retraining indication information includes an analysis ID, an identifier of the VFL model, filter information, and indication information of training samples used for retraining.

9. The performance monitoring method according to claim 8, further comprising: Receiving intermediate result information sent by the one or more clients participating in the retraining, wherein the intermediate result information sent by each client participating in the retraining includes the analysis ID, the identifier of the VFL model and the intermediate result of the retraining; Determining the gradient parameters of each client participating in the retraining according to the intermediate result information; Sending gradient indication information to each client participating in the retraining, wherein the gradient indication information corresponding to each client participating in the retraining includes the analysis ID, an identifier of the VFL model, and a gradient parameter of each client participating in the retraining.

10. The performance monitoring method according to claim 8, further comprising: Sending retraining training information to the data analysis logical network element AnLF, wherein the retraining training information includes at least one of the retraining training progress and training results.

11. The performance monitoring method according to claim 1, further comprising: When the MTLF is a client and determines to retrain the VFL model, a retraining request is sent to a server, wherein the retraining request includes an analysis ID, an identifier of the VFL model, and retraining request indication information; Receiving a retraining response sent by the server, wherein the retraining response includes the analysis ID, an identifier of the VFL model, filter information, and indication information of training samples used for retraining; The VFL model is retrained according to the retraining response.

12. The performance monitoring method according to claim 11, wherein: Retraining the VFL model according to the retraining response includes: Inputting the training samples used for the retraining into the VFL model to obtain an intermediate result of the retraining; Sending intermediate result information to the server, wherein the intermediate result information sent by each client participating in the retraining includes the analysis ID, the identifier of the VFL model and the intermediate result of the retraining; Receiving gradient indication information sent by the server, wherein the gradient indication information includes the analysis ID, the identifier of the VFL model, and gradient parameters; Parameters of the VFL model are adjusted according to the gradient parameters.

13. The performance monitoring method according to any one of claims 1 to 12, wherein: The MTLF is a server, and the performance monitoring method further includes: Receiving an analysis subscription request sent by a data analysis logic network element AnLF, wherein the analysis subscription request includes an analysis ID, filter information, information of an analysis report target, and an identifier of the VFL model; Select one or more clients participating in reasoning from the clients participating in training according to the VFL association ID and the filter information; Sending reasoning indication information to the one or more clients participating in the reasoning, wherein the reasoning indication information includes the analysis ID and an identifier of the VFL model.

14. The performance monitoring method according to claim 13, further comprising: Receiving an analysis response message sent by each of the one or more clients participating in the reasoning, wherein the analysis response message includes an intermediate reasoning result of each client participating in the reasoning, the analysis ID, and an identifier of the VFL model; The reasoning result of the VFL model is generated according to the intermediate reasoning result of each client participating in the reasoning.

15. The performance monitoring method according to claim 13, further comprising: An analysis result message is sent to the AnLF, wherein the analysis result message includes the inference result, the analysis ID, and an identifier of the VFL model.

16. The performance monitoring method according to any one of claims 1 to 12, wherein: The MTLF is a client, and the performance monitoring method further includes: Receiving reasoning indication information sent by the server, wherein the reasoning indication information includes an analysis ID, filter information, information of an analysis report target, and an identifier of the VFL model; According to the reasoning indication information, reasoning is performed using a local model corresponding to the VFL model to obtain an intermediate reasoning result; Sending an analysis response message to the server so that the server generates an inference result of the VFL model according to the analysis response message, wherein the analysis response message includes the intermediate inference result of each client participating in the inference, the analysis ID and the identifier of the VFL model.

17. The performance monitoring method according to any one of claims 1 to 12, further comprising: Sending a registration request to a network repository function NRF, wherein the registration request includes at least one of network function NF profile information, analysis ID, address information, service area, and capability type; A registration response sent by the NRF is received, wherein the registration response includes information indicating whether the registration is successful.

18. The performance monitoring method according to any one of claims 1 to 12, wherein: The MTLF is a server, and the performance monitoring method further includes: Receiving an analysis request sent by a data analysis logic network element AnLF, wherein the analysis request includes an analysis ID, information of an analysis report target, filter information, and an analysis target period; Determine whether to support becoming a server according to the capability type of the MTLF; In response to supporting becoming a server, sending a discovery request to a network repository function NRF, wherein the discovery request includes the analysis ID and the filter information; Receiving a discovery response sent by the NRF based on the discovery request, wherein the discovery response includes identifications of one or more clients participating in the training; Sending a training preparation message to the one or more clients participating in the training, wherein the training preparation message includes the analysis ID, information of the analysis report target, filter information, analysis target period, and VFL association ID; A training preparation response is received from the one or more clients participating in the training, wherein the training preparation response includes indication information of whether to participate in the training.

19. The performance monitoring method according to claim 18, further comprising: Sending a training instruction message to the one or more clients participating in the training, wherein the training instruction message includes an analysis ID, an identifier of the VFL model, filter information, and instruction information of a training sample; Performing feature alignment of training samples with the one or more clients participating in the training; Receiving intermediate result information sent by each of the one or more clients participating in the training, wherein the intermediate result information sent by each client participating in the training includes the analysis ID, the identifier of the VFL model and the intermediate result of the training; Determine the gradient parameters of each client participating in the training according to the intermediate result information; Sending gradient indication information to each client participating in the training, wherein the gradient indication information corresponding to each client participating in the training includes the analysis ID, an identifier of the VFL model, and a gradient parameter of each client participating in the training.

20. The performance monitoring method according to any one of claims 1 to 12, wherein: The MTLF is a client, and the performance monitoring method further includes: Receiving a training preparation message sent by the server, wherein the training preparation message includes the analysis ID, information of the analysis report target, filter information, analysis target period, and VFL association ID; Send a training preparation response to the server, wherein the training preparation response includes indication information on whether to participate in the training.

21. The performance monitoring method according to claim 20, further comprising: Receiving a training instruction message sent by the server, wherein the training instruction message includes an analysis ID, an identifier of the VFL model, filter information, and instruction information of a training sample; Acquire a training sample according to the instruction information of the training sample; Performing feature alignment of training samples with the one or more clients participating in the training; Inputting the training sample into a local model corresponding to the VFL model to obtain an intermediate result; The intermediate result information sent to the server includes the analysis ID, the identifier of the VFL model and the intermediate result of the training; Receive gradient indication information sent by the server, wherein the gradient indication information includes the analysis ID, an identifier of the VFL model, and gradient parameters.

22. A model training logic function MTLF, comprising: An acquisition module, configured to acquire data for performance monitoring of the VFL model from one or more devices in the network, wherein the data for performance monitoring of the VFL model corresponds to an inference result of the VFL model; A first determining module, configured to determine a performance monitoring result of the VFL model according to the data used for performance monitoring of the VFL model; The second determination module is used to determine whether to retrain the VFL model according to the performance monitoring result.

23. An electronic device comprising: processor; as well as A memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the performance monitoring method of the vertical federated learning (VFL) as described in any one of claims 1 to 21.

24. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the performance monitoring method of the vertical federated learning (VFL) as described in any one of claims 1 to 21 is implemented.

25. A computer program product comprising instructions, which, when executed by a processor, cause the processor to perform the performance monitoring method of the vertical federated learning (VFL) according to any one of claims 1 to 21.

26. A VFL performance monitoring system, comprising: The MTLF according to claim 22; and one or more devices in a network, wherein: The one or more devices are used to send data for performance monitoring of the VFL model to the MTLF, wherein the data for performance monitoring of the VFL model corresponds to the inference result of the VFL model.

Images