Performance monitoring of machine learning models

By actively monitoring and reporting client performance during the horizontal federated learning training loop, the problem of client performance degradation is solved, the training process is optimized, and the efficiency and quality of model training are improved.

CN123068237APending Publication Date: 2026-11-10NOKIA TECHNOLOGIES OY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610623589.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-05-08
Filing Date
2026-05-08
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In horizontal federated learning, participating clients suffer from performance degradation due to a lack of data and computing resources, and existing performance monitoring schemes increase communication overhead, making it impossible to effectively monitor and improve client performance.

Method used

A machine learning model performance monitoring scheme is proposed. By actively monitoring and reporting client performance during the HFL training cycle, performance degradation can be detected, and an adaptation mechanism can be introduced to optimize the training process to improve the performance of poorly performing clients.

Benefits of technology

Effectively monitor and improve client performance in horizontal federated learning training, reduce communication overhead, and improve model training efficiency and quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068237A_ABST
    Figure CN123068237A_ABST
Patent Text Reader

Abstract

The example implementations of this disclosure relate to a solution for performance monitoring of a machine learning model. One method includes: receiving from a first server a report request for reporting the performance of a machine learning model, the report request including at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance; evaluating the performance of the machine learning model based on the report request using local data from the device; and sending a report response to the report request to the first server, the report response relating to the evaluated performance of the machine learning model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Various exemplary embodiments disclosed herein relate generally to the telecommunications field, and more particularly to methods, apparatus, devices, and computer-readable storage media for performance monitoring of machine learning models. Background Technology

[0002] Due to privacy concerns, federated learning techniques have been introduced to enable multiple parties to jointly train machine learning (ML) models while maintaining their data locally and protecting their privacy. More specifically, federated learning techniques are divided into two categories: horizontal federated learning (HFL) and vertical federated learning (VFL). Performance monitoring is a crucial stage for ML models, ensuring that the model's output is effective. Summary of the Invention

[0003] In a first aspect of this disclosure, an apparatus is provided. The apparatus includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: receive from a first server a report request for reporting the performance of a machine learning model, the report request including at least one of: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance; evaluate the performance of the machine learning model based on the report request using local data of the apparatus; and send a report response to the report request to the first server, the report response relating to the evaluated performance of the machine learning model.

[0004] In a second aspect of this disclosure, a first server is provided. The first server includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the first server to at least: send a report request to a device for reporting the performance of a machine learning model, the report request including at least one of: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance; receive a report response from the device in response to the report request, the report response relating to the performance of the machine learning model evaluated by the device; and detect performance degradation of the machine learning model based on the performance of the evaluated machine learning model.

[0005] In a third aspect of this disclosure, a second server is provided. The second server includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the second server to at least: send a monitoring request to a first server for monitoring the performance of a machine learning model, the monitoring request including a reference dataset for evaluating the performance of the machine learning model; and receive a notification from the first server including the performance of the machine learning model evaluated by the first server using the reference dataset.

[0006] In a fourth aspect of this disclosure, a method is provided. The method includes: receiving from a first server a report request for reporting the performance of a machine learning model, the report request including at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance; evaluating the performance of the machine learning model based on the report request using local data of the device; and sending a report response to the report request to the first server, the report response relating to the evaluated performance of the machine learning model.

[0007] In a fifth aspect of this disclosure, a method is provided. The method includes: sending a report request to a device for reporting the performance of a machine learning model, the report request including at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance; receiving a report response from the device in relation to the report request, the report response being related to the performance of the machine learning model evaluated by the device; and detecting performance degradation of the machine learning model based on the performance of the evaluated machine learning model.

[0008] In a sixth aspect of this disclosure, a method is provided. The method includes: sending a monitoring request to a first server for monitoring the performance of a machine learning model, the monitoring request including a reference dataset for evaluating the performance of the machine learning model; and receiving a notification from the first server including the performance of the machine learning model evaluated by the first server using the reference dataset.

[0009] In a seventh aspect of this disclosure, an apparatus is provided. The apparatus includes: components for receiving from a first server a report request for reporting the performance of a machine learning model, the report request including at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance; components for evaluating the performance of the machine learning model based on the report request using local data of the apparatus; and components for sending a report response to the report request to the first server, the report response relating to the evaluated performance of the machine learning model.

[0010] In an eighth aspect of this disclosure, a first server is provided. The first server includes: components for sending a report request to a device for reporting the performance of a machine learning model, the report request including at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance; components for receiving a report response from the device in response to the report request, the report response relating to the performance of the machine learning model evaluated by the device; and components for detecting performance degradation of the machine learning model based on the performance of the evaluated machine learning model.

[0011] In a ninth aspect of this disclosure, a second server is provided. The second server includes: components for sending a monitoring request to a first server for monitoring the performance of a machine learning model, the monitoring request including a reference dataset for evaluating the performance of the machine learning model; and components for receiving a notification from the first server, the notification including the performance of the machine learning model evaluated by the first server using the reference dataset.

[0012] In a tenth aspect of this disclosure, a computer-readable medium is provided. The computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to a fourth aspect.

[0013] In the eleventh aspect of this disclosure, a computer-readable medium is provided. The computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to the fifth aspect.

[0014] In a twelfth aspect of this disclosure, a computer-readable medium is provided. The computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to the sixth aspect.

[0015] It should be understood that the summary portion is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0016] Some exemplary embodiments will now be described with reference to the accompanying drawings, in which: Figure 1 An example communication environment in which example embodiments of this disclosure may be implemented is shown; Figure 2A The signaling flow for HFL training is shown; Figure 2B The signaling flow for ML model performance monitoring is shown; Figure 3A and Figure 3B Signaling flows for performance monitoring according to some example embodiments of this disclosure are shown; Figure 4 Signaling flows for performance monitoring according to some example embodiments of this disclosure are shown; Figure 5 A flowchart is shown illustrating a method implemented at an apparatus according to some example embodiments of the present disclosure; Figure 6 A flowchart is shown illustrating a method implemented at a first server according to some example embodiments of the present disclosure; Figure 7 A flowchart is shown illustrating a method implemented at a second server according to some example embodiments of the present disclosure; Figure 8 A flowchart is shown illustrating a method implemented at an apparatus according to some example embodiments of the present disclosure; Figure 9 A flowchart is shown illustrating a method implemented at a first server according to some example embodiments of the present disclosure; Figure 10 A flowchart is shown illustrating a method implemented at a second server according to some example embodiments of the present disclosure; Figure 11 A simplified block diagram of a device suitable for implementing example embodiments of the present disclosure is shown; and Figure 12 A block diagram of an example computer-readable medium according to some example embodiments of the present disclosure is shown.

[0017] In all the accompanying drawings, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0018] The principles of this disclosure will now be described with reference to some exemplary embodiments. It should be understood that these embodiments are described for illustrative purposes only and to help those skilled in the art to understand and implement this disclosure, without implying any limitation on the scope of this disclosure. The embodiments described herein can be implemented in various ways other than those described below.

[0019] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0020] References to "an embodiment," "an embodiment," "an example embodiment," etc., in this disclosure indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment includes that particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. In addition, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is to be noted that those skilled in the art will recognize, whether explicitly described or not, that such features, structures, or characteristics apply in conjunction with other embodiments.

[0021] It should be understood that although terms such as "first," "second," etc., may be used before names (or similar designations) to describe various elements herein, these elements should not be limited by these terms. These terms are used only to distinguish one element from another, and they do not restrict the order of the nouns (or similar designations). For example, without departing from the scope of the exemplary embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0022] As used herein, “at least one of the following: ” and “at least one of ” and similar expressions, wherein the list of two or more elements is connected by “and” or “or”, means at least any one of these elements, or at least any two or more of these elements, or at least all of these elements.

[0023] As used herein, unless explicitly stated otherwise, the action “in response to A” does not indicate that the action is performed immediately after “A” occurs and may include one or more intervention steps.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “possessing,” “containing,” and / or “covering,” as used herein, specify the presence of the stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0025] As used in this application, the term "circuit" may refer to one or more of the following: (a) Hardware circuit implementation only (such as implementations in analog, digital and / or quantum circuits) and (b) A combination of (multiple) hardware circuits and software, such as (if applicable): (i) A combination of (multiple) analog, digital, and / or quantum hardware circuits and software / firmware, and (ii) Any or all portions of (multiple) hardware processors (including (multiple) digital and / or quantum signal processors) having software, and (multiple) memories, which work together to enable a device (such as a mobile device, computing device, or server) to perform various functions, and (c) Any or all parts of the hardware circuitry, such as microprocessors, processors and / or quantum processors, that require software (e.g., firmware) to operate, but which may not exist when no software is required to operate.

[0026] This definition of "circuit" applies to all uses of the term in this application. As a further example, as used in this application, the term "circuit" also covers only hardware circuitry or processors (or processors), or portions of hardware circuitry or servers and their accompanying software and / or firmware implementations. For example, where applicable to certain claim elements, the term "circuit" also covers baseband integrated circuits or processor integrated circuits for mobile devices or similar integrated circuits in servers, cellular network devices, or other computing or networking devices.

[0027] As used herein, the term "communication network" refers to a network that conforms to any suitable communication standard, such as New Radio (NR), Long Term Evolution (LTE), LTE-A Advanced (LTE-A), Wideband Code Division Multiple Access (WCDMA), High-Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. Furthermore, communication between terminal devices and network devices in a communication network can be performed according to any suitable generation of communication protocol, including but not limited to first-generation (1G), second-generation (2G), 2.5G, 2.75G, third-generation (3G), fourth-generation (4G), 4.5G, fifth-generation (5G), sixth-generation (6G) communication protocols, and / or any other currently known or under development protocols. Embodiments of this disclosure can be applied to various communication systems. Given the rapid development of communications, future types of communication technologies and systems that can implement this disclosure will inevitably emerge. The scope of this disclosure should not be considered limited to the systems described above.

[0028] As used herein, the term "server" refers to a node in a communications network that can serve a client / UE. Depending on the terminology and technology applied, network equipment can refer to a base station (BS) or access point (AP), such as a Node B (NodeB or NB), an evolved Node B (eNodeB or eNB), an NR NB (also known as a gNB), a Remote Radio Unit (RRU), a Radio Header End (RH), a Remote Radio Header End (RRH), a relay, an Integrated Access and Backhaul (IAB) node, a low-power node (such as a femtosecond or picosecond), a non-terrestrial network (NTN) or non-terrestrial network equipment (such as satellite network equipment, low Earth orbit (LEO) satellites, and geostationary orbit (GEO) satellites), an aircraft network equipment, etc. In some example embodiments, the Radio Access Network (RAN) separation architecture includes a centralized unit (CU) and a distributed unit (DU) at the IAB donor node. The IAB node includes a mobile terminal (IAB-MT) portion that behaves like a UE towards its parent node, and the DU portion of the IAB node behaves like a base station towards the next-hop IAB node.

[0029] As used herein, the term "server" can also refer to a core network (CN) entity / node / function / device / equipment. Example core network nodes may include functions such as: Location Management Function (LMF), Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier Dehiding Function (SIDF), Unified Data Management (UDM), Security Edge Protection Agent (SEPP), Network Exposure Function (NEF), and / or User Plane Function (UPF), etc.

[0030] The term "device / client" refers to any end device with wireless communication capabilities. As an example and not a limitation, "end device" can also refer to communication equipment, user equipment (UE), subscriber station (SS), portable subscriber station, mobile station (MS), or access terminal (AT). End devices can include, but are not limited to, mobile phones, cellular phones, smartphones, Voice over IP (VoIP) phones, wireless local loop phones, tablets, wearable end devices, personal digital assistants (PDAs), portable computers, desktop computers, image acquisition end devices (such as digital cameras), gaming end devices, music storage and playback devices, in-vehicle wireless end devices, wireless endpoints, mobile stations, laptop embedded devices (LEE), laptop mounted devices (LME), USB dongles, smart devices, wireless client devices (CPE), Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. The terminal device may also correspond to the mobile terminal (MT) portion of an IAB node (e.g., a relay node). In the following description, the terms "terminal device," "communication device," "terminal," "user equipment," and "UE" are used interchangeably.

[0031] As mentioned above, federated learning techniques fall into two categories: HFL and VFL. In HFL, all collaborating parties share the same feature set, but they have different samples distributed among them. For example, all wearable devices collect the same set of measurements (e.g., heart rate, temperature, etc.) from different users (i.e., different samples) to predict health-related issues, such as heart attack prediction. Therefore, this type of federated learning (i.e., HFL) is suitable when all collaborating parties have the same data structure.

[0032] On the other hand, in VFL (Virtual Functional Framework), collaborators possess different features of the same sample, and they collectively complement the feature space of that same sample to train an ML model together to predict something or provide ML-based services about that same sample. For example, a wearable device collects some health-related characteristics (e.g., heart rate, temperature, etc.) of a specific person (i.e., a specific sample), and the medical records of the same specific person (i.e., the same specific sample) also bring other characteristics of the same person (i.e., the same sample) (medical examinations, medications, etc.). Both parties jointly train a model to predict, for example, a heart attack for that specific person.

[0033] Since edge artificial intelligence (AI) / machine learning (ML) applications are the primary consumers of wireless communication data analytics services, the vertical application enablement layer (VAL) is an integral part of exposing AI / ML services to third-party vertical application service providers. Essentially, the VAL enables third-party servers and UE consumers to interact or coordinate with each other and execute AI / ML model operations. Generally, a VAL server provides server-side application functionality for a specific VAL service, and a VAL client provides client-side functionality for that specific VAL service. AI / ML enablement (AIMLE) is a service that assists in executing or enabling one or more AI / ML operations. The AIMLE server performs server-side operations for the AI / ML service, and the AIMLE client performs client-side operations for the AI / ML service.

[0034] In some solutions, a third-party application service provider acts as a VAL server, delivering HFL tasks to its service consumers (i.e., VAL clients). Therefore, the VAL server requests assistance from the AIMLE server for its HFL tasks and performance monitoring.

[0035] A major challenge of HFL is the lack of diverse data (e.g., all possible categories in a classification problem) across participating clients. Furthermore, data is private to each participating client and cannot be shared. Therefore, some clients with only a limited number of categories in their local datasets are prone to significant performance degradation during HFL training iterations, as other clients with more diverse local datasets may bias the aggregation model during HFL training, leading to significant performance drops for other clients. This problem becomes even more pronounced when UE devices have limited computational resources for participating in ML model training scenarios. Therefore, a solution is desired to proactively monitor the performance of UE clients participating in HFL training in a manner that not only reports UE client performance but also provides a mechanism for improving parameter aggregation, allowing underperforming clients to achieve better results and improve their performance in subsequent iterations after performance degradation is detected.

[0036] Based on some example embodiments of this disclosure, a solution for performance monitoring of machine learning models is proposed. Specifically, a solution for performance monitoring during HFL training cycles is proposed, which enhances the HFL training process in various aspects of HFL training cycle optimization, client (i.e., UE) performance monitoring, and client performance improvement.

[0037] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0038] Figure 1 An example communication environment 100 in which exemplary embodiments of the present disclosure can be implemented is shown. Communication environment 100 includes device 110-1 and optional devices 110-2 to 110-N, where N is an integer greater than 1 (collectively referred to as device 110 or individually referred to as device 110). Communication environment 100 also includes a first server 120 and a second server 130. Figure 1 As described above, the device (110) and the first server 120 can communicate with each other, and the first server 120 and the second server 130 can communicate with each other.

[0039] In some example embodiments, device 110 may include an Artificial Intelligence Machine Learning Enabled Client / UE (AIMLE Client / UE), and first server 120 includes an Artificial Intelligence Machine Learning Enabled Server (AIMLE Server), and second server 130 includes a Vertical Application Layer Server (VAL Server).

[0040] It should be understood that Figure 1 The number of devices and their connections shown are for illustrative purposes only and do not imply any limitation. Communication environment 100 may include any suitable number of devices configured to implement the exemplary embodiments of this disclosure.

[0041] Communication in communication environment 100 can be implemented according to any suitable communication protocol, including but not limited to cellular communication protocols such as first-generation (1G), second-generation (2G), third-generation (3G), fourth-generation (4G), fifth-generation (5G), 5.5G, and sixth-generation (6G), wireless local network communication protocols such as IEEE 802.11, and / or any other currently known or future-developed protocols. Furthermore, communication can utilize any suitable wireless communication technology, including but not limited to: Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Frequency Division Duplex (FDD), Time Division Duplex (TDD), Multiple-Input Multiple-Output (MIMO), Orthogonal Frequency Division Multiple Access (OFDM), Discrete Fourier Transform Extended OFDM (DFT-s-OFDM), and / or any other currently known or future-developed technologies.

[0042] Reference Figure 2A Describe an example process for HFL training. Figure 2A The prerequisites for the example include that AIMLE client discovery and selection have been performed, and that the dataset is available and ready for AI / ML training at the AIMLE client, where the dataset is assigned an identifier for HFL training.

[0043] like Figure 2A As shown, at Action 1, the AIMLE server receives an ML model training request from the VAL server. If the AIMLE server determines that HFL training is to be used, the process continues to Action 2. At Action 2, the AIMLE server uses an appropriate ML model retrieval procedure to retrieve the indicated ML model. Then, at Action 3, for HFL training, the AIMLE server performs AIMLE client selection using either the AIMLE client selection criteria or the AIMLE client list provided at Action 1. If the AIMLE client selection criteria are provided at Action 1, the AIMLE server continuously monitors and selects AIMLE clients for HFL training. If the list of AIMLE clients is in Action 1, the AIMLE server selects the provided AIMLE client for HFL training.

[0044] At Action 4, based on the AIMLE client set determined at Action 3, the AIMLE server checks the capabilities and participation in HFL training of the selected AIMLE clients. As a result, at Action 5, the AIMLE server configures the training schedule for each AIMLE client and sends an HFL training subscription request with relevant information.

[0045] At action 6, each AIMLE client sends an HFL training subscription response with relevant information. If an AIMLE client cannot approve the subscription (e.g., training cannot be performed), the AIMLE client sends a response with a failure status, and the process jumps to action 8.

[0046] At action 7, each AIMLE client performs local training using the configured AI / ML model, model parameters, and prepared local data associated with a dataset identifier for a specified number of samples, according to the operation schedule. At action 8, upon completion or due to errors during training, each AIMLE client sends an HFL training notification to the AIMLE server. The notification includes relevant information. The AIMLE client provides its AIMLE service ID for the HFL training operation.

[0047] In action 9, if an error is encountered, the process jumps to action 10. Therefore, if training is successful, the AIMLE server aggregates the model parameters received from the AIMLE client (e.g., by averaging). If the training schedule is not complete (e.g., there are remaining training epochs), the AIMLE server configures the next set of training schedules and repeats actions 3 through 8 for the next training epoch. When the training schedule has been exhausted (e.g., there are no remaining training epochs) and if configured, the AIMLE server initiates an ML model information storage request to store the AI / ML model in the ML repository. At action 10, the AIMLE server sends an ML model training notification to the VAL server.

[0048] Reference Figure 2B This describes an example process for monitoring the performance of an ML model.

[0049] like Figure 2B As shown, at Action 1, the VAL server sends an ML model performance monitoring subscription request to the AIMLE server, requesting assistance in monitoring ML model performance. In some examples, the ML model performance monitoring subscription request may include ML model information and optional AIML service information. At Action 2, the AIMLE server checks whether the VAL server is authorized to execute the ML model performance monitoring request.

[0050] At Action 3, if the VAL server is authorized, the AIMLE server returns a success response to the VAL server; otherwise, the AIMLE server returns a failure response indicating the reason for the failure. Then, at Action 4, the AIMLE server identifies the AIMLE service that is utilizing the requested ML model. Such an AIMLE service can be an ML model training service or an HFL service at the AIMLE server or client. In some examples, the identification of the AIMLE service can be performed by retrieving ML model information from an ML repository using an ML model management process. The AIMLE server then begins monitoring the performance of the AIMLE service (e.g., accuracy, key performance indicators (KPIs) or KPI metrics related to the AIML operation). Action 4 includes receiving information from one or more AIMLE clients performing operations based on the target ML model, containing expected or experienced deviations from the desired performance of the AIMLE service.

[0051] At Action 5, the AIMLE server detects anticipated ML model degradation (e.g., model drift, data drift) based on deviations in the performance of the AIMLE service as indicated in Action 4. At Action 6, based on the anticipated ML model degradation, the AIMLE server can also instruct and execute triggering actions to ensure that AIMLE service requirements are met. Such triggering actions can be adaptation of the AIMLE service, such as training a new ML model for AIMLE by the same or different AIMLE clients, or retraining the ML model by the same or different AIMLE clients; or terminating the AIMLE service and initiating a new AIMLE service using the new ML model. In some cases, Action 6 may involve reselecting the AIMLE client for the AIMLE service.

[0052] At action 7, the AIMLE server notifies the VAL server of the expected ML model degradation, and if requested, notifies the AIMLE service to trigger adaptation.

[0053] Figure 2A and Figure 2B The example process requires some enhancements. For one example, it continuously subscribes the UE client to each training iteration without any performance monitoring applicable to the defined HFL. For another example, in addition to performance monitoring and correction issues, Figure 2A and Figure 2B In the example, the client can still obtain AI / ML model parameters even if it does not subscribe to a specific HFL training iteration, which significantly increases the overhead in communication.

[0054] Based on some example embodiments of this disclosure, an enhanced performance monitoring and adaptation process for HFL training is proposed. Specifically, it provides AIMLE with the ability to proactively monitor and report the performance of collaborating UE clients during HFL training. Furthermore, performance degradation of AIMLE clients can be detected, and an adaptation (correction) method is introduced so that HFL training can be adapted to the aggregation method in such a way that performance-impaired UE clients can improve their performance during the next iteration of HFL training. This is achieved by prioritizing their selection in the next HFL subscription round, and their parameters can be aggregated by the server to increase client gain.

[0055] Reference Figure 3A and Figure 3B Discuss example performance monitoring process, Figure 3A and Figure 3B Signaling streams 300A and 300B for communication according to some embodiments of this disclosure are shown. For discussion purposes, reference will be made to... Figure 1For example, signaling flows 300A and 300B are discussed using device 110, first server 120 and second server 130.

[0056] exist Figure 3A and Figure 3B In some example embodiments, device 110 may be an AI machine learning enabled client, first server 120 may be an AI machine learning enabled server, and second server 130 may be a vertical application layer server.

[0057] like Figure 3A As shown, in operation, the first server 120 sends (310-1) a report request to the device 110 for reporting the performance of a machine learning model, and the device 110 accordingly receives (310-2) the report request. Specifically, the report request includes at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance. It should be noted that in other example embodiments, multiple other parameters that can be used for performance reporting may also be included in the report request.

[0058] Based on the report request, the device evaluates (312) the performance of the machine learning model using local data. The device 110 then sends (314-1) a report response to the report request to the first server 120, and the first server 120 accordingly receives (314-2) a report response relating to (i.e., including) the performance of the evaluated machine learning model. In some example embodiments, in addition to the performance of the evaluated machine learning model, the report response may also include an identifier associated with the device 110, or a service identifier used to train the machine learning model. Additionally, in some example embodiments, training the machine learning model may include lateral federated learning training of the machine learning model. It should be noted that in other example embodiments, several other parameters that may indicate performance information may also be included in the report response.

[0059] For the first server 120, based on the performance of the evaluated machine learning model, the first server 120 can detect (316) performance degradation of the machine learning model.

[0060] In some example embodiments, in response to detecting performance degradation of the machine learning model, the first server 120 may send a configuration request (318-1) to the device 110, and the device 110 may accordingly receive a configuration request (318-2). In some examples, the configuration request may indicate one or more configurations applied to increase the collaboration between the device 110 and at least one other device when training the machine learning model.

[0061] According to some example embodiments of this disclosure, before triggering a performance report, the first server 120 may subscribe to the training of a machine learning model at device 110, as described below.

[0062] In some example embodiments, the first server may send a subscription request (304-1) to device 110 for subscribing to the training of a machine learning model, and device 110 may accordingly receive a subscription request (304-2). In one example, the subscription request may include information about the machine learning model. Alternatively or additionally, in another example, the subscription request may include a requested training iteration of the machine learning model. Alternatively or additionally, in another example, the subscription request may include at least one tracked performance record of the machine learning model. Alternatively or additionally, in another example, the subscription request may include a dataset identifier associated with a dataset used by the first server 120 for the initial training of the machine learning model, or a description of the dataset to be used for the initial training of the machine learning model.

[0063] In response to receiving a subscription request, device 110 can train (306) a machine learning model. The device can then send (308-1) a subscription response to the subscription request to a first server 120, and the first server 120 can accordingly receive (308-2) a subscription response, wherein the subscription response indicates the success or failure of the subscription for training the machine learning model.

[0064] According to some example embodiments of this disclosure, the first server 120 may trigger a performance report in response to a request from the second server 130. For example... Figure 3A As shown, the second server 130 can send a monitoring request (302-1) to the first server 120 to monitor the performance of the machine learning model, and the first server 120 can receive a monitoring request (302-2) accordingly, wherein the monitoring request includes a reference dataset for evaluating the performance of the machine learning model.

[0065] Based on the monitoring request, if the first server 120 has obtained performance information, the first server 120 may send (320-1) a notification to the second server 130 including the performance of the machine learning model evaluated by the first server 120, and the second server 130 may receive (320-2) the notification accordingly.

[0066] In the following text, reference will be made to Figure 3B Let's discuss another example of a performance monitoring process.

[0067] like Figure 3BAs shown, in operation, the first server sends (354-1) a subscription request to device 110 for subscribing to the training of a machine learning model, and device 110 accordingly receives (354-2) the subscription request. In one example, the subscription request may include information about the machine learning model (e.g., the model size of the machine learning model, the model architecture of the machine learning model). Alternatively or additionally, in another example, the subscription request may include a requested training iteration of the machine learning model. Alternatively or additionally, in another example, the subscription request may include at least one tracked performance record of the machine learning model. Alternatively or additionally, in another example, the subscription request may include a dataset identifier associated with the dataset used by the first server 120 for the initial training of the machine learning model, or a description of the dataset to be used for the initial training of the machine learning model.

[0068] Based on the subscription request, device 110 determines (356) whether training of the machine learning model is permitted. Based on this determination, device 110 sends (358-1) a subscription response to the subscription request to first server 120, and first server 120 receives (358-2) a subscription response accordingly. In some example embodiments, the subscription response may include an identifier for a subscription to training the machine learning model.

[0069] According to some example embodiments of this disclosure, the subscription response can indicate the success or failure of the subscription for training the machine learning model. In some example embodiments, if the subscription response indicates that the subscription for training the machine learning model was successful, a performance report can be configured by the first server 120. Specifically, the first server 120 can send (360-1) a report request to the device 110 for reporting the performance of the machine learning model, and the device 110 can accordingly receive (360-2) the report request.

[0070] Specifically, a report request may include at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of its performance. It should be noted that in other example embodiments, multiple other parameters that can be used for performance reporting may be included in the report request.

[0071] In some example embodiments, the reporting request may be sent based on a monitoring request from a second server 130. For example... Figure 3B As shown, the second server 130 can send a monitoring request (344-1) to the device 110 for monitoring the performance of the machine learning model, and the device 110 can receive the monitoring request accordingly. Based on the monitoring request, the first server 120 can send a report request to the device 110.

[0072] In response to a report request, device 110 may evaluate (362) the performance of a machine learning model based on the report request using its local data. The local data of device 110 may be stored within device 110 (but not obtained from the network) and therefore maintained locally by device 110. The local data may include any relevant data that can be used to evaluate the performance of the machine learning model. The types of the local data may depend on the capabilities of the machine learning model. Device 110 then sends (364-1) a report response to the report request to first server 120, and first server 120 accordingly receives (364-2) a report response relating to the performance of the evaluated machine learning model (i.e., including the evaluation of that performance). In some example embodiments, in addition to the performance of the evaluated machine learning model included in the performance report of the machine learning model, the report response may also include an identifier associated with device 110 or a service identifier used to train the machine learning model. In some example embodiments, the training of the machine learning model may include lateral federated learning training of the machine learning model. It should be noted that in other example embodiments, multiple other parameters that may indicate performance information may also be included in the report response.

[0073] For the first server 120, based on the performance of the evaluated machine learning model, the first server 120 can detect (366) performance degradation of the machine learning model.

[0074] In some example embodiments, when the first server 120 detects (366) performance degradation of the machine learning model, the first server 120 may send (368-1) a configuration request to the device 110, and the device 110 may accordingly receive (368-2) a configuration request. In one example, the configuration request may indicate one or more configurations applied to increase the collaboration of the device 110 with at least one other device during the training of the machine learning model (also referred to as at least one feature required to increase the contribution of the device 110 to the training of the machine learning model). In another example, the configuration request may indicate an identifier of the requester of the configuration request and / or quantitative information for increasing the contribution of the device 110. In yet another example, the configuration request may indicate the number of training iterations used to prioritize the selection of the device 110.

[0075] Furthermore, if the configuration request indicates at least one function required to increase the contribution of device 110 to the training of the machine learning model, after receiving the configuration request, device 110 can determine at least one configuration from one or more configurations for training the machine learning model. Device 110 can then send (370-1) a configuration response to the configuration request to first server 120, and first server 120 can accordingly receive (370-2) a configuration response, wherein the configuration response can indicate the determined at least one configuration. Note that the training of the machine learning model is performed using the determined at least one configuration.

[0076] As described above, the second server 130 can transmit monitoring requests to the first server 120. Therefore, the first server 120 needs to respond to the monitoring requests.

[0077] Specifically, the first server 120 may receive parameters of a machine learning model trained by a corresponding device 110 among a plurality of devices. Then, the first server 120 may obtain (376) an aggregated machine learning model based on the received parameters. Next, the first server 120 may evaluate (378) the performance of the aggregated machine learning model using a reference dataset. Finally, the first server 120 may send (380-1) a notification including the evaluated performance of the aggregated machine learning model to the second server 130, and the second server 130 may accordingly receive (380-2) the notification.

[0078] According to some example embodiments of this disclosure, training may be triggered by a second server 130, as described below.

[0079] As shown in the figure, the second server 130 can send (342-1) a training request for training a machine learning model to the device 110, and the device 110 can correspondingly receive (342-2) the training request. In one example, the training request may include at least one of a baseline dataset to be used by the first server 120 for initial training of the machine learning model or a dataset identifier associated with at least one dataset owned by the second server 130. In response to the training request, the device 110 can perform (346) initial training of the machine learning model using the baseline dataset or at least one of the at least one dataset owned by the second server 130.

[0080] To better understand the reference Figure 3A and Figure 3B The example process discussed will refer to Figure 4 Further examples and embodiments are discussed below. It should be noted that the following examples and embodiments are for illustrative purposes only and do not imply any limitation.

[0081] Figure 4An example signaling flow 400 for performance monitoring during a horizontal federated training cycle 410, according to some example embodiments of this disclosure, is illustrated. Signaling flow 400 relates to device(s) 110, a first server 120, and a second server 130. For better understanding, in Figure 4 In the examples, the VAL client / AIMLE client is used as an example of device 110, the AIMLE server is used as an example of first server 120, and the VAL server is used as an example of second server 130.

[0082] At action 1, the AIMLE server can receive ML model training requests from the VAL server (e.g., specifying the training type as HFL). The AIMLE server can then identify HFL as the training type and send a response message to the VAL server.

[0083] In some example embodiments, an augmented table is introduced (which shows the example information elements (IEs) included in the ML model training request), for example, indicating the HFL baseline training dataset, where the baseline dataset can be used in the following process (e.g., action 5). See reference... Figure 3A The training request discussed may include at least one of a baseline dataset to be used by the first server for initial training of a machine learning model, or a dataset identifier associated with at least one dataset owned by the second server. An example of an augmentation table is shown below.

[0084]

[0085] At action 2, the VAL server can send an ML model performance monitoring subscription request to the AIMLE server (i.e., refer to...). Figure 3A and Figure 3B (The monitoring request is discussed). Based on some examples in this disclosure, the performance monitoring subscription request can be enhanced to specifically assist in performance monitoring during the HFL training process, such as by including the HFL server baseline validation dataset. This dataset is used in the next process during performance evaluation. An example of the enhanced table is shown below.

[0086]

[0087] At Action 3, the AIMLE server can perform AIMLE client selection using either the AIMLE client selection criteria or the AIMLE client list provided at Action 1. If the AIMLE client selection criteria are provided at Action 1, the AIMLE server can continuously monitor and select AIMLE clients for HFL training. If a list of AIMLE clients is provided at Action 1, the AIMLE server can select the provided AIMLE clients for HFL training.

[0088] Based on the AIMLE client set (determined at Action 3), at Action 4, the AIMLE server can check the capabilities and participation of the selected AIMLE clients in HFL training. At Action 5, the AIMLE server can perform initial training of the global HFL model using the baseline training dataset provided at Action 1.

[0089] At action 6, the AIMLE server can configure training scheduling for each AIMLE client and can send HFL training subscription requests (e.g., see reference). Figure 3A and Figure 3B The subscription request is discussed. An enhancement table (which shows example information elements (IEs) included in the HFL training subscription request) is introduced to specifically assist UE client performance monitoring during the HFL training process. An example of the enhancement table is shown below.

[0090]

[0091] In Action 7, each AIMLE client can send an HFL training subscription response (e.g., see reference). Figure 3A and Figure 3B The discussion includes the subscription response and the HFL subscription ID. In some example embodiments, if the AIMLE client cannot authorize a subscription (e.g., cannot perform training, or its performance is compromised), the AIMLE client can send a response with a failure status, regardless of the remaining actions within that HFL training loop 410.

[0092] At action 8, the AIMLE server can send performance report requests only to the AIMLE clients(s) that subscribed to HFL training in action 7 with a "success" response (e.g., refer to...). Figure 3A and Figure 3B (Discussion of report requests). By doing so, (multiple) AIMLE clients can update (multiple) related models and run (multiple) related models on the corresponding local data. Performance report requests may include example information elements (IEs) as shown in the table below.

[0093]

[0094] In Action 9, each AIMLE client can evaluate the performance of the received aggregated global model on local data based on the parameters and metrics received in Action 8.

[0095] In Action 10, each AIMLE client can send a performance report of the aggregated global model on its local data as a response message to the AIMLE server. Performance report response messages (such as, see reference...) Figure 3A and Figure 3B The report response to the discussion includes example information elements (IE) as shown in the table below.

[0096]

[0097] At Action 11, the AIMLE server receives performance evaluations for each AIMLE client in the current subscription and updates its corresponding performance tracker. It also detects potential performance degradation in each AIMLE client.

[0098] At action 12, the AIMLE server can send requests for increased training collaboration configuration only to AIMLE clients with degraded performance records (e.g., reference...). Figure 3A and Figure 3B (Discussion of configuration requests). By doing so, the AIMLE client can choose how to increase its collaboration in HFL training in a way that enhances its performance evaluation. The configuration request for increased training collaboration includes example information elements (IEs) as described in the table below.

[0099]

[0100] At action 13, each AIMLE client experiencing performance degradation selects the increased collaboration option based on its capacity and resource constraints, and they send an increased training collaboration configuration response message, as described in the table below.

[0101]

[0102] At Action 14, for a specified number of samples, each AIMLE client performs local training on its local data using the global aggregated model parameters received in Action 8, along with the quantitative information and required functionality from Action 12. At Action 15, upon completion or due to errors during training, each AIMLE client may send an HFL training notification to the AIMLE server.

[0103] At action 16, the AIMLE server can receive locally trained AI / ML model parameters from each client (either a degraded or high-performance client), and it performs global HFL model aggregation. At action 17, the AIMLE server can evaluate the performance of the globally aggregated model on the validation dataset received from the VAL server, as in action 2.

[0104] At action 18, the AIMLE server can notify the VAL server of the global aggregated model's performance. If the HFL training iteration is not yet complete, the process continues to action 6. At actions 19 and 20, the AIMLE server initiates an ML model information storage request to store the AI / ML model in the ML repository, and the AIMLE server sends an ML model training notification to the VAL server.

[0105] Figure 5 A flowchart of an example method 500 implemented at a device according to some example embodiments of the present disclosure is shown. For the purposes of discussion, [the following will be discussed]. Figure 1 The angle description method of the device 110 in the middle is 500.

[0106] At box 510, the apparatus receives from a first server a subscription request for subscribing to the training of a machine learning model, the subscription request including at least one of the following: the model size of the machine learning model, the model architecture of the machine learning model, the requested training iterations of the machine learning model, or at least one tracked performance record of the machine learning model.

[0107] At box 520, the device determines whether to allow training of the machine learning model based on the subscription request.

[0108] At box 530, the device sends a subscription response to the first server based on this determination, the subscription response indicating the success or failure of the subscription for training the machine learning model.

[0109] In some example embodiments, the subscription request may also include at least one of the following: a dataset identifier associated with the dataset used by the first server for the initial training of the machine learning model, or a dataset description for the initial training of the machine learning model.

[0110] In some example implementations, the subscription response may include an identifier for a subscription to the training of a machine learning model.

[0111] In some example embodiments, method 500 may further include: receiving a report request from a first server for reporting the performance of a machine learning model; evaluating the performance of the machine learning model using local data of the device based on the report request; and sending a report response to the report request to the first server, the report response relating to the evaluated performance of the machine learning model.

[0112] In some example embodiments, a report request may include at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance.

[0113] In some example embodiments, the report response may include at least one of the following: an identifier associated with the device, a service identifier used to train a machine learning model, or a performance report of the machine learning model.

[0114] In some example embodiments, method 500 may further include: receiving a configuration request from a first server, the configuration request indicating one or more configurations to be applied to increase the collaboration between the device and at least one other device in the training of a machine learning model; training the machine learning model based on the configuration request; and sending a notification to the first server including parameters of the trained machine learning model.

[0115] In some example embodiments, a configuration request may include at least one of the following: an identifier of the requester of the configuration request, at least one function required to increase the device's contribution to the training of the machine learning model, quantitative information for increasing the device's contribution, or the number of training iterations for prioritizing the selection of the device.

[0116] In some example embodiments, method 500 may further include: in response to receiving a configuration request, determining at least one configuration from one or more configurations for training a machine learning model; and sending a configuration response to the configuration request to a first server, the configuration response indicating the determined at least one configuration, wherein training of the machine learning model uses the determined at least one configuration.

[0117] In some example embodiments, training a machine learning model may include horizontal federated learning training of the machine learning model.

[0118] In some example embodiments, the device may include an AI machine learning enabled client, and the first server may include an AI machine learning enabled server.

[0119] Figure 6 A flowchart of an example method 600 implemented at a first server according to some example embodiments of the present disclosure is shown. For the purposes of discussion, [the following will be discussed]. Figure 1 The first server in the 120 angle description method 600.

[0120] At box 610, the first server sends a subscription request to the device for subscribing to the training of a machine learning model, the subscription request including at least one of the following: the model size of the machine learning model, the model architecture of the machine learning model, the requested training iterations for training the machine learning model, or at least one tracked performance record of the machine learning model.

[0121] At box 620, the first server receives a subscription response from the device in response to a subscription request, which indicates whether the subscription for training the machine learning model was successful or failed.

[0122] In some example embodiments, the subscription request may also include at least one of the following: a dataset identifier associated with the dataset used by the first server for the initial training of the machine learning model, or a dataset description for the initial training of the machine learning model.

[0123] In some example implementations, the subscription response may include an identifier for a subscription to the training of a machine learning model.

[0124] In some example embodiments, method 600 may further include: receiving a training request from a second server for training a machine learning model, the training request including at least one of a baseline dataset to be used by the first server for initial training of the machine learning model or a dataset identifier associated with at least one dataset owned by the second server; and performing initial training of the machine learning model based on the training request by using the baseline dataset or at least one of the at least one dataset owned by the second server.

[0125] In some example embodiments, method 600 may further include: sending a report request to the device for reporting the performance of a machine learning model; receiving a report response from the device in relation to the performance of the machine learning model evaluated by the device; and detecting performance degradation of the machine learning model based on the performance of the evaluated machine learning model.

[0126] In some example embodiments, a report request may include at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance.

[0127] In some example embodiments, the report response may include at least one of the following: an identifier associated with the device, a service identifier used to train a machine learning model, or a performance report of the machine learning model.

[0128] In some example embodiments, method 600 may further include: in response to detected performance degradation of the machine learning model, sending a configuration request to the device, the configuration request indicating one or more configurations to be applied to increase the collaboration between the device and at least one other device during the training of the machine learning model.

[0129] In some example embodiments, a configuration request may include at least one of the following: an identifier of the requester of the configuration request, at least one function required to increase the device's contribution to the training of the machine learning model, quantitative information for increasing the device's contribution, or the number of training iterations for prioritizing the selection of the device.

[0130] In some example embodiments, method 600 may further include: receiving a configuration response from the device in response to a configuration request, the configuration response indicating at least one configuration determined from one or more configurations used for training a machine learning model.

[0131] In some example embodiments, method 600 may further include: receiving a monitoring request from a second server for monitoring the performance of a machine learning model, the monitoring request including a reference dataset for evaluating the performance of the machine learning model, wherein sending a report request to the device is based on the monitoring request.

[0132] In some example embodiments, method 600 may further include: receiving parameters of a machine learning model trained by a corresponding device among a plurality of devices; obtaining an aggregated machine learning model based on the received parameters; evaluating the performance of the aggregated machine learning model using a reference dataset; and sending a notification to a second server including the performance of the evaluated aggregated machine learning model.

[0133] In some example embodiments, the second server may include a vertical application layer server.

[0134] In some example embodiments, training a machine learning model may include horizontal federated learning training of the machine learning model.

[0135] In some example embodiments, the device may include an AI machine learning enabled client, and the first server may include an AI machine learning enabled server.

[0136] Figure 7 A flowchart of an example method 700 implemented at a second server according to some example embodiments of the present disclosure is shown. For discussion purposes, [the following will be discussed]. Figure 1 The second server 130 in the method of angle description 700.

[0137] At box 710, the second server sends a training request to the first server for training a machine learning model. The training request includes at least one of a baseline dataset to be used by the first server for initial training of the machine learning model or a dataset identifier associated with at least one dataset owned by the second server.

[0138] In some example embodiments, method 700 may further include: sending a monitoring request to a first server for monitoring the performance of a machine learning model, the monitoring request including a reference dataset for evaluating the performance of the machine learning model; and receiving a notification from the first server including the performance of the machine learning model evaluated by the first server using the reference dataset.

[0139] In some example embodiments, training a machine learning model may include horizontal federated learning training of the machine learning model.

[0140] In some example embodiments, the first server may include an artificial intelligence machine learning enabling server, and the second server may include a vertical application layer server.

[0141] Figure 8 A flowchart of an example method 800 implemented at a device according to some example embodiments of the present disclosure is shown. For the purposes of discussion, [the following will be discussed]. Figure 1 The angle description method of device 110 in the middle is 800.

[0142] At box 810, the apparatus receives a report request from a first server for reporting the performance of a machine learning model. The report request includes at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance.

[0143] At box 820, the device evaluates the performance of the machine learning model using the device’s local data based on a report request.

[0144] At box 830, the device sends a report response to a first server in response to a report request, which relates to the performance of the evaluated machine learning model.

[0145] In some example embodiments, the report response may include at least one of the following: an identifier associated with the device, a service identifier used for training a machine learning model, or a performance report of the machine learning model.

[0146] In some example embodiments, method 800 may further include: receiving a configuration request from a first server, the configuration request indicating one or more configurations to be applied to increase the collaboration between the device and at least one other device during the training of a machine learning model.

[0147] In some example embodiments, method 800 may further include: receiving from a first server a subscription request for subscribing to training a machine learning model, the subscription request including at least one of the following: information about the machine learning model, a requested training iteration of the machine learning model, or at least one tracked performance record of the machine learning model; training the machine learning model in response to receiving the subscription request; and sending a subscription response to the first server in response to the subscription request, the subscription response indicating the success or failure of the subscription to training the machine learning model.

[0148] In some example embodiments, the subscription request may also include at least one of the following: a dataset identifier associated with the dataset used by the first server for the initial training of the machine learning model, or a dataset description for the initial training of the machine learning model.

[0149] In some example embodiments, training a machine learning model may include horizontal federated learning training of the machine learning model.

[0150] In some example embodiments, the device may include an AI machine learning enabled client, and the first server may include an AI machine learning enabled server.

[0151] Figure 9 A flowchart of an example method 900 implemented at a first server according to some example embodiments of the present disclosure is shown. For discussion purposes, [the following will be discussed]. Figure 1 The first server in the 120 angle description method is 900.

[0152] At box 910, the first server sends a report request to the device for reporting the performance of a machine learning model. The report request includes at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of performance.

[0153] At box 920, the first server receives a report response from the device in response to a report request, which relates to the performance of the machine learning model evaluated by the device.

[0154] At box 930, the first server detects performance degradation of the machine learning model based on the performance of the evaluated machine learning model.

[0155] In some example embodiments, the report response may include at least one of the following: an identifier associated with the device, a service identifier used for training a machine learning model, or a performance report of the machine learning model.

[0156] In some example embodiments, method 900 may further include: in response to detected performance degradation of the machine learning model, sending a configuration request to the device, the configuration request indicating one or more configurations to be applied to increase the collaboration between the device and at least one other device during the training of the machine learning model.

[0157] In some example embodiments, method 900 may further include: receiving from a second server a monitoring request for monitoring the performance of a machine learning model, the monitoring request including a reference dataset for evaluating the performance of the machine learning model; and in response to the monitoring request, sending to the second server a notification including the performance of the machine learning model evaluated by the first server.

[0158] In some example embodiments, the second server may include a vertical application layer server.

[0159] In some example embodiments, method 900 may further include: sending a subscription request to a device for subscribing to training of a machine learning model, the subscription request including at least one of the following: information about the machine learning model, a requested training iteration of the machine learning model, or at least one tracked performance record of the machine learning model; and receiving a subscription response from the device in response to the subscription request, the subscription response indicating the success or failure of the subscription to the training of the machine learning model.

[0160] In some example embodiments, the subscription request may also include at least one of the following: a dataset identifier associated with the dataset used by the first server for the initial training of a machine learning model, or a dataset description to be used for the initial training of a machine learning model.

[0161] In some example embodiments, training a machine learning model may include horizontal federated learning training of the machine learning model.

[0162] In some example embodiments, the device may include an AI machine learning enabled client, and the first server may include an AI machine learning enabled server.

[0163] Figure 10 A flowchart of an example method 1000 implemented at a second server according to some example embodiments of the present disclosure is shown. For discussion purposes, [the following will be discussed]. Figure 1 The second server 130 in the method of angle description 1000.

[0164] At box 1010, the second server sends a monitoring request to the first server to monitor the performance of the machine learning model. The monitoring request includes a reference dataset for evaluating the performance of the machine learning model.

[0165] At box 1020, the second server receives a notification from the first server, which includes the performance of the machine learning model evaluated by the first server using a reference dataset.

[0166] In some example embodiments, the first server may include an artificial intelligence machine learning enabling server, and the second server may include a vertical application layer server.

[0167] In some example embodiments, any of the means capable of performing methods 500 and 800 (e.g., Figure 1The device 110 may include components for performing corresponding operations of methods 500 and 800 and any embodiments thereof. The components may be implemented in any suitable form. For example, the components may be implemented in a circuit or software module. The components may be implemented as or included in... Figure 1 In device 110.

[0168] In some example embodiments, any of the first servers in methods 600 and 900 (e.g., Figure 1 The first server 120 may include components for performing corresponding operations of methods 600 and 900 and any embodiments thereof. The components may be implemented in any suitable form. For example, the components may be implemented in a circuit or software module. The first server may be implemented as or included in Figure 1 The first server in the series is 120.

[0169] In some example embodiments, any of the second servers in methods 700 and 1000 (e.g., Figure 1 The second server 130 may include components for performing corresponding operations of methods 700 and 1000 and any embodiments thereof. The components may be implemented in any suitable form. For example, the components may be implemented in a circuit or software module. The second server may be implemented as or included in... Figure 1 In the second server 130.

[0170] Figure 11 This is a simplified block diagram of a device 1100 suitable for implementing exemplary embodiments of the present disclosure. The device 1100 can be provided to implement a communication device, such as... Figure 1 The device 110 shown is either a first server 120 or a second server 130. As shown, the device 1100 includes one or more processors 1110, one or more memories 1120 coupled to the processors 1110, and one or more communication modules 1140 coupled to the processors 1110.

[0171] Communication module 1140 is used for bidirectional communication. Communication module 1140 has one or more communication interfaces to facilitate communication with one or more other modules or devices. The communication interface can represent any interface required for communication with other network elements. In some example embodiments, communication module 1140 may include at least one antenna.

[0172] As a non-limiting example, processor 1110 can be any type suitable for a local technology network and can include one or more of the following: general-purpose computer, special-purpose computer, microprocessor, digital signal processor (DSP), and processor based on a multi-core processor architecture. Device 1100 can have multiple processors, such as application-specific integrated circuit chips that are time-dependent on a clock of a synchronous main processor.

[0173] Memory 1120 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 1124, electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), optical disc, laser disc, and other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, random access memory (RAM) 1122 and other volatile memories that will not be maintained during power outages.

[0174] Computer program 1130 includes computer-executable instructions that are executed by an associated processor 1110. The instructions of program 1130 may include instructions for performing operations / actions of some example embodiments of this disclosure. Program 1130 may be stored in memory (e.g., ROM 1124). Processor 1110 can perform any suitable actions and processes by loading program 1130 into RAM 1122.

[0175] Example embodiments of this disclosure can be implemented by means of program 1130, enabling device 1100 to perform as described in the reference. Figures 3A to 10 Any process discussed in this disclosure. Exemplary embodiments of this disclosure may also be implemented by hardware or by a combination of software and hardware.

[0176] In some example embodiments, program 1130 may be tangibly included in a computer-readable medium, which may be included in device 1100 (such as in memory 1120) or in other storage devices accessible by device 1100. Device 1100 may load program 1130 from the computer-readable medium into RAM 1122 for execution. In some example embodiments, the computer-readable medium may include any type of non-transitory storage medium, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc. As used herein, the term "non-transitory" is a limitation of the medium itself (i.e., tangible, not tactile), rather than a limitation of the persistence of data storage (e.g., RAM versus ROM).

[0177] Figure 12An example of a computer-readable medium 1200 is shown, which may be in the form of a CD, DVD, or other optical storage disc. The computer-readable medium 1200 has a program 1130 stored thereon.

[0178] In general, the various embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, and others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof, as non-limiting examples.

[0179] Some exemplary embodiments of this disclosure also provide at least one computer program product tangibly stored on a computer-readable medium, such as a non-transitory computer-readable medium. The computer program product includes computer-executable instructions, such as those included in a program module, which are executed in a device on a target physical or virtual processor to perform any of the methods described above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform a particular task or implement a particular abstract data type. In various embodiments, the functionality of a program module can be combined or split among program modules as needed. The machine-executable instructions for a program module can execute within a local or distributed device. In a distributed device, the program module can reside on both local and remote storage media.

[0180] Program code for performing the methods of this disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that, when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be performed. The program code may be executed entirely on a machine, partially on a machine, as a stand-alone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0181] In the context of this disclosure, computer program code or related data may be carried by any suitable carrier wave to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carrier waves include signals, computer-readable media, etc.

[0182] Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable media can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples of computer-readable storage media will include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0183] Furthermore, although operations are described in a specific order, this should not be construed as requiring that such operations be performed in the specific order shown or sequentially, or requiring that all shown operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure, but rather as a description of features that may be specific to particular embodiments. Unless explicitly stated otherwise, certain features described in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, unless explicitly stated otherwise, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0184] Although this disclosure has been described in language specific to structural features and / or methodological actions, it should be understood that the disclosure as defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms for implementing the claims.

Claims

1. A device for communication, comprising: At least one processor; as well as At least one memory storing instructions that, when executed by the at least one processor, cause the device to at least: Receive a report request from a first server for reporting the performance of a machine learning model, the report request including at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of the performance; Based on the report request, the performance of the machine learning model is evaluated using local data from the device; as well as A report response to the report request is sent to the first server, the report response relating to the performance of the evaluated machine learning model.

2. The apparatus of claim 1, wherein the report response includes at least one of the following: an identifier associated with the apparatus, a service identifier for training the machine learning model, or a performance report of the machine learning model.

3. The apparatus according to claim 1 or 2, wherein the apparatus is further configured to: A configuration request is received from the first server, the configuration request indicating one or more configurations to be applied to increase the collaboration between the device and at least one other device during the training of the machine learning model.

4. The apparatus according to claim 1 or 2, wherein the apparatus is further configured to: The first server receives a subscription request for subscribing to the training of a machine learning model, the subscription request including at least one of the following: information about the machine learning model, a requested training iteration of the training of the machine learning model, or at least one tracked performance record of the machine learning model; In response to receiving the subscription request, the machine learning model is trained; as well as A subscription response to the subscription request is sent to the first server, the subscription response indicating the success or failure of the subscription for the training of the machine learning model.

5. The apparatus of claim 4, wherein the subscription request further comprises at least one of the following: A dataset identifier associated with the dataset used by the first server for the initial training of the machine learning model, or A description of the dataset used for the initial training of the machine learning model.

6. The apparatus of claim 2, wherein the training of the machine learning model comprises lateral federated learning training of the machine learning model.

7. The apparatus of claim 1 or 2, wherein the apparatus includes an AI machine learning enabled client, and the first server includes an AI machine learning enabled server.

8. The apparatus according to claim 1 or 2, wherein the local data of the apparatus is stored in the apparatus.

9. A first server, comprising: At least one processor; as well as At least one memory storing instructions that, when executed by the at least one processor, cause the first server to at least: Send a report request to the device for reporting the performance of a machine learning model, the report request including at least one of the following: an identifier of the requester of the report request, parameters of the machine learning model, or at least one metric of the performance; The device receives a report response in response to the report request, the report response being related to the performance of the machine learning model evaluated by the device. as well as Performance degradation of the machine learning model is detected based on the evaluated performance of the machine learning model.

10. The first server of claim 9, wherein the report response includes at least one of the following: an identifier associated with the device, a service identifier for training the machine learning model, or a performance report of the machine learning model.

11. The first server according to claim 9 or 10, wherein the first server is further configured to: In response to the detected performance degradation of the machine learning model, a configuration request is sent to the device, the configuration request indicating one or more configurations to be applied to increase the collaboration between the device and at least one other device during the training of the machine learning model.

12. The first server according to claim 9 or 10, wherein the first server is further configured to: Receive a monitoring request from a second server for monitoring the performance of the machine learning model, the monitoring request including a reference dataset for evaluating the performance of the machine learning model; and In response to the monitoring request, a notification including the performance of the machine learning model evaluated by the first server is sent to the second server.

13. The first server according to claim 12, wherein the second server comprises a vertical application layer server.

14. The first server according to claim 9 or 10, wherein the first server is further configured to: Sending a subscription request to the device for subscribing to the training of a machine learning model, the subscription request including at least one of the following: information about the machine learning model, a requested training iteration of the machine learning model's training, or at least one tracked performance record of the machine learning model; and The device receives a subscription response in response to the subscription request, the subscription response indicating the success or failure of the subscription for the training of the machine learning model.

15. The first server of claim 14, wherein the subscription request further comprises at least one of the following: A dataset identifier associated with the dataset used by the first server for the initial training of the machine learning model, or A description of the dataset used for the initial training of the machine learning model.

16. The first server of claim 14, wherein the training of the machine learning model comprises lateral federated learning training of the machine learning model.

17. The first server of claim 9 or 10, wherein the apparatus includes an AI machine learning enabled client, and the first server includes an AI machine learning enabled server.

18. A second server, comprising: At least one processor; as well as At least one memory storing instructions that, when executed by the at least one processor, cause the second server to at least: A monitoring request for monitoring the performance of a machine learning model is sent to a first server, the monitoring request including a reference dataset for evaluating the performance of the machine learning model; as well as A notification is received from the first server, the notification including the performance of the machine learning model evaluated by the first server using the reference dataset.

19. The second server of claim 18, wherein the first server includes an artificial intelligence machine learning enabled server, and the second server includes a vertical application layer server.