Methods for federated learning

Abstract

The present disclosure relates to methods for federated learning. A method, apparatus and computer program are described comprising: receiving an input; and determining a classification of the input using a machine learning model, the machine learning model comprising a local portion and a portion of collaborative learning, the determining comprising: determining extracted features of the input using a feature extractor of the portion of collaborative learning, the feature extractor caused to extract features of the input; determining a set of similarity scores using a prototype layer of the local portion of the model, the prototype layer caused to determine similarities between the extracted features of the input and a set of trained prototypes of the prototype layer; and determining the classification of the input using a prototype classifier of the local portion of the model, the prototype classifier caused to determine the classification based on the similarity scores.

Description

Technical Field

[0001] Example embodiments may relate to apparatus and methods for collaborative machine learning. Background Technology

[0002] In federated learning (FL) methods, machine learning steps are performed decentralizedly across multiple clients, each of which can train a model based on its own local dataset(s). The results of this decentralized learning can be aggregated across multiple clients to obtain an updated model. Models trained in federated learning methods can be personalized to some extent for specific clients. There is interest in providing federated learning methods that strike a balance between generalization ability, personalization, and privacy. Summary of the Invention

[0003] The scope of protection sought by the various embodiments of the present invention is stated in the independent claims. Embodiments and features described in this specification that are not within the scope of the independent claims, if any, are to be interpreted as examples useful for understanding the various embodiments of the invention.

[0004] A first aspect provides an apparatus comprising: components for training a machine learning model, the training including: training a collaborative learning portion of the model by training a first set of trainable parameters to obtain local updates to the first set of trainable parameters, the first set of trainable parameters parameterizing the collaborative learning portion, wherein the collaborative learning portion includes a feature extractor configured to extract features from an input; and training a local portion of the model by training a second set of trainable parameters to obtain local updates to the second set of trainable parameters, the second set of trainable parameters parameterizing the local portion, wherein the local portion includes a local classifier configured to receive extracted features and output a classification of the input, the local classifier including: a prototype layer configured to determine similarity scores between the extracted features and a plurality of trainable prototypes; and a prototype classifier configured to receive similarity scores and output a classification of the input based on the similarity scores.

[0005] In some examples, the apparatus further includes: components for sending an instruction to a device participating in collaborative learning for a local update of a first set of trainable parameters; components for updating a second set of trainable parameters of a machine learning model based on the obtained local update of the second set of trainable parameters; components for receiving an instruction from the device participating in collaborative learning for an update of the first set of trainable parameters; and components for updating the first set of trainable parameters based on the indicated update.

[0006] In some examples, the device also includes components for receiving the feature extractor of the collaborative learning portion of the model and the local classifier of the local portion of the model.

[0007] In some examples, the components used to train the machine learning model are configured such that, at least in the initial rounds of training, a first set of trainable parameters is trained without training a second set of trainable parameters.

[0008] In some examples, the collaborative learning portion of the machine learning model also includes a collaborative learning classifier, which is configured to receive representations of the input data from the feature extractor and output a classification of the input data; and The machine learning model also includes a joint predictor, which is configured to determine the classification based on the classification output by the local classifier and the classifier learned collaboratively.

[0009] In some examples, the components used to train the machine learning model are configured to alternate between training a feature extractor and a local classifier on a first dataset, and training a feature extractor and a collaboratively learned classifier.

[0010] In some examples, the feature extractor for the collaborative learning part is configured to extract features by defining multiple regions of the input for the input and outputting multiple feature vectors, where the corresponding feature vector among the multiple feature vectors represents the features of the data from the corresponding defined region of the input.

[0011] In some examples, the prototype layer is configured to determine the similarity score between the feature vector of the extracted features and the trainable prototype.

[0012] In some examples, the trainable prototype includes corresponding trainable vectors in the same vector space as the multiple feature vectors. In some examples, the trainable prototype includes a corresponding trainable radius, and determining the similarity between a feature vector and a trainable prototype includes determining a similarity score based at least in part on whether the feature vector lies within the trainable radius of the trainable vectors of the trainable prototype.

[0013] In some examples, the classification includes a probability distribution, which comprises multiple probabilities indicating the probability that the input corresponds to a specific category in a set of categories.

[0014] In some examples, the input includes image data.

[0015] In some examples, the input includes sensor data.

[0016] The second aspect provides a method comprising: training a machine learning model, including: training a collaborative learning portion of the model by training a first set of trainable parameters to obtain local updates of the first set of trainable parameters, the first set of trainable parameters parameterizing the collaborative learning portion, wherein the collaborative learning portion includes a feature extractor configured to extract features; and training a local portion of the model by training a second set of trainable parameters to obtain local updates of the second set of trainable parameters, the second set of trainable parameters parameterizing the local portion, wherein the local portion includes a local classifier configured to receive extracted features and output a classification of the input, the local classifier including: a prototype layer configured to determine similarity scores between the extracted features and a plurality of trainable prototypes; and a prototype classifier configured to receive similarity scores and output a classification of the input based on the similarity scores.

[0017] In some examples, the method further includes: sending an instruction to a device participating in collaborative learning for a local update of a first set of trainable parameters; updating a second set of trainable parameters of a machine learning model based on the obtained local update of a second set of trainable parameters; receiving an instruction from the device participating in collaborative learning for an update of the first set of trainable parameters; and updating the first set of trainable parameters based on the indicated update.

[0018] In some examples, the method also includes a feature extractor that receives the collaboratively learned portions of the model and a local classifier for the local portions of the model.

[0019] In some examples, training a machine learning model involves training a first set of trainable parameters without training a second set of trainable parameters, at least in the initial rounds of training.

[0020] In some examples, the collaborative learning portion of the machine learning model also includes a collaborative learning classifier, which is configured to receive representations of the input data from the feature extractor and output a classification of the input data; and The machine learning model also includes a joint predictor, which is configured to determine the classification based on the classification output by the local classifier and the classifier learned collaboratively.

[0021] In some examples, training a machine learning model involves alternating between training a feature extractor and a local classifier on a first dataset, and training a feature extractor and a collaboratively learned classifier.

[0022] In some examples, the feature extractor for the collaborative learning part is configured to extract features by defining multiple regions of the input for the input and outputting multiple feature vectors, where the corresponding feature vector among the multiple feature vectors represents the features of the data from the corresponding defined region of the input.

[0023] In some examples, the prototype layer is configured to determine the similarity score between the feature vector of the extracted features and the trainable prototype.

[0024] In some examples, the trainable prototype includes corresponding trainable vectors in the same vector space as the multiple feature vectors. In some examples, the trainable prototype includes a corresponding trainable radius, and determining the similarity between the feature vector and the trainable prototype includes determining a similarity score based at least in part on whether the feature vector lies within the trainable radius of the trainable vector of the trainable prototype.

[0025] In some examples, the classification includes a probability distribution, which comprises multiple probabilities indicating the probability that the input corresponds to a specific category in a set of categories.

[0026] In some examples, the input includes image data.

[0027] In some examples, the input includes sensor data.

[0028] A third aspect provides a computer program including a set of instructions configured, when executed on a device, to cause the device to perform a method comprising: training a machine learning model, including: training a collaborative learning portion of the model by training a first set of trainable parameters to obtain local updates to the first set of trainable parameters, the first set of trainable parameters parameterizing the collaborative learning portion, wherein the collaborative learning portion includes a feature extractor configured to extract features; and training a local portion of the model by training a second set of trainable parameters to obtain local updates to the second set of trainable parameters, the second set of trainable parameters parameterizing the local portion, wherein the local portion includes a local classifier configured to receive extracted features and output a classification of the input, the local classifier including: a prototype layer configured to determine similarity scores between the extracted features and a plurality of trainable prototypes; and a prototype classifier configured to receive similarity scores and output a classification of the input based on the similarity scores.

[0029] In some example embodiments, the third aspect may include any other features mentioned in relation to the method of the second aspect.

[0030] A fourth aspect provides a non-transitory computer-readable medium having computer-readable code stored thereon, which, when executed by at least one processor, causes at least one processor to perform a method comprising: training a machine learning model, including: training a collaborative learning portion of the model by training a first set of trainable parameters to obtain local updates to the first set of trainable parameters, the first set of trainable parameters parameterizing the collaborative learning portion, wherein the collaborative learning portion includes a feature extractor configured to extract features; and training a local portion of the model by training a second set of trainable parameters to obtain local updates to the second set of trainable parameters, the second set of trainable parameters parameterizing the local portion, wherein the local portion includes a local classifier configured to receive extracted features and output a classification of the input, the local classifier including: a prototype layer configured to determine similarity scores between the extracted features and a plurality of trainable prototypes; and a prototype classifier configured to receive similarity scores and output a classification of the input based on the similarity scores.

[0031] The fourth aspect may include any other features mentioned in relation to the methods of the second aspect.

[0032] A fifth aspect provides an apparatus having at least one processor and at least one memory having computer-readable code stored thereon, the code, when executed, controlling the at least one processor to perform a method comprising: training a machine learning model, including: training a collaborative learning portion of the model by training a first set of trainable parameters to obtain local updates to the first set of trainable parameters, the first set of trainable parameters parameterizing the collaborative learning portion, wherein the collaborative learning portion includes a feature extractor configured to extract features; and training a local portion of the model by training a second set of trainable parameters to obtain local updates to the second set of trainable parameters, the second set of trainable parameters parameterizing the local portion, wherein the local portion includes a local classifier configured to receive extracted features and output a classification of the input, the local classifier including: a prototype layer configured to determine similarity scores between the extracted features and a plurality of trainable prototypes; and a prototype classifier configured to receive similarity scores and output a classification of the input based on the similarity scores.

[0033] The fifth aspect may include any other features mentioned in relation to the method of the second aspect.

[0034] A sixth aspect provides an apparatus comprising: a component for receiving an input; and a component for determining a classification of the input using a machine learning model, the machine learning model including a local part and a collaborative learning part, the component for determining being configured to: determine extracted features of the input using a feature extractor of the collaborative learning part, the feature extractor being configured to extract features of the input; determine a set of similarity scores using a prototype layer of the local part of the model, the prototype layer being configured to determine similarity between the extracted features of the input and a set of trained prototypes of the prototype layer; and determine a classification of the input using a prototype classifier of the local part of the model, the prototype classifier being configured to determine the classification based on the similarity scores.

[0035] A seventh aspect provides a method comprising: receiving an input; and determining a classification of the input using a machine learning model, the machine learning model including a local part and a collaborative learning part, the determination comprising: using a feature extractor of the collaborative learning part to determine extracted features of the input, the feature extractor being configured to extract features of the input; using a prototype layer of the local part of the model to determine a set of similarity scores, the prototype layer being configured to determine similarity between the extracted features of the input and a set of trained prototypes of the prototype layer; and using a prototype classifier of the local part of the model to determine a classification of the input, the prototype classifier being configured to determine the classification based on the similarity scores.

[0036] An eighth aspect provides a computer program including a set of instructions configured, when executed on a device, to cause the device to perform a method comprising: receiving an input; and determining a classification of the input using a machine learning model, the machine learning model including a local part and a collaborative learning part, the determination comprising: using a feature extractor of the collaborative learning part to determine extracted features of the input, the feature extractor being configured to extract features of the input; using a prototype layer of the local part of the model to determine a set of similarity scores, the prototype layer being configured to determine similarity between the extracted features of the input and a set of trained prototypes of the prototype layer; and using a prototype classifier of the local part of the model to determine a classification of the input, the prototype classifier being configured to determine the classification based on the similarity scores.

[0037] In some example embodiments, the eighth aspect may include any other features mentioned in relation to the method of the seventh aspect.

[0038] A ninth aspect provides a non-transitory computer-readable medium having computer-readable code stored thereon, which, when executed by at least one processor, causes at least one processor to perform a method, the method comprising: receiving an input; and determining a classification of the input using a machine learning model, the machine learning model including a local part and a collaborative learning part, the determination comprising: determining extracted features of the input using a feature extractor of the collaborative learning part, the feature extractor being configured to extract features of the input; determining a set of similarity scores using a prototype layer of the local part of the model, the prototype layer being configured to determine similarity between the extracted features of the input and a set of trained prototypes of the prototype layer; and determining a classification of the input using a prototype classifier of the local part of the model, the prototype classifier being configured to determine the classification based on the similarity scores.

[0039] The ninth aspect may include any other features mentioned in relation to the method of the seventh aspect.

[0040] A tenth aspect provides an apparatus having at least one processor and at least one memory having computer-readable code stored thereon, the computer-readable code, when executed, controlling the at least one processor to perform a method, the method comprising: receiving an input; and determining a classification of the input using a machine learning model, the machine learning model including a local part and a collaborative learning part, the determination comprising: using a feature extractor of the collaborative learning part to determine extracted features of the input, the feature extractor being configured to extract features of the input; using a prototype layer of the local part of the model to determine a set of similarity scores, the prototype layer being configured to determine similarity between the extracted features of the input and a set of trained prototypes of the prototype layer; and using a prototype classifier of the local part of the model to determine a classification of the input, the prototype classifier being configured to determine the classification based on the similarity scores. Attached Figure Description

[0041] Exemplary embodiments will now be described by way of non-limiting examples with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of a communication network according to an example embodiment; Figure 2 and Figure 3 This is a schematic diagram of the example input; Figure 4 This is a schematic diagram of the example model architecture; Figure 5 It is based on the example message sequence diagram; Figure 6 and Figure 7 It is a flowchart based on the example method; Figure 8 This is a block diagram of the components of a system according to an example embodiment; and Figure 9 An example of a tangible medium for storing computer-readable code is shown, which, when run by a computer, can execute methods according to the above example embodiments. Detailed Implementation

[0042] The following embodiments are exemplary. Although this specification may refer to "an," "a," or "some" embodiments in various places throughout the text, this does not necessarily mean that every reference refers to the same embodiment(s), or that a particular feature applies only to a single embodiment. Individual features of different embodiments may also be combined to provide other embodiments. Furthermore, when a particular feature, structure, or characteristic is described in connection with embodiments, those skilled in the art will understand that the feature, structure, or characteristic can be applied in combination with other embodiments, whether explicitly described or not. It should be understood that although terms such as "first," "second," etc., may be used herein to describe various elements, these elements should not be limited by such terms. These terms are used only to distinguish one element from another.

[0043] For the purposes of this disclosure, the phrases "at least one of A or B", "at least one of A and B", and "A and / or B" mean (A), (B), or (A and B). For the purposes of this disclosure, the phrases "A, B and / or C" mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C).

[0044] The described embodiments can be implemented in any combination of wired communication protocols and / or wireless communication protocols and / or networks, such as any of the following radio access technologies (RATs): WiMAX, GSM (2G), GSM EDGE Radio Access Network (GERAN), GRPS, UMTS (3G) (based on basic Wideband Code Division Multiple Access W-CDMA), HSPA, LTE, LTE-Advanced, and enhanced LTE (eLTE), 5G (also known as NR), or any future RAT (such as 6G). Furthermore, communications in the communication network can utilize any suitable wireless communication technology, including but not limited to: Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Frequency Division Duplex (FDD), Time Division Duplex (TDD), Multiple Input Multiple Output (MIMO), Orthogonal Frequency Division Multiplexing (OFDM), and / or Discrete Fourier Transform Spread Spectrum OFDM (DFT-s-OFDM). The implementation can also be implemented in wired networks, such as wired local area networks (LANs), wide area networks (WANs), and the Internet. The implementation can also be implemented using wired communication protocols or technologies, such as Ethernet technology.

[0045] As used herein, the term "network device" or "network node" refers to a node in a communication network through which user equipment can access the network and / or through which the node can control radio communications within the cell and manage radio resources. This network node or network device may be referred to as a base station (BS), access point (AP), or access node. Depending on the technology applied, the network device may be, for example, a Node B (NodeB or NB), an evolved Node B (eNodeB or eNB), an NR NB (also known as a gNB), a Remote Radio Unit (RRU), a Radio Header (RH), a Remote Radio Header (RRH), a relay, an Integrated Access and Backhaul (IAB) node, a low-power node, a non-terrestrial network (NTN) or non-terrestrial network equipment (such as satellite network equipment, low Earth orbit (LEO) satellites, and geostationary orbit (GEO) satellites), or an aircraft network device.

[0046] The term "terminal device" refers to any terminal device capable of wired and / or wireless communication. For example, a terminal device may be referred to as a communication device, user equipment (UE), subscriber station (SS), or mobile station (MS). Terminal devices can include mobile phones, cellular phones, smartphones, Voice over IP (VoIP) phones, wireless local loop phones, tablets, wearable terminal devices, personal digital assistants (PDAs), portable computers, desktop computers, image acquisition terminal devices (such as digital cameras), gaming terminal devices, music storage and playback devices, in-vehicle wireless terminal devices, USB dongles, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (such as remote surgery), industrial equipment and applications (such as robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronics devices, devices operating on commercial and / or industrial wired or wireless networks, and any combination thereof.

[0047] Figure 1 An example of a communication network to which the examples disclosed herein can be applied is shown. This communication network, or cellular communication network, may include network node 110 and network node 112, with network node 110 providing one or more cells (such as cell 100) and network node 112 providing one or more other cells (such as cell 102). Cell 102 may be, for example, a macro cell, micro cell, femtocell, or picocell. The cell may define the coverage area or service area of ​​the corresponding access node.

[0048] Network node 110 can provide wireless access to a communication network to user equipment (UE) 120 (one or more UEs). This wireless access may include downlink (DL) communication from the network node to UE 120 and uplink (UL) communication from UE 120 to the network node.

[0049] Multiple UEs 120 and 122 can exist in the system. They can be served by the same or different network nodes 110 and 112. UEs can be configured with dual connectivity (DC), where a UE (e.g., UE 120) can connect to multiple network nodes 110 and 112. They can communicate with each other when a device-to-device (D2D) communication interface is established between UEs 120 and 122 via a so-called side link (SL). Such D2D communication can be referred to as machine-to-machine, peer-to-peer (P2P) communication, or vehicle-to-vehicle (V2V) communication, for example.

[0050] Network nodes 110 and 112 can be further connected to the core network 116 of the communication network via another interface.

[0051] AI / ML is expected to enable real-time analytics and automated operation and control in 5G and subsequent RANs. This requires the ability to acquire streaming data from wireless devices in a timely manner, especially in highly time-sensitive applications such as real-time video surveillance and extended reality (XR). This may manifest in network architecture, such as by placing and moving ML agents within the network to desired locations (e.g., for data collection). User equipment (mobile devices) can assist the network in resource management decisions, thus acting as an infrastructure resource.

[0052] As networks evolve towards programmable and flexible cloud-native implementations, AI / ML-based network automation will be used to simplify network management and optimization. It is anticipated that parts of the air interface (particularly signal processing algorithms) will be supported by machine learning models and eventually even replaced by them. Therefore, 6G wireless communication standards will natively support AI-based air interfaces.

[0053] The aspects related to communication and / or networking described herein can be embodied by wired and / or wireless communication devices or methods. Therefore, the sending, transmitting, and / or receiving steps can utilize radio access technologies (such as those described above) and / or wired networking technologies. "Wired" networking technologies can include technologies that enable the transmission of data over physical lines. This can include wire-based networking or optical-based networking.

[0054] Federal Learning Federated learning (FL) facilitates collaborative training of global models while maintaining data privacy. However, its practical application faces significant challenges due to data heterogeneity caused by the non-independent and identically distributed (IID) nature of data among clients. Clients typically have different data distributions due to differences in user behavior, demographics, and environment, meaning the global model may not adequately meet the specific needs of each client.

[0055] Personalized FL addresses the challenge of data heterogeneity by adapting the global model to the unique data distribution of each client, allowing customized models to perform better locally. However, striking a balance between personalizing the local model and the need for global collaboration remains a challenge, as extreme personalization can reduce the benefits of shared knowledge from other clients.

[0056] The following section will discuss aspects of personalized federated learning using prototype layer ensembles. This allows for a balance between generalization and personalization.

[0057] Generalization can refer to the model’s performance on new clients (which may have datasets that are not IID with existing clients) or to the model’s performance on data at existing clients that do not follow the data distribution at that client (e.g., in cases where there are outliers or offsets in the data collected at the client).

[0058] Personalization can involve the model’s performance on data from clients or subsets of clients participating in FL (i.e., how well the global model fits the data encountered by a local client, which may not encounter the same data distribution as the global average).

[0059] By leveraging conceptual prototypes to capture deeper category characteristics, aspects of this disclosure enable personalized models to benefit from global knowledge sharing, thereby ultimately improving both local and global performance in heterogeneous federated learning environments.

[0060] In some example methods for personalization in FL, prototype-based approaches are used where clients do not share the full model parameters, but only send the mean of the class representations, or a combination of the mean representation of their local data and the corresponding soft prediction of the mean. While these methods aim to enhance local accuracy, they may struggle to achieve strong global performance and contribute little to improving the global model, thus undermining the broader goals of federated learning. This paper refers to these methods as prototype-based approaches. Statistical prototype The method involves calculating the prototype as the mean of the categorical feature representation. Statistical prototypes represent the central tendency or average feature vector of a category, but lack deep interpretability.

[0061] Therefore, the aspects disclosed in this paper address a key challenge in federated learning: the problem of data heterogeneity. Clients typically have diverse and non-uniformly distributed data. This variability makes it difficult for the global model to perform well across all clients, as it cannot adapt to the unique data distributions of each client.

[0062] Personalized Flare Graph (FL) approaches attempt to adapt the global model to local data. However, they often prioritize individual client performance at the expense of the global model, thus undermining the collaborative advantages of FL. Prototype-based approaches (such as those based on statistical prototypes) share simplified representations of data and typically lack the depth needed to effectively enhance both local and global performance.

[0063] The aspects disclosed in this paper integrate conceptual prototypes into the FL process. Unlike methods that use statistical averages, conceptual prototypes can be explicitly learned (e.g., represented by a set of explicit trainable parameters) and represent key and interpretable features of the data (e.g., “patches” of the input data). This allows for more accurate and meaningful personalization without compromising the effectiveness of the global model.

[0064] The method for personalized federated learning disclosed in this paper uses prototype learning via conceptual prototypes. Under the conceptual prototype approach, prototypes are explicitly learned (e.g., prototypes may have a corresponding set of trainable parameters or be defined by a corresponding set of trainable parameters) to represent unique and interpretable features or concepts associated with a category. These prototypes capture important semantic elements and provide meaningful insights into classification beyond mere averages.

[0065] The aspects disclosed in this article apply to three key aspects of FL: (1) Generalization (e.g., by creating feature extractors that work for both existing and new clients); (2) Personalization (e.g., updates based on local data from the client); and (3) Privacy (e.g., by keeping prototypes locally without sharing them, and using privacy-preserving techniques for shared components).

[0066] Conceptual Prototyping Method Prototype-based classification is based on the idea that features extracted from test data can be associated with similar features learned from samples in the training set. Feature extractor It is to input Mapping to latent representation The model. Assume the input... It can be divided into smaller chunks that capture localized information from corresponding parts of the input data. These chunks can be non-overlapping. Therefore, potential chunks This represents the extracted features of a corresponding small block of input data. A latent block is designed to encode the features of a specific input block, and is the set of all latent block vectors. The entire input can be represented in the latent space. The feature extractor can be represented by a convolutional neural network.

[0067] Figure 2 This is a schematic diagram of example input 200. Input 200 is two-dimensional input data (e.g., image data). Patches of input 200 can be defined (e.g., according to predefined or trained rules—such as preset or trained regions or sizes for the patch) to produce patches such as patch 210. Patch 210 can capture specific features of the input data, which can be reflected in the corresponding vector (or feature vector) of the corresponding potential patch. For example, in the case of patch 210, features extracted from the “bottom” portion of the schematic diagram of the base station can be reflected in the corresponding feature vector.

[0068] Figure 3 This is a schematic diagram of the signal s of example input 300 over time. Example input 300 is an example of time-series input data (e.g., sensor data). Small blocks of input 300 can be defined (e.g., according to predefined or trained rules—such as preset or trained fixed time intervals for the blocks). Small blocks 310 can capture specific features of the input sensor data.

[0069] In the case of time-series medical sensor data (such as electrocardiogram (ECG) or photoplethysmography (PPG) data), different portions of the data may vary depending on the patient's condition. For example, different portions of an ECG may conform to or deviate from a healthy example in different ways and to different degrees, and some aspects of the patient's condition can be inferred from this. Potential small pieces corresponding to one of these features can therefore correspond to meaningful (and interpretable) features (such as features indicative of the patient's physiological condition).

[0070] In the case of audio data, potential small blocks can contain features corresponding to different audio sources (e.g., different bird calls may exist at different times, or elements of the sound environment may indicate normal or abnormal operation of a machine or its components).

[0071] The input may include time-series image data (e.g., video data), and potential patches may correspond to defined spatiotemporal regions of the input data. Additionally or alternatively, the input may include multiple image and / or time-series data sources. For example, a combination of multiple sensor readings (such as readings from multiple leads of an ECG, or a combination of ECG and PPG data) may serve as input data.

[0072] When feature extractor The training process involves associating small patches of input data from samples in the training set with learned prototypes (e.g., by associating the feature vectors output by the feature extractor and associated with the input patches with the learned prototypes). For example, a small patch of input data from a set of learned prototypes can be associated with a learned prototype. From potential small blocks Learning, in small pieces These prototypes are obtained from samples in the training set. A prototype can be represented by a prototype vector (e.g., this prototype vector can be compared to a feature vector). A prototype can be represented as a sphere in the latent space, which can contain various learned prototypes from the training set. Prototype sphere It is a sphere in potential space, with its center being... , radius is The prototype sphere can be defined around the prototype center. A fixed region is defined, and all potential patches within that region are considered to have equal similarity. In some examples, the prototype sphere can be defined by (trainable) vectors in the latent space (e.g., prototype vectors in the latent space) and a (trainable) radius in the latent space. Feature vectors within the sphere defined by the center and radius in the latent space can be considered to have equal similarity.

[0073] In some examples, the feature extractor f may include a convolutional layer. It has one dimension... Each pixel, in another dimension 1 pixel and has An input image with multiple channels (such as color channels) can be processed by a feature extractor (using at least one convolutional layer) to obtain a feature image with... Small blocks (where each small block has) The representation of (a channel). Therefore, small blocks can be represented by the latent space. It is represented by a dimensional vector.

[0074] Novel feature extractors can be designed to extract patches from the input, or existing architectures such as ResNet (e.g., ResNet18), DenseNet, VGG, etc.) can be adapted to provide feature extractors. For example, the outputs of layers in a convolutional network with a desired size (e.g., number of patches) and depth can be selected as the output of the feature extractor. For instance, in some examples, layers from the first to the penultimate layer of a ResNet model can be viewed as forming the feature extractor, while the final layer can be viewed as receiving the extracted features and determining the classification.

[0075] The prototype sphere represents a broader concept of input data for a category and allows for the capture of diverse features without requiring an exact match with the training set (e.g., without giving excessive weight to exact matches with or lack thereof with the prototype center).

[0076] Prototype layer It is a model parameterized from M trainable prototypes. For example, these trainable prototypes could be prototype spheres. The prototype layer will represent the latent representation. Mapped to Similarity scores. Output of the prototype layer (i.e., potential patches) With the prototype ball The similarity can be represented as These similarity scores can be calculated using logarithmic similarity, cosine similarity, or any other suitable method for calculating similarity between vectors. A limit can be placed on the maximum similarity to ensure that all vectors within the prototype sphere have the same similarity score. In some examples, the prototype is compared to multiple potential patches, and the corresponding similarity score for that prototype is based on the potential patch most similar to that prototype.

[0077] Prototype classifier It is a model that receives similarity scores generated by the prototype layer as input. The output can be called the evidence logit because, semantically, the output... This indicates the following evidence: by input Potential small blocks and categories Each learned prototype ball is compared, input Category This classifier can be generated by... Each category is assigned a weight parameter, where It is the number of categories in the training set.

[0078] By combining the above three models (i.e.) This allows for prototype-based classification. Prototype-based classification has an advantage over conventional deep learning methods in that it provides interpretable prototypes that offer insights into the reasons behind the final classification.

[0079] For example, in the context of prototype-based time-series medical data classification, medical professionals can be provided with the classification of the input data output by the model. Medical professionals may be interested in determining how the classification was arrived at, so the model could also output prototype spheres that the model identifies as similar to the input data, or output example (e.g., training) data corresponding to those prototype spheres. From the example data, medical professionals can determine that the prototype spheres represent certain features (such as specific deviations from normal in portions of ECG, PPG, etc.) and infer the "reason" for the model's conclusion (e.g., professionals can infer that a particular patient condition was determined based on a specific combination of identified features). This can enhance trust in the model, provide insights into patterns in the input data, and / or allow monitoring of the model's accuracy and / or reliability.

[0080] Prototypes can be used to enhance interpretability and explainability in centralized environments. Prototypes can also be applied to personalization in FL models (personalization may be less relevant in centralized environments without independent clients for personalization).

[0081] However, in some FL implementations, , and The prototype needs to be distributed across multiple clients for training and subsequent aggregation. However, distributing all of these components could lead to unacceptable privacy issues, as the interpretability of the prototype could reveal sensitive information to other clients, such as the presence of certain features in the training set.

[0082] Model Architecture This disclosure relates to prototype-based learning in the FL context.

[0083] Figure 4 This is a schematic diagram of the architecture of Model 400. Model 400 includes a private / local part (local model 410) and a collaborative part (global model 420).

[0084] Model 400 integrates prototype-based classification into FL. Model 400 includes Model 420 (or the "global" model), which is collaboratively trained in FL and is represented as follows. As described above, f outputs a latent representation Z, which includes feature vectors corresponding to the latent patches of the input, while s generates a classification (or evidence logit) based on the input latent representation.

[0085] Global models (such as classifiers for global models) do not necessarily have to use conceptual prototype methods. Instead, alternative methods may be better suited for privacy protection or for efficient federated learning.

[0086] In some examples, the feature extractor f may include one or more convolutional layers, while s may include one or more pooling layers, one or more fully connected layers, etc. For example, the feature extractor f may include the first part of a layer in a ResNet-18 architecture, which is configured to receive an input image and output... Small blocks (where each small block has) (Number of channels), and the classifiers include a second part of the layers of the ResNet-18 architecture, which is configured to receive data with... Each channel The ResNet-18 architecture can be divided into small blocks and output classifications. For example, a set of layers in the ResNet-18 architecture (which takes an input image and generates a 7×7 output with D channels) can be used as a feature extractor, and the remaining layers can be used as a classifier. In some examples, layers from the first to the penultimate layer of the ResNet-18 architecture can be used as feature extractors, and the final layer can be used as a global classifier.

[0087] By introducing two additional components and To extend / supplement this global model, these two components remain private on the client side. and Apply prototyping methods (such as the conceptual prototyping method using a prototype ball mentioned above) and use the feature extractor f of the global model to generate latent representations.

[0088] In order to leverage both local and global knowledge, and The outputs can be combined to construct a joint predictor that performs the final classification. During the FL round, the client can iteratively train all components, but only... and Shared with the server.

[0089] When training model 400, the global model 420 (i.e., including a shared feature extractor f and classifier heads s) is trained collaboratively across multiple clients (e.g., based on local datasets at the respective clients). The global classifier can perform classification using a neural network. Additionally or alternatively, logistic regression, support vector machines, decision trees, and / or gradient boosting can be used.

[0090] The local model 410 (especially the prototype layer g and prototype classifier h of the local model) is trained locally to retain local data characteristics for personalization. The training process can be structured in several key phases, including an initial warm-up phase for the prototype-based model and alternating local training phases during federated learning rounds.

[0091] By aggregating f across multiple clients, the feature extractor is generalized and can be applied to both existing and new clients. Generalization can be further enhanced by using global classifiers and a joint predictor to provide classifications based on multiple datasets (i.e., datasets from multiple clients) for joint predictions with the outputs of local classifiers.

[0092] Personalized classification paths can be provided based on the client-specific data distribution and characteristics through the client-based local data update components g and h.

[0093] By updating prototypes based on local data (e.g., prototypes in components g and h), there is no need to share the prototypes with other clients. This allows the personalization and interpretability advantages of the conceptual prototyping approach to be preserved without compromising privacy through prototype sharing. Components f and s can benefit from privacy-preserving techniques in FL (e.g., classifiers s can be used without employing the conceptual prototyping approach), which can be used in parallel with model 400. Any one or more of the following privacy-preserving techniques can be used, such as differential privacy, secure multi-party computation, homomorphic encryption, federated averaging with model compression, and segmentation learning.

[0094] train Figure 5 This is a message sequence diagram, illustrating an example method generally indicated by reference numeral 500. In the training process 500, client 510... Training process 500 begins at step 520, in which server 512 initializes the model. and And distribute them among the clients (including client 510).

[0095] In some examples, g and h can be initialized without using knowledge from f and s. For example, the parameters of g and h can be initialized using random or predefined values. In other examples, g and h can be initialized based on knowledge from f and s.

[0096] For example, pre-trained parameters from f and s can be used to initialize g and h. The client can initialize g and h based on representations learned from f and s. For instance, the client can apply the learned feature representations of f to derive relevant local prototypes, then use these representations to initialize g and h, and tune them during local training.

[0097] In some examples, layer adaptation or transformations can be used to initialize the values ​​of g and h. Clients can initialize g by applying transformations (such as dimensionality reduction or clustering) to the output space of f to identify meaningful prototype centers. Similarly, h can inherit initial weights from s to maintain compatibility between global and local classification decisions and can be fine-tuned based on the local data distribution.

[0098] In some examples, local layer fine-tuning can be used when initializing the values ​​of g and h. After receiving f and s from the global model, the client can initialize g and h using the same architecture or a simplified variant, and fine-tune them locally. This allows each client to adapt to its specific data distribution while still utilizing the shared knowledge encapsulated in f and s.

[0099] In step 522, client 510 first freezes and training and To "warm up" and The connection between them. This "warm-up" process can be performed in one or more training steps, utilizing a local dataset.

[0100] In steps 526 and 524, after warm-up, client 510 trains (i) and With (ii) training and Alternating between the two, both are based on local datasets. Step 526 (Training) and ) or step 524 (training) and Any of the steps in ) can be executed first, and steps 524 and 526 can be alternated once or multiple times. In some examples, during training and At that time, for Training involves mini-batches of samples, including minimizing the cross-entropy loss (by updating the model's trainable parameters using any suitable iterative method, such as stochastic gradient descent).

[0101]

[0102] in It is for categories samples The label (0 or 1), and It is by For category The predicted softmax output.

[0103] When in step 526 Training shared models ( During step 524, the freeze can be maintained. (or ).

[0104] For private models ( and Multi-objective loss can be used to ensure that the learned prototype ball is representative, distinct, and properly aligned with its corresponding category:

[0105] in It is to control the loss of the corresponding target to multiple targets. The hyperparameters contributing to the calculation can be fine-tuned through cross-validation.

[0106] Cross-entropy loss with mixup Measure the difference between predicted labels and combinations of true and predicted labels to encourage generalization, where It is by For category The predicted softmax output, and λ controls the mixing.

[0107]

[0108] Orthogonal loss By encouraging the centers of the prototype spheres to be orthogonal, it is ensured that different prototype spheres learn different characteristics.

[0109]

[0110] Clustering loss The prototype center is brought closer to the example of its corresponding category to ensure that it represents a meaningful feature.

[0111]

[0112] Separation loss The prototype center is pushed away from examples of other categories, so that they do not overlap with features of other categories.

[0113]

[0114] L1 regularization Promote the sparsity of the model's parameters and exclude weights associated with the correct class to avoid penalizing accurate predictions.

[0115]

[0116] minimize The prototype sphere is encouraged to remain compact, thereby allowing the learned concepts to capture more specific semantic features that are crucial for accurate classification.

[0117]

[0118] Private models g and h can be trained in step 524 (e.g., while keeping the feature extractor f frozen). In other examples, a shared feature extractor and private models f, g, and h can be trained in step 524.

[0119] When the shared models (g and h) are in step 524 by During training, h (or g and h) can remain frozen during step 526.

[0120] Distillation process In some examples, knowledge distillation (KD) can be used to incorporate knowledge of the global model 420 into the local model 410. However, this distillation is selective and is applied only if any of the following conditions are met: 1) The global model is correct, but the local model is incorrect:

[0121] 2) Both models are correct, but the global model has a higher confidence level:

[0122] For a sample that satisfies any of these conditions, the probability of the local branch (e.g., output by h) can be replaced with the probability of the global branch. q l ( x ) ← q g ( x (e.g., output by s).

[0123] Adjusted local probabilities can be used q l ( x ) and global probability q g ( x Distillation losses are calculated using the Kullback-Leibler (KL) divergence between the two groups.

[0124] Here, B represents a batch of samples that meets one of the conditions. This loss encourages the local model to mimic the output distribution of the global branch for selected samples, thereby helping the local model improve its performance by learning from stronger or more confident predictions from the global model. When the conditions are met, this distillation loss can be included as an additional term in the multi-objective loss.

[0125] Federal Aggregation In step 528, after local training, client 510 sends locally determined updates for the global model f and s to server 512. In step 530, server 512 aggregates updates from multiple clients. For example, server 512 can determine the average of the weight updates received from the respective clients. The relative contribution of the weight updates from the clients to the average can be weighted based on the number of samples at that client (i.e., performing aggregation can include calculating a weighted average). In this example, the weight updates can be determined as follows:

[0126] in K It is the number of clients. n k It is a client k The number of data samples, and It represents the total number of samples across all clients. It is the local dataset of the client. The associated cross-entropy loss.

[0127] After aggregation, in step 532, the weight update determined by the server is sent to the client 510, and the client 510 uses the updated weights in models f and s.

[0128] The training and aggregation steps 524-532 can be performed once, or steps 524-532 can be repeated until certain conditions are met (e.g., convergence of the global model). When a new client joins as a participant in FL, the training and aggregation process can be repeated (to improve the global model using that client's local data). When participating client 510 obtains new training data, the training and aggregation process can also be repeated (e.g., to train a new classifier h using the new data).

[0129] In some examples, steps 528-532 can be omitted. For example, if it is determined that the overall model or a specific classifier h has achieved sufficient accuracy (e.g. on a local dataset), steps 528-532 can be omitted from (at least some) subsequent training rounds.

[0130] In some examples, steps 528-532 can be omitted from all training rounds. In some examples, client 510 can benefit from federated training of models f and s performed by other clients (e.g., performed before step 520) without providing the server with locally determined updates to these models for aggregation.

[0131] infer During inference, once the model has been trained, the classifier h can be used in step 534 to classify the input.

[0132] Alternatively, during the estimation period, the input can be fed into both classifiers s and h, and the final prediction can be given by the joint predictor q.

[0133] The joint predictor utilizes the outputs of the global model 420 and the local model 410. The joint prediction is computed by averaging the outputs of the global classifier and the local prototype-based branch. :

[0134] in, It is the predicted probability distribution from the global classifier s, and This is the predicted probability distribution from the local model h. The average combines the strengths of both models to produce the final prediction.

[0135] Figure 6 It is a flowchart of a method according to an example embodiment, generally indicated by reference numeral 600.

[0136] In step 610, a machine learning model is trained (e.g., trained by a device or client device). The machine learning model includes a collaboratively learned portion parameterized by a first set of parameters, and a local portion including a second set of trainable parameters.

[0137] The collaborative learning component includes a feature extractor configured to extract features from the input, such as image data, time-series data, etc. Using a collaborative learning feature extractor can provide improved generalization of the model and allow new clients to accelerate the initial training process by leveraging the work of existing clients. The extracted features can be extracted as vectors, and these vectors can correspond to corresponding portions, patches, or regions within multiple regions of the input. Extracting features as vectors corresponding to portions or patches of the input allows for comparison with a trainable prototype. Feature vectors can represent features of data from corresponding defined regions within multiple defined regions of the input. Training the machine learning model involves training the collaborative learning component to obtain local updates to the first set of trainable parameters.

[0138] The local component includes a local classifier configured to receive extracted features and output a classification of the input. The local classifier includes a prototype layer configured to determine similarity scores between the extracted features and multiple trainable prototypes. Using trainable prototypes enhances interpretability (because prototypes can be interpreted based on training data corresponding to them). Using trainable prototypes also enhances personalization, as once trained, they can represent features of the local client data. For example, given a set of trainable prototypes and feature vectors, the local classifier can determine a corresponding similarity score for combinations of feature vectors and trainable prototypes. Thus, the similarity score characterizes which features (corresponding to trainable prototypes) are present in small patches of input data. Trainable prototypes can include trainable vectors in the same vector space as the feature vectors. Similarity scores can include or are based on log-based or cosine-based similarity. Trainable prototypes can also include a trainable radius to define a prototype sphere. For all feature vectors located within the trainable radius of a trainable prototype, the similarity score can be the same (e.g., log-based or cosine-based similarity scores can have an upper bound such that all vectors within the trainable radius have the same similarity). Using a prototype ball avoids attaching too much weight to an exact match with the prototype, allowing the prototype to reflect features of a broader category. Training the machine learning model involves training the local parts to obtain local updates to a second set of trainable parameters. In some examples, the corresponding prototype has a corresponding similarity score based on the similarity between the prototype and the most similar latent patch.

[0139] The local classifier includes a prototype classifier configured to receive similarity scores and output a classification of the input. The prototype classifier may include one or more fully connected layers. In some examples, the fully connected layers of the prototype classifier output a logit, evidence associated with the corresponding class. The prototype classifier may also include layers configured to generate probability or confidence scores associated with the corresponding class based on the logit.

[0140] In step 612, a local update to the first set of trainable parameters is sent to the device. This could be, for example, a server managing collaborative learning, or another client device (e.g., in the case of decentralized collaborative learning). These can be aggregated by sending local updates or instructions for local updates to the trainable parameters of the collaborative portion of the model to the device, allowing global updates to be shared with multiple participating clients.

[0141] In some examples, the input to the model may include image data. In some examples, the image data is medical image data.

[0142] In some examples, the input includes time-series data. Time-series data can include sensor data. Sensor data can include electrocardiogram (ECG), photoplethysmography (PPG), or inertial measurement unit (IMU) data.

[0143] In some examples, the collaborative learning portion of the machine learning model also includes a collaborative learning classifier. The collaborative learning classifier can be configured to receive a representation of the input data from a feature extractor and output a classification of the input data. The machine learning model may also include a joint predictor, which is configured to determine the classification based on the classifications output by the local classifier and the collaborative learning classifier. Using a global classifier and a joint predictor can provide better performance and generalization. Using a global classifier (which can be trained on a larger dataset than the local classifier) ​​can improve performance overall, and in particular, improve generalization (e.g., because a global classifier can perform better on data that does not follow the local distribution on which the local classifier was trained).

[0144] In some examples, training the machine learning model involves alternating between training a local classifier and training a feature extractor and a collaboratively learned classifier on a first dataset.

[0145] In some examples, the classification includes a probability distribution comprising multiple probabilities, where each probability indicates the probability that the input corresponds to a specific category from a set of categories. This provides more information about the classification confidence level than simply indicating the category.

[0146] In some examples, training a machine learning model involves training a first set of trainable parameters in at least the initial training rounds without training a second set of trainable parameters. Since the second set of trainable parameters is learned collaboratively, while the first set is trained locally, the second set of trainable parameters can be "frozen" to retain the parameters learned collaboratively, while the first set of trainable parameters is "warmed up" to establish connections with the second set.

[0147] Figure 7 It is a flowchart of a method according to an example embodiment, which is generally indicated by reference numeral 700.

[0148] Method 700 includes steps 610 and 612 of method 600.

[0149] In step 710 of method 700, a machine learning model is received from a device. This device may also be a device that receives parameter updates. The received machine learning model (or, in particular, the parameters of a local portion of the machine learning model) may include placeholder parameters or uninitialized parameters to be initialized at the device performing training. The collaboratively learned portion of the received machine learning model may include parameters that have already been trained (e.g., these parameters may have been collaboratively trained by other devices / client devices, if those parameters are available).

[0150] The next steps are 610 and 612, which use the received machine learning model.

[0151] In step 712 of the method, updates to the second set of trainable parameters (or indications of such updates) are received from the device. For example, updates to the second set of trainable parameters can be received from the device from which the local updates are sent, if the device to which they are sent is a server that aggregates the results of training rounds in federated learning.

[0152] The model can be updated to use a second set of trainable parameters received from or indicated by the device. In contrast, the model can use locally updated parameters from the first set of trainable parameters.

[0153] In some examples, the trained model can be used for inference.

[0154] Inference may include receiving input (e.g., at a client device). The input data used for inference may have a similar form to the input data used for training. For example, the model may be trained using image data (such as medical image data), and the input during inference may also be image data (such as medical image data). The input data during inference may come from the same or similar sources as the input data during training. For example, training data may be data from an electrocardiogram (ECG), and the input data may also be data from an ECG.

[0155] During inference, the input is classified using a trained model. For example, the trained model can output labels or sets of labels that classify the input, such as the label with the highest associated confidence, or by outputting labels and associated confidence values, or by outputting a set of labels and confidence values.

[0156] Classifying the input can include: using a collaboratively trained feature extractor to determine the extracted features of the input, using a local prototype layer to determine the similarity score between the extracted features and the prototype, and using a local prototype classifier to determine the classification of the input.

[0157] In some examples, the classification determined by the local prototype classifier can be output by the model. In some examples, classifying the input may also include using a collaboratively trained classifier to determine the input's classification, and using a joint predictor to determine the classification from the classifications output by the local prototype classifier and the collaboratively trained classifier.

[0158] Feature extractors can extract features corresponding to small patches of input, and the extracted features can be expressed as feature vectors. For example, a specific feature vector can correspond to a specific texture, pattern, edge, or combination thereof in a small patch of an image.

[0159] The prototype layer compares feature vectors to prototypes. A prototype can be a vector or a sphere in the same vector space as the feature vectors, and the similarity score can be calculated using a suitable measure of vector similarity, such as cosine similarity. In some examples, the prototype is a "sphere" with a radius (relative to a center, which can be defined by a vector). In this case, all vectors within the radius of the center can have the same similarity score, for example, by restricting the similarity score so that the similarity score of vectors within the prototype sphere does not exceed the similarity score of vectors on the boundary of the prototype sphere. In some examples, the similarity score can be determined for pairs of prototypes and feature vectors; therefore, if the prototype layer includes M prototypes and extracts Z features from Z patches of input, then M×Z similarity scores, or at most M×Z similarity scores, can be determined.

[0160] Classification can include probability distributions or confidence scores, which indicate how confident the classification is of the input. For example, a high confidence or high probability that a classification or label is applicable to the input may indicate that the classification or label is likely applicable to the input, while a low confidence or low probability may indicate that the classification or label is likely not applicable to the input.

[0161] Example Application The aspects disclosed herein can be applied, for example, to medical image analysis. The application of the above aspects in the classification of moles is discussed below.

[0162] Clients (such as client devices at healthcare providers) may have a dataset of categorized mole images (e.g., images of moles classified as cancerous, non-cancerous, etc.). This dataset may include images from patients that have already been categorized by healthcare professionals.

[0163] There might be interest in training models to classify mole images. Using federated learning, multiple client devices (e.g., client devices from multiple healthcare providers) can train a global model, which is then updated using their own classified mole images. However, the data encountered by different client devices may not be independent and identically distributed. For example, patients from different healthcare providers may have different average sun exposures (e.g., based on latitude), which could lead to different conditions with varying frequencies. Therefore, personalization might be of interest to address the non-IID nature of data across different client devices. Furthermore, healthcare provider data can be sensitive, thus privacy protection might be a concern.

[0164] Applying the above method to this scenario, the client can participate in federated learning and train the classifier together with other clients (as described above). In federated learning, the feature extractor and the global classifier can be trained collaboratively.

[0165] Conceptual prototyping methods can be applied, and prototype layers (e.g., including trainable prototypes such as prototype balls) and prototype classifiers can be trained on local data. Prototype balls are associated with features of potential patches. Patches of training data corresponding to potential patches having feature vectors located within these prototype balls can be associated with prototype balls to aid in interpretability.

[0166] Once trained, the classifier can classify mole images (e.g., classify them as containing cancerous or non-cancerous moles).

[0167] In some cases, it may be obvious to medical professionals that a particular image has been correctly classified as containing a cancerous nevus or a non-cancerous nevus. However, the "reasoning" behind the inferred classification may be of interest, especially when the incorrect or incorrect classification is not immediately obvious. For example, inferring the reasoning behind a model can help quickly distinguish between incorrect results and correct results derived from features that are not immediately apparent. Inferring the reasoning can also allow for investigating aspects of the model or training data that lead to incorrect results, thus enabling updates to the training data or model accordingly.

[0168] Training data samples can correspond to prototype spheres (e.g., small patches of training data can correspond to potential small patches with feature vectors located within the prototype sphere). Features of these training data samples represented by the sphere can be inferred (e.g., if all samples corresponding to a particular prototype sphere contain the same pattern or texture, then the sphere can represent that pattern or texture feature). A medical professional can then infer from the similarity score and the prototype sphere a specific combination of features (such as texture or pattern) that leads to the classification. The medical professional can then determine whether the model's inference "cause" is reasonable and, accordingly, agree or disagree with the classification.

[0169] This conceptual prototype-based approach can deliver good performance on local datasets based on local data, while the collaborative learning parts of the model (such as the collaborative learning feature extractor f and global classifiers s) can provide: a) better performance on data that does not necessarily follow the local distribution trained on the client (e.g. due to outliers or local distribution), and b) better performance on new clients.

[0170] The aspects of this disclosure are similarly applicable to other types of medical data from any medical or physiological sensor. For example, images from specialized medical imaging equipment (e.g., ultrasound, magnetic resonance imaging (MRI), computed tomography (CAT), positron emission tomography (PET), etc.) can be analyzed. In the case of a three-dimensional imaging device, the potential patch can be three-dimensional. Similarly, in the case of time-series image data (e.g., video), the potential patch can also be three-dimensional. In other examples, the data can be time-series medical sensor data, such as electrocardiogram (ECG), photoplethysmography (PPG), blood glucose meter, blood pressure monitor, etc., where the potential patch is a time period.

[0171] Furthermore, aspects of this disclosure are applicable to any type of sensor data, such as data from positioning / position sensors, IMU (inertial measurement unit) sensors, camera sensors, lidar (light detection and ranging) sensors, etc., where the potential small blocks are time periods.

[0172] Furthermore, aspects of this disclosure are applicable to analyzing performance data of any type of electrical component or device, such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), memory unit, transmitter, receiver, antenna, microphone, speaker, display, camera sensor, etc., where the potential small blocks are time periods.

[0173] Furthermore, aspects of this disclosure are applicable to analyzing performance data of any type of wireless or wired communication network equipment or components, such as NodeB (or NB), evolved NodeB (eNodeB or eNB), NR NB (also known as gNB), remote radio unit (RRU), radio head (RH), remote radio head (RRH), relay, integrated access and backhaul (IAB) node, low-power node, non-terrestrial network (NTN) or non-terrestrial network equipment (such as satellite network equipment, low Earth orbit (LEO) satellites and geostationary orbit (GEO) satellites), aircraft network equipment, user equipment (UE), user station (SS), mobile station (MS), or Internet of Things (IoT) equipment. User equipment (UE) can include mobile phones, cellular phones, smartphones, IP-based voice (VoIP) phones, wireless local loop phones, tablets, wearable devices, personal digital assistants (PDAs), laptops, desktop computers, image acquisition devices (such as digital cameras), gaming devices, playback devices, in-vehicle wireless devices, USB dongles, watches or other wearable devices, head-mounted displays (HMDs), etc. Internet of Things (IoT) devices can also include vehicles, drones, medical devices and applications (such as remote surgery), industrial devices and applications (such as robots and / or other wireless devices operating in industrial and / or automated processing chain environments), etc.

[0174] The aspects of this disclosure are also applicable to fault detection. For example, image data can be used for corrosion detection, or audio data can be used to detect mechanical faults in machinery. The methods described herein can also be applied to wildlife classification (which can also be based on image or audio data). The occurrence of corrosion and the presence of wildlife may depend on climate and may therefore be non-IID.

[0175] Example training data and training hyperparameters The following describes an example of a training process that can be performed based on the above aspects.

[0176] In one example, the method described in this paper is applied to the CIFAR-10 training set (which consists of labeled 32×32 pixel images). The feature extractor uses convolutional layers based on the ResNet-18 architecture, and the collaborative learning classifier uses a classifier based on the ResNet-18 architecture (e.g., the classifier may correspond to the last layer of the ResNet-18 architecture, and the feature extractor may correspond to the preceding layers). A pre-trained ResNet-18 model (trained on the CIFAR-10 training set) can be used as the initial global model, which simulates new clients joining federated learning and achieving the effect of a collaborative learning model already trained on other data.

[0177] In this example, training begins with a 10-round local warm-up phase (during which the collaborative learning feature extractor and global classifier are frozen, and the parameters of the local model are trained). The parameters of the local model are initialized with random values ​​and then trained.

[0178] During each federated round, the global model and the local model are trained alternately for one round. The feature extractor, classifier, and prototype are all optimized with a learning rate of 0.001. Stochastic gradient descent (SGD) is used for the feature extractor and the adaptive moment estimation (Adam) is used for the prototype. A momentum of 0.9 is used across all optimizers and no weight decay is used.

[0179] The batch size was set to 64 for training. The following weighting scheme / hyperparameters were used for the loss function of the local model: α = 1, β = 1, γ = 0.8, δ = 0.08, = 10 -4 , and ζ = 1. For A top-k algorithm with k = 10 and radius 4.5; for The confidence score was set to 0.8. A total of 200 prototypes and 10 descriptive units were used to represent the underlying structure of the data.

[0180] For completeness, Figure 8 This is a schematic diagram of components of one or more example embodiments previously described, which are generally referred to below as processing system 1800. Processing system 1800 may, for example, be comprised of the devices mentioned in the following claims.

[0181] The processing system 1800 may have a processor 1802, a memory 1804 tightly coupled to the processor and including random access memory (RAM) 1814 and read-only memory (ROM) 1812, and optionally, a user input 1810 and a display 1818. In addition to or instead of the user input 1810, the processing system may include a sensor 1820. The processing system 1800 may acquire data (e.g., sensor data) via the sensor 1820 and / or via a network interface 1808. The processing system 1800 may include one or more network / device interfaces 1808 for connecting to a network / device (e.g., a wired or wireless modem). The network / device interface 1808 may also operate as a connection to other devices, such as devices that are not network-side devices. Therefore, direct connections between devices without network involvement are possible.

[0182] The processor 1802 is connected to other components to control their operation.

[0183] Memory 1804 may include non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD). The ROM 1812 of memory 1804 stores the operating system 1815 and may also store software applications 1816. The RAM 1814 of memory 1804 is used by the processor 1802 for temporary data storage. The operating system 1815 may contain code that, when executed by the processor, implements aspects of methods 500, 600, and 700. Note that in the case of small devices, the memory may be best suited for small size applications; that is, a hard disk drive (HDD) or a solid-state drive (SSD) is not always used.

[0184] The processor 1802 can take any suitable form. For example, it can be a microcontroller, multiple microcontrollers, a processor, or multiple processors.

[0185] The processing system 1800 can be a standalone computer, server, console, or its network. The processing system 1800 and the required structural components can be any internal devices / devices such as IoT devices, i.e., embedded in a very small size.

[0186] In some example embodiments, the processing system 1800 may also be associated with external software applications. These may be applications stored on a remote server device / device and may run partially or exclusively on the remote server device / device. These applications may be referred to as cloud-hosted applications. The processing system 1800 may communicate with the remote server device / device to utilize the software applications stored there.

[0187] Figure 9 A tangible medium is shown storing computer-readable code in the form of a removable memory unit 1910, which, when run by a computer, can execute the methods according to the example embodiments described above. The removable memory unit 1910 may be a memory stick, such as a Universal Serial Bus (USB) memory stick, having internal memory 1930 for storing computer-readable code. Internal memory 1930 can be accessed by a computer system via connector 1920. Of course, other forms of tangible storage media can be used, as will be apparent to those skilled in the art. The tangible medium can be any device / apparatus capable of storing data / information that can be exchanged between devices / devices / networks.

[0188] Embodiments of the present invention can be implemented in software, hardware, application logic, or a combination of software, hardware, and application logic. The software, application logic, and / or hardware can reside on memory or any computer medium. In example embodiments, the application logic, software, or instruction set is maintained on any of a variety of conventional computer-readable media. In the context of this document, "memory" or "computer-readable medium" can be any non-transitory medium or device that can contain, store, transmit, propagate, or transfer instructions for use by or in connection with an instruction execution system, apparatus, or device (such as a computer).

[0189] In relevant contexts, references to “computer-readable medium,” “computer program product,” “tangible computer program,” or “processor” or “processing circuitry” should be understood to encompass not only computers with different architectures (such as single / multiprocessor architectures and sequencer / parallel architectures) but also special-purpose circuits such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), signal processing devices / apparatus, and other devices / apparatus. References to computer programs, instructions, code, etc., should be understood to express software (such as programmable content of hardware devices / apparatus) used for programmable processor firmware as instructions for a processor or as configuration or configuration settings for fixed-function devices / apparatus, gate arrays, programmable logic devices / apparatus, etc.

[0190] As used in this application, the term "circuit" may refer to one or more or all of the following: (a) Hardware circuit implementation only (e.g., implemented with purely analog and / or digital circuits) and (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of (multiple) analog and / or digital hardware circuits and software / firmware, and (ii) Any part of a hardware processor(s) having software (including (multiple) digital signal processors, software, and (multiple) memories, which work together to enable a device (such as a mobile phone or server) to perform various functions) and (c) The operation requires software (e.g., firmware) for the operation of (multiple) hardware circuits and / or (multiple) processors, such as (multiple) microprocessors or parts thereof, but the software may be absent when the operation does not require the software.

[0191] This definition of "circuit" applies to all uses of the term in this application (including in any claim). As a further example, as used in this application, the term "circuit" also covers only hardware circuitry or processors (or processors), or portions of hardware circuitry or servers and their accompanying software and / or firmware implementations. For example, where applicable to a particular claim element, the term "circuit" also covers baseband integrated circuits or processor integrated circuits for mobile devices or similar integrated circuits in servers, cellular network devices, or other computing or networking devices.

[0192] In some examples, the circuit implements aspects of the methods disclosed herein, such as methods 500, 600, and 700. For example, the circuit may be configured to perform at least one step of at least one of methods 500, 600, and 700 or any other method disclosed herein.

[0193] If necessary, the different functions discussed herein can be executed in different orders and / or simultaneously with each other. Furthermore, one or more of the above functions can be optional or can be combined, if needed. Similarly, it should be understood that... Figures 5 to 7 The diagram is merely an example, and the various operations depicted therein can be omitted, reordered, and / or combined.

[0194] It should be understood that the above exemplary embodiments are purely illustrative and do not limit the scope of the invention. Other variations and modifications will be apparent to those skilled in the art after reading this specification.

[0195] Furthermore, the disclosure of this application should be understood to include any novel feature or any novel combination of features or any generalization thereof explicitly or implicitly disclosed herein, and new claims may be formulated during the examination of this application or any application derived therefrom to cover any such feature and / or combination of such features.

[0196] Although various aspects of the invention are set forth in the independent claims, other aspects of the invention include other combinations of features from the described exemplary embodiments and / or dependent claims with features of the independent claims, and not just combinations explicitly set forth in the claims.

[0197] It should also be noted in this document that while various examples have been described above, these descriptions should not be viewed in a restrictive sense. Rather, several changes and modifications may be made without departing from the scope of the invention as defined in the appended claims.

Claims

1. An apparatus for model training, comprising: Components for training a machine learning model, wherein the training includes: The collaborative learning portion of the model is trained by training a first set of trainable parameters to obtain local updates of the first set of trainable parameters, the first set of trainable parameters parameterizing the collaborative learning portion, wherein the collaborative learning portion includes a feature extractor configured to extract features from the input; and The local part of the model is trained by training a second set of trainable parameters to obtain local updates to the second set of trainable parameters, the second set of trainable parameters parameterizing the local part, wherein the local part includes a local classifier configured to receive the extracted features and output a classification of the input, the local classifier comprising: A prototype layer, wherein the prototype layer is configured to determine similarity scores between extracted features and multiple trainable prototypes; and A prototype classifier, which is configured to receive the similarity score and output a classification of the input based on the similarity score.

2. The apparatus according to claim 1, further comprising: A component for sending instructions to devices participating in collaborative learning regarding the local updates of the first set of trainable parameters; A component for updating the second set of trainable parameters of the machine learning model based on the obtained local update of the second set of trainable parameters; A component for receiving an indication of an update to the first set of trainable parameters from a device participating in the collaborative learning; as well as A component for updating the first set of trainable parameters based on the indicated update.

3. The apparatus according to claim 1 or claim 2, further comprising: The component for receiving the feature extractor of the collaborative learning portion of the model and the local classifier of the local portion of the model.

4. The apparatus of claim 1 or claim 2, wherein the component for training the machine learning model is configured to: train the first set of trainable parameters without training the second set of trainable parameters, at least in the initial round of training.

5. The apparatus according to claim 1 or claim 2, wherein: The collaborative learning portion of the machine learning model further includes a collaborative learning classifier configured to receive representations of the input data from the feature extractor and output a classification of the input data; and The machine learning model also includes a joint predictor configured to determine a classification based on the classification output by the local classifier and the collaboratively learned classifier.

6. The apparatus of claim 5, wherein the component for training the machine learning model is configured to alternate between the following on a first dataset: Training the feature extractor and the local classifier; and The feature extractor and the collaboratively learned classifier are trained.

7. The apparatus of claim 1 or claim 2, wherein the feature extractor of the collaborative learning portion is configured to extract features by defining a plurality of regions of the input for the input and outputting a plurality of feature vectors, wherein a corresponding feature vector among the plurality of feature vectors represents a feature of data from a corresponding defined region of the defined plurality of regions of the input.

8. The apparatus of claim 7, wherein the prototype layer is configured to: determine a similarity score between the feature vector of the extracted feature and the trainable prototype.

9. The apparatus of claim 8, wherein the trainable prototype comprises a corresponding trainable vector in the same vector space as the plurality of feature vectors.

10. The apparatus of claim 9, wherein the trainable prototype includes a corresponding trainable radius, and wherein determining the similarity between the feature vector and the trainable prototype comprises: The similarity score is determined at least in part based on whether the feature vector lies within the trainable radius of the trainable vector of the trainable prototype.

11. The apparatus of claim 1 or claim 2, wherein the classification comprises a probability distribution comprising a plurality of probabilities, wherein the plurality of probabilities indicate the probability of an input corresponding to a corresponding category in a set of categories.

12. The apparatus according to claim 1 or claim 2, wherein the input comprises image data.

13. The apparatus of claim 1 or claim 2, wherein the input comprises sensor data.

14. A method for model training, comprising: Training a machine learning model includes: The collaborative learning portion of the model is trained by training a first set of trainable parameters to obtain local updates of the first set of trainable parameters, the first set of trainable parameters parameterizing the collaborative learning portion, wherein the collaborative learning portion includes a feature extractor configured to extract features from the input; and The local part of the model is trained by training a second set of trainable parameters to obtain local updates to the second set of trainable parameters, the second set of trainable parameters parameterizing the local part, wherein the local part includes a local classifier configured to receive the extracted features and output a classification of the input, the local classifier comprising: A prototype layer, wherein the prototype layer is configured to determine similarity scores between extracted features and multiple trainable prototypes; and A prototype classifier, which is configured to receive the similarity score and output a classification of the input based on the similarity score.

15. A method for classification, comprising: Receive input; as well as The input is classified using a machine learning model, which includes a local component and a collaborative learning component. The determination includes: The feature extractor of the collaborative learning part is used to determine the extracted features of the input, and the feature extractor is made to extract features of the input; The prototype layer of the local part of the model is used to determine a set of similarity scores, the prototype layer being configured to determine the similarity between the extracted features of the input and a set of trained prototypes of the prototype layer; and The prototype classifier of the local part of the model is used to determine the classification of the input, and the prototype classifier is made to determine the classification based on the similarity score.

Images