Semantic Learning in Federated Learning Systems

Abstract

A method, a computer system, and a computer program product are provided for federated learning enhanced with semantic learning. An aggregator may receive cluster information from a plurality of distributed computing devices. The cluster information may relate to a plurality of clusters identified in sample data from the plurality of distributed computing devices. The aggregator may include at least one processor. The aggregator aggregates the cluster information to define a plurality of classes. The aggregation may include identifying potentially redundant clusters among the identified plurality of clusters. The number of the plurality of classes may correspond to a total number of clusters from the plurality of distributed computing devices reduced by the potentially redundant clusters. A deep learning model may be transmitted from the aggregator to the plurality of distributed computing devices. The deep learning model may include an output layer having a plurality of nodes. The plurality of nodes may correspond to the defined plurality of classes. The aggregator may receive one or more results of federated learning performed by the plurality of distributed computing devices. The federated learning may train the deep learning model.

Description

[Technical field]

[0001] The present invention relates generally to the field of deep learning models, and more particularly to federated learning for deep learning models. [Background technology]

[0002] In a federated learning system, a group of devices or parties work together to develop and jointly train a deep learning model, e.g., a predictive model, without sharing or exposing the raw data of each party to other devices or parties participating in training the model. Training deep learning using multiple parties and their data will result in better training and a better model, since the more data that is processed, the better the model will be trained. An aggregator receives information from each device or party and compiles or aggregates the information to fine-tune the deep learning model. In some cases, the aggregator averages the information from the parties to adjust the deep learning model. In a federated learning system, data from various devices in a network can be used to train the model without the devices sharing their individual raw data on their individual computing devices or phones through the cloud. The aggregator can send the adjusted deep learning model to each device without those devices receiving raw data from other devices. Thus, federated learning helps achieve privacy for individual device raw data while still allowing multiple parties to collaborate to train deep learning models. Federated learning prevents an adversary from undoing the data transformations and finding the original raw training data that matches a particular party.

[0003] "Towards Federated Learning At Scale System Design" by Bonawitz et al. describes a distributed machine learning approach that enables training on a large corpus of distributed data residing on devices, such as mobile phones. Bonawitz et al. also state that FL addresses fundamental issues of data privacy, ownership, and locality.

[0004] U.S. Patent Application Publication No. 2019 / 0138934 A1 to Prakash et al. discloses that for federated learning, each client computing node fetches a global model, uses its local data to update the global model, and communicates the updated model to a central server. However, Prakash et al. disclose that computation should be balanced among heterogeneous computing nodes based on knowledge of network conditions and operational constraints experienced by the heterogeneous computing nodes.

[0005] Known federated learning systems have the drawback that they are built on the assumption that different parties label data, that different parties use the same data labels, and that different parties know all possible data classes. In practice, parties often do not know the labels used by other parties. A first party may label a sample differently than a second party would label that sample. For example, one party may consider rabbits to be pets, while another party may consider rabbits to be food. Parties, such as parties at a hospital, may not be allowed to share data and labels in advance. Also, new samples may emerge that do not fit well into the static model structure. For example, in a model for recognizing pictures of food, a local food, such as arepa, may be understood by a local party, but not by other parties scattered around the world. Updating the model structure would require all local models to be retrained, which would consume a huge amount of time and energy. Summary of the Invention [Means for solving the problem]

[0006] According to one exemplary embodiment, a method, a computer system, and a computer program product may perform federated learning. An aggregator may receive cluster information from a plurality of distributed computing devices. The cluster information may relate to a plurality of clusters identified in sample data of the plurality of distributed computing devices. The aggregator may include at least one processor. The aggregator may aggregate the cluster information to define a plurality of classes. The integrating may include identifying possible redundant clusters among the identified plurality of clusters. The number of the plurality of classes may correspond to a total number of clusters from the plurality of distributed computing devices reduced by the possible redundant clusters. A deep learning model may be transmitted from the aggregator to the plurality of distributed computing devices. The deep learning model may include an output layer having a plurality of nodes. The plurality of nodes may correspond to the defined plurality of classes. The aggregator may receive one or more results of the federated learning performed by the plurality of distributed computing devices. The federated learning may train the deep learning model.

[0007] In this embodiment, labels for data samples can be accurately and seemingly encoded in the deep learning system, even if different parties give the same sample type different names. The deep learning models in federated learning are tuned to the semantic meanings of the different participants, so that the participants can independently use their correct native semantic labels while still contributing to the federated learning system.

[0008] In additional exemplary implementations, the aggregator may provide an autoencoder to the plurality of distributed computing devices via the aggregator. Each computing device of the plurality of distributed computing devices may run its sample data through the autoencoder to generate an autoencoder output. Each distributed computing device may run the autoencoder output through a clustering algorithm to identify the plurality of clusters from the sample data. The cluster information may be transmitted from the plurality of distributed computing devices to the aggregator.

[0009] In this embodiment, the federated learning system may be improved to have increased accuracy for correctly identifying labels and classes for data samples fed into the system.

[0010] Another example embodiment may further include, via a computing device of the plurality of distributed computing devices, naming one of the classes according to a semantic meaning for the computing device.

[0011] In this embodiment, each party that is part of a system of multiple distributed computing devices in a federated learning deep learning network may provide its own semantic definition for labeling groups that are shared by other parties in the network.

[0012] Exemplary supplemental implementations may further include anonymizing the cluster information via the plurality of distributed computing devices before the cluster information is transmitted to the aggregator.

[0013] In this embodiment, privacy of customer data is maintained so that individual parties can participate in federated learning without exposing private customer and other individual raw data.

[0014] Further example implementations may further comprise checking new classes during the federated learning via a first computing device of the plurality of distributed computing devices, where the checking may comprise the first computing device feeding new samples to the autoencoder and performing anomaly detection to detect deviations of the new samples from the plurality of classes by exceeding a predefined threshold.

[0015] In this way, new classes that emerge during associative learning can be added to the deep learning model in an accelerated manner, thereby saving computing resources.

[0016] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which should be read in conjunction with the accompanying drawings, in which various features of the drawings are not to scale for clarity's sake that the drawings, together with the detailed description of the invention, will facilitate understanding of the invention by those skilled in the art. The drawings are as follows: [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 illustrates a networked computing environment in accordance with at least one embodiment. [Diagram 2] FIG. 2 is an operational flow chart diagram illustrating a process for a federated learning process workflow enhanced with semantic learning in accordance with at least one embodiment. [Diagram 3] FIG. 3 illustrates an autoencoder structure and clustering technique implemented in accordance with at least one embodiment. [Figure 4] FIG. 4 illustrates an example of cluster information integration in accordance with at least one embodiment. [Diagram 5]FIG. 5 illustrates a networked computer environment according to at least one embodiment in which individual computing devices of a federated learning network load data samples organized into clusters. [Figure 6] FIG. 6 is a diagram illustrating an example of layers of a deep learning model. [Figure 7] FIG. 7 is a block diagram of internal and external components of the computer and server depicted in FIG. 1 according to at least one embodiment. [Figure 8] FIG. 8 is a block diagram of an exemplary cloud computing environment including the computer system depicted in FIG. 1 according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a block diagram of functional layers of the example cloud computing environment of FIG. 7 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Detailed embodiments of the claimed structures and methods are disclosed herein. However, it can be understood that the disclosed embodiments are merely examples of the claimed structures and methods, which may be embodied in various forms. However, the present invention may be embodied in various forms and should not be construed as being limited to the exemplary embodiments described herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. In the detailed description of the invention, details of well-known features and techniques may be omitted in order to avoid unnecessarily obscuring the presented embodiments.

[0019] The exemplary embodiments described below provide a system, method, and program product for federated learning enhanced with semantic learning. The embodiments have the ability to accurately and seemingly code labels for each participant or party in a federated learning system, even when a sample has different semantic meanings for different participants or parties. The embodiments also allow new classes to be added to a deep learning model in an accelerated manner without requiring full retraining to be performed across all parties. Thus, the embodiments save computing and training resources for the federated learning system, and allow the federated learning system to improve its adaptability to unique samples and private semantic labels for the samples of each participating party.

[0020] Referring to FIG. 1, an exemplary networked computing environment 100 according to one embodiment is illustrated. In some embodiments, the networked computing environment 100 may be considered a federated learning system. The networked computing environment 100 may include multiple computers, namely, a first computer 102a, a second computer 102b, and a third computer 102c. Although FIG. 1 illustrates three such client computers in the networked computing environment 100, in practice, there may be many more such client computers in the networked computing environment 100. As described below with respect to FIG. 7, each of the first, second, and third computers 102a, 102b, and 102c may include one or more processors and memories capable of executing and storing the federated learning programs 110a, 110b, and 110c. The networked computing environment 100 may also include a server 112 that may execute the federated learning program 110d, which may interact with a database 114 and a communication network 116. The networked computing environment 100 may include multiple servers 112, only one of which is shown. The communication network 116 may include various types of communication networks, such as a wide area network (WAN), a local area network (LAN), a telephone network, a wireless network, a public switched network, or a satellite network, or combinations thereof. It should be understood that FIG. 1 provides only an example of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0021] The first, second, and third computers 102a, 102b, and 102c may communicate with the server computer 112 via a communication network 116. The communication network 116 may include connections, such as wired communication links, wireless communication links, or fiber optic cables. As described with reference to FIG. 7, the server computer 112 may include internal components 902a and external components 904a, respectively, and the first, second, and third client computers 102a, 102b, and 102c may include internal components 902b and external components 904b, respectively. The server computer 112 may also operate in a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server 112 may also be located in a cloud computing deployment model, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud. The first, second, and third computers 102a, 102b, and 102c may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running a program, accessing a network, and accessing the database 114. According to various implementations of the present embodiment, the federated learning programs 110a, 110b, 110c, and 110d may interact with the database 114, which may be embedded within various storage devices, such as, but not limited to, the computer / mobile device 102, the networked server 112, or a cloud storage service.

[0022] According to this embodiment, a user using a client or server computer 112 may use the associative learning programs 110a, 110b, 110c, and 110d to perform semantic learning-enhanced associative learning to train a deep learning model. The semantic learning-enhanced associative learning method is described in detail below with reference to Figures 2-9.

[0023] The server computer 112 may be considered an aggregator that is part of a federated learning system. In federated learning, the aggregator may issue a query to all available parties in the federated learning system, for example, the aggregator may issue a query to each of the first, second and third computers 102a, 102b and 102c in the networked computing environment 100 shown in FIG. 1 and FIG. 5. Each of the parties has their own data set. In a federated learning system, the parties cannot see the raw data of the other parties in the system. Each party may have its own local learning model. The query made by the aggregator may be an inquiry as to what the current model parameters of the local learning model for each party are. The query may result in the parties providing some information to the aggregator. The aggregator may ask what the answer is to a particular question based on the parties' data sets. The parties may generate their answers based on their own local data. In some cases, the parties may have local learning models stored and running in their client computers that help generate responses. The parties generate their own responses and send the responses back to the aggregator. After the aggregator receives responses from all parties, or from a sufficient number of parties above a threshold, the aggregator performs an aggregation or synthesis of all responses and uses the results to update a machine learning model maintained by the aggregator and stored and run in the aggregator computer. After several of these training / tuning sessions or cycles have been performed, a final machine learning model, e.g., a global model or neural network, is then generated by the aggregator and shared with all parties, e.g., the first, second and third computers 102a, 102b and 102c. The raw data remains at each party's site.In the embodiment shown in FIG. 1 implemented in a federated learning system, the server computer 112 is configured to send queries to the first, second, and third computers 102a, 102b, and 102c, perform this aggregation process to generate a global model or neural network, and then transmit the global model or neural network to the first, second, and third computers 102a, 102b, and 102c.

[0024] Federated learning can be implemented, for example, in an Internet of Things (IoT) network or in a network of smartphones. In these cases, many parties may participate in the training. Data from various devices in the network can be used to train the deep learning model without the devices sharing their individual raw data on the individual devices or smartphones through the cloud. The raw data includes identity data of the parties. In at least some embodiments, anonymized data from which identity data has been removed may not be considered raw data.

[0025] In another possible implementation, competitors in the market can cooperate and use federated learning to train a model while also protecting the privacy of their customers' information. This scenario of competitors cooperating may involve fewer parties helping to train the system compared to the system described above. In this scenario, for example, federated learning could be used by multiple banks to train a system to detect money laundering without the banks having to share or disclose the individual raw banking data of each individual customer.

[0026] In other situations, federated learning may also be performed when individual devices are constrained by connectivity and have few opportunities to share their information to the cloud. For example, a robot on Mars does not have many data transmission opportunities to send its data to Earth or satellites orbiting Earth. Instead of sending its complete compilation of raw data, the local device may train a model locally and then transmit the trained model during the few transmission opportunities. The local dataset including features and labels is held by the owner / individual party, and other entities in the federated learning system do not have access to the local dataset of other parties in the federated learning system.

[0027] The associative learning system, in some cases, is implemented in predictive typing or predictive speech. The associative learning system may be implemented to classify images or may be implemented to classify audio voice signals.

[0028] The federated learning system may protect the raw data of individual parties from being shared with other parties by using secure aggregation. When a party has a unique sample to add to the deep learning model, that information may be provided anonymously to the aggregator. Thus, the aggregator may be able to recognize which of the parties in the federated learning system the sample came from without knowing which particular party or machine in the federated learning system it came from. A party may share model parameters or model weights of a local model or training sample size or a combination thereof with the aggregator. A party may also share gradients calculated based on the local dataset with the aggregator. If only these parameters or weights or gradients are shared with the aggregator, the raw data, including features and individual labels, may remain private on the party's computing device.

[0029] 2, an operational flow chart diagram is shown illustrating an exemplary process 200 for deep learning augmented with semantic learning, in which the federated learning programs 110a, 110b, 110c, and 110d may act in accordance with at least one embodiment.

[0030] As depicted in Figure 2, process 200 includes an exemplary deep learning process augmented with semantic learning. Process 200 may begin at step 202, where an aggregator transmits an autoencoder to a computing device of the federated learning system. The autoencoder may be transmitted over a communications network 116, which may include various types of communications networks, such as a wide area network (WAN), a local area network (LAN), a telephone network, a wireless network, a public switched network, or a satellite network, or combinations thereof, as described above with respect to Figure 1, and may include connections, such as wired communications links, wireless communications links, or fiber optic cables.

[0031] In at least some embodiments, the autoencoder may be a pre-trained autoencoder. To train a deep learning model, the total number of classes should be fixed and static, and definitions for the classes should be provided. Therefore, to pre-train an autoencoder, an operator may input the number of classes and definitions for those classes based on experience with previous deep learning models or based on educated guesses about potential samples expected to be received and analyzed by the deep learning model. However, training an autoencoder may be unsupervised, such that labels are not required during training. When an image is fed into the autoencoder, the encoder of the autoencoder extracts high-level features from the image. The decoder of the autoencoder may reconstruct the image using the extracted features. The encoder and decoder together help to generate the autoencoder. Using a pre-trained autoencoder helps to reduce training time and improve the performance of the autoencoder. In some cases, the autoencoder may be pre-trained without the number of classes being known. Using a pre-trained autoencoder helps the autoencoder generalize.

[0032] The autoencoder, described below with respect to Figure 3, may include a deep learning model. The autoencoder receives data as input and uses the deep learning model to extract a high-level feature vector from each data sample. The deep learning model may be a neural network, such as a convolutional neural network.

[0033] In step 204 of process 200, each computing device of the federated learning system, such as the first, second, and third computers 102a, 102b, and 102c shown in FIG. 1, trains the autoencoder received in step 202 and generates a trained autoencoder as a result. FIG. 3, described below, illustrates an example of an autoencoder and its components. This training of the autoencoder is done by each party running or feeding their samples or sample data to the autoencoder as input to the autoencoder's learning model. A loss function is then used for each party to gradually adjust the parameters of the autoencoder's learning model. The loss function is run to produce a loss amount, the parameters are gradually adjusted to try to reduce the loss amount, and the loss function is run again. The loss function measures the performance of the autoencoder.

[0034] Step 204 may include sequential training of the autoencoder by individual computing devices of the federated learning system, for example, by the first, second, and third computers 102a, 102b, and 102c. In the sequential training, the autoencoder may be first sent to a particular computing device, for example, the first computer 102a, which first trains the autoencoder to generate a partially trained autoencoder. The first computer 102a may then send the partially trained autoencoder to another computing device, for example, the second computer 102b. This other computing device may further train the partially trained autoencoder by feeding its own data to the deep learning model and repeatedly running a loss function for guidance to gradually adjust the parameters of the deep learning model. The other computing device will generate a further trained autoencoder due to this further training.

[0035] This passing of the partially trained autoencoder from one computing device to another may be done directly over the communication network 116, or it may be done via an aggregator that receives the partially trained autoencoder and forwards it to the next computing device. The latter scenario will often be done because in many federated learning systems, the party will not know who the other parties in the federated learning system are and therefore will not know how to send its partially trained model to the other parties. Therefore, in the embodiment, the aggregator will be used to facilitate this transfer because it knows and has stored information about who the parties / computing devices in the federated learning system are. Regardless of whether the aggregator is used as a facilitator for the transfer or the party knows about the other parties and can transfer directly, the training of this autoencoder by one party after the other parties may be considered sequential training.

[0036] The second computer 102b may pass or transmit the further trained autoencoder to an additional computing device, such as a third computer 102c. This passing may be done directly or through the aggregator, as described above, either way using a communications network 116. The additional computing device may feed its samples to the further trained autoencoder and run a loss function to guide incremental adjustment of the deep learning model parameters. This additional training may produce a trained autoencoder, or an autoencoder ready to be distributed to all parties.

[0037] In some embodiments, this training or sequential training of the autoencoder may be performed across all of the computing devices or parties of the federated learning system before the aggregator requests the autoencoder. Alternatively, the aggregator may request the autoencoder after it has been trained by the majority of the devices / parties of the federated learning system. This use of majority training may save some time and resources compared to embodiments in which all parties help train the autoencoder.

[0038] During the incremental training of the autoencoder, the autoencoder itself, including its neural network / learning model, is passed from one party to another, but the raw data, including the raw features and raw labels, is not passed. The incremental learning of the autoencoder by the parties may also be considered as collaborative training of the autoencoder by the parties.

[0039] In step 206, the computing device sends the trained autoencoder back to the aggregator. This transmission may occur over a communications network 116. In at least some embodiments, the last computing device, such as the third computer 102c shown in FIG. 1, will send the autoencoder back to the aggregator. If the aggregator requests the autoencoder after most but not all of the parties' autoencoders have been trained, the party that sends the autoencoder to the aggregator will not be the last of all the parties to the autoencoder.

[0040] In at least some embodiments, clustering techniques are not required or performed for steps 202-206, which are all related to training the autoencoder. In these embodiments for process 200, these steps 202-206 do not require submitting sample data through a clustering algorithm.

[0041] After the aggregator receives the trained autoencoder from the computing devices of the federated learning system, in step 208, the aggregator shares the trained autoencoder with the computing devices, for example, with all of the computing devices of the federated learning system. This sharing may be done via a communication network 116. If the training of the autoencoder by the parties is done sequentially, sharing the final trained autoencoder with all parties gives all parties the benefit of a complete training of the autoencoder done by all parties, or by a majority of the parties, or by a certain number of the parties. This sharing gives each party an improved autoencoder compared to an autoencoder trained on data by one party alone.

[0042] In at least some embodiments, the autoencoder will include a clustering algorithm. In various embodiments, the clustering algorithm may be a K-means clustering algorithm, a mean-shift clustering algorithm, density-based spatial clustering of applications with noise (DBSCAN), expectation-maximization (EM) clustering with Gaussian mixture models (GMM), or agglomerative hierarchical clustering.

[0043] At step 210, each computing device, e.g., first, second, and third computers 102a, 102b, and 102c, runs its sample data through the trained autoencoder to generate an autoencoder output. The autoencoder output may be a high-level representation of the input data, and may in particular be a vector. The vector may include multiple variables or parameters, e.g., three or more variables or parameters, or even up to 100 or more variables or parameters. These variables or parameters may be referred to as features. In a deep learning model that identifies pictures of animals, the autoencoder may recognize various features about each image that may help the deep learning model classify the animal, e.g., size, number of appendages, shape of ears, etc. These features may be variables or parameters that are determined by feeding samples through an autoencoder that analyzes the samples. In embodiments where an image is fed to the autoencoder, the autoencoder may analyze the image and analyze the pixels of the image.

[0044] In step 212, each computing device, e.g., each of the first, second, and third computers 102a, 102b, and 102c, runs the autoencoder output through a clustering algorithm to identify clusters in the sample data. A clustering algorithm may reside within each computing device. Alternatively, a clustering algorithm may reside within the autoencoder received from the aggregator. In some implementations, the clustering algorithm may be a K-means clustering algorithm, a mean shift clustering algorithm, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering with Gaussian mixture models (GMM), or agglomerative hierarchical clustering. Figure 3 shows an example of an autoencoder paired with the feature of passing the autoencoder output through a clustering algorithm. The top part of Figure 3 shows an example of clusters generated from output samples run through a clustering algorithm.

[0045] Feeding the autoencoder output through the clustering algorithm will generate a respective data point for each autoencoder output, i.e., a data point representing each sample. These data points can be graphed. For example, in a three feature autoencoder model, a three-dimensional graph can be used to graph the data points. Data points that are generally close together on the three-dimensional graph will generally be considered to belong to a particular cluster of the data. In at least some embodiments, each cluster of sample data or each cluster of the autoencoder output will have a centroid, i.e., the center of the cluster, and a radius. Data points that are located in the chart at a distance from the centroid less than the radius will be considered to belong to that cluster. The top part of FIG. 3 is an example of the number of clusters of data points, showing the centroid and radius of a particular cluster. An autoencoder model with three features that can be paired with a three-dimensional graph in a clustering algorithm is used as an example for simplicity and to more easily explain the concept. In many autoencoder models that will be used, the autoencoder model will use more than three features. For example, the autoencoder model may contain up to 100 features, or even more. Clustering techniques are still achievable with these models that have many more features.

[0046] In step 214 of process 200, the parties, e.g., first, second, and third computers 102a, 102b, and 102c, perform an anonymization technique on the cluster information generated by the clustering algorithm. This anonymization results in a private representation of the autoencoder output. The anonymization technique may be a generalization technique. In the case of the anonymization technique, identity attributes, e.g., explicit identifiers, may be removed from the cluster information. Thus, when the aggregator receives a set of anonymized cluster information from a party of the federated learning system, the aggregator may be able to recognize that the sending party belongs to the federated learning system, but may not be able to recognize which party sent the set of cluster information. For example, any name of the party that generated the cluster information may be removed from the cluster information before the cluster information is sent to the aggregator.

[0047] Various anonymization techniques may be performed according to step 214. In the suppression anonymization technique, tuples or attribute values ​​are replaced with a special anonymous symbol, e.g., "**". Thus, in suppression anonymization, the original data values ​​are replaced with some anonymous value throughout the cluster information. In the generalization anonymization technique, the attribute values ​​are replaced with values ​​that do not change semantically but are less specific. For example, if the party identification information contains information about the location of the party / participant, the city or state of the party may be replaced with the country of the location of the party / participant. In the bucketization anonymization technique, the sensitive information is maintained but is separated or partitioned from any identifying information. Thus, in bucketization anonymization, the aggregator may receive features from sample data but would not be able to recognize which party a particular sample data came from. In the perturbation anonymization technique, the sensitive information is not removed but is randomly altered. Therefore, the modified information received by the aggregator will be incorrect, and the aggregator will know what information has been modified, but external parties will not be able to know what information has been modified. Slicing anonymization may be performed by splitting columns / rows of information horizontally and vertically, and then randomly sorting the splits, so that the final information transmitted has features grouped together, but not in a way that completely links the information set. Slicing anonymization may be further complemented by encryption of sensitive features.

[0048] In step 216 of process 200, the parties, such as the first, second, and third computers 102a, 102b, and 102c, transmit the cluster information to the aggregator, such as server computer 112. The parties may use communication network 116 to transmit the cluster information. The cluster information may be anonymized before being transmitted.

[0049] In step 218 of process 200, the aggregator, e.g., server computer 112, aggregates the cluster information to define multiple classes. The aggregation is a method of processing cluster information. The aggregation performed by the aggregator may include identifying possible redundant clusters among the identified clusters, and the number of classes defined may be characterized as corresponding to the total number of clusters from the multiple distributed computing devices reduced by the possible redundant clusters. In at least some embodiments, the cluster information may include centroid information regarding the centroids of the clusters, and the aggregator may compare the centroid information to identify possible redundant clusters. For example, if centroids from various parties are within a distance less than a predefined new cluster threshold distance, the aggregator may consider the centroids to belong to redundant clusters that should be aggregated or merged for class aggregation. Additionally or alternatively, if the radius of a first cluster overlaps with the radius of a second cluster, the aggregator may consider centroids with the respective radii as belonging to redundant clusters that should be merged for class aggregation. The aggregator may determine the distance between each received centroid and all other centroids that are part of the cluster information of all groups of the federated learning system. The aggregator may also perform a new clustering technique on the cluster information to identify the total number of clusters. This new clustering is especially likely if the cluster information was anonymized in step 214, because the aggregator does not know which party a particular feature information came from, but it does have access to the feature information. In the case of the K-means clustering technique performed here, a predefined threshold distance between centroids does not need to be selected, because the K-means clustering algorithm can hash out and determine what groups of data points constitute separate clusters.

[0050] In step 220 of process 200, the aggregator, e.g., server computer 112, transmits the deep learning model incorporating the multiple classes to all computing devices in the federated learning system, e.g., all of first, second, and third computers 102a, 102b, and 102c. The aggregator may use communication network 116 to transmit the deep learning model. In at least some embodiments, the deep learning model may be a neural network, e.g., a convolutional neural network. The deep learning model has an output layer with multiple nodes. The nodes correspond to the above-defined multiple classes defined in step 218. FIG. 6, described below, illustrates an example of a layer of a deep learning model with an output layer with multiple nodes. With the integration from step 218 and the insertion of class data into the model transmitted in step 220, the parties may be considered to be jointly training the deep learning model. In the first example of performing step 220, the deep learning model will typically be transmitted alone. As part of this step 220 in a subsequent cycle of executing process 200, in addition to sending the updated deep learning model, the aggregator may also send an updated autoencoder in which the new class has been incorporated. Step 226, described below, illustrates the possible need for another updated autoencoder to be sent to each party if a new class is later discovered.

[0051] In step 222 of process 200, the computing devices, e.g., first, second, and third computers 102a, 102b, and 102c, assign new semantic meanings to at least some of the classes. In the example of a model learning to recognize images, e.g., pictures of animals, if a party considers rabbits to be food and the aggregator identifies rabbits as one of the classes for the model, the party may assign "food" as the label for this category after receiving the class from the aggregator. If another party in the same federated learning system considers rabbits to be pets, the party may assign "pet" as the label for this category after receiving the class from the aggregator. Then, even if the first party and the other party have unique and different semantic labels for samples of the same class, the system will still function to perform federated learning.

[0052] In step 224 of the process 200, the computing devices, e.g., the first, second, and third computers 102a, 102b, and 102c, perform federated learning on the multiple classes. Thus, the computing devices receive new samples and feed these new samples into the deep learning model received from the aggregator. Periodically, the computing devices will transmit the results of their deep learning training to the aggregator. This transmission of the results of the deep learning training may be performed via the communication network 116. The federated learning described above occurs when parties in a system share data to train a model / system while still maintaining the privacy of their raw data. For federated learning, anonymized data may be transmitted instead of raw data. Thus, although some information is shared to achieve the model learning, raw sample data is not shared with the aggregator or with other parties. As part of step 224, the aggregator may receive from the plurality of distributed computing devices one or more results of the federated learning performed by the plurality of distributed computing devices.

[0053] In step 226 of process 200, the party periodically and locally checks to determine if new clusters have been added. To perform this check, the party may feed the new samples to the updated autoencoder received from the aggregator in step 220. In the feature space of the autoencoder, anomaly detection is performed to detect significant deviations of possible new data points / features from the clusters for known classes. For the anomaly detection, the distance from the data points for the new sample to the centroids of other clusters may be measured. If the distance exceeds a predefined threshold, the new data point may be considered anomalous. The distance from the data points of the new sample to the nearest other known data points of other clusters may be measured, and if the distance exceeds a predefined threshold, the new data point may be considered anomalous. Other clustering algorithms may be used to perform the anomaly detection. Numerous anomaly detection techniques may be implemented, such as density-based techniques. If the number of anomalies is significantly higher relative to the average number of anomalies over a period of time, or relative to the number of total test samples, a flag or warning message may be sent to the aggregator to alert it to the need to redefine the classes. Alternatively, when a single new data point that is outside of the existing classes is locally identified by a party, a flag or warning message may be sent to the aggregator to alert it to the need to redefine the classes. If the deviation of a sample from the existing classes exceeds a predefined threshold, the sample may be considered to belong to a new cluster. Information for the new data class itself may be sent together with the warning signal. This new data class information may be used to inform the aggregator, may be used by the aggregator to help retrain the autoencoder starting at step 202, or this new data class information may be part of the reintegration of cluster information in step 218 to skip the initial autoencoder training and define a new set of classes.This cluster information about the new potential cluster may be anonymized by the party before being sent with a warning flag to the aggregator. In other words, step 214 may be repeated at this time before the cluster information about the new possible cluster is sent to the aggregator. The aggregator may add a new class to the classes to form a new set of classes. The new class corresponds to the new sample. The new set of classes may be sent from the aggregator to the multiple distributed computing devices. Additional federated learning may then be performed via the aggregator and the multiple distributed computing devices to further train the deep learning model. Performing the additional federated learning includes using the new set of classes as new nodes of an output layer of the deep learning model.

[0054] To perform step 226, the data samples are provided to the updated autoencoder. To perform step 224, the data samples are provided to the deep learning model received from the aggregator. Thus, in some instances and at times during process 200, the data samples may be provided in parallel to both the updated autoencoder and the deep learning model.

[0055] In step 228 of process 200, a check is performed as to whether the deep learning should be stopped for the system or for a particular device of the federated learning system. If the check results in a positive decision to stop, the deep learning and the federated learning may be stopped as a whole for the entire federated learning system or may be stopped for an individual device of the federated learning system. If the check results in a negative decision to stop, the process may loop back to step 224 where the federated learning continues and then to step 226 where another check for a new cluster is performed. In a default setting, step 228 will be answered in the affirmative to continue the deep learning / federated learning. As part of step 228, the federated learning program 110a, 110b, 110c may generate and provide a graphical user interface (GUI) prompt to ask the user of the computing device whether the deep learning / federated learning should be stopped. If the user provides an affirmative answer in the GUI, the individual computing device may send a message to the aggregator to inform the aggregator of the same. In many embodiments, identifying new labels / classes will be an automated process performed by the individual computing devices. The identifying may be the result of new data points being added to the training dataset, some data samples being consistently misclassified, or the party deciding that they want to increase the number of local clusters with external knowledge.

[0056] In process 200, a model can be trained in a federated learning process while allowing semantic labels to be provided by individual parties and avoiding additional manual intervention or pre-processing that can be expensive. Process 200 can also be implemented for two different domains where samples of interest have been collected but do not have the same labels.

[0057] The autoencoder may be, for example, a vanilla autoencoder, a sparse autoencoder, a multilayer autoencoder, or a convolutional autoencoder. In at least one embodiment, the autoencoder may be a feedforward neural network, a non-recurrent neural network, having an input layer, an output layer, and one or more hidden layers connecting the input layer to the output layer. The output layer may have the same number of nodes as the input layer. The output layer helps reconstruct the input. In at least some embodiments, an autoencoder may be considered an unsupervised learning model that does not require labeled input to enable deep learning.

[0058] 3 shows the structure of an autoencoder 300 used in at least one embodiment according to the present invention. The autoencoder 300 typically comprises an encoder 304 and a decoder 317 that encode and decode sample data 302.

[0059] The encoder 304 encodes the sample data 302 to generate vectors and embeds the sample data 302 into a latent space. The vectors are a high-level representation of the sample data 302. In one embodiment, the sample data 302 is an image of an animal, and the autoencoder generates vectors that classify features of the animal image. The encoder 304 includes a first filter 306a, a second filter 306b, and a third filter 306c as an input layer of the autoencoder 300. The sample data is gradually reduced as it passes through the series of filters, the first filter 306a, the second filter 306b, and the third filter 306c. In the flattening layer 308, the autoencoder 300 may flatten the output of the encoder 304 to a dimension required for the output. The flattening layer 308 generates vectors that are fed to the embedding layer 310. The embedding layer 310 may generate embedded points, which are fed into a clustering algorithm 311 to generate a clustering diagram 312. The embedded points may be considered as an autoencoder output. In at least some embodiments, the dimensionality of the autoencoder output is reduced before the autoencoder output is passed to the clustering algorithm 311.

[0060] The embedded data may be passed through a clustering algorithm 311 to generate a clustering diagram 312. The clustering algorithm 311 may be included as part of the autoencoder 300 or may be a separate component of the individual host computing device running the autoencoder. In some embodiments, a clustering algorithm layer may be connected to the embedding layer 310. The clustering diagram 312 shown in FIG. 3 is generated via a k-means clustering algorithm. When a k-means clustering algorithm is used, a k value may be selected or calculated as a heuristic that represents the expected number of classes. Adjustments to the k value may then be performed as necessary to reduce the loss function. In one example, k is selected or calculated as 10. When the k is selected, the k value may be input as an argument to the clustering algorithm. At least one embodiment may comprise a fuzzy k-means clustering approach because deep clustering may require knowing the number of classes in advance. If k is calculated by an algorithm, a partition entropy algorithm, a partition coefficient algorithm, or other algorithms may be used.

[0061] The vector may be passed back through an enhancement layer 315 and then fed to a decoder 317 having a first decoding layer 318a, a second decoding layer 318b, and a third decoding layer 318c. Feeding the enhanced vector through the first decoding layer 318a, the second decoding layer 318b, and the third decoding layer 318c helps to reconstruct the original input data. The reconstructed image is output as an output sample data set 320.

[0062] FIG. 3 illustrates a clustering diagram 312, according to at least one embodiment. The clustering diagram may be generated by individual computing devices of the federated learning system. By generating the clustering diagram, the individual computing devices may generate cluster information for each cluster. The cluster information may be passed to the aggregator, for example, in step 216 of process 200 shown in FIG. 2. Alternatively, as shown in FIG. 4, a system clustering diagram 400 may be generated by the aggregator after the aggregator receives clustering information from the individual computing devices of the federated learning system.

[0063] The clustering diagram 312 shown at the top of FIG. 3 shows that when a party fed its samples into the autoencoder 300, it generated ten clusters, namely clusters 314a-314j. Cluster 314a is shown as having a centroid 316 and a radius 318. All sample points / vectors whose distance from the centroid 316 is less than the length of the radius 318 may be considered to belong to cluster 314a. The ends of the radius 318 represent the boundaries of cluster 314a. Note that since the centroid 316 is the average value of the sample points / vectors that belong to cluster 314a, it is possible that no sample points / vectors that belong to cluster 314a are present in the centroid 316. Although in FIG. 3 only cluster 314a is shown as having a centroid 316 and a radius 318, in reality all of the other clusters 314b-314j also have their own respective centroids and their own respective radii. Cluster information for clusters 314a-314j, as well as all of their centroids and radii, may be sent from each computing device to the aggregator in step 216 of process 200 shown in Figure 2. The feature of using centroid information instead of using individual raw data samples as part of the cluster information helps to protect the privacy of each party's individual raw data. Even if the aggregator is able to reconstruct centroids using centroid information, the aggregator typically still cannot find individual data points or match individual data points to possible specific parties participating in the federated learning system.

[0064] FIG. 4 shows an example of a system clustering diagram 400 in which clusters or cluster information from all devices of a federated learning system, or a combination thereof, are combined or aggregated by the aggregator. In the embodiment shown in FIG. 4, the aggregator first aggregated nine clusters received from all distributed computing devices or all individual computing devices. To ensure that the raw data is not shared with other parties, the individual computing devices passed centroid and radius information to the aggregator, but did not pass individual raw data, and did not pass individual data points from the clustering diagram 312. Therefore, FIG. 3 shows centroids and radii for the clusters, but not individual data points. The individual computing devices may also pass information about the number of data points that belong to each cluster. In FIG. 4, centroids 402a-402i are shown. Each of the centroids 402a-402i also has its own radius, but for simplicity, radii 404a, 404b, 404c, and 404h associated with centroids 402a, 402b, 402c, and 402h are shown. In one embodiment with a particular threshold, the aggregator checks, and the two clusters associated with centroids 402b and 402c will be identified as being redundant with each other because the distance between their respective centroids 402b and 402c is less than the threshold. In some embodiments, a threshold value of 0.5, 1.0, or 1.5 may be selected. Alternatively, the aggregator may recognize that the radius 404b for the cluster associated with centroid 402b overlaps with the radius 404c for the cluster associated with centroid 402c, and therefore, the two clusters associated with centroids 402b and 402c may be considered to be redundant with each other.

[0065] Therefore, in generating a deep learning model for transmission to all parties or participants of the federated learning system or individual distributed computing devices, as is done in steps 218 and 220, the aggregator will combine the two clusters associated with the two centroids 402b and 402c into a single cluster or single centroid. The aggregator may simply designate the new centroid to be located at the midpoint between the two centroids 402b and 402c. Alternatively, the aggregator may give more weight to the centroid of the two centroids 402b and 402c that has more data points. Thus, the aggregator will provide a total of nine output nodes in the output layer of the deep learning model shown in the embodiment of FIG. 4. In some embodiments, the output layer will be referred to as a logits layer. These nine output nodes correspond to the total number of clusters identified by the individual parties or by the individual distributed computing devices, except that number is reduced by one because clusters and their center points 402b and 402c are considered redundant and are considered to be the same cluster.

[0066] 4 shows that radii 404a and 404h are larger than radii 404b and 404c. The size of the radii may depend on the spread of the data points and the presence or absence of other clusters or centroids nearby.

[0067] FIG. 5 shows the networked computing environment 100 seen in FIG. 1, but in this case the first, second and third computers 102a, 102b and 102c are loaded with image groups from the data sample. In this embodiment, a deep learning model is to be trained to recognize pictures of animals. Each image will have its own label provided by the individual computing devices. The first computer 102a has a first computer first image group 502a, a first computer second image group 502b and a first computer third image group 502c. In the illustrated embodiment, the first computer first image group 502a is a collection of pictures of cats. The first computer second image group 502b is a collection of pictures of rabbits, but the participant operating the first computer 102a considers rabbits to be "food". The first computer third image group 502c is a collection of pictures of dogs. The second computer 102b has a second computer first image group 504a, a second computer second image group 504b, and a second computer third image group 504c. In the illustrated embodiment, the second computer first image group 504a is a collection of photos of cats. The second computer second image group 504b is a collection of photos of rabbits, although the participant operating the second computer 102b considers rabbits to be "pets". The second computer third image group 504c is a collection of photos of fish. The third computer 102c has a third computer first image group 506a and a third computer second image group 506b. In the illustrated embodiment, the third computer first image group 506a is a collection of photos of dogs. The third computer second image group 506b is a collection of photos of fish.

[0068] These image groups have been created by individual computing devices, for example, first computer 102a, second computer 102b, and third computer 102c, by running their individual data samples of images through a trained autoencoder received from the aggregator, for example, a trained autoencoder received from server computer 112 having federated learning program 110d, which provides an autoencoder output that, when fed through a clustering algorithm 311, produces clusters 314 representative of the image groups.

[0069] When the cluster information for the clusters 314 is passed to the aggregator, the aggregator aggregates the cluster information to identify the total number of classes for the deep learning model. Because the cat image from the first image group 502a of the first computer is not identical to the cat image from the first image group 504a of the second computer, the data points and their cluster centroids of the two received clusters are not identical, but the aggregator recognizes that the two clusters are overlapping and redundant and combines them into a single cluster. Because the fish image from the third image group 504c of the second computer is not identical to the fish image from the second image group 506b of the third computer, the data points and their cluster centroids are not identical, but the aggregator recognizes that the two clusters are overlapping and redundant and combines them into a single cluster. Although the image of the dog from the third image group 502c of the first computer is not identical to the image of the dog from the first image group 506a of the third computer, because the data points and their cluster centroids are not identical, the aggregator recognizes that these two clusters are overlapping and redundant and combines them into a single cluster.

[0070] The rabbit images from the second image group of the first computer 502b and the second image group of the second computer 504b are similar or identical to each other, but the first computer 102a named the rabbit group "food" and the second computer 102b named the rabbit group "pets". The aggregator may analyze the cluster information, e.g., the cluster information described above including the centroid and optionally the radius and the number of data samples, to recognize that the first computer's second image group 502b and the second computer's second image group 504b belong to the same cluster because the two clusters overlap and are redundant. The aggregator combines the two groups into a single cluster. Alternatively, in the initial passing of cluster information from the first, second and third computers 102a, 102b and 102c to the aggregator, the clusters are not labeled so that the aggregator can correctly recognize overlaps and redundancies of clusters by analyzing and integrating the cluster information, even if the operators of the first computer 102a and the second computer 102b have their own semantic meanings for the groups / clusters. In that regard, class labeling does not need to be performed to train the autoencoder 300.

[0071] The aggregator recognizes that the total number of clusters received from the distributed computing devices, in this case the first, second, and third computers 102a, 102b, and 102c, is eight: three from the first computer 102a, three from the second computer 102b, and two from the third computer 102c. Although the total number of clusters received is eight, the aggregator reduces that number based on the number of redundant pairs of clusters identified. Because the aggregator determines that four pairs of clusters are redundant, the aggregator reduces the total number of clusters (eight) by four, resulting in a total number of classes for deep learning in the federated learning system of four. Two cat groups were deemed redundant with each other. Two rabbit groups were deemed redundant with each other. Two dog groups were deemed redundant with each other. Two fish groups were deemed redundant with each other. Therefore, by performing aggregation and analysis of the cluster information, the aggregator recognizes that the total number of clusters for this embodiment of the deep learning model is four.

[0072] The aggregator then generates a deep learning model with four output nodes to pass to multiple distributed computing devices, in this case first, second, and third computers 102a, 102b, and 102c. For example, FIG. 6 shows a deep learning model 600 with a first input layer 602 and a second input layer 604, and an output layer 606. The output layer 606 in the first instance would have four nodes to correspond to the four classes determined through the merging performed by the aggregator. A first node 608a would be for the cat class of the image. A second node 608b would be for the rabbit class of the image. A third node 608c would be for the dog class of the image. A fourth node 608d would be for the fish class of the image.

[0073] After the aggregator passes the deep learning model 600 in step 220 to the plurality of distributed computing devices, in this case the first, second and third computers 102a, 102b and 102c according to Figures 5 and 6, the plurality of distributed computing devices have the opportunity in step 224 to name the received classes according to their own semantic preferences or meanings.

[0074] The first computer 102a may name the first three classes as "cat", "food" and "dog" while leaving the fourth class blank since it does not have any samples belonging to the fourth class. The aggregator may also send generic labels for the various classes, e.g., class 1, class 2, class 3, class 4, etc. Alternatively, if the aggregator sends classes filled with names, the first computer 102a may leave the name for the fourth class provided by the aggregator, e.g., leave the name "fish" for the fourth class, while still renaming the second group (rabbit group) according to its own semantic meaning for "food" since it believes rabbits are things to be eaten.

[0075] The second computer 102b can name the first, second and fourth classes as "cat", "pet" and "fish", respectively, while leaving the third class blank since it has no samples belonging to the third class, in this case the dog class. Alternatively, if the aggregator sends a class filled with names, the first computer 102a can leave the name "dog" for the third class provided by the aggregator, while still renaming the second group (rabbit group) according to its own meaning that rabbits should be kept as pets. Such naming is applied for the sample fitting performed by the computing device.

[0076] The third computer 102c may leave the first and second classes blank and name the third and fourth classes "dog" and "fish" because it has no samples belonging to the first class or the second class, and no samples belonging to the rabbit cluster or the cat cluster. Alternatively, if the aggregator sends classes filled with names, the first computer 102a may leave the names of the first and second classes provided by the aggregator, e.g., "cat" and "rabbit", while also accepting the names "dog" and "fish" provided by the aggregator for the third and fourth classes.

[0077] According to the embodiment shown in Figures 5 and 6 and described above, in a further step 226, when one of the distributed computing devices, for example the first computer 102a, receives a sample of bird images and feeds those images to the updated autoencoder received from the aggregator, the updated autoencoder at the first computer 102a can recognize that the bird samples are not near any of the other four clusters. Anomaly detection may be performed to detect significant deviations from the four clusters for the four known classes in the feature space of the updated autoencoder. If the number of anomalies is significantly high relative to the average number of bird images input over a period of time or relative to the number of total test samples, a flag may be sent to the aggregator to alert it to the need to redefine the classes. In some embodiments, a flag may be sent to the aggregator when a first data point is detected as an anomaly to alert it to the need to redefine the classes.

[0078] The aggregator may integrate the clustering information for the new bird image cluster with other cluster information to verify that a new cluster should be created. If the aggregator is satisfied with the verification, for example due to agreement that the centroid of the bird image is not close to any of the other centroids, the aggregator may add a fifth node 608e to the deep learning model 600 and to the updated autoencoder, and may send this updated deep learning model and the updated autoencoder to all of the multiple distributed computing devices in the federated learning system, for example, the first, second and third computers 102a, 102b and 102c for the embodiment shown in Figures 1 and 5. Each distributed computing device will then again have the opportunity to provide their own semantic local definition for the fifth node 608e and its associated cluster, to accept the label or name provided by the aggregator for the fifth node 608e, or to not select a name for the fifth node 608e. This may be advantageous when the local computing device does not have a sample that falls within the cluster associated with that fifth node 608e.

[0079] In the embodiment described above with respect to Figures 5 and 6, the sample data provided to and recognized by the federated learning system includes images of animals. In an alternative embodiment of recognizing simple spoken audio phrases, the principles for following the process 200 shown in Figure 2 would be similar. A party in a federated learning system with enhanced semantic learning to recognize audio phrases may divide the phrases into groups of questions, greetings, insults, or complements. Different parties may determine the same spoken audio phrase to be an insult, greeting, or complement depending on the cultural background or cultural environment in which the particular participant / party operates. The individual parties and the aggregator may collaborate to perform steps 202-228 with respect to forming clusters and classes, deep learning models for determining the type of audio phrase, and nodes corresponding to the identified clusters. The system may rely on tone recognition and speech-to-text conversion to generate vectors for the particular spoken and recorded audio phrases, and to analyze the spoken audio phrases. Process 200 may be applied to a computing device, such as a personal computer or a smartphone, to predictive input for words. To analyze text or audio data, methods may be implemented, such as word2vec, to map raw data to numerical vectors so that a neural network can understand and analyze the data.

[0080] It may be understood that Figures 2-6 provide only examples of certain embodiments and do not imply any limitations as to how different embodiments may be implemented. Many modifications to one or more of the illustrated embodiments may be made based on design and implementation requirements.

[0081] As described in the above embodiments, the semantic learning enhanced associative learning programs 110a, 110b, 110c, and 110d may improve the functionality of a computer or computer system by allowing deep learning systems to be trained more accurately while also improving the agility of the learning system and reducing the computational power required to add model classes, as well as tuning the associative learning more efficiently.

[0082] Figure 7 is a block diagram 900 of the internal and external components of the computer shown in Figure 1 in accordance with one exemplary embodiment of the present invention. It should be understood that Figure 7 is only intended to provide an example of one implementation and is not intended to imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0083] Data processing systems 902 and 904 represent possible electronic devices capable of executing machine-readable program instructions. The data processing systems 902 and 904 may represent a smart phone, a computer system, a PDA, or other electronic device. Examples of computing systems, environments, or configurations, or combinations thereof, that may be represented by the data processing systems 902 and 904 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0084] The user client computer 102 and the network server 112 may include multiple sets of internal components 902a and 902b and external components 904a and 904b, respectively, as shown in Figure 7. Each of the multiple sets of internal components 902a and 902b includes one or more processors 906, one or more computer-readable RAMs 908 and one or more computer-readable ROMs 910, as well as one or more operating systems 914 and one or more computer-readable tangible storage devices 916 on one or more buses 912. One or more operating systems 914, software programs 108, and federated learning programs 110a, 110b, and 110c in the first, second, and third computers 102a, 102b, and 102c, respectively, and the federated learning program 110d in the network server 112 are stored on one or more computer-readable tangible storage devices 916 for execution by one or more processors 906 via one or more RAMs 908 (which typically include cache memory). In the embodiment shown in FIG. 7, each of the computer-readable tangible storage devices 916 is a magnetic disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices 916 is a semiconductor storage device, such as a ROM 910, an EPROM, a flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0085] Each set of internal components 902a and 902b also includes a R / W drive or interface 918 for reading from and writing to one or more portable computer-readable tangible storage devices 920, such as CD-ROMs, DVDs, memory sticks, magnetic tapes, magnetic disks, optical disks, or semiconductor storage devices. Software programs, such as software program 108 and associative learning programs 110a, 110b, 110c, and 110d, can be stored in one or more of the respective portable computer-readable tangible storage devices 920, read via the respective R / W drive or interface 918, and loaded into the respective hard drives 5916.

[0086] Each set of internal components 902a and 902b may also include a network adapter (or switch port card) or interface 922, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G or 4G wireless interface card, or other wired or wireless communication link. The software programs 108 and federated learning programs 110a, 110b, and 110c in the first, second, and third computers 102a, 102b, and 102c, and the federated learning program 110d in the network server computer 112, may be downloaded from an external computer (e.g., a server) via a network (e.g., the Internet, a local area network, or other wide area network) and the respective network adapter or interface 922. From the network adapter (or switch port adapter) or interface 922, the software program 108 and the federated learning programs 110a, 110b and 110c in the first, second and third computers 102a, 102b and 102c, respectively, are loaded into the respective hard drives 916. The network may comprise copper wire, fiber optic, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or combinations thereof.

[0087] Each of the sets of external components 904a and 904b may include a computer display monitor 924, a keyboard 926, and a computer mouse 928. External components 904a and 904b may also include touch screens, virtual keyboards, touch pads, pointing devices, and other human interface devices. Each of the sets of internal components 902a and 902b includes a device driver 930 for interfacing to the computer display monitor 924, the keyboard 926, and the computer mouse 928. The device driver 930, the R / W drive or interface 918, and the network adapter or interface 922 include hardware and software (stored in the storage device 916 or ROM 910, or a combination thereof).

[0088] The present invention may be a system, method, or computer program product, or combination thereof, at any level of technical detail that may be integrated. The computer program product may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0089] The computer readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a ridge structure in a groove in which instructions are recorded, or any suitable combination thereof. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0090] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing device / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may be comprised of copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing device / processing device receives the computer-readable program instructions from the network and transmits the computer-readable program instructions to the respective computing device / processing device for storage in a computer-readable storage medium.

[0091] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, such as object-oriented programming languages, such as Smalltalk, C++, and the like, conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any kind of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0092] Aspects of the present invention are described herein with reference to flowchart illustrations or block diagrams, or combinations thereof, of methods, apparatus (systems), and computer program products or computer programs according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations or block diagrams, or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams, or combinations thereof, can be implemented by computer readable program instructions.

[0093] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof, to produce a machine. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer programmable data processing apparatus or other device, or combinations thereof, to function in a particular manner, such that a computer readable storage medium having stored thereon includes an article of manufacture including instructions that implement aspects of the functions / operations specified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof.

[0094] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on a computer, other programmable data processing apparatus, or other device, implement the functions / acts identified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof, to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process.

[0095] The flowchart diagrams and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products or computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart diagrams or block diagrams may represent a module, segment, or portion of an instruction, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the drawings. For example, two blocks shown in succession may actually be accomplished as one step performed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, depending on the functions involved, or the blocks may be performed in the reverse order. It should be noted that each block of the block diagrams or flowchart diagrams or combinations thereof, as well as combinations of multiple blocks of the block diagrams or flowchart diagrams or combinations thereof, may be implemented by a special purpose hardware-based system that performs the specified functions or operations, or may execute a combination of special purpose hardware and computer instructions.

[0096] Although the present disclosure includes detailed descriptions of cloud computing, it should be understood in advance that implementation of the teachings recited herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0097] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0098] Its features are as follows.

[0099] On-demand self-service: A cloud consumer can unilaterally provision computing capacity, such as server time and network storage, as needed, without the need for human interaction with the provider of the service.

[0100] Broad Network Access: Capabilities are available over the network and accessed via standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs).

[0101] Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, and various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the provided resources, but are said to be location-independent in that they may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0102] Rapid Elasticity: Capabilities can be provisioned quickly and elastically, sometimes automatically, quickly scaled out, quickly released and quickly scaled in. To the consumer, the capabilities available for provisioning are often unlimited and can be purchased in any quantity at any time.

[0103] Measured Services: Cloud systems automatically control and optimize resource usage by using metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being utilized.

[0104] The service model is as follows:

[0105] Software as a Service (SaaS): The capability offered to consumers to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface, e.g., a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings.

[0106] Platform as a Service (PaaS): A capability offered to a consumer to deploy consumer-created or acquired applications, created using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, e.g., including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application-hosting environment configuration.

[0107] Infrastructure as a Service (IaaS): The capability offered to consumers to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over selecting network components (e.g., host firewalls).

[0108] The deployment models are as follows:

[0109] Private Cloud: Cloud infrastructure is operated exclusively for an organization. The cloud infrastructure may be managed by the organization or a third party, and may be on-premises or off-premises.

[0110] Community Cloud: Cloud infrastructure is shared by several organizations and supports a particular community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). The cloud infrastructure may be managed by the organizations or a third party, and may reside on-premise or off-premise.

[0111] Public Cloud: The cloud infrastructure is available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0112] Hybrid Cloud: The cloud infrastructure remains a unique entity, but is a hybrid of two or more clouds (private, community, or public) brought together by standardized or specialized technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0113] A cloud computing environment is service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0114] Referring now to FIG. 8, an exemplary cloud computing environment 1000 is illustrated. As illustrated, the cloud computing environment 1000 comprises one or more cloud computing nodes 800 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 1000A, a desktop computer 1000B, a laptop computer 1000C, or an automobile computer system 1000N, or the like, or combinations thereof, may communicate. The nodes 800 may communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described herein (not shown). This allows the cloud computing environment 1000 to provide infrastructure, platform, or software, or combinations thereof, as a service without the cloud consumer having to maintain resources on the local computing device. It will be understood that the types of computing devices 1000A-1000N shown in Figure 8 are intended to be illustrative only, and that computing node 800 and cloud computing environment 1000 communicate with any type of computerized device (e.g., using a web browser) over any type of network or network addressable connection or combination thereof. Cloud computing node 800 may have a similar or identical structure and internal and external components as client computer 102 shown in Figure 7 and computers 102a, 102b, and 102c shown in Figure 1.

[0115] Referring now to Figure 9, a set of functional abstraction layers provided by cloud computing environment 1000 (Figure 8) is shown. It should be understood that the components, layers and functions shown in Figure 9 are intended to be merely exemplary, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0116] The hardware and software layer 1102 includes hardware and software components. Examples of hardware components include mainframe 1104, RISC (Reduced Instruction Set Computer) architecture based servers 1106, servers 1108, blade servers 1110, storage devices 1112, and networks and networking components 1114. In some embodiments, the software components include network application server software 1116 and database software 1118.

[0117] The virtualization layer 1120 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 1122; virtual storage 1124; virtual networks 1126, including, for example, virtual private networks; virtual applications and operating systems 1128; and virtual clients 1130.

[0118] In one example, the management layer 1132 may provide multiple functions described below. Resource provisioning 1134 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 1136 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection for data and other resources. User portal 1138 provides access to the cloud computing environment for consumers and system administrators. Service level management 1140 provides allocation and management of cloud computing resources such that required service levels are met. Service level agreement (SLA) planning and fulfillment 1142 provides pre-provisioning and procurement of cloud computing resources where future requirements are predicted according to SLAs.

[0119] Workload tier 1144 provides examples of functions for which the cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 1146; software development and lifecycle management 1148; virtual classroom instruction provision 1150; data analytics processing 1152; transaction processing 1154; and semantic federated learning 1156. Federated learning programs 110a, 110b, 110c, and 110d provide a method for accurately performing federated learning even when dealing with unique semantic naming preferences for individual computing devices in the federated learning system.

[0120] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural unless the context clearly indicates otherwise. It will be further understood that as used herein, the words "comprises", "comprising", "includes", "including", "has", "have", "having", "with" and the like specify the presence of stated features, integers, steps, operations, elements, or components or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components or groups thereof or combinations thereof.

[0121] The description of various embodiments of the present disclosure is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method for associative learning, comprising: receiving cluster information from a plurality of distributed computing devices via an aggregator, where the cluster information relates to a plurality of identified clusters within the sample data of the plurality of distributed computing devices, and where the aggregator comprises at least one processor; aggregating the cluster information via the aggregator to define a plurality of classes, where the aggregating includes identifying possible redundant clusters among the identified plurality of clusters, and a number of the plurality of classes corresponds to a total number of clusters from the plurality of distributed computing devices reduced by possible redundant clusters; transmitting a deep learning model from the aggregator to the plurality of distributed computing devices, where the deep learning model comprises an output layer having a plurality of nodes, the plurality of nodes corresponding to the defined plurality of classes; and receiving, by the aggregator, one or more results of federated learning performed by the plurality of distributed computing devices, where the federated learning trains the deep learning model; The method comprising:

2. providing an autoencoder to the plurality of distributed computing devices via the aggregator; processing the sample data of each computing device through the autoencoder via each computing device of the plurality of distributed computing devices to generate an autoencoder output; processing the autoencoder output via the respective computing devices through a clustering algorithm to identify the plurality of clusters from the sample data; and transmitting the cluster information from the plurality of distributed computing devices to the aggregator; The method of claim 1 further comprising:

3. performing said federated learning via said plurality of distributed computing devices; The method of claim 1 further comprising:

4. Each cluster has a centroid, the cluster information includes centroid information, where the centroid information relates to the centroid; and identifying possible redundant clusters includes comparing the centroid information for the centroids of the plurality of clusters; The method of claim 1.

5. The method of claim 4 , wherein the aggregator defines two clusters as redundant if the distance between their centroids is less than a predefined threshold.

6. Naming a class of the plurality of classes via a computing device of the plurality of distributed computing devices according to a semantic meaning for the computing device of the plurality of distributed computing devices. The method of claim 1 further comprising:

7. anonymizing the cluster information across the plurality of distributed computing devices before the cluster information is transmitted to the aggregator. The method of claim 2 further comprising:

8. transmitting, by the aggregator, an initial autoencoder to the plurality of distributed computing devices for execution; sequentially training the initial autoencoder via the plurality of distributed computing devices to generate the autoencoder; and transmitting the autoencoder from a last computing device of the plurality of distributed computing devices to the aggregator; The method of claim 2 further comprising:

9. The sequential training comprises: training the initial autoencoder via a first computing device of the plurality of distributed computing devices to generate a partially trained autoencoder; transmitting the partially trained autoencoder from the first computing device to a second computing device of the plurality of distributed computing devices; training the partially trained autoencoder via the second computing device to generate a further trained autoencoder; transmitting the further trained autoencoder from the second computing device to an additional computing device; and training the further trained autoencoder via the additional computing device to generate the autoencoder, where the additional computing device is the final computing device; The method of claim 8 , comprising:

10. The method of claim 8 , wherein the initial autoencoder is a pre-trained autoencoder.

11. checking new classes during the federated learning via a first computing device of the plurality of distributed computing devices, where the checking includes the first computing device feeding new samples to the autoencoder, and performing anomaly detection to detect deviations of the new samples from the plurality of classes, where the deviations exceed a predefined threshold. The method of claim 2 further comprising:

12. notifying the aggregator via the first computing device regarding the deviation; adding a new class to the plurality of classes via the aggregator to form a new set of classes, where the new class corresponds to the new sample; transmitting the new set of classes from the aggregator to the plurality of distributed computing devices; and performing additional federated learning via the aggregator and the plurality of distributed computing devices to further train the deep learning model, where performing the additional federated learning includes using the new set of classes as new nodes in the output layer of the deep learning model. The method of claim 11 further comprising:

13. The method of claim 2 , wherein the clustering algorithm is part of the autoencoder.

14. A computer system for semantic learning enhanced associative learning, the computer system comprising one or more processors, one or more computer readable memories, one or more computer readable tangible storage media, and a plurality of program instructions stored on at least one of the one or more computer readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer readable memories, the computer system being capable of performing a method comprising: receiving cluster information from a plurality of distributed computing devices, wherein the cluster information relates to a plurality of identified clusters within the sample data of the plurality of distributed computing devices; aggregating the cluster information to define a plurality of classes, where the aggregating includes identifying possible redundant clusters among the identified plurality of clusters, and a number of the plurality of classes corresponds to a total number of clusters from the plurality of distributed computing devices reduced by possible redundant clusters; transmitting a deep learning model to the plurality of distributed computing devices, where the deep learning model comprises an output layer having a plurality of nodes, the plurality of nodes corresponding to the defined plurality of classes; and receiving one or more results of federated learning performed by the plurality of distributed computing devices, where the federated learning trains the deep learning model; The computer system comprising:

15. The method further comprising: Providing the autoencoder to the plurality of distributed computing devices. The computer system of claim 14 further comprising:

16. The computer system of claim 15 , wherein the autoencoder comprises a clustering algorithm.

17. Each cluster has a centroid, the cluster information includes centroid information, where the centroid information relates to the centroid; and identifying possible redundant clusters includes comparing the centroid information for the centroids of the plurality of clusters; 15. The computer system of claim 14.

18. 20. The computer system of claim 17, wherein if the distance between the centroids of two clusters is less than a predefined threshold, the computer system defines the two clusters as redundant.

19. The method further comprising: sending the initial autoencoder to a first computing device of the plurality of distributed computing devices for execution; receiving a partially trained autoencoder from the first computing device; transmitting the partially trained autoencoder to a second computing device of the plurality of distributed computing devices; receiving a further trained autoencoder from the second computing device; transmitting the further trained autoencoder to an additional computing device of the plurality of distributed computing devices; and Receiving the autoencoder from the additional computing device.

16. The computer system of claim 15, further comprising:

20. 1. A computer program for semantic learning enhanced associative learning, comprising: receiving cluster information from a plurality of distributed computing devices, wherein the cluster information relates to a plurality of identified clusters within the sample data of the plurality of distributed computing devices; aggregating the cluster information to define a plurality of classes, where the aggregating includes identifying possible redundant clusters among the identified plurality of clusters, and a number of the plurality of classes corresponds to a total number of clusters from the plurality of distributed computing devices reduced by possible redundant clusters; transmitting a deep learning model to the plurality of distributed computing devices, where the deep learning model comprises an output layer having a plurality of nodes, the plurality of nodes corresponding to the defined plurality of classes; and receiving one or more results of federated learning performed by the plurality of distributed computing devices, where the federated learning trains the deep learning model; The computer program causes one or more computers to execute each step of the method including the steps of:

Images