Methods, computer systems, and program products for federated learning

Abstract

A method, computer system, and computer program product for leveraging semantic learning-enhanced joint learning are provided. An aggregator can receive cluster information from a distributed computing device. The cluster information may relate to identified clusters in sample data from the distributed computing device. The aggregator can integrate the cluster information to define categories. This integration may include identifying any redundant clusters among the identified clusters. The number of categories may correspond to the total number of clusters from the distributed computing device minus any redundant clusters. A deep learning model can be sent from the aggregator to the distributed computing device. The deep learning model may include an output layer with nodes that may correspond to the defined categories. The aggregator can receive the results of joint learning performed by the distributed computing device. Joint learning can train the deep learning model.

Description

Background Technology

[0001] This invention relates generally to the field of deep learning models, and more specifically to federated learning for deep learning models.

[0002] In a federated learning system, a group of multiple devices or parties work together to develop and collaboratively train a deep learning model, such as a prediction model, without sharing or revealing the raw data of individual parties with other devices or parties involved in the model training. Because the more data processed, the better the model will be trained, using multiple parties and their data for deep learning training leads to better training and a better model. An aggregator receives information from various devices or parties and compiles or aggregates that information to fine-tune the deep learning model. In some cases, the aggregator averages the information from the parties to adjust the deep learning model. In a federated learning system, data from various devices in the network can be used to train the model without the devices sharing their individual raw data on their individual computing devices or phones via the cloud. The aggregator can send the adjusted deep learning model to the various devices without those devices receiving raw data from other devices. Therefore, federated learning helps maintain the privacy of the raw data of individual devices while still allowing multiple parties to work together to train a deep learning model. Federated learning prevents adversaries from refactoring data transformations to find raw training data that matches a specific party.

[0003] Bonawitz et al.'s "Towards Federated Learning At Scale System Design" describes federated learning ("FL") as a distributed machine learning approach that enables training on large amounts of dispersed data residing on devices such as mobile phones. Bonawitz et al. also state that FL addresses fundamental issues of data privacy, ownership, and locality.

[0004] U.S. Patent Application Publication No. 2019 / 0138934A1 by Prakash et al. discloses that, for joint learning, each client computing node obtains a global model, updates the global model using its local data, and transmits the updated model to a central server. However, Prakash et al. disclosed that computation should be balanced among heterogeneous computing nodes based on knowledge of the network conditions and operational constraints experienced by the heterogeneous computing nodes.

[0005] A drawback of known joint learning systems is that they are built upon the assumptions that different parties label the data, that different parties will use the same data labels, and that different parties know all possible data categories. In reality, parties often do not know the labels used by other parties. The label a first party can provide for a sample may differ from the label a second party provides for that sample. For example, some parties may consider a rabbit a pet, while others may consider it food. Sometimes, parties, such as those in hospitals, are not allowed to share their data and labels in advance. Furthermore, new samples may emerge that do not fit the static model structure well. For example, in a model used to identify food images, a local party will understand local foods such as AREPA (tacos), while other parties scattered around the world will not. Updating the model structure requires retraining all local models, which consumes significant time and energy. Summary of the Invention

[0006] According to an exemplary embodiment, a method, computer system, and computer program product can perform joint learning. An aggregator can receive cluster information from a distributed computing device. The cluster information may relate to identified clusters in sample data from the distributed computing device. The aggregator may include at least one processor. The aggregator can integrate the cluster information to define categories. This integration may include identifying any redundant clusters among the identified clusters. The number of categories may correspond to the total number of clusters from the distributed computing device minus any redundant clusters. A deep learning model can be sent from the aggregator to the distributed computing device. The deep learning model may include an output layer with nodes. Nodes may correspond to the defined categories. The aggregator can receive one or more results of joint learning performed by the distributed computing device. Joint learning can train the deep learning model.

[0007] Using this embodiment, even if different parties give different names to the same sample type, the labels used for data samples can be accurately and appropriately codified in the deep learning system. The deep learning model in joint learning is customized to adapt to the semantic meaning of different participants, allowing each participant to independently use its correct and unique semantic label while still contributing to the joint learning system.

[0008] In an additional exemplary embodiment, the aggregator may provide an autoencoder to the distributed computing devices. Each computing device within the distributed computing devices can run its own sample data using the autoencoder to generate autoencoder output. Each computing device can then run the autoencoder output using a clustering algorithm to identify clusters from the sample data. Cluster information can be sent from the distributed computing devices to the aggregator.

[0009] Using this embodiment, joint learning systems can be improved to have improved accuracy, thereby correctly identifying the labels and categories of data samples fed into the system.

[0010] Another exemplary embodiment may additionally include the step of naming categories in a category according to the semantic meaning of the computing device via a computing device in a distributed computing device.

[0011] Using this embodiment, individual participants, as part of a system of distributed computing devices in a federated deep learning network, can provide their own semantic definitions to label groups shared by other participants in the network.

[0012] Additional exemplary embodiments may include the step of anonymizing cluster information via a distributed computing device before the cluster information is sent to the aggregator.

[0013] This embodiment can maintain the privacy of customer data, allowing parties to participate in collaborative learning without exposing their private customer and other individuals' raw data.

[0014] Another exemplary embodiment may additionally include a step of checking for new categories via a first computing device in a distributed computing device during joint learning. This check may include the first computing device feeding new samples to an autoencoder and performing anomaly detection to detect new samples deviating from the category. This deviation exceeds a predefined threshold.

[0015] In this way, new categories that emerge during joint learning can be added to the deep learning model in a faster manner that saves computing resources. Attached Figure Description

[0016] These and other objects, features, and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is taken in conjunction with the accompanying drawings. The various features in the drawings are not to scale, as the illustrations are intended to clearly assist those skilled in the art in understanding the invention in conjunction with the detailed description. In the drawings:

[0017] Figure 1 A networked computer environment according to at least one embodiment is shown;

[0018] Figure 2 This is an operation flowchart illustrating the workflow of a joint learning process enhanced by semantic learning according to at least one embodiment;

[0019] Figure 3 The structure of an autoencoder and clustering technique implemented according to at least one embodiment is shown;

[0020] Figure 4 An example of cluster information integration according to at least one embodiment is shown;

[0021] Figure 5 A networked computer environment according to at least one embodiment is shown, wherein the various computing devices of the joint learning network are loaded with data samples organized into clusters;

[0022] Figure 6 An example of a layer in a deep learning model is shown;

[0023] Figure 7 According to at least one embodiment Figure 1 A block diagram depicting the internal and external components of a computer and server;

[0024] Figure 8 Includes embodiments according to this disclosure Figure 1 A block diagram illustrating a cloud computing environment for a computer system; and

[0025] Figure 9 According to embodiments of this disclosure Figure 7 A block diagram illustrating the functional layers of an illustrative cloud computing environment. Detailed Implementation

[0026] This document discloses detailed embodiments of the claimed structures and methods; however, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which can be implemented in various forms. The invention can be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0027] The exemplary embodiments described below provide a system, method, and program product for leveraging semantic learning to enhance joint learning. This embodiment has the ability to accurately and appropriately code the labels of each participant or party in a joint learning system, even if some samples have different semantic meanings for different participants or parties. This embodiment also enables the addition of new categories to a deep learning model in an accelerated manner without requiring a complete retraining across all parties. Therefore, this embodiment saves computational and training resources for the joint learning system and allows the joint learning system to improve its adaptability to the private semantic labels and unique samples of each participating party.

[0028] refer to Figure 1This document describes an exemplary networked computer environment 100 according to one embodiment. In some embodiments, the networked computer environment 100 may be considered a federated learning system. The networked computer environment 100 may include multiple computers, namely a first computer 102a, a second computer 102b, and a third computer 102c. Figure 1 Three such client computers are shown in the networked computer environment 100, but in reality, there can be many more such client computers in the networked computer environment 100. See below for reference. Figure 7 As explained, each of the first computer 102a, the second computer 102b, and the third computer 102c may include one or more processors and memories capable of running and storing the co-learning programs 110a, 110b, and 110c. The networked computer environment 100 may also include a server 112 capable of running the co-learning program 110d, which interacts with the database 114 and the communication network 116. The networked computer environment 100 may include multiple servers 112, only one of which is shown. The communication network 116 may include various types of communication networks, such as wide area networks (WANs), local area networks (LANs), telecommunications networks, wireless networks, public switched networks, and / or satellite networks. It should be understood that... Figure 1 This is merely an illustration of an implementation and does not imply any limitation on the environments in which different embodiments may be implemented. Many modifications can be made to the described environment based on design and implementation requirements.

[0029] First computer 102a, second computer 102b, and third computer 102c can communicate with server computer 112 via communication network 116. Communication network 116 may include connections such as wired, wireless communication links, or fiber optic cables. (See reference...) Figure 7As discussed, server computer 112 may include internal component 902a and external component 904a, and first client computer 102a, second client computer 102b, and third client computer 102c may each include internal component 902b and external component 904b. Server computer 112 may also operate in a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). Server 112 may also reside in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud. First computer 102a, second computer 102b, and third computer 102c may be, for example, mobile devices, telephones, personal digital assistants, netbooks, laptops, tablets, desktop computers, or any type of computing device capable of running programs, accessing networks, and accessing database 114. According to various implementations of this embodiment, the joint learning programs 110a, 110b, 110c, and 110d can interact with a database 114 that can be embedded in various storage devices, such as, but not limited to, a computer / mobile device 102, a network server 112, or a cloud storage service.

[0030] According to this embodiment, a user using a client computer or server computer 112 can use joint learning programs 110a, 110b, 110c, and 110d to perform joint learning enhanced by semantic learning to train a deep learning model. See below for reference. Figures 2-9 This section explains in more detail the joint learning method that leverages semantic learning enhancement.

[0031] Server computer 112 can be considered an aggregator as part of a federated learning system. For federated learning, the aggregator can issue queries to all available parties in the federated learning system; for example, the aggregator can... Figure 1 and Figure 5Each of the first computer 102a, the second computer 102b, and the third computer 102c in the networked computer environment 100 shown issues a query. Each party has its own dataset. In the federated learning system, parties cannot see the raw data of other parties in the system. Each party may have its own local learning model. The query made by the aggregator may be an inquiry about what the current model parameters of the local learning model are for each party. The query may cause each party to provide some information to the aggregator. The aggregator may ask what the answer to a specific question is based on the party's dataset. Each party may generate its own answer based on its own local data. In some cases, each party may have a local learning model that is stored and operated on within the client computer, which helps generate the response. Each party generates its response and sends it back to the aggregator. After the aggregator has received responses from all parties or from enough parties to exceed a threshold, the aggregator performs an aggregation or combination of all responses and uses the result to update the machine learning model maintained by the aggregator and stored and operated on in the aggregator's computer. After several of these training / tuning sessions or loops have been executed, the aggregator then produces the final machine learning model, such as a global model or neural network, and shares it with all parties, for example, with the first computer 102a, the second computer 102b, and the third computer 102c. The raw data remains at each party's site. This is for implementations in a federated learning system. Figure 1 In the embodiment shown, server computer 112 is configured to submit queries to first computer 102a, second computer 102b, and third computer 102c to perform the aggregation step to generate a global model or neural network, and then send the global model or neural network to the first computer 102a, second computer 102b, and third computer 102c.

[0032] Federation learning can be implemented, for example, in IoT (Internet of Things) networks or in smartphone networks. In these cases, many parties can participate in the training. Data from various devices in the network can be used to train deep learning models, without the devices needing to share individual raw data on their respective devices or smartphones via the cloud. The raw data will include identity data of the parties. Anonymous data, where identity data has been removed, may not be considered raw data, at least in some embodiments.

[0033] In another possible implementation, competitors in the market can work together and use joint learning to train the model while protecting the privacy of their customers' information. Compared to the systems described earlier, this scenario of competitors working together can involve fewer parties helping to train the system. In this case, for example, joint learning could be used by multiple banks to train the system to detect money laundering, without the banks having to share or disclose the individual raw banking data of individual customers.

[0034] In another scenario, federated learning can also be implemented when individual devices have connectivity constraints and few opportunities to share their information to the cloud. For example, a robot on Mars doesn't have many data transfer opportunities to transmit its data to Earth or a satellite orbiting Earth. Instead of transmitting its entire compilation of the raw data, a local device can train its model locally and then transfer its trained model during its limited transfer opportunities. The local dataset, including features and labels, will be maintained by the owner / individual party, and other entities in the federated learning system will not have access to the local datasets of other parties in the system.

[0035] Joint learning systems are implemented in certain situations, such as predictive typing or predictive speech. They can also be implemented for classifying images or audio signals.

[0036] Joint learning systems can protect the raw data of each party from being shared with other parties by using secure aggregation. When a party has a unique sample to add to a deep learning model, this information can be anonymously provided to the aggregator. Therefore, the aggregator may be able to identify where the sample comes from in the joint learning system, without knowing which specific party or machine within the system it originated from. Each party can share its local model and / or model parameters or weights based on the size of the training samples with the aggregator. Each party can also share gradients computed based on its local dataset with the aggregator. If these parameters or weights or gradients are shared separately with the aggregator, the raw data, including features and individual labels, can remain private on that party's computing device.

[0037] Now for reference Figure 2 The diagram illustrates an operational flowchart of an exemplary process 200 for leveraging semantic learning to enhance deep learning. In this process 200, joint learning procedures 110a, 110b, 110c, and 110d will take actions according to at least one embodiment.

[0038] like Figure 2 As shown, process 200 includes an exemplary deep learning process enhanced with semantic learning. Process 200 may begin in step 202 by sending an autoencoder to the computing device of the federated learning system via an aggregator. The autoencoder may be sent via communication network 116, as referenced above. Figure 1 The communication network under discussion may include various types of communication networks, such as wide area networks (WANs), local area networks (LANs), telecommunications networks, wireless networks, public switching networks, and / or satellite networks, and may include connections such as wired, wireless communication links, or fiber optic cables.

[0039] In at least some embodiments, the autoencoder may be a pre-trained autoencoder. For training a deep learning model, the total number of categories should be fixed and static, and definitions of those categories should be provided. Therefore, for pre-training an autoencoder, the operator may input multiple categories and their definitions based on experience with previous deep learning models or on educated guesses about the potential samples expected to be received and analyzed by the deep learning model. However, training the autoencoder can be unsupervised, so that no labeling is required during training. When an image is fed into the autoencoder, the encoder extracts high-level features from the image. The decoder can use the extracted features to reconstruct the image. The encoder and decoder together help form the autoencoder. Using a pre-trained autoencoder helps reduce training time and improve the performance of the autoencoder. In some cases, the autoencoder can be pre-trained without knowing the number of categories. Using a pre-trained autoencoder facilitates the generalization of the autoencoder.

[0040] The following is for reference Figure 3 The described autoencoder may include a deep learning model. The autoencoder receives data as input and uses a deep learning model to extract a high-level feature vector from each data sample. The deep learning model may be a neural network, such as a convolutional neural network.

[0041] In step 204 of process 200, the various computing devices of the joint learning system, such as Figure 1 The first computer 102a, the second computer 102b, and the third computer 102c shown train the autoencoder received in step 202, and thus generate a trained autoencoder. The following will describe... Figure 3 An example of an autoencoder and its components is shown. This training of the autoencoder is performed by having each party run or feed its samples or sample data into the autoencoder as input to the autoencoder's learning model. Then, a loss function is applied to each party to gradually adjust the parameters of the autoencoder's learning model. The loss function is run and a loss amount is generated; the parameters are gradually adjusted to attempt to reduce the loss amount, and the loss function is run again. The loss function measures the autoencoder's performance.

[0042] Step 204 may include sequential training of the autoencoder by various computing devices, such as a first computer 102a, a second computer 102b, and a third computer 102c of the joint learning system. In sequential training, the autoencoder may first be sent to a specific computing device, such as the first computer 102a, which first trains the autoencoder to form a partially trained autoencoder. The first computer 102a may then send this partially trained autoencoder to another computing device, such as the second computer 102b. This other computing device may further train the partially trained autoencoder by feeding its own data into a deep learning model and repeatedly running a loss function for guidance to gradually adjust the parameters of the deep learning model. Due to this further training, the other computing device will produce a further trained autoencoder.

[0043] This transfer of partially trained autoencoders from one computing device to another can occur directly via the communication network 116, or via an aggregator that receives the partially trained autoencoders and forwards them to the next computing device. The latter often occurs because in many joint learning systems, the parties do not know who the other parties are, and therefore will not know how to send their partially trained models to the other party. Therefore, in this embodiment, the aggregator will be used to facilitate the transfer, as it will know and store information about who the other parties / computing devices are in the joint learning system. Whether the aggregator acts as a facilitator of the transfer, or whether the parties know the other parties and are able to transfer directly, this autoencoder training performed by one party after another can be considered sequential training.

[0044] The second computer 102b can transfer or send the further trained autoencoder to an additional computing device, such as a third computer 102c. This transfer can occur directly or via the aggregator described above, and either method can utilize the communication network 116. The additional computing device can feed its samples into this further trained autoencoder and run a loss function to guide the gradual adjustment of the deep learning model parameters. This additional training can produce a trained autoencoder or an autoencoder ready for distribution to all parties.

[0045] In some embodiments, such training or sequential training of the autoencoder can be performed on each computing device or each party of the joint learning system before the aggregator requests the autoencoder. Alternatively, the aggregator can request the autoencoder after it has been trained by most devices / party of the joint learning system. Using training performed by a majority of parties will save some time and resources compared to embodiments where all parties help train the autoencoder.

[0046] During sequential training of an autoencoder, the autoencoder itself (including its neural network / learning model) is passed from one party to another, but the raw data, including the original features and original labels, is not passed. Sequential training of the autoencoder by each party can also be considered as collaborative training of the autoencoder by each party.

[0047] In step 206, the computing device sends the trained autoencoder back to the aggregator. This transmission can occur via communication network 116. In at least some embodiments, the final computing device, for example... Figure 1 The third computer 102c shown will send the autoencoder back to the aggregator. If the aggregator requests the autoencoder after most, but not all, parties have trained it, the party that sends the autoencoder to the aggregator will not be the last party among all parties to train the autoencoder.

[0048] In at least some embodiments, clustering techniques are not required or performed for all steps 202-206 related to the training of the autoencoder. In these embodiments of process 200, sample data does not need to be submitted via a clustering algorithm in steps 202-206.

[0049] After the aggregator receives the trained autoencoder from the computing devices of the joint learning system, in step 208, the aggregator shares the trained autoencoder with the computing devices (e.g., with all computing devices in the joint learning system). This sharing can occur via communication network 116. This sharing of the finally trained autoencoder with all parties provides all parties with the benefit of complete training of the autoencoder performed by all parties, a majority of parties, or a certain number of parties, as training of the autoencoder occurs sequentially by each party. This sharing provides all parties with an improved autoencoder compared to an autoencoder trained on data by only one party.

[0050] In at least some embodiments, the autoencoder will include a clustering algorithm. In various embodiments, the clustering algorithm may be a K-means clustering algorithm, a mean-shift clustering algorithm, a density-based spatial clustering algorithm for noisy applications (DBSCAN), an expectation-maximization (EM) clustering algorithm using a Gaussian mixture model (GMM), or agglomerative hierarchical clustering.

[0051] In step 210, each computing device, such as each of the first computer 102a, the second computer 102b, and the third computer 102c, runs its sample data through a trained autoencoder to produce an autoencoder output. The autoencoder output can be a high-level representation of the input data, and specifically, it can be a vector. The vector can include multiple variables or parameters, such as three or more, or even up to 100 or more. These variables or parameters can be referred to as feature values. In a deep learning model for recognizing images of animals, the autoencoder can identify various features about each image, such as size, number of appendages, ear shape, etc., which can help the deep learning model classify the animal. These features can be variables or parameters determined by feeding samples through an autoencoder that analyzes the samples. In embodiments where images are fed to the autoencoder, the autoencoder analyzes the image and can analyze the pixels of the image.

[0052] In step 212, each computing device, such as each of the first computer 102a, the second computer 102b, and the third computer 102c, runs an autoencoder output using a clustering algorithm to identify clusters in the sample data. The clustering algorithm may reside in each computing device. Alternatively, the clustering algorithm may reside in the autoencoder received from the aggregator. In some embodiments, the clustering algorithm may be a K-means clustering algorithm, a mean-shifted clustering algorithm, density-based spatial clustering with noisy applications (DBSCAN), expectation-maximization (EM) clustering using a Gaussian mixture model (GMM), or agglomerative hierarchical clustering. Figure 3 An example of an autoencoder is shown, which is paired with features passed through a clustering algorithm to the autoencoder output. Figure 3 The upper part shows an example of clusters generated from the output samples by running a clustering algorithm.

[0053] The autoencoder output feed is processed by a clustering algorithm to generate corresponding data points for each autoencoder output, representing data points for each sample. These data points can be plotted. For example, in an autoencoder model with three features, a 3D plot can be used to plot the data points. Data points that generally fall together on the 3D plot are generally considered to belong to a specific data cluster. In at least some embodiments, each cluster of sample data or autoencoder output will have a centroid, i.e., the center of the cluster, and will have a radius. Data points in the plot that are less than the radius from the centroid are considered to belong to that cluster. Figure 3The upper part shows an example of multiple clusters of data points, and illustrates the centroid and radius of a specific cluster. For simplicity and to more easily explain the concept, an autoencoder model with three features that can be paired with a 3D graph in a clustering algorithm is used as an example. In many autoencoder models that will be used, the autoencoder model will use far more than three features. For example, an autoencoder model may include up to one hundred or more features. Clustering techniques can still be implemented using these models with even more features.

[0054] In step 214 of process 200, the parties, such as the first computer 102a, the second computer 102b, and the third computer 102c, perform anonymization techniques on the cluster information generated by the clustering algorithm. This anonymization results in the generation of a private representation of the autoencoder output. The anonymization technique can be a generalized technique. For anonymization techniques, identity attributes, such as explicit identifiers, can be removed from the cluster information. Therefore, when the aggregator receives an anonymized set of cluster information from one party of the joint learning system, the aggregator may not be able to identify which party sent the set of cluster information, even though the aggregator may be able to identify that the sender belongs to their joint learning system. For example, any name of the party that generated the cluster information can be removed from the cluster information before it is sent to the aggregator.

[0055] Various anonymization techniques can be performed according to step 214. In suppression anonymization, tuples or attribute values ​​are replaced with special anonymization symbols such as "**". Therefore, through suppression anonymization, the original data values ​​are replaced with a certain anonymous value throughout the cluster information. In generalized anonymization, attribute values ​​are replaced with semantically invariant but less specific values. For example, if participant identification information includes information about the location of a participant, the participant's city or state can be replaced with the country of the participant's location. In bucketing anonymization, sensitive information is preserved but separated or isolated from any identification information. Therefore, using bucketing anonymization, the aggregator can receive features from the sample data but will not be able to identify which party some specific sample data comes from. In perturbation anonymization, sensitive information is not deleted but is randomly altered. Therefore, the altered information received by the aggregator is incorrect, and the aggregator knows which information has been altered, but external parties will not know which information has been altered. Slice anonymization can be performed by partitioning the information horizontally and vertically into columns / rows and then randomly sorting the partitions, so that the final sent information has the characteristic of being grouped together, but not in a fully linked set of information. Slice anonymization can be further supplemented with encryption of sensitive features.

[0056] In step 216 of process 200, the parties, such as the first computer 102a, the second computer 102b, and the third computer 102c, send cluster information to the aggregator, for example, to the server computer 112. The parties may use the communication network 116 to send this cluster information. The cluster information may have been anonymized before being sent.

[0057] In step 218 of process 200, the aggregator, such as server computer 112, integrates cluster information to define multiple categories. This integration is one way of processing cluster information. The integration performed by the aggregator may include identifying any redundant clusters in the identified clusters, and may include the number of defined categories corresponding to the total number of clusters from the distributed computing devices minus the characteristics of any redundant clusters. In at least some embodiments, the cluster information may include centroid information related to the centroids of the clusters, and the aggregator may compare the centroid information to identify any redundant clusters. For example, if the centroids from each party are within a distance less than a predefined new cluster threshold distance, the aggregator may consider the centroids to belong to redundant clusters that should be merged or merged based on the count of categories. Additionally or alternatively, if the radius of a first cluster overlaps with the radius of a second cluster, the aggregator may consider the centroids with the corresponding radii to belong to redundant clusters that should be merged based on the count of categories. The aggregator may determine the distance between each received centroid and each other centroid that is part of the cluster information of all groups in the joint learning system. Moreover, the aggregator may run new clustering techniques on the cluster information to identify the total number of clusters. This new clustering is particularly possible if the cluster information has been anonymized in step 214, because the aggregator will be able to access the feature information, although it will not know from which sources the specific feature information comes. For the K-means clustering technique performed here, there is no need to choose a predefined threshold distance between centroids, because the K-means clustering algorithm can solve and determine which groups of data points constitute separate clusters.

[0058] In step 220 of process 200, an aggregator, such as server computer 112, sends a deep learning model incorporating multiple categories to all computing devices in the federated learning system, such as all computing devices in the first computer 102a, the second computer 102b, and the third computer 102c. The aggregator may use communication network 116 to send the deep learning model. In at least some embodiments, the deep learning model may be a neural network, such as a convolutional neural network. The deep learning model includes an output layer with nodes. The nodes correspond to the defined categories defined in step 218. Figure 6Examples of layers in a deep learning model, including an output layer with nodes, will be discussed and illustrated below. Due to the integration from step 218 and the insertion of category data into the model sent in step 220, the parties can be considered to be collaboratively training the deep learning model. In the first instance of performing step 220, the aforementioned deep learning model will typically be sent alone. As part of this step 220 in a subsequent loop of execution process 200, in addition to sending an updated deep learning model, the aggregator may also send an updated autoencoder in which the new categories have been incorporated. Step 226, described below, explains that if a new category is discovered later, it may be necessary to send another updated autoencoder to the parties.

[0059] In step 222 of process 200, computing devices, such as first computer 102a, second computer 102b, and third computer 102c, assign new semantic meanings to at least some of the multiple categories. In an example of learning a model to recognize images such as animal pictures, if one party considers a rabbit to be food and the aggregator has already identified rabbits as one of the categories in the model, that party can assign "food" as the label for that category after receiving the category from the aggregator. If another party in the same joint learning system considers a rabbit to be a pet, that party can assign "pet" as the label for that category after receiving the category from the aggregator. Even if the first party and the other party have unique and different semantic labels for the same sample category, the system will still run to perform joint learning.

[0060] In step 224 of process 200, computing devices, such as first computer 102a, second computer 102b, and third computer 102c, perform joint learning using multiple categories. Therefore, the computing devices receive new samples and feed these new samples into a deep learning model received from an aggregator. The computing devices periodically transmit the results of their deep learning training to the aggregator. This transmission of deep learning results can occur via communication network 116. As described above, joint learning occurs when parties in the system share data to train the model / system while still retaining the privacy of their original data. Anonymous data can be sent instead of the original data for joint learning. Thus, some information is shared to enable model training, but the original sample data is not shared with the aggregator or other parties. As part of step 224, the aggregator can receive one or more results of the joint learning performed by the distributed computing devices.

[0061] In step 226 of process 200, the parties periodically and locally check to determine if a new cluster has been added. To perform this check, the parties can feed new samples into the updated autoencoder they received from the aggregator in step 220. Anomaly detection is performed in the autoencoder feature space to detect any new data point / feature that deviates significantly from clusters of known categories. For anomaly detection, the distance from the new sample's data point to the centroids of other clusters can be measured. When this distance exceeds a predefined threshold, the new data point can be considered an anomaly. The distance from the new sample's data point to the nearest other known data point in other clusters can be measured, and when this distance exceeds a predefined threshold, the new data point can be considered an anomaly. Other clustering algorithms can be used to perform anomaly detection. Many anomaly detection techniques, such as density-based techniques, can be implemented. When the number of anomalies is significantly high, or relative to the average number of anomalies over a period of time or relative to the total number of test samples, a flag or warning message can be sent to the aggregator to warn it that the categories need to be redefined. Alternatively, when a party locally identifies a single new data point outside of existing categories, it can send a flag or warning message to the aggregator to alert it that the categories need to be redefined. If a sample deviates from an existing category by more than a predefined threshold, the sample can be considered to belong to a new cluster. Information about the new data category itself can be sent along with the warning signal. This new data category information can be used to notify the aggregator and can be used by the aggregator to help retrain the autoencoder starting from step 202, or the new data category information can skip the first autoencoder training and be included as part of the re-integration of cluster information to define a new set of categories in step 218. This cluster information about new potential clusters can be anonymized by the party before being sent to the aggregator along with the warning flag. In other words, step 214 can be repeated at this point before cluster information about new possible clusters is sent to the aggregator. The aggregator can add new categories to the categories to form a new set of categories. The new categories correspond to new samples. The new set of categories can be sent from the aggregator to the distributed computing device. Additional joint learning can then be performed via the aggregator and the distributed computing device to further train the deep learning model. Performing additional joint learning involves using a new set of categories as new nodes in the output layer of the deep learning model.

[0062] To perform step 226, data samples are fed into the updated autoencoder. To perform step 224, data samples are fed into the deep learning model received from the aggregator. Thus, at some instances and times during process 200, data samples can be fed into both the updated autoencoder and the deep learning model in parallel.

[0063] In step 228 of process 200, a check is performed regarding whether deep learning should be stopped for the entire system or only for a specific device within the federated learning system. If the check results in a positive stop decision, deep learning and federated learning can be stopped for the entire federated learning system as a whole, or for individual devices within the federated learning system. If the check results in a negative stop decision, the process can loop back to step 224, where federated learning continues, and then loop back to step 226, where another check is performed for the new cluster. By default, a positive response to step 228 is given to continue deep learning / federated learning. As part of step 228, federated learning procedures 110a, 110b, 110c can generate and provide a graphical user interface (GUI) prompt to ask the user of the computing device whether they want to stop deep / federated learning. If the user provides a positive response to the GUI, the individual computing device can send a message to the aggregator to notify the aggregator of the positive response. In many embodiments, the identification of new tags / categories will be an automated process performed by the individual computing device. Identification can be the result of new data points being added to the training dataset, some data samples being consistently misclassified, or one party deciding, based on external knowledge, that they want to increase the number of local clusters.

[0064] Using process 200, models can be trained during joint learning, allowing semantic labels to be provided by each party, while avoiding potentially costly additional manual intervention or preprocessing. Process 200 can also be implemented for two different domains that have already collected samples of interest but do not have the same labels.

[0065] An autoencoder can be, for example, a vanilla autoencoder, a sparse autoencoder, a multilayer autoencoder, or a convolutional autoencoder. In at least one embodiment, the autoencoder can be a feedforward, a non-recurrent neural network with an input layer, an output layer, and one or more hidden layers connecting the input layer to the output layer. The output layer can have the same number of nodes as the input layer. The output layer helps reconstruct the input. In at least some embodiments, the autoencoder can be considered an unsupervised learning model that does not require labeled input to perform deep learning.

[0066] Figure 3 The structure of an auto encoder 300 used in at least one embodiment of the invention is shown. The auto encoder 300 typically includes an encoder 304 and a decoder 317 for encoding and decoding sample data 302.

[0067] Encoder 304 encodes sample data 302 to generate vectors and embeds the sample data 302 into a latent space. The vectors are high-level representations of the sample data 302. In one embodiment, the sample data 302 is an image of an animal, and the autoencoder generates vectors that classify features of the animal image. Encoder 304 includes a first filter 306a, a second filter 306b, and a third filter 306c as input layers to autoencoder 300. As the sample data progresses through the filter series of first filter 306a, second filter 306b, and third filter 306c, the sample data gradually decreases. In flattening layer 308, autoencoder 300 flattens the output of encoder 304 to the desired dimension for output. Flattening layer 308 produces vectors that are fed into embedding layer 310. Embedding layer 310 can generate embedding points, which can be fed into clustering algorithm 311 to produce a cluster map 312. The embedding points can be considered as autoencoder outputs. In at least some embodiments, the dimensionality of the autoencoder output is reduced before the autoencoder output passes through clustering algorithm 311.

[0068] The embedded data can be processed by clustering algorithm 311 to generate a clustering graph 312. Clustering algorithm 311 may be included as part of autoencoder 300 or may be a separate component of an individual host computing device running the autoencoder. In some embodiments, the clustering algorithm layer may be connected to the embedding layer 310. Figure 3 The clustering graph 312 shown was generated using the K-means clustering algorithm. When using the K-means clustering algorithm, a value of k can be selected or computed as a heuristic representing the expected number of clusters. Adjustments using the k value can then be performed as needed to reduce the loss function. In one example, k is selected or computed to be ten. When k is selected, the k value can be input as an independent variable into the clustering algorithm. At least one embodiment may include a fuzzy k-means clustering method, as deep clustering may require prior knowledge of the number of clusters. When calculating k using the algorithm, partition entropy algorithms, partition coefficient algorithms, or other algorithms can be used.

[0069] The vector can be passed back through the extension layer 315 and then fed into the decoder 317, which has a first decoding layer 318a, a second decoding layer 318b, and a third decoding layer 318c. Feeding the extended vector through the first decoding layer 318a, the second decoding layer 318b, and the third decoding layer 318c helps reconstruct the original input data. The reconstructed image is output as the output sample dataset 320.

[0070] Figure 3 A clustering graph 312 according to at least one embodiment is shown. This clustering graph can be generated by individual computing devices of a federated learning system. By generating the clustering graph, the individual computing devices can generate cluster information for each cluster. For example, in Figure 2 In step 216 of the illustrated process 200, cluster information can be passed to the aggregator. Alternatively, as... Figure 4 As shown, after the aggregator receives cluster information from the various computing devices of the federated learning system, the aggregator generates a system clustering graph 400.

[0071] Figure 3 The clustering diagram 312 shown at the top illustrates the generation of ten clusters, namely clusters 314a-j, when one party feeds its samples into the autoencoder 300. Cluster 314a is shown as having a centroid 316 and a radius 318. All sample points / vectors whose distance from the centroid 316 is less than the length of the radius 318 can be considered to belong to cluster 314a. The end of the radius 318 represents the boundary of cluster 314a. It is possible that no sample point / vector belonging to cluster 314a lies at the centroid 316, because the centroid 316 is the average of the sample points / vectors belonging to cluster 314a. Although in Figure 3 The diagram only shows cluster 314a as having a centroid 316 and a radius 318, but in reality, all other clusters 314b-j will also have their own corresponding centroids and their own corresponding radii. Figure 2 In step 216 of the illustrated process 200, cluster information about clusters 314a-j, as well as about all their centroids and radii, can be sent from individual computing devices to the aggregator. This feature of using centroid information as part of the cluster information, rather than using individual raw data samples, helps protect the privacy of each party's individual raw data. Even if the aggregator can use the centroid information to reconstruct the centroids, the aggregator will still typically not be able to find the individual data points or match them with any particular party participating in the joint learning system.

[0072] Figure 4 An example of a system clustering graph 400 is shown, where cluster and / or cluster information of all devices from the joint learning system has been combined or integrated together by an aggregator. Figure 4 In the illustrated embodiment, the aggregator initially integrates nine clusters received from either the entire distributed system or all individual computing devices. To prevent the raw data from being shared with other parties, each computing device has passed centroid and radius information to the aggregator, but not the individual data points from cluster graph 312, nor the individual raw data. Therefore, Figure 3 The centroids and radii of the clusters are shown, rather than the individual data points. Each computing device can also transmit information about the number of data points belonging to each cluster. Figure 3The diagram shows centroids 402a-402i. Each centroid 402a-402i will also have its own radius, although for simplicity, radii 404a, 404b, 404c, and 404h are shown as associated with centroids 402a, 402b, 402c, and 402h. The aggregator will check, and in one embodiment, using a specific threshold, two clusters associated with centroids 402b and 402c will be identified as redundant to each other because the distance between their respective centroids 402b and 402c is less than the threshold. In some embodiments, the threshold can be selected as 0.5, 1.0, or 1.5. Alternatively, the aggregator can identify that the radius 404b of the cluster associated with centroid 402b overlaps with the radius 404c of the cluster associated with centroid 402c, and for this purpose, the two clusters associated with centroids 402b and 402c can be considered redundant to each other.

[0073] Therefore, when generating a deep learning model to send to all parties or participants in the joint learning system or to individual distributed computing devices, as occurs in steps 218 and 220, the aggregator combines the two clusters associated with the two centroids 402b and 402c into a single cluster or a single centroid. The aggregator can simply specify a new centroid to be placed at an intermediate point between the two centroids 402b and 402c. Alternatively, the aggregator can assign a larger weight to the centroid with a larger number of data points in the two centroids 402b and 402c. Thus, the aggregator will... Figure 4 The deep learning model shown in the embodiment provides a total of nine output nodes in its output layer, which in some cases will be referred to as the logit layer. These nine output nodes correspond to the total number of clusters identified by the various parties or by the various distributed computing devices, but this number is reduced by one because the clusters and their centroids 402b and 402c are considered redundant and are considered to be the same cluster.

[0074] Figure 4 The figures show that radii 404a and 404h are larger than radii 404b and 404c. The size of the radius can depend on the distribution of the data points and the presence of other clusters or centroids nearby.

[0075] Figure 5 It shows Figure 1The networked computer environment 100 is shown, but in this example, the first computer 102a, the second computer 102b, and the third computer 102c are loaded with image sets from data samples. In this embodiment, a deep learning model will be trained to recognize animal pictures. Each image will have its own label provided by the individual computing device. The first computer 102a has a first computer first image set 502a, a first computer second image set 502b, and a first computer third image set 502c. In the illustrated embodiment, the first computer first image set 502a is a collection of cat pictures. The first computer second image set 502b is a collection of rabbit pictures, but the participant operating the first computer 102a considers rabbits to be "food". The first computer third image set 502c is a collection of dog pictures. The second computer 102b has a second computer first image set 504a, a second computer second image set 504b, and a second computer third image set 504c. In the illustrated embodiment, the second computer first image set 504a is a collection of cat pictures. The second computer's second image group 504b is a collection of rabbit pictures, but the participant operating the second computer 102b considers rabbits to be "pets." The second computer's third image group 504c is a collection of fish pictures. The third computer 102c has a third computer first image group 506a and a third computer second image group 506b. In the illustrated embodiment, the third computer first image group 506a is a collection of dog pictures. The third computer second image group 506b is a collection of fish pictures.

[0076] These image groups are formed by various computing devices, such as a first computer 102a, a second computer 102b, and a third computer 102c, running individual data samples of their images through a trained autoencoder received from an aggregator, for example from a server computer 112 having a joint learning program 110d. The autoencoder provides autoencoder output, which, when fed through a clustering algorithm 311, generates clusters 314 representing the image groups.

[0077] When cluster information about cluster 314 is passed to the aggregator, the aggregator integrates the cluster information to identify the total number of categories for the deep learning model. Although the cat images from the first image group 502a of the first computer are different from those from the first image group 504a of the second computer, resulting in different data points and cluster centroids for the two received clusters, the aggregator identifies these two clusters as overlapping and redundant, and combines them into a single cluster. Similarly, although the fish images from the third image group 504c of the second computer and the second image group 506b of the third computer are different, resulting in different data points and cluster centroids, the aggregator identifies these two clusters as overlapping and redundant, and combines them into a single cluster. Likewise, although the dog images from the third image group 502c of the first computer and the first image group 506a of the third computer are different, resulting in different data points and cluster centroids, the aggregator identifies these two clusters as overlapping and redundant, and combines them into a single cluster.

[0078] Rabbit images from the second image group 502b of the first computer and the second image group 504b of the second computer are similar or identical to each other, but the first computer 102a names its rabbit group "food" and the second computer 102b names its rabbit group "pets". The aggregator can analyze cluster information, including the centroid of the data samples and optionally the radius and number of data samples, to identify that the first computer's second image group 502b and the second computer's second image group 504b belong to the same cluster because these two clusters overlap and are redundant. The aggregator combines these two groups into a single cluster. Alternatively, in the first transmission of cluster information from the first computer 102a, the second computer 102b, and the third computer 102c to the aggregator, the clusters are not labeled, such that even if the operators of the first computer 102a and the second computer 102b have unique semantic meanings for their groups / clusters, the aggregator correctly identifies cluster overlap and redundancy by analyzing and integrating the cluster information. In this respect, it is not necessary to perform category labeling for training the autoencoder 300.

[0079] The aggregator identifies a total of eight clusters received from the distributed computing devices (in this case, from the first computer 102a, the second computer 102b, and the third computer 102c) – three from the first computer 102a, three from the second computer 102b, and two from the third computer 102b. Although the total number of received clusters is eight, the aggregator reduces this total based on the number of redundant cluster pairs identified. Because the aggregator determines that four pairs of clusters are redundant, it reduces the total number of clusters (eight) by four to achieve the total number of the four categories of deep learning in the joint learning system. Two cat pairs are considered redundant with each other. Two rabbit pairs are considered redundant with each other. Two dog pairs are considered redundant with each other. Two fish pairs are considered redundant with each other. Therefore, by performing the integration and analysis of cluster information, the aggregator identifies a total of four clusters for this embodiment of the deep learning model.

[0080] The aggregator then generates a deep learning model with four output nodes, which is then passed to distributed computing devices, in this case, to the first computer 102a, the second computer 102b, and the third computer 102c. For example, Figure 6 A deep learning model 600 with a first input layer 602, a second input layer 604, and an output layer 606 is shown. In a first case, the output layer 606 will have four nodes corresponding to four categories determined by ensemble operation performed by an aggregator. A first node 608a will be used for images of the cat category. A second node 608b will be used for images of the rabbit category. A third node 608c will be used for images of the dog category. A fourth node 608d will be used for images of the fish category.

[0081] After the aggregator passes the deep learning model 600 to the distributed computing device in step 220, according to... Figure 5 and Figure 6 In this case, the data is passed to the first computer 102a, the second computer 102b, and the third computer 102c. In step 224, the distributed computing devices have the opportunity to name the received category according to their own semantic preferences or meanings.

[0082] The first computer 102a can name the first three categories—"cat," "food," and "dog"—while leaving the fourth category blank because it has no samples belonging to that category. The aggregator can also send generic tags for various categories such as category 1, category 2, category 3, category 4, etc. Alternatively, if the aggregator sends categories with names filled in, the first computer 102a can leave the name of the fourth category provided by the aggregator; for example, it can leave the name "fish" for the fourth category, while still renaming the second group (the rabbit group) according to its own semantics for "food," since they believe rabbits are meant to be eaten.

[0083] The second computer 102b can name the first, second, and fourth categories—"cat," "pet," and "fish"—while leaving the third category blank because it has no samples belonging to the third category (in this case, the dog category). Alternatively, if the aggregator sends categories filled with names, the first computer 102a can leave the name "dog" for the third category provided by the aggregator, while still renaming the second group (the rabbit group) according to its own meaning of rabbits being kept as pets. Naming applies to sample fitting performed by the computing device.

[0084] The third computer 102a can name the third and fourth categories—"dog" and "fish"—while leaving the first and second categories blank because it has no samples belonging to either category and no samples belonging to either the rabbit or cat clusters. Alternatively, if the aggregator sends categories filled with names, the first computer 102a can leave the names of the first and second categories provided by the aggregator, such as "cat" and "rabbit," while also accepting the names "dog" and "fish" provided by the aggregator for the third and fourth categories.

[0085] exist Figure 5 and Figure 6 Following the embodiments described above, if in a further step 226, one of the distributed computing devices, such as the first computer 102a, receives samples of bird images and feeds these images into an updated autoencoder received from the aggregator, the updated autoencoder at the first computer 102a can identify that the bird samples are not located near any of the other four clusters. In the feature space of the updated autoencoder, anomaly detection can be performed to detect significant deviations from the four clusters of the four known categories. When the number of anomalies is significantly high, either relative to the average number of bird images input over a period of time or relative to the total number of test samples, a flag can be sent to the aggregator to warn it that the category needs to be redefined. In some embodiments, when a first data point is detected as an anomaly, a flag can be sent to the aggregator to warn it that the category needs to be redefined.

[0086] The aggregator can integrate cluster information about the new bird image cluster with other cluster information to verify that a new cluster should be created. If the aggregator is satisfied with the verification, for example, because it agrees that the bird image centroid is not close to any other centroid, the aggregator can add the fifth node 608e to the deep learning model 600 and the updated autoencoder, and can send the updated deep learning model and the updated autoencoder to all distributed computing devices in the federated learning system, for example, to... Figure 1 and Figure 5All of the first computer 102a, second computer 102b, and third computer 102c in the illustrated embodiment. Then, each distributed computing device will again have the opportunity to provide its own semantically localized definition for the fifth node 608e and the cluster associated with it, accepting a tag or name provided by the aggregator for the fifth node 608e, or not choosing a name for the fifth node 608e, which may be advantageous when the local computing device does not fall within the cluster associated with the fifth node 608e.

[0087] The above references Figure 5 and Figure 6 In the described embodiments, the sample data fed into and recognized by the joint learning system includes images of animals. In an alternative embodiment for recognizing spoken simple audio phrases, following... Figure 2 The principle of process 200 is similar. Parties in a joint learning system with enhanced semantic learning for recognizing audio phrases can group phrases into groups of questions, greetings, insults, or complements. Different parties can determine whether the same spoken audio phrase is an insult, greeting, or complement based on the cultural background or environment in which the specific participant / participant operates. Parties and an aggregator can work together to perform steps 202 to 228 regarding the formation of clusters and categories, and a deep learning model for determining the type of audio phrase and having nodes corresponding to the identified clusters. The system can rely on tone recognition and speech-to-text conversion to generate vectors of specific spoken and recorded audio phrases and analyze the spoken audio phrases. Process 200 can also be applied to predictively typing words into computing devices such as personal computers or smartphones. To analyze text or speech data, methods such as word2vec can be implemented to map raw data to numerical vectors, allowing neural networks to understand and analyze the data.

[0088] Understandable. Figures 2-6 This description provides only illustrations of certain embodiments and does not imply any limitation on how different embodiments can be implemented. Many modifications can be made to the described embodiments based on design and implementation requirements.

[0089] As described in the above embodiments, the joint learning procedures 110a, 110b, 110c, and 110d enhanced by semantic learning can improve the functionality of a computer or computer system by allowing for more accurate training of deep learning systems, while also increasing the flexibility of the learning system and reducing the computational power required to add model categories, and coordinating joint learning more efficiently.

[0090] Figure 7 This is an illustrative embodiment of the present invention. Figure 1 Block diagram 900 of the computer's internal and external components is shown. It should be understood that... Figure 7This is merely an illustration of an implementation and does not imply any limitation on the environments in which different embodiments may be implemented. Many modifications can be made to the described environment based on design and implementation requirements.

[0091] Data processing systems 902 and 904 represent any electronic device capable of executing machine-readable program instructions. Data processing systems 902 and 904 may represent smartphones, computer systems, PDAs, or other electronic devices. Examples of computing systems, environments, and / or configurations that can be represented by data processing systems 902 and 904 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the aforementioned systems or devices.

[0092] The user client computer 102 and the network server 112 may include Figure 7 The internal components 902a, b and the corresponding sets of external components 904a, b are shown. Each of the sets of internal components 902a, b includes one or more processors 906, one or more computer-readable RAMs 908 and one or more computer-readable ROMs 910 on one or more buses 912, one or more operating systems 914 and one or more computer-readable tangible storage devices 916. One or more operating systems 914, software programs 108 and co-learning programs 110a, 110b, 110c in the first computer 102a, the second computer 102b and the third computer 102c respectively, and co-learning program 110d in the network server 112 may be stored on one or more computer-readable tangible storage devices 916 for execution by one or more processors 906 via one or more RAMs 908 (typically including cache memory). Figure 7 In the illustrated embodiment, each computer-readable tangible storage device 916 is a disk storage device of an internal hard disk drive. Alternatively, each computer-readable tangible storage device 916 is a semiconductor storage device, such as ROM 910, EPROM, flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0093] Each set of internal components 902a, b also includes an R / W drive or interface 918 for reading from and writing to one or more portable computer-readable physical storage devices 920, such as CD-ROMs, DVDs, Memory Sticks, magnetic tapes, disks, optical discs, or semiconductor storage devices. Software programs such as software program 108 and co-learning programs 110a, 110b, 110c, 110d can be stored on one or more corresponding portable computer-readable physical storage devices 920, read via the corresponding R / W drive or interface 918, and loaded into the corresponding hard disk drive 916.

[0094] Each set of internal components 902a, b may also include a network adapter (or switch port card) or interface 922, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G or 4G wireless interface card, or other wired or wireless communication links. Software programs 108 and co-learning programs 110a, 110b, 110c in the first computer 102a, the second computer 102b, and the third computer 102c, as well as co-learning program 110d in the network server computer 112, can be downloaded from an external computer (e.g., a server) via a network (e.g., the Internet, a local area network, or another wide area network) and the corresponding network adapter or interface 922. From the network adapter (or switch port adapter) or interface 922, software programs 108 and co-learning programs 110a, 110b, 110c in the first computer 102a, the second computer 102b, and the third computer 102c, and co-learning program 110d in the network server computer 112, are loaded into the corresponding hard disk drives 916. The network may include copper wires, fiber optics, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers.

[0095] Each of the sets of external components 904a, b may include a computer display monitor 924, a keyboard 926, and a computer mouse 928. External components 904a, b may also include a touchscreen, a virtual keyboard, a touchpad, a pointing device, and other human-machine interface devices. Each of the sets of internal components 902a, b further includes a device driver 930 for interfacing with the computer display monitor 924, keyboard 926, and computer mouse 928. Device driver 930, R / W driver or interface 918, and network adapter or interface 922 include hardware and software (stored in storage device 916 and / or ROM 910).

[0096] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0097] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0098] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0099] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information from the computer-readable program instructions.

[0100] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0101] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0102] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be implemented as a single step, executed simultaneously, substantially simultaneously, with partial or complete time overlap, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0104] It should be understood that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings set forth herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0105] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0106] The features are as follows:

[0107] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring manual interaction with the service provider.

[0108] Wide Area Network (WAN) Access: Capabilities are available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms, such as mobile phones, laptops, and PDAs.

[0109] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is a sense of location independence because consumers typically do not control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0110] Rapid and flexible: Features can be delivered quickly and flexibly, and in some cases, automatically, to scale out quickly and release quickly to scale in. For consumers, the available capabilities typically appear unlimited and can be purchased at any time and in any quantity.

[0111] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both service providers and consumers.

[0112] The service model is as follows:

[0113] Software as a Service (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through a thin client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even the individual application capabilities, with possible exceptions of limited user-specific application configuration settings.

[0114] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of possible application hosting environments.

[0115] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over the operating system, storage, deployed applications, and possibly limited control over chosen networking components (e.g., host firewalls).

[0116] The deployment model is as follows:

[0117] Private cloud: Cloud infrastructure operated solely by an organization. It can be managed by the organization or a third party, and can exist internally or externally.

[0118] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist internally or externally.

[0119] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0120] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported together, such as cloud bursts for load balancing between clouds.

[0121] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0122] Now for reference Figure 8 The diagram illustrates an illustrative cloud computing environment 1000. As shown, the cloud computing environment 1000 includes one or more cloud computing nodes 800 with which local computing devices used by cloud consumers can communicate, such as personal digital assistants (PDAs) or cellular phones 1000A, desktop computers 1000B, laptop computers 1000C, and / or automotive computer systems 1000N. The nodes 800 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 1000 to provide infrastructure, platform, and / or software as a service, without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 8 The types of computing devices 1000A-N shown are for illustrative purposes only, and the computing node 800 and cloud computing environment 1000 can communicate with any type of computing device via any type of network and / or network-addressable connectivity (e.g., using a web browser). The cloud computing node 800 may have... Figure 7 The client computer 102 shown is Figure 1 The computers 102a, 102b, and 102c shown have similar or identical structures and internal and external components.

[0123] Now for reference Figure 9 This illustrates a set of functional abstraction layers 1100 provided by the cloud computing environment 1000. It should be understood beforehand that... Figure 9 The components, layers, and functions shown are for illustrative purposes only, and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0124] The hardware and software layer 1102 includes hardware and software components. Examples of hardware components include: a host 1104; a server 1106 based on a RISC (Reduced Instruction Set Computer) architecture; a server 1108; a blade server 1110; a storage device 1112; and a network and networking component 1114. In some embodiments, the software components include network application server software 1116 and database software 1118.

[0125] The virtualization layer 1120 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 1122; virtual storage 1124; virtual network 1126, including virtual private network; virtual application and operating system 1128; and virtual client 1130.

[0126] In one example, management layer 1132 can provide the functionality described below. Resource provisioning 1134 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 1136 provides cost tracking when utilizing resources in the cloud computing environment, as well as billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, and protection for data and other resources. User portal 1138 provides access to the cloud computing environment for consumers and system administrators. Service level management 1140 provides cloud resource allocation and management to ensure the required service level is met. Service level agreement (SLA) planning and fulfillment 1142 provides pre-scheduling and procurement of cloud resources, where future needs are anticipated according to the SLA.

[0127] Workload layer 1144 provides examples of functionalities that can be leveraged from a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 1146; software development and lifecycle management 1148; virtual classroom education delivery 1150; data analytics and processing 1152; transaction processing 1154; and semantic federated learning 1156. The federated learning procedures 110a, 110b, 110c, and 110d provide a way to accurately perform federated learning even when dealing with unique semantic naming preferences for individual computing devices within a federated learning system.

[0128] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms “comprising,” “including,” “containing,” “having,” etc., as used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0129] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for collaborative learning, the method comprising: Cluster information is received from a distributed computing device via an aggregator, wherein the cluster information relates to an identified cluster in sample data of the distributed computing device, and wherein the aggregator includes at least one processor, and wherein the sample data includes image data, audio data, or text data. The cluster information is integrated via the aggregator to define categories, wherein the integration includes identifying any redundant clusters among the identified clusters, and wherein the number of categories corresponds to the total number of clusters from the distributed computing device minus any redundant clusters; The aggregator sends a deep learning model to the distributed computing device, wherein the deep learning model includes an output layer with nodes, wherein the nodes correspond to defined categories; as well as The aggregator receives one or more results of joint learning performed by the distributed computing device, wherein the joint learning trains the deep learning model.

2. The method according to claim 1, further comprising: The aggregator provides an autoencoder to the distributed computing device; The automatic encoder output is generated by running sample data from each computing device of the distributed computing device through the automatic encoder. The autoencoder output is run via a clustering algorithm through the respective computing devices to identify the cluster from the sample data; as well as The cluster information is sent from the distributed computing device to the aggregator.

3. The method according to claim 1, further comprising: The joint learning is performed via the distributed computing device.

4. The method of claim 1, wherein each cluster includes a centroid; The cluster information includes centroid information, wherein the centroid information is related to the centroid; and Identifying any redundant clusters includes comparing the centroid information of the clusters.

5. The method of claim 4, wherein if the distance between the centroids of two clusters is less than a predefined threshold, the aggregator defines the two clusters as redundant.

6. The method according to claim 1, further comprising: The categories within the category are named according to the semantic meaning of the computing devices in the distributed computing device.

7. The method according to claim 2, further comprising: The cluster information is anonymized via the distributed computing device before it is sent to the aggregator.

8. The method according to claim 2, further comprising: The aggregator sends an initial autoencoder to the distributed computing device for execution; The initial autoencoder is sequentially trained via the distributed computing device to generate the autoencoder; as well as The autoencoder is sent from the last computing device in the distributed computing device to the aggregator.

9. The method of claim 8, wherein the sequential training comprises: The initial autoencoder is trained via a first computing device in the distributed computing device to generate a partially trained autoencoder; The partially trained autoencoder is sent from the first computing device to a second computing device within the distributed computing device; The partially trained autoencoder is trained via the second computing device to generate a further trained autoencoder; The further trained autoencoder is sent from the second computing device to the additional computing device; as well as The further trained autoencoder is trained by the additional computing device to produce the autoencoder, wherein the additional computing device is the final computing device.

10. The method of claim 8, wherein the initial autoencoder is a pre-trained autoencoder.

11. The method according to claim 2, further comprising: During the joint learning process, a new category is checked via a first computing device in the distributed computing device, wherein the check includes the first computing device feeding new samples to the autoencoder and performing anomaly detection to detect the new samples that deviate from the category, and wherein the deviation exceeds a predefined threshold.

12. The method of claim 11, further comprising: The deviation is notified to the aggregator via the first computing device; New categories are added to the categories via the aggregator to form a new set of categories, wherein the new categories correspond to the new samples; Sending the new set of categories from the aggregator to the distributed computing device; and Additional joint learning is performed via the aggregator and the distributed computing device to further train the deep learning model, wherein the execution of the additional joint learning includes using the new set of categories as new nodes of the output layer of the deep learning model.

13. The method of claim 2, wherein the clustering algorithm is part of the autoencoder.

14. A computer system for joint learning enhanced by semantic learning, the computer system comprising: One or more processors, one or more computer-readable storage devices, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media, the program instructions being executable by at least one of the one or more processors via at least one of the one or more computer-readable storage devices, wherein the computer system is capable of performing a method comprising: Cluster information is received from a distributed computing device, wherein the cluster information is related to a cluster identified in sample data of the distributed computing device, wherein the sample data includes image data, audio data, or text data. The cluster information is integrated to define categories, wherein the integration includes identifying any redundant clusters in the identified clusters, and wherein the number of categories corresponds to the total number of clusters from the distributed computing device minus any redundant clusters; Send a deep learning model to the distributed computing device, wherein the deep learning model includes an output layer with nodes, wherein the nodes correspond to defined categories; and Receive one or more results of joint learning performed by the distributed computing device, wherein the joint learning trains the deep learning model.

15. The computer system of claim 14, wherein the method further comprises: An automatic encoder is provided to the distributed computing device.

16. The computer system of claim 15, wherein the autoencoder includes a clustering algorithm.

17. The computer system of claim 14, wherein each cluster includes a centroid; The cluster information includes centroid information, wherein the centroid information is related to the centroid; and Identifying any redundant clusters includes the computer system comparing the centroid information of the clusters.

18. The computer system of claim 17, wherein if the distance between the centroids of two clusters is less than a predefined threshold, the computer system defines the two clusters as redundant.

19. The computer system of claim 15, wherein the method further comprises: The initial autoencoder is sent to the first computing device in the distributed computing device for execution; Receive a partially trained autoencoder from the first computing device; The partially trained autoencoder is sent to a second computing device in the distributed computing device; Receive the autoencoder for further training from the second computing device; The further trained autoencoder is sent to an additional computing device in the distributed computing device; as well as The automatic encoder is received from the additional computing device.

20. A computer program product for joint learning enhanced by semantic learning, the computer program product comprising program instructions executable by a processor to cause the processor to perform a method comprising: Cluster information is received from a distributed computing device, wherein the cluster information is related to a cluster identified in sample data of the distributed computing device, wherein the sample data includes image data, audio data, or text data. The cluster information is integrated to define categories, wherein the integration includes identifying any redundant clusters in the identified clusters, and wherein the number of categories corresponds to the total number of clusters from the distributed computing device minus any redundant clusters; Send a deep learning model to the distributed computing device, wherein the deep learning model includes an output layer with nodes, wherein the nodes correspond to defined categories; as well as Receive one or more results of joint learning performed by the distributed computing device, wherein the joint learning trains the deep learning model.

Images