Adaptive co-distillation model

Abstract

A method for use with a computing device is provided. The method can include inputting an input dataset into a first private artificial intelligence model generated using a first private dataset and a second private artificial intelligence model generated using a second private dataset. The method can also include receiving a first result dataset from the first private artificial intelligence model and receiving a second result dataset from the second private artificial intelligence model. The method can also include training an adaptive co-distillation model with the input dataset and the first result dataset. The method can also include training the adaptive co-distillation model with the input dataset and the second result dataset. The adaptive co-distillation model can not be trained on the first private dataset or the second private dataset.

Description

Technical Field

[0001] This invention relates to an adaptive co-distillation model. Background Technology

[0002] Computer models are used across various fields to enable computer systems to analyze data in specific situations and make decisions based on that data. Many such models have been developed. For example, symbolic models such as expert systems and decision trees, and non-symbolic models such as neural networks and support vector machines are well-known. In machine learning, models are trained on datasets, while in symbolic systems, models are often heuristically designed, i.e., using human decision-making. With the wide variety of models used by different entities, opportunities for collaboration exist.

[0003] In distributed machine learning, machine learning algorithms are trained using data distributed across multiple computing devices. Each of these devices can store its own training dataset, which is used to train the machine learning model locally. These machine learning models can then be combined into a centralized model. Distributed machine learning can be used, for example, when users of the computing devices do not want to send private data to a central server to train the machine learning model.

[0004] However, the inventors have identified the following technical drawbacks of the aforementioned distributed machine learning method. In this method, all client-side machine learning models share the same model architecture. Therefore, multiple client-side machine learning models with different architectures may not be integrated into a central machine learning model. Furthermore, users of distributed client computing devices may wish to maintain the privacy of the specific machine learning model architecture they use for their client-side machine learning models for reasons such as privacy, competitive advantage, licensing, or other legal constraints. However, the aforementioned distributed machine learning method shares the same model architecture among distributed users, thus failing to maintain the privacy of the client-side model architecture. Therefore, there is a technical challenge in providing an AI system that facilitates the integration of different AI models developed by independent parties while protecting the data privacy of the data used to train each of these different AI models.

[0005] Industrial automation is one example of a technology where these shortcomings present specific challenges. Different manufacturers have equipped manufacturing equipment, such as robotic arms and conveyors, with motion sensors and cameras that collect performance data from the equipment during the manufacturing process. This performance data is used by each manufacturer to train artificial intelligence (AI) models, which are then used to operate the manufacturing equipment in future manufacturing processes. The lack of interoperability between these different AI models and the challenge of data privacy regarding the datasets on which these different models have been trained pose potential obstacles to the rapid growth and development of AI models used in such industrial automation systems. Summary of the Invention

[0006] According to one aspect of this disclosure, a method for use with a computing device is provided. The method may include inputting an input dataset into a first private artificial intelligence model generated using a first private dataset and a second private artificial intelligence model generated using a second private dataset. The method may further include receiving a first result dataset from the first private artificial intelligence model as a result of applying the first private artificial intelligence model to the input dataset. The method may further include receiving a second result dataset from the second private artificial intelligence model as a result of applying the second private artificial intelligence model to the input dataset. In a first training phase, the method may further include training an adaptive co-distillation model using the input dataset as input and the first result dataset as a first target output. In a second training phase, the method may further include training an adaptive co-distillation model using the input dataset as input and the second result dataset as a second target output. The adaptive co-distillation model may not be trained on either the first or second private dataset.

[0007] This summary is provided to introduce, in a simplified form, some concepts that will be further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that address any or all of the shortcomings pointed out in any part of this disclosure. Attached Figure Description

[0008] [ Figure 1 ] Figure 1 An exemplary computational system capable of training an adaptive co-distillation model is illustrated according to an exemplary embodiment.

[0009] [ Figure 2 ] Figure 2 It is shown in accordance with Figure 1The adaptive co-distillation model of the implementation method is the training of the adaptive co-distillation model in the instance of a deep neural network.

[0010] [ Figure 3 ] Figure 3 It shows according to Figure 1 The implementation method is an adaptive co-distillation model that selects classification labels from a set of combined classification labels.

[0011] [ Figure 4 ] Figure 4 It shows according to Figure 1 The implementation method uses multiple series of ordered input values ​​to train an adaptive co-distillation model.

[0012] [ Figure 5 ] Figure 5 It is shown in Figure 1 A schematic diagram illustrating the use of a weighted loss function in the training of an adaptive co-distillation model.

[0013] [ Figure 6 ] Figure 6 It is shown in Figure 5 A schematic diagram illustrating the calculation of the weighting factors used in the weighted loss function.

[0014] [ Figure 7 ] Figure 7 The calculation of the training data distance between each of two exemplary images in a private dataset and a public dataset is shown.

[0015] [ Figure 8 ] Figure 8 It shows Figure 1 An exemplary adaptive co-distillation model generation interface for a graphical user interface of the implementation method.

[0016] [ Figure 9 ] Figure 9 It shows according to Figure 1 An exemplary interface for editing classification confidence levels in the implementation of the method.

[0017] [ Figure 10 ] Figure 10 It shows according to Figure 1 An exemplary interface for editing training data distances in the implementation of the above method.

[0018] [ Figure 11A ] Figure 11A It shows the basis Figure 1 Example diagrams of the implementation of the method for the error, unweighted loss, and weighted loss between the public model output and the private model output for a set of given model inputs.

[0019] [ Figure 11B ] Figure 11B It shows the basis Figure 1 Another example diagram of the implementation of the method is shown, illustrating the error between the public model output and the private model output for a given set of model inputs, and the loss between the public model output and the weighted private model output.

[0020] [ Figure 12 ] Figure 12 It shows according to Figure 1 The implementation method is illustrated by a graph of the weighted and unweighted loss surfaces in one dimension.

[0021] [ Figure 13 ] Figure 13 This illustrates the process of training a template machine learning model on a server computing device. Figure 1 An exemplary computing system.

[0022] [ Figure 14 ] Figure 14 It shows according to Figure 1 The implementation method is the training of a template machine learning model in an instance of a deep neural network.

[0023] [ Figure 15A ] Figure 15A It shows according to Figure 1 A flowchart illustrating an exemplary method for training an adaptive co-distillation model.

[0024] [ Figure 15B ] Figure 15B It shows when execution Figure 15A The method is an additional step that can be performed in some implementations.

[0025] [ Figure 16 ] Figure 16 It shows that it can be executed Figure 1 A schematic diagram of an exemplary computing environment for a computing system. Detailed Implementation

[0026] To solve the above problems, according to Figure 1One exemplary implementation shown provides a computing system 1. As a summary, computing system 1 is configured to enable public or shared artificial intelligence models to be trained to simulate the behavior of one or more private artificial intelligence models generated using private datasets. This is achieved by inputting data from the public or shared datasets into each private artificial intelligence model and training the public artificial intelligence model using the outputs of the private artificial intelligence models. Weighted loss functions can be used to mitigate the training effects on less relevant data in the public or shared datasets, as explained in detail below. As described below, the weighted loss function can be, for example, an output-weighted loss function or an input-weighted loss function. By using such a system, the learning of each private artificial intelligence model can be shared while maintaining the privacy of the data in each private dataset.

[0027] exist Figure 1 In this example, computing system 1 includes a server computing device 10, a first client computing device 80, and a second client computing device 90. The server computing device 10 can be configured to communicate with the first client computing device 80 and the second client computing device 90 via a network. Although in Figure 1 The example shows only two client computing devices, but in some implementations, server computing device 10 may be configured to communicate with more than two client computing devices. In some implementations, the functionality of server computing device 10 may be distributed across multiple communication-coupled physical computing devices.

[0028] Server computing device 10 may include a processor 12. Additionally, server computing device 10 may include a memory 14 for storing instructions executable by processor 12. Memory 14 may include volatile storage and / or non-volatile storage. First client computing device 80 and second client computing device 90 may also include corresponding processors 82 and 92 and memories 84 and 94. In some embodiments, server computing device 10 may include one or more corresponding input devices 16, such as a keyboard, mouse, touchscreen, touchpad, microphone, optical sensor, accelerometer, or other types of input devices 16. Server computing device 10 may also include one or more corresponding output devices 18, such as a display, speaker, haptic feedback device, or other types of output devices 18. First client computing device 80 and second client computing device 90 may also include one or more corresponding input devices and / or one or more corresponding output devices.

[0029] The processor 12 of the server computing device 10 can be configured to send an input dataset 50 to a first client computing device 80 and a second client computing device 90. Because the input dataset 50 is shared among the server computing device, the first computing device, and the second computing device, each of which may reside at different entities, the input dataset 50 can also be referred to as a shared dataset, and in some cases, it may be a public dataset. The input dataset 50 may include multiple input data entries 52. As discussed further in detail below, the input dataset 50 can be used as input for a first private AI model 30 executed at the first client computing device 80 and a second private AI model 40 executed at the second client computing device 90, enabling outputs from these models that can be used to train an adaptive co-distillation model 60 at the server computing device. The input data entries 52 included in the input dataset 50 may be unlabeled data. Unlabeled data has the advantage of being generally more readily available than labeled data. Furthermore, unlabeled data does not specify what classification labels the first private AI model 30 and the second private AI model 40 may assign to the input data entries 52. In other implementations, input dataset 50 may be a partially labeled dataset in which a subset of input data entries 52 has corresponding classification labels, as discussed in further detail below. For example, by using unlabeled or partially labeled data in input dataset 50, a private AI model can freely classify the data in input dataset 50 according to classifications known to each private model but not included in input dataset 50.

[0030] like Figure 1 As shown in the example, a first private AI model 30 has been generated using a first private dataset 32. The first private dataset 32 ​​may include multiple first private entries 34, which may include multiple exemplary inputs and corresponding exemplary outputs for the first private AI model 30. The first private AI model 30 may be a machine learning model trained using the first private dataset 32 ​​as a training dataset. Alternatively, the first private AI model may be a heuristic model programmed based on the first private dataset 32 ​​without using machine learning techniques, such as a symbolic AI model. In some implementations, the first private AI model 30 may be generated before receiving the input dataset 50, such that the input dataset 50 is the runtime input of the first private AI model 30. Since the first private AI model 30 is generated at the first client computing device 80, the first private dataset 32 ​​need not be sent to the server computing device 10 and can be retained as a private dataset.

[0031] like Figure 1As shown in the example, a second private AI model 40 has been generated using a second private dataset 42. The second private dataset 42 may include multiple second private entries 44, which may include multiple exemplary inputs and corresponding exemplary outputs for the second private AI model 40. In an implementation where the second private AI model 40 is a machine learning model, the second private dataset 42 can be used as a training dataset to train the second private AI model 40. Alternatively, the second private AI model 40 may be a heuristic model programmed based on the second private dataset 42 without using machine learning model training techniques. Similar to the first private AI model 30, the second private AI model 40 may have been generated using the second private dataset 42 before receiving the input dataset 50.

[0032] In some implementations, the first private AI model 30 may have a first model architecture, and the second private AI model 40 may have a second model architecture different from the first model architecture. For example, each of the first private AI model 30 and the second private AI model 40 may be a deep neural network, a kernel machine, or a random forest. When a deep neural network is used in the first private AI model 30 or the second private AI model 40, the deep neural network may be, for example, a multilayer perceptron, a convolutional neural network, a recurrent neural network, or some other type of deep neural network. If desired, different types or architectures of deep neural networks may be used in each private AI model. When a kernel machine is used in the first private AI model 30 and / or the second private AI model 40, the kernel machine may be a support vector machine or some other type of kernel machine. The difference between the first model architecture and the second model architecture may be a difference in the type of machine learning algorithm used, or alternatively, a difference in parameters (such as the number of neurons or the number of layers) when using the same type of machine learning algorithm.

[0033] Figure 2 A first private AI model 30 and a second private AI model 40, both of which are deep neural networks, are shown. The first private AI model 30 has multiple first private model weights 31 for the connections between its neurons. Furthermore, the first private AI model 30 has a corresponding first private model bias 33 for each neuron. Multiple pairs of exemplary inputs and corresponding exemplary outputs included in the first private dataset 32 ​​can be used via a first training algorithm 100. Figure 5 (As shown in the diagram), a first private AI model 30 is trained, such as using backpropagation with stochastic gradient descent. In such an instance, the first private model can use a cost function 36 employing a weighted loss function. The weighted loss function can be about... Figure 11A The output-weighted loss function shown and described, or about Figure 11B The input-weighted loss function is shown and described. Cost function 36 is used for backpropagation during machine learning and can be, for example, the mean squared error cost function. Alternatively, another suitable cost function, such as the cross-entropy cost function, the exponential cost function, etc., can be used, as described below. Alternatively, the first private AI model 30 can be trained using other methods such as coordinate descent or subgradient descent.

[0034] Figure 2 The second private AI model 40 shown has multiple second private model weights 41 for the connections between its neurons. Furthermore, the second private AI model 40 has corresponding second private model biases 43 for each neuron. As in the example provided above for the first private AI model 30, the second private AI model 40 can be trained using a second training algorithm 102 that utilizes backpropagation with stochastic gradient descent. Figure 5 (as shown in the diagram) for training. When using backpropagation, the second private model cost function 46 can be the mean squared error cost function or any other suitable cost function, such as the cross-entropy cost function, the exponential cost function, etc. Other training methods, such as coordinate descent or subgradient descent, can be used alternatively to train the second private AI model 40.

[0035] return Figure 1 As a result of applying the first private AI model 30 to the input dataset 50, the first client computing device 80 can generate a first result dataset 20. The first result dataset 20 may include a plurality of first result entries 22. In some embodiments, the first result dataset 20 may also include a plurality of first confidence scores 23 associated with the first result entries 22. For example, when each first result entry 22 is selected from a predetermined list of potential outputs, the first result dataset 20 may include the estimated probability of each of those potential outputs as a first confidence score 23. In some embodiments, each first result entry 22 may be a vector of the plurality of first confidence scores 23. Additionally, the first result dataset 20 may include a plurality of first data distances 21, each first data distance being, for example, the training data distance between a corresponding input data entry 52 in the input dataset 50 and the first private dataset 32, as calculated by the first data similarity algorithm 35. For example, the first data distances 21 may be used to calculate a weighted loss function, as described below. Figure 5 As described. Regarding Figure 6 An exemplary data similarity algorithm is discussed.

[0036] See also Figure 1Furthermore, as a result of applying the second private AI model 40 to the input dataset 50, the second client computing device 90 can generate a second result dataset 24. The second result dataset 24 may include multiple second result entries 26. The second result dataset 24 may also include multiple second confidence scores 27 associated with each second result entry 26. The processor 82 of the first client computing device 80 and the processor 92 of the second client computing device 90 may be configured to send the first result dataset 20 and the second result dataset 24 to the server computing device 10, respectively. Additionally, the second result dataset 24 may include multiple second data distances 25, each second data distance being, for example, the training data distance between the corresponding input data entry 52 in the input dataset 50 and the second private dataset 42, as calculated by the second data similarity algorithm 45. It should be understood that the first data similarity algorithm 35 and the second data similarity algorithm 45 may be the same or different algorithms as needed. For example, the second data distance 25 may also be used to calculate a weighted loss function, as described below. Figure 5 As described.

[0037] See also Figure 1 The processor 12 of the server computing device 10 can also be configured to train an adaptive co-distillation model 60 with an input dataset 50 as input and a result dataset 20 (specifically, the first data entry 22 of the first dataset 20) as the first target output. During the training of the adaptive co-distillation model 60, the first target output may also include a first confidence level 23. For example, the adaptive co-distillation model 60 can be a machine learning model, such as a deep neural network, kernel machine, or random forest, or an algorithm heuristically designed by humans. Figure 2 In the example, the adaptive co-distillation model 60 is a deep neural network with multiple adaptive co-distillation model weights 61 between its neurons. The adaptive co-distillation model 60 may also have a corresponding adaptive co-distillation model bias 63 for each neuron. Training using a first result dataset 20 can be performed during a first training phase 62. For example, the adaptive co-distillation model 60 can be trained using an adaptive co-distillation model cost function 66 (such as a mean squared error cost function) via stochastic gradient descent. Alternatively, some other cost functions may be used as discussed above. In some implementations, in addition to stochastic gradient descent, some other methods for training the adaptive co-distillation model 60 may be used, such as coordinate descent or subgradient descent.

[0038] In the second training phase 64, processor 12 may also be configured to train the adaptive co-distillation model 60 with input dataset 50 as input and second result dataset 24 as the second target output. The second training phase 64 may be executed after the first training phase 62. Backpropagation using stochastic gradient descent can be used to train the adaptive co-distillation model 60 with input dataset 50 and second result dataset 24. In some embodiments, the adaptive co-distillation model cost function 66 used in the first training phase 62 may be reused in the second training phase 64. Alternatively, as discussed above, other cost functions may be used. As described below, the cost function may include a weighted loss function that enables the adaptive co-distillation model to learn more from training data similar to the private training data in the first private training dataset 32 ​​and the second private training dataset 42.

[0039] The training of the adaptive co-distillation model 60 can be symbolically represented by the following example. The input dataset 50 can be represented as D dist = (x1, x2, ...), where x n This is input data entry 52. ​​The first private AI model 30 and the second private AI model 40 can each output their respective model values ​​{M}. i (x n )|x n ∈D dist The output is sent to server computing device 10, where M i It is D dist Each input value x n The model output value. For a classification model, the output value can be a classification, and in addition to classification, the model can also be configured to output confidence values. When training the adaptive co-distillation model 60, the processor 12 of the server computing device 10 can be configured to minimize the following error expression:

[0040] [Mathematical Expression 1]

[0041]

[0042] The following terms have the following meanings:

[0043] WeightedLoss: The M value minimized during training. server The cost function;

[0044] M server : Shared or public models (e.g., adaptive co-distillation model 60);

[0045] M i Private models (e.g., first private AI model 30 and second private AI model 40);

[0046] Loss: Any suitable loss function, such as mean squared error, mean error, mean bias error, support vector machine loss, cross-entropy loss;

[0047] x n Shared or public dataset D dist (For example, the element n of the input dataset 50);

[0048] w: weighting factor; and

[0049] [Mathematical Expression 2]

[0050]

[0051] The above expression can be called the output-weighted loss function because the output of the loss function is weighted and can be compared with the input-weighted loss function described below. The weighting factors can be calculated using data similarity algorithms 35 and 45, which use public or shared datasets D... dist Each x in n With the generation of M i Private dataset D i The data similarity algorithms 35 and 45 can also compare all elements in the input data element x. n The data distance between the dataset and the public dataset outputs the corresponding weighting factor w. i，n Data similarity algorithms 35 and 45 can calculate data distance in a single dimension (i.e., for a single independent variable) or in multiple dimensions (i.e., for multiple independent variables). The weighting factor w is used as... i，n Based on the replacement of this data distance, or otherwise, the weighting factor w i，n This can be based on another metric, such as confidence values ​​from the corresponding private AI models 30, 40, or manually specified input, for example, as described below regarding... Figure 8 The discussed algorithms, suitable for data similarity algorithms 35 and 45, include, for example, Euclidean distance, Manhattan distance, Minkowski distance, nth root similarity, cosine similarity, square root similarity, and Jaccard distance. Data similarity algorithms 35 and 45 can first calculate the data distances, and then calculate a weighting factor w based on these distances. i，n For example, this is achieved by normalizing the data distance to a value between zero and one (the larger the distance, the closer to one), and then calculating the weighting factor as equal to 1 minus the normalized data distance value. This is used to train a shared or public model M. server (For example, the training algorithm of the adaptive co-distillation model 60) attempts to minimize the weighted loss by learning more from a larger loss, sharing or public model M. server(For example, adaptive co-distillation model 60) will combine the public dataset with each private dataset M i It learns more from data similar to the data in the dataset and will not learn as much from data that is different from the data in each private dataset from public datasets. See the following references. Figure 9 The above equation will be described in more detail.

[0052] exist Figure 1 and Figure 2 In this instance, the adaptive co-distillation model 60 is not directly trained on the first private dataset 32 ​​or the second private dataset 42, and cannot access them. Since the server computing device 10 does not need to receive the first private dataset 32 ​​or the second private dataset 42 for training, the privacy of the private datasets can be maintained. Furthermore, since the processor 12 receives the first result dataset 20 and the second result dataset 24 instead of the parameters of the first private AI model 30 and the second private AI model 40, the corresponding users of the first client computing device 80 and the second client computing device 90 can also maintain the architecture of their respective machine learning models as private, thereby contributing to privacy, competitive advantage, licensing, or other legal constraints. The corresponding users of the first client computing device 80 and the second client computing device 90 could, for example, be employees of the respective companies or researchers of the respective research entities. Using the first result dataset 20 and the second result dataset 24 to train the adaptive co-distillation model 60 also allows the generation of adaptive co-distillation models 60 for private AI models with different architectures. Conversely, existing methods of combining distributed machine learning models by forming shared layers between models are generally not applicable to combining models with different architectures.

[0053] In some implementations, such as Figure 1As shown, the adaptive co-distillation model 60 can be trained at least partially through supervised learning. In such embodiments, during the training of the adaptive co-distillation model 60, the processor 12 may also be configured to transmit one or more adaptive co-distillation model outputs 68 generated by the adaptive co-distillation model 60 to a user via one or more output devices 18 of the server computing device 10. The processor 12 may also be configured to receive one or more supervised learning user inputs 56 via one or more input devices 16. The one or more supervised learning user inputs 56 may include corresponding evaluations of one or more adaptive co-distillation model outputs 68. For example, the supervised learning user input 56 may indicate whether the adaptive co-distillation model 60 has applied accurate classification labels to input data entries 52 included in the input dataset 50. Based on this supervised learning user input 56, the processor 12 may also be configured to update one or more parameters of the adaptive co-distillation model 60. For example, in embodiments where the adaptive co-distillation model 60 is a deep neural network, the processor 12 may update one or more adaptive co-distillation model weights 61 and / or one or more adaptive co-distillation model biases 63.

[0054] In some implementations, such as Figure 3 As shown, the adaptive co-distillation model 60 can be a classification model. In such an implementation, the first result dataset 20 and the second result dataset 24 can each include multiple corresponding classification labels. Figure 3 Multiple first classification labels 126 and multiple second classification labels 128 are shown, respectively included in the first result dataset 20 and the second result dataset 24. Figure 3 The adaptive co-distillation model 60 in the example includes a combined classification label set 120, which includes each of the first classification label 126 and the second classification label 128. At runtime, the adaptive co-distillation model 60 may receive a runtime input dataset 110 including one or more runtime input data entries 112. In response to receiving the runtime input dataset 110, the adaptive co-distillation model may be configured to output a runtime output dataset 140 including one or more runtime classification labels 142 corresponding to one or more runtime input data entries 112. In some embodiments, multiple runtime classification labels 142 may be matched with each runtime input data entry 112. In such an embodiment, the adaptive co-distillation model 60 may also be configured to output a corresponding output confidence score 144 for each runtime classification label 142. Therefore, the adaptive co-distillation model 60 may be configured to perform a classification task performed by both the first private AI model 30 and the second private AI model 40, without training on either the first private dataset 32 ​​or the second private dataset 42.

[0055] like Figure 3As shown, in an implementation where the adaptive co-distillation model 60 is a classification model, the input dataset 50 can be a partially labeled dataset. In such an implementation, the input dataset 50 may include a first subset of input data entries 52 with corresponding input classification labels 54 and a second subset of input data entries 52 without corresponding input classification labels 54. In such an implementation, multiple input classification labels 54 may be included in a combined classification label set 120.

[0056] Alternatively, in some embodiments, the adaptive co-distillation model 60 may be a regression model. In such an embodiment, the first result dataset 20 and the second result dataset 24 may each include a corresponding plurality of numerical values. Both the first private AI model 30 and the second private AI model 40 may be configured to map their respective inputs to corresponding first result entries 22 and second result entries 26 that include one or more numerical values. For example, the first private AI model 30 and the second private AI model 40 may each output a scalar numerical value, a vector of numerical values, or some other data structure that includes one or more numerical values. In such an embodiment, the adaptive co-distillation model 60 may perform regression on the first result entries 22 and second result entries 26 based on the input data entries 52.

[0057] In other implementations, such as Figure 4 As shown in the example, the adaptive co-distillation model 60 can be a recurrent neural network. Figure 4 In the example, input dataset 250 includes multiple input sequences 252, each input sequence 252 including multiple ordered input values ​​254. The ordered input values ​​254 included in each input sequence 252 may be ordered temporally, spatially, or according to some other variable. In an implementation where the adaptive co-distillation model 60 is a recurrent neural network, the first private dataset 232 for training the first private AI model 30 may include multiple first private sequences 234. Each first private sequence 234 may include multiple first private ordered input values ​​236, which may be ordered according to the same variable as the ordered input values ​​254 included in the input sequence 252 of input dataset 250. Furthermore, the second private dataset 242 may include multiple second private sequences 244. Each second private sequence 244 may include multiple second private ordered input values ​​246, which may be ordered according to the same variable as the ordered input values ​​254 included in the input sequence 252 of input dataset 250. Therefore, the first private AI model 30, the second private AI model 40, and the adaptive co-distillation model 60 may each be generated using sequence data.

[0058] In some implementations... Figure 1The first result dataset 20 or the second result dataset 24 can be homomorphically encrypted. When the first result dataset 20 or the second result dataset 24 is homomorphically encrypted, one or more computations can be performed on the first result entry 22 or the second result entry 26 at the processor 12 of the server computing device 10, while the plaintext values ​​of those entries remain hidden. The first result dataset 20 or the second result dataset 24 can be encrypted using a partially homomorphic encryption algorithm, a specific homomorphic encryption algorithm, a hierarchical fully homomorphic encryption algorithm, or a fully homomorphic encryption algorithm. For example, as a partially homomorphic scheme, the Paillier cryptosystem, the Rivest-Shamir-Adleman (RSA), the ElGamal scheme, or the Goldwasser-Micali scheme can be used, and as a fully homomorphic scheme, the Brakerski-Gentry-Vaikuntahan scheme or its variants can be used. The encryption of the first result dataset 20 or the second result dataset 24 can be performed at the first client computing device 80 or the second client computing device 90, respectively. An adaptive co-distillation model 60 can be trained using the homomorphically encrypted first result dataset 20 or the second result dataset 24.

[0059] Figures 5 to 12 The overall picture shows aspects of computer system 1 that enable it to utilize a weighted loss function. For example... Figure 5 As shown, the input dataset is 50(D) dist Each element x of ) n Private datasets (such as the first private dataset 32(D)) can be used. i ) and the second private dataset 42 (D i After generating each of those models, they are fed as input into multiple private AI models (such as the first private AI model 30 and the second private AI model 40). In the case of element x... n When inputting each element into the private models 30 and 40, a data similarity algorithm (such as the first data similarity algorithm 35 and the second data similarity algorithm 45) will apply this algorithm to the element x. n Data D in private datasets 32 and 42 i Comparisons are made. Data similarity algorithms can include, for example, Euclidean distance, Manhattan distance, Minkowski distance, nth root similarity, cosine similarity, square root similarity, and Jaccard distance. Private models 30 and 40 each output corresponding result datasets 20 and 24, which include the model output values ​​(M... i The model input value (x) includes the confidence value in the model output and the distance between the training data and the model input value (x). nThe results datasets 20 and 24 can also be passed in the result datasets 20 and 24, or retrieved directly from the input dataset 50 at the server. The result datasets 20 and 24 output by the private models 30 and 40 are used in the corresponding training phases to train the adaptive co-distillation model 60 (i.e., a shared or public model) using the first training algorithm 100 and the second training algorithm 102. Using x from each private model 30 and 40 n As input and M i The adaptive co-distillation model 60 is trained as its output. The first training algorithm 100 and the second training algorithm 102 can be selected from a variety of training algorithms, such as backpropagation utilizing gradient descent as described above. Each of the training algorithms 100 and 102 has an associated weighted loss function that attempts to be minimized during training. The weighted loss function may be based, for example, on the corresponding training data distances 21 and 25 output from the corresponding similarity algorithms 35 and 45, on the corresponding confidence values ​​23 and 27 output from the associated private models 30 and 40, and / or on human input, as explained in more detail below.

[0060] Figure 6 This illustrates input dataset 50 (i.e., shared or public datasets (D)). dist x in )) n Specific examples (images of eggs being caught by robots) and the first private dataset D i Comparison. In this example, D i It is a set of labels for images of golf balls that have been caught by the robot or not (depending on the situation), and these labels are ground truth labels indicating whether the image shows the robot correctly catching the ball. Feature extraction algorithm 108 is applied to element x. n and private dataset D i And it generates parameterized representations 110, 112 for each of them. Typically, these features are scale-invariant features, and they can be detected, for example, by a convolutional neural network. Alternatively, other types of features can be extracted. For example, the parameterized representations 110, 112 can be multidimensional feature vectors. The feature vectors are fed into a first similarity algorithm 35, in this instance, where the first similarity algorithm 35 calculates a first training data distance 21 and then converts it into a weighting factor 114 for element x. n The adaptive co-distillation model 60 (i.e., the shared or common model M) will be applied. server The learning effect of the training data is weighted based on the values ​​between zero and one. As the distance 21 between the training data increases, the weighting factor decreases, and as the distance 21 between the training data decreases, the weighting factor increases. Both values ​​are typically within the range of zero and one.

[0061] Figure 7This demonstrates how two images (one image of a robot catching an egg and the other image of a robot catching a baseball) are compared with a private dataset D using the first similarity algorithm 35. i The comparisons are shown separately for instances with low training data distance of 21 and high training data distance of 21. This difference can be attributed to the private dataset D. i The difference in shape between a medium-sized egg and a golf ball, and the private dataset D i The similarity between the shapes of baseballs and golf balls in the regions of interest (ROI) of the images is calculated. Therefore, similarity algorithm 35 calculates a corresponding weighting factor 114 that is low for images with eggs and high for images with baseballs. It should be understood that this is merely an illustrative example, as many variations are possible.

[0062] Figure 8 A private model owner view of a graphical user interface (GUI) 116 for configuring the adaptive co-distillation model 60 is shown. As shown, GUI 116 includes multiple input fields 118 for the user to specify paths to a public model, a public dataset, a first private dataset, and a first private data model. A component 120 is provided to add additional private datasets and private models owned by the owner as needed. It should be understood that other owners of different private datasets will use different views of GUI 116 to train the public model on the additional private models. GUI 116 also includes a training algorithm selection tool 122, as shown, which is configured to select backpropagation using gradient descent as the training algorithm. GUI 116 also includes a selector 124 to specify one or more weighting factors 114 used by the first training algorithm 100. As shown, the weighting factor selector 124 includes a multi-select checkbox tool, through which the user can select between one or more of training data distance, model output confidence values, or manually specified factors (such as manually edited confidence values, manually edited data distance values, or manually assigned algorithms). Links 126, 128, and 130 are displayed for defining these manually specified values. Once the parameters are entered, the user can select the training model command selector 132 to begin processing the public dataset (e.g., input dataset 50) by the first private data model 30, so that the output will be fed into the first result dataset 20 in the adaptive co-distillation model 60, and thus, the adaptive co-distillation model 60 is trained.

[0063] Figure 9 Showing the selection Figure 8 The manually edited confidence value user interface 134 is displayed when link 126 is mentioned. As shown, the manually edited confidence value user interface 134 includes elements x from the input dataset 50 as input to the first private model 30. nThe first result dataset 20 contains the output value (ball-catching classification) and output confidence score (0.88). Through the operator interface portion of interface 134, the operator can provide input to modify the output of the first private model 30 (e.g., by selecting "Yes" to indicate that the ball was caught correctly), and can enter 1.00 as the classification confidence score, since the classification has been determined to be correct as the ground truth for training. This value can also be manually edited by typing a specific value. In the example shown, the private model computed a ball-catching classification with a confidence score of 0.88, which was manually edited to a confidence score of 1.00 via the interface.

[0064] Figure 10 A user interface 136 for providing manually edited training data distance values ​​is shown, which can be traversed... Figure 8 Access it via link 128. As shown in the figure, user interface 136 displays element x from input dataset 50. n And display from the first private dataset 32 ​​(D i The element is typically a loop through D. i Each of the elements in the array is configured to receive manually edited input, thereby enabling x to... n With D i Each element in the image is compared. In response to each pair of images displayed, the operator can select a "Yes" or "No" selector to verify whether the images are similar or dissimilar. The default value of "Yes" is selected for a training data distance of 0.01, and the default value of "No" is selected for a training data distance of 1.0; both values ​​can be further edited by the operator as needed. Selecting "Submit" loops to the next image pair until the training data distances for all image pairs have been manually edited. In the example shown, the first similarity algorithm 35 calculated a training data distance of 0.14 between the image pairs, which was manually edited to 0.01.

[0065] Figure 11A This is an annotated diagram illustrating the relationship between model input, model output, error, unweighted loss, and weighted loss used to weight the output of the loss function used in training the common model. The top of the diagram shows the model input, model output, error, unweighted loss, and weighted loss used for adaptive co-distillation of the model (common model M). server ) and the first private artificial intelligence model 30 (private model M) i This is a graph showing the model input (each of the x1 to x5 sample elements from the public dataset 50) to the model output. The difference between the two outputs is the error E = M. server -M i For ease of illustration in a single dimension, the error E is shown, but more typically it is multidimensional, such as M. server and M iThe output. The loss is usually based on M. server and M i The output value is calculated based on the loss function. If a loss function that squares the error is used, such as mean squared error, then a large error will have a greater impact on the loss. In the graph, x1, x2, x4, and x5 are those that have passed the first similarity algorithm 35. Figure 6 The processing shown is determined to be related to the first private dataset 32 ​​(D) i Similar to (i.e., similar to the first private dataset 32(D)) i () elements with low data distance), while x3 has been identified as being related to the first private dataset 32 ​​(D i The data has high data distance (i.e., low similarity). For this reason, the dashed-line unweighted loss (e.g., MSE) vector is shown as a solid line error vector greater than x3, while the dashed-line unweighted loss vector is shown as a solid line error vector approximately equal in length to x1, x2, x4, and x5. Without weighting, the influence of data element x3 on learning through the adaptive co-distillation model 60 (i.e., a shared or common data model) will be greater than the influence of any of the data elements x1, x2, x4, and x5. Furthermore, due to x3 to D... i The training data distance is high, therefore the larger learning effect will not be achieved, leading to erroneous results from the adaptive co-distillation model 60 (i.e., the shared or common data model). To address this issue, according to Figure 6 The processing described herein calculates a weighting factor for x3 that is relatively lower than that for x1, x2, x4, and x5, and this weighting factor is used to calculate the weighted loss of the data elements. The weighting factor w... i，n Input in Figure 11A The formula below the graph is represented here; the first term is the weighted average of the weighting factors. This weighted average is multiplied by the output of the loss function (which can be any suitable loss function as described above), and the result is the weighted loss. Since... Figure 11A The weighted loss in the algorithm is weighted by the distance to the training data, so it is also called the training data weighted loss. It should be noted that the distance to D... i High data distance data element x i The weighted loss is less than that shown in the curve with a value of D. i The weighted losses for other data elements with lower data distances are shown as the relative magnitude differences in the dashed vector representing the weighted losses. It should be understood that the relative magnitudes of the unweighted and weighted loss vectors are for illustrative purposes, and the loss and error are shown on the same axis only for illustrative purposes.

[0066] It should be understood that, as an alternative to calculating the weighted loss based on the expression given above, alternative forms of weighting can be used to minimize the learning effect of data elements in the public dataset that have a high data distance to the private dataset. For example, an input-weighted loss function can be used. Figure 11B The following expression, which shows a similarity when using an input-weighted loss function, is shown. Figure 11A Annotated diagram:

[0067] [Mathematical Expression 3]

[0068]

[0069] In this expression, a weighted average is applied to each model input x. i Private model output value M i Instead of the output of the loss function. Besides the actual M... i value, Figure 11B The weighted M, used as input to the loss function, is also shown. i Value. For example, in Figure 11A In this instance, the loss function can be mean squared error or any other suitable loss function. This is achieved by taking the private model output value M... i The processor 12 weights the private model output value before it is output to the loss function, thus reducing the private model output value M that is propagated to the loss function. i The amount of noise in the dataset is reduced, and therefore less is learned from data elements in the public dataset that have a high data distance from the private dataset compared to data elements in the private dataset that have a low data distance.

[0070] Figure 12 An exemplary two-dimensional loss function surface is shown as a curve. This two-dimensional representation is for illustrative purposes; more typically, the loss function is multi-dimensional. The loss for each of x1 to x5 is illustrated graphically. Figure 11A The same data elements are shown in the image. Because... Figure 11A The unweighted loss of the data element shown is relatively large, so the unweighted loss surface is represented by a dashed line with a peak at x3. The weighted loss surface is represented by a solid line, and its peak is not at x3. By weighting the loss surface, inappropriate learning effects from high unweighted losses near x3 can be avoided. In this way, the adaptive co-distillation model can learn from the data in the input dataset 50 without learning too much from data in the input dataset that is dissimilar to the data in the private dataset.

[0071] Figure 13 This illustrates the training of a template machine learning model 70 at the processor 12 of the server computing device 10. Figure 1 Computing system 1. For example... Figure 13As illustrated in the example, processor 12 can also be configured to train a template machine learning model 70 on a template dataset 72 before training a first private AI model 30 and a second private AI model 40 on a first client computing device 80 and a second client computing device 90 using a first private dataset 32 ​​and a second private dataset 42. The template dataset 72 may include a plurality of template input entries 74 associated with corresponding plurality of template output entries 78. Each of the template input entries 74 may have the same data type as the input data entries 52 included in the input dataset 50. Furthermore, each template output entry 78 may each have the data type of an adaptive co-distillation model output 68, which is subsequently generated by training the adaptive co-distillation model 60.

[0072] Figure 14 The template machine learning model 70 is shown in more detail as an implementation of a deep neural network. Figure 14 In the example, the template machine learning model 70 has multiple template weights 71 for the connections between its neurons. Furthermore, the template machine learning model 70 has a corresponding template bias 73 for each neuron. The template machine learning model 70 can be trained using backpropagation with stochastic gradient descent, and it can use a template cost function 76. The template cost function 76 can be, for example, a mean squared error cost function. Alternatively, other training methods and other cost functions can be used, as discussed above.

[0073] After the template machine learning model 70 has been trained, the processor 12 of the server computing device 10 can be further configured to send a copy of the template machine learning model 70 to the first client computing device 80 and the second client computing device 90. Figure 14 In this example, a first copy 75 and a second copy 77 of the template machine learning model 70 are sent to a first client computing device 80 and a second client computing device 90, respectively. Figure 14 In the example, the first private AI model 30 is a first copy 75 of the template machine learning model 70 after it has been further trained on the first private dataset 32. Similarly, the second private AI model 40 is a second copy 77 of the template machine learning model 70 after it has been further trained on the second private dataset 42. Therefore, the template machine learning model 70 can be used as a template for the first private AI model 30 and the second private AI model 40, which were initially trained on the template dataset 72 and further trained on the first client computing device 80 and the second client computing device 90. It should be understood that, according to Figure 13 and Figure 14In this implementation, after the first private machine learning model 30 and the second private machine learning model 40, which have been trained based on both the template machine learning model 70 and the private datasets 32 and 42, are fed with an unlabeled or partially labeled input dataset 50, they produce a first result dataset 20 and a second result dataset 24. The first result dataset 20 and the second result dataset 24 are then used to... Figure 1 The adaptive co-distillation model 60 is trained in the same manner as described in the implementation method.

[0074] Figure 15A A flowchart of an exemplary method 300 that can be used with a computing system is shown. Figure 15A Method 300 can be used with Figure 1 The exemplary computing system 1 can be used with other computing systems. In step 302, method 300 may include inputting an input dataset into a first private AI model generated using a first private dataset and a second private AI model generated using a second private dataset. The input dataset may include multiple input data entries. In some embodiments, the first private AI model and the second private AI model may be executed at corresponding first client computing devices and second client computing devices. In this embodiment, the input dataset may be sent from a server computing device to the first client computing device and the second client computing device. The first private dataset may include multiple first private entries, which may be exemplary inputs and corresponding exemplary outputs for training the first private AI model. Similarly, the second private dataset may include multiple second private entries, which may be exemplary inputs and corresponding exemplary outputs for training the second private AI model.

[0075] In step 304, method 300 may further include receiving a first result dataset from the first private AI model as a result of applying the first private AI model to the input dataset. The first result dataset may include a plurality of first result entries. The first result dataset may also include one or more first confidence values ​​associated with each first result entry. Furthermore, in step 306, method 300 may further include receiving a second result dataset from the second private AI model as a result of applying the second private AI model to the input dataset. The second result dataset may include a plurality of second result entries. In some embodiments, the second result dataset may include one or more second confidence values ​​associated with each second result entry.

[0076] In some implementations, the first private AI model may have a first model architecture, and the second private AI model may have a second model architecture different from the first model architecture. For example, each of the first and second private AI models may be a deep neural network, a kernel machine, or a random forest. The type of machine learning algorithm used for the first private AI model may differ from the type of machine learning algorithm used for the second private AI model. Alternatively, if the same type of machine learning model is used for both the first and second private AI models, parameters such as the number of neurons or the number of layers between the two models may differ.

[0077] In step 308, method 300 may further include: training an adaptive co-distillation model in a first training phase using an input dataset as input and a first result dataset as the first target output. Therefore, given an input dataset as input, the adaptive co-distillation model can be trained to produce the same result data as that produced by the first proprietary AI model. For example, the adaptive co-distillation model can be trained using backpropagation with stochastic gradient descent. When backpropagation is used to train the adaptive co-distillation model, the adaptive co-distillation model may have an adaptive co-distillation model cost function.

[0078] In step 310, method 300 may further include: in a second training phase, further training the adaptive co-distillation model using the input dataset as input and the second result dataset as the second target output. The second training phase may be performed after the first training phase. Similar to the training of the adaptive co-distillation model in the first training phase, training of the adaptive co-distillation model in the second phase can be performed via backpropagation. In the steps of method 300 for training the adaptive co-distillation model, the adaptive co-distillation model is not trained on either the first private dataset or the second private dataset. Therefore, the first private dataset and the second private dataset can remain private to the server computing device. Furthermore, since the adaptive co-distillation model can be trained using the result dataset generated by the first private AI model and the second private AI model without explicitly referencing the model architecture of the first private AI model and the second private AI model, the structure of the first private AI model and the second private AI model can also remain private.

[0079] In method 300, it should be understood that, in at least the first training phase of 308 and typically the second training phase of 310, an adaptive co-distillation model is trained using a training algorithm that utilizes a weighted loss function, as described above. The weighted loss function weights the loss between the predicted output of the adaptive co-distillation model and the target data output of the first and second result datasets using a weighting factor based on one or more of the following: (a) the data distance between elements in the first (or second) input dataset determined by the first (or second) similarity algorithm and the first (or second) private dataset; (b) the confidence value in the first result dataset output by the first or second private AI model; and manually specified inputs, as described above. Figure 8 As shown in the image.

[0080] Figure 15B It shows when execution Figure 15A Method 300 is an additional step that can be performed in some embodiments. In step 312, method 300 may further include training a template machine learning model on a template dataset. The template dataset may include multiple template input entries and multiple template output entries. The template machine learning model may be, for example, a deep neural network, a kernel machine, or a random forest. In some embodiments, the template machine learning model may be trained at a server computing device. In step 314, the method may further include training a first copy of the template machine learning model on a first private dataset. Similarly, in step 316, method 300 may further include training a second copy of the template machine learning model on a second private dataset. Steps 314 and 316 may be performed at a first client computing device and a second client computing device, and the processor of the server computing device may be configured to send copies of the template machine learning model to the first client computing device and the second client computing device. Therefore, the first private dataset and the second private dataset can be used to further train copies of the template machine learning model at the client computing devices.

[0081] In step 318, method 300 may further include homomorphically encrypting the first result dataset or the second result dataset. The encryption algorithm used for homomorphically encrypting the first result dataset or the second result dataset may be a partially homomorphic encryption algorithm, a partially homomorphic encryption algorithm, a hierarchically homomorphic encryption algorithm, or a fully homomorphic encryption algorithm. The first result dataset or the second result dataset may be homomorphically encrypted at the first client computing device or the second client computing device respectively before being sent to the server computing device. Therefore, plaintext versions of the first result entries and the second result entries can remain private.

[0082] Although the exemplary computing system and method described above are illustrated with reference to the first client computing device 80 and the second client computing device 90, the system and method described above can be used with more than two client computing devices. Each client computing device may have a corresponding private dataset for training a private artificial intelligence model. Furthermore, each client computing device may send a corresponding result dataset to the server computing device 10. The processor 12 of the server computing device 10 can train the adaptive co-distillation model 60 sequentially on the corresponding result datasets received from each of the multiple client computing devices. In some embodiments, the processor 12 may use one or more batches comprising two or more result datasets to train the adaptive co-distillation model 60, while simultaneously training the adaptive co-distillation model 60 using two or more result datasets.

[0083] In an exemplary use case scenario, the users of the first and second client computing devices are industrial manufacturers that produce electronic equipment such as inspection devices. In this example, the first client computing device executes a support vector machine to distinguish resistors and capacitors in an image when classifying circuit components. The second client computing device executes a convolutional neural network to distinguish between intact and damaged electronic components. An adaptive co-distillation model is trained using the outputs of both the support vector machine and the convolutional neural network. At runtime, the adaptive co-distillation model distinguishes between intact resistors, intact capacitors, damaged resistors, and damaged capacitors. In this example, neither the support vector machine nor the convolutional neural network is shared between industrial manufacturers, and the data used to train each is kept private for each industrial manufacturer.

[0084] In this example, as an alternative to convolutional neural networks, a heuristic model can be used to distinguish between intact and damaged electronic components. For example, a heuristic model could be an edge detector configured to determine whether an electronic component (resistor or capacitor) has a cracked outer surface. In this example, a second private dataset can be used as the test dataset instead of the training dataset. Although a heuristic edge detector is not a machine learning model, its output can be used together with the output of a support vector machine to train an adaptive co-distillation model.

[0085] In another exemplary use case scenario, the first private AI model is a recurrent neural network configured to generate a text transcription of speech. The second private AI model is another recurrent neural network configured to distinguish between the voices of multiple people whose speech is included in the audio input. Given a shared set of audio input, an adaptive co-distillation model can be trained using the outputs produced by the two recurrent neural networks to generate a text transcription of the speech included in the audio input, and to label each utterance in the transcription as belonging to which person. This is achieved without sharing the individual recurrent neural networks themselves or the data used to train each recurrent neural network.

[0086] In another exemplary use case scenario, a first private AI model is a recurrent neural network configured to control the movement of a robotic arm to move manufactured goods from one area of ​​a factory to another. A second private AI model is another recurrent neural network configured to move an autonomous robot as it navigates the physical environment. A shared input dataset provided to the first and second private AI models includes layout data indicating the size, shape, and position of objects in the factory environment. Using the corresponding outputs of the first and second private AI models, an adaptive co-distillation model is trained to output a combined movement path that moves manufactured goods from one area of ​​the factory environment to another. In each combined movement path, the manufactured goods are moved from an initial position to the autonomous robot by the robotic arm and then to a final position by the autonomous robot. The adaptive co-distillation model is trained to generate combined movement paths without requiring the manufacturers of the robotic arm and the autonomous robot to grant access to their private machine learning models to the users training the adaptive co-distillation model.

[0087] In other exemplary use case scenarios, an adaptive co-distillation model is trained for use in a medical setting. In this example, a first private AI model is a support vector machine configured to identify which bones are present in an X-ray image. A second private AI model is a convolutional neural network configured to determine whether bones in the X-ray image are fractured. An input dataset comprising multiple X-ray images is fed into both the first and second private AI models, and the outputs of the private models are used to train the adaptive co-distillation model. The trained adaptive co-distillation model is configured to receive X-ray images and output corresponding labels indicating which bones (if any) appearing in the X-ray image are fractured.

[0088] In other exemplary use case scenarios, an adaptive co-distillation model is trained to perform traffic flow analysis. In this example, a first private AI model is a recurrent neural network configured to estimate the flow velocity of a group of people moving through a space depicted in video feeds from on-site cameras or stored in the video (e.g., a train station). The first private AI model is a regression model configured to output numerical values. A second private AI model is another regression model configured to estimate the density of people within the space shown in the video. Both the first and second private AI models receive an input dataset comprising multiple videos of crowds moving through various physical environments. Based on the outputs of the first and second private AI models, an adaptive co-distillation model is trained to estimate the degree of traffic congestion (defined as a function of flow velocity and density) of multiple people moving through a physical area. The adaptive co-distillation model is a regression model configured to receive video footage as input and output a numerical estimate of the degree of traffic congestion.

[0089] In some implementations, the methods and processes described herein can be attached to a computing system of one or more computing devices. Specifically, these methods and processes can be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0090] Figure 16 A non-limiting embodiment of a computing system 400 capable of performing one or more of the methods and processes described above is schematically illustrated. The computing system 400 is shown in a simplified form. The computing system 400 may embody the above and... Figure 1 The computing system 1 shown is a computing system 400. The computing system 400 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, and wearable computing devices (such as smartwatches and head-mounted augmented reality devices).

[0091] The computing system 400 includes a logic processor 402, volatile memory 404, and non-volatile storage device 406. The computing system 400 may optionally include a display subsystem 408, an input subsystem 410, a communication subsystem 412, and / or... Figure 16 Other components not shown.

[0092] The logic processor 402 includes one or more physical devices configured to execute instructions. For example, the logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions can be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise obtain desired results.

[0093] The logic processor may include one or more physical processors (hardware) configured to execute software instructions. Alternatively, the logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of logic processor 402 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic processor may optionally be distributed across two or more separate devices that may be remotely located and / or configured for collaborative processing. Aspects of the logic processor may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In this context, it should be understood that these virtualized aspects run on different physical logic processors on various different machines.

[0094] The non-volatile storage device 406 includes one or more physical devices configured to hold instructions executable by a logic processor to implement the methods and processes described herein. When implementing these methods and processes, the state of the non-volatile storage device 406 can be transitioned—for example, to maintain different data.

[0095] Non-volatile storage device 406 may include removable and / or built-in physical devices. Non-volatile storage device 406 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.) and / or magnetic memory (e.g., hard disk drive, floppy disk drive, magnetic tape drive, MRAM, etc.), or other high-capacity storage device technologies. Non-volatile storage device 406 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that non-volatile storage device 406 is configured to retain instructions even when power to the non-volatile storage device is cut off.

[0096] Volatile memory 404 may include a physical device including random access memory. Volatile memory 404 is typically used by logic processor 402 to temporarily store information during the processing of software instructions. It should be understood that volatile memory 404 typically does not continue storing instructions when power to volatile memory 404 is cut off.

[0097] The logic processor 402, volatile memory 404, and non-volatile storage device 406 can be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SOCs), and complex programmable logic devices (CPLDs).

[0098] The terms "module," "program," and "engine" can be used to describe aspects of a computing system 400 typically implemented in software by a processor to perform specific functions using portions of volatile memory, involving translational processing specifically configured for that function. Thus, a module, program, or engine can be instantiated by a logical processor 402 executing instructions held by non-volatile memory 406, or by using portions of volatile memory 404. It should be understood that different modules, programs, and / or engines can be instantiated from the same applications, services, code blocks, objects, libraries, routines, APIs, functions, etc. Similarly, the same modules, programs, and / or engines can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass single or grouped executable files, data files, libraries, drives, scripts, database records, etc.

[0099] When a display subsystem 408 is included, it can be used to present a visual representation of data held by the non-volatile storage device 406. The visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein alter the data held by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 408 can also be transformed to visually represent changes in the underlying data. The display subsystem 408 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic processor 402, the volatile memory 404, and / or the non-volatile storage device 406 in a shared housing, or such display devices may be peripheral display devices.

[0100] When the input subsystem 410 is included, it may include or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or interface with a selected Natural User Input (NUI) component. Such a component may be integrated or peripheral, and the translation and / or processing of input actions may be performed on-board or off-board. Examples of NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0101] When a communication subsystem 412 is included, the communication subsystem 412 may be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 412 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network (such as an HDMI connection via Wi-Fi). In some embodiments, the communication subsystem may allow the computing system 400 to send messages to and / or receive messages from other devices via a network (such as the Internet).

[0102] According to one aspect of this disclosure, a method for use with a computing device is provided. The method may include inputting an input dataset into a first private artificial intelligence model generated using a first private dataset and a second private artificial intelligence model generated using a second private dataset. The method may further include receiving a first result dataset from the first private artificial intelligence model as a result of applying the first private artificial intelligence model to the input dataset. The method may further include receiving a second result dataset from the second private artificial intelligence model as a result of applying the second private artificial intelligence model to the input dataset. In a first training phase, the method may further include training an adaptive co-distillation model using the input dataset as input and the first result dataset as a first target output. In a second training phase, the method may further include training the adaptive co-distillation model using the input dataset as input and the second result dataset as a second target output. The adaptive co-distillation model may not be trained on either the first or second private dataset.

[0103] According to this aspect, the first private AI model may have a first model architecture, and the second private AI model may have a second model architecture that is different from the first model architecture.

[0104] According to this, each of the first and second private AI models is a deep neural network, a kernel machine, or a random forest.

[0105] Based on this aspect, the adaptive co-distillation model can be a classification model. The first and second result datasets can each include multiple corresponding classification labels.

[0106] According to this aspect, the input dataset can be a partially labeled dataset, which includes a first subset of input data entries with corresponding input classification labels and a second subset of input data entries without corresponding input classification labels.

[0107] Based on this, the adaptive co-distillation model is a regression model. The first and second result datasets can each include multiple numerical values.

[0108] Based on this, the adaptive co-distillation model can be a recurrent neural network. The input dataset may include multiple input sequences, each containing multiple ordered input values.

[0109] Based on this, at least in the first training phase, an adaptive co-distillation model can be trained using a training algorithm that utilizes a weighted loss function.

[0110] According to this aspect, the weighted loss function can weight the loss between the predicted output of the adaptive co-distillation model and the target data output of the first result dataset by a weighting factor based on one or more of the following: the data distance between the elements in the input dataset and the first private dataset determined by the first similarity algorithm, the confidence value in the first result dataset, and the manually specified input.

[0111] Based on this, either the first result dataset or the second result dataset can be homomorphically encrypted.

[0112] Based on this, adaptive co-distillation models can be trained, at least in part, through supervised learning.

[0113] According to another aspect of this disclosure, a computing system is provided, comprising a server computing device including a processor. The processor is configured to send an input dataset to a first client computing device configured to execute a first private artificial intelligence model generated using a first private dataset. The processor is also configured to send an input dataset to a second client computing device configured to execute a second private artificial intelligence model generated using a second private dataset. The processor is further configured to receive a first result dataset from the first private artificial intelligence model executed at the first client computing device as a result of applying the first private artificial intelligence model to the input dataset. The processor is also configured to receive a second result dataset from the second private artificial intelligence model executed at the second client computing device as a result of applying the second private artificial intelligence model to the input dataset. In a first training phase, the processor is further configured to train an adaptive co-distillation model using the input dataset as input and the first result dataset as a first target output. In a second training phase, the processor is further configured to further train the adaptive co-distillation model using the input dataset as input and the second result dataset as a second target output. The adaptive co-distillation model may not be trained on either the first or second private dataset.

[0114] According to this aspect, the first private AI model may have a first model architecture, and the second private AI model may have a second model architecture that is different from the first model architecture.

[0115] According to this, each of the first private AI model and the second private AI model can be a deep neural network, a kernel machine, or a random forest.

[0116] Based on this, the adaptive co-distillation model can be a classification model. The first and second result datasets can each include multiple corresponding classification labels.

[0117] Based on this, the adaptive co-distillation model can be a regression model. The first and second result datasets can each include multiple corresponding numerical values.

[0118] Based on this, the adaptive co-distillation model can be a recurrent neural network. The input dataset may include multiple input sequences, each containing multiple ordered input values.

[0119] According to this aspect, the processor can also be configured to train a template machine learning model on a template dataset. The processor can also be configured to send the template machine learning model to a first client computing device and a second client computing device. The first private AI model can be a first copy of the template machine learning model that has been further trained on the first private dataset. The second private AI model can be a second copy of the template machine learning model that has been further trained on the second private dataset.

[0120] Based on this, adaptive co-distillation models can be trained, at least in part, through supervised learning.

[0121] According to another aspect of this disclosure, a method for use with a computing device is provided. The method may include inputting an input dataset into a first private artificial intelligence model generated using a first private dataset and a second private artificial intelligence model generated using a second private dataset. The method may also include receiving a first result dataset from the first private artificial intelligence model as a result of applying the first private artificial intelligence model to the input dataset. The first result dataset may include a plurality of first classification labels. The method may further include receiving a second result dataset from the second private artificial intelligence model as a result of applying the second private artificial intelligence model to the input dataset. The second result dataset may include a plurality of second classification labels. In a first training phase, the method may further include training an adaptive co-distillation model using the input dataset as input and the first result dataset as a first target output. In a second training phase, the method may further include further training the adaptive co-distillation model using the input dataset as input and the second result dataset as a second target output. The adaptive co-distillation model may not be trained on either the first or second private dataset. The method may further include receiving a runtime dataset comprising a plurality of runtime input data entries. For each of the plurality of runtime input data entries, the method may further include outputting a runtime classification label selected from a combined set of classification labels comprising a plurality of first classification labels and a plurality of second classification labels.

[0122] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific implementations or examples should not be considered limiting, as many variations are possible. The particular routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions illustrated and / or described may be performed in the illustrated and / or described order, in another order, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0123] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of different processes, systems and configurations, as well as other features, functions, actions, and / or properties disclosed herein, and any and all their equivalents.

Claims

1. A method for use with a computing device, the computing device including a processor, the method comprising: Using the processor: The input dataset is input to a first client computing device and a second client computing device, the first client computing device being configured to execute a first private artificial intelligence model generated using the first private dataset, and the second client computing device being configured to execute a second private artificial intelligence model generated using the second private dataset; As a result of applying the first private artificial intelligence model to the input dataset, a first result dataset is received from the first private artificial intelligence model; As a result of applying the second private AI model to the input dataset, a second result dataset is received from the second private AI model; In the first training phase, an adaptive co-distillation model is trained using the input dataset as input and the first result dataset as the first target output, wherein... At least in the first training phase, the adaptive co-distillation model is trained using a training algorithm that utilizes a weighted loss function; and The weighted loss function weights the loss between the predicted output of the adaptive co-distillation model and the first result dataset using a weighting factor. This weighting factor is based on one or more of the following: the data distance between elements in the input dataset determined by a first similarity algorithm and the first private dataset; the confidence value in the first result dataset; and manually specified input. In the second training phase, the adaptive co-distillation model is further trained using the input dataset as input and the second result dataset as the second target output. The adaptive co-distillation model is not trained on either the first private dataset or the second private dataset.

2. The method according to claim 1, wherein, The first private AI model has a first model architecture, and the second private AI model has a second model architecture that is different from the first model architecture.

3. The method according to claim 2, wherein, Each of the first private AI model and the second private AI model is a deep neural network, a kernel machine, or a random forest.

4. The method according to claim 1, wherein: The adaptive co-distillation model is a classification model; and The first result dataset and the second result dataset each include a number of corresponding classification labels.

5. The method according to claim 4, wherein, The input dataset is a partially labeled dataset, which includes a first subset of input data entries with corresponding input category labels and a second subset of input data entries without corresponding input category labels.

6. The method according to claim 1, wherein: The adaptive co-distillation model is a regression model; and The first result dataset and the second result dataset each include a number of corresponding numerical values.

7. The method according to claim 1, wherein: The adaptive co-distillation model is a recurrent neural network; and The input dataset includes multiple input sequences, and each input sequence includes multiple ordered input values.

8. The method according to claim 1, wherein, Homomorphic encryption is applied to either the first result dataset or the second result dataset.

9. The method according to claim 1, wherein, The adaptive co-distillation model is trained, at least in part, through supervised learning.

10. A computing system, comprising: A server computing device, including a processor, the processor being configured to: Send the input dataset to the following devices: A first client computing device is configured to execute a first private artificial intelligence model generated using a first private dataset; as well as The second client computing device is configured to execute a second private artificial intelligence model generated using a second private dataset. As a result of applying the first private artificial intelligence model to the input dataset, a first result dataset is received from the first private artificial intelligence model executed at the first client computing device; As a result of applying the second private artificial intelligence model to the input dataset, a second result dataset is received from the second private artificial intelligence model executed at the second client computing device; In the first training phase, an adaptive co-distillation model is trained using the input dataset as input and the first result dataset as the first target output, wherein... At least in the first training phase, the adaptive co-distillation model is trained using a training algorithm that utilizes a weighted loss function; and The weighted loss function weights the loss between the predicted output of the adaptive co-distillation model and the first result dataset using a weighting factor. This weighting factor is based on one or more of the following: the data distance between elements in the input dataset determined by a first similarity algorithm and the first private dataset; the confidence value in the first result dataset; and manually specified input. In the second training phase, the adaptive co-distillation model is further trained using the input dataset as input and the second result dataset as the second target output. The adaptive co-distillation model is not trained on either the first private dataset or the second private dataset.

11. The computing system according to claim 10, wherein, The first private AI model has a first model architecture, and the second private AI model has a second model architecture that is different from the first model architecture.

12. The computing system according to claim 11, wherein, Each of the first private AI model and the second private AI model is a deep neural network, a kernel machine, or a random forest.

13. The computing system according to claim 10, wherein: The adaptive co-distillation model is a classification model; and The first result dataset and the second result dataset each include a number of corresponding classification labels.

14. The computing system according to claim 10, wherein: The adaptive co-distillation model is a regression model; and The first result dataset and the second result dataset each include a number of corresponding numerical values.

15. The computing system according to claim 10, wherein: The adaptive co-distillation model is a recurrent neural network; and The input dataset includes multiple input sequences, and each input sequence includes multiple ordered input values.

16. The computing system according to claim 10, wherein, The processor is also configured to: Train a template machine learning model on a template dataset; and The template machine learning model is sent to the first client computing device and the second client computing device, wherein: The first private artificial intelligence model is a first copy of the template machine learning model that has been further trained on the first private dataset; and The second private artificial intelligence model is a second copy of the template machine learning model that has been further trained on the second private dataset.

17. The computing system according to claim 10, wherein, The adaptive co-distillation model is trained, at least in part, through supervised learning.

18. A method for use with a computing device, the computing device including a processor, the method comprising: Using the processor: The input dataset is input to a first client computing device and a second client computing device, the first client computing device being configured to execute a first private artificial intelligence model generated using the first private dataset, and the second client computing device being configured to execute a second private artificial intelligence model generated using the second private dataset; As a result of applying the first private artificial intelligence model to the input dataset, a first result dataset is received from the first private artificial intelligence model, wherein the first result dataset includes a plurality of first category labels; As a result of applying the second private AI model to the input dataset, a second result dataset is received from the second private AI model, wherein the second result dataset includes a plurality of second classification labels; In the first training phase, an adaptive co-distillation model is trained using the input dataset as input and the first result dataset as the first target output, wherein... At least in the first training phase, the adaptive co-distillation model is trained using a training algorithm that utilizes a weighted loss function; and The weighted loss function weights the loss between the predicted output of the adaptive co-distillation model and the first result dataset using a weighting factor, which is based on one or more of the following: the data distance between the elements in the input dataset determined by the first similarity algorithm and the first private dataset, the confidence value in the first result dataset, and the manually specified input; In the second training phase, the adaptive co-distillation model is further trained using the input dataset as the input and the second result dataset as the second target output, wherein the adaptive co-distillation model is not trained on the first private dataset or the second private dataset. It receives a runtime dataset that includes multiple runtime input data entries; and For each of the plurality of runtime input data entries, the output is selected from a runtime classification label set that includes the plurality of first classification labels and the plurality of second classification labels.

Images