System and method for generating an integrated model

Abstract

A system and method for generating an integrated model is provided. The system includes a central server having a plurality of clients, each client privately storing data accessible to the central server. The model trainer is configured to receive model suggestions from the first client and train the integrated model based on the model suggestions and private storage data of the plurality of clients without exposing the private storage data of the plurality of clients to the first client. The model deployer is configured to propagate the trained integrated model to one or more of the plurality of clients.

Description

Technical Field

[0001] The present invention relates to the field of federated machine learning, and more particularly to securely federating models across data stored in separate cloud-based environments. Background Art

[0002] Aircraft and / or other machine component damage or failure is associated with costs. In the case of commercial airlines, one cost manifests itself in flight delays, unhappy customers, and / or lost revenue. Therefore, successfully repairing and / or replacing components prior to damage or failure can help reduce such costs. Summary of the invention

[0003] The present invention provides a system for generating an integrated model. The system includes a central server with multiple clients, each client privately storing data accessible to the central server. A model trainer is configured to receive a model suggestion from a first client, and train the integrated model based on the model suggestion and the privately stored data of the multiple clients without exposing the privately stored data of the multiple clients to the first client. A model deployer is configured to propagate the trained integrated model to one or more of the multiple clients. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 An example aircraft is depicted.

[0005] Figure 2 An example system for generating an integrated model is schematically illustrated.

[0006] Figure 3 Features of an example test system that can generate data for performing model training and federation are schematically depicted.

[0007] Figure 4 Aspects of a system for training an ensemble model are schematically illustrated.

[0008] Figure 5 Aspects of the system employed in the training of multiple local models are schematically illustrated.

[0009] Figure 6 Aspects of a system employed in fusing local models into a federated model are schematically illustrated.

[0010] Figure 7 Aspects of the system employed in training an assisted joint model are schematically illustrated.

[0011] Figure 8 A method for generating an integrated model is shown.

[0012] Fig. 9 An example computing system is schematically illustrated. DETAILED DESCRIPTION

[0013] Prognostic and predictive modeling can be used in many applications, including engineering, automotive, hydrological engineering, electronics, medical devices, medical treatments, etc. Such prognostic models may include predictive models, regression models, classifiers, anomaly detection models, fault prediction models, remainder prediction models, and / or other types of machine learning and / or other artificial intelligence modules. Modern modeling algorithms (such as logistic regression, decision trees, and / or neural networks) are more effective when they parse data that incorporates a large number of events as input / training data. For sparse event occurrences (e.g., rare diseases), the available data sets may not be able to separate accurate predictions from noise.

[0014] As an example, prognostic and predictive models of aircraft components can be used to predict safe replacement scenarios. One example is predicting safe replacement scenarios for the spoiler power control unit. Figure 1 An exemplary aircraft 100 is shown including a right wing 102A and a left wing 102B. The right wing 102A includes a wing spoiler 104A, while the left wing 102B includes a wing spoiler 104B. The wing spoilers 104A and 104B can be used to extend into or retract from the airflow as the aircraft 100 moves, thereby changing the drag and lift characteristics of the wing, allowing changes in speed, descent rate, roll, etc. The positioning of the spoiler power control units (106A and 106B) on the wings of the aircraft exposes the control units to moisture, and therefore preventative maintenance / replacement can enhance the safety and profitability of the aircraft.

[0015] However, the application of predictive maintenance in commercial aviation is limited by the lack of sufficient data related to failures in training prognostic models. Prognostic models trained on a moderate amount of data associated with such failures tend to be unreliable with poor precision and / or poor recall.

[0016] In an ideal world, different airlines using the same aircraft would pool their data, generating a dataset with more total failure events, which would allow for a more accurate model to be produced. However, while customers expect a better set of predictors in the future, airlines may be reluctant to share sensitive data with each other.

[0017] As such, most current solutions are limited to developing prognostic models based on data collected by a single airline. Therefore, the opportunity to implement predictive maintenance is limited to the problem of having a sufficient number of example failure events in the airline's historical data. This typically limits model generation to airlines with relatively large fleet sizes.

[0018] In this article, systems and methods are described that utilize federated machine learning to address these and other problems. Federated machine learning provides an opportunity to improve the performance of these predictive models by combining models trained across multiple clients (e.g., different airlines) without the need to disclose private data to other clients (e.g., competitor airlines). This federated machine learning approach addresses challenges associated with the training of reliable prognostic models in commercial aviation and other fields, including limited availability of data associated with device failures and restrictions that prevent data sharing between clients.

[0019] In some examples, for example, clients such as airlines are encouraged to submit candidate models for different problems via an accessible web portal. The selected candidate models will then be federated across the individual environments of the participating clients. Each of the participating clients may be notified of the new candidate model and contribute data to further model training securely and privately without making the data accessible to other clients. Each of the participating clients may then be notified of the new federated model and may then select the new federated model for deployment to their own environment.

[0020] Such an approach is beneficial in facilitating the development of reliable prognostic models for use in a variety of environments. Model training and data preparation can be maintained in separate environments, providing secure training and rapid federation of prognostic models without requiring clients to lose the privacy and / or security of their data. Prognostic models can be rapidly federated and deployed for use by clients.

[0021] Figure 2 A system 200 for training an integrated model is schematically depicted. The system 200 includes a central server 205 having a plurality of clients. In this example, three clients are shown—a first client 210, a second client 212, and a third client 214. However, any number of at least two clients may be communicatively coupled to the central server 205.

[0022] Each client privately stores data accessible to the central server 205 (represented by the dashed lines). For example, each client may store data locally (e.g., data 220, 222, and 224) and provide selective access to at least some of the data of the client to the central server 205. In some examples, each client may selectively upload data to the central server 205, and the central server 205 may securely store at least some of the data (e.g., credentialed and / or encrypted) on behalf of the client. In some examples, the central server itself or a portion thereof may not have access to the decoded data values. In other words, the central server 205 can be used as a data warehouse for multiple different clients, allowing each client to privately maintain its own data relative to each other client within the architecture of the system 200. In any case, the system 200 encourages each client to actively participate in model training. Data and pre-models can be intentionally contributed by each client, allowing the system 200 to generate and train integrated models based on their contributions.

[0023] Although primarily described with respect to the example of aircraft mechanical maintenance, clients 210, 212, and 214 may be organizations in a common field such as engineering, meteorology, medicine, languages, etc. Figure 1 As described, multiple clients may be airlines that employ a common aircraft model. In such an example, the data stored by each client may have common characteristics that allow the data to be integrated into a larger group. Each client may operate in a unique environment, for example, an airline's aircraft may be exposed to a unique set of environmental conditions and operating conditions. In other words, each client's data may be more common internally than other clients' data.

[0024] Many central servers offer multiple services and have already established data pipelines with their clients. The system herein can leverage this pre-existing architecture and set of digital relationships. Such an arrangement can lend itself to subscription services. For example, data can be uploaded from each client as part of a subscription model or in response to a specific prompt (such as a new model coming online).

[0025] The central server 205 includes at least a model trainer 230, a model federator 232, and a model deployer 234. The central server 205 also includes at least a logic machine 236 and a storage machine 238. The logic machine 236 may include, for example, one or more processors, and the storage machine 238 may include any suitable combination of non-volatile storage media and volatile storage media. A more detailed description of the logic machine and the storage machine may be found in this document and with reference to Fig. 9 turn up.

[0026] The model trainer 230 may be configured to train the machine learning model according to any suitable training process. Non-limiting examples of training processes for machine learning models include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on classes derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning), and / or generative adversarial neural network-based training methods.

[0027] The machine learning model can be trained on an objective function via supervised training on an input data set, which measures the accuracy, precision and / or recall of the likelihood of a specified event determined by the machine learning model compared to the actual occurrence of the specified event indicated in the input data. In some instances, the training is supervised or partially supervised using indications of ground truth. As used herein, "ground truth" refers to information that is known to be true or real and is typically input by a human analyst or user. Once trained, the machine learning model can be any of a plurality of model types. The trained machine learning model can be deployed to predict events that occur, predict what may happen after an event, classify events, detect anomalies, perform regression, perform inference, and the like. In this way, the model trainer 230 can support different types of integrated models required to characterize different events, such as a device safety replacement indicator.

[0028] The model trainer 230 can randomly initialize the weights and biases of the untrained models so that a unique configuration of weights and biases is applied to each model. The training function of the model trainer 230 uses the input data and the randomly initialized weights and biases to train one or more untrained models and generate one or more trained models.

[0029] The training function may use one or more back-propagation algorithms to update or adjust weights and biases of one or more untrained models based on a comparison between prediction coefficients output by one or more trained models and labels in the input data. The training function may include a loss function to measure the error between the prediction coefficients output by the one or more models and the labels in the input data, such that the measured error may be minimized with each iteration of the back-propagation algorithm. This iterative process of updating or adjusting weights and biases may continue until the output of the prediction coefficients output by the one or more models converges with the labels in the input data, thereby indicating convergence or minimization of the loss function. The proposed model is trained using private feature data of the corresponding client without sharing or abandoning the feature data.

[0030] Model deployer 234 can package, compare, and / or develop interfaces for machine learning models before delivering them or publishing them for client retrieval. Model deployer can deploy both local (e.g., single client / environment) models and federated models, e.g., to make predictions on new data. Model deployer 234 can be configured to push model files and / or indications of new and / or updated models. In some examples, model deployer 234 can evaluate different candidate models for a given problem, propagating the best performing model to participating clients.

[0031] The model unionizer 232 may combine multiple individual models into a single union model. The model unionizer 232 receives two or more trained local models that have been trained on existing private data of the corresponding client. Each received trained model reflects the model attributes of the corresponding client by applying the local existing private feature data to a feature mapper that maps the feature data of the corresponding client to the trained local model. For example, the trained model attributes may include weights given to model variables and hidden layers included in a neural network model. The trained local model may be a common analysis model across all clients, thereby achieving the union of models.

[0032] The model unionizer 232 may generate an integrated aggregate model by unionizing models from a group of similarly trained local models using a federated learning technique (e.g., federated stochastic gradient descent, federated averaging, federated learning with dynamic regularization, dynamic aggregation such as reverse distance aggregation, and hybrid federated dual coordinate ascent).

[0033] The model federator 232 can receive models of similar client assets of the group and, using a federated learning technique, federate the multiple models into a single aggregate model. The model federator 232 uses a federated learning technique to produce a merger of the multiple models, thereby producing a model that reflects a higher quality level obtained by applying the model to a larger feature data set, but in this case, without sharing or abandoning the corresponding client feature data set.

[0034] The client asset models received by the model federator 232 have been trained by processing existing local client (e.g., private) asset feature data mapped to canonical input features of the model using a feature mapper previously trained on a seed model by each respective client. The respective client asset models received by the model federator 232 reflect an output level of accuracy based solely on the respective feature data of the individual client assets. Each received model includes updated parameters and weights based on the respective feature data of the respective client data. Federating the parameters and weights based on a technique such as federated averaging results in a single aggregate model having the combined machine learning accuracy of the respective feature data sets of multiple client assets without exposing or sharing the feature data.

[0035] Thus, the central server 205 can enable rapid federation of integrated models by automating data preparation and model training within individual client environments, combining the trained local models into a federated model, and delivering the federated model to each airline.

[0036] The central server 205 can act as a federated learning portal that will allow clients to submit candidate models for different problems. These candidate models can then be trained and federated across the individual environments of the participating clients. Each of the participating clients can be notified of the newly federated model, which can then be optionally deployed to their own environment.

[0037] As such, the central server 205 provides a community benefit where even if some clients do not employ their own data scientists, they can sign and contribute the value of their data without exposing their sensitive data to other clients. Each participating client still gains the benefit of the person developing the model. Clients with data scientists but with limited data gain the value of leveraging data from other operators. In this way, the client is the direct beneficiary of the system, rather than the service provider, who does not gain value from the generated model.

[0038] Figure 3 is a schematic diagram depicting features of an example test system 300 that can generate data for performing model training and federation as disclosed herein. The test system 300 can be applied to an electromechanical test object 310 that is Figure 3 In the example shown in FIG. 3 , this takes the form of aircraft 312 .

[0039] The test system 300 includes a test computing system 320 of one or more computing devices 322. The one or more computing devices 322 collectively include a logic subsystem 324 of one or more logic devices 326 (e.g., a processor), a storage subsystem 328 of one or more data storage devices 330, and an input / output subsystem 332 through which the test computing system 320 can communicate with other devices. Fig. 9 Find additional description of an example computing device.

[0040] As an example, the test computing system 320 can communicate with the electromechanical component 314, which is integrated or otherwise interfaced with the electromechanical test object 310. As an example, Figure 3 Aircraft 312 of the type shown may take the form of a multi-passenger, commercial airliner, and / or cargo aircraft having onboard electromechanical components integrated with the aircraft. In at least some examples, communications between electromechanical components 314 and test computing system 320 may be through one or more intermediate communication networks 304 that provide one or more wired communication links and / or one or more wireless communication links utilizing wired and / or wireless communication protocols.

[0041] A human user, schematically represented at 316, may access the computer via one or more computer terminals 318 (in Figure 3 310) interacts with a test computing system 320 and / or electromechanical components 314 associated with the test object 310. Communications between the terminal 318, the test computing system 320, and the electromechanical components 314 can traverse the network 304. In at least some examples, one or more terminals 318 can be integrated with the test computing system 320, and / or one or more terminals 318 can be integrated with the electromechanical components 314 of the test object.

[0042] Electromechanical components 314 Figure 3 310 , and includes an onboard computing system 340 , an onboard data network 342 , a set of one or more sensors 344 associated with the test object 310 , and one or more integrated electromechanical components 346 of the test object including electronic components 348 and mechanical components 350 .

[0043] The onboard computing system 340 may similarly include one or more of the components previously described with reference to the test computing system 320. In an example, the onboard computing system 340 may be integrated with the test object 310 and may be used to control the operation of the test object. Within the scope of the example aircraft 312, an onboard data network 342 may be used to manage data flows between various systems and subsystems of the aircraft, including the onboard computing system 340, and power subsystems, electromechanical subsystems, cockpit subsystems, attendant subsystems, entertainment subsystems, etc. of integrated electromechanical components 346, and sensor subsystems including sensors 344. As an example, the onboard data network 342 may include an integrated aircraft data network, such as a common data network (CDN). The onboard data network 342 may include bidirectional fiber optic and / or copper network path components, bridges, switches, routers, hubs, etc., over which communications and / or power are transmitted using a set of communication protocols and standards.

[0044] Sensors 344 may be integrated with various integrated electromechanical components 346 of electromechanical test object 310 and / or physically interfaced with the test object by a technician during production or maintenance. Sensors 344 may be used to measure the operation and performance of integrated electromechanical components 346 during testing, maintenance, and / or operation of test object 310. Sensors 334 may include various sensor types that measure electrical properties of electronic components 348, such as voltage, resistance, current, impedance, capacitance, power, etc. As an example, sensors 344 may include a battery voltage sensor that measures the voltage of a battery located onboard the test object. Sensors 344 may include various sensor types that measure physical, non-electrical properties of mechanical components 350 or other physical characteristics of the test object, such as position, velocity, acceleration, force, work, pulse, pressure, temperature, quantity, presence, etc. As an example, sensors 344 may include a hydraulic pressure sensor in a hydraulic actuator responsible for positioning a flight control surface and a position sensor that measures the positioning of the flight control surface.

[0045] Figure 4 An example scenario 400 is depicted in which the system 200 generates and deploys an integrated model 405. The model trainer 230 may be a neural network model trainer configured to train one or more untrained models based on model proposals 410. The first client 210 may submit the model proposals 410 to the model trainer 230 along with related data 420. For example, the first client 210 may submit model proposals related to an event category and data related to the event category. For example, if the event category includes aircraft component failures, the related data may include instances of that failure type and data collected from events that led to those failures.

[0046] Data 420 may include statistical data, such as Figure 3Data for an aircraft description. Data 420 may also include tags, such as event tags, data type tags, coefficients, etc. Model trainer 230 may be configured to train the model on a per-client basis using private information from a single client to train the model. Model trainer 230 may also coordinate the training of integrated model 405 within the separate environment of each client (e.g., first client 210, second client 212, and third client 214).

[0047] The system 200 may support various types of integrated models 405, such as those required to characterize maintenance triggers for different devices. A maintenance trigger may include repairing or replacing a component based on one or more indicators that the component exhibits reduced performance, an impending failure, and / or a complete failure. The model trainer 230 may be configured to receive model suggestions 410 from the first client 210. For example, the first client 210 may submit a model suggestion 410 that predicts the likelihood of a future event occurring, such as a maintenance trigger for an aircraft component. As a non-limiting example, the model suggestion 410 may suggest a correlation between average operating temperature and spoiler power control unit performance degradation. The model trainer 230 may then train a suggestion model based on the model suggestion 410 of the first client 210 and private storage data 420. The resulting model may be considered to be a first trained local model.

[0048] The model deployer 234 can propagate the model suggestion and / or the first trained local model to at least some of the multiple clients (e.g., the second client 212 and the third client 214) without exposing the private storage data (e.g., data 422, 424) of the multiple clients to the first client 210 before training the integrated model 405 (represented by the dashed line). In response to the propagation of the model suggestion, one or more additional clients can access (e.g., upload, change permissions) the private storage data. The central server 205 can thus store some information that is not available to other clients, while anonymously revealing data that is useful in training a particular integrated model to allow the largest possible group of training data. Clients can thus contribute the value of their private storage data without exposing their data to other clients.

[0049] Each client can manage how their data is used and accessible. In some examples, clients that do not have relevant event data may not participate in the generation of the integrated model. In other examples, there may be other peripheral data that can enhance the negative correlation when submitted by the client. When the model trainer 230 is provided with a larger data set from multiple clients, then the model trainer 230 can train the integrated model 405 based on the model recommendations 410. Training the integrated model 405 may include retraining the first trained local model.

[0050] The integrated model 405 can then be propagated to clients via the model deployer 234. Clients 210, 212, and 214 gain the benefit of operating a well-trained integrated model without having to do any training locally. In addition, the model recommendations 410 are trained using a large data set without requiring any client to expose its sensitive data to any other client.

[0051] In some examples, the model trainer can train multiple local models and establish parameters for each local model. Thus, the model trainer can coordinate the training of local models within the individual environments of each client.

[0052] Figure 5 An example scenario 500 is shown for training multiple local models based on a single model suggestion using a model trainer 505. The model trainer 505 can be an example of the model trainer 230. A first client 510 submits a model suggestion 512 and accompanying related data 514 to the model trainer 505 via a central server. The model trainer 505 trains a first trained local model 516 based on the model suggestion 512 and the data 514, and sets parameters for the first trained local model 516.

[0053] Model suggestion 512 and / or first trained local model 516 can be propagated to multiple clients via a model deployer. A second client 520 can then provide access to data 524, and a third client 530 can provide access to data 534.

[0054] The model trainer 505 can then train one or more additional trained local models, each additional trained local model based on the model suggestion and the privately stored data of one additional client, without exposing the privately stored data of the one additional client to the first client. In other words, the model trainer 505 trains the local model of each client based on the contribution data of the client's environment. Data 524 produces a second trained local model 526, and data 534 produces a third trained local model 536. The first client 510, the second client 520, and the third client 530 cannot access each other's data (represented by the dotted lines). In some examples, the model trainer 505 can act within each client space to run the model training process. The model trainer 505 can then return from the client space to obtain the model parameters of the client. As such, the model training process itself can be maintained as a secure and private process.

[0055] The privately stored data used to train the additional trained local model includes one or more instances of previous occurrences of the event being modeled. For example, the initial set of local data 514 includes one or more instances of events in the event category. The data of the auxiliary client (data 524, data 534) also includes one or more instances of events in the event category.

[0056] Clients may not share strict specifications for how they label their data, or even how they identify and classify key events. The central server may also be configured to align metadata from two or more clients prior to model training. For example, if certain signals are labeled differently, those signals may be given a common descriptor. If one client restricts its dataset so that a particular set of signals is not included, then the client's datasets may be aligned so that equivalent signals are sorted together. Clients may have differences in opinions and, therefore, in labeling regarding what constitutes an event (e.g., a part maintenance trigger) and other target variables. These events may be aligned to a common definition prior to initiating model training.

[0057] As an example, Table 1 shows data for four example clients, in this example airline operators Air1, Air2, Air3, and Air4. Each operator contributes data for multiple events (e.g., maintenance triggers) for a given fleet size. The precision and recall of the trained local model for each operator are shown.

[0058] Table 1 - Local Model

[0059]

[0060] In this example, each of Air1, Air2, Air3, and Air4 contributes data from a unique environment and thus generates a unique trained local model. While Air4 has the largest fleet size, Air1 has the largest number of maintenance triggers and thus can be considered to contribute the largest dataset.

[0061] In an example of establishing a client network, a first client 510 can push model recommendations 512 to a second client 520 and a third client 530 using direct peer-to-peer sharing, thereby bypassing the model trainer 505 and any model deployer. In an example where the first client 510 is configured to train a model locally, a first trained local model 516 can be pushed directly to other clients. In these examples, "pushing" a model or model recommendation can include pushing a notification about the availability of a new model or model recommendation. In an example where one or more clients employ their own model trainer, each of the one or more clients can upload parameters for a trained local model to a central server as an alternative or in addition to raw local data.

[0062] As about Figure 6 As described, once multiple local models have been trained, the model federator can federate the local models into a global federated model with a single set of parameter values, and the central server can propagate the global model to clients via the model deployer.

[0063] Figure 6 Scenario 600 is depicted in which a model unionizer 605 combines multiple individual locally trained prognostic models into a single union model. Scenario 600 follows scenario 500, such that a first trained local model 516, a second trained local model 526, and a third trained local model 536 have been trained by model trainer 505.

[0064] First trained local model 516, second trained local model 526, and third trained local model 536 are fed to model federator 605. Model federator 605 may be an example of model federator 232. Model federation may allow for the development of better performing prognostic models.

[0065] The model federator 605 is configured to federate the first trained local model 516 and one or more additional trained local models (526, 536) to generate a trained federated model 610. The federated model 610 can be considered an example of an integrated model (such as the integrated model 405) because data from multiple sources are integrated during training. Each trained local model can include a set of parameters and a set of results. The trained federated model 610 can include one or more parameters that are different from or otherwise refined than the parameters of the first trained local model 516. This can generate a federated model 610 that performs better than any trained local model, taking advantage of the benefits of data from each client without exposing the data to other clients in the group. The model federator 605 can use different aggregation protocols depending on the initial data, model parameters, and model proposals. In some examples, a voting system can increase the likelihood of obtaining a monotonic benefit using a trained prognostic model.

[0066] As an example, Table 2 shows the recall and precision evaluation of local and prognostic models for four airlines. In this example, four relatively weak-performing local models are combined into a higher-performing prognostic model. As shown, the combined prognostic model shows at least equal accuracy relative to each local model, outperforming the local models for three of the four clients.

[0067] Table 2 - Local model and joint model

[0068]

[0069] The model deployer 612 is configured to propagate the trained federated model 610 to one or more of a plurality of clients (e.g., clients 510, 520, 530). The model deployer 612 may also propagate the first trained local model 516 to the first client 510, and may propagate each additional trained local model (526, 536) to the respective additional clients (520, 530). The propagated models may then be employed at least for the purpose of making predictions on new data. Each client that contributes data and / or has a local model trained on its data may be provided with a copy of the federated model because the federated model has been trained on that client's data.

[0070] In the classic example of federated machine learning, the model is deployed directly to the service provider, while in this example, there is peer-to-peer sharing of the model. In addition, the model is usually targeted at the user's behavior - e.g., typing patterns, predictive text. In this community-based setting, the privacy standards are only relaxed so that client data can be shared with the central server while maintaining privacy from other clients. In addition, the scenario is that the modeling device itself, rather than the user or user habits, adjusts the data required to train an accurate federated model. In addition, the benefits of an accurate federated model are passed on to the client using a community-based architecture, rather than being retained in a central server.

[0071] In some examples, a central server can evaluate different candidate models for a given problem, propagating the best performing model to participating clients. This scenario can take advantage of an automated model federation process.

[0072] Figure 7 An example scenario 700 is shown for training a modified prognostic model based on a modified model proposal by a model trainer 505. A first client 510 previously submitted a model proposal 512, which resulted in the federation and deployment of a federated model 610.

[0073] Once the federated model 610 has been propagated to the second client 520 and the third client 530, those clients can review the model and their corresponding local models and as part of a community driven effort, can propose improvements to the federated model 610. Modified models can be proposed, trained, and federated, which can produce an improved model over the original federated model.

[0074] For example, the second client 520 may submit a modified model suggestion 712 and accompanying relevant data 714 to the model trainer 505 via the central server. The second client 520 may propose a different model for predicting the same event as the first model suggestion 512. The modified model suggestion 712 may be based on the results and observations of the initial model. As an example, the modified model suggestion may indicate that the maximum in-flight temperature is more strongly correlated with spoiler control unit performance degradation than the median temperature.

[0075] The model trainer 505 trains a first modified trained local model 716 based on the modified model suggestion 712 and the data 714, setting parameters for the first modified trained local model 716. The model trainer 505 can then train one or more additional trained local models, each additional trained local model based on the model suggestion and the privately stored data of an additional client, without exposing the privately stored data of the additional client to the first client (indicated by the dotted line). In other words, the model trainer 505 trains each client's local model based on the contributed data of the client's environment. The data 724 produces a second modified trained model 726, and the data 734 produces a third modified trained local model 736. The first client 510, the second client 520, and the third client 530 cannot access each other's data.

[0076] In some examples, during training of the first trained local model, data required to train the modified trained local model may be made accessible, e.g., data 714 is included in data 524. However, in some examples, additional data may need to be accessed in order to train the modified trained local model.

[0077] The model federator 605 is also configured to federate the first modified trained local model 716 and one or more additional modified trained local models (726, 736) to generate a modified trained federated model 740. The model deployer 612 is configured to compare the trained federated model 610 and the modified trained federated model 740 and propagate the higher performing federated model to each of the plurality of clients.

[0078] As an example, recall and precision estimates for the initial joint model and the modified joint model for four airlines are shown in Table 3. In this example, the modified joint model has at least equal precision and at least equal recall relative to the initial joint model.

[0079] Table 3 - Retraining

[0080]

[0081] Figure 8 A flow chart of an example method 800 for generating an integrated model is shown. The method 800 may be performed by a logic subsystem, such as the logic machine 236 of the central server 205.

[0082] At 810, method 800 includes accessing privately stored data of a plurality of clients. For example, each client may selectively store data at a central server, and / or may store data locally and selectively provide access to at least some of the stored data to the central server.

[0083] At 820, method 800 includes receiving a model suggestion from a first client. At 830, method 800 includes training an integrated model based on the model suggestion and privately stored data of the plurality of clients without exposing the privately stored data of the plurality of clients to the first client.

[0084] In some examples, training the integrated model includes training a suggestion model based on a model suggestion from the first client and the privately stored data to generate a first trained local model, such as a reference Figure 4 The training of the integrated model may further include training one or more additional trained local models, each additional trained local model based on the model suggestion and privately stored data of an additional client, without exposing the privately stored data of an additional client to the first client, as described in reference Figure 5 described.

[0085] Training the integrated model may further include combining the first trained local model and one or more additional trained local models to generate a trained combined model, as shown in FIG. Figure 6 as described.

[0086] At 840, method 800 includes propagating the trained ensemble model to one or more of the plurality of clients. In some examples, the trained local models may be propagated to their respective clients.

[0087] In some examples, clients can upload data that is mislabeled, omit data relevant to the ensemble model, or generate new relevant data. Relabeled or redrawn data groups can be automatically identified by the model trainer, which can trigger automatic retraining and / or re-federation of the ensemble model. The model deployer can propagate the updated global model.

[0088] Given a trained local model and a trained federated model, each client can decide to use whichever model has the higher value on the spot. As shown in Table 2, the federated model typically has higher precision and / or recall than its local counterpart. However, a local model for a unique environment may be more accurate, even with fewer data points. For example, if the environmental features cause a strong correlation - e.g., flight duration, atmospheric conditions, load, then the local model may outperform the federated model.

[0089] Large clients with significantly larger datasets than their smaller client counterparts may need to be incentivized to contribute data to a central server. For example, model syndication may be monetized, whereby a client is paid if its model is syndicated and other clients use the model.

[0090] The methods and processes described herein may be bound to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as an executable computer application, a network accessible computing service, an application programming interface (API), a library, or a combination of the above and / or other computing resources.

[0091] Fig. 9 Schematically illustrated is a simplified representation of a computing system 900 that is configured to provide any of the computing functions described herein. The computing system 900 may take the form of one or more personal computers, network accessible server computers, tablet computers, home entertainment computers, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), virtual / augmented / mixed reality computing devices, wearable computing devices, Internet of Things (IoT) devices, embedded computing devices, and / or other computing devices.

[0092] The computing system 900 includes a logic subsystem 910 and a storage subsystem 920. The computing system 900 may optionally include a display subsystem 930, an input subsystem 940, a communication subsystem 950, and / or Fig. 9 Other subsystems not shown.

[0093] The logic subsystem 910 includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions as part of one or more applications, services, or other logical structures. The logic subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processor of the logic subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic subsystem may be optionally distributed in two or more separate devices, which may be remotely located and / or configured for collaborative processing. Various aspects of the logic subsystem may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0094] The storage subsystem 920 includes one or more physical devices configured to temporarily and / or permanently hold computer information such as instructions and data that can be executed by the logic subsystem. When the storage subsystem includes two or more devices, the devices can be collocated and / or remotely located. The storage subsystem 920 can include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices. The storage subsystem 920 can include removable and / or built-in devices. When the logic subsystem executes instructions, the state of the storage subsystem 920 can be converted - for example, to hold different data.

[0095] Aspects of logic subsystem 910 and storage subsystem 920 may be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, program specific and application specific integrated circuits (PASIC / ASIC), program specific and application specific standard products (PSSP / ASSP), systems on chips (SOC), and complex programmable logic devices (CPLD).

[0096] The logic subsystem and the storage subsystem can collaborate to instantiate one or more logic machines. As used herein, the term "machine" is used to collectively refer to a combination of hardware, firmware, software, instructions, and / or any other components that collaborate to provide computer functionality. In other words, a "machine" is never an abstract concept and always has a tangible form. A machine may be instantiated by a single computing device, or a machine may include two or more subcomponents instantiated by two or more different computing devices. In some implementations, a machine includes a local component (e.g., a software application executed by a computer processor) that collaborates with a remote component (e.g., a cloud computing service provided by a network of server computers). The software and / or other instructions that give a particular machine its functionality may optionally be saved as one or more unexecuted modules on one or more suitable storage devices.

[0097] The machine may be implemented using any suitable combination of existing and / or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) techniques. Non-limiting examples of techniques that may be incorporated into one or more implementations of the machine include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural network configured to convolve and pool features in one or more temporal and / or spatial dimensions, recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memory), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases), and / or natural language processing techniques (e.g., tokenization, stemming, composition and / or dependency parsing, and / or intent identification, segmentation models, and / or hyper-segmentation models (e.g., hidden dynamics models)).

[0098] In some examples, the methods and processes described herein can be implemented using one or more differentiable functions, where the gradient of the differentiable function can be calculated and / or estimated with respect to the input and / or output of the differentiable function (e.g., with respect to training data and / or with respect to an objective function). Such methods and processes can be determined at least in part by a set of trainable parameters. Thus, the trainable parameters for a particular method or process can be adjusted by any suitable training process to continuously improve the functionality of the method or process.

[0099] Non-limiting examples of training processes for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot methods, few-shot methods, unsupervised learning methods (e.g., classification based on classes derived from unsupervised clustering methods), enhanced learning (e.g., feedback-based deep Q-learning), and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), contextual banding methods, maximum likelihood methods, and / or expectation maximization. In some examples, multiple methods, processes, and / or components of the systems described herein may be trained simultaneously with respect to an objective function that measures the performance of the collective function of multiple components (e.g., with respect to enhanced feedback and / or with respect to labeled training data). Training multiple methods, processes, and / or components simultaneously may improve such collective function. In some examples, one or more methods, processes, and / or components may be trained independently of other components (e.g., offline training on historical data).

[0100] When display subsystem 930 is included, display subsystem 930 can be used to present a visual representation of the data stored by storage subsystem 920. The visual representation can take the form of a graphical user interface (GUI). Display subsystem 930 can include one or more display devices using almost any type of technology. In some implementations, the display subsystem may include one or more virtual displays, augmented displays, or mixed reality displays.

[0101] When an input subsystem 940 is included, the input subsystem 940 may include or interface with one or more input devices. The input device may include a sensor device or a user input device. Examples of user input devices include a keyboard, a mouse, a touch screen, or a game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral devices, and the conversion and / or processing of input actions may be processed on-board or off-board. Examples of NUI components may include microphones for voice and / or speech recognition; infrared, color, stereo and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers and / or gyroscopes for motion detection and / or intent recognition.

[0102] When included, the communication subsystem 950 can be configured to communicatively couple the computing system 900 with one or more other computing devices. The communication subsystem 950 can include wired and / or wireless communication devices compatible with one or more different communication protocols. The communication subsystem can be configured to communicate via a personal network, a local area network, and / or a wide area network.

[0103] The present disclosure is presented by way of example and with reference to the associated drawings. Parts, processing steps, and other elements that may be substantially the same in one or more of the drawings are coordinately identified and described with minimal repetition. However, it will be noted that the coordinately identified elements may also differ to some extent. It should be further noted that some of the figures may be schematic and not drawn to scale. The various drawing scales, aspect ratios, and numbers of parts shown in the drawings may be purposefully deformed to make it easier to see certain features or relationships.

[0104] It will be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered to have a restrictive meaning, because many variations are feasible. The specific routine or method described herein can represent one or more of any number of processing strategies. As such, the illustrated and / or described different actions can be performed or omitted in the illustrated and / or described order, in other orders, in parallel. Similarly, the order of the above-mentioned processing can be changed.

[0105] Furthermore, the present disclosure includes configurations according to the following clauses.

[0106] Clause 1. A system for generating an integrated model, comprising: a central server having multiple clients, each client privately storing data accessible to the central server; a model trainer configured to: receive a model suggestion from a first client; and train the integrated model based on the model suggestion and the privately stored data of the multiple clients without exposing the privately stored data of the multiple clients to the first client; and a model deployer configured to propagate the trained integrated model to one or more of the multiple clients.

[0107] Clause 2. A system according to clause 1, wherein the model trainer is further configured to: train a suggestion model based on the model suggestion and privately stored data from the first client to generate a first trained local model; and train one or more additional trained local models, each additional trained local model being based on the model suggestion and privately stored data of an additional client, without exposing the privately stored data of an additional client to the first client.

[0108] Clause 3. The system of clause 2, further comprising a model unionizer configured to union the first trained local model and one or more additional trained local models to generate a trained union model.

[0109] Clause 4. A system according to clauses 1 to 3, wherein the model trainer is further configured to: receive a modified model suggestion from a second client; train a modified trained local model based on the modified model suggestion and privately stored data from the second client to generate a first modified trained local model; and train one or more additional modified trained local models, each additional modified trained local model being based on the modified model suggestion and privately stored data of an additional client, such that data from each additional client is not exposed to any other client, and wherein the model unionizer is further configured to union the modified trained local model and the one or more additional modified trained local models to generate a modified trained union model.

[0110] Clause 5. The system of clauses 1 to 4, wherein the model deployer is configured to compare the trained federated model and the modified trained federated model and propagate the higher performing federated model to each of the plurality of clients.

[0111] Clause 6. The system of clauses 1 to 5, wherein the model deployer propagates the model recommendations to at least some of the plurality of clients prior to training each additional trained local model.

[0112] Clause 7. The system of clauses 1 to 6, wherein data is uploaded by one or more additional clients in response to the propagation of the model recommendations.

[0113] Clause 8. The system of clauses 1 to 7, wherein the model deployer propagates the first trained local model to the first client and propagates each additional trained local model to a corresponding additional client.

[0114] Clause 9. A system according to clauses 1 to 8, wherein the proposed model predicts the likelihood of a future event occurring.

[0115] Clause 10. The system of clauses 1 to 9, wherein the privately stored data used to train the additional trained local model includes one or more instances of previous occurrences of the event.

[0116] Clause 11. A system according to clauses 1 to 10, wherein the plurality of customers are airlines that employ a common aircraft type.

[0117] Clause 12. The system of clauses 1 to 11, wherein the future event comprises a maintenance trigger for an aircraft component.

[0118] Clause 13. The system of clauses 1 to 12, wherein the trained integrated model comprises one or more parameters that are different from parameters of the first trained local model.

[0119] Clause 14. A method for generating an integrated model, comprising: accessing private storage data of multiple clients; receiving a model suggestion from a first client; training the integrated model based on the model suggestion and the private storage data of the multiple clients without exposing the private storage data of the multiple clients to the first client; and propagating the trained integrated model to one or more of the multiple clients.

[0120] Clause 15. The method according to clause 14 further includes: training a suggestion model based on the model suggestion and privately stored data from the first client to generate a first trained local model; and training one or more additional trained local models, each additional trained local model being based on the model suggestion and privately stored data of an additional client, without exposing the privately stored data of an additional client to the first client.

[0121] Clause 16. The method of clause 15, further comprising combining the first trained local model and one or more additional trained local models to generate a trained combined model.

[0122] Clause 17. The method according to clauses 14 to 16 also includes receiving a modified model suggestion from a second client; training a modified trained local model based on the modified model suggestion and privately stored data from the second client to generate a modified trained local model; and training one or more additional modified trained local models, each additional modified trained local model based on the modified model suggestion and privately stored data of an additional client, so that data from each additional client is not exposed to any other client; and combining the modified mirror trained local model and one or more additional modified trained local models to generate a modified trained combined model.

[0123] Clause 18. A system for generating an integrated model, comprising: a central server having multiple clients, each client privately storing data accessible to the central server; a model trainer configured to: receive a model suggestion from a first client; train a suggestion model based on the model suggestion and the privately stored data of the first client to generate a first trained local model; and train one or more additional trained local models, each additional trained local model based on the model suggestion and the privately stored data of an additional client without exposing the privately stored data of an additional client to the first client; a model federator configured to federate the first trained local model and the one or more additional trained local models to generate a trained federated model; and a model deployer configured to propagate the trained federated model to one or more of the multiple clients.

[0124] Clause 19. A system according to clause 18, wherein each client operates in a unique environment.

[0125] Clause 20. A system according to clauses 18 to 19, wherein data from each client is made accessible to the central server as part of a subscription model.

Claims

1. A system for generating an integrated model, the system comprising: A central server having a plurality of clients, each client privately storing data accessible by the central server; The model trainer is configured as follows: receiving a model suggestion from a first client; as well as training the integrated model based on the model suggestions and the privately stored data of the plurality of clients without exposing the privately stored data of the plurality of clients to the first client; as well as A model deployer is configured to propagate the trained ensemble model to one or more of the plurality of clients.

2. The system according to claim 1, wherein: The model trainer is also configured to: training a suggestion model based on the model suggestions and privately stored data from the first client to generate a first trained local model; as well as One or more additional trained local models are trained, each additional trained local model being based on the model suggestion and privately stored data of an additional client without exposing the privately stored data of the additional client to the first client.

3. The system according to claim 2, further comprising: A model combiner is configured to combine the first trained local model with the one or more additional trained local models to generate a trained combined model.

4. The system according to claim 3, wherein: The model trainer is also configured to: receiving a modified model suggestion from a second client; training a modified trained local model based on the modified model suggestion and privately stored data from the second client to generate a first modified trained local model; as well as training one or more additional modified trained local models, each additional modified trained local model being based on the modified model proposal and privately stored data of an additional client, such that data from each additional client is not exposed to any other client, and wherein the model federator is further configured to federate the first modified trained local model and the one or more additional modified trained local models to generate a modified trained federated model.

5. The system according to claim 4, wherein: The model deployer is configured to compare the trained federated model and the modified trained federated model, and propagate the high-performance federated model to each of the plurality of clients.

6. The system according to claim 2, wherein: The model deployer propagates the model recommendations to at least some of the plurality of clients prior to training each additional trained local model.

7. The system according to claim 6, wherein: In response to the propagation of the model suggestion, data is uploaded by one or more additional clients.

8. The system according to claim 2, wherein: The model deployer propagates the first trained local model to the first client and propagates each additional trained local model to a corresponding additional client.

9. The system according to claim 2, wherein: The proposed model predicts the likelihood of future events occurring.

10. The system according to claim 9, wherein: The privately stored data used to train the additional trained local model includes one or more instances of previous occurrences of the event.

11. The system according to claim 9, wherein: The multiple clients are airlines that use a common aircraft model.

12. The system according to claim 11, wherein: The future event includes a maintenance trigger for an aircraft component.

13. The system of claim 2, wherein: The trained integrated model includes one or more parameters that are different from parameters of the first trained local model.

14. A method for generating an integrated model, the method comprising: Access private storage data of multiple clients; receiving a model suggestion from a first client; training the integrated model based on the model suggestions and the privately stored data of the plurality of clients without exposing the privately stored data of the plurality of clients to the first client; as well as The trained ensemble model is propagated to one or more of the plurality of clients.

15. The method according to claim 14, further comprising: training a suggestion model based on the model suggestions and privately stored data from the first client to generate a first trained local model; as well as One or more additional trained local models are trained, each additional trained local model being based on the model suggestion and privately stored data of an additional client without exposing the privately stored data of the additional client to the first client.

16. The method according to claim 15, further comprising: The first trained local model is combined with the one or more additional trained local models to generate a trained combined model.

17. The method according to claim 16, further comprising: receiving a modified model suggestion from a second client; training a modified trained local model based on the modified model suggestion and privately stored data from the second client to generate a modified trained local model; as well as training one or more additional modified trained local models, each additional modified trained local model being based on the modified model proposal and privately stored data of an additional client such that data from each additional client is not exposed to any other client; as well as The modified trained local model and the one or more additional modified trained local models are combined to generate a modified trained combined model.

18. A system for generating an integrated model, the system comprising: A central server having a plurality of clients, each client privately storing data accessible by the central server; The model trainer is configured as follows: receiving a model suggestion from a first client; training a suggestion model based on the model suggestion and privately stored data of the first client to generate a first trained local model; as well as training one or more additional trained local models, each additional trained local model based on the model suggestion and privately stored data of an additional client without exposing the privately stored data of the additional client to the first client; a model unionizer configured to union the first trained local model with the one or more additional trained local models to generate a trained union model; as well as A model deployer is configured to propagate the trained federated model to one or more of the plurality of clients.

19. The system of claim 18, wherein: Each client operates in a unique environment.

20. The system of claim 18, wherein: Data from each client is made accessible to the central server as part of a subscription model.

Images