Computer-implemented method and computing device for managing machine learning models

Abstract

The present disclosure provides systems and methods for managing and / or evaluating machine learning models based on locally recorded data. In one example, a user computing device can obtain a machine learning model (e.g., from a server computing device) and can evaluate at least one performance metric for the machine learning model. Specifically, the at least one performance metric for the machine learning model can be evaluated with respect to data locally stored at the user computing device. The user computing device and / or the server computing device can determine whether to activate the machine learning model on the user computing device based at least in part on the at least one performance metric. In another example, a user computing device can evaluate a plurality of machine learning models with respect to locally stored data. At least one of the models can be selected based on the evaluated performance metrics.

Description

[0001] This application is a divisional application of the invention patent application filed on September 25, 2017, with application number 201780071956.2. Technical Field

[0002] This disclosure generally relates to user data and machine learning. More specifically, this disclosure relates to the management and / or evaluation of machine learning models based on locally recorded data. Background Technology

[0003] User computing devices, such as smartphones, tablets, and / or other mobile computing devices, continue to become increasingly: (a) ubiquitous; (b) powerful; (c) endowed with significant local storage; and (d) possess potentially sensitive data about users, their behavior, and their environment. Furthermore, applications delivered on mobile devices are becoming increasingly data-driven. For example, in many scenarios, data collected from user computing devices is used to train and evaluate new machine learning models, personalization features, and computational metrics to assess product quality.

[0004] Traditionally, many of these tasks (e.g., via server computing devices) are performed centrally. Specifically, in some scenarios, data can be uploaded from user computing devices to server computing devices. The server computing devices can then train various machine learning models on the centrally collected data and evaluate the trained models. The trained models can be used by the server computing devices or downloaded to user computing devices for use there. Additionally, in some scenarios, personalized features can be transferred from server computing devices. Similarly, server computing devices can compute cross-user metrics on centrally recorded data for quality assessment.

[0005] However, it is often impossible to know exactly how useful the data will be in the future, and especially which data will be useful. Therefore, without a sufficient history of the recorded data, certain machine learning models or other data-driven applications may not be feasible. In other words, without recording a set or class of data needed to train a model, personalize features, or compute metrics of interest, there is still a significant waiting time before enough data is generated to enable such training, personalization, or computation, even after determining that some data is useful and should be recorded.

[0006] One possible response to this problem is to centrally log all and all data. However, this response has its own drawbacks. Specifically, users engage in a variety of privacy-sensitive activities with their mobile devices. Mobile devices also have increasingly sophisticated sensors, which can lead to devices accessing further privacy-sensitive data streams from their surroundings. Therefore, privacy considerations recommend that logging should be done cautiously—rather than on a large scale—to minimize privacy risks to users.

[0007] Beyond privacy concerns, the data streams these devices can generate are becoming increasingly high-bandwidth. Therefore, in many cases, streaming any and all user data to a centralized database is impractical, even if it is desirable. Summary of the Invention

[0008] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0009] One example aspect of this disclosure relates to a computer implementation method for managing machine learning models, the method comprising: obtaining a plurality of machine learning models from a server computing device by a user computing device, wherein the user computing device is a client computing device; evaluating at least one performance metric for each of the plurality of machine learning models by the user computing device, wherein the at least one performance metric for each machine learning model is evaluated relative to data locally stored and recorded at the user computing device; reporting one or more of the performance metrics evaluated for the plurality of machine learning models to the server computing device by the user computing device; determining, at least in part, a first machine learning model from the plurality of machine learning models by the user computing device based on the performance metrics evaluated for the plurality of machine learning models; using the selected first machine learning model by the user computing device to obtain one or more predictions; and outputting one or more predictions; wherein evaluating at least one performance metric for each of the plurality of machine learning models by the user computing device comprises: using historical data previously recorded at the user computing device and not uploaded for storage on a central server to evaluate at least one performance metric for each of the plurality of machine learning models.

[0010] Evaluating at least one performance metric for each of a plurality of machine learning models using historical data previously recorded at the user computing device and not uploaded for storage in a central service includes: the user computing device inputting at least a first portion of the historical data into each of the plurality of machine learning models; in response to the input of the first portion of the historical data, the user computing device receiving at least one prediction from each of the plurality of machine learning models; and the user computing device comparing the at least one prediction from each machine learning model with a second portion of the historical data to determine at least one performance metric for each of the plurality of machine learning models, respectively.

[0011] Evaluating at least one performance metric for each of a plurality of machine learning models by a user computing device includes: after obtaining a plurality of machine learning models by the user computing device, the user computing device uses newly recorded data at the user computing device to evaluate at least one performance metric for each of the plurality of machine learning models.

[0012] Evaluating at least one performance metric for each of a plurality of machine learning models using newly recorded data at the user computing device includes: the user computing device inputting at least a first portion of the new data into each of the plurality of machine learning models; in response to the input of the first portion of the new data, the user computing device receiving at least one prediction from each of the plurality of machine learning models; and the user computing device comparing the at least one prediction from each machine learning model with a second portion of the new data to determine at least one performance metric for each of the plurality of machine learning models.

[0013] The process of a user computing device determining the selection of a first machine learning model from multiple machine learning models includes: the user computing device selecting the first machine learning model from multiple machine learning models based at least in part on performance metrics evaluated for each of the multiple machine learning models, or the user computing device providing performance metrics evaluated for each of the multiple machine learning models to a server computing device; and the user computing device receiving the selection of the first machine learning model from the server computing device.

[0014] The evaluation of at least one performance metric by the user computing device for each of the multiple machine learning models includes: the evaluation of the average probability of a correct prediction by the user computing device for each of the multiple machine learning models.

[0015] Multiple machine learning models obtained from a user computing device include: multiple machine learning models that have been trained on different types of training data, respectively, obtained from a user computing device.

[0016] The computer-implemented method also includes: aggregating one or more performance metrics received from a user's computing device by a server computing device with other performance metrics reported by other computing devices to assess the global performance for each of the multiple machine learning models.

[0017] Another exemplary aspect of this disclosure relates to a computing device. The computing device includes one or more processors and one or more non-transitory computer-readable media, which jointly store instructions that, when executed by the one or more processors, cause the computing device to perform the following operations: obtaining a machine learning model from a server computing device; evaluating at least one performance metric against the machine learning model, wherein the at least one performance metric against the machine learning model is evaluated relative to data locally stored and recorded at the computing device; reporting one or more of the performance metrics evaluated against a plurality of machine learning models to the server computing device; determining whether to activate the machine learning model based at least in part on the at least one performance metric evaluated against the machine learning model; when it is determined that the machine learning model should be activated, using the machine learning model to obtain one or more predictions; and outputting one or more predictions, wherein the computing device evaluates at least one performance metric against the machine learning model using historical data previously recorded at a user computing device prior to receiving the machine learning model and not uploaded for storage on a central server.

[0018] The computing device uses newly recorded data at the user's computing device after receiving the machine learning model to evaluate at least one performance metric for each of the plurality of machine learning models.

[0019] To determine whether to activate a machine learning model based at least in part on at least one performance metric, the computing device: compares at least one performance metric with at least one threshold; and determines whether at least one performance metric is more favorable than at least one threshold, wherein when at least one performance metric is more favorable than at least one threshold, it is determined that the machine learning model should be activated; or the computing device: provides at least one performance metric evaluated for the machine learning model to a server computing device; and receives from the server computing device a determination of whether to activate the machine learning model.

[0020] The at least one performance metric includes the average probability of a correct prediction.

[0021] Another exemplary aspect of this disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors of a computing system, cause the computing system to perform the operations of the aforementioned computer-implemented method for managing a machine learning model.

[0022] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0023] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the specification, serve to explain the relevant principles. Attached Figure Description

[0024] A detailed discussion of embodiments for those skilled in the art is set forth in the specification with reference to the accompanying drawings, in which:

[0025] Figure 1A A block diagram of an example computing system for managing and evaluating data based on local records, according to an example embodiment of the present disclosure, is depicted.

[0026] Figure 1B A block diagram of an example computing device for managing and evaluating a data execution model based on locally recorded data according to an example embodiment of the present disclosure is depicted.

[0027] Figure 1C A block diagram of an example computing device for managing and evaluating a data execution model based on locally recorded data according to an example embodiment of the present disclosure is depicted.

[0028] Figure 2 A flowchart depicts an example method for managing a machine learning model according to an example embodiment of the present disclosure.

[0029] Figure 3 A flowchart is depicted illustrating an example method for evaluating a machine learning model according to an example embodiment of the present disclosure.

[0030] Figure 4 A flowchart is depicted illustrating an example method for managing multiple machine learning models according to an example embodiment of this disclosure.

[0031] Figure 5 A flowchart is depicted illustrating an example method for managing multiple machine learning models according to an example embodiment of this disclosure. Detailed Implementation

[0032] Typically, this disclosure relates to managing and / or evaluating machine learning models based on locally recorded data. Specifically, compared to traditional centralized data collection, the systems and methods of this disclosure can leverage the availability of large amounts of data already locally recorded at a user computing device. In one example, the user computing device can obtain a machine learning model (e.g., from a server computing device) and can evaluate at least one performance metric of the machine learning model. Specifically, at least one performance metric for the machine learning model can be evaluated relative to data locally stored at the user computing device. The user computing device and / or the server computing device can determine whether to activate a machine learning model on the user computing device based at least in part on at least one performance metric. In another example, the user computing device can obtain multiple machine learning models and can evaluate at least one performance metric for each of the multiple machine learning models relative to data locally stored at the user computing device. The user computing device and / or the server computing device can select at least one machine learning model for use at the user computing device based at least in part on the performance metric. For example, a model with one or more optimal performance metrics can be selected for activation and use at the user computing device.

[0033] Therefore, this disclosure provides improved techniques for model management and evaluation that utilize existing locally recorded user data. For example, one or more models can be evaluated before activation to determine how much value each such model would bring if enabled as a user-visible feature. As a result of this evaluation process, the system of this disclosure can make decisions regarding the use and activation of more informed and user-specific machine learning models. Furthermore, because user data is stored locally on the user's computing device, rather than uploaded for storage on a central server, user privacy is enhanced, and the amount of data that needs to be uploaded is reduced. This reduces bandwidth and power requirements for data processing.

[0034] More specifically, aspects of this disclosure are based on the understanding that the traditional conflict between "centralized recording" and "non-recording" is a flawed dichotomy. Instead, according to aspects of this disclosure, user data can be recorded locally on device storage. In this way, the data coexists with the user, thereby increasing user privacy.

[0035] Specifically, according to one aspect of this disclosure, an application and / or operating system implemented on a user computing device (e.g., a mobile computing device such as a smartphone) may provide a local on-device database to which log entries are written. In some implementations, the database(s) may or may not be user-inspectable. Log entries in the database may or may not expire (e.g., due to time-to-live constraints, storage space constraints, or other constraints). Furthermore, log entries may or may not be user-expungable. Records to the database(s) may logically resemble records to a database stored on a central server, but the records may be placed on a secure local storage device that does not leave the device.

[0036] According to one aspect of this disclosure, when it is desired to train a model, personalize features, or compute metrics, computation can be sent to the data, rather than the other way around. According to this approach, only the minimum amount of information needed to perform a specific action is centrally collected. In some cases, using locally recorded data to provide personalized features and / or perform model training and / or management may result in no central collection of user data at all. Specifically, in some implementations, the application libraries and / or operating systems of the user's computing device can provide support for running analytics against a local database, thus eliminating the need for centralized data uploads for analysis or use.

[0037] As an example, in some implementations, a new machine learning model can be trained by a user computing device based on locally recorded data. Specifically, an untrained model structure and training algorithm can be provided to the user computing device (e.g., from a server computing device). The user computing device can be instructed to use entries from a local database (or a subset thereof) as training data for the algorithm. The newly trained machine intelligence model can then be immediately used to deliver predictions, which can be useful in any number of scenarios. Alternatively, as will be discussed further below, the trained model can be evaluated before activation.

[0038] In some implementations, the machine learning model can be pre-trained, either partially or completely, before being delivered to the user's computing device. For example, the server computing device can train the model partially or completely on centrally collected data before sending it to the user's computing device. In some such implementations, the user computing device can further train the model with reference to a local database of records. In some cases, this additional local training can be viewed as personalizing the model.

[0039] In some implementations, information about the parameters of the trained model can be passed back from the user's computing device to the server's computing device. For example, the server's computing device can use the returned parameter information to build a cross-population model.

[0040] According to another aspect of this disclosure, in some implementations, a user computing device can evaluate at least one performance metric for a machine learning model relative to data stored locally at the user computing device. For example, the performance metric can be used to determine how much value a new model would bring if activated as a user-visible feature. One example performance metric is the average probability of a correct prediction. Another example performance metric is the model's expected utility. For example, expected utility could include a reduction / elimination of the number of user interactions. Statistics regarding prediction performance can be recorded locally for future analysis or centrally recorded.

[0041] As an example, a user computing device can evaluate a machine learning model based on its ability to predict future user behavior. Specifically, the user computing device can use newly recorded local data after receiving and / or training the machine learning model to evaluate at least one performance metric of the model. For example, the model can be used to make predictions based on newly received data. In some implementations, the predictions are not used to provide user characteristics or otherwise influence the performance of the device or application, but rather to be compared with new ground truth data to assess the correctness of the predictions.

[0042] Alternatively or additionally, the user computing device may evaluate the machine learning model by referring to previously recorded local data. Specifically, the user computing device may use historical data previously recorded at the user computing device to evaluate at least one performance metric. Therefore, the machine learning model can be evaluated by referring to previously collected historical data and / or by referring to newly collected data.

[0043] According to another aspect of this disclosure, the user computing device and / or the server computing device can determine whether to activate a machine learning model based at least in part on at least one performance metric. For example, if one or more performance metrics are more favorable than one or more performance thresholds, the model can be activated for use at the user computing device. In various implementations, the user computing device can determine whether to activate the model, or the user computing device can provide one or more performance metrics to the server computing device, and the server computing device can determine whether to activate the model.

[0044] According to another aspect of this disclosure, in some implementations, the user computing device may receive and / or train multiple machine learning models, rather than receiving and / or training a single machine learning model. For example, the server computing device may have already trained multiple machine learning models on different types of training data.

[0045] The user computing device can evaluate at least one performance metric for each of a plurality of machine learning models. Specifically, the user computing device can evaluate at least one performance metric for each machine learning model relative to data locally stored at the user computing device. For example, as mentioned above, the locally stored data can be newly received data or previously recorded data.

[0046] User computing devices and / or server computing devices may select one of the machine learning models, at least in part, based on performance metrics. For example, a model with the best performance metrics (one or more) may be selected for activation and use at the user computing device. As noted, the user computing device may perform the selection, or the user computing device may provide performance metrics to the server computing device, and the server computing device may perform the selection.

[0047] To provide a specific example, the server computing device may have already trained three different machine learning models to predict (e.g., based on previously typed strings) the next word to be typed into a text messaging application. However, these machine learning models may have already been trained on different types of training data. For example, the first model may have been trained on publicly available literature (e.g., books); the second model may have been trained on publicly available social media posts; and the third model may have been trained on publicly available email conversations. The server computing device can then provide these three models to the user computing device.

[0048] Before activating any model, the user computing device can evaluate at least one performance metric for each of the three models relative to locally stored data. For example, the locally stored data may include instances of previous records of text entered by the user into a text messaging application and / or instances of new records of text entered by the user into the text messaging application. As an example, the user computing device can determine, for each model, the average probability of correctly predicting the next word when a given set of instances of text input is present.

[0049] User computing devices and / or server computing devices can select one of three models based on evaluated performance metrics. As an example, if a model trained on a publicly available email session is found to have the highest average probability of correctly predicting the next word, that model can be activated and used on the user computing device. For instance, the selected model could be used to predict the next word that appears within the autocomplete feature of a text messaging application on the user computing device.

[0050] In a further implementation, the selected model can be personalized based on locally stored data before activation. In an even further implementation, if two or more models perform favorably relative to one or more thresholds, these two or more models can be aggregated to form a single model, which is then activated (or personalized and activated). As an example, two or more models performing favorably relative to one or more thresholds can be aggregated based on a weighted average, where the weights applied to each of the two or more models are functions of the corresponding performance metrics evaluated for that model (e.g., the model performing more favorably may receive increased weights).

[0051] According to another aspect of this disclosure, metrics based on typical aggregation queries can also be calculated with reference to locally recorded data. Each user's contribution to the overall metric can be calculated on the user's computing device. In some cases, these per-user aggregates are small enough and de-anonymized that they can be passed to a central server for further cross-user aggregation without exposing sensitive data. In other cases, to enhance user privacy, differential privacy can be used to introduce noise into the per-user aggregates before passing them to the central server for cross-user aggregation.

[0052] In some implementations, per-user aggregations can be securely combined across multiple users using cryptographic techniques before the resulting cross-user aggregations are passed to a central server. In other cases, to further enhance user privacy, differential privacy can be used to add noise to the cryptographically computed cross-user aggregations during the cryptographic aggregation process before the results are passed to the central server.

[0053] According to another aspect of this disclosure, user privacy can be further enhanced by not maintaining a complete stream of certain forms of data on the user's computing device. For example, in some implementations, a summary of locally recorded data, rather than the user data itself, can be stored on the user's computing device. As an example, the summary could be a standard aggregate metric, such as maximum, minimum, average, histogram count, etc. As another example, the summary could be a machine learning model trained from streaming data.

[0054] As yet another example, the digest itself can be stored and updated in a privacy-preserving manner using pan-private techniques. For instance, pan-private techniques can be used to inject noise into a streaming algorithm to achieve differential privacy guarantees regarding the algorithm's internal state and its output.

[0055] According to another aspect of this disclosure, in some implementations, a local database or a digest version of a local database on a user computing device may be stored in encrypted form. In some implementations, the local database or a digest version of a local database may only be directly accessed via privileged software (e.g., a task in an operating system operating under special permissions) or privileged hardware (e.g., trusted computing hardware, such as a trusted platform module).

[0056] Exemplary embodiments of this disclosure will now be discussed in more detail with reference to the accompanying drawings.

[0057] Example devices and systems

[0058] Figure 1A A block diagram of an example computing system 100 performing model management and evaluation based on locally recorded data according to an example embodiment of the present disclosure is depicted. System 100 includes a user computing device 102, a server computing device 130, and a training computing system 150 communicatively coupled via a network 180.

[0059] User computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, a personal assistant computing device, or any other type of computing device.

[0060] User computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118 executed by processor 112 to cause user computing device 102 to perform operations.

[0061] Furthermore, according to aspects of this disclosure, the memory 114 of the user computing device 102 may include locally recorded data 119. Specifically, applications and / or operating systems implemented on the user computing device 102 may provide a local on-device database in the memory 114 to which record entries are written. One database or multiple databases may be used. In some implementations, the database(s) may or may not be user-inspectable. Record entries in the database may or may not expire (e.g., due to time-to-live constraints, storage space constraints, or other constraints). Furthermore, record entries may or may not be user-deletable. Records to the database(s) may logically resemble records to a database stored on a central server, but the records may be placed in a secure local storage device that does not leave the device.

[0062] According to another aspect of this disclosure, the user computing device 102 may store or include one or more machine learning models 120. The machine learning model 120 may be or may additionally include one or more neural networks (e.g., deep neural networks); Markov models (e.g., hidden Markov models); classifiers; regression models; support vector machines; Bayesian networks; multilayer nonlinear models; or other types of machine learning models. The neural network (e.g., deep neural network) may be a feedforward neural network, a convolutional neural network, an autoencoder, a recurrent neural network (e.g., a long short-term memory neural network, a gated recurrent unit, etc.) and / or various other types of neural networks.

[0063] In some implementations, one or more machine learning models 120 may be received from server computing device 130 via network 180, stored in user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 may implement multiple parallel instances of a single machine learning model 120.

[0064] More specifically, one or more machine learning models 120 can be implemented to provide predictions in various situations or applications. As an example, one or more machine learning models 120 may be employed within the context of a text messaging application on user computing device 102, an email application on user computing device 102, a text synthesis interface of an application (e.g., a word processing application), or any other application. As other examples, one or more machine learning models 120 may be included as a browser plugin, as a keyboard overlay or feature (e.g., a virtual keyboard on user computing device 102), or in other contexts. Therefore, one or more machine learning models 120 can receive and analyze data from various applications or other computing device components and can generate predictions based on such data. These predictions can be used for various features.

[0065] Alternatively or concurrently, one or more machine learning models 140 may be included in or otherwise stored and implemented by server computing device 130, which communicates with user computing device 102 according to a client-server relationship. For example, machine learning model 140 may be implemented by server computing device 140 as part of a web service (e.g., a web email service). Thus, one or more models 120 may be stored and implemented at user computing device 102 and / or one or more models 140 may be stored and implemented at server computing device 130.

[0066] User computing device 102 may also include model trainer 122. Model trainer 122 may train or retrain one or more of the machine learning models 120 stored at user computing device 102 using various training or learning techniques, such as, for example, backpropagation of error (e.g., truncated backpropagation over time). Specifically, model trainer 122 may use locally recorded data 119 as training data to train or retrain one or more of the machine learning models 120. Model trainer 122 may perform many generalization techniques (e.g., weight decay, loss of weight, etc.) to improve the generalization ability of the model being trained.

[0067] As an example, in some implementations, a new machine learning model 120 can be trained by model trainer 122 based on locally recorded data 119. Specifically, an untrained model structure and training algorithm can be provided to user computing device 102 (e.g., from server computing device 130). Model trainer 122 can use locally recorded data 119 (or a subset thereof) as training data for training the algorithm. The newly trained machine learning model 120 can then be used immediately to deliver predictions, which can be useful in any number of scenarios. Alternatively, as will be discussed further below, the trained model 120 can be evaluated before activation.

[0068] In some implementations, machine learning model 120 may be pre-trained, either partially or fully, before being delivered to user computing device 102. For example, server computing device 130 may partially or fully train model 140 on centrally collected data 162 (e.g., in cooperation with training computing system 150) before sending model 140 to user computing device 102, and then store it as model 120 in user computing device 102. In some such implementations, user computing device 102 may implement model trainer 122 to further train model 120 with reference to local record database 119. In some cases, this additional local training can be viewed as personalizing model 120.

[0069] In some implementations, the user computing device 102 can pass some information about the parameters of the trained model back to the server computing device 130. For example, the server computing device 130 can use the returned parameter information to build a cross-population model.

[0070] According to another aspect of this disclosure, user computing device 102 may further include model manager 124, which performs management and / or evaluation of one or more of the machine learning models 120. For example, model manager 124 may evaluate at least one performance metric for each machine learning model 120 based on locally recorded data 119. For example, the performance metric may be used to determine how much value model 120 will deliver if activated as a user-visible feature. An example performance metric is the average probability of a correct prediction. Statistics regarding prediction performance may be recorded locally for future analysis or centrally recorded (e.g., at server computing device 130).

[0071] As an example evaluation technique, model manager 124 can evaluate machine learning model 120 based on its ability to predict future user behavior. Specifically, model manager 124 can use newly recorded local data 119, which is received and / or trained after machine learning model 120, to evaluate at least one performance metric for machine learning model 120. For example, model 120 can be used to make predictions based on the newly recorded data 119. In some implementations, the predictions are not used to provide user characteristics or otherwise influence the performance of the device or application, but rather to be compared with new real data to assess the correctness of the predictions.

[0072] Alternatively or additionally, the model manager 124 may evaluate the machine learning model 120 by referring to previously recorded local data 119. Specifically, the model manager 124 may use previously recorded historical data 119 at the user computing device 102 to evaluate at least one performance metric. Thus, the machine learning model 120 can be evaluated by referring to previously collected historical data and / or by referring to newly collected data.

[0073] According to another aspect of this disclosure, model manager 124 can determine whether to activate machine learning model 120 based at least in part on at least one performance metric. For example, if one or more performance metrics are more favorable than one or more performance thresholds, model 120 can be activated for use at user computing device 102. In other implementations, user computing device 102 can provide one or more performance metrics to server computing device 130, and model manager 142 located at server computing device 130 can determine whether to activate model 120 on user computing device 102. Therefore, the decision regarding the activation of a particular model 120 can be performed at either user computing device 102 or server computing device 130.

[0074] According to another aspect of this disclosure, in some implementations, user computing device 102 may receive and / or train multiple machine learning models 120, rather than receiving and / or training a single machine learning model 120. For example, server computing device 130 may have already (e.g., in cooperation with training computing system 150) trained multiple machine learning models 140 on different types of training data, and then send such models 140 to user computing device 102, where they are stored as model 120.

[0075] As described above, model manager 124 can evaluate at least one performance metric for each of the plurality of machine learning models 120. Model manager 124 and / or model manager 142 can select one of the machine learning models 120 based at least in part on the performance metric. For example, the model 120 with the best performance metrics(s) can be selected for activation and use at user computing device 102. As noted, model manager 124 can perform the selection, or user computing device 102 can provide performance metrics to server computing device 130, and model manager 142 can perform the selection.

[0076] In a further implementation, the selected model 120 can be personalized based on locally stored data 119 (e.g., via model trainer 122) before activation. In an even further implementation, if two or more models 120 perform favorably relative to one or more thresholds, the two or more models 120 can be aggregated to form a single model 120, which is then activated (or personalized and activated).

[0077] Server computing device 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138 executed by processor 132 to cause server computing device 130 to perform operations.

[0078] In some implementations, server computing device 130 includes one or more server computing devices or is otherwise implemented by one or more server computing devices. Where server computing device 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0079] As described above, the server computing device 130 may store or additionally include one or more machine learning models 140. The machine learning model 140 may be or may additionally include one or more neural networks (e.g., deep neural networks); Markov models (e.g., hidden Markov models); classifiers; regression models; support vector machines; Bayesian networks; multilayer nonlinear models; or other types of machine learning models. The neural network (e.g., deep neural network) may be a feedforward neural network, a convolutional neural network, an autoencoder, a recurrent neural network (e.g., a long short-term memory neural network, a gated recurrent unit, etc.) and / or various other types of neural networks.

[0080] Server computing device 130 can train machine learning model 140 via interaction with training computing system 150, which is communicatively coupled to network 180. Training computing system 150 may be separate from server computing device 130 or may be part of server computing device 130.

[0081] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0082] The training computing system 150 may include a model trainer 160 that trains a machine learning model 140 stored at a server computing device 130 using various training or learning techniques, such as, for example, backpropagation of error (e.g., truncated backpropagation over time). The model trainer 160 may perform various generalization techniques (e.g., weight decay, loss of weight, etc.) to improve the generalization ability of the model being trained. Specifically, the model trainer 160 may train the machine learning model 140 based on a set of training data 162. The training data 162 may include centrally collected data.

[0083] Each of model trainer 122, model manager 124, model manager 142, and model trainer 160 may include computer logic for providing the desired functionality. Each of model trainer 122, model manager 124, model manager 142, and model trainer 160 may be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, each of model trainer 122, model manager 124, model manager 142, and model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, each of model trainer 122, model manager 124, model manager 142, and model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0084] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Typically, communication over network 180 can be carried via any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0085] Figure 1A An example computing system that can be used to implement this disclosure is shown. Other computing systems may also be used. For example, in some implementations, only the user computing device 102 includes a model trainer and a model manager. For example, in some implementations, the server computing device 130 does not include a manager 142, and the system 100 does not include a training computing system 150. In some implementations, the system 100 includes only the user computing device 102.

[0086] Figure 1B A block diagram of an example computing device 10 for managing and evaluating a data execution model based on locally recorded data according to an example embodiment of the present disclosure is depicted.

[0087] The computing device 10 includes many applications (e.g., applications 1 to N). Each application may contain its own machine learning library and one or more machine learning models. Example applications include text messaging applications, email applications, virtual keyboard applications, browser applications, photo management applications, music streaming applications, ride-sharing applications, or any other type of application.

[0088] Each application 1 to N also includes local application data. Specifically, each application may provide a local on-device database in memory to which record entries are written or communicate with it in other ways. One database may be available for all applications, or different corresponding databases may be available for each application. In some implementations, the database(s) may or may not be user-inspectable. Record entries in the database(s) may or may not expire (e.g., due to time-to-live constraints, storage space constraints, or other constraints). Furthermore, record entries may or may not be user-deletable. Records to the database(s) may logically resemble records to a database stored on a central server, but the records may be placed in secure local storage that does not leave the device.

[0089] User computing device 10 may also include model manager 12, which performs management and / or evaluation of one or more machine learning models included in the application. In some implementations, model manager 12 may be implemented by the operating system of user computing device 10. In other implementations of this disclosure, each application 1 to N may include its own dedicated model manager, rather than a single model manager 12 for the entire device 10.

[0090] Model Manager 12 can evaluate at least one performance metric for each machine learning model based on the corresponding local application data. For example, machine learning model 1 can be evaluated against local application 1 data, and so on. The evaluated performance metrics can be used to determine how much value machine learning model 1 will bring if it is activated as a user-visible feature within application 1. The local application 1 data used for evaluation purposes can be previously recorded data or newly recorded data.

[0091] Model manager 12 can determine whether to activate a machine learning model based at least in part on at least one performance metric. For example, if the performance metric evaluated for machine learning model 1 (e.g., relative to local application 1 data) is more favorable than one or more performance thresholds, machine learning model 1 can be activated for use within application 1.

[0092] Figure 1C A block diagram of an example computing device 50 for managing and evaluating a data execution model based on locally recorded data according to an example embodiment of the present disclosure is depicted.

[0093] The computing device 50 includes many applications (e.g., application 1 to N). Each application communicates with a central intelligence layer. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0094] The central intelligence layer includes many machine learning models. For example, such as... Figure 1C As shown, a corresponding machine learning model can be provided for each application and managed by a central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model for all applications.

[0095] The central intelligence layer may also include a model manager 52, which performs management and / or evaluation of one or more machine learning models included in the application. In some implementations, the model manager 52 may be implemented by the operating system of the user computing device 50.

[0096] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized repository for locally recorded data used by computing device 50. For example... Figure 1C As shown, the central device data layer can communicate with many other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0097] Model Manager 52 can evaluate at least one performance metric for one or more machine learning models included in the central intelligence layer based on data from locally recorded data included in the central device data layer. For example, machine learning model 1 can be evaluated with reference to data from locally recorded data included in the central device data layer. The evaluated performance metric can be used to determine how much value machine learning model 1 will bring if it is activated as a user-visible feature within application 1. The locally recorded data used for evaluation purposes can be previously recorded data or newly recorded data.

[0098] Model manager 52 can determine whether to activate a machine learning model based at least in part on at least one performance metric. For example, if the performance metric evaluated for machine learning model 1 is more favorable than one or more performance thresholds, machine learning model 1 can be activated to provide predictions for use by application 1.

[0099] Figure 1B and Figure 1C The computing devices shown are merely examples. Combinations of various aspects of these devices can also be used to implement this disclosure.

[0100] Example Method

[0101] Figures 2-5 An example method of this disclosure is shown. Although Figures 2-5 The steps performed in a specific order are described for illustrative and discussion purposes, but the method of this disclosure is not limited to the specifically shown order or arrangement. Without departing from the scope of this disclosure, steps may be omitted, rearranged, combined, and / or adapted in various ways. Figures 2-5 The various steps of the method. Furthermore, although... Figures 2-4 The methods are discussed as being primarily performed by user computing devices, but these methods can be performed by other computing devices (e.g., server computing devices) or a combination of computing devices.

[0102] Figure 2 A flowchart is depicted for an example method 200 for managing a machine learning model according to an example embodiment of the present disclosure.

[0103] At point 202, the user computing device obtains the machine learning model. As an example, in some implementations, the user computing device can train a new machine learning model based on locally recorded data. Specifically, an untrained model structure and training algorithm can be provided to the user computing device (e.g., from a server computing device). The user computing device can be instructed to use entries from a local database (or a subset thereof) as training data for training the algorithm.

[0104] As another example, machine learning models can be delivered to a user's computing device pre-trained, either partially or completely. For instance, a server computing device can train the model, either partially or completely, on centrally collected data before sending it to the user's computing device. In some such implementations, the user's computing device can further train the model with reference to a local database of records. In some cases, this additional local training can be viewed as personalizing the model.

[0105] In some implementations, information about the parameters of the trained model can be passed back from the user's computing device to the server's computing device. For example, the server's computing device can use the returned parameter information to build a cross-population model.

[0106] At 204, the user computing device evaluates at least one performance metric against the machine learning model relative to data locally stored on the user computing device. One example performance metric is the average probability of a model correctly predicting a value. Another example performance metric is the model's expected utility. For example, expected utility could include a reduction / elimination of the number of user interactions. As an example, in the context of predicting keyboard input, the use of the model could result in a reduction in the number of user taps or other interactions, which could be reflected in expected utility.

[0107] As an example, at 204, the user computing device can evaluate a machine learning model based on its ability to predict future user behavior. Specifically, the user computing device can use newly recorded local data, newly recorded after the machine learning model has been received and / or trained at 202, to evaluate at least one performance metric against the machine learning model. For example, the model can be used to make predictions based on the newly received data, which can then be compared with other newly received data.

[0108] Alternatively or additionally, at point 204, the user computing device may evaluate the machine learning model by referring to previously recorded local data. Specifically, the user computing device may use historical data previously recorded at the user computing device to evaluate at least one performance metric. Thus, at point 204, the machine learning model may be evaluated by referring to previously collected historical data and / or by referring to newly collected data.

[0109] As an example evaluation technique, Figure 3A flowchart is depicted for an example method 300 for evaluating a machine learning model according to an example embodiment of the present disclosure.

[0110] At point 302, the user computing device obtains the first portion of the actual data stored on the user computing device. The actual data may be previously recorded local data or newly recorded local data.

[0111] Here's an example of how to evaluate a machine learning model based on its ability to accurately predict the next word to be typed into a text messaging application, based on a previously typed string. Therefore, in such an example, the real data could include the first part corresponding to the text string typed by the user.

[0112] At position 304, the user computing device feeds the first portion of the real data into the machine learning model. Continuing the example, at position 304, the user computing device feeds the text string typed by the user into the machine learning model.

[0113] At 306, the user computing device receives a second portion of the real data stored on the user computing device. Continuing this example, the second portion of the real data may include words typed by the user after typing the text string. Therefore, given the first portion, the model can be evaluated regarding its ability to correctly predict the second portion of the real data.

[0114] Box 306 can be completed before or after box 304. Specifically, as noted, the real data can be local data from a previously recorded record or local data from a new record. Thus, when using local data from a previously recorded record, obtaining the real data can simply involve accessing the data from memory. However, in some implementations using local data from a new record, the model can be evaluated in real time. For example, continuing with this example, evaluation can be performed using model predictions as the user types text.

[0115] At 308, the user computing device receives a prediction of the second portion of the real data from the machine learning model. At 310, the user computing device determines the error between the second portion of the real data obtained at 306 and the prediction received at 308. As an example, this error can be binary in nature (e.g., whether the prediction received at 308 matches the second portion of the real data obtained at 306). As another example, the error can be determined based on an error function that evaluates the magnitude of the error between the second portion of the real data obtained at 306 and the prediction received at 308 (non-binary). Any error function can be used.

[0116] At point 312, the user computing device updates the performance metric of the machine learning model based on the error determined at point 310. As an example, the performance metric could be the average of the error over time. Therefore, at point 312, the average can be updated based on the newly determined error.

[0117] After 312, method 300 can terminate or return to box 302 and evaluate the machine learning model again relative to new first and second parts of the real data. Therefore, method 300 can be executed iteratively with many corresponding first and second parts of the real data.

[0118] Refer again Figure 2 At point 206, the user computing device determines whether to activate the machine learning model based at least in part on one performance metric. For example, if one or more performance metrics are favorable to one or more performance thresholds, the model can be activated for use at the user computing device. In some implementations, the user computing device may determine whether to activate the model. In other implementations, the user computing device may provide one or more performance metrics to the server computing device, and the server computing device may determine whether to activate the model.

[0119] At point 208, the user computing device evaluation is determined at point 206. If the machine learning model should be activated, method 200 proceeds to 210. At 210, the user computing device uses the machine learning model to obtain one or more predictions. For example, the predictions made by the machine learning model can be used by one or more applications or other computing device components to provide user characteristics. As an example, the activated model can be used to predict the next word that might appear within the autocomplete feature of a text messaging application on the user computing device.

[0120] However, referring again to box 208, if the machine learning model should not be activated, method 200 proceeds to 212. At 212, the user computing device does not activate the model. Therefore, if the model cannot perform advantageously, it will not be used to provide user-facing features.

[0121] Figure 4 A flowchart is depicted of an example method 400 for managing multiple machine learning models according to an example embodiment of the present disclosure.

[0122] At point 402, the user computing device acquires multiple machine learning models. Specifically, the user computing device may receive and / or train multiple machine learning models, rather than as... Figure 2 Box 202 describes receiving and / or training a single machine learning model. For example, the server computing device may have already trained multiple machine learning models on different types of training data.

[0123] To provide a specific example, the server computing device may have already trained three different machine learning models to predict (e.g., based on previously typed strings) the next word to be typed into a text messaging application. However, the machine learning models may have already been trained on different types of training data. For example, the first model may have already been trained on publicly available literature (e.g., books); the second model may have already been trained on publicly available social media posts; and the third model may have already been trained on publicly available email conversations. The server computing device can then provide these three models to the user computing device.

[0124] At position 404, the user computing device evaluates at least one performance metric for each of a plurality of machine learning models relative to data locally stored on the user computing device. For example, as mentioned above, the locally stored data could be newly received data or previously recorded data. As an example, at position 404, the user computing device could perform the evaluation separately for each of the plurality of machine learning models. Figure 3 Method 300.

[0125] Continuing with the specific example provided above, the user computing device can evaluate at least one performance metric for each of the three models received at 402 relative to locally stored data. For example, the locally stored data may include instances of previous records of text entered by the user into a text messaging application and / or instances of new records of text entered by the user into the text messaging application. As an example, the user computing device can determine, for each model, the average probability of correctly predicting the next word when a given set of instances of text input is present.

[0126] At 406, the user computing device determines the selection of a first machine learning model from multiple machine learning models based at least in part on the performance metrics evaluated at 404. In some implementations, determining the selection at 406 may include performing the selection locally at the user computing device. In other implementations, determining the selection at 406 may include sending the performance metrics to a server computing device and then receiving the selection from the server computing device. As an example, the model with the best performance metrics(s) may be selected at 406.

[0127] Continuing with the specific example provided above, the user computing device and / or server computing device can select one of three models based on evaluated performance metrics. As an example, if a model trained on a publicly available email session is found to have the highest average probability of correctly predicting the next word, that model can be activated and used on the user computing device. For instance, the selected model could be used to predict the next word that might appear within the autocomplete feature of a text messaging application on the user computing device.

[0128] At 408, the user computing device uses the selected first machine learning model to obtain one or more predictions. Continuing with the specific example provided above, the selected model can be used to predict the next word that might appear within the autocomplete feature of a text messaging application on the user computing device. Many different types of models can provide many different types of predictions, which can be used for many different purposes, including both user-oriented and non-user-oriented features.

[0129] In a further implementation, the locally stored data can be used before being used at 408, and the model selected at 406 can be personalized. In an even further implementation, if two or more models perform favorably with respect to one or more thresholds, the two or more models can be aggregated to form a single model, which is then activated (or personalized and activated). In one example, two or more models performing favorably with respect to one or more thresholds can be aggregated based on a weighted average, where the weights applied to each of the two or more models are functions of the corresponding performance metrics evaluated for such models.

[0130] Figure 5 A flowchart is depicted for an example method 500 for managing multiple machine learning models according to an example embodiment of the present disclosure.

[0131] At position 502, the server computing device obtains multiple performance metrics, each associated with a different machine learning model. The performance metric for each model indicates its performance when evaluated against a set of data stored locally on the user's computing device.

[0132] At position 504, the server computing device selects at least one of multiple machine learning models based at least in part on multiple performance metrics. As an example, at position 504, the model with the best performance metrics(s) could be selected.

[0133] At point 506, the server computing device causes at least one selected machine learning model to be used at the user computing device. For example, the server computing device can instruct the user computing device to activate and use at least one selected machine learning model.

[0134] Additional Public Content

[0135] The technologies discussed here refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from these systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions between and within components. For example, the processes discussed here can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0136] While the subject matter has been described in detail with respect to various specific exemplary embodiments, each example is provided by way of explanation rather than limitation. Upon gaining an understanding of the foregoing, those skilled in the art will readily generate modifications, variations, and equivalents to these embodiments. Therefore, this disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, as will be apparent to those skilled in the art. For example, features shown or described as part of one embodiment may be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. A computer-implemented method for managing machine learning models, the method comprising: Multiple machine learning models are obtained by the user's computing device from one or more server computing devices; The user computing device evaluates at least one performance metric for each of the plurality of machine learning models, wherein the at least one performance metric for each machine learning model is evaluated relative to data stored locally at the user computing device. The user computing device provides performance metrics, which are evaluated for the multiple machine learning models, to one or more server computing devices respectively. The user computing device receives, at least in part, a selection of a first machine learning model from one or more server computing devices based on performance metrics evaluated for each of the plurality of machine learning models; and The user's computing device uses a selected first machine learning model to obtain one or more predictions.

2. The computer-implemented method according to claim 1, wherein, Evaluating at least one performance metric for each of the plurality of machine learning models by the user computing device includes evaluating the at least one performance metric for each of the plurality of machine learning models by the user computing device using historical data previously recorded at the user computing device.

3. The computer-implemented method according to claim 2, wherein, Evaluating the at least one performance metric for each of the plurality of machine learning models using previously recorded historical data at the user computing device includes: The user's computing device inputs at least a first portion of the historical data into each of the plurality of machine learning models; In response to the input of a first portion of historical data, the user computing device receives at least one prediction from each of the plurality of machine learning models; and The user computing device compares the at least one prediction from each machine learning model with a second portion of historical data to determine the at least one performance metric for each of the plurality of machine learning models.

4. The computer-implemented method according to any of the preceding claims, wherein, The evaluation of at least one performance metric for each of the plurality of machine learning models by the user computing device includes, after the user computing device obtains the plurality of machine learning models, the user computing device using newly recorded data at the user computing device to evaluate the at least one performance metric for each of the plurality of machine learning models.

5. The computer-implemented method according to claim 4, wherein, Evaluating at least one performance metric for each of the plurality of machine learning models using newly recorded data at the user computing device includes: The user computing device inputs at least a first portion of the new data into each of the plurality of machine learning models; In response to the input of the first portion of new data, the user computing device receives at least one prediction from each of the plurality of machine learning models; and The user computing device compares the at least one prediction from each machine learning model with a second portion of new data to determine the at least one performance metric for each of the plurality of machine learning models.

6. The computer-implemented method according to claim 1, wherein, The selection of a first machine learning model from the plurality of machine learning models is based at least in part on the aggregated global performance from performance metrics reported by other computing devices.

7. The computer-implemented method according to claim 1, wherein, The user computing device evaluates at least one performance metric for each of the plurality of machine learning models, including the average probability of a correct prediction evaluated by the user computing device for each of the plurality of machine learning models.

8. The computer-implemented method according to claim 1, wherein, The multiple machine learning models are evaluated before any machine learning model is activated. and After the user computing device receives the selection of the first machine learning model, the first machine learning model is activated and used on the user computing device.

9. The computer-implemented method according to claim 1, further comprising: The user computing device reports one or more performance metrics evaluated for the plurality of machine learning models to one or more server computing devices.

10. The computer-implemented method according to claim 9, further comprising: One or more server computing devices aggregate the one or more performance metrics received from the user computing device with other performance metrics reported by other computing devices to evaluate the global performance of each of the plurality of machine learning models.

11. A computing device, the computing device comprising: One or more processors; and One or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause a computing device to perform the method according to any of the preceding claims.

12. A computer program product comprising computer-readable instructions that, when executed by one or more processors of a server computing system, cause the server computing system to perform the method according to any one of claims 1 to 10.

Images