Application development platforms and software development kits that provide comprehensive machine learning services

Abstract

This disclosure provides an application development platform and an associated software development kit (SDK) that offer comprehensive services for generating, deploying, and managing machine learning models used by computer applications, such as mobile applications executed by mobile computing devices. Specifically, the application development platform and SDK can provide, or otherwise fully utilize, a unified cross-platform application programming interface (API), which enables access to all the different machine learning services required for the full machine learning functionality within the application. In this way, developers can access a single SDK for all machine learning services.

Description

[0001] Priority requirements

[0002] This application claims priority and benefit to U.S. Provisional Application 62 / 667,959, filed May 7, 2018, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to systems for developing and managing computer applications. More specifically, this disclosure relates to an application development platform and an associated software development kit (SDK) that provides comprehensive services for generating, deploying, and managing machine learning models used by computer applications, such as mobile applications executed by mobile computing devices. Summary of the Invention

[0004] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0005] One example aspect of this disclosure relates to a mobile computing device. The mobile computing device includes one or more processors and one or more non-transitory computer-readable media, which jointly store: a computer application; and a machine intelligence software development kit (MIK). The MKI is configured to store one or more machine learning models and machine learning libraries. The MKI is configured to communicate with the computer application using an application programming interface (API) to receive input data from the computer application. The MKI is configured to implement one or more machine learning models and machine learning libraries on the device to generate inferences at least in part based on the input data. The MKI is configured to communicate with the computer application using the API to provide inferences to the computer application.

[0006] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0007] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. Exemplary embodiments of this disclosure are illustrated in conjunction with the accompanying drawings, which are incorporated in and form part of this specification, and serve to explain the relevant principles in conjunction with the description. Background Technology

[0008] A computer application generally refers to a structured collection of computer-readable instructions that, when executed, cause a computing device to perform certain tasks. Computer applications can be stored in the memory of a computing device and executed by one or more processors of the computing device.

[0009] Example computer applications include mobile applications and web applications. Mobile applications are generally computer applications specifically designed to be executed primarily by mobile computing devices, such as smartphones, tablets, watches, etc. Web applications are generally computer applications that are partially executed by a web server and interface with computing devices (e.g., mobile computing devices) via the Internet. For example, a user can interact with a web application interface via a mobile web browser executed by a mobile computing device.

[0010] Developers of computer applications (e.g., mobile applications and / or web applications) often rely on the use of software development kits (SDKs) to develop their applications. An SDK can include a collection of software development tools that allow the creation of applications for certain software packages, software frameworks, hardware platforms, computer systems, operating systems, and / or other considerations.

[0011] SDKs often enable the implementation of one or more application programming interfaces (APIs) as on-device libraries, allowing applications to interface with other computing applications or systems that provide various services to that application. Furthermore, as recent advancements in machine learning become increasingly integrated with and widely used across various applications, machine learning services offer application developers another aspect to consider and implement.

[0012] However, tools and services used in mobile applications (such as machine learning services) are highly fragmented. For example, analysis shows that a typical mobile application uses an average of about 16 different SDKs. This fragmented nature of services is costly at runtime because the application / device must load and execute many different libraries, consuming significant processing and memory resources. Furthermore, it requires application developers to digest different documentation, download and integrate each SDK separately, call different methods of each SDK, learn the corresponding console for each SDK, and perform other tasks, greatly complicating the development process.

[0013] Furthermore, countless applications today utilize deep neural networks and other complex machine learning models. However, their accuracy is partly due to the large number of parameters that result in high computational overhead. It would be beneficial if these models could be run (and their accuracy benefits be realized) on mobile and other resource-constrained devices, such as smart devices.

[0014] While running these models on mobile devices would be extremely useful for a variety of reasons (e.g., faster user-application interaction, no need for an internet connection, no need for data to be sent to the cloud), many challenges remain. In particular, complex models often require significant network bandwidth for transmission, storage resources for storage, and processing resources for execution. Consequently, in some scenarios, complex models may be too large or resource-intensive to run on resource-constrained devices. Attached Figure Description

[0015] The specification provides a detailed discussion of the embodiments for those skilled in the art, with reference to the accompanying drawings, wherein:

[0016] Figure 1 A block diagram of an example computing system according to an example embodiment of the present disclosure is depicted.

[0017] Figure 2 A stack diagram of an example computing system according to an example embodiment of the present disclosure is depicted.

[0018] Figure 3 A functional diagram of an example console according to an example embodiment of the present disclosure is depicted.

[0019] Figure 4 A workflow diagram of an example computing system according to an example embodiment of the present disclosure is depicted.

[0020] Figure 5 A workflow diagram of an example computing system according to an example embodiment of the present disclosure is depicted.

[0021] Figure 6 A flowchart depicts an example process for generating and deploying a machine learning model according to an example embodiment of the present disclosure.

[0022] Figure 7 A workflow diagram depicts an example process for generating and deploying a machine learning model according to an example embodiment of the present disclosure.

[0023] Figure 8 A swimlane diagram depicts an example process for uploading a machine learning model for distribution according to an example embodiment of this disclosure.

[0024] Figure 9 A swimlane diagram depicts an example process for downloading a machine learning model according to an example embodiment of the present disclosure.

[0025] Figure 10 A block diagram of an example computing system including a fat SDK according to an example embodiment of the present disclosure is depicted.

[0026] Figure 11A block diagram of an example computing system including a thin SDK according to an example embodiment of the present disclosure is depicted.

[0027] Figure 12 A block diagram of an example computing system, including a mobile vision model as part of an updatable first-party supporting application, is depicted according to an example embodiment of the present disclosure.

[0028] Figure 13 A block diagram of an example computing system, including mobile vision, text, and speech models as part of an updatable first-party supporting application, is depicted according to an example embodiment of the present disclosure.

[0029] Figure 14 A block diagram of an example computing system comprising an updatable first-party support application for executing a runtime and training both, according to an example embodiment of the present disclosure, is depicted.

[0030] Figure 15 A workflow diagram of an example training pipeline according to an example embodiment of the present disclosure is depicted.

[0031] Figure 16A A graphical representation of an example joint training scheme according to an example embodiment of the present disclosure is provided.

[0032] Figure 16B A graphical representation of an example joint training scheme for training multiple student models, based on an example aspect of this disclosure, is depicted.

[0033] Figure 17 An example user interface according to an example embodiment of this disclosure is depicted.

[0034] Figure 18 An example user interface according to an example embodiment of this disclosure is depicted. Detailed Implementation

[0035] Overview

[0036] Generally, this disclosure relates to an application development platform and an associated software development kit (SDK) that provides comprehensive services for generating, deploying, and managing machine learning models used by computer applications, such as mobile applications executed by mobile computing devices. Specifically, the application development platform and SDK can provide, or otherwise fully utilize, a unified cross-platform application programming interface (API), enabling access to all the different machine learning services required for all machine learning functionalities within the application. In this way, developers can access a single SDK for all machine learning services. Therefore, developers will have a single documentation set, a common way to obtain machine learning products, a single console to access, and a single initialization call to serve the different machine learning needs of all applications.

[0037] According to one aspect of this disclosure, an application development platform enables developers to generate custom models for their applications. In one example, developers can upload or otherwise make access to training data, and then use the application development platform to create and train machine learning models for use with their applications. In another example, developers can upload pre-trained models to the platform. In yet another example, developers can select a pre-trained model from a set of available pre-trained models made available to users of the platform. Developers can use the pre-trained model as is, or they can retrain the model on additional training data.

[0038] Additionally, after the model has been trained, or in some implementations, as part of the training process itself, the application development platform can enable developers to compress and / or transform the model to optimize it for use on resource-constrained devices (e.g., mobile or embedded devices) or in resource-constrained environments. For example, model compression can include performing quantization (e.g., scalar quantization, vector quantization / weight sharing, product quantization, etc.), pruning (e.g., value pruning, L1 regularization, etc.), low-rank representations (e.g., circulate matrix, Kronecker structure, SVD decomposition, etc.), refining, and / or other compression techniques, including novel learning techniques described further herein for directly learning memory-efficient and computationally efficient (e.g., small and fast) machine learning models that can be directly used for inference on edge devices such as mobile phones, wearable devices, and IoT devices. Therefore, various compression tools can be optionally accessed and used to compress learned or uploaded models.

[0039] More specifically, aspects of this disclosure relate to an end-to-end compressed learning framework that allows for the compression of an input model (e.g., a large pre-trained model) through a learning process, during which a compact model is learned based on the input model. The compact model can be an off-the-shelf, on-device model that is smaller, more memory-efficient, more power-efficient, faster inference, and suffers minimal loss of accuracy. Additional state-of-the-art techniques for compressing machine learning models can also be combined with the compressed learning framework.

[0040] Model conversion can include transforming a model from a standard version to a mobile-optimized version, compatible with lightweight machine learning libraries specifically designed for mobile and embedded devices. In one example, the platform can use a conversion tool called the TensorFlow Lite Optimizing Converter (TOCO) to convert a model's standard TensorFlow graph to a TensorFlow Lite graph, where TensorFlow Lite is a lightweight machine learning library designed specifically for mobile applications. Therefore, various conversion tools can be optionally accessed and used to convert learned or uploaded models into mobile-optimized versions.

[0041] Compression / conversion services also offer a variety of tools and dashboards to enable developers to explore and control compression options. For example, tools may display trade-offs with quality, enable parameter tuning, and / or other controls that allow for fine-grained control over the compression results. For instance, developers can use the compression service to generate multiple models with different sizes and corresponding trade-offs. These multiple models can be used as part of a model target determination scheme, as described further below.

[0042] According to another aspect of this disclosure, as mentioned above, this disclosure provides an end-to-end framework and cloud-based solution, as well as an API, for learning memory-efficient and computationally efficient (i.e., small and fast) machine learning models that can be directly used for inference on resource-constrained devices. In particular, in some embodiments, the end-to-end learning framework may be provided as part of or a feature of the application development platform described herein. Therefore, the application development platform can employ the end-to-end framework to train compact machine learning models that are useful for different problem domains where more complex machine learning models are typically used.

[0043] When an end-to-end learning framework is combined with or provided as a feature of an application development platform, the result is an end-to-end cloud-packaged solution that enables developers to easily train their own high-quality, custom on-device models. This allows developers to directly learn models optimized for size and quality using advanced machine learning techniques, starting from raw training data or checkpoints of their pre-trained models, if available. However, end-to-end learning frameworks can also be used outside the context of an application development platform, or they can be used separately from the application development platform. Therefore, while this disclosure focuses on the use of end-to-end frameworks within the context of an application development platform, end-to-end frameworks are not limited to such a context.

[0044] Therefore, the end-to-end framework enables the direct learning of compact custom models, unlike (but optionally complementing) other techniques aimed at compressing post-trained model weights or performing model format transformations (e.g., TOCO) (e.g., quantization). The trained compact model can be used to perform tasks on computing devices with limited computational power and memory, addressing diverse problem domains that demand high computational power and memory for complex models. In this way, a trained compact machine learning model can perform prediction tasks on computing devices with limited computational power and memory.

[0045] The end-to-end framework can: learn from a variety of input sources; cover a wide range of prediction tasks; support multiple use cases; be robust in terms of learnable machine learning models and / or supported architectures; be compatible with infrastructure requirements such as TensorFlow Lite integration, fast training (e.g., by using distributed technologies), access for internal and / or external users (e.g., developers), and platform integration for the SDK; and be easy to use.

[0046] In some implementations, compact machine learning models describe machine learning models that use only relatively limited computational power and limited memory, such as memory-efficient and computationally efficient machine learning models, or any other suitable machine learning model with small size and fast computation time. Complex machine learning models describe machine learning models that use relatively higher computational power and more memory to predict complex tasks, such as feedforward neural networks, long short-term memory recurrent neural networks, or other machine learning models with high computational costs. In some specific scenarios, complex machine learning models may be too large or too inefficient to run effectively on resource-constrained computing devices.

[0047] According to one aspect of this disclosure, the application development platform can be flexible and enable learning from a variety of inputs to generate models for prediction tasks with a wide range of predictions. That is, the end-to-end framework provided by the platform can receive various inputs from developers and can train compact models and / or train compact models that satisfy those inputs based on these inputs.

[0048] In some implementations, the input for training a compact machine learning model can be data, one or more input functions, and / or one or more parameters defining the compact machine learning model for training. Examples of inputs may include: labeled data for a supervised compact machine learning model, unlabeled data for a compact semi-supervised machine learning model, a pre-trained machine learning model (which can be used as a trainer model), desired inference speed, and / or desired model size.

[0049] One example input source is labeled data. For instance, the platform can train a compact model directly from raw data provided in the typical format of machine learning libraries (e.g., tensorflow.Example). The platform can also process unlabeled data in a semi-supervised setting. Another example input source is a pre-trained model. For instance, the size of a pre-trained machine learning model can be larger than the size of the compact machine learning model to be learned. In one example, the pre-trained model could be a checkpoint from a production model deployed for cloud-based inference.

[0050] Inference speed can be a flexible parameter in the input space (e.g., it can be defined by the developer). Inference speed describes the computational efficiency of a compact machine learning model running on a wide range of computing devices (e.g., mobile devices, wearable devices, embedded devices, etc.). Inference speed can be used to guide the architecture selection of compact machine learning models. Inference speed can depend on the computing resources available on a specific device.

[0051] Model size can be a flexible parameter in the input space (e.g., it can be defined by the developer). Model size describes the size of the compact machine learning model after the training process is complete. In some implementations, the application development platform can jointly train multiple compact machine learning models of different sizes (e.g., small: <1-2 million (M), medium: 5-10M, large: >10M) and make the trained compact machine learning models available to developers / users so that users can select a model of a custom size for a specific use case. In some implementations, the application development platform can train compact machine learning models with distinguishable architectures (e.g., this comes from a series of efficient architectural frameworks or a collection of efficient software operations (ops) such as projection, depth-first convolution, etc.) rather than pre-specifying model specifications.

[0052] In some implementations, the input may also include an indication of the type or class of the input. Example types of input include text, images, video, multimodal (text combined with images), sensor signals from a computing device (e.g., a device on which a model will be deployed), or some combination thereof.

[0053] In some implementations, the input may also include an instruction on the prediction task to be performed by the trained model. Example prediction tasks performed by a compact machine learning model (and provided as input) may include classification (e.g., the number of classes may range from small (e.g., binary or 10-100) to large output spaces (e.g., 10k or 1M classes)); regression; or structured prediction (e.g., sequences).

[0054] According to another aspect of this disclosure, an application development platform can improve the predictive accuracy of a compact machine learning model by jointly training a compact machine learning model and a trainer model (which in some cases may be a pre-trained machine learning model). The pre-trained machine learning model may be a machine learning model previously trained by the application development platform or by an external system. The size of the pre-trained machine learning model may be larger than that of the compact machine learning model (such as checkpoints from production models deployed in the cloud). In some implementations, the pre-trained machine learning model may be a complex model that requires higher computational power and more memory than the compact machine learning model.

[0055] Joint training enables compact machine learning models to learn from (and / or from) trainer models, thereby improving the prediction accuracy of compact machine learning models. Therefore, joint training can follow a teacher-student joint training architecture.

[0056] Therefore, in some implementations, the application development platform may include and implement a training pipeline to train a compact machine learning model. The training pipeline may train the compact machine learning model independently and / or jointly with a trainer model (e.g., a pre-trained model). Thus, the trainer or teacher model may be fixed or may be jointly optimized with the student model.

[0057] The trainer or teacher model can be any type of model, including, for example, feedforward neural networks, recurrent neural networks (e.g., long short-term memory networks), quasi-RNNs, convolutional neural networks, ProjectionNet (e.g., dense and sparse versions), BiLSTM (bidirectional LSTM), depth-separable ConvNet, MobileNet, ProjectionCNN, NASNet, Inception (e.g., Inception v3), ResNet, and / or other types of machine learning models. The compact or student model can be any type of model, but its weights are typically lighter than the trainer model. Example student models include feedforward neural networks, recurrent neural networks (e.g., long short-term memory networks), quasi-RNNs, convolutional neural networks, ProjectionNet (e.g., dense and sparse versions), BiLSTM (bidirectional LSTM), depth-separable ConvNet, MobileNet, ProjectionCNN, NASNet, Inception (e.g., Inception v3), ResNet, and / or other types of machine learning models.

[0058] In some implementations, the training pipeline may receive one or more inputs from a user. For example, the training pipeline may receive training data along with corresponding input functions for training and for evaluation. The training pipeline may create a schema specifying how one or more training methods for a compact machine learning model will be performed. The training pipeline may also provide an experiment-based API (e.g., tf.Experiment) for building networks. The training pipeline may invoke training (e.g., the wrapper code and / or training below to begin training).

[0059] A training pipeline can train a compact model until a desired number of steps are reached. The training pipeline can export the trained compact machine learning model in a specific format (e.g., TF-Lite format). In some implementations, the trained compact machine learning model can then be used to run on a computing device (e.g., on a mobile device).

[0060] In some implementations, the created schema may include several fields, such as experiment name, features (e.g., field name, feature type, one or more dimensions of the feature, etc.), hyperparameters (e.g., learning rate, number of steps, optimizer, activation layer, loss weights for the pre-trained model, loss weights for the compact model, cross loss weights, etc.), a model specification for a compact model that includes multiple fields to build a compact model, and a model specification for a pre-trained model that includes multiple fields to build a pre-trained model.

[0061] In some implementations, the training pipeline can remove or add fields to the model specification based on ongoing development and / or new use cases. In other implementations, the training pipeline can validate the schema to ensure that the training pipeline behaves as expected. Each field in the schema is validated before the pipeline starts. Fields in the model specification can also be validated using the validation methods of the corresponding model class.

[0062] In some implementations, a training pipeline can jointly train a compact machine learning model with a pre-trained machine learning model. For example, a training pipeline can receive a pre-trained machine learning model, a compact machine learning model, training data, and / or a corresponding input function. The training pipeline can create patterns specifying how training will proceed. The training pipeline can provide an experiment-based API to specify joint training. The training pipeline can invoke joint training.

[0063] The training pipeline can jointly train a compact machine learning model and a pre-trained machine learning model until the joint training loss function indicates that the difference between the output of the compact machine learning model and the expected output is less than a threshold. The training pipeline can extract and export the trained compact machine learning model in a specific format.

[0064] In some implementations, the training pipeline can implement and / or store one or more common machine learning models (e.g., feedforward networks, projective networks, quasi-recurrent neural networks, convolutional neural networks, long short-term memory networks, etc.) for joint training. Accordingly, instead of inputting pre-trained machine learning models or custom compact model specifications, users can choose which models to store and / or implement in the training pipeline, and can input these choices into the training pipeline. In response to the user's selection, the training pipeline can create a pattern specifying how training should be performed with the selected model(s).

[0065] In some implementations, the training pipeline may include one or more debugging metrics, such as various losses, accuracies, confusion matrices for the pre-trained machine learning model, and the compact machine learning model. In some implementations, these added metrics are separate from other metrics (e.g., metrics integrated with Tensorboard). In some implementations, the training pipeline may include example implementations for using wrappers for the tf.Estimator API and other plugins. In some implementations, the training pipeline may include integration with TOCO to export the trained compact machine learning model to TF-Lite format.

[0066] Example uses of compact models generated via the end-to-end framework include: testing applications (e.g., MNIST, CIFAR10 / CIFAR100), smart replies, handwriting recognition, wearable health (e.g., heart rate prediction), wearable notifications (e.g., content-based classification), text classification / ranking, emotion detection, sensitive content detection, gesture recognition, image classification, and / or multimodal learning (e.g., photo replies).

[0067] According to another aspect of this disclosure, the application development platform can enable and perform machine learning model management. For example, after a model has been trained, compressed, and / or transformed, developers can use the platform to store the model in a cloud storage database. Based on the cloud storage database, developers can use the platform to download the model to devices on which their applications are already stored. The platform can also enable developers to perform various other management functions, including, for example, version control, compatibility, A / B testing, download management, etc.

[0068] Therefore, the platform can provide and execute a complete, end-to-end model generation workflow, which can include the platform: receiving uploaded models or learning new models based on uploaded training data; automatically compressing models using benchmarks; converting models into mobile-optimized formats; and hosting models for download and use on devices.

[0069] According to another aspect of this disclosure, in some embodiments, a custom third-party model can be included in a machine intelligence SDK that forms part of an application and communicates with the application using platform APIs. For example, the machine intelligence SDK can be included in an application package downloaded from an app store. Therefore, the custom model can be downloaded to the device and run on the device to provide inferences to the application.

[0070] Therefore, in some implementations, the application development platform can support so-called "fat" SDKs, where services run during application execution. This architecture can provide security and privacy benefits for third-party data and models. Alternatively, as will be further described below, the platform can allow the exposure of existing on-device models not included in the application via APIs.

[0071] Various aspects of this disclosure also implement out-of-band model updates, where an updated version of the model can be downloaded to the machine intelligence SDK on the device without requiring a complete application reinstallation. In other embodiments, the model can be downloaded at runtime and inserted into the machine intelligence SDK. Separating the model download from the download of the entire application or, more generally, the application update download can help reduce the size of the application package download or application update download, thereby reducing latency and application downtime.

[0072] On the other hand, an application development platform can enable applications to fully utilize first-party models provided as a feature of the platform. In some implementations, as described above, the first-party model can be included in the Machine Intelligence SDK. Alternatively or additionally, the first-party model can be included in a first-party support application separate from the developer's application. In other implementations, first and / or third-party models can be maintained in the cloud, and inferences can be obtained from the models over the network via APIs.

[0073] Therefore, application development platforms can enable applications to access (e.g., via API calls for inference) first-party models (e.g., generic models provided / maintained by the platform operator) and custom third-party models (e.g., application-specific models associated with a particular application developer). Specifically, these models can be included in and dynamically updated within the machine intelligence SDK included in the application, enabling on-device inference. For example, first-party models can be accessed using a basic API, while third-party models can be accessed via a custom API.

[0074] More specifically, as used herein, the term "first-party" refers to system components (e.g., machine learning models) that are generally generated, maintained, and / or controlled by the same entity operating a large platform. Therefore, as a feature of application development platforms, entities may also provide access to one or more first-party models. These first-party models can be general-purpose machine learning models that deliver high-quality performance on commonly required tasks, such as speech analysis (e.g., natural language processing, speech recognition, etc.), text analysis, image analysis (e.g., object detection, barcode / QR code reading, optical character recognition, and / or other tasks that can be categorized as "mobile vision," etc.). Another example of a first-party machine learning model could be an intelligent response model (e.g., accessed via an API) that predicts possible text responses a user might expect in response to an input set of text.

[0075] Conversely, the term "third party" refers to system components (e.g., machine learning models) generated by entities other than those operating the large platform. In one example, a third-party model could be generated and / or controlled by application developers. For instance, application developers could use various aspects of the platform to generate and deploy their own custom machine learning models. In another example, a third-party model could be generated and / or controlled by an additional entity besides the application developer. For example, application developers could use an application development platform to receive models from such an additional entity.

[0076] According to another aspect of this disclosure, the Machine Intelligence SDK may also include a dedicated machine learning library, which can be implemented by an application to run and / or train models included in the Machine Intelligence SDK on the device. In some implementations, this machine learning library may be a lightweight library designed for mobile applications (e.g., TensorFlow Lite). Alternatively or additionally, copies of the same or different machine learning libraries may also be included in a first-party supporting application. Models included in the first-party supporting application can be run and / or trained on the device by the first-party supporting application using such a copy of the machine learning library.

[0077] In some implementations, the machine intelligence SDK included in the developer's application and / or first-party applications can perform on-device data logging. This on-device logging can support on-device inference and / or training. Regarding on-device training, in some implementations, the machine intelligence SDK can perform validation of the on-device training data as part of the logging process. For example, the SDK can detect anomalies and prevent anomaly logging into the training data. On-device training can enable the personalization of models based on user-specific data. In a further implementation, on-device training can enable the device to participate in federated learning schemes.

[0078] Therefore, this disclosure addresses device-wide model training management (e.g., scheduling, storage, etc.). In some implementations, batch training can be performed according to scheduling rules. Scheduling rules can be default and / or customized according to configurations specified by the developer. As an example, training can occur once a day, at night, and when the device is not in active use, is actively charging, and is connected to a network.

[0079] Logging can also enable on-device performance monitoring. For example, in some implementations, the Machine Intelligence SDK can also perform on-device training model quality verification (e.g., performance monitoring). These quality statistics can be forwarded to developers via dashboards provided by the application development platform.

[0080] In addition to on-device inference and / or training, or alternatives to on-device inference and / or training, the platform can also enable applications to receive inference and / or training services via the cloud. In one example, developers can specify (e.g., for a subset of devices and / or models) whether inference and / or training occurs on-device or via a cloud service. Both options can be supported by a single machine intelligence SDK, providing dynamically controllable flexibility around the location of inference / training. In some implementations, different APIs or parameters can be used to support each of these options. In some implementations, developers can specify rules governing when inference and / or training should occur on-device or in the cloud, and the machine intelligence SDK can implement these rules. For example, the use of rules can enable the platform to automatically handle the transition / balancing between on-device and cloud services (e.g., to implement workload balancing schemes). In some implementations, whether inference and / or training occurs on-device or in the cloud can be transparent to the device user.

[0081] Beyond dynamic model downloading and deployment, application development platforms can also implement other advanced machine learning services, including version control, compatibility, A / B testing, download management, and / or various other management functions. As an example, an application development platform can enable developers to perform or allow automated model target determination. For instance, different versions of machine learning models can be distributed to different types of devices. In one example, larger, more complex models can be distributed to devices with higher computing power (e.g., larger memory size, faster processors, etc.), while smaller, less complex models can be distributed to devices with lower computing power. In other examples, different model versions can be downloaded to different devices based on the device's geographic location (e.g., an English model for the UK versus a French model for France), the characteristics of the device's users (e.g., paid membership versus free trial), and / or other device and / or user attributes. For example, developers can specify rules controlling model target determination, and the platform can implement these rules. Therefore, the platform can enable the optimal version of the model to be sent (e.g., dynamically downloaded) to each different device.

[0082] As another example, application development platforms can provide monitoring and dashboards for model and system health, displaying assessments of these health metrics (i.e., model evaluation services). For instance, the platform can provide analytics on model usage, performance, download status, and / or other metrics. Performance statistics may include descriptions of accuracy, under-curve accuracy, precision relative to recall, confusion matrix, speed (e.g., #FLOP, milliseconds per inference), and model size (e.g., before and after compression). Developers can use this analytics to make decisions regarding model retraining and compression.

[0083] The platform can also provide dashboards that allow developers to explore system status and health, such as model compatibility, availability of stored models, download status, etc. Furthermore, the platform enables developers and devices to ensure (and, if necessary, resolve) forward and backward compatibility of model updates.

[0084] As another example, application development platforms enable developers to conduct A / B experiments. For instance, developers can make two models usable for different traffic volumes or different sets of devices. The platform can collect the resulting data and provide analytics that allow developers to explore performance statistics and results.

[0085] The systems and methods disclosed herein offer numerous technical effects and benefits. As an example of a technical effect, the application development platform and associated SDK enable on-device use of mobile-optimized machine learning models. These mobile-optimized models offer the following benefits: smaller model size; less memory usage; faster computation; and more efficient power utilization, all contributing to competitive model performance. Furthermore, lacking the platform tools described herein (which facilitate the easy generation and deployment of mobile-optimized models), application developers might choose to perform inference in the cloud. Therefore, by implementing on-device use, the systems and methods of this disclosure eliminate the need for cloud inference calls, thereby reducing network traffic. Additionally, by implementing on-device training, the systems and methods of this disclosure can lead to improved model performance, including, for example, personalized models and / or models already generated through joint learning.

[0086] As another example of technical effect and benefit, the systems and methods disclosed herein implement out-of-band model updates, where an updated version of the model can be dynamically downloaded to the machine intelligence SDK on the device without reinstalling the entire application. Separating the model download from the download of the entire application or, more generally, the application update download helps reduce the size of the application package download or application update download, thereby reducing wait time and application downtime. Additionally, the updated model can be distributed in a less intrusive manner and with a smaller download size.

[0087] Exemplary embodiments of this disclosure will now be discussed in more detail with reference to the accompanying drawings.

[0088] Example computing system

[0089] Figure 1 A block diagram of an example computing system according to an exemplary embodiment of the present disclosure is depicted. The example computing system includes a developer computing device 130, an application development computing system 102, and a user computing device 140 communicating via one or more networks.

[0090] The developer computing device 130 can be any type of computing device, including, for example, laptops, smartphones, desktops, server computing devices, etc. The developer computing device 130 may include one or more processors and memory. Application developers can use the developer computing device 130 to communicate with and / or control the application development platform 116 implemented by the application development computing system 102. As an example, the developer computing device 130 may include a dedicated computer program or application designed to communicate with and / or control the application development platform 116. As another example, the developer computing device 130 may include a browser application that communicates with the application development platform 116. For example, the application development platform 116 may provide services to the developer computing device 130 via a browser application.

[0091] Application development computing system 102 may include one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 may store data and instructions executed by processor 112 to cause user computing device 102 to perform operations.

[0092] In some implementations, the application development computing system 102 includes or is otherwise implemented with one or more server computing devices. In instances where the application development computing system 102 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0093] Application development computing system 102 provides application development platform 116. Application development platform 116 can provide comprehensive services for the generation, deployment, and management of machine learning models. The application development platform may include console manager 118, model manager 120, and machine learning manager 122.

[0094] The console manager 118 can control and / or manage the presentation of the console at the developer computing device 130, including tasks such as presenting the console interface, modifying the console interface, and receiving user input for the console interface. In some embodiments, the console manager 118 may be included in an application stored at the developer computing device 130, or may be a portion of both the developer computing device 130 and the application development computing system 102.

[0095] Model Manager 120 can provide a variety of model management services, including, for example, model compression services, model conversion services, model evaluation services, model hosting / download management services and / or other model management services, including, for example, version control, compatibility and / or A / B testing services.

[0096] Machine Learning Manager 122 can provide many machine learning services, such as model training services and / or training data management services. Machine Learning Manager 122 can include and use machine learning libraries to train models. Example machine learning libraries include TensorFlow and TensorFlow Lite libraries.

[0097] Each of the console manager 118, model manager 120, and machine learning manager 122 includes computer logic for providing desired functionality. Each of the console manager 118, model manager 120, and machine learning manager 122 may be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, each of the console manager 118, model manager 120, and machine learning manager 122 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, each of the console manager 118, model manager 120, and machine learning manager 122 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0098] Application development platform 116 can communicatively couple to one or more databases, including, for example, cloud storage device 124 and / or application data database 126. Application data database 126 can be a globally distributed relational and / or non-relational database. Cloud storage device 124 and application data database 126 can be a single database or multiple distributed databases.

[0099] Application development platform 116 can also communicate with one or more user computing devices 140. Each user computing device 140 can be any type of computing device, including desktop computing devices, server computing devices, mobile computing devices (e.g., smartphones, laptops, tablets, wearable devices, etc.), embedded devices, smart devices, Internet of Things devices, game consoles, computing devices in vehicles, and / or any other form of computing device. Each user computing device 140 may include one or more processors (e.g., as described at 112) and memory (e.g., as described at 114).

[0100] In some implementations, user computing device 140 may include application 142. For example, application 142 may have been developed by a developer associated with the developer computing device (e.g., using application development platform 116). Application 142 may include machine intelligence SDK 144 associated with the application development platform. Machine intelligence SDK 144 may include computer-readable instructions that enable application 142 to interface with application development platform 116 and / or other system components (e.g., via the use of one or more platform APIs).

[0101] According to various aspects of this disclosure, in some embodiments, the Machine Intelligence SDK 144 may include one or more machine learning models 146 and machine learning libraries 148. For example, model 146 may have been downloaded (e.g., dynamically and out-of-band) from cloud storage device 124 (e.g., according to instructions received from model manager 120). The Machine Intelligence SDK 144 may use machine learning libraries 148 to run machine learning models 146 to generate inferences for application 142 on user computing device 140.

[0102] In some implementations, the user computing device 140 may further include a first-party support application 152. The first-party support application 152 may be a standalone application or may be provided as part of the operating system of the user computing device 140.

[0103] First-party support application 152 may also include a machine intelligence SDK 154 associated with an application development platform. The machine intelligence SDK 154 may include computer-readable instructions that enable application 152 to interface with application 152 (e.g., machine intelligence SDK 144), application development platform 116 and / or other system components (e.g., via the use of one or more platform APIs).

[0104] According to various aspects of this disclosure, in some embodiments, the Machine Intelligence SDK 154 may include one or more machine learning models 156 and machine learning libraries 158. For example, model 158 may have been downloaded (e.g., dynamically and out-of-band) from cloud storage device 124 (e.g., according to instructions received from model manager 120). The Machine Intelligence SDK 154 may use machine learning libraries 158 to run machine learning models 156 to generate inferences for application 142 on user computing device 140.

[0105] Figure 2 A stack diagram of an example computing system according to an example embodiment of the present disclosure is depicted. In particular, Figure 2 The diagram illustrates a console interface that communicates / interacts with the model manager 120, machine learning manager 122, cloud storage device 124, and application data database 126 via one or more platform APIs.

[0106] As shown in the figure, the model manager 120 can provide model compression services, model conversion services, model evaluation services, and model hosting services. The model compression and / or model conversion services enable developers to compress and / or convert models to optimize them for use on mobile devices or in mobile environments. For example, model compression may include performing quantization (e.g., scalar quantization, vector quantization / weight sharing, product quantization, etc.), pruning (e.g., value pruning, L1 regularization, etc.), low-rank representation (e.g., circular matrices, Kronecker structures, SVD decomposition, etc.), distillation, and / or other compression techniques.

[0107] Pruning reduces model size by removing weights or operations that are least useful for prediction, including, for example, low-scoring weights. This can be particularly effective for on-device models involving sparse inputs. For example, session models on some devices can be further pruned to achieve up to a 2x size reduction while only reducing the trigger rate by 25%, and retaining 97% of the original prediction quality.

[0108] Quantization techniques can improve inference speed by reducing the number of bits used for model weights and activations. For example, using 8-bit fixed-point representations instead of floating-point numbers can speed up model inference, reduce power consumption, and reduce size by a factor of four.

[0109] Therefore, various compression tools can be optionally accessed and used to compress the learned or uploaded models.

[0110] Model conversion can include transforming a model from a standard version to a mobile-optimized version, which is compatible with lightweight machine learning libraries specifically designed for mobile and embedded devices. In one example, the platform can use a conversion tool called the TensorFlow Lite Optimizing Converter (TOCO) to convert a model's standard TensorFlow graph into a TensorFlow Lite graph, where TensorFlow Lite is a lightweight machine learning library designed for mobile applications. Therefore, various conversion tools can be selectively accessed and used to convert learned or uploaded models into mobile-optimized versions.

[0111] The model evaluation service provides analytics on model usage, performance (e.g., accuracy, speed, and model size), download status, and / or other metrics. Developers can use this analytics to make decisions regarding model retraining and compression. The service also provides dashboards that allow developers to explore system status and health, such as model compatibility, availability of stored models, download status, and more. Furthermore, the service enables developers and devices to ensure (and, if necessary, resolve) backward and forward compatibility of model updates.

[0112] As another example, model evaluation services enable developers to perform A / B experiments. For instance, developers can make two models usable for different traffic volumes or different sets of devices. The platform can collect the resulting data and provide analytics that allow developers to explore performance statistics and results.

[0113] Model hosting services enable models to be downloaded to devices (e.g., devices that store the developer's application). Model hosting services also allow developers to perform various other management functions, such as version control, compatibility, download management, and model target determination.

[0114] The Machine Learning Manager 122 can provide training data management services. This service can include training data streams with comprehensive services, such as data augmentation, data visualization, and data labeling crowdsourcing. Training data augmentation can include data rotation, perturbation, and supplementation (e.g., via image search).

[0115] The Machine Learning Manager 122 can provide model training services. These services can perform model training and / or retraining. For example, standard training techniques can be used to train models based on training data.

[0116] In some embodiments, the model training service may execute training techniques provided by this disclosure, resulting in the learning of computationally or memory-efficient models. In particular, in some embodiments, the model training service may execute novel training schemes that enable the direct learning of compact machine learning models. For example, the model training service may jointly train compact machine learning models and larger, more complex models in a teacher-student training architecture, enabling the compact machine learning model to learn from and approximate the more complex machine learning models.

[0117] In some implementations of the scheme described herein that enables direct learning of compact models, quantization can be applied during training. That is, the model (e.g., the network) learns to optimize weights and activations in the quantization space using gradients computed via backpropagation. This is more efficient than applying this approach after training (e.g., quantizing pre-training weights only for inference).

[0118] Some of the joint training and refinement methods disclosed herein follow a teacher-student setting, in which the knowledge of the trainer model is used to learn an equivalent compact student model with minimal loss of accuracy. During training, the parameters of either the teacher or trainer model can be kept fixed (e.g., as in refinement) or jointly optimized to improve both models simultaneously.

[0119] In some implementations, the trainer model can also be jointly trained with multiple student models of different sizes. Therefore, instead of providing a single compressed model, the machine learning manager 122 can generate multiple on-device models with different sizes and inference speeds, and developers can choose the model best suited to their application needs (e.g., providing the most appropriate trade-off between size and performance). Furthermore, jointly training multiple compact models with shared parameters typically takes only slightly more time than training a single large model, but produces multiple compressed / compact models in a single shot that are smaller, faster, and less expensive than more complex models, while still providing good predictive accuracy. These techniques can be combined with other techniques such as transfer learning to make the learning / compression process more efficient and scalable to large-scale datasets.

[0120] More generally, the machine learning manager 122 may include and implement a training pipeline to train a compact machine learning model. The training pipeline may train the compact machine learning model individually and / or jointly train the compact machine learning model with a pre-trained machine learning model.

[0121] In some implementations, the training pipeline may receive one or more inputs from a user. For example, the training pipeline may receive training data along with corresponding input functions for training and for evaluation. The training pipeline may create patterns specifying how one or more training operations will be performed on a compact machine learning model. The training pipeline may also provide an experiment-based API (e.g., tf.Experiment) for building networks. The training pipeline may invoke training (e.g., the wrapper code and / or training below to begin training).

[0122] A training pipeline can train a compact model until a desired number of steps are reached. The training pipeline can export the trained compact machine learning model in a specific format (e.g., TF-Lite format). In some implementations, the trained compact machine learning model can then be used to run on a computing device (e.g., on a mobile device).

[0123] In some implementations, the created schema may include several fields, such as experiment name, features (e.g., field name, feature type, one or more dimensions of the feature, etc.), hyperparameters (e.g., learning rate, number of steps, optimizer, activation layer, loss weights for the pre-trained model, loss weights for the compact model, cross loss weights, etc.), and a model specification for a compact model that includes multiple fields to construct a compact model.

[0124] In some implementations, the model specification may include some or all of the following example information:

[0125] id: A unique identifier for this model instance.

[0126] model_type: (e.g., "feed_forward", "projection").

[0127] params: A general dictionary of parameters specific to the model type. For example, projection_size will be the parameter for the projection net, not the parameter for the feedforward net.

[0128] load_checkpoint: If this parameter is set, the model will be loaded from the path where this parameter has a value.

[0129] `freeze_model_from_checkpoint`: This parameter should only be set to `true` if `load_checkpoint` is set. If this parameter is `true`, gradients will not be backpropagated through this model. For `pod_model_spec`, this parameter should not be set to `true` (because freezing the PoD model and training the trainer model would be pointless).

[0130] tflite_output_path: If this parameter is set, the pipeline will output this model as a TFLite model (assuming that all operations in the model are supported by TFLite) to the given path.

[0131] If both `pod_model_spec` and `trainer_model_spec` are defined, joint training will occur. If only `pod_model_spec` is specified, the pipeline will train individual models. Specifying only `trainer_model_spec` will be an error.

[0132] In some implementations, the training pipeline can remove or add fields to the model specification based on ongoing development and / or new use cases. In other implementations, the training pipeline can validate the schema to ensure that the training pipeline behaves as expected. Each field in the schema can be validated before starting the pipeline. Fields in the model specification can also be validated using the validation methods of the corresponding model class.

[0133] In some implementations, the training pipeline can implement and / or store one or more common machine learning models (e.g., feedforward networks, projective networks, quasi-recurrent neural networks, convolutional neural networks, long short-term memory networks, etc.) for joint training. Accordingly, instead of inputting pre-trained machine learning models or custom compact model specifications, users can choose which models to store and / or implement in the training pipeline, and can input these choices into the training pipeline. In response to the user's selection, the training pipeline can create a pattern specifying how training will be performed with the selected model(s).

[0134] In some implementations, the training pipeline may include one or more debugging metrics, such as various losses, accuracies, confusion matrices for the pre-trained machine learning model, and the compact machine learning model. In some implementations, these added metrics are separate from other metrics (e.g., metrics integrated with Tensorboard). In some implementations, the training pipeline may include example implementations for using wrappers for the tf.Estimator API and other plugins. In some implementations, the training pipeline may include integration with TOCO to export the trained compact machine learning model to TF-Lite format.

[0135] As an example, Figure 15 A workflow diagram for an example training pipeline according to an example embodiment of this disclosure is depicted. The training pipeline validates DeepPodSpec for shared fields between models (e.g., compact models and pre-trained models). In some implementations, the training pipeline may invoke validation methods (e.g., validation methods (one or more) for the corresponding model type) to validate the model specification of the corresponding model, and if such validation fails, the training pipeline will encounter an error.

[0136] The training pipeline creates a `model_fn` to be used by a `tf.Estimator` object, which performs the following operations: 1) instantiates the corresponding model class for each model, causing the model to perform bookkeeping and sanity checks; 2) calls the `forward_pass` method (e.g., `create_graph`) for each model; and 3) constructs and readjusts the loss function and training operations based on the predictions and corresponding specifications from each model. The training pipeline constructs the `tf.Estimator` based on the output from the `model_fn` and / or other input-related flags in the specifications.

[0137] The training pipeline constructs a tf.Experiment object with hooks to call persist for each model. The training pipeline returns a tf.Experiment object.

[0138] The training pipeline can also convert the trainer GraphDef into an inference GraphDef. It can extract the PoD model (if jointly trained) into separate networks for inference. The training pipeline can also convert frozen inference graphs into TF-Lite format.

[0139] Figure 16A A graphical representation of an example joint training scheme according to an exemplary embodiment of this disclosure is provided. Figure 16A In this approach, a projection network is jointly trained with a trainer network (e.g., a feedforward network, a quasi-recurrent neural network, a convolutional neural network, a long short-term memory network, etc.) to learn to mimic the predictions made by the trainer network, which has many more parameters and therefore stronger predictive power. The projection network is provided as an example type of model that can be compactly learned as a student model. Other types of compact models can be jointly learned according to the training scheme shown, including, for example, feedforward neural networks, recurrent neural networks (e.g., long short-term memory networks), quasi-RNNs, convolutional neural networks, ProjectionNet (e.g., dense and sparse versions), BiLSTM (bidirectional LSTM), depth-separable ConvNet, MobileNet, ProjectionCNN, NASNet, Inception (e.g., Inception v3), ResNet, and / or other types of machine learning models.

[0140] A central aspect of the joint training initiative is the novel goal of jointly training two distinct models—a complete trainer model (e.g., using existing architectures such as feedforward NNs or LSTM RNNs) combined with a simpler student model. For example, the student model could be a projective network that fully leverages random projections to transform the input or intermediate representation into bits. Thus, in some implementations, the simpler model can encode lightweight and computationally efficient operations in a bit space with a low memory footprint.

[0141] In some implementations, backpropagation can be used to jointly train the two models, similar to apprenticeship learning, where the student network learns from the teacher network. Once trained, the smaller network can be used directly for inference with lower memory and computational costs.

[0142] The student model can optionally leverage any existing deep network (such as a feedforward or recurrent neural network) to teach a lightweight model (e.g., a projected model) in a joint optimization setting that is trained end-to-end using backpropagation. In some implementations, projections based on locality-sensitive hashing can be used to represent the hidden units of the lightweight network, which encodes operations that are computationally very efficient during inference.

[0143] This framework allows for efficient distributed training, but can be optimized to produce neural network models with low memory footprints that can run on devices at low computational cost.

[0144] The model size can be parameterized and is configurable based on task or device capacity. This framework is effective in achieving significant model size reduction while delivering competitive performance on a variety of vision and language classification tasks.

[0145] Figure 16A The framework is illustrated, and its use in training projection networks will be discussed. However, other model types besides projection networks can also be used as student models.

[0146] In some implementations, the trainer network uses Used as input feature vectors to predict the prediction result. Projection networks use projection functions ( ) Input feature vector eigenvectors transformed into projections Projection networks use the feature vectors of the projection to predict the outcome. .

[0147] During joint training, the loss function of the trainer network can be optimized. and projection loss function The combination of these elements allows the projection network to mimic and learn from a trained network. As shown in Figure 16, there is a label loss. It was also used for training. It has labeled loss. Based on the prediction results Compared to ground truth A comparison between them.

[0148] Once learning is complete, the transformation function P(.) and the corresponding training weights W can be extracted from the projection network. p B p This allows for the creation of a compact model that can be pushed onto the device. During inference, the compact model and its corresponding operations can be applied to a given input. To generate predictions .

[0149] More specifically, neural networks are a type of non-linear model that learns from inputs. To output The mapping, where This represents the input feature vector or sequence (in the case of a recurrent neural network), and These networks are used for classification tasks, outputting categories, predicted sequences, regression values, etc. Typically, they consist of multiple layers of hidden units or neurons, with connections between pairs of layers. For example, in a fully connected feedforward neural network, the number of trained weighted connections or network parameters is... , where n is the number of hidden units in each layer. Other models can similarly receive input and produce output.

[0150] This disclosure provides a novel objective and joint optimization framework for training compact on-device models for inference. Figure 16A The diagram illustrates a neural projection network architecture using a feedforward neural network (NN) for training the network. Coupled networks are jointly trained to optimize the combined loss function.

[0151]

[0152] in , and It is the loss function corresponding to the two models defined below.

[0153]

[0154]

[0155]

[0156] N indicates the number of training instances in the dataset. This represents the input feature vector in a feedforward network or the sequence input in an RNN, and This refers to the ground truth output class used for network training. This represents the parameterized representation of the hidden units in the trainer network, which will... Transform into output prediction Similarly, the table This means transforming the input into the corresponding prediction. The projection network parameters. Softmax activation can be applied to the last layer of both networks to compute the prediction. and .

[0157] D represents the distance function of the prediction error used in the measurement loss function. This can be decomposed into three parts: trainer prediction error, projection simulation error, and projection prediction error. Reducing the first term results in a better trained trainer network, while reducing the last term leads to a better projected network, which is simpler but has roughly equivalent prediction capabilities. In practice, cross-entropy can be used... For the projection Lp in Equation 2, we can optimize it by following the extraction method. Because it has been shown that it is more than just on the label. The trained model produces better generalization ability. and These are the hyperparameters that influence the trade-offs between these different types of errors. These parameters are tuned on a small reserved development set, and as an example, they are set to λ1 = 1.0, λ2 = 0.1, and λ3 = 1.0 in the experiments.

[0158] The trainer network (θ). The trainer model can be a fully neural network (feedforward, RNN, or CNN), and its choice can be flexible and task-dependent. Figure 16A A trainer using a feedforward network is shown, but this can be interchanged with LSTM RNNs or other models such as deep neural networks. For the network shown in the figure, layers are used... In The activation can be calculated as follows:

[0159]

[0160] Where σ is the ReLU activation function applied in every layer except the last layer, and A represents the calculated activation value for the hidden unit.

[0161] Weights / bias parameters in this network , The number can be arbitrarily large, as this will only be used during the training phase, which can be efficiently accomplished using high-performance distributed computing with CPUs or GPUs.

[0162] Projection network (p). A projection model is a simple network that encodes a set of computationally efficient operations that will be executed directly on the device for inference. The model itself defines an efficient "projection" function. set A set of ) that will contain each input instance Projected into different spaces Then, learning is performed in this space to map it to the corresponding output. .like Figure 16A The example shown uses a simplified projection network with almost no operations. An example projection network is illustrated. Other structures (e.g., additional layers, etc.) can also be used. A series of T projection functions are used... , , To transform the input Then, it can be followed by single-layer activation.

[0163]

[0164]

[0165] Projective transformations use pre-computed parameterized functions; that is, they are not trained during the learning process, and their outputs are concatenated to form hidden units for subsequent operations. During training, simpler projection networks learn to select and apply specific projection operations. (Through activation) these specific projection operations become more predictive for a given task. It is possible to superimpose additional layers connected to the bit layers in this network to achieve nonlinear combinations of projections.

[0166] The projection model can be jointly trained with a trainer and learns to mimic the predictions made by a fully trained network, which has many more parameters and therefore stronger predictive power. Once learning is complete, the transformation function from the projection network is extracted. and the corresponding trained weights , This allows for the creation of lightweight models that can be pushed to devices. During inference, the lightweight model and its corresponding operations can then be applied to a given input. To generate predictions .

[0167] Projection matrix in settings Type selection and projection space The representation directly impacts computational cost and model size. A modified version of locality-sensitive hashing (LSH) is used to fully leverage the efficient randomized projection method to define... In combination, for Using bit representation That is, the hidden units of the network are represented using projected bit vectors. Compared to a complete network, this significantly reduces memory usage in terms of both the number and size of parameters. Some key features of this method include:

[0168] Unlike typical machine learning approaches that rely on a smaller vocabulary size as a scaling mechanism, this approach does not require a commitment to a predefined vocabulary or feature space. For example, LSTM RNN models typically apply pruning and use a smaller, fixed-size vocabulary in the input encoding step to reduce model complexity.

[0169] The proposed learning method scales efficiently to large data sizes and high-dimensional spaces. This is particularly useful for natural language applications involving sparse, high-dimensional feature spaces. For dense feature spaces (e.g., image pixels), existing operations, such as fully connected layers (or even convolutions), can be efficiently approximated for prediction without relying on a large number of parameters. Such operations can also be combined with projection functions to produce more complex projection networks while limiting memory requirements.

[0170] The calculation can be independent of the training data size.

[0171] It can ensure It can efficiently perform real-time calculations for inference on the device.

[0172] The projection method and associated operations are described in more detail below.

[0173] Locally sensitive projection networks: The projection networks described above depend on the transformation function A set of transformation functions that will input Projection to hidden unit representation In this context, different types of functions can be used to perform the projection operation outlined in Equation 4. One possibility is to use a feature nesting matrix pre-trained using word2vec or similar techniques, and... Modeling for The feature is nested in a lookup table, and then aggregation operations such as vector averaging are performed. However, this requires storing the nested matrices, which leads to additional memory complexity.

[0174] Alternatively, efficient randomized projection methods can be used for this step. Locality-Sensitive Hash (LSH) can be used to model the underlying projection operation. LSH is commonly used as a dimensionality reduction technique for applications such as clustering. The motivation for using LSH in Projection Nets is that it allows for similar inputs to be modeled. Alternatively, intermediate network layers can be projected onto nearby hidden unit vectors in the metric space. This allows for the transformation of the input and the learning of an efficient and compact network representation that depends only on the inherent dimensionality of the data (i.e., the observed features), and not on the number of instances or the dimensionality of the actual data vectors (i.e., the overall features or vocabulary size). This can be used for... It is implemented using a binary hash function.

[0175] Theorem 1: For , and from Vectors drawn from a spherically symmetric distribution on The sign of the inner product and the angle between the vectors The relationship between them can be represented as:

[0176]

[0177] This property is based on a simple geometric principle, namely, as long as it comes from the projection matrix. The row vector in and If the direction of the vectors falls within the angle between the unit vectors, they will produce opposite signs. (This is in contrast to vectors containing...) Any projection vector orthogonal to the plane will not have any effect. Since the inner product can be used to determine the parameter representation of the vicinity, this is because... Therefore, the hidden activation unit vectors of the network can be modeled and stored in an efficient manner by using symbolic signatures of vectors.

[0178] Calculate the projection: Following the above properties, binary hashes can be reused and applied... The projection vector in the input will be Transformed into The binary hash representation, where This results in a d-bit vector representation, with one bit corresponding to each projected row. correspond.

[0179] Projection matrix This can be fixed before training and inference. Note that there is no need to explicitly store the random projection vectors. This is because hash functions can be used to compute them at runtime, instead of calling a random number generator. Furthermore, this allows for projection operations that are linear on the observed feature size rather than on the overall feature size (which can be excessively large for high-dimensional data), thus saving memory and computational costs. Binary representation can be important because it results in a very compact representation of the projected network parameters, which in turn significantly reduces the model size compared to the trainer network.

[0180] Note that other techniques (such as quantization or weight sharing) can be layered on this approach to provide further benefits in terms of memory reduction.

[0181] Projection parameters: In practice, T different projection functions can be used. ,like Figure 16A As shown, each projection function produces a d-bit vector, which can be concatenated to form the activation unit of the projection in Equation 4. T and d depend on the fact that... The specified projection network parameters can be configured and varied, and can be adjusted to strike a tradeoff between prediction quality and model size.

[0182] Training and Inference: Compact bit cells can be used to represent projection networks, as described earlier. During training, this network learns in the bit space of the projection. The gradient moves in the same direction for points nearby to each other. The direction and magnitude of the gradient can be determined by a trainer network with access to a larger set of parameters and a more complex architecture.

[0183] Two networks are jointly trained using backpropagation. Although there is a joint optimization objective, stochastic gradient descent can make training efficient when performing distributed computation on high-performance CPUs or GPUs.

[0184] Once trained, the two networks will be decoupled and used for different purposes. The trainer model can be deployed anywhere a standard model can be used. The weights and transformation function of the simpler projective network model are extracted. This allows for the creation of lightweight models that can be pushed to devices. This is achieved by applying the same operations from equations 4 and 5 to the new input. And generate predictions This model can be used directly "on" the device during inference.

[0185] As a bit vector representation An alternative method is to use a projection matrix instead. This is used to generate sparse representations of hidden units in the projected network. Each d-bit block can be encoded as an integer instead of a bit vector. This results in a larger parameter space overall. However, it can still be beneficial for applications where the actual number of learned parameters is small and inference can be performed through efficient sparse search operations.

[0186] Any compact machine learning model can be jointly trained with any trainer model, including, for example, pre-trained machine learning models.

[0187] In some implementations, the trainer model can also be jointly trained with multiple student models of different sizes. As an example, Figure 16B A graphical representation depicts an example joint training scheme for training multiple student models according to an example aspect of this disclosure. The multiple student models can be...

[0188] The compressed learning framework described in this paper has been used in example experimental tests and has successfully generated small, fast models with good prediction accuracy suitable for mobile applications. For example, on the ImageNet task, the compressed learning framework can generate models that are 22 times smaller than the Inception v3 baseline and 4 times smaller than the MobileNet v1 baseline, while the accuracy drops by only 4.6-7%. As another example, on CIFAR-10, as described in this paper, jointly training multiple models with shared parameters takes only 10% more time than training a single large model, but produces 3 compressed models that are up to 94 times smaller, up to 27 times faster, and up to 36 times less expensive, while maintaining good prediction quality (top-1 accuracy of 90-95%).

[0189] Figure 17 An example user interface according to an exemplary embodiment of this disclosure is depicted. The user interface may be referred to as Tensorboard. Once training begins, the user can monitor the training progress on Tensorboard. Tensorboard has several options (e.g., scalar, graph, distribution, histogram, and projector) to present the training progress.

[0190] like Figure 17As shown, the "Scalar" option can be selected to present the accuracy of the compact model (also known as the PoD model) and the trainer model. The left side of the user interface displays several tools for manipulating the charts on the right side of the interface, such as "Show Data Download Links" to show download links for the data associated with the PoD model and / or the trainer model, "Ignore Outliers in Chart Zoom" to remove outliers from the charts on the right side of the interface, "Tooltip Sort Method" to allow the user to choose the sorting method, "Smooth" to allow the user to choose the smoothing level to smooth the accuracy curves on the right side of the interface, "Horizontal Axis" to allow the user to choose the type of horizontal axis for the accuracy curve (e.g., run step, relative time, and wall time), and "Run" to write regular expressions to filter runs.

[0191] Figure 18 An example user interface according to an example embodiment of this disclosure is depicted. Figure 18 As shown, select the "Graphics (GRAPHS)" option. The graph on the right side of the user interface shows an example structure for training. This structure illustrates how joint training of the PoD model and trainer can be performed. The structure shows the data flow from bottom to top and illustrates each function that can be used to jointly train the PoD model and trainer. Several tools are listed on the left side of the user interface. Users can select tools to manipulate the graph on the right side of the user interface. For example, input can be tracked by activating "Track Input".

[0192] Refer again Figure 2 The application data database 126 can store model metadata. The cloud storage device 124 can store machine learning models (including both third-party and first-party models) and training data. Training data can be uploaded by developers and may include validation data. Alternatively or additionally, training data can be exported from large public training datasets.

[0193] As shown in the figure, the console interface can communicate / interact with the model manager 120, machine learning manager 122, cloud storage device 124, and application data database 126 via one or more platform APIs. Sample APIs are provided in the appendix. These APIs are provided as examples only. Different APIs may be used as a supplement or alternative to the sample APIs.

[0194] Appendix A provides sample custom model APIs. Appendix A is incorporated into and forms part of this specification.

[0195] Appendix B provides an example visual model API. Appendix B is incorporated into and forms part of this specification.

[0196] Appendix C provides a sample visual model API. Appendix C is incorporated in and forms part of this specification.

[0197] Now for reference Figure 3 , Figure 3 A functional diagram illustrating example console interface modes / states according to exemplary embodiments of the present disclosure is provided. As shown, in one workflow, developers can perform operations through various console interfaces that enable them to upload models, check compatibility, deploy models, and then monitor model performance. In another workflow, the console may provide interfaces for uploading training data (e.g., uploading to a cloud storage bucket), optionally augmenting the training data, training and / or compressing the model, then deploying the model and monitoring performance.

[0198] In some implementations, the console can provide multiple tools or user interfaces to help developers generate the correct machine learning models. As an example, the console can provide a set of use cases for developers to choose from, which will determine the target model architecture. In another example, the console can enable developers to select or explore trade-offs between accuracy and model size requirements.

[0199] Figure 4 A workflow diagram of an example computing system according to an example embodiment of this disclosure is depicted. More specifically, in some implementations, model compression and / or transformation are long-running tasks and are therefore not processed directly within the front-end server. In such cases, processing is performed using a dedicated back-end job. Figure 4 An example layout of such a system is shown, which includes a job scheduler with a database queue to handle job scheduling between the front end and the back end.

[0200] Backend operations may require running models provided by developers. As a result, the process of running the service can be sandboxed to provide privacy and security. For example, a cloud machine learning engine can be used.

[0201] In some implementations, at a high level, the platform can treat model compression and transformation as jobs. Therefore, in some implementations, when a developer wants to compress or transform a model, the platform front-end server can create a job and insert it into a database queue. A scheduler can handle the actual job scheduling, while the platform front-end server continues to poll the job's status and report it to the developer when it enters the terminal state. Developers can also initiate queries to check the status of their jobs.

[0202] Figure 5 A workflow diagram of an example computing system according to an example embodiment of the present disclosure is depicted. Specifically, Figure 5An example of basic device-side flow is depicted. Specifically, the application development platform and machine intelligence SDK allow developers to use custom models on the device. The API allows applications to perform inference using custom models already present on the device; developers can manually download model files from a location or package them within the application itself.

[0203] The platform also provides a model file hosting solution tightly integrated with the client-side SDK, which will offer the following features: a user interface section of the platform console where developers can manage their models; model version control, allowing developers to specify active (or production) models and / or easily perform rollbacks; automatic determination of the compatibility of uploaded models with different machine learning library versions, and providing only compatible models to compatible devices; and / or best-in-class A / B testing support, allowing developers to run experiments on different versions of their models.

[0204] Example model generation and deployment workflow

[0205] Figure 6 A flowchart depicts an example process for generating and deploying a machine learning model according to an example embodiment of this disclosure, while Figure 7 A workflow diagram illustrating an example implementation of an example process for generating and deploying machine learning models according to an example embodiment of the present disclosure is provided.

[0206] Specifically, aspects of this disclosure relate to an on-device machine intelligence SDK that forms part of a cross-platform third-party mobile machine learning platform operated via API. For mobile developers building on-device machine learning features, training models for mobile applications remains a major challenge for most. Furthermore, training workflows are very similar for many mobile use cases. Therefore, this disclosure provides tools to facilitate the entire model training workflow in the cloud.

[0207] Step 1: Model Selection and Design: For most use cases (e.g., mobile use cases), the training process begins with pre-trained models (e.g., Inception v3, MobileNets, etc.). The platform can provide access to a set of pre-trained models that cover some typical computer vision use cases and can be guaranteed to be converted into a mobile-optimized format. The optional model set can also cover other domains or custom models.

[0208] Step 2: Training Data Creation: The platform helps developers augment and manage their training data. Data augmentation can include adding more training samples (e.g., via image search), leveraging crowdsourcing platforms to provide labels, and introducing / transforming existing samples (e.g., adding noise, rotation, perturbation, etc.). The platform can also provide data cleaning and visualization tools.

[0209] The platform offers multiple data format options, including, for example, labeled raw data and / or TF.example. The platform also offers multiple model format options, including, for example, SavedModel and / or Checkpoint.

[0210] Step 3: Model Training: Training may include refining the pre-trained model (e.g., top-level only or more than top-level only). Training may also include fully automated training from scratch. Alternatively, training may be performed according to the joint training techniques described herein.

[0211] Step 4: Model Optimization for Mobile: Since model size is critical for mobile applications, platforms can provide tools to automatically compress models and evaluate performance and accuracy. Automated model optimization may include retraining the model, which may require training data.

[0212] Step 5: Mobile-Optimized Model Conversion: Once a model meeting the required performance and accuracy is obtained, the platform can convert it into a mobile-optimized format for deployment. Using command-line tools, the conversion can include several steps (e.g., model freezing, TOCO conversion, etc.). These steps can be streamlined, and error information can be provided via the platform console. In some implementations, the output model from the compression step can be automatically converted to a mobile format. Depending on the TOCO tool's readiness, the model can be forced into a standard architecture, for example, through POD compression (e.g., specifying the output model architecture).

[0213] Step 6: Model Deployment and Management: The application development platform provides developers with a target-setting mechanism, specifying which models should be applied to which application versions, device types, and / or user segments. The platform can also provide compatibility checks to ensure that the models are runtime-compatible with the machine learning libraries being targeted. Finally, the platform can host all models for developers, and applications can download / upgrade models on demand.

[0214] While the platform provides tools to help complete the training process, developers can also choose to fully utilize specific parts of the process. For example, if a developer is already ready to deploy their model, they can upload their model and use only step 6. If a developer already has a TensorFlow model for cloud inference, they can upload their TensorFlow model and also use steps 4 and / or 5.

[0215] Figure 8 A swimlane diagram depicts an example process for uploading a machine learning model for distribution according to an example embodiment of this disclosure. As shown, a user (developer) can initiate a model file upload (e.g., via their browser). The platform backend API can create a new object using cloud storage and can return the cloud storage upload URL to the user. The user can upload the file to the cloud storage and then notify the platform API when the upload is complete. The backend API can queue model compatibility check requests.

[0216] The model compatibility service can dequeue model compatibility check requests and retrieve model information from a backend API. The API can return a download URL for a cloud storage device. The model compatibility service can download model files from cloud storage, check compatibility, and extract model metadata. The model compatibility service can then use the compatibility information to update the model metadata.

[0217] Therefore, in some implementations, model compatibility can be implemented as a separate asynchronous service. When a model file is uploaded, a compatibility check request is added to a database queue. The compatibility check service then performs the check asynchronously. In some implementations, the model file may not be downloaded until the compatibility check is complete.

[0218] Figure 9 A swimlane diagram depicts an example process for downloading a machine learning model according to an example embodiment of this disclosure. As shown, upon initial loading, the Machine Intelligence SDK can request a model from an API. The API can look up a compatible version, authorize it, and then obtain a download URL from a cloud storage device. The API can return model information (e.g., the download URL) to the Machine Intelligence SDK. The Machine Intelligence SDK can then download the model from the cloud storage device using the URL and can cache the downloaded model.

[0219] During subsequent loading, the SDK can retrieve model information from the API. The API can return model information (including the download URL, if applicable). If needed, the SDK can then download the new model.

[0220] Example Machine Intelligence SDK

[0221] Figure 10-14 Example computing systems, including various example implementations of the Machine Intelligence SDK, are described. The Machine Intelligence SDK can be "fat," "partially fat," or "thin."

[0222] The application development platform allows developers to either bundle models with the SDK or dynamically download models to the SDK. In one example, developers can upload their models to the platform console, which will then be synchronized with and cached on the device. Once on the device, the models can be easily accessed by services on the device via simple API calls.

[0223] Regarding the extension of the SDK itself, the SDK can be extended to some of its components that have an "on-device backend," i.e., calls to other modules (e.g., first-party modules) or applications as part of its implementation (e.g., ...). Figure 11-14 (As shown in the image). Alternatively, all models and services can be included in a single machine intelligence module (such as...). Figure 10 (As shown in the diagram) Avoid duplicate machine learning library runtimes; optimize implementations and maintain a single stack (e.g., a C / C++ stack) for cross-platform use.

[0224] In particular, Figure 10 A block diagram of an example computing system including a fat SDK according to an example embodiment of this disclosure is depicted. Specifically, in the fat SDK, all machine intelligence models and services (e.g., vision, speech, text, custom, etc.) are unified into a single SDK. More specifically, all models and machine learning libraries / runtimes (e.g., TFLite) can be contained within a single machine intelligence module, such as... Figure 10 As shown in the image.

[0225] Generally speaking, a "fat" SDK means pulling all dependencies into a single, thick binary and statically linking it to the application. Typically, a fat SDK doesn't dynamically update any parts via updates. However, a fat SDK can interact with first-party modules or the application. Fat SDKs avoid any machine learning library compatibility issues.

[0226] Aside from the ability to dynamically update the SDK without requiring a full application update, a "fat updatable" SDK is similar to a fat SDK (e.g., a fat binary statically linked to the application). In some implementations, updated versions of first-party models are dynamically loaded and run during application execution, as long as an updated version is available on the device. This enables rapid updates to machine learning runtimes and models, but also introduces the possibility of machine learning library compatibility issues.

[0227] A "thin" SDK means that the SDK only contains client libraries and depends on whether the model runs in a first-party support module or within the application. These can still optionally run during application execution. The benefits of a thin SDK are a smaller application package size and the ability for multiple applications to share the implementation of modules.

[0228] As an example, Figure 11 A block diagram of an example computing system including a thin SDK according to an example embodiment of the present disclosure is depicted. Figure 11 As shown, the machine intelligence module can invoke other first-party modules suitable for cost-sharing among clients. For some of those parts of the SDK, the client library may reside within the SDK, while the implementation resides in the first-party module or application.

[0229] Therefore, various possible arrangements also allow for model management, including decisions regarding bundling versus dynamic downloading. In particular, in some instances, the model (plus any database required for inference) can be quite large (~MB), thus enabling developers to decide whether to bundle the model or download it dynamically for simplicity. For third-party models, developers can choose the appropriate option.

[0230] For first-party models, the model can be fully bundled in the SDK or downloaded dynamically, which is controlled by the first party. Bundling the model to the SDK offers the following benefits: the model is available from application installation; there is no need to wait for the download to occur; network and power usage is reduced; and there is greater stability, with dynamically downloaded models having a lower chance of compatibility breaks.

[0231] In some implementations, large first-party models can be shared between two or more applications. In one example, model sharing between applications can be achieved by making the model part of a first-party module (i.e., having a separate updatable module that handles the model / download). The first-party module is not part of a "fat" or "fat updatable" SDK, but rather in a "thin" part of the SDK (i.e., the SDK only pulls the client library of the module). The first-party module can be exposed to other parts of the SDK via a first-party API.

[0232] In some implementations, a separate machine learning library runtime may exist for the first-party model, which is updated along with the first-party model and is not used for models owned by the developers. This can help reduce compatibility issues.

[0233] Sharing models between applications provides storage benefits / savings. That is, if not shared, redundant versions of the model would have a huge impact on device storage (and if bundled, on application package size, or on power / data if they are downloaded).

[0234] In some implementations, the application development platform can distribute the SDK via a static framework. The static framework can be a single statically linked library coupled with a header (used in Objective-C and Swift) and some additional metadata (e.g., a module graph). It is also possible to distribute the SDK as a dynamically loaded library.

[0235] In some implementations, distribution can be done via CocoaPods, where CocoaPods is a JSON file specifying how to fetch the framework. The pod specification can point to a downloadable binary blob, or alternatively, to source code, such as source code hosted externally.

[0236] CocoaPods can contain one or more frameworks, but it typically packages only one framework into a pod. The platform can automate most of the build, test, and deployment pipelines.

[0237] In some implementations, if a module already exists in the SDK (e.g., mobile vision), it is easy to create a new SDK that can pull this dependency as is (e.g., instead of making a tighter integration).

[0238] In some implementations, it's possible to reference only symbols from another library, and it can be ensured that users of that library receive both. This is easily achieved using CocoaPods. Some examples can use several dependencies, such as Protobuf. The platform's build infrastructure can have clauses to ensure that these dependencies are not statically linked at build time, thus preventing symbol duplication and breaking.

[0239] Figure 12 A block diagram of an example computing system, according to an example embodiment of the present disclosure, includes a mobile vision model as part of an updatable first-party supporting application. Specifically, Figure 12 An architecture is described in which mobile vision models are provided to first-party supporting applications only via API, while third-party models are included in the machine intelligence SDK.

[0240] therefore, Figure 12 An example of a partially "fat" and partially "updatable" SDK is shown, where the custom model and TFLite runtime are located in the fat part, while the mobile vision module (and its tfmini runtime) are located in the updatable part.

[0241] The reason for making the mobile vision module a thin updatable (as opposed to fat updatable) model is to share storage space and download all model data across all applications using the SDK. Custom third-party models can be updated via the platform console and cloud storage. The platform can provide compatibility tests that can be run from the platform console to help mitigate potential compatibility issues.

[0242] Regarding bundled models: For mobile vision APIs, models can be downloaded based on their size. For some third-party custom models, developers can choose to bundle them with the model or update the model later via dynamic download.

[0243] Figure 13 A block diagram of an example computing system, according to an example embodiment of this disclosure, includes mobile vision, text, and speech models as part of an updatable first-party supporting application. That is, relative to... Figure 12 ,exist Figure 13 In this framework, more basic APIs (such as text and voice) are included in supporting applications, and each API may have its own module structure. Redundant common runtimes can be optionally removed.

[0244] Figure 14 A block diagram of an example computing system including an updatable first-party support application according to an example embodiment of the present disclosure is depicted. This updatable first-party support application also performs both runtime and training. That is, all first-party models and modules in the support application use the same runtime within the first-party support application, thereby eliminating redundant runtime, logging, and library downloads.

[0245] First-party supporting applications can also perform training. In other implementations, training can be part of a fat updatable SDK (as opposed to thin updatable SDKs and shared across all applications) to provide privacy benefits for application-specific data storage.

[0246] This paper discusses technologies related to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes discussed in this paper can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0247] Figure 10-14 The example SDK and application layouts shown are provided by way of example only. Many different variations and combinations of these example layouts can be made according to various aspects of this disclosure.

[0248] While the subject matter has been described in detail with respect to various specific exemplary embodiments of the present disclosure, each example is provided by way of explanation and not as a limitation thereof. Modifications, variations, and equivalents of such embodiments will be readily apparent to those skilled in the art upon understanding the foregoing. Therefore, this disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, which will be readily apparent to those skilled in the art. For example, features shown or described as part of one embodiment may be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such changes, variations, and equivalents.

[0249]

[0250]

[0251]

[0252]

[0253]

[0254]

[0255]

[0256]

[0257]

[0258]

[0259]

[0260]

[0261]

[0262]

[0263]

[0264]

[0265]

[0266]

[0267]

[0268]

[0269]

[0270]

[0271]

[0272]

[0273]

[0274]

[0275]

[0276]

[0277]

[0278]

[0279]

[0280]

[0281]

[0282]

[0283]

[0284]

[0285]

[0286]

[0287]

[0288]

[0289]

[0290]

[0291]

[0292]

[0293]

[0294]

[0295]

[0296]

[0297]

[0298]

[0299]

[0300]

Claims

1. A mobile computing device comprising: one or more processors; and one or more non-transitory computer-readable media collectively storing: a computer application; and a machine intelligence software development kit included in and forming a part of the computer application, the machine intelligence software development kit configured to: store one or more machine learning models and a machine learning library, wherein the machine intelligence software development kit is configured to download the one or more machine learning models at runtime of the computer application; communicate with the computer application using an application programming interface to receive input data from the computer application; perform on-device data logging and on-device model training based on the input data; implement the one or more machine learning models and the machine learning library on the device to produce inferences based at least in part on the input data; and communicate with the computer application using the application programming interface to provide the inferences to the computer application.

2. The mobile computing device of claim 1, wherein the machine intelligence software development kit is configured to perform the on-device model training according to a custom set of scheduling rules associated with the computer application.

3. The mobile computing device of claim 1, wherein the machine intelligence software development kit is configured to perform on-device performance monitoring of the one or more machine learning models.

4. The mobile computing device of claim 1, wherein the machine intelligence software development kit is configured to implement a rule set that specifies whether inferences occur using the one or more machine learning models stored in the machine intelligence software development kit or using one or more cloud-based versions of the one or more machine learning models stored in a cloud-based computing system.

5. One or more non-transitory computer-readable media collectively storing: a computer application, wherein the computer application includes a machine intelligence software development kit included in and forming a part of the computer application, and wherein the machine intelligence software development kit is configured to: download one or more machine learning models at runtime of the computer application; communicate with the computer application using an application programming interface to receive input data from the computer application; perform on-device data logging and on-device model training based on the input data; implement the one or more machine learning models and a machine learning library on the device to produce inferences based at least in part on the input data; and communicate with the computer application using the application programming interface to provide the inferences to the computer application.

6. The one or more non-transitory computer-readable media of claim 5, wherein the machine intelligence software development kit is configured to perform the on-device model training according to a custom set of scheduling rules associated with the computer application.

7. The one or more non-transitory computer-readable media of claim 5, wherein the machine intelligence software development kit is configured to perform on-device performance monitoring of the one or more machine learning models. ​ ​ 8. The one or more non-transitory computer-readable media of claim 5, wherein the machine intelligence software development kit is configured to implement a ruleset specifying whether inferences are to occur using the one or more machine learning models stored in the machine intelligence software development kit or one or more cloud-based versions of the one or more machine learning models stored in a cloud-based computing system.

9. A computer-implemented method comprising: receiving, by a machine intelligence software development kit included in and forming a part of a computer application stored by a mobile computing device, input data from the computer application via an application programming interface, wherein the machine intelligence software development kit is configured to perform on-device data logging and on-device model training based on the input data; in response to receipt of the input data, causing, by the machine intelligence software development kit, implementation of one or more machine learning models on the mobile computing device via a machine learning library to produce inferences based at least in part on the input data; and providing, by the machine intelligence software development kit, the inferences to the computer application via the application programming interface, wherein causing, by the machine intelligence software development kit, implementation of the one or more machine learning models on the mobile computing device via the machine learning library comprises performing, by the machine intelligence software development kit, a call to a first-party support application separate from the computer application causing the first-party support application to implement the one or more machine learning models using the machine learning library, wherein the machine intelligence software development kit is configured to download the one or more machine learning models at runtime of the computer application.

10. The computer-implemented method of claim 9, further comprising performing, by the machine intelligence software development kit, on-device performance monitoring of the one or more machine learning models.

11. The computer-implemented method of claim 9, further comprising performing, by the machine intelligence software development kit, on-device model training according to a custom set of scheduling rules associated with the computer application.

12. The computer-implemented method of claim 9, further comprising implementing a ruleset specifying whether inferences are to occur using the one or more machine learning models stored in the machine intelligence software development kit or one or more cloud-based versions of the one or more machine learning models stored in a cloud-based computing system.

13. A computing system comprising: one or more processors; and one or more non-transitory computer-readable media collectively storing: a computer application, and a machine intelligence software development kit included in and forming a part of the computer application, and the machine intelligence software development kit is configured to: store one or more machine learning models and a machine learning library, wherein the machine intelligence software development kit is configured to download the one or more machine learning models at runtime of the computer application, communicate with the computer application using an application programming interface to receive input data from the computer application; perform on-device data logging and on-device model training based on the input data; ​ implementing one or more machine learning models and machine learning libraries on a device to produce inferences based at least in part on input data; and communicate with the computer application using an application programming interface to provide inferences to the computer application.

14. The computing system of claim 13, wherein the machine intelligence software development kit is configured to perform on-device model training according to a custom set of scheduling rules associated with the computer application.

15. The computing system of claim 13, wherein the machine intelligence software development kit is configured to perform on-device performance monitoring of the one or more machine learning models.

16. The computing system of claim 13, wherein the machine intelligence software development kit is configured to implement a ruleset that specifies whether inferences are to occur using the one or more machine learning models stored in the machine intelligence software development kit or using one or more cloud-based versions of the one or more machine learning models stored in a cloud-based computing system.

Images