Machine Learning Model Compiler

Abstract

The present disclosure relates to a machine learning model compiler. The subject technology provides a framework for executable machine learning models that can be executed in a zero-runtime operating environment. This allows machine learning models to be deployed in limited memory environments such as embedded domains. A machine learning compiler is provided to generate executable machine learning models.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 041,766, filed on June 19, 2020, and entitled “Machine Learning Model Compiler,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This specification relates generally to machine learning, and more particularly to machine learning model compilers. Background Art

[0004] Software engineers and scientists have been using computer hardware for machine learning to improve applications across various industries. To aid in the design and development of machine learning models, machine learning libraries are sometimes provided that include machine learning functions that can be obtained and interpreted at runtime. Summary of the Invention

[0005] According to some embodiments of the present disclosure, a method is provided, comprising: providing a machine learning model to a machine learning compiler; generating, using the machine learning compiler, compilable source code for all operations of the machine learning model, the compilable source code being configured to be compiled by a separate compiler; and generating, using the machine learning compiler, an executable machine learning model using compiled machine code generated by the separate compiler from the compilable source code, the executable machine learning model being executable in a zero runtime operating environment.

[0006] According to some embodiments of the present disclosure, a non-transitory computer-readable medium is provided, which stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: providing a machine learning model to a machine learning compiler; generating, using the machine learning compiler, compilable source code for all operations of the machine learning model, the compilable source code being configured to be compiled by a separate compiler, the compilable source code including code for only a subset of a runtime library used by the machine learning model; and generating, using the machine learning compiler, an executable machine learning model using compiled machine code generated by the separate compiler from the compilable source code.

[0007] According to some embodiments of the present disclosure, a device is provided, comprising: a memory storing an executable machine learning model; and one or more processors configured to execute the executable machine learning model to generate a model output without accessing a runtime library. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Certain features of the subject technology are set forth in the appended claims.For purposes of explanation, however, several embodiments of the subject technology are set forth in the following figures.

[0009] Figure 1 An exemplary network environment is shown according to one or more implementations.

[0010] Figure 2 An exemplary computing architecture for a system for generating and running executable machine learning models according to one or more implementations is shown.

[0011] Figure 3 A schematic flowchart for generating a machine learning model according to one or more specific implementations is shown.

[0012] Figure 4 A schematic flow chart for executing a machine learning model according to one or more specific implementations is shown.

[0013] Figure 5 A schematic flow diagram for executing a compiled ML model using a runtime library is shown according to one or more implementations.

[0014] Figure 6 A schematic flowchart for generating an executable machine learning model according to one or more specific implementations is shown.

[0015] Figure 7 A schematic flow chart for executing an executable machine learning model according to one or more specific implementations is shown.

[0016] Figure 8 Aspects of an ML compiler tool for an ML compiler according to one or more implementations are shown.

[0017] Figure 9 Aspects of an executable ML model according to one or more implementations are shown.

[0018] Figure 10 A flowchart illustrating an exemplary process for generating an executable machine learning model according to one or more specific implementations is shown.

[0019] Figure 11 An electronic system is shown that can be used to implement one or more implementations of the subject technology. DETAILED DESCRIPTION

[0020] The specific embodiments shown below are intended to be descriptions of various configurations of the subject technology and are not intended to represent the only configuration by which the subject technology can be put into practice. The accompanying drawings are incorporated herein and constitute a part of the specific embodiments. The specific embodiments include specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein, but can be put into practice using one or more other specific implementations. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concepts of the subject technology.

[0021] The popularity of machine learning has increased significantly in recent years due to the availability of large amounts of training data and advances in more powerful and efficient computing hardware. Machine learning can utilize models such as neural networks that are trained and then executed to provide predictions in specific applications (e.g., analyzing images and videos, sound, object detection and / or tracking, etc.) among many other types of applications. However, providing machine learning in hardware-constrained environments, such as in devices with low power or limited memory processing environments, can be challenging.

[0022] For example, embedded domains present unique challenges to existing machine learning frameworks, which can hinder the deployment of machine learning models. Limited power budgets impose constraints on processing power and memory bandwidth. In these embedded domains, traditional operating systems are bypassed, making the system calls traditionally relied upon unavailable, and file systems through which external resources can be loaded are often absent.

[0023] Despite these and other hardware limitations, it is often desirable to embed machine learning directly into firmware with real-time constraints (e.g., restrictions on computation timing so that computations occur once relative to other computations). One of the most challenging constraints is size: in some cases, machine learning models must fit into an extremely small memory footprint to be feasible in embedded devices. Every byte of runtime required to execute a machine learning model affects the number of available weights in the machine learning model and, therefore, indirectly affects the accuracy of models deployed in these environments.

[0024] According to aspects of the subject disclosure, a machine learning compiler is provided to generate a self-sufficient executable machine learning model that substantially reduces or completely eliminates the need for and use of a runtime environment.

[0025] According to aspects of the subject disclosure, an executable machine learning model is provided that can be run in a small memory and / or low power environment (e.g., including on a wearable device such as a smartwatch and / or on an earbud with a limited power-constrained memory processor). The executable machine learning model can be generated by compiling a machine learning model generated using a machine learning library to compile a compilable (e.g., object-oriented) code version of the model. The compilable code can then be compiled and packaged to provide an executable machine learning model that can be executed without runtime calls to library functions and / or other runtime environment features.

[0026] Compiling all operations of a machine learning model in this manner also provides additional opportunities for optimization of the model. For example, the process of including all operations of a machine learning model in the compilable source code that can execute the machine learning model allows knowledge about the connections between layers of the model to be exploited to optimize operations across layers. Existing machine learning models are typically executed using an interpreter that can optimize individual operations but does not know about upcoming operations in the next layer and therefore cannot optimize across layers. Reducing or eliminating the runtime environment, as described in further detail below, and the resulting additional optimization opportunities (e.g., including arbitrary fusion capabilities) provide the ability to deploy machine learning to extremely small embedded devices.

[0027] Figure 1 An exemplary network environment 100 is shown according to one or more implementations. However, not all depicted components may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Variations in the arrangement and types of these components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.

[0028] The network environment 100 includes an electronic device 110 and a server 120. The network 106 can be communicatively coupled (directly or indirectly) to the electronic device 110 and / or the server 120. In one or more specific implementations, the network 106 can be an interconnected network that can include the Internet or devices communicatively coupled to the Internet. For the purpose of explanation, the network environment 106 is described in detail below. Figure 1 1 and 2. Network environment 100 is shown as including electronic device 110 and server 120; however, network environment 100 may include any number of electronic devices and any number of servers.

[0029] The electronic device 110 may be, for example, a desktop computer, a portable computing device such as a laptop computer, a smart phone, a peripheral device (e.g., a digital camera, headphones, earbuds), a tablet device, a wearable device such as a smart watch, a smart band, etc. Figure 1In the embodiment, the electronic device 110 is depicted as a mobile electronic device (e.g., a smart phone) by way of example. The electronic device 110 may be and / or may include the following with respect to Figure 2 and / or Figure 11 All or part of the electronic system.

[0030] In one or more specific implementations, the electronic device 110 may provide a system for generating and / or executing an executable machine learning model that can be executed in a zero-runtime operating environment (e.g., an operating environment that does not include a runtime environment and / or runtime libraries that are separate from the executable machine learning model and / or provided by an operating system). Executing the executable machine learning model in the zero-runtime operating environment may include executing the executable machine learning model without access to a runtime library (e.g., in an operating environment that does not include a runtime library). In addition, the electronic device 110 may provide one or more machine learning frameworks for training machine learning models and / or developing applications using such machine learning models (e.g., machine learning models that implement neural networks). In one example, such machine learning frameworks may provide various machine learning algorithms and models for different problem domains in machine learning. In one example, the electronic device 110 may include a deployed executable machine learning model that provides an output of data corresponding to a prediction or some other type of machine learning output. In one specific implementation, the electronic device 110 utilizes a trained executable machine learning model and continuously learns / retrains the model over time.

[0031] Figure 2 An exemplary computing architecture for a system for generating and / or executing an executable machine learning model according to one or more implementations is shown. For purposes of explanation, the computing architecture is described as being provided by electronic device 110; however, the computing architecture may be implemented by any other electronic device, such as a desktop computer, a laptop computer, a wearable device, a tablet computer, etc. However, not all depicted components may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and types of these components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.

[0032] As shown, the electronic device 110 may include a memory 200 that stores an executable machine learning (ML) model 220. In one or more specific implementations, the electronic device 110 may include a memory 202 that stores training data 210 and / or other input data for training the executable machine learning model 220 and / or training a machine learning model that can be compiled to form the executable machine learning model 220. In various specific implementations, the training of the machine learning model can be performed before or after compiling the machine learning model to form the executable machine learning model. In one or more specific implementations, the executable ML model 220 can be a trained machine learning model that has been compiled by the machine learning compiler 218 at the electronic device 110 or at another device and includes parameters (e.g., weights, biases, etc. associated with nodes of a neural network) that have been trained at the electronic device 110 and / or at another device / system and provided to the electronic device 110.

[0033] exist Figure 2 In the example of FIG, the electronic device 110 includes a processing circuit 208. As shown, the processing circuit 208 may include a central processing unit 204 (eg, a CPU), a graphics processing unit 206 (eg, a GPU), and a low-power processor 212.

[0034] In one example, the low-power processor 212 can be a small, low-power auxiliary processor (and / or ASIC) that can be continuously (or constantly) powered on to detect motion of the device or to detect and / or interpret touch, audio, or other input data for controlling the device, as provided in electronic devices such as smart phones, wearable devices such as smart watches, or audio devices such as headphones or earbuds. In one or more specific implementations, the low-power processor 212 can be configured to use less power than the CPU 204 and / or GPU 206. In one or more specific implementations, the low-power processor 212 can have less available on-chip cache than the CPU 204 or GPU 206. In one example, the low-power processor 212 can be an always-on processor (AOP), which is a small, low-power auxiliary processor implemented as an embedded motion co-processor, as provided in electronic devices such as or The executable machine learning model 220 may be executed by the low-power processor 212 in a zero-runtime operating environment (e.g., without accessing runtime libraries or resources that are not self-contained within the executable machine learning model 220).

[0035] like Figure 2As shown, processing circuitry 208 may also include local memory 211, such as a memory cache accessible to one or more of CPU 204, GPU 206, and / or low-power processor 212. In one or more implementations, processing circuitry 208 may be implemented using multiple separate chips corresponding to CPU 204, GPU 206, and low-power processor 212. In one or more implementations, low-power processor 212 may be provided without CPU 204 or GPU 206. In one or more implementations, low-power processor 212 may be configured to remain active (e.g., executing executable ML models 220 for audio trigger detection, motion detection, etc.) when CPU 204 and GPU 206 are inactive (e.g., in a sleep state of electronic device 110). In one or more implementations, processing circuitry 208 may be formed from a single processor complex having different core types or multiple processors of different types. For example, a processor complex may include a multi-processing system having multiple core clusters, each cluster having one or more cores of a core type interconnected by one or more buses and / or memory fabric interconnects.

[0036] like Figure 2 As shown, a machine learning composer 214 (ML composer) may be provided (e.g., for execution by one or more of the CPU 204 or the GPU 206 from code stored in the memory 200). The ML composer 214 may be a development tool for machine learning developers, by which one or more previously stored ML library functions may be obtained and / or combined from the ML library 216 stored in the memory 200 to build and / or train a machine learning model. For example, the ML composer may include tools to include calls to ML library functions in the ML library 216 in an ML developer language document, which may be executed by the interpreter. At runtime of an ML model such as that implemented in the document, the interpreter may obtain and execute (e.g., using the CPU 204 and / or the GPU 206) the ML library functions according to the machine learning model, as described in the ML developer language document.

[0037] However, the ML developer language document and the ML library 216 and / or runtime library for running the associated model may be too large to be implemented using, for example, the low-power processor 212 (e.g., too large to be implemented using the memory resources and / or processing resources of the low-power processor). For example, the low-power processor 212 may not have sufficient memory for the entire runtime library. A machine learning compiler 218 (ML compiler) may be provided (e.g., for execution by one or more of the CPU 204 or GPU 206 from code stored in the memory 200) to compile (e.g., using an ML compiler tool 219) the machine learning model generated (or otherwise generated) by the ML composer 214. Compiling the ML developer language document generates an executable ML model 220 that has a smaller memory footprint than the ML developer language document and associated library and can be executed using the low-power processor 212 (e.g., in a zero-runtime operating environment). For example, the ML compiler 218 may generate the kernel required to customize the ML model 220 (e.g., by creating records on the fly based on the model creator's requirements in the ML model 220). In one or more implementations, for part of the compilation process, ML compiler 218 can operate a separate compiler, such as compiler 221 (e.g., a conventional compiler for compiling code, such as structure-oriented code, such as object-oriented code, functional code, or code in any language that allows compilation of stand-alone executable programs) to generate machine code for a machine learning kernel and / or a wrapper around a machine learning kernel that can be packaged by ML compiler 218 to form executable ML model 220.

[0038] Figure 3 A schematic flow chart of an exemplary process for generating an ML model using the ML composer 214 is shown. As shown, user input 300 may be provided to the ML composer 214 to select and / or integrate one or more ML library functions (e.g., ML library functions for computer vision, natural language processing, sound classification, activity classification, labeling, estimation, finding similarity, visualization, activation layers, convolutional layers, fully connected layers, pooling layers, memory management, etc.) from the ML library 216 into an ML model 302. For example, the ML composer 214 may generate the ML model 302 by generating an ML language document including function calls to the ML library functions from the ML library 216, which may be executed by the interpreter. The ML composer 214 may also provide tools for training the model.

[0039] Figure 4An exemplary process is shown in which an ML model 302 (e.g., from an ML writer 214) is executed by an interpreter 400 (e.g., an interpreter at an electronic device 110 or another electronic device or system). As shown, the interpreter 400 may receive a model input 401 (e.g., image data for a computer vision ML model, audio data for a speech recognition ML model, etc.) and the ML model 302, and may (e.g., based on the ML model) obtain, optimize, and / or execute ML library functions from the ML library 216 to generate a model output 402 using the CPU 204.

[0040] like Figure 5 As shown, to reduce the memory footprint of the ML model, in one embodiment, the ML model 302 can be compiled for execution by the CPU 204 (e.g., without using the interpreter 400), such as by inserting calls to the runtime environment 502 into the compiled ML model 500 (e.g., a compiled executable binary that includes calls to the runtime environment 502). The runtime environment 502 can include runtime library functions corresponding to any of the ML library functions from the ML library 216 that can be used by any ML model, and / or additional runtime library functions, such as graphics functions, math functions, etc.

[0041] However, even if Figure 5 The reduced memory footprint shown still requires sufficient memory for loading the runtime environment 502 into memory during execution of the compiled ML model 500 , and for communication between the executable binary and the runtime environment, which may be prohibitive for execution by a low-power processor such as the low-power processor 212 .

[0042] According to one or more specific implementations, a solution is provided to completely reduce or eliminate the need and use of a runtime environment. Figure 6As shown, in one or more specific implementations, the ML compiler 218 converts the ML model 302 (e.g., from the ML writer 214) into (e.g., self-contained) compilable source code 600 that implements the network (e.g., compilable source code such as object-oriented code such as C code, and / or intermediate representation code for later use by a separate compiler 221). For example, the ML compiler 218 may identify and include (e.g., in the compilable source code 600) only a portion of a runtime library or another code library accessible for developing the model. For example, the portion of the code library included in the compilable source code 600 by the ML compiler 218 may include only the portion of the code library actually used by the ML model 302 (e.g., not including other unused portions of the library). The resulting compilable source code 600 can then be independently compiled by a separate compiler, such as compiler 221, to generate executable code 604 (e.g., a set of machine learning kernels and wrapper code for the set of machine learning kernels) that can be packaged by the ML compiler 218 and deployed to any environment. For example, the compiler 221 can be controlled by the ML compiler 218 (e.g., by the ML compiler tool 219) to generate one or more executable codes 604 (e.g., machine code) and return them to the ML compiler 218. It packages a container of executable code to form an executable ML model 220 (e.g., a container that includes a set of kernels and machine code wrappers for the kernels and provides a user-callable network interface). In this way, the ML compiler 218 can eliminate the need to manually develop custom ML solutions (e.g., in a structured language or other compilable language), thereby providing productivity gains for teams deploying ML in limited environments. This can be particularly valuable when exploring multiple iterations of a model to optimize between accuracy, model size, and model performance.

[0043] Neural network performance bottlenecks can generally be divided into compute-bound regions (where the processor's ability to execute instructions is the limiting factor in performance) and memory-bound regions (where the memory subsystem's ability to provide data is the limiting factor in performance). The ML compiler 218 can utilize ML compiler tools 219 to optimize the performance of the executable ML model 220. For example, the ML compiler tools 219 may include front-end tools for optimizing operation-level aspects of the compilable source code 600 (e.g., by combining ML and / or low-level operations, and / or performing memory layout and alignment operations) and / or for determining how loop operations (e.g., a collection of nested loops) should be optimized. The ML compiler tools 219 may include a loop optimizer, a loop interface tool for operating the loop optimizer to convert a compute-bound network into a memory-bound network, and / or one or more additional tools such as a profile tool.

[0044] The loop interface of the ML compiler tool 219 (see, e.g. Figure 8The loop interface 802 of the ML compiler 218 may provide a loop optimization framework that generates custom kernels for input neural networks and provides a large pool of optimizations to extract maximum performance. The loop interface may allow the ML compiler 218 to fuse arbitrary operations into a single kernel, thereby keeping tensor data in registers and significantly reducing memory bandwidth. For some network configurations, fusing (e.g., arbitrary) operations in this way eliminates the need to instantiate explicit tensors, thereby reducing memory requirements.

[0045] Using these methods together (e.g., (i) generating a compilable version of the ML model, (ii) multi-level optimization of the structure-oriented code, including generating kernels from loop interfaces, and (iii) compiling all code and resources of the ML model into a standalone executable ML model, such as executable ML model 220) enables the ML compiler to create high-performance zero-overhead implementations of machine learning models.

[0046] Figure 7 The process by which the executable ML model 220 may be executed by a processor, such as the low-power processor 212, is schematically illustrated. As shown, the executable ML model 220 and a model input 401 of the executable ML model may be provided to the low-power processor 212, which may generate a model output 402 using only the executable ML model 220 without accessing a runtime environment, such as Figure 5 The runtime environment 502 of the ML compiler 218 may be configured to include a subset of the runtime library used by the machine learning model in the executable ML model 220 (e.g., so that the entire runtime library does not need to be loaded into memory for executing the ML model). In this way, memory is used only for the precise operations performed by the executable ML model 220, without needing to store runtime library information not used by the executable ML model 220, and without requiring memory bandwidth usage to access the runtime environment.

[0047] Compiler-based approaches to optimizing machine learning, zero-runtime solutions generated by the ML compiler 218 (e.g., where the resulting executable model includes only the resources it uses, compared to the interpreted model), and kernel generation, optimization, and profiling techniques of the ML compiler tool 219 may provide methods that allow Figure 6 and Figure 7 The various advantages of the process shown produce high-performance, minimal-overhead ML solutions.

[0048] Figure 8is a schematic diagram illustrating aspects of the ML compiler tool 219. As shown, the ML compiler tool 219 may include a front end 800, a loop interface 802, and a loop optimizer 804. In one or more specific implementations, the ML compiler 218 (e.g., in the ML compiler tool 219) may also include a profile tool to help users tune machine learning models.

[0049] The front end 800 can be an interface for the ML compiler 218 between the ML model 302, the ML library 216, one or more other code libraries (such as a runtime library including code referenced by the ML model 302 and / or the ML library), and other ML compiler tools 219. For example, the ML compiler 318 can operate the front end 800 to optimize compilable source code such as the compilable source code 600, and perform other processes to compile the ML model including the optimized code into one (e.g., so that the executable ML model 220 is a standalone, zero-runtime ML model).

[0050] Because all operations of a model (including library operations) can be included by the front end 800 in the compilable source code 600 that is compiled to form an executable ML model, optimization of operations within that code can be based on operations before and / or after a particular operation, rather than limited optimization of every operation that can be performed by the interpreter.

[0051] For example, the front end 800 of the ML compiler 218 can describe the model, optimize the description at the operation level, and decompose the optimized operations into smaller steps for optimization by the loop interface 802. Optimizing at the operation level can include, for example, determining whether to pad tensors (e.g., padding tensors with empty space to align vector operations) and / or where to place transpose operations. The front end 800 can generate instructions for how to optimize nested loops within the operations of the ML model, for example, by selecting scheduling instructions for mapping kernels into loops and for combining and / or modifying loops (e.g., reducing decisions). Scheduling instructions can include instructions for unrolling or unrolling loops and / or instantiating one or more internal tensors. The front end 800 can generate instructions in part by using heuristics within the ML compiler itself to optimize at the operation level, decomposing the operations into nested loops and finding optimization options between loops, and detecting opportunities for optimization further down the loops (e.g., down to the level of machine instructions / machine code).

[0052] The loop interface 802 may receive instructions to optimize loops from the front end 800 and, based on the instructions, operate a loop optimizer 804 for performing optimizations at the loop level. The front end 800 and / or the loop interface 802 may generate instructions (e.g., for the loop interface 802 and / or for the loop optimizer 804) to optimize a set of nested loops across multiple operations and / or multiple layers of an ML model, which would not be possible without the ML compiler front end operating to pull in all the code for a separate model.

[0053] The loop interface 802 can be a low-level kernel generator within the ML compiler 218. The loop interface 802 can be configured to generate a set of kernels 900 (e.g., machine learning kernels) and wrapper code 902 (e.g., as shown in the example) that binds the kernels together for use in the executable ML model 220. Figure 9 shown).

[0054] The loop interface 802 can operate the loop optimizer 804 to transform, fuse, reorder, and / or restructure loops based on instructions from the front end 800. This allows the ML compiler 218 to select optimizations at each loop level. This fine-grained selection opportunity provided by the ML compiler 218 can be important for optimizing performance while reducing the memory footprint of the ML model.

[0055] In one or more specific implementations, the loop interface 802 may receive as input a loop nest (e.g., a set of nested loops) and instructions for optimizing the loop nest (e.g., scheduling instructions) (e.g., from the front end 800). The loop interface 802 may map the ML-specific loop nest received from the front end 800 onto a different domain that may be the optimization focus of a loop optimizer in the absence of the loop interface 802. The loop interface 802 may obtain an optimized version of the loop nest from the loop optimizer 804 and may also decompose the output of the loop optimizer into compilable code, such as C code, intermediate representation code of the compiler 221, and / or other compilable code.

[0056] In one or more implementations, the front end 800 and / or loop interface 802 can use multiple optimizations to fuse operations and subsequent operations into a single, more complex operation. This can reduce or eliminate each memory traffic generated by each individual operation reading from memory, writing results, etc., rather than allowing intermediate values ​​to remain in hardware registers. In one or more implementations, the front end 800 and / or loop interface 802 can simplify operations, such as arithmetic simplification (e.g., through folding).

[0057] In one or more specific implementations, the front end 800 and / or the loop interface 802 can determine (e.g., for controlling the loop optimizer 804 via the loop interface 802) an effective balance between fusion, unrolling, and instantiation based on competing code size and performance requirements. For example, determining the balance can include fusing bias+activation operations into convolutions to reduce memory traffic, and / or fusing all pointwise convolutions into surrounding (e.g., preceding and / or following) operations.

[0058] In one or more embodiments, the front end 800 and / or the loop interface 802 may determine (e.g., for controlling the loop optimizer 804 via the loop interface 802) instructions for contextual transformation (e.g., unfolding) of depthwise convolutions, such as contextual fusion of operations (separable convolution + bias + activation + dotted convolution) into the computation of subsequent depthwise convolutions, unfolding the innermost two layers of dotted convolutions, and / or statically setting limit information about tensors. Statically setting limit information about tensors can coordinate with unfolding optimization operations by completely removing additional memory instructions and runtime tests for size. In one or more embodiments, the front end 800 and / or the loop interface 802 may provide an efficient application programming interface (API) to pass buffers with known bounds to kernels generated by the loop interface 802 (e.g., using the loop optimizer 804 and / or the compiler 221). In one or more embodiments, the front end 800 and / or the loop interface 802 may provide aggressive memory reuse for the stack allocator to optimize executable machine learning models. In one or more implementations, the front end 800 and / or loop interface 802 can utilize, for example, callback-based allocation to ensure bounded memory usage (e.g., to prevent the executable machine learning model from allocating memory during execution time) for optimizing the executable machine learning model.

[0059] Figure 10 A flowchart of an exemplary process for generating an executable machine learning model using a machine learning compiler according to one or more specific implementations is shown. For the purpose of explanation, this article mainly refers to Figure 1 The process 1000 is described with reference to the electronic device 110. However, the process 1000 is not limited to Figure 1 1000, and one or more blocks (or operations) of process 1000 may be performed by one or more components of server 120 and / or by other suitable devices. Further for the purpose of explanation, the blocks of process 1000 are described herein as occurring sequentially or linearly. However, multiple blocks of process 1000 may occur in parallel. In addition, the blocks of process 1000 do not need to be performed in the order shown, and / or one or more blocks of process 1000 do not need to be performed and / or may be replaced by other operations.

[0060] At block 1002, a machine learning model, such as ML model 302, may be provided to a machine learning compiler, such as ML compiler 218. For example, the ML model may have been generated using an ML composer, such as ML composer 214 and include ML library functions from ML library 216, or generated using other machine learning model generation tools and / or libraries / resources.

[0061] At block 1004, for all operations of the machine learning model, the machine learning compiler may generate compilable source code (e.g., compilable source code such as object-oriented code, other structure-oriented code, or functional code), such as Figure 6 6, which is configured to be compiled by a separate compiler such as compiler 221. In one or more implementations, the machine learning model may have been generated using a library of previously stored machine learning library functions (e.g., ML library 216). Generating the compilable source code (e.g., compilable source code 600) may include creating a set of machine learning kernels; and optimizing the generated compilable source code for one or more of the operations using a front end of a machine learning compiler. In one or more implementations, optimizing the generated compilable source code may include performing memory layout and alignment operations.

[0062] Optimizing the generated compilable source code for one or more of the operations may also include using the front end of the machine learning compiler to divide the one or more operations in the optimized operation into sub-operations, such as loops (e.g., a set of nested loops), determining instructions for optimizing the loops, and providing the loops and the instructions for optimizing the loops to a loop interface of the machine learning compiler (e.g., loop interface 802). The instructions for optimizing may include scheduling instructions. The loop interface uses the instructions to optimize the loops and may further operate a loop optimizer, such as loop optimizer 804, to generate compilable code for one or more of the loops (e.g., as part of the compilable source code 600).

[0063] The set of machine learning kernels may also be optimized. For example, in one or more implementations, optimizing the generated compilable source code for one or more of the operations and optimizing the machine learning kernels include balancing performance metrics and size metrics for the executable machine learning model. In one or more implementations, optimizing the machine learning kernels includes implementing a solution (e.g., a callback solution, a solution for notifying the loop interface 802 and / or the loop optimizer 804 what memory to reuse, a solution for providing the loop interface 802 and / or the loop optimizer 804 with additional buffers for such memory in advance, or other solutions) to limit memory usage, thereby preventing the executable machine learning model from allocating memory during execution time. In one or more implementations, optimizing the machine learning kernels includes removing runtime checks directed by the front end of the machine learning compiler.

[0064] At block 1006, the machine learning compiler may use compiled machine code generated by a separate compiler from the compilable source code to generate an executable machine learning model, such as executable ML model 220, that is executable in a zero-runtime operating environment. In one or more implementations, the executable machine learning model may include platform-specific executable code. For example, the executable machine learning model may be specific to be executed by specific hardware. In one or more implementations, the machine learning compiler may generate a machine learning model that includes a set of machine learning kernels (e.g., Figure 9 The scheduling instructions may include instructions for optimizing one or more loops in the loop.

[0065] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, personal information data can be used as input to a neural network or output from a neural network.

[0066] This disclosure contemplates that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information will adhere to established privacy policies and / or practices. Specifically, such entities are expected to implement and consistently apply privacy practices generally recognized as meeting or exceeding industry or government requirements for maintaining user privacy. Such information regarding the use of personal data should be prominently displayed and easily accessible to users and updated as the collection and / or use of data changes. Users' personal information should be collected only for lawful uses. Furthermore, such collection / sharing should occur only after receiving user consent or other lawful basis as provided in applicable law. Furthermore, such entities should consider taking any necessary steps to safeguard and secure access to such personal information and ensure that others with access to such personal information adhere to their privacy policies and procedures. Furthermore, such entities may subject themselves to third-party assessments to demonstrate compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be tailored to the specific types of personal information being collected and / or accessed and adapted to applicable laws and standards, including specific considerations specific to jurisdictions that may impose higher standards. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0067] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates providing hardware elements and / or software elements to prevent or block access to such personal information data. For example, in the case of inputs to or outputs from a neural network, the subject technology may be configured to allow a user to choose to "opt in" or "opt out" to participate in the collection and / or sharing of personal information data during registration for a service or at any time thereafter. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0068] Furthermore, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risk can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated where appropriate by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at a city level rather than at an address level or at a level that is insufficient for facial recognition), controlling how data is stored (e.g., aggregating data across users), and / or other methods such as differential privacy.

[0069] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, this disclosure also contemplates that various embodiments may be implemented without access to such personal information data. That is, various embodiments of the present technology will not be unable to function properly due to the lack of all or part of such personal information data.

[0070] Figure 11 An electronic system 1100 is shown that can be used to implement one or more implementations of the subject technology. The electronic system 1100 can be Figure 1 The electronic device 110 and / or server 120 are shown and / or may be part thereof. The electronic system 1100 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1100 includes a bus 1108, one or more processing units 1112, a system memory 1104 (and / or buffers), a ROM 1110, a permanent storage device 1102, an input device interface 1114, an output device interface 1106, and one or more network interfaces 1116, or subsets and variations thereof.

[0071] The bus 1108 generally represents all system, peripheral, and chipset buses that communicatively connect the many internal devices of the electronic system 1100. In one or more implementations, the bus 1108 communicatively connects the one or more processing units 1112 with the ROM 1110, the system memory 1104, and the permanent storage device 1102. The one or more processing units 1112 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed herein. In different implementations, the one or more processing units 1112 can be a single processor or a multi-core processor.

[0072] The ROM 1110 stores static data and instructions required by the one or more processing units 1112 and other modules of the electronic system 1100. On the other hand, the permanent storage device 1102 can be a read-write memory device. The permanent storage device 1102 can be a non-volatile memory unit that stores instructions and data even when the electronic system 1100 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) can be used as the permanent storage device 1102.

[0073] In one or more implementations, a removable storage device (such as a floppy disk, a flash drive, and its corresponding disk drive) may be used as the permanent storage device 1102. Like the permanent storage device 1102, the system memory 1104 may be a read-write memory device. However, unlike the permanent storage device 1102, the system memory 1104 may be a volatile read-write memory, such as random access memory. The system memory 1104 may store any of the instructions and data that one or more processing units 1112 may need during execution. In one or more implementations, the processes disclosed herein are stored in the system memory 1104, the permanent storage device 1102, and / or the ROM 1110. The one or more processing units 1112 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more implementations.

[0074] The bus 1108 is also connected to an input device interface 1114 and an output device interface 1106. The input device interface 1114 enables a user to transmit information and select commands to the electronic system 1100. Input devices that can be used with the input device interface 1114 may include, for example, an alphanumeric keyboard and a pointing device (also known as a "cursor control device"). The output device interface 1106 may enable, for example, the display of images generated by the electronic system 1100. Output devices that can be used with the output device interface 1106 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid-state display, a projector, or any other device for outputting information. One or more specific implementations may include a device that acts as both an input device and an output device, such as a touch screen. In these specific implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user may be received in any form, including acoustic, voice, or tactile input.

[0075] Finally, if Figure 11As shown, bus 1108 also couples electronic system 1100 to one or more networks and / or to one or more network nodes, such as a network interface 1116. Figure 1 . In this manner, electronic system 1100 can be part of a computer network, such as a LAN, a wide area network ("WAN"), or an intranet, or can be part of a network of networks, such as the Internet. Any or all components of electronic system 1100 can be used with the subject disclosure.

[0076] According to aspects of the present disclosure, a method is provided, comprising: providing a machine learning model to a machine learning compiler; generating, using the machine learning compiler, compilable source code for all operations of the machine learning model, the compilable source code being configured to be compiled by a separate compiler; and generating, using the machine learning compiler, an executable machine learning model using compiled machine code generated by the separate compiler from the compilable source code, the executable machine learning model being executable in a zero-runtime operating environment.

[0077] According to aspects of the present disclosure, a device is provided, comprising: a memory storing an executable machine learning model; and one or more processors configured to execute the executable machine learning model to generate a model output without accessing a runtime library.

[0078] According to aspects of the present disclosure, a non-transitory computer-readable medium storing instructions is provided that, when executed by one or more processors, causes the one or more processors to perform operations including: providing a machine learning model to a machine learning compiler; generating, using the machine learning compiler, compilable source code for all operations of the machine learning model, the compilable source code being configured to be compiled by a separate compiler, the compilable source code including code for only a subset of a runtime library used by the machine learning model; and generating, using the machine learning compiler, executable machine learning using compiled machine code generated by the separate compiler from the compilable source code.

[0079] Implementations within the scope of the present disclosure may be implemented in part or in whole using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) having one or more instructions programmed thereon. The tangible computer-readable storage medium may also be non-transitory in nature.

[0080] Computer-readable storage media can be any storage medium that can be read, written, or otherwise accessed by a general-purpose or special-purpose computing device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, and without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.

[0081] Furthermore, the computer-readable storage medium may include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more implementations, the tangible computer-readable storage medium may be directly coupled to the computing device, while in other implementations, the tangible computer-readable storage medium may be indirectly coupled to the computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0082] Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or can be implemented as high-level language instructions that can be compiled to produce executable or non-executable machine code. In addition, instructions can also be implemented as data, or can include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As those skilled in the art will appreciate, the details including but not limited to the number, structure, sequence and organization of instructions can be significantly different without changing the underlying logic, function, processing and output.

[0083] While the above discussion primarily relates to microprocessors or multi-core processors that execute software, one or more implementations are performed by one or more integrated circuits such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions stored on the circuits themselves.

[0084] Those skilled in the art will recognize that the various illustrative blocks, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative blocks, modules, elements, parts, methods and algorithms have been generally described above in terms of functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Technicians can implement the described functionality in different ways for each specific application. Various components and blocks can be arranged differently (e.g., arranged in different orders, or divided in different ways) without departing from the scope of the present subject technology.

[0085] Should be understood that the specific order or the hierarchical structure of the frames in the process disclosed by the present invention are illustrations of exemplary methods. Based on design preference requirements, it should be understood that the specific order or the hierarchical structure of the frames in the process can be rearranged or all the frames shown are executed. Any frame in these frames can be executed simultaneously. In one or more specific implementations, multitasking and parallel processing may be advantageous. In addition, the division of each system component in the above-mentioned specific implementation should not be understood as requiring this type of division in all specific implementations, and should be understood that program components and systems can generally be integrated together in a single software product or be packaged in multiple software products.

[0086] As used in this specification and any claims of this patent application, the terms "base station," "receiver," "computer," "server," "processor," and "memory" refer to electronic devices or other technical equipment. These terms exclude people or groups of people. For the purposes of this specification, the terms "display" or "displaying" mean displaying on an electronic device.

[0087] As used herein, the phrase "at least one of" following a list of items, any of which is separated by the terms "and" or "or," modifies the list as a whole, rather than each member (i.e., each item) of the list. The phrase "at least one of" does not require selection of at least one of each item listed; rather, the phrase allows for a meaning that includes at least one of any one item and / or at least one of any combination of items and / or at least one of each item. For example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" each refers to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0088] The predicate words "configured to," "operable to," and "programmed to" do not imply any specific tangible or intangible modification of a subject matter and are intended to be used interchangeably. In one or more implementations, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation or that the processor is operable to monitor and control the operation. Similarly, a processor configured to execute code may be interpreted as a processor that is programmed to execute code or operable to execute code.

[0089] Phrases such as aspect, this aspect, another aspect, some aspects, one or more aspects, an implementation, this implementation, another implementation, some implementations, one or more implementations, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, configuration, this configuration, other configurations, some configurations, one or more configurations, subject technology, disclosure, the present disclosure, other variations thereof, and the like are used for convenience and do not imply that disclosure involving such one or more phrases is essential to the subject technology or that such disclosure applies to all configurations of the subject technology. Disclosure involving such one or more phrases may apply to all configurations or one or more configurations. Disclosure involving such one or more phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to the other aforementioned phrases.

[0090] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the terms "including," "having," and the like are used in the specification or claims, such terms are intended to be inclusive, similar to the way the term "comprising" is interpreted when used as a transitional word in a claim.

[0091] All structural and functional equivalents to the elements of various aspects described throughout this disclosure that are known or later come to be known to one of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. In addition, nothing disclosed herein is intended to be made available to the public regardless of whether such disclosure is explicitly recited in the claims. No claim element shall be construed under 35 U.S.C. §112(f) unless the element is explicitly recited using the phrase “means for” or, in the case of a method claim, the phrase “step for”.

[0092] The previous description is provided to enable those skilled in the art to practice various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the present claims are not intended to be limited to the aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein reference to elements in singular values ​​is not intended to mean "only one", but refers to "one or more", unless specifically noted. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used only for convenience and do not limit this subject disclosure.

Claims

1. A method for generating an executable machine learning model, comprising: Provide the machine learning model to the machine learning compiler; generating, using the machine learning compiler, compilable source code for all operations of the machine learning model, the compilable source code being configured to be compiled by a separate compiler to generate compiled machine code, the compilable source code including code for a subset of a runtime library used by the machine learning model; as well as generating, with the machine learning compiler, an executable machine learning model using the compiled machine code generated by the separate compiler from the compilable source code, the executable machine learning model being executable in a zero runtime operating environment, wherein executing the executable machine learning model in the zero runtime operating environment comprises executing the executable machine learning model without accessing the runtime library.

2. The method of claim 1 , further comprising generating the machine learning model, wherein generating the compilable source code comprises: Create a set of machine learning kernels; as well as A front end of the machine learning compiler is utilized to optimize the generated compilable source code for one or more of the operations.

3. The method of claim 2 , further comprising optimizing the machine learning kernel, wherein optimizing the generated compilable source code for one or more of the operations and optimizing the machine learning kernel comprises balancing a performance metric and a size metric for the executable machine learning model.

4. The method of claim 2 , further comprising optimizing the machine learning kernel, wherein optimizing the machine learning kernel comprises implementing a callback-based solution to limit memory usage, thereby preventing the executable machine learning model from allocating memory during execution time.

5. The method of claim 2 , further comprising optimizing the machine learning kernel, wherein optimizing the machine learning kernel comprises removing runtime checks directed by the front end of the machine learning compiler.

6. The method of claim 2, wherein optimizing the generated compilable source code for the one or more of the operations comprises: separating, with the front end of the machine learning compiler, the one or more of the operations into loops; determining instructions for optimizing the loop; as well as The loop and the instructions for optimizing the loop are provided to a loop interface of the machine learning compiler. 7 . The method of claim 6 , further comprising operating a loop optimizer to optimize one or more of the loops using the loop interface and the instructions to optimize the loops. The method of claim 6 , wherein the instructions for optimizing comprise scheduling instructions.

9. The method of claim 2, wherein optimizing the generated compilable source code comprises performing memory layout and alignment operations.

10. The method of claim 1, wherein the executable machine learning model comprises platform-specific executable code.

11. The method of claim 1 , wherein the portion of the compilable source code comprises C code.

12. The method of claim 1, wherein the portion of the compilable source code comprises an intermediate representation code for the separate compiler.

13. The method of claim 1 further comprising executing the executable machine learning model in the zero runtime operating environment to generate a model output.

14. The method according to claim 1 also includes using the machine learning compiler to provide a configuration file tool to help users tune the machine learning model.

15. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: Provide the machine learning model to the machine learning compiler; generating, using the machine learning compiler, compilable source code for all operations of the machine learning model, the compilable source code being configured to be compiled by a separate compiler to generate compiled machine code, the compilable source code including code for a subset of a runtime library used by the machine learning model; and generating, with the machine learning compiler, an executable machine learning model using the compiled machine code generated by the separate compiler from the compilable source code, the executable machine learning model being executable in a zero runtime operating environment, wherein executing the executable machine learning model in the zero runtime operating environment comprises executing the executable machine learning model without accessing the runtime library.

16. The non-transitory computer-readable medium of claim 15, the operations further comprising generating the machine learning model, wherein generating the compilable source code comprises: Create a set of machine learning kernels; as well as A front end of the machine learning compiler is utilized to optimize the generated compilable source code for one or more of the operations.

17. The non-transitory computer-readable medium of claim 16, further comprising optimizing the machine learning kernel, wherein optimizing the machine learning kernel comprises implementing a callback-based solution to limit memory usage, thereby preventing the executable machine learning model from allocating memory during execution time.

18. The non-transitory computer-readable medium of claim 16, the operations further comprising optimizing the machine learning kernel, wherein optimizing the machine learning kernel comprises removing runtime checks directed by the front end of the machine learning compiler.

19. An electronic device comprising: A memory storing an executable machine learning model generated by the method according to any one of claims 1 to 14; and One or more processors configured to execute the executable machine learning model to generate a model output without accessing a runtime library.

20. The electronic device of claim 19, wherein the one or more processors comprises a low-power processor.

Images