Systems and methods for providing machine learning models with adjustable computing requirements

Abstract

A computing device is disclosed, the computing device comprising at least one processor and a machine learning model. The machine learning model may include a plurality of blocks and one or more residual connections between two or more of the plurality of blocks. The machine learning model may be configured to receive a model input and, in response to receiving the model input, output a model output. The machine learning model may be configured to perform operations, the operations comprising determining a resource allocation parameter corresponding to a desired allocation of system resources to the machine learning model at inference time; deactivating a subset of the plurality of blocks of the machine learning model based on the resource allocation parameter; inputting the model input into the machine learning model in which the subset of the plurality of blocks is deactivated; and receiving the model output as an output of the machine learning model.

Description

[0001] Related Applications

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 739,584, filed on October 1, 2018. The entire contents of U.S. Provisional Patent Application No. 62 / 739,584 are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to machine learning models. More specifically, the present disclosure relates to systems and methods for providing machine learning models with adjustable computing requirements. Background Art

[0004] On-device machine learning models have become more common. For example, deep neural networks have been deployed on "edge" devices such as mobile phones, embedded devices, other "smart" devices, or other resource-constrained environments. Such on-device models can provide benefits including reduced latency and improved privacy when compared to cloud-based configurations, in which machine learning models are stored and accessed remotely, for example, in a server accessed via a wide area network.

[0005] However, the computing resources of such edge devices can vary significantly. Furthermore, for a particular device, the amount of computing resources available at a given time for executing such an on-device machine learning model can vary based on various factors. As a result, on-device machine learning models may exhibit poor performance, such as increased latency or delay, and / or require suboptimal allocation of device resources. Summary of the invention

[0006] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description which follows, or may be learned from the description, or may be learned through practice of the embodiments.

[0007] An example aspect of the present disclosure is directed to a computing device. The computing device may include at least one processor and a machine learning model. The machine learning model may include a plurality of blocks and one or more residual connections between two or more of the plurality of blocks. The machine learning model may be configured to receive a model input and, in response to receiving the model input, output a model output. The machine learning model may include at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by at least one processor, cause the at least one processor to perform an operation. The operation may include determining a resource allocation parameter that corresponds to a desired allocation of system resources to the machine learning model at inference time. The operation may include deactivating a subset of the plurality of blocks of the machine learning model based on the resource allocation parameter. The operation may include inputting the model input into the machine learning model in which a subset of the plurality of blocks is deactivated, and receiving the model output as an output of the machine learning model.

[0008] Another example aspect of the present disclosure is directed to a computer-implemented method for reducing computational costs associated with a machine learning model. The method may include determining, by one or more computing devices, resource allocation parameters that describe a desired allocation of system resources to the machine learning model at inference time. The method may include deactivating, by one or more computing devices, a subset of a plurality of blocks of the machine learning model based on the resource allocation parameters. The method may include inputting, by one or more computing devices, an input set into the machine learning model, and receiving, by one or more computing devices, an output set as an output of the machine learning model.

[0009] Another example aspect of the present disclosure is directed to a method for training a machine learning model to be robust to deactivating at least some of a plurality of blocks of a neural network of the machine learning model at inference time. The method may include iteratively training the machine learning model using a training data set by one or more computing devices. The method may include deactivating, by one or more computing devices, an iteration-specific subset of a plurality of blocks of the machine learning model prior to at least one iteration of the iterative training of the machine learning model.

[0010] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The specification sets forth a detailed discussion of embodiments for those of ordinary skill in the art with reference to the accompanying drawings, in which:

[0013] Figure 1A A block diagram of an example computing system is depicted according to an example embodiment of the present disclosure.

[0014] Figure 1B A block diagram of an example computing system is depicted according to an example embodiment of the present disclosure.

[0015] Figure 1C A block diagram of an example computing system is depicted according to an example embodiment of the present disclosure.

[0016] Figure 2A Depicts a machine learning model according to an example embodiment of the present disclosure.

[0017] Figure 2B Depicted is a machine learning object recognition model according to an example embodiment of the present disclosure.

[0018] Figure 2C Depicts a machine learning speech recognition model according to an example embodiment of the present disclosure.

[0019] Figure 3A An example block depicting the layers of a machine learning model according to an example embodiment of the present disclosure.

[0020] Figure 3B Depicting another example block of layers of a machine learning model according to an example embodiment of the present disclosure.

[0021] Figure 4 Depicting an example machine learning model in which a subset of multiple blocks have been deactivated according to an example embodiment of the present disclosure.

[0022] Figure 5 Depicted is another example machine learning model in which a subset of multiple blocks have been deactivated according to an example embodiment of the present disclosure.

[0023] Figure 6 A flowchart depicting an example method of reducing computational costs associated with a machine learning model according to an example embodiment of the present disclosure.

[0024] Figure 7 A flow chart depicting an example method for training a machine learning model to be robust to deactivating at least some of a plurality of blocks of a neural network of the machine learning model at inference time.

[0025] Repeated reference numerals in the various drawings are intended to identify like features in the various embodiments. DETAILED DESCRIPTION

[0026] Overview

[0027] In general, the present disclosure is directed to systems and methods for providing machine learning models with adjustable computing requirements. Example aspects of the present disclosure are directed to computing systems and related methods that include or in addition utilize machine learning models that can be adjusted to adjust the computing requirements of executing machine learning models on computing devices. In some embodiments, the machine learning model can be stored and / or executed on a computing device (such as an "edge" device). Example devices include smartphones, "smart" devices, embedded devices, and any computing device that can have limited computing power and / or access to cloud computing. Before reasoning time, the computing device can select a subset of the machine learning model (e.g., a set of one or more layers and / or blocks of layers) based on resource allocation parameters that correspond to the desired allocation of system resources to the machine learning model at reasoning time. The computing device can deactivate a subset of the selected model in a manner that reduces or eliminates the computing requirements associated with the deactivated portion. As a result, the total computing requirements for the computing device at reasoning time can be reduced. More specifically, the total computational demand on the computing device at inference time can be intelligently controlled to match a desired allocation of system resources, where the desired allocation is based on various environmental and / or contextual factors (such as, one or more metrics associated with the available processing power or memory of the device at inference time) and / or based on user input. While deactivating portions of a machine learning model can provide faster processing times, such deactivation can also reduce the quality of the results output by the model. Therefore, a trade-off between processing time and quality often exists. Therefore, aspects of the present disclosure provide an adjustable model that can be intelligently and granularly adjusted to provide a desired trade-off between speed and quality.

[0028] A machine learning model can be trained to be robust to deactivating blocks at inference time. More specifically, during iterative training of the model, an iteration-specific subset of blocks can be deactivated. The subset of blocks that are deactivated during training can be selected in a similar manner to those blocks that are deactivated at inference time. Thus, the model can be trained to be robust to the deactivation of blocks that may be deactivated at inference time, which can improve the quality of the output of the machine learning model at inference time.

[0029] In addition, such an adjustable machine learning model can be suitable for deployment across a range of computing devices with various resource levels. Each computing device can adjust the machine learning model as needed (e.g., based on the resources of the corresponding computing device). Alternatively, a single machine learning model can be trained, and then various adjusted machine learning models can be created and distributed based on the single, trained machine learning model. The various adjusted machine learning models may require different levels of computing resources at inference time. Therefore, a machine learning model according to aspects of the present disclosure can be adjusted or customized according to the specific computing device on which the machine learning model will be executed.

[0030] In one example, a user may request an operation (such as object recognition) that utilizes an on-device machine learning model resident on a smartphone. Prior to executing the machine learning model, the computing device may, for example, deactivate portions of the machine learning model based on context-specific considerations associated with the smartphone to reduce the computing resources required to execute the machine learning model. Examples include the battery state of the smartphone at inference time, the currently available processor power, and the number of currently executed applications. Such adjustments to the machine learning model may reduce the time required to execute the machine learning model and provide output (e.g., recognized text) to the user.

[0031] Specifically, according to one aspect of the present disclosure, a computing device may include a machine learning model including a plurality of blocks. Each block may include one or more layers, and each layer may include one or more nodes. For example, in some embodiments, the machine learning model may be or include a convolutional neural network. The machine learning model may include one or more residual connections between two or more blocks in the plurality of blocks. The residual connection may be configured to pass information to a "downstream" block, for example, by bypassing a disabled block. Thus, the model may include any number of blocks and any number of residual connections between the blocks. In one example, there is a residual connection between each adjacent block, while in other examples, the residual connection is sparse.

[0032] The computing device can be configured to determine a resource allocation parameter that corresponds to a desired allocation of system resources to the machine learning model at inference time. The computing device can deactivate a subset of the multiple blocks of the machine learning model based on the resource allocation parameter. As a result, the computational requirements associated with executing the resulting machine learning model can be reduced. For example, the reduction in the computational requirements associated with executing the machine learning model can be inversely proportional to the magnitude of the resource allocation parameter. Finally, the computing device can be configured to input the model input into the machine learning model in which the subset of the multiple blocks is deactivated, and receive the model output as the output of the machine learning model.

[0033] In some embodiments, resource allocation parameters can be determined before the model inputs are input to the machine learning model (e.g., before inference time). Resource allocation parameters can also be determined based on the current state of the computing device. As an example, when the battery state of the computing device is low, the resource allocation parameters can correspond to a low expected allocation of system resources to the machine learning model at inference time to preserve the remaining battery power of the computing device. Similarly, when the currently available processor power is low and / or a large number of applications are currently executing, the resource allocation parameters can correspond to a low expected allocation of system resources to avoid long processing times. Therefore, the resulting machine learning model can be adjusted based on the current state of the computing device to quickly provide a solution and / or preserve the resources of the computing device.

[0034] In some embodiments, a user can provide input indicating the amount of computing resources the user would like to allocate to the machine learning model. As an example, the user can interact with a touch-sensitive display screen of a computing device (e.g., a smart phone) to provide input. For example, the user can enter a value (e.g., via a keyboard) or adjust a slider (or other virtual control object) to indicate her preference for faster results but potentially less accurate results or slower but potentially more accurate results.

[0035] In some embodiments, a machine learning model can have a structural arrangement that provides resilience or robustness to deactivating various blocks. More specifically, the blocks can be connected by various residual connections so that information can be passed "bypassing" the deactivated blocks to subsequent blocks or layers (e.g., to a classification or other (multiple) output layers). As an example, a subset of multiple blocks can be selected so that at least one of the residual connections bypasses each block included in the subset of multiple blocks. In other words, each block included in the subset of multiple blocks can be located between at least one of the residual connections of the machine learning model. Therefore, deactivating a subset of blocks can reduce the computational requirements associated with executing the machine learning model without rendering the machine learning model inoperable or unacceptably reducing the quality of the model's output.

[0036] The residual connections of a machine learning model can have various configurations. As an example, multiple blocks can be "densely" connected so that a residual connection is provided from the output of each block to the input of the block directly following the next sequential block, so that each block is residually connected to a block outside of one block (for example, the connection from block 1 to block 3 skips block 2). In such an example, each block is eligible for deactivation. As another example, residual connections can be formed only between some blocks. Each residual connection can skip one block or can skip multiple blocks. Residual connections can have different connections and skip amounts in the network. In such a configuration, only blocks for which residual connections are provided (for example, blocks that can be skipped by residual connections) can be eligible for deactivation. However, aspects of the present disclosure can be applied to machine learning models with different suitable configurations of any number of residual connections.

[0037] As used herein, a "block" may refer to a group of one or more consecutive layers, and each layer may include one or more nodes. In some embodiments, the layers within a block may be arranged in a general sequential configuration in which the output of one layer is passed to the next layer as input. In some embodiments, the machine learning model may include a convolutional neural network, and at least one of the multiple blocks may include a convolutional block. The convolutional block may apply at least one convolutional filter. The convolutional block may also include one or more pooling layers or other suitable layers found in a convolutional neural network. Additional residual connections may be included within the convolutional block, for example, the residual connection bypasses one or more of the convolutional filters. In addition, in some embodiments, the machine learning model may include one or more fully connected layers and / or classification layers, such as a softmax layer.

[0038] Various suitable techniques may be used to deactivate a subset of blocks. As an example, a subset of blocks may be disconnected so that information is not input into the subset of blocks. However, deactivating a subset of blocks may include any suitable technique such that consumption of computational resources for the subset of blocks at inference time is significantly reduced or eliminated.

[0039] In some embodiments, the size of the subset of the plurality of blocks may be selected based on the magnitude of the resource allocation parameter such that the size of the subset of the plurality of blocks is negatively correlated with the magnitude of the resource allocation parameter. For example, a small resource allocation parameter may result in a large number of blocks being deactivated before inference time. However, in other embodiments, depending on the convention selected, the size of the subset of the plurality of blocks may be positively correlated with the magnitude of the resource parameter.

[0040] In some embodiments, multiple blocks (or subsets thereof) may be residually connected in a "residual chain" that may extend from the input of the machine learning model to the output of the machine learning model. Deactivating a subset of multiple blocks of the machine learning model may include deactivating a starting residual block within the residual chain and a residual tail portion of the residual chain. The residual tail portion may include blocks within the residual chain extending from the starting residual block to the output of the machine learning model. Thus, the residual tail portion may include a continuous string of blocks that are "after" the starting residual block. When the residual tail portion is disabled, the residual connection may pass information to subsequent layers or blocks (e.g., "bypassing" the disabled portion). Thus, the subset of blocks may include a continuous chain of blocks within multiple blocks of the machine learning model.

[0041] In some embodiments, a subset of blocks can be selected in a semi-random manner that prefers to deactivate blocks located near the output of the model relative to deactivating blocks located near the input of the machine learning model. For example, blocks can be selected based on the corresponding probabilities of each block. These probabilities can be assigned to the blocks and can correspond to the likelihood that each block is selected for deactivation. The corresponding probability of each block can be positively correlated with the corresponding position of each block in the neural network. More specifically, blocks located near the input of the machine learning model can have low associated probabilities. The probabilities associated with the corresponding blocks can increase toward the output. Therefore, the subset of blocks can include non-contiguous blocks that can be dispersed among multiple blocks of the machine learning model.

[0042] According to another aspect of the present disclosure, a method for reducing computational costs associated with a machine learning model is disclosed. The method may include determining a resource allocation parameter that describes a desired allocation of system resources to the machine learning model at inference time. A subset of a plurality of blocks of the machine learning model may be deactivated based on the resource allocation parameter. The method may include inputting an input set into the machine learning model and receiving an output set as an output of the machine learning model.

[0043] In some embodiments, the method may include receiving a machine learning model at a user computing device (e.g., an "edge" device) after deactivating a subset of the plurality of blocks (e.g., at a server computing system). The machine learning model may be trained (e.g., at a server computing system). During training, portions of the machine learning model may be deactivated, e.g., as described below, so that the machine learning model is robust to deactivated blocks. Before the trained machine learning model is sent to the user computing device, resource allocation parameters may be determined (e.g., at the server computing system), for example, based on computing resources of the user computing device to which the machine learning model is to be sent. A subset of the plurality of blocks of the machine learning model may then be deactivated (e.g., at the server computing system) based on the resource allocation parameters, and the adjusted machine learning model may be sent to the user computing system. The user computing system may then utilize the machine learning model by inputting the model inputs and receiving the model outputs as outputs of the machine learning model.

[0044] Such an implementation may provide for more efficient training and distribution of machine learning models. Instead of training a variety of similar machine learning models of varying complexity for an array of devices with varying levels of computing resources, a single machine learning model may be trained. Then, by deactivating portions of the trained machine learning model, multiple copies of the trained machine learning model may then be adjusted to require different levels of computing resources (e.g., corresponding to various devices). The resulting machine learning model may then be distributed to an array of devices with varying levels of computing resources. However, it should be understood that in other implementations, each step of the above method may be performed by a single user computing device.

[0045] In some embodiments, for example, as described above, a subset of the plurality of blocks may be selected such that at least one of the residual connections bypasses each of the subset of the plurality of blocks. Similarly, in some embodiments, for example, as described above, a resource allocation parameter may be determined based on at least one of a battery state, currently available processor power, or a number of currently executed applications.

[0046] According to another aspect of the present disclosure, a method is disclosed for training a machine learning model to be robust to at least some of a plurality of blocks of a neural network that deactivates the machine learning model at inference time. The method may include iteratively training the machine learning model using a training data set. The training data set may include any suitable training input data set, and may optionally include a training output data set. An example training data set may include an audio file and recognized text of spoken words in the audio file. Another example training data set may include an image and object recognition output describing the location and / or labels of the recognized objects depicted in the image.

[0047] The method may include deactivating an iteration-specific subset of multiple blocks of the machine learning model before at least one iteration of iterative training of the machine learning model. The iteration-specific subset may be selected in various suitable ways. As an example, the iteration-specific subset may be selected based on a corresponding probability associated with each block that is positively correlated with the corresponding position of each block within the neural network. It should be understood that the iteration-specific subset may be selected using a method similar to the method described above regarding deactivating a subset of multiple blocks based on a resource allocation parameter before inference time. For example, during training, a subset of iteration-specific blocks may be selected in a semi-random manner that prefers to deactivate blocks located near the output end of the machine learning model relative to deactivating blocks located near the input end of the machine learning model. Therefore, the machine learning model can be trained to be robust to deactivating blocks that may be deactivated at inference time. This training method can improve the quality of the output of the machine learning model at inference time.

[0048] The systems and methods of the present disclosure provide many technical effects and benefits, including, for example, reducing the computational resources required by a machine learning model at inference time. In addition, where variants of a machine learning model with different computational requirements are desired, the storage of the machine learning model on the device can be reduced to an adjustable machine learning model that can adjust its computational requirements and can replace multiple machine learning models. The described systems and methods can also reduce the computational resources required to train a machine learning model, because an adjustable machine learning model can be trained to replace multiple machine learning models with different computational requirements.

[0049] As an example, the systems and methods of the present disclosure may be included in or otherwise employed in the context of an application, a browser plug-in, or in other contexts. Thus, in some embodiments, the models of the present disclosure may be included in or otherwise stored and implemented by a user computing device (such as a laptop, tablet, or smart phone). As yet another example, the models may be included in or otherwise stored by a server computing device that communicates with the user computing device in accordance with a client-server relationship.

[0050] Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.

[0051] Example devices and systems

[0052] Figure 1AA block diagram depicts an example computing system 100 that reduces computational costs associated with machine learning models according to an example embodiment of the present disclosure. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0053] In some implementations, the user computing device 102 may be an "edge" device. Example "edge" devices include smartphones, "smart" devices, embedded devices, and any computing device that may have limited computing capabilities and / or access to cloud computing. However, the user computing device 102 may be any type of computing device, such as a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0054] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0055] The user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may include, among other things, various machine learning models, such as a neural network (e.g., a deep neural network) or other multi-layer nonlinear model. The neural network may include a convolutional neural network, a recurrent neural network (e.g., a long-short term memory recurrent neural network), a feedforward neural network, or other forms of neural networks. Figure 2A to Figure 2C Discuss example machine learning model 120.

[0056] In some implementations, one or more machine learning models 120 may be received from a server computing system 130 via a network 180, stored in a user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, a user computing device 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., perform parallel operations across multiple instances of a model 120).

[0057] Additionally or alternatively, the one or more machine learning models 140 may be included in or otherwise stored by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. Thus, they may be stored and implemented at the user computing device 102. In some implementations, the one or more models 120 may be sent from the server computing system 130 to the user computing device 102.

[0058] The user computing device 102 may also include a model controller 122, for example, as described in reference Figures 2A to 7 As described, the model controller 122 is configured to deactivate a subset of the plurality of blocks of the machine learning model 120. The model controller 122 may correspond to a computer program (e.g., data 116 and instructions 118) configured to deactivate blocks as described herein.

[0059] The user computing device 102 may also include one or more user input components 124 that receive user input. For example, the user input component 124 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user may input communications.

[0060] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0061] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0062] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the model 140 may be or may otherwise include various machine learning models, such as a neural network (e.g., a convolutional neural network, a deep recurrent neural network, etc.) or other multi-layer nonlinear model. Figure 2A to Figure 2C Example model 140 is discussed.

[0063] Server computing system 130 may train model 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130, or may be a part of server computing system 130.

[0064] The server computing system 130 may also include a model controller 142, for example, as described in reference Figures 2A to 7 As described, the model controller 142 is configured to deactivate a subset of the plurality of blocks of the machine learning model 140. The model controller 122 may correspond to a computer program (e.g., data 116 and instructions 118) configured to deactivate blocks as described herein.

[0065] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be one processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes one or more server computing devices, or is otherwise implemented by one or more server computing devices.

[0066] The training computing system 150 may include a model trainer 160 that trains the machine learning model 140 stored at the server computing system 130 using various training or learning techniques. For example, as described below with reference to Figure 7 As described, the model trainer 160 can be configured to deactivate an iteration-specific subset of multiple blocks of the machine learning model before at least one iteration of iterative training of the machine learning model. Example training techniques include back propagation of errors, which can include performing truncated back propagation through time. The model trainer 160 can perform many generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0067] Specifically, the model trainer 160 can train the machine learning model 140 based on the training data 162 set. For example, as shown below Figure 2A to Figure 2C As described above, the training data 162 may include, for example, model input data (e.g., images, audio, etc.). In some embodiments, for example, as described below with reference to Figure 2A to Figure 2C As described above, the training data 162 may be “labeled” and also include model output data (eg, object recognition data, speech recognition data, etc.).

[0068] In some implementations, if the user has provided permission, training examples may be provided by the user computing device 102 (e.g., based on communications previously provided by the user of the user computing device 102). Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 on user-specific communication data received from the user computing device 102. In some instances, this process may be referred to as personalizing the model.

[0069] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some embodiments, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium (such as a RAM hard disk or an optical or magnetic medium).

[0070] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications through the network 180 may be performed via any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0071] Figure 1A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some embodiments, the user computing device 102 may include a model trainer 160 and a training data set 162. In such embodiments, the model 120 may be trained and used locally at the user computing device 102. In some of such embodiments, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0072] Figure 1B Depicted is a block diagram of an example computing device 10 according to an example embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0073] Computing device 10 includes many applications (e.g., applications 1 to N). Each application includes its own machine learning library and (multiple) machine learning models. For example, each application can include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some embodiments, each application can also include its own model controller, which can intelligently deactivate blocks within the corresponding machine learning model included in the application.

[0074] like Figure 1B As shown, each application can communicate with many other components of the computing device (such as one or more sensors, context managers, device state components, and / or additional components). In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to the application.

[0075] Figure 1C Depicted is a block diagram of an example computing device 50 according to an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0076] The computing device 50 includes a number of applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some embodiments, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a public API across all applications).

[0077] The central intelligence layer includes many machine learning models. For example, Figure 1C As shown, a corresponding machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included in the operating system of the computing device 50 or is implemented by the operating system of the computing device 50 in addition thereto.

[0078] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized repository for data of computing devices 50. Figure 1CAs shown, the central device data layer can communicate with many other components of the computing device (such as one or more sensors, context managers, device state components, and / or additional components). In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API). In some embodiments, the central intelligence layer can also include a model controller that intelligently deactivates blocks in the machine learning model provided for a given application via the API.

[0079] Example Model Layout

[0080] Figure 2A A block diagram depicts an example machine learning model 200 according to an example embodiment of the present disclosure. In some implementations, the machine learning model 200 is trained to receive a set of input data 202 and, as a result of receiving the input data 202, provide output data 204. The machine learning model can be trained to perform a wide variety of functions. As an example, refer to Figure 2B , object recognition machine learning model 240 may receive one or more images 242 (e.g., a single image or a video file) and output object recognition data 244 (e.g., describing the location and / or label of the recognized objects depicted in the image). As another example, referring to Figure 2C , speech recognition machine learning model 260 can be trained to receive audio file 262 and output speech recognition data 264 (e.g., recognized text including spoken words in the audio file). However, it should be understood that aspects of the present disclosure can be applied to machine learning model 200 configured to perform any suitable functions, including functions performed by an on-device machine learning model (e.g., text prediction, personal assistant functions, etc.).

[0081] In general, the present disclosure is directed to systems and methods for providing a machine learning model 200 with adjustable computing requirements. Example aspects of the present disclosure are directed to computing systems and related methods that include or in addition utilize a machine learning model that can be adjusted to adjust the computing requirements of executing the machine learning model on a computing device. In some embodiments, the machine learning model can be stored and / or executed on a computing device (such as an "edge" device). Example devices include smartphones, "smart" devices, embedded devices, and any computing device that may have limited computing power and / or access to cloud computing. Before reasoning time, the computing device can select a subset of the machine learning model (e.g., a set of one or more layers and / or blocks of layers) based on resource allocation parameters that correspond to the desired allocation of system resources to the machine learning model at reasoning time. The computing device can deactivate the selected model subset in a manner that reduces or eliminates the computing requirements associated with the deactivated portion. As a result, the total computing requirements for the computing device at reasoning time can be reduced. More specifically, the total computational demand on the computing device at inference time can be intelligently controlled to match a desired allocation of system resources, where the desired allocation is based on various environmental and / or contextual factors (such as, for example, one or more metrics associated with the available processing power or memory of the device at inference time) and / or based on user input. While deactivating portions of a machine learning model can provide faster processing times, such deactivation can also reduce the quality of the results output by the model. Therefore, a trade-off between processing time and quality often exists. Accordingly, aspects of the present disclosure provide an adjustability model that can be intelligently and granularly adjusted to provide a desired trade-off between speed and quality.

[0082] A machine learning model can be trained to be robust against deactivating blocks at inference time. More specifically, during iterative training of the model, an iteration-specific subset of blocks can be deactivated. The subset of blocks that are deactivated during training can be selected in a similar manner to those blocks that are deactivated at inference time. As a result, the model can be trained to be robust against deactivating blocks that may be deactivated at inference time, which can improve the quality of the output of the machine learning model at inference time.

[0083] In addition, such an adjustable machine learning model can be suitable for deployment across a range of computing devices with various resource levels. Each computing device can adjust the machine learning model as needed, for example, based on the resources of the corresponding computing device. Alternatively, a single machine learning model can be trained, and then various adjusted machine learning models can be created and distributed based on the single, trained machine learning model. The various adjusted machine learning models may require different levels of computing resources at inference time. Therefore, the machine learning model according to aspects of the present disclosure can be adjusted or customized according to the specific computing device on which the machine learning model will be executed.

[0084] In one example, a user may request an operation (such as object recognition) that utilizes an on-device machine learning model resident on a smartphone. Prior to executing the machine learning model, the computing device may, for example, deactivate portions of the machine learning model based on context-specific considerations associated with the smartphone to reduce the computing resources required to execute the machine learning model. Examples include the battery state of the smartphone at inference time, the currently available processor power, and the number of currently executed applications. Such adjustments to the machine learning model may reduce the time required to execute the machine learning model and provide output (e.g., recognized text) to the user.

[0085] As used herein, a "block" may refer to a group of one or more consecutive layers, and each layer may include one or more nodes. In some embodiments, the layers within a block may be arranged in a general sequential configuration in which the output of one layer is passed to the next layer as input. In some embodiments, the machine learning model may include a convolutional neural network, and at least one of the multiple blocks may include a convolutional block. The convolutional block may apply at least one convolutional filter. The convolutional block may also include one or more pooling layers or other suitable layers found in a convolutional neural network. Additional residual connections may be included within the convolutional block, for example, the additional residual connections bypass one or more of the convolutional filters. In addition, in some embodiments, the machine learning model may include one or more fully connected layers and / or classification layers, such as a softmax layer.

[0086] For example, refer to Figure 3A , showing an example block 300. The blocks included in the model described herein may be as shown in the example block 300, and / or may have other configurations / structures different from the example block 300. The block 300 may have an input 302, an output 304, and include one or more layers 306, 308. Each layer 306, 308 may include one or more nodes. The residual connection 310 may be configured to pass information from the input 302 to the output 304 of the block 300 (e.g., "bypassing" the layers 306, 308 of the block 300). In some embodiments, one or more "internal" residual connections 312 may be provided, and the residual connection 312 passes information around fewer blocks than all of the layers 306, 308 of the block 300. For example, the internal residual connection 312 may pass information around the layer 306 or around the layer 308.

[0087] Figure 3BAnother example block 350 is shown that includes a "parallel" structure. For example, the block 350 may include multiple layers 354 configured as convolutional filters or other types of parallel neural network structures. One or more residual connections 356 may be configured to pass information from the input 352 to the output 360 of the block 350. In addition, an "internal" residual connection 358 may be configured to pass information around one or more individual layers 354 of the block 350.

[0088] Figure 4 An example machine learning model 400 is shown having an input 402, an output 404, and a plurality of blocks 405. The machine learning model 400 may be similar to the one described above with reference to Figure 2A to Figure 2C The machine learning model 400 may also include one or more fully connected layers and / or classification layers 426, such as a softmax layer. The blocks 405 may generally be arranged in a linear configuration, with the output of the first block 406 being input to the second block 408, and the output of the second block 408 being input to the third block 410, and so on. The output of the last block 424 may be input to the classification layer 426. The output of the classification layer 426 may correspond to the output 404 of the model 400.

[0089] The machine learning model 400 may include one or more residual connections 427 between two or more blocks in the plurality of blocks 406-424. The residual connection 427 may be configured to pass information to a "downstream" block, for example, by bypassing certain blocks. Thus, the model may include any number of blocks and any number of residual connections 427 between various blocks 405.

[0090] Computing devices (e.g. Figure 1A The device 102 or system 130 shown in FIG. 4 may be configured to determine a resource allocation parameter that corresponds to a desired allocation of system resources for the machine learning model 400 at inference time. The computing device may deactivate a subset of the plurality of blocks 405 of the machine learning model 400 based on the resource allocation parameter. Figure 4 In the example shown, the subset of the plurality of blocks 405 includes blocks 414, 418, 422, 424. As a result, the computational requirements associated with executing the resulting machine learning model 400 can be reduced. For example, the reduction in the computational requirements associated with executing the machine learning model 400 can be inversely proportional to the magnitude of the resource allocation parameter. Finally, the computing device can be configured to input the model input 402 to the machine learning model 400 in which a subset of the plurality of blocks 405 (e.g., blocks 414, 418, 422, 424) is disabled, and receive the model output 404 as an output of the machine learning model.

[0091] In some embodiments, the computing device may select blocks for deactivation based at least in part on the amount of computing resources associated with executing each corresponding block in the machine learning model 400. More specifically, the resource allocation parameter may correspond to the maximum target resource allocation for the machine learning model. As an example, the maximum target resource allocation may correspond to a percentage of total processor power, a percentage of total available memory (e.g., random access memory), or any other suitable metric describing the use of computing resources. The corresponding resource consumption associated with each block may vary, for example, based on the number and / or type of layers and / or nodes in each block. The computing device may be configured to select blocks based on the corresponding resource consumption of each block (e.g., expected computing requirements) to adjust the resources required to execute the machine learning model to less than the total target resource allocation. Therefore, the computing device may be configured to intelligently select deactivated blocks so that the resulting model 400 requires resources less than the maximum target resource allocation at inference time.

[0092] For example, given the knowledge of the corresponding resource consumption amount and the total target resource allocation associated with each block, the computing device can perform one or more optimization techniques to select which blocks to deactivate. For example, the computing device can minimize the difference between the implemented resource allocation and the target resource allocation by iteratively searching for the combination of blocks to be deactivated. In another example, a lookup table associating resource allocation parameter values ​​with corresponding subsets of blocks to be deactivated can be stored in the memory of the computing device and accessed at inference time to determine the blocks to be deactivated. In yet another example, a variable number of "final" blocks in the model can be simply deactivated, where the variable number is a function of the resource allocation parameter. Thus, for example, as the resource allocation parameter increases, an increasing number of "final" blocks in the model can be deactivated.

[0093] Still reference Figure 4 , to provide an example, blocks 406-416 may have respective resource consumptions that are each equal to 10 units. Blocks 418-424 may have respective resource consumptions that are equal to 20 units. In this example, if the total target resource allocation is equal to 75 units, the computing device may select shaded blocks 414, 418, 422, 424. The sum of the respective resource consumptions of the remaining blocks 406-412, 416, 420 is equal to 70, which is less than the total target resource allocation of 75 units.

[0094] In some embodiments, resource allocation parameters may be determined before the model input 402 is input to the machine learning model 400 (e.g., before inference time). Resource allocation parameters may also be determined based on the current state of the computing device. As an example, when the battery state of the computing device is low, the resource allocation parameters may correspond to a low expected allocation of system resources to the machine learning model 400 at inference time to maintain the remaining battery power of the computing device. As another example, the current state of the computing device may include a mode of the computing device (e.g., a battery saving mode). When the computing device is in battery saving mode, the resource allocation parameters may correspond to a low expected allocation of system resources to the machine learning model 400 at inference time to reduce the battery power used at inference time. Similarly, when the currently available processor power is low and / or a large number of applications are currently executing, the resource allocation parameters may correspond to a low expected allocation of system resources to avoid long processing times. Therefore, the resulting machine learning model 400 may be adjusted based on the current state of the computing device to quickly provide a solution and / or maintain the resources of the computing device.

[0095] In some implementations, a user may provide input indicating the amount of computing resources the user would like to allocate to the machine learning model 400. As an example, refer to Figure 1A , a user may interact with a user input component 124 (e.g., a touch-sensitive display) of a computing device 102 (e.g., a smartphone) to provide input. For example, a user may enter a value (e.g., via a keyboard) or adjust a slider (or other virtual control object) to indicate her preference for a faster result but potentially less accurate result or a slower but potentially more accurate result.

[0096] Reference again Figure 4 In some embodiments, the machine learning model 400 can have a structural arrangement that provides resilience or robustness to the deactivation of various blocks. More specifically, the blocks of the model 400 can be connected by various residual connections 427 so that information can be passed to subsequent blocks or layers (e.g., to the classification layer 524 or other (multiple) output layers) "bypassing" the deactivated blocks 414, 418, 422, 424. As an example, a subset of the plurality of blocks 405 can be selected so that at least one residual connection in the residual connection 427 bypasses each block 414, 418, 422, 424 included in the subset of the plurality of blocks 405. In other words, each of the blocks 414, 418, 422, 424 included in the subset of the plurality of blocks 405 can be located between at least one of the residual connections 427 of the machine learning model. Thus, deactivating a subset of blocks 414 , 418 , 422 , 424 can reduce the computational requirements associated with executing the machine learning model 400 without rendering the machine learning model inoperable or unacceptably degrading the quality of the output of the model 400 .

[0097] The residual connections of the machine learning model 400 can have various configurations. For example, Figure 4 As shown, residual connections 427 may be formed only between some of the blocks 405. Figure 4 As shown, each residual connection 427 can skip a single block. In other embodiments, the residual connection can skip multiple blocks. The residual connection can have different connections and skip amounts in the network. In such a configuration, only the blocks for which the residual connection is provided (e.g., blocks that can be skipped by the residual connection) can be eligible for deactivation. However, aspects of the present disclosure can be applied to machine learning models with different suitable configurations of any number of residual connections.

[0098] For example, refer again to Figure 4 , residual connections 427 may be formed only between blocks 412-424. Other blocks 406, 408, 410 may not have associated residual connections. In this example, only blocks 412-424 with associated residual connections 427 may be eligible for deactivation.

[0099] In some embodiments, a subset of blocks 405 can be selected in a semi-random manner that favors deactivating blocks located near the output 428 (near the output 404) of the machine learning model 400 over deactivating blocks located near the input 430 (near the input 402) of the machine learning model 400. For example, some of the blocks 405 can be selected based on the corresponding probabilities of each block. These probabilities can be assigned to the blocks and can correspond to the likelihood that each block is selected for deactivation. The corresponding probability of each of the blocks 405 can be positively correlated with the corresponding position of each of the blocks 405 in the neural network. More specifically, blocks located near the input 430 of the machine learning model 400 can have low associated probabilities. The probabilities associated with the corresponding blocks 405 can increase toward the output 428. Thus, for example, Figure 4 As shown, a subset of blocks 405 may include non-contiguous blocks 414, 418, 422, 424 that may be dispersed among multiple blocks 405 of the machine learning model 400.

[0100] For example, refer to Figure 4 , blocks 412, 414, 416 may have relatively low respective associated probabilities (e.g., less than 25%); blocks 418, 420 may have medium respective associated probabilities (e.g., 25% to 50%); and blocks 422, 424 may have high respective associated probabilities (e.g., greater than 50%). The computing device may then select blocks for deactivation based on their respective probabilities. As a result, for example, Figure 4As shown, blocks selected for deactivation may be funneled toward output 428 of machine learning model 400 .

[0101] refer to Figure 5 In some embodiments, multiple blocks 503 may be "densely" connected such that residual connections 528, 529 are provided from the output of each block to the input of the block that directly follows the next sequential block, such that each block is residually connected to a block other than one block (e.g., the connection from the output of the first block 504 to the input of the third block 508 skips the second block 506). In such an example, each of the blocks 503 may be eligible for deactivation.

[0102] However, as indicated above, a subset of the plurality of blocks 503 may be selected such that at least one of the residual connections 528, 529 bypasses each block 512, 530 included in the subset of the plurality of blocks 503. In other words, each block 512, 530 included in the subset of the plurality of blocks 503 may be located between at least one of the residual connections 528, 529 of the machine learning model 500. Thus, the eligibility of various blocks for deactivation may depend on the arrangement of the residual connections 528, 529. For example, referring to Figure 5 , if all residual connections 528, 529 shown are present, each of the blocks 503 will be eligible for deactivation in any combination. However, if the residual connection 529 shown in dashed lines is not present, only the shaded blocks 512, 530 will be eligible for deactivation.

[0103] In some embodiments, multiple blocks (or subsets thereof) can be residually connected in a "residual chain" (e.g., block 503), which can extend from an input 532 of the machine learning model 500 to an output 534 of the machine learning model 500. Deactivating a subset of multiple blocks of the machine learning model 500 can include deactivating a starting residual block 512 within the residual chain and a residual tail portion 530 of the residual chain. The residual tail portion 530 can include blocks (e.g., blocks 514-522) that are within the residual chain and extend from the starting residual block 512 to the output 534 of the machine learning model 500. Therefore, the residual tail portion 530 can include a continuous string of blocks 514-522 that are located "after" the starting residual block 512. When the residual tail portion 530 is disabled, the residual connection 528 can pass information to a subsequent layer or block (e.g., to the classification layer 524) (e.g., "bypassing" the disabled portion 512, 530). Thus, the subset of blocks 512, 530 can include a continuous chain of blocks within the plurality of blocks 503 of the machine learning model 500.

[0104] Example Method

[0105] Figure 6A flowchart of an example method of reducing computational costs associated with a machine learning model according to an example embodiment of the present disclosure is depicted. Although for purposes of illustration and discussion, Figure 6 The steps are depicted as being performed in a particular order, but the method of the present disclosure is not limited to the particular illustrated order or arrangement. Various steps of method 600 may be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0106] At (602), method 600 may include determining resource allocation parameters that correspond to a desired allocation of system resources to the machine learning model at inference time. In some embodiments, the resource allocation parameters may be determined before the model input is input to the machine learning model (e.g., before inference time). The resource allocation parameters may also be determined based on the current state of the computing device. As an example, when the battery state of the computing device is low, the resource allocation parameters may correspond to a low desired allocation of system resources to the machine learning model at inference time to preserve the remaining battery power of the computing device. Similarly, when the currently available processor power is low and / or a large number of applications are currently executing, the resource allocation parameters may correspond to a low desired allocation of system resources to avoid long processing times. Thus, the resulting machine learning model can be adjusted based on the current state of the computing device to quickly provide a solution and / or preserve resources of the computing device.

[0107] In some embodiments, a user can provide input indicating the amount of computing resources the user would like to allocate to the machine learning model. As an example, the user can interact with a touch-sensitive display screen of a computing device (e.g., a smart phone) to provide input. For example, the user can enter a value (e.g., via a keyboard) or adjust a slider (or other virtual control object) to indicate her preference for faster results but potentially less accurate results or slower but potentially more accurate results.

[0108] At (604), the method may include deactivating a subset of the plurality of blocks of the machine learning model based on the resource allocation parameter. Various suitable techniques may be used to deactivate the subset of blocks. As an example, the subset of blocks may be disconnected so that information is not input into the subset of blocks. As another example, deactivating the subset may include adjusting a corresponding activation function associated with the subset of blocks. The activation function may be adjusted so that the output of each node is set to a constant value, such as 1 or 0. Deactivating the subset of blocks may include any suitable technique so that consumption of computational resources of the subset of blocks at inference time is substantially reduced or eliminated.

[0109] In some embodiments, the size of the subset of the plurality of blocks may be selected based on the magnitude of the resource allocation parameter such that the size of the subset of the plurality of blocks is negatively correlated with the magnitude of the resource allocation parameter. For example, a small resource allocation parameter may result in a large number of blocks being deactivated before inference time. However, in other embodiments, depending on the convention selected, the size of the subset of the plurality of blocks may be positively correlated with the magnitude of the resource parameter.

[0110] In some embodiments, for example, as described above with reference to Figure 4 As explained, the computing system can select blocks for deactivation based at least in part on an amount of computing resources associated with executing each corresponding block in the machine learning model.

[0111] At (606), for example, as referenced Figures 1A to 2C As described, method 600 may include inputting a model input to a machine learning model in which a subset of the plurality of blocks are disabled.

[0112] At (608), for example, as referenced Figures 1A to 2C As described, method 600 may include receiving a model output as an output of a machine learning model.

[0113] Figure 7 A flowchart for an example method for training a machine learning model to be robust to deactivating at least some of a plurality of blocks of a neural network of the machine learning model at inference time is depicted. Although for purposes of illustration and discussion, Figure 7 The steps are depicted as being performed in a particular order, but the method of the present disclosure is not limited to the particular order or arrangement described. The various steps of method 700 may be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0114] At (702), method 700 may include iteratively training a machine learning model using a training data set (e.g., training data 162). The training data set may include any suitable training input data set, and may optionally include a training output data set. An example training data set may include an audio file and recognized text of spoken words in the audio file. Another example training data set may include an image and object recognition output describing the location and / or label of the recognized object depicted in the image.

[0115] At (704), method 700 may include deactivating an iteration-specific subset of a plurality of blocks of the machine learning model prior to at least one iteration of iterative training of the machine learning model. The iteration-specific subset may be selected in a variety of suitable ways. As one example 110, the iteration-specific subset may be selected based on a respective probability associated with each block that is positively correlated with a respective position of each block within the neural network. It should be understood that, for example, referring to Figure 4 and Figure 5 , a method similar to the method described above with respect to deactivating a subset of a plurality of blocks based on resource allocation parameters prior to inference time may be used to select a particular subset of iterations. Referring again to Figure 4 During training, a subset of iteration-specific blocks can be selected in a semi-random manner that favors deactivating blocks located near the output of the machine learning model over deactivating blocks located near the input of the machine learning model. For example, in a first iteration, a subset of first-iteration-specific blocks (e.g., shadow blocks 414, 418, 422, and 424) can be deactivated; in a second iteration, a subset of second-iteration-specific blocks (e.g., blocks 412, 416, 418, and 422) can be deactivated; in a third iteration, a subset of third-iteration-specific blocks (e.g., blocks 408, 414, and 424) can be deactivated; and so on. As another example, referring again to Figure 5 , in a first iteration, a subset of first iteration specific blocks (e.g., starting residual block 512 and associated residual tail portion 530) may be disabled. In a second iteration, a different block (e.g., block 518) may be selected as the starting residual block so that a subset of second iteration specific blocks (e.g., blocks 518, 520, 522) are disabled. In a third iteration, another block (e.g., block 508) may also be selected as the starting residual block so that a subset of third iteration specific blocks (e.g., blocks 508-522) are disabled, and so on. Thus, the machine learning model may be trained to be robust to deactivating blocks that may be disabled at inference time. This training method may improve the quality of the output of the machine learning model at inference time.

[0116] The systems and methods of the present disclosure provide many technical effects and benefits, including, for example, reducing the computational resources required by a machine learning model at inference time. In addition, in the case where variants of a machine learning model with different computational requirements are desired, the storage of the machine learning model on the device can be reduced as an adjustable machine learning model, which can adjust its computational requirements and can replace multiple machine learning models. The described systems and methods can also reduce the computational resources required to train a machine learning model, because an adjustable machine learning model can be trained to replace multiple machine learning models with different computational requirements.

[0117] Additional disclosure

[0118] This paper discusses servers, databases, software applications and other computer-based systems, as well as the actions taken and the information sent to and from these systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions between and in components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0119] Although the subject matter has been described in detail for various specific example embodiments of the subject matter, each example is provided as an explanation, not as a limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, can easily produce changes, variations, and equivalents to these embodiments. Therefore, the subject matter disclosure does not exclude such modifications, variations, and / or additions to the subject matter that are obvious to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, the purpose of the present disclosure is to cover such changes, variations, and equivalents.

Claims

1. A computing device, comprising: at least one processor; a machine learning model comprising a plurality of blocks and one or more residual connections between two or more of the plurality of blocks, wherein the machine learning model is configured to receive a model input and, in response to receipt of the model input, output a model output; at least one tangible, non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: Before the machine learning model is used at inference time to generate model output: determining resource allocation parameters corresponding to a desired allocation of system resources for the machine learning model at inference time; selecting a subset of the plurality of blocks of the machine learning model based on the resource allocation parameters; and deactivating a subset of the plurality of blocks of the machine learning model to produce an adjusted machine learning model; and At inference time: inputting the model input to the adjusted machine learning model with a subset of the plurality of blocks disabled; and The model output is received as an output of the adjusted machine learning model. 2 . The computing device of claim 1 , wherein the subset of the plurality of blocks is selected such that at least one of the residual connections bypasses each of the subset of the plurality of blocks.

3. A computing device according to claim 1, wherein each of the subsets of the plurality of blocks is located between at least one residual connection of the machine learning model. 4 . The computing device of claim 1 , wherein the resource allocation parameter is determined based on one or more of: a battery status, a mode of the computing device, currently available processor power, or a number of currently running applications. 5 . The computing device of claim 1 , wherein determining the resource allocation parameter comprises receiving input from a user of the computing device.

6. A computing device according to claim 1, wherein determining the resource allocation parameters includes determining the resource allocation parameters before inputting the model input into the machine learning model.

7. The computing device of claim 1, wherein: At least two blocks of the plurality of blocks are residually connected in a residual chain; and Deactivating a subset of the plurality of blocks of the machine learning model includes deactivating a starting residual block and a residual tail portion of the residual chain.

8. The computing device of claim 7, wherein: The residual chain extends from an input end of the machine learning model to an output end of the machine learning model; and The residual tail portion includes blocks within a residual chain extending from the starting residual block to the output end of the machine learning model.

9. A computing device according to claim 1, wherein selecting a subset of the plurality of blocks comprises selecting blocks based on a respective probability of each block being positively correlated with a respective position of each block within the machine learning model.

10. The computing device of claim 1, wherein: The resource allocation parameter corresponds to the maximum target resource allocation amount of the machine learning model; as well as Selecting a subset of the plurality of blocks includes selecting blocks based on a respective resource consumption amount of each block to adjust an amount of resources required to execute the machine learning model to be less than the maximum target resource allocation amount.

11. The computing device of claim 1 , further comprising selecting a size of the subset of the plurality of blocks for deactivation based on a magnitude of the resource allocation parameter such that the size of the subset of the plurality of blocks is negatively correlated to the magnitude of the resource allocation parameter.

12. The computing device of claim 1, wherein deactivating a subset of the plurality of blocks of the machine learning model comprises deactivating a layer of the machine learning model.

13. The computing device of claim 1, wherein the machine learning model comprises a convolutional neural network and at least one of the plurality of blocks comprises a convolutional block that applies at least one convolutional filter.

14. A computer-implemented method of reducing computational costs associated with a machine learning model, the method comprising: Before the machine learning model is used at inference time to generate model output: determining, by one or more computing devices, resource allocation parameters describing a desired allocation of system resources to the machine learning model at inference time; selecting, by the one or more computing devices, a subset of the plurality of blocks of the machine learning model based on the resource allocation parameters; deactivating, by the one or more computing devices, a subset of the plurality of blocks of the machine learning model to produce an adjusted machine learning model; and At inference time: inputting, by the one or more computing devices, an input set into the adapted machine learning model; as well as An output set is received by the one or more computing devices as an output of the adjusted machine learning model.

15. The computer-implemented method of claim 14 further comprising receiving the machine learning model at a user computing device by the one or more computing devices after deactivating the subset of the plurality of blocks.

16. The computer-implemented method of claim 14, wherein the subset of the plurality of blocks is selected such that at least one of the residual connections bypasses each of the subset of the plurality of blocks.

17. The computer-implemented method of claim 14, wherein the resource allocation parameter is determined based on one or more of: a battery status, currently available processor power, or a number of currently running applications.

18. The computer-implemented method of claim 14, wherein determining the resource allocation parameters comprises receiving, by the one or more computing devices, input from a user of the one or more computing devices.

19. A method for training a machine learning model to be robust to deactivating at least some of a plurality of blocks of a neural network of the machine learning model at inference time, the method comprising: iteratively training the machine learning model using the training dataset by one or more computing devices; as well as Prior to at least one iteration of iterative training of the machine learning model, the one or more computing devices determine resource allocation parameters corresponding to a desired allocation of system resources to the machine learning model at inference time, the one or more computing devices select a subset of multiple blocks of the machine learning model based on the resource allocation parameters, and the one or more computing devices deactivate an iteration-specific subset of the multiple blocks of the machine learning model to produce an adjusted machine learning model.

20. The method of claim 19, further comprising selecting an iteration-specific subset of the plurality of blocks based on a respective probability associated with each block that is positively correlated with a respective position of each block within the neural network.

Images