Constraint mask for sparsification in machine learning

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for feature masks. A feature tensor is accessed in a neural network, and a feature mask is generated by processing the feature tensor using a mask sub-network trained to generate the feature mask based at least in part on a polarization constraint and an activation constraint. A mask feature tensor is generated based on the feature tensor and the feature mask, and an output inference is generated using the neural network based at least in part on the mask feature tensor.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 17 / 932,941 filed on September 16, 2022, which is hereby incorporated by reference into this application.

[0003] introduction

[0004] Aspects of the present disclosure relate to feature masks in machine learning.

[0005] Many machine learning architectures involve the extraction and analysis of features of input data (guided by trained parameters). A wide variety of machine learning models rely on this feature extraction to perform useful tasks, such as classification. However, not all features are equally useful. That is, while the model can learn to extract relevant features, it is common for a certain subset of learned features to have little or no use in performing the desired task. Therefore, there is a need for a way to identify and / or select useful features (such as by masking), thereby improving the efficiency of the architecture without unduly compromising the performance of the model.

[0006] Therefore, techniques for improved feature masking are needed. Summary of the invention

[0007] One aspect provides a method comprising: accessing a feature tensor in a neural network; generating a feature mask by processing the feature tensor using a mask subnetwork, wherein the mask subnetwork is trained at least in part based on a polarization constraint and an activation constraint to generate the feature mask; generating a mask feature tensor based on the feature tensor and the feature mask; and generating an output inference using the neural network based at least in part on the mask feature tensor.

[0008] Other aspects provide processing systems configured to perform the aforementioned methods and those described herein; non-transitory computer-readable media comprising instructions that, when executed by a processor of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium, the computer program product comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising components for performing the aforementioned methods and those further described herein.

[0009] The following description and associated drawings set forth in detail certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings depict certain aspects of the one or more aspects and therefore are not to be considered limiting of the scope of the present disclosure.

[0011] Figure 1 Depicted is an example workflow for constrained feature masking using a trained mask generator.

[0012] Figure 2 Depicted is an example workflow for training a mask generator for constrained feature masking.

[0013] Figure 3 An example flow chart illustrating a method for generating a constrained feature mask is depicted.

[0014] Figure 4 An example flow chart illustrating a method for training a mask generator for constrained feature mask generation is depicted.

[0015] Figure 5 An example flow chart illustrating a method for generating inferences based on a mask feature tensor is depicted.

[0016] Figure 6 An example processing system configured to perform various aspects of the present disclosure is depicted.

[0017] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation. DETAILED DESCRIPTION

[0018] Aspects of the present disclosure provide techniques for improving feature masks using constraints and training mask generators. In some aspects, a feature mask is said to be "constrained" to indicate that the feature mask is generated by components trained using one or more constraints such as polarization constraints and / or activation constraints, as discussed in more detail below.

[0019] Some architectures and systems use feature masking, which typically involves analyzing only a subset of features to select only relevant features for the task. This can help with efficient computation, removing spurious features, preventing overfitting, and / or for diagnostic and model accuracy purposes.

[0020] In some systems, deep learning-based feature masks (e.g., for some computer vision tasks, such as occlusion masks used in optical flow estimation, 3D scene flow estimation, depth estimation, etc.) have been introduced in some architectures to sparse models. As used herein, "sparseness" or "sparseness" may generally refer to the process of reducing the density of features or data or otherwise softening (e.g., reducing extreme values) or reducing values ​​in data (e.g., where a mask is applied so that some values ​​are replaced with zero or values ​​closer to zero). Using learning-based masking methods can produce selectivity and / or sparsification in network features. However, while learning-based feature masks are useful, conventional methods involve indirect and cumbersome techniques, such as the Gumbel-softmax trick or techniques, for neural networks to learn and adapt during a period of training.

[0021] Furthermore, some techniques for learning-based masks are often based at least in part on randomness and are not flexible enough to support additional constraints, regularization, and / or loss terms in training. That is, such methods are non-deterministic (e.g., where the sigmoid operation of the mask is enhanced with random noise and the corresponding temperature scaling is through an annealing scheme).

[0022] Typically, in various aspects of the present disclosure, a mask generator (or module) is used to provide a constrained feature mask. The mask generator can be an individual model or sub-model (or sub-network) of an existing model that is configured to generate a feature mask as described herein. Specifically, the mask generator processes an input feature map and an output feature mask that, when applied to the input feature map itself, generates mask features that are propagated to downstream components (e.g., to the next layer in a neural network). In some aspects, the network and the mask generator are trained end-to-end using a loss function having multiple components (e.g., multiple loss terms), which include task losses (specific to potential computational tasks, such as image segmentation) and constraints on the mask (such as to control polarization and activation rates).

[0023] As compared to conventional mask methods, feature masks generated according to aspects described herein may include many beneficial features, including, for example, eliminating inherent randomness and increasing flexibility by allowing the use of hyperparameters or learned parameters to control the activation rate and polarization of the mask. In some aspects of the present disclosure, this flexibility can be achieved using various constraints and additional loss functions during training, which improves model performance.

[0024] The improved feature masks described in this disclosure can improve machine learning in a wide variety of ways. For example, the learning-based feature masks described herein enable more efficient computation because the masks allow the model to select a subset of features, thereby allowing faster computation with reduced computational resources. These improvements are particularly effective for dense prediction tasks (e.g., tasks involving processing significant data such as to generate pixel-level predictions) such as for optical flow, segmentation, etc.

[0025] Additionally, in some aspects, the techniques described herein can be used for adaptive computing, thereby enabling dynamic selection of network modules based on available computing resources (e.g., memory, power, and / or computing budget). For example, in a system with limited computing resources available for training and / or using a machine learning model, polarization rates and / or activation rates can be defined to reduce the computational load to fit within the available resources. In some aspects, polarization rates and activation rates can be defined differently at different points in the model (e.g., at different layers in a neural network), thereby enabling dynamic and adaptive resource usage.

[0026] In one aspect, activation rate constraints can generally control the amount of a feature tensor that is masked (e.g., whether 40% or 70% of the feature tensor is masked). Polarization constraints can generally affect the discreteness or hardness of the values ​​of the mask. For example, a higher polarization rate can result in mask values ​​being generated that are closer to their extremes (e.g., closer to zero and one, with few or no values ​​near the middle of the range, resulting in highly polarized or sharp edges of the mask), while lower polarization results in a more balanced distribution (with more values ​​away from the ends of the range, resulting in a softer mask / a mask with softer edges).

[0027] In some aspects, learned masking techniques enable improved feature selection, where the most relevant features are selected for a particular task (and other features may be masked, ignored, or otherwise not used). Thus, aspects of the present disclosure may prevent irrelevant features from interfering with recognition / inference.

[0028] Additionally, aspects of the present disclosure may enable improved model personalization (e.g., for speaker verification or anti-spoofing). That is, in some aspects, a pre-trained model may be quickly adapted to a new domain (e.g., to a new speaker) using a few samples from a new data distribution. The masking techniques described herein may reduce feature dimensionality complexity and help prevent or reduce overfitting for these model adaptations.

[0029] In addition, aspects of the present disclosure can achieve improved interpretability. For example, because masks enable the model to select regions in the input feature map that influence a particular model output (e.g., inference, such as classification), masks can be used as a diagnostic tool to find the reasoning behind the model output, such as what features dominate in reaching the model output.

[0030] Furthermore, by controlling the polarization constraints, soft values ​​may be obtained for improved uncertainty measurements. For example, soft (i.e., continuous) values ​​(e.g., those between zero and one) may carry more information than discrete extreme values ​​(e.g., zero and one). Thus, in some aspects, such soft values ​​may not only serve the purpose of the mask (e.g., by rounding the soft value to the nearest value of zero or one), but such soft values ​​may also be used as confidence (or certainty) for the mask. For example, both 0.98 and 0.53 may be rounded to one, resulting in the mask having a value of "one" at that position (thereby not masking the latent tensor). However, 0.98 may be used to indicate a significantly higher confidence (e.g., a higher confidence that the pixel should not be masked) than 0.53, even though both have the same result.

[0031] Additionally, in a federated learning implementation, user-specific masks may be generated to obtain relevant features for a particular user (thereby allowing the system to ignore or mask features that are not relevant or less relevant to that user). That is, each user may locally train and use a specific mask generator based on local training data. Thus, these user-specific mask generators may generate user-specific masks, thereby allowing each user to identify specific features that are relevant to the user's specific data set.

[0032] Example workflow for constraining feature masks

[0033] Figure 1 An example workflow 100 for constraining feature masking using a trained mask generator is depicted. In the illustrated example, the workflow 100 is performed by a machine learning system 102. The machine learning system 102 may generally correspond to a physical or virtual computing device (e.g., hardware and / or software) configured to generate output inferences using a machine learning model such as a neural network. That is, the machine learning system 102 may process input data using one or more neural networks to generate inferences that may be used for a variety of tasks depending on the particular implementation.

[0034] The depicted workflow 100 generally corresponds to a single portion of a model (e.g., a feature extraction phase). That is, the feature extractor 110, the mask generator 120, and the operations 130 may be used as part of a feature extraction phase of a model (e.g., a neural network), and there may be any number of operations performed before the workflow 100 (e.g., to generate input to the workflow 100 and / or the machine learning model) and any number of operations performed after the workflow 100 (e.g., to use the remainder of the model to generate output inferences).

[0035] In the illustrated workflow 100, input tensor 105 is processed by feature extractor 110 to generate feature tensor 115 (also referred to in some aspects as feature map). In general, feature extractor 110 may correspond to one or more layers in a neural network for extracting features from input data (e.g., a ResNet backbone). Input tensor 105 may include output data from a previous layer (e.g., a set of activations from a previous layer), or may include raw input to a model (e.g., where feature extractor 110 is the first or initial component of the model).

[0036] As shown, the feature tensor 115 is then processed by the mask generator 120 to generate a mask 125 (sometimes referred to as a feature mask). Although the illustrated example depicts the mask generator 120 operating on the feature tensor 115 (generated by the feature extractor 110), and depending on the particular implementation, the mask generator 120 may generally process any tensor (e.g., the input tensor 105) to generate a corresponding tensor mask, and need not operate only on the feature tensor 115 (to generate a feature mask).

[0037] In some aspects, the mask generator 120 generally corresponds to one or more layers of a neural network. During training, the mask generator 120 learns to generate a mask 125 based on the input feature tensor 115. As discussed in more detail below, the mask generator 120 can be trained with one or more constraints such as an activation rate constraint and / or a polarization constraint.

[0038] As discussed above, activation rate constraints may generally control the amount of feature tensor 115 that is masked (e.g., whether 40% or 70% of feature tensor 115 is masked). Polarization constraints may generally affect the discreteness or hardness of the values ​​of mask 125. For example, a higher polarization rate may result in mask values ​​being generated that are closer to their extremes (e.g., closer to zero and one, with few or no values ​​near the middle of the range, while a lower polarization rate results in a more balanced distribution (with more values ​​away from the ends of the range).

[0039] In some aspects, mask 125 generally has the same size and / or dimensions as feature tensor 115. Values ​​in mask 125 indicate whether corresponding elements in feature tensor 115 are masked, and / or how many elements should be masked.

[0040] In some aspects, a binary mask is used, where all entries (e.g., elements) in the mask have a binary value of zero or one. These binary values ​​may be derived directly from the polarization constraint, or in some cases, by rounding off "soft" values ​​(e.g., continuous values ​​between zero and one), as discussed above. In some aspects, regardless of whether the mask is soft (e.g., non-binary), the system may perform an element-by-element multiplication between the feature tensor and the mask (since the mask tensor has the same shape and dimension as the feature tensor) to obtain a mask feature tensor. For a soft mask, this may result in partial mask elements (e.g., multiplying an element in the feature tensor by a value of 0.75, thereby partially masking the element). In the extreme case of a soft mask (e.g., where the mask element is zero or one), the mask acts as a binary mask. A binary mask may be similarly applied, where a mask value of one passes the feature through unchanged, and a mask value of zero causes the corresponding feature to be completely masked out. As discussed above, the mask may be referred to as sparsification in some aspects.

[0041] In the depicted workflow 100, both the mask 125 and the feature tensor 115 are input to an operation 130 to generate a mask feature tensor 135. In some aspects, the operation 130 is an element-wise product operation. That is, the operation 130 may include multiplying each element in the feature tensor 115 by the corresponding element in the mask 125.

[0042] The mask feature tensor 135 may then be output for downstream processing. In some aspects, outputting the mask feature tensor 135 includes providing the mask feature tensor 135 as an input to an activation function, and / or as an input to a subsequent layer or component in a machine model (e.g., a neural network or other model using the depicted feature extraction stage) to generate output inferences. In some aspects, outputting the mask feature tensor 135 includes returning the mask feature tensor 135 as the final output from the model. Additionally, although not depicted in the illustrated example, in some aspects, the mask 125 itself may be used as an output from the model. For example, a specific model to which the mask generator 120 belongs may be used to generate the mask 125, and another module (e.g., another neural network) may utilize the generated mask 125 to perform further processing.

[0043] As discussed above, the mask generator 120 may thereby improve the efficiency and operation of the model, at least through reduced computational cost, improved model accuracy, etc. For example, downstream layers or other components may be able to process less data, such as by processing smaller tensors corresponding to selected features, or by avoiding or refraining from performing computations involving masked portions of the feature tensor.

[0044] Example workflow for training a constrained mask generator

[0045] Figure 2 An example workflow 200 for training a mask generator for constrained feature masking is depicted. In the illustrated example, the workflow 200 is performed by a machine learning system 202. The machine learning system 202 may generally correspond to a physical or virtual computing device (e.g., hardware and / or software) configured to train a machine learning model, such as a neural network, including one or more mask generators, such as 120. That is, the machine learning system 202 may use training data to generate losses, and then use these losses to refine the model (including one or more mask generators, such as 120).

[0046] The depicted workflow 200 generally corresponds to a single portion of a model. That is, there may be any number of operations performed before the workflow 200 (e.g., to generate inputs to the workflow 200) and any number of operations performed after the workflow 200 (e.g., to generate output inferences and refine the model). In some aspects, the machine learning system 202 is similar to the one described above with reference to Figure 1 The machine learning system 102 discussed above is the same system. In other aspects, the machine learning system 102 and the machine learning system 202 may be different and discrete systems or components. That is, Figure 1 Workflow 100 (corresponding to training a model, including use of one or more mask generators) and workflow 200 (corresponding to training the model, including one or more mask generators) may be performed on the same system or device, or may be performed on separate systems.

[0047] In the illustrated workflow 200, during training, the input tensor 205 is generated by the feature extractor 110 (which may correspond to Figure 1 The feature extractor 110 of FIG. 110 is processed to generate a feature tensor 215. As discussed above, the feature extractor 110 may generally correspond to one or more layers in a neural network. Figure 1 , input tensor 105, input tensor 205 may include output data from a previous layer (e.g., a set of activations from a previous layer), or may include the original input to the model (e.g., where feature extractor 110 is the first or initial component of the model).

[0048] As shown and discussed above, the feature tensor 215 is then generated by the mask generator 120 (which may correspond to Figure 1 The mask generator 120 of FIG. 110 processes the feature tensor 215 to generate the corresponding mask 225. Although the illustrated example depicts the mask generator 120 operating on the feature tensor 215 (generated by the feature extractor 110), the mask generator 120 may generally process any tensor (e.g., the input tensor 205), and need not operate only on the feature tensor 215, depending on the particular implementation.

[0049] As shown, the mask 225 and the feature tensor 215 are combined by operation 130 (e.g., element-wise multiplication) to obtain the mask feature tensor 235. Figure 1 As discussed, during inference, the mask feature tensor 235 can be provided to one or more subsequent components (e.g., subsequent layers in the model). In some aspects, the mask feature tensor 135 can similarly be provided to downstream components during training, such as during a forward pass of the training process.

[0050] Additionally, during training, mask feature tensor 235 and mask 225 may be processed by loss component 240 to generate a loss 245 for the model. Although depicted as discrete components for conceptual clarity, in various aspects, the operation of loss component 240 may be implemented in a variety of ways using hardware, software, or a combination of hardware and software.

[0051] In some aspects, loss 245 may include multiple components of loss. In at least one aspect, loss component 240 may generate a task loss based at least in part on mask feature tensor 235. For example, mask feature tensor 235 may be used to generate an output inference (potentially through one or more downstream layers of the model), and the inference may be compared to the true value label of the input sample (e.g., using a cross entropy loss). The task loss may then be back-propagated through the model (including through mask generator 120 and feature extractor 110) to refine the weights and / or biases. In at least one aspect, the task loss is defined as L task (f s ), where L task is any suitable loss function, such as the cross entropy loss, and f s is the mask feature tensor 235, which can be defined as f s =m*f, where m is the mask 225 and f is the feature tensor 215.

[0052] Additionally, to provide one or more constraints to the mask generator 120, the loss component 240 can generate activation loss and / or polarization loss using the mask 225. As discussed above, the polarization loss can affect the extreme degree of the generated value of the mask 225. In at least one aspect, the polarization loss for the mask m is defined using the following equation 1, where E[·] is the expected value or expected value of ·. In this way, the polarization loss is defined based on the expected value of the product of the mask 225 (m) and the inverse of the mask 225 (1-m).

[0053] L pol (m) = E[m(1-m)] (1)

[0054] In some aspects, the activation loss can generally affect how much of the feature tensor is masked. In at least one aspect, the activation loss for mask m is defined using Equation 2 below, where a is the activation rate (which can be a defined hyperparameter or a learned parameter). In this way, the activation loss is defined based on the activation rate and the expected value of the mask 225 (m).

[0055] L act (m) = (R[m] - a) 2 (2)

[0056] In some aspects, the loss 245 is calculated by using a weighted sum of the individual loss components. For example, the total loss 245 may be defined as L total =λ pol L pol +λ act L act +λ task L task , where λ pol , act and λ task are defined weights. In various aspects, these loss weights may be defined as hyperparameters, or may be learned during training. Modifying each of these weights may have different effects on the mask generator 120. For example, increasing the weight of the polarization constraint may increase the polarization (e.g., discreteness or hardness) of the generated mask 225. In some aspects, the "hardness" or "discreteness" of the mask may be used to describe the polarization, in that more polarized values ​​(also referred to as more hard values) are also more discrete (e.g., closer to zero and one), while less polarized values ​​(e.g., in the case of a soft mask) may generally be away from the discrete ends of the range (e.g., away from zero and one).

[0057] In some aspects, only two loss weights may be used because controlling the weights of any two loss components may affect the contribution of a third loss component, thereby implicitly defining the weight of the third loss component.

[0058] In some aspects, the loss weights and / or activation rates may be modified during training. That is, the activation rates, the weights of the polarization constraints, and / or the weights of the activation rate constraints may vary during training (e.g., using different values ​​at different stages of training). For example, these hyperparameters may be reconfigured based on changes in loss values ​​seen at different times. Similarly, the activation rates need not be fixed and may vary throughout training based on feedback from the task loss values.

[0059] As an example, if a particular loss term (e.g., activation loss, polarization loss, or task loss) is relatively high, the system can increase the weight coefficient for that particular loss term so that the loss term is more important during training, thereby causing the particular loss to decrease more quickly over a training round or epoch. This can allow each loss term to eventually have a similar value (e.g., be on a similar scale). In some aspects, the activation rate also varies (e.g., if the observed activation rate is different from the expected activation rate). For example, if the preferred activation rate is 0.1, but the observed activation rate of the generated mask is 0.05, the activation rate or weight can be increased to compensate.

[0060] In some aspects, the output of the mask generator 120 is generated by a sigmoid function, and the system can use a temperature hyperparameter to control the discreteness of the mask 225, which can have an effect similar to changing the weight of the polarization constraint. As discussed above, the discreteness of the mask can generally refer to the proximity of the mask value to zero or one. If these values ​​are close to either extreme value, the mask is said to be more discrete or polarized. In some aspects, high temperature means that the mask is less discrete (e.g., closer to an intermediate value, such as 0.5). Therefore, in some aspects, the temperature can be controlled using the weight of the polarization loss term, for example, by observing the average polarization / discreteness of the output and adjusting the weight accordingly.

[0061] In various aspects, the mask generator 120 may be generally applicable to one or more layers of the model. That is, depending on the computational limitations of the system, a separate mask generator 120 may be trained and used for each layer in a set of multiple layers.

[0062] Additionally, in some aspects, a multi-level mask (rather than a binary mask) can be used to implement multi-level selection by using multi-hot encoding and polarization constraints on each dimension. For example, rather than a binary mask (having two states: zero and one), the mask can include multiple states (e.g., four states, which can be encoded as [0,0], [0,1], [1,0], [1,1]). To allow for such multiple states, in some aspects, the system can use additional dimensions in the mask (e.g., two dimensions in this example). In this aspect, polarization constraints can also be applied to multiple (e.g., two) dimensions.

[0063] As discussed above, in one aspect, the mask generator 120 and the feature extractor 110 are typically trained end-to-end using the loss 245 (e.g., through a training or updating component), as depicted by arrow 250. That is, the loss 245 may be back-propagated (e.g., through one or more downstream layers), through the mask generator 120, and then through the feature extractor 110. This enables joint training of the feature extractor 110 and the mask generator 120, resulting in improved model performance and accuracy.

[0064] Example Method for Constraining Feature Masks

[0065] Figure 3 An example flow chart illustrating a method 300 for generating a feature mask is depicted. In some aspects, the method 300 provides the Figure 1 Additional details of the discussed workflow 100. In some aspects, the method 300 is performed by a machine learning system such as Figure 1 The machine learning system 102 is used to perform.

[0066] At block 305, access the input tensor (e.g., Figure 1 As used herein, "accessing" data (such as an input tensor) may generally include receiving data (e.g., from a memory or storage device), requesting data, retrieving data, etc. As discussed above, depending on the specific implementation, the input tensor may generally have any size and dimension. That is, the received input tensor may have any height and width and any channel depth. In some aspects, the input tensor is the output (e.g., a set of activations) from a previous layer in the neural network. In some aspects, the input tensor is the raw input (e.g., an image) to the neural network.

[0067] At block 310, the input tensor is used to generate a feature tensor (such as Figure 1 For example, a feature extractor (e.g., Figure 1 The feature extractor 110 of FIG. 110 processes the input tensor. As discussed above, in some aspects, generating the feature tensor includes processing the input tensor using one or more layers of a neural network. Depending on the specific implementation, the feature tensor can generally have any dimension and size.

[0068] At block 315, a mask generator (e.g., Figure 1 The mask generator 120 of ) then uses the feature tensor to generate a mask (such as Figure 1As discussed above, in some aspects, the mask generator may generate a mask based on an input tensor rather than a feature tensor. That is, in some aspects, block 310 may be skipped. Such a mask may be used to mask the input tensor itself, or to mask a generated feature tensor (potentially using convolution or other operations to ensure that the mask is the same size as the feature tensor).

[0069] In one aspect, the mask generator corresponds to one or more layers of a neural network. In some aspects, the mask generator can be referred to as a subnet or subnetwork of a larger neural network. Figure 4 As discussed in more detail, one or more constraints (eg, related to polarization and / or activation rate) may be used to train a mask generator to achieve an improved and dynamic mask.

[0070] In some aspects, the mask is represented using a tensor having the same size and / or dimension as the feature tensor (or input tensor, if the input tensor is used to generate the mask), where each element in the mask has a corresponding element in the feature tensor. Typically, the value of each element in the mask indicates whether the corresponding element in the feature tensor should be masked. In some aspects, the mask is a binary tensor, where a value of zero may indicate that the corresponding feature element should be masked, and a value of one may indicate that the corresponding feature element should not be masked. In other aspects, the mask may be non-binary, as discussed above.

[0071] At block 320, the mask and feature tensor are used to generate a mask feature tensor (such as Figure 1 For example, the feature mask and the feature tensor may be combined using element-wise multiplication (e.g., using Figure 1 Operation 130) where each element in the feature tensor is multiplied by the corresponding element in the mask. In one aspect, the mask feature tensor can thus have the same size as the feature tensor and the mask. In aspects where an input tensor is used to generate a mask, the system can generate a mask input tensor by combining the input tensor and the resulting mask (e.g., using element-wise multiplication).

[0072] At block 335, the mask feature tensor is then output for any downstream processing. For example, in some aspects, the mask feature tensor is processed using an activation function. In some aspects, the mask feature tensor is output for use as an input to a subsequent layer in a neural network. The subsequent layer can use the mask feature tensor to generate output inferences for the neural network. In at least one aspect, the mask feature tensor itself is used as an output from the neural network.

[0073] Although the depicted method 300 includes the generation of a single mask, in some aspects, the neural network may include multiple mask generation stages. That is, multiple layers of the neural network may each include a respective mask generator for generating a corresponding mask for a particular layer.

[0074] It should be noted that Figure 3 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0075] Example method for training a mask generator for constrained feature masking

[0076] Figure 4 An example flow chart illustrating a method 400 for training a mask generator for constrained feature mask generation is depicted. In some aspects, the method 400 provides the above reference Figure 2 Additional details of the discussed workflow 200. In some aspects, the method 400 is performed by a machine learning system such as Figure 2 The machine learning system 202 is used to perform.

[0077] At block 405, a mask (e.g., Figure 2 In some aspects, a loss component (e.g., Figure 2 The mask may be received by the loss component 240 of the training mask generator. In one aspect, the mask may be generated during a forward pass of training the mask generator and / or the wider neural network. For example, the training samples may be used as input to the network during a forward pass, resulting in the generation of the mask (e.g., after feature extraction).

[0078] At block 410, the loss component generates a polarization loss based on the generated mask. For example, as described above with reference to Figure 2 As discussed above, the loss component can generate the polarization loss based on an expected value of the product of the mask and the inverse of the mask. That is, the loss component can determine the inverse of the mask, multiply the inverse by the mask itself, and then calculate the expected value of the resulting product (e.g., using Equation 1 above). As discussed above, the polarization loss can generally be used to affect the degree of polarization of the mask.

[0079] At block 415, the loss component generates an activation loss based on the mask. In some aspects, as discussed above, the activation loss is generated based on the mask and a defined activation rate (which may be a configurable hyperparameter). In at least one aspect, as discussed above, the activation loss may be calculated based on the calculated expected value of the mask and the defined activation rate (e.g., using Equation 2 above). As discussed above, the activation loss may generally be used to influence how much percentage or amount of the feature tensor is masked (e.g., how many non-zero elements the generated mask contains).

[0080] At block 420, the loss component may receive a mask feature tensor generated using the mask received at block 405. That is, the mask feature tensor may have been generated by computing an element-wise multiplication of the feature tensor and the mask.

[0081] At block 425, the loss component generates a task loss based at least in part on the mask feature tensor. In some aspects, the mask feature tensor is used directly to calculate the task loss. In other aspects, the mask feature tensor is used to generate an output inference (e.g., by one or more subsequent layers of the model), and the task loss can be calculated based on the resulting inference. In some aspects, the task loss is generated based on a cross entropy function (e.g., based on the difference between the output inference generated using the mask feature tensor and the true value label of the input data used to generate the inference).

[0082] At block 430, the loss components may aggregate polarization loss, activation loss, and task loss. For example, using a weighted sum (where two or more of the loss components have corresponding weights), the loss components may aggregate the losses to generate a total loss for the mask generator / neural network. As discussed above, these weights may be configurable hyperparameters (which may be changed during training), or may be learned weights. In general, adjusting these weights may affect the polarization and / or activation rate of the generated mask.

[0083] At block 435, the total loss is used to refine the mask generator (e.g., via back-propagation). In some aspects, the total loss is also used to refine previous layers of the model, such as the feature extractor (e.g., Figure 2 In some aspects, as discussed above, the feature extractor and mask generator are refined end-to-end such that the loss is propagated through the mask generator and then through the feature extractor.

[0084] Although the illustrated examples suggest a sequential process for conceptual clarity, in some aspects, these operations may be performed in a different order and may be performed in whole or in part in parallel. For example, in some aspects, blocks 410, 415, and 425 may be performed in parallel. As discussed above, although the depicted method 400 includes generating polarization loss and activation loss based on a single mask, in some aspects, the neural network may include multiple mask generation stages. That is, multiple layers of the neural network may each include a corresponding mask generator for generating a corresponding mask for a particular layer. In some such aspects, the loss component may similarly generate a corresponding polarization loss, activation loss, and / or task loss for each such mask generator. In at least one aspect, the system generates a corresponding polarization loss and activation loss for each mask generator, and generates an overall task loss based on the final output of the model.

[0085] In some aspects, if each mask generator provides a mask operation for a corresponding layer, all losses may be summed for all loss terms and all layers. In at least one aspect, for intermediate layers (e.g., any layer before the final layer that generates the output of the model), cross entropy or any other task loss is not used, and only mask constraints or losses are used to refine these intermediate layers. In one such aspect, the task loss is only used to refine the final output layer. In some aspects, using backpropagation, the loss term for a particular layer will only affect that particular layer and layers shallower (e.g., before that layer) in the model. That is, the loss term from a given layer will not be used to modify any downstream layers.

[0086] Additionally, although the illustrated examples depict refining a mask generator and feature extractor using a single mask generated using a single input example (e.g., using stochastic gradient descent), in various aspects, the machine learning system may instead use method 400 to process multiple examples using batches of input data to refine model parameters (e.g., using batch gradient descent).

[0087] It should be noted that Figure 4 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0088] Example method for machine learning using constrained masks

[0089] Figure 5 An example flow chart illustrating a method for generating inferences based on a mask feature tensor is depicted. In one aspect, method 500 may be performed by a machine learning system such as Figure 1 The machine learning system 102 and / or Figure 2 The machine learning system 202 is used to perform.

[0090] At block 505, a feature tensor (e.g., Figure 1 The feature tensor 115).

[0091] At block 510, a mask subnetwork (e.g., Figure 1 The mask generator 120 of ) processes the feature tensor to generate a feature mask (e.g., Figure 1 125), wherein the mask sub-network is trained at least in part based on polarization constraints and activation constraints to generate feature masks.

[0092] At block 515, a mask feature tensor (e.g., Figure 1 The mask feature tensor 135).

[0093] At block 520 , an output inference is generated using a neural network based at least in part on the mask feature tensor.

[0094] In some aspects, generating the output inference includes outputting the mask feature tensor to a subsequent layer in the neural network.

[0095] In some aspects, generating output inference is performed while processing input data during a runtime phase of the neural network.

[0096] In some aspects, generating output inferences is performed while processing training data during a training phase of the neural network.

[0097] In some aspects, method 500 also includes generating a polarization loss based at least in part on the feature mask, and refining one or more parameters of the mask sub-network based on the polarization loss.

[0098] In some aspects, the polarization loss is generated based at least in part on an expected value of a product of a feature mask and an inverse of the feature mask.

[0099] In some aspects, method 500 also includes generating an activation loss based at least in part on the feature mask, and further refining one or more parameters of the mask sub-network based on the activation loss.

[0100] In some aspects, an activation loss is generated based at least in part on a desired value of a feature mask and an activation rate, where the activation rate is a hyperparameter that controls what percentage of the feature tensor is masked.

[0101] In some aspects, method 500 also includes generating a task loss based at least in part on the mask feature tensor, and further refining one or more parameters of the mask sub-network based on the task loss.

[0102] In some aspects, method 500 further includes aggregating the polarization loss, the activation loss, and the task loss using a defined set of weights prior to refining one or more parameters of the mask sub-network, wherein the defined set of weights controls polarization of the feature mask.

[0103] In some aspects, method 500 also includes refining one or more previous layers of the neural network (e.g., Figure 1 feature extractor 110).

[0104] It should be noted that Figure 5 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0105] Example Processing System for Constrained Feature Masks

[0106] In some respects, reference Figures 1 to 5The described workflows, techniques, and methods may be implemented on one or more devices or systems. Figure 6 An example processing system 600 is depicted that is configured to perform various aspects of the present disclosure, including, for example, with respect to Figures 1 to 5 In one aspect, the processing system 600 may correspond to a machine learning system that trains and / or uses a machine learning model, including a mask generator, such as Figure 1 The machine learning system 102 and / or Figure 2 's machine learning system 202. In at least some aspects, as discussed above, the operations described below with respect to the processing system 600 may be distributed across any number of devices.

[0107] Processing system 600 includes a central processing unit (CPU) 602, which in some examples may be a multi-core CPU. Instructions executed at CPU 602 may be loaded, for example, from a program memory associated with CPU 602, or may be loaded from memory 624.

[0108] The processing system 600 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 604 , a digital signal processor (DSP) 606 , a neural processing unit (NPU) 608 , a multimedia processing unit 610 , and a wireless connectivity component 612 .

[0109] An NPU, such as NPU 608, is typically a dedicated circuit configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. An NPU is sometimes alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0110] NPUs such as NPU 608 are configured to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip such as a system on a chip (SoC), while in other examples, the NPU may be part of a dedicated neural network accelerator.

[0111] The NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPUs that can perform both training and inference, the two tasks can generally still be performed independently.

[0112] NPUs designed to accelerate training are typically configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (usually labeled or tagged), iterating on the dataset, and then adjusting model parameters such as weights and biases to improve model performance. Typically, optimization based on error predictions involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0113] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs can be configured to input new data slices and quickly process the new data slices through the trained model to generate model outputs (e.g., inferences).

[0114] In one specific implementation, the NPU 608 is part of one or more of the CPU 602 , the GPU 604 , and / or the DSP 606 .

[0115] In some examples, wireless connectivity component 612 may include, for example, subcomponents for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. Wireless connectivity component 612 is further connected to one or more antennas 614.

[0116] The processing system 600 may also include one or more sensor processing units 616 associated with any manner of sensors, one or more image signal processors (ISPs) 618 associated with any manner of image sensors, and / or a navigation component 620, which may include a satellite-based positioning system component (e.g., GPS or GLONASS) and an inertial positioning system component.

[0117] The processing system 600 may also include one or more input and / or output devices 622, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0118] In some examples, one or more of the processors of processing system 600 may be based on the ARM or RISC-V instruction set.

[0119] The processing system 600 also includes a memory 624, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 624 includes computer-executable components that are executable by one or more of the aforementioned processors of the processing system 600.

[0120] Specifically, in this example, memory 624 includes feature components 624A, mask components 624B, and training components 624C. Memory 624 also includes a set of mask parameters 624D and model parameters 624E. Model parameters 624E may generally correspond to parameters of all or a portion of a machine learning model, such as weights and / or biases for one or more layers of a neural network. Mask parameters 624D generally correspond to parameters (or hyperparameters) for controlling masks, such as activation rates, polarization weights, and / or activation weights discussed above. The depicted components, as well as other components not depicted, may be configured to perform various aspects of the techniques described herein. Although for conceptual clarity, the descriptions in Figure 6 Although depicted as discrete components in FIG. 6A , feature component 624A, mask component 624B, and training component 624C may be implemented jointly or separately in various aspects.

[0121] Processing system 600 also includes characterization circuitry 626, masking circuitry 627, and training circuitry 628. The depicted circuitry, as well as other circuitry not depicted, may be configured to perform various aspects of the techniques described herein.

[0122] For example, feature component 624A and feature circuit 626 may correspond to Figure 1 and Figure 2 The feature extractor 110 is used to generate a feature tensor or map based on the input data, as described above with reference to Figures 1 to 3 and Figure 5 The mask assembly 624B and the mask circuit 627 may correspond to Figure 1 and Figure 2 The mask generator 120 is used to generate a feature mask based on the feature tensor, as described above with reference to Figures 1 to 3 and Figure 5 The training component 624C and the training circuit 628 may correspond to Figure 2 The training component 240 can be used to calculate losses (such as polarization loss, activation loss, and task loss) to refine the machine learning model, as described above with reference to Figure 2 , Figure 4 and Figure 5 discussed.

[0123] Although for clarity in Figure 6 Although depicted as separate components and circuits in FIG. 6 , the feature circuit 626 , the mask circuit 627 , and the loss circuit may be implemented together or separately in other processing devices of the processing system 600 , such as within the CPU 602 , the GPU 604 , the DSP 606 , the NPU 608 , etc.

[0124] Generally, the processing system 600 and / or its components may be configured to perform the methods described herein.

[0125] It is worth noting that in other aspects, aspects of the processing system 600 may be omitted, such as where the processing system 600 is a server computer, etc. For example, in other aspects, the multimedia processing unit 610, the wireless connectivity component 612, the sensor processing unit 616, the ISP 618, and / or the navigation component 620 may be omitted. Furthermore, aspects of the processing system 600 may be distributed among multiple devices.

[0126] Sample Clauses

[0127] Item 1: A method comprising: accessing a feature tensor in a neural network; generating a feature mask by processing the feature tensor using a mask subnetwork, wherein the mask subnetwork is trained at least in part based on a polarization constraint and an activation constraint to generate the feature mask; generating a mask feature tensor based on the feature tensor and the feature mask; and generating an output inference using the neural network based at least in part on the mask feature tensor.

[0128] Clause 2: The method of clause 1, wherein generating the output inference comprises outputting the mask feature tensor to a subsequent layer in the neural network.

[0129] Clause 3: A method according to any of clauses 1 to 2, wherein generating the output inference is performed while processing input data during a runtime phase of the neural network.

[0130] Clause 4: A method according to any of clauses 1 to 3, wherein generating the output inference is performed while processing training data during a training phase of the neural network.

[0131] Clause 5: The method of any one of clauses 1 to 4, further comprising: generating a polarization loss based at least in part on the feature mask; and refining one or more parameters of the mask sub-network based on the polarization loss.

[0132] Clause 6: The method of any of clauses 1 to 5, wherein the polarization loss is generated based at least in part on an expected value of a product of the feature mask and an inverse of the feature mask.

[0133] Clause 7: The method of any one of clauses 1 to 6, further comprising: generating an activation loss based at least in part on the feature mask; and further refining one or more parameters of the mask sub-network based on the activation loss.

[0134] Clause 8: A method according to any one of clauses 1 to 7, wherein the activation loss is generated based at least in part on an expected value of the feature mask and an activation rate, wherein the activation rate is a hyperparameter that controls what percentage of the feature tensor is masked.

[0135] Clause 9: The method of any one of clauses 1 to 8, further comprising: generating a task loss based at least in part on the mask feature tensor; and further refining the one or more parameters of the mask subnetwork based on the task loss.

[0136] Clause 10: The method according to any one of clauses 1 to 9 further includes: before refining the one or more parameters of the mask subnetwork, aggregating the polarization loss, the activation loss and the task loss using a defined weight set, wherein the defined weight set controls the polarization of the feature mask.

[0137] Clause 11: A method according to any one of clauses 1 to 10, further comprising: refining one or more previous layers of the neural network based at least in part on the polarization loss, the activation loss and the task loss.

[0138] Clause 12: A system comprising: a memory including computer executable instructions; and one or more processors configured to execute the computer executable instructions and cause the processing system to perform the method according to any one of clauses 1 to 11.

[0139] Clause 13: A system comprising: means for performing the method according to any one of clauses 1 to 11.

[0140] Clause 14: A non-transitory computer readable medium comprising: computer executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method of any one of clauses 1 to 11.

[0141] Clause 15: A computer program product embodied on a computer readable storage medium, the computer program product comprising code for performing the method according to any one of clauses 1 to 11.

[0142] Additional considerations

[0143] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Various examples may omit, replace, or add various procedures or components as appropriate. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. In addition, the features described for some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality, or structures and functionality that are supplementary or alternative to the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the present claims.

[0144] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0145] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). For example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0146] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Furthermore, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Furthermore, "determining" may include resolving, selecting, choosing, establishing, etc.

[0147] As used herein, in the context of sharing electronic signals and data between the various elements described herein, the term "connected to" may generally mean data communication between corresponding elements connected to each other. In some cases, the elements may be directly connected to each other, such as via one or more conductive traces, lines, or other conductive carriers capable of carrying signals and / or data between corresponding elements directly connected to each other. In other cases, the elements may be indirectly connected to each other, such as via one or more data buses or similar shared circuits and / or integrated circuit elements for conveying signals and data between corresponding elements indirectly connected to each other.

[0148] The method disclosed herein includes one or more steps or actions for implementing the method. The steps and / or actions of the method can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. In addition, the various operations of the method described above can be performed by any appropriate component capable of performing the corresponding function. The component may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Typically, in the case of operations illustrated in the drawings, those operations may have corresponding corresponding components with similar numbers plus functional components.

[0149] The following claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless otherwise specified, the reference to the singular element is not intended to mean "one and only one", but "one or more". Unless otherwise specified, the term "some" refers to one or more. No claim element should be interpreted according to the provisions of 35 U.S.C. § 112 (f), unless the phrase "parts for..." is used to expressly record the element, or in the case of a method claim, the phrase "step for..." is used to record the element. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known later to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recorded in the claims.

Claims

1. A processor-implemented method, comprising: Accessing feature tensors in neural networks; generating a feature mask by processing the feature tensor using a mask subnetwork, wherein the mask subnetwork is trained based at least in part on a polarization constraint and an activation constraint to generate the feature mask; generating a mask feature tensor based on the feature tensor and the feature mask; as well as The neural network is used to generate output inferences based at least in part on the mask feature tensor.

2. The processor-implemented method of claim 1 , wherein generating the output inference comprises: The masked feature tensor is output to a subsequent layer in the neural network.

3. The processor-implemented method of claim 1 , wherein generating the output inference is performed while processing input data during a runtime phase of the neural network.

4. The processor-implemented method of claim 1 , wherein generating the output inference is performed while processing training data during a training phase of the neural network.

5. The processor-implemented method of claim 1 , further comprising: generating a polarization loss based at least in part on the feature mask; as well as One or more parameters of the mask sub-network are refined based on the polarization loss. 6 . The processor-implemented method of claim 5 , wherein the polarization loss is generated based at least in part on an expected value of a product of the feature mask and an inverse of the feature mask.

7. The processor-implemented method of claim 5 , further comprising: generating an activation loss based at least in part on the feature mask; as well as One or more parameters of the mask sub-network are further refined based on the activation loss.

8. The processor-implemented method of claim 7, wherein the activation loss is generated based at least in part on an expected value of the feature mask and an activation rate, wherein the activation rate is a hyperparameter that controls what percentage of the feature tensor is masked.

9. The processor-implemented method of claim 7, further comprising: generating a task loss based at least in part on the mask feature tensor; as well as The one or more parameters of the mask sub-network are further refined based on the task loss.

10. The processor-implemented method of claim 9, further comprising: Prior to refining the one or more parameters of the mask subnetwork, the polarization loss, the activation loss, and the task loss are aggregated using a set of defined weights, wherein the set of defined weights controls polarization of the feature mask.

11. The processor-implemented method of claim 9, further comprising: One or more previous layers of the neural network are refined based at least in part on the polarization loss, the activation loss, and the task loss.

12. A system comprising: a memory including computer executable instructions; and a processor configured to execute the computer-executable instructions and cause the system to perform operations comprising: Accessing feature tensors in neural networks; generating a feature mask by processing the feature tensor using a mask subnetwork, wherein the mask subnetwork is trained based at least in part on a polarization constraint and an activation constraint to generate the feature mask; generating a mask feature tensor based on the feature tensor and the feature mask; and The neural network is used to generate output inferences based at least in part on the mask feature tensor.

13. The system of claim 12, wherein generating the output inference is performed while processing input data during a runtime phase of the neural network.

14. The system of claim 12, the operations further comprising: generating a polarization loss based at least in part on the feature mask; as well as One or more parameters of the mask sub-network are refined based on the polarization loss.

15. The system of claim 14, wherein the polarization loss is generated based at least in part on an expected value of a product of the feature mask and an inverse of the feature mask.

16. The system of claim 14, the operations further comprising: generating an activation loss based at least in part on the feature mask; as well as One or more parameters of the mask sub-network are further refined based on the activation loss.

17. The system of claim 16, wherein the activation loss is generated based at least in part on a desired value of the feature mask and an activation rate, wherein the activation rate is a hyperparameter that controls what percentage of the feature tensor is masked.

18. The system of claim 16, the operations further comprising: generating a task loss based at least in part on the mask feature tensor; as well as The one or more parameters of the mask sub-network are further refined based on the task loss.

19. The system of claim 18, the operations further comprising: Prior to refining the one or more parameters of the mask subnetwork, the polarization loss, the activation loss, and the task loss are aggregated using a set of defined weights, wherein the set of defined weights controls polarization of the feature mask.

20. The system of claim 18, the operations further comprising: One or more previous layers of the neural network are refined based at least in part on the polarization loss, the activation loss, and the task loss.

21. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform operations comprising: Accessing feature tensors in neural networks; generating a feature mask by processing the feature tensor using a mask subnetwork, wherein the mask subnetwork is trained based at least in part on a polarization constraint and an activation constraint to generate the feature mask; generating a mask feature tensor based on the feature tensor and the feature mask; as well as The neural network is used to generate output inferences based at least in part on the mask feature tensor.

22. The one or more non-transitory computer-readable media of claim 21, wherein generating the output inference is performed while processing input data during a runtime phase of the neural network.

23. The one or more non-transitory computer readable media of claim 21, the operations further comprising: generating a polarization loss based at least in part on the feature mask; as well as One or more parameters of the mask sub-network are refined based on the polarization loss.

24. The one or more non-transitory computer-readable media of claim 23, wherein the polarization loss is generated based at least in part on an expected value of a product of the feature mask and an inverse of the feature mask.

25. The one or more non-transitory computer readable media of claim 23, the operations further comprising: generating an activation loss based at least in part on the feature mask; as well as One or more parameters of the mask sub-network are further refined based on the activation loss.

26. The one or more non-transitory computer-readable media of claim 25, wherein the activation loss is generated based at least in part on an expected value of the feature mask and an activation rate, wherein the activation rate is a hyperparameter that controls what percentage of the feature tensor is masked.

27. The one or more non-transitory computer readable media of claim 25, the operations further comprising: generating a task loss based at least in part on the mask feature tensor; as well as The one or more parameters of the mask sub-network are further refined based on the task loss.

28. The one or more non-transitory computer readable media of claim 27, the operations further comprising: Prior to refining the one or more parameters of the mask subnetwork, the polarization loss, the activation loss, and the task loss are aggregated using a set of defined weights, wherein the set of defined weights controls polarization of the feature mask.

29. The one or more non-transitory computer readable media of claim 27, the operations further comprising: One or more previous layers of the neural network are refined based at least in part on the polarization loss, the activation loss, and the task loss.

30. A system comprising: Components for accessing feature tensors in neural networks; means for generating a feature mask by processing the feature tensor using a mask subnetwork, wherein the mask subnetwork is trained based at least in part on a polarization constraint and an activation constraint to generate the feature mask; means for generating a mask feature tensor based on the feature tensor and the feature mask; and Means for generating output inferences using the neural network based at least in part on the mask feature tensor.

Images