Scaling forward gradients with local optimization

Abstract

A plurality of model portions are determined from a machine learning model based on at least one criterion. A plurality of local optimization functions are determined for the plurality of model portions, respectively. Forward mode differentiation is performed for each of the plurality of model portions. Performing the forward mode differentiation includes applying a perturbation to an output of one or more model units of the model portion. Performing forward mode differentiation includes determining a gradient of the local optimization function for the model portion based at least in part on the perturbation. Performing the forward mode differentiation includes modifying one or more parameters of the model portion based on the gradient.

Description

[0001] Priority claim

[0002] This application is based on U.S. Provisional Application No. 63 / 410,584, filed on September 27, 2022, and claims priority to the U.S. Provisional Application, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to training machine learning models. More specifically, the present disclosure relates to forward gradient learning via activity perturbations. Background Art

[0004] Machine learning models are partly inspired by the structure and function of the human brain. In particular, some types of machine learning models (such as neural networks) include nonlinear model units (e.g., activation functions, etc.) that mimic the biological functions of neurons in the brain. Conventionally, many of these models are trained by adjusting the values ​​associated with these model units using backpropagation. However, many people believe that backpropagation is biologically unreasonable because it would require the brain to form symmetrical backward connections. Summary of the invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] An example aspect of the present disclosure relates to a computer-implemented method. The method includes determining, by a computing system including one or more computing devices, a plurality of model parts from a machine learning model based on at least one criterion. The method includes determining, by the computing system, a plurality of local optimization functions for the plurality of model parts, respectively. The method includes performing, by the computing system, a forward mode differential for each of the plurality of model parts. Performing the forward mode differential includes applying, by the computing system, a perturbation to an output of one or more model units of the model part. Performing the forward mode differential includes determining, by the computing system, a gradient of the local optimization function for the model part based at least in part on the perturbation. Performing the forward mode differential includes modifying, by the computing system, one or more parameters of the model part based on the gradient.

[0007] Another example aspect of the present disclosure relates to a computing system comprising one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations include determining a plurality of model portions from a machine learning model based on at least one criterion. The operations include determining a plurality of local optimization functions for the plurality of model portions, respectively. The operations include performing forward mode differentiation for each of the plurality of model portions. Performing the forward mode differentiation includes applying a perturbation to an output of one or more model units of the model portion. Performing the forward mode differentiation includes determining a gradient of the local optimization function for the model portion based at least in part on the perturbation. Performing the forward mode differentiation includes modifying one or more parameters of the model portion based on the gradient.

[0008] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations include determining multiple model parts from a machine learning model based on at least one criterion. The operations include determining multiple local optimization functions for the multiple model parts, respectively. The operations include performing forward mode differentiation for each of the multiple model parts. Performing the forward mode differentiation includes applying a perturbation to the output of one or more model units of the model part. Performing the forward mode differentiation includes: determining a gradient of the local optimization function for the model part based at least in part on the perturbation. Performing the forward mode differentiation includes modifying one or more parameters of the model part based on the gradient.

[0009] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0012] Figure 1A A block diagram of an example computing system that performs feed-forward gradient learning is depicted according to an example embodiment of the present disclosure.

[0013] Figure 1BDepicted is a block diagram of an example computing device performing forward-mode automatic differentiation according to an example embodiment of the present disclosure.

[0014] Figure 1C A block diagram of an example computing device that performs training of a machine learning model via forward-mode automatic differentiation is depicted according to an example embodiment of the present disclosure.

[0015] Figure 2 is a data flow diagram of an example architecture of a machine learning model including multiple parts and multiple corresponding local optimization functions according to some implementations of the present disclosure.

[0016] Figure 3 is a detailed data flow diagram of a local mixer residual block according to some implementations of the present disclosure, which local mixer residual block is one or more parts of a machine learning model or otherwise includes one or more parts of a machine learning model.

[0017] Figure 4 An example feature aggregator architecture according to some implementations of the present disclosure is shown.

[0018] Figure 5 Depicted is a flowchart illustration of an example method for performing forward gradient learning according to an example embodiment of the present disclosure.

[0019] Reference numerals repeated across multiple drawings are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0020] Overview

[0021] In general, the present disclosure relates to training machine learning models via forward gradient learning. More particularly, the present disclosure relates to performing forward mode differentiation in conjunction with activity perturbations for training machine learning models without backpropagation. In particular, forward mode differentiation has been proposed as a biologically plausible alternative to backpropagation for training machine learning models. However, standard forward gradient algorithms suffer from the curse of dimensionality (i.e., they cannot scale efficiently with the size of the machine learning model). Therefore, implementations of the present disclosure propose scaling the forward gradient using a localized optimization function.

[0022] For example, a computing system (e.g., a system providing a model training service) may determine a plurality of model parts from a machine learning model based on at least one criterion (e.g., a blockwise loss criterion, a patchwise loss criterion, a channel-wise loss criterion, etc.). The computing system may determine a plurality of local optimization functions (e.g., a localized greedy loss function, etc.) for the plurality of model parts, respectively. The computing system may perform forward mode differentiation for each of the plurality of model parts.

[0023] To perform forward mode differentiation, the computing system may apply a perturbation to the output of one or more model units of the model portion (e.g., adding noise to the input of an activation function, etc.). Based at least in part on the perturbation, the computing system may determine a gradient of the local optimization function for the model portion and modify one or more parameters of the model portion based on the gradient. In such a manner, the computing system may efficiently determine local gradients for each portion of the machine learning model in an efficient and accurate manner.

[0024] Various aspects of the present disclosure provide a variety of technical effects and benefits. As an example technical effect and benefit, the standard forward gradient algorithm suffers from the curse of dimensionality, which results in excessive computational requirements for training larger models. However, implementations of the present disclosure propose scaling the forward gradient by adding multiple local optimization functions for discrete portions of a machine learning model. By scaling the forward gradient using local optimization functions, implementations of the present disclosure can significantly reduce the computational resources (e.g., power, computational cycles, memory, storage, etc.) required to perform forward gradient training of a machine learning model.

[0025] Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.

[0026] Example Apparatus and Systems

[0027] Figure 1A A block diagram of an example computing system 100 that performs forward gradient learning according to an example embodiment of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0028] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0029] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operably connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0030] In some implementations, the user computing device 102 may store or include one or more models 120. For example, the model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a transformer model).

[0031] In some implementations, one or more models 120 may be received from a server computing system 130 via a network 180, stored in a user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, a user computing device 102 may implement multiple parallel instances of a single model 120 (e.g., to perform parallel services across multiple instances of a model).

[0032] Additionally or alternatively, one or more models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the models 140 may be implemented by the server computing system 140 as part of a web service. Thus, one or more models 120 may be stored and implemented at the user computing device 102 and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0033] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0034] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0035] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In the case where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0036] As described above, the server computing system 130 may store or otherwise include one or more models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a transformer model).

[0037] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0038] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or multiple processors that are operably connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0039] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques such as, for example, forward error propagation. For example, a loss function may be back-propagated or forward-propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters through multiple training iterations.

[0040] In particular, the model trainer 160 can train the machine learning model 120 / 140 via forward gradient learning. For example, the model trainer 160 can perform forward mode automatic differentiation to train the model 120 / 140 according to the forward gradient. In order to find the forward gradient, the model trainer 160 can perform active perturbations to determine the forward gradient. For example, the model trainer 160 can divide the model 120 / 140 into multiple parts and determine an optimization function for each part (e.g., according to certain criteria, etc.). The model trainer 160 can perturb the output of the model unit (e.g., neuron, etc.) included in each part of the model to determine the forward gradient according to the local optimization function associated with the part.

[0041] In some implementations, performing reverse error propagation may include performing truncated back propagation through time. Model trainer 160 may perform a variety of generalization techniques (e.g., weight decay, backoff, etc.) to improve the generalization ability of the model being trained. In particular, model trainer 160 may train models 120 and / or 140 based on a set of training data 162.

[0042] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0043] The model trainer 160 includes computer logic that is utilized to provide the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0044] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0045] In some implementations, the input of the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of image data, potential embedding of image data, encoded representation of image data, hash of image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change in image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded representation and / or a compressed representation of image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0046] In some implementations, the input of the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a conversion output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data with higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a predicted output.

[0047] In some implementations, the input of the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech conversion output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded representation and / or a compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data with a higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0048] In some implementations, the input of the machine learning model of the present disclosure may be latent coded data (e.g., a latent space representation of an input, etc.). The machine learning model may process the latent coded data to generate an output. As an example, the machine learning model may process the latent coded data to generate a recognition output. As another example, the machine learning model may process the latent coded data to generate a reconstruction output. As another example, the machine learning model may process the latent coded data to generate a search output. As another example, the machine learning model may process the latent coded data to generate a re-clustering output. As another example, the machine learning model may process the latent coded data to generate a prediction output.

[0049] In some implementations, the input to the machine learning model of the present disclosure may be statistical data. The statistical data may be, represent, or otherwise include data calculated and / or computed from some other data source. The machine learning model may process the statistical data to generate an output. As an example, the machine learning model may process the statistical data to generate an identification output. As another example, the machine learning model may process the statistical data to generate a prediction output. As another example, the machine learning model may process the statistical data to generate a classification output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a visualization output. As another example, the machine learning model may process the statistical data to generate a diagnostic output.

[0050] In some implementations, the input of the machine learning model of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0051] In some cases, the machine learning model can be configured to perform a task including encoding input data to achieve reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task can be an audio compression task. The input can include audio data, and the output can include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task can include generating an embedding for the input data (e.g., input audio or visual data).

[0052] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines the corresponding likelihood of each category in a predetermined set of categories for each pixel in one or more images. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted at the pixel between images in the network input for each pixel of one of the input images.

[0053] In some cases, the input includes audio data representing a spoken utterance, and the task is a speech recognition task. The output may include a text output mapped to the spoken utterance. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes a microprocessor performance task, such as branch prediction or memory address translation.

[0054] Figure 1A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training data set 162. In such implementations, the model 120 may be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0055] Figure 1B Depicted is a block diagram of an example computing device 10 that performs forward mode automatic differentiation according to an example embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0056] Computing device 10 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application can include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0057] like Figure 1B As shown, each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, context managers, device state components, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0058] Figure 1C A block diagram of an example computing device 50 performing training of a machine learning model via forward-mode automatic differentiation is depicted according to an example embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0059] The computing device 50 includes a plurality of applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0060] The central intelligence layer includes multiple machine learning models. For example, Figure 1C As shown, a corresponding machine learning model can be provided for each application, and the corresponding machine learning model can be managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50 or is implemented by the operating system in other ways.

[0061] The central intelligence layer may communicate with the central device data layer. The central device data layer may be a centralized data repository for computing devices 50. Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device (e.g., such as one or more sensors, context managers, device state components, and / or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0062] Figure 2is a data flow diagram of an example architecture of a machine learning model 200 including multiple parts and multiple corresponding local optimization functions according to some implementations of the present disclosure. In particular, the machine learning model 200 includes multiple model parts 202A, 202B, and 202C. As depicted, each model part 202A to 202C is a mixer residual block. However, it should be noted that the model parts can be determined in any manner and can therefore include any particular layer of the machine learning model. Parts 202A to 202C can be trained using multiple local optimization functions 204A, 204B, and 204C, respectively. For example, each local optimization function 204A to 204C can be attached to their corresponding model parts 202A to 202C using a linear projection layer.

[0063] More specifically, the model portions 202A-202C may be mixer residual blocks that together form the local mixer network 200. The local mixer network 200 may be trained using local losses 204A-204C. The local losses 204A-204C may include any type or manner of loss or optimization function.

[0064] For example, the local loss 204A can be or otherwise include a block-wise loss. The block-wise loss can be a loss function calculated at the "end" of each model portion 202A to 202C. The block-wise loss can be utilized to update the parameters of that particular model portion. In this way, the block-wise loss can be used as a "stop gradient" operator between modules.

[0065] For another example, the local loss 204B can be or otherwise include a per-tile loss applied along the spatial dimensions of a sensory input signal (such as an image). Here, each spatial "token" can represent a tile in the image. Parameters from each spatial location can be shared to improve data efficiency and reduce memory bandwidth utilization. In some implementations, this weight sharing effect can be simulated or otherwise achieved via knowledge distillation techniques.

[0066] For yet another example, the local loss 204C may be or otherwise include a groupwise loss that evaluates the channel dimension. To this end, the channels may be divided into multiple groups, and each group may be attached to a loss function (e.g., one of the local losses in the local loss 204). To prevent channel groups from communicating with each other, channels are only connected to other channels within the same group. For an individual model part g, the grouped linear layer may be calculated as .

[0067] In some implementations, a feature aggregator may be utilized to aggregate information from other groups and spatial patches before the local loss function 204 .

[0068] The local loss 204 may be applied to meet a specific learning goal. More specifically, the local loss 204 may jointly implement or be included in the supervised classification loss and the contrastive loss. These losses, which are most commonly used in image representation learning, may be applied to optimize the model in the manner described herein.

[0069] In some implementations, the local loss 204 implements a supervised classification loss. In some implementations, if the local loss 204 implements a supervised classification loss, the supervised classification loss can be obtained by attaching a shared linear layer (e.g., shared across p, g) to the cross entropy loss. This loss can have the same value across each group and patch position.

[0070] Additionally or alternatively, in some implementations, the local loss 204 implements a contrastive loss function. If the local loss 204 implements a contrastive loss function, the linear layer can become a linear feature projector. For example, assuming and are two different views of the nth example, then the loss of contrastive learning can be expressed as:

[0071]

[0072] It should be noted that a stop gradient operator is added on the second view to force the loss to decrease. This is due to the sharing of perturbations between the two views, and therefore, having the same perturbation will increase the dot product between the two views, which is undesirable from a representation learning perspective.

[0073] Figure 3 Detailed data flow diagram of a local mixer residual block according to some implementations of the present disclosure, which is one or more parts of a machine learning model or otherwise includes one or more parts of a machine learning model. In particular, as depicted, labeled mixing can be performed at 302, and the labeled mixing can consist of a linear layer and channels grouped in a channel mixing layer. In particular, layer 304 can include LN (layer norm), FC (fully connected layer), A (activation function), and T (transpose).

[0074] More specifically, at 302, a tag mix may be performed. This method of tag mix 302 may be similar to the method described in the previous section. Figure 2 To perform labeled blending, the channel

[0075] Figure 4An example feature aggregator architecture 400 is shown according to some implementations of the present disclosure. In particular, in implementations of the present disclosure, rather than performing average pooling to aggregate features from different spatial locations as performed in conventional designs, features are first concatenated across groups and then averaged across spatial locations. Copies of the same feature may be created using different stop gradient masks so that more local losses are obtained rather than global losses. The stop gradient mask ensures that perturbations in a spatial group correspond to its loss function. The numerical value of the calculated loss function may be similar or identical to that of a more conventional design.

[0076] More specifically, in order to perform aggregation without reducing the total dimensionality, the channel groups 402 may be first copied and transferred to each other. Feature aggregation may be performed without reducing the total dimensionality. To this end, each group other than the active group (e.g., the group in which communication is occurring) may be masked using a stop gradient mask so that the other groups do not affect the forward gradient calculation, which may be expressed as:

[0077]

[0078] Where p and g index the patch and group, respectively. Similarly, each spatial position can also be replicated, transmitted, masked, and then locally averaged, which is expressed as:

[0079]

[0080] The output of feature aggregation can be the same as the output of a regular global average pooling layer. The difference is that the above losses are repeated and different groups of patches are activated in each loss.

[0081] Example Method

[0082] Figure 5 Depicted is a flowchart illustration of an example method 500 for performing forward gradient learning according to an example embodiment of the present disclosure. Figure 5 The steps are depicted in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific order or arrangement described. The various steps of method 500 can be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0083] At 502, the computing system may determine a plurality of model portions from a machine learning model based on at least one criterion.

[0084] At 504 , the computing system may determine a plurality of local optimization functions for the plurality of model parts, respectively.

[0085] At 506, the computing system may perform forward mode differentiation for each of the plurality of model portions. Performing forward mode differentiation includes applying a perturbation to outputs of one or more model units of the model portion. Performing forward mode differentiation includes determining, by the computing system, a gradient of a local optimization function for the model portion based at least in part on the perturbation. Performing forward mode differentiation includes modifying one or more parameters of the model portion based on the gradient.

[0086] In some implementations, the perturbation applied may be a weight perturbation for performing a weight-perturbed forward gradient. i,j Let be the weighted connection between units i and j, and let f be the loss function. This can be done by extracting IId elements v from a zero-mean unit variance Gaussian distribution. i,j The gradient is estimated by sampling a random matrix of . The estimator can be expressed as:

[0087]

[0088] The estimator can be used to predict the random disturbance direction v ij is sampled and tested by performing a dot product using the forward mode to see how it compares to the true gradient Align, and then multiply the scalar alignment with the perturbation direction.

[0089] Additionally or alternatively, in some implementations, the perturbation applied may be an active perturbation for performing an active perturbation forward gradient. For example, assuming that a discrete-time rate-based formula is utilized. Further assuming that x i represents the activity of the ith presynaptic neuron and z i denotes the activity of the jth postsynaptic neuron before the nonlinear activation function, and let u j Indicates z j The forward gradient estimator perturbed by activity can be expressed as:

[0090]

[0091] in The inner product between and u is computed using forward mode automatic differentiation.

[0092] In some implementations, the first model portion includes multiple model units, and applying a perturbation to the output of one or more model units of the first model portion includes identifying a first portion of the multiple model units as rectified linear unit (ReLU) activation units, and identifying a second portion of the multiple model units as units other than ReLU activation units, and applying a perturbation to the output of the second portion of the multiple model units.

[0093] In some implementations, the plurality of model portions include a first model portion and a second model portion. Performing forward mode differentiation for each of the plurality of model portions includes applying a first perturbation to outputs of the one or more model units of the first model portion and applying a second perturbation to outputs of the one or more model units of the second model portion, wherein the second perturbation is different from the first perturbation.

[0094] In some implementations, based on one or more parameters right Each parameter in is modified, where x i is the presynaptic activity (e.g., activity before the model cell, etc.), is the rate of change of postsynaptic activity, and is the rate of change of the reward associated with the optimization function.

[0095] In some implementations, the at least one criterion includes a block-wise loss criterion for evaluating a depth dimension of the machine learning model. Determining a plurality of model parts from the machine learning model based on the at least one criterion includes determining a plurality of model parts from the machine learning model based on the block-wise loss criterion for evaluating a depth dimension of the machine learning model. Determining a plurality of local optimization functions for the plurality of model parts, respectively, includes determining a plurality of localized greedy loss functions for the plurality of model parts, respectively.

[0096] In some implementations, the at least one criterion includes a patch-wise loss criterion for evaluating a spatial dimension of an input to the machine learning model. Determining a plurality of model portions from the machine learning model based on the at least one criterion includes determining a plurality of model portions from the machine learning model based on the patch-wise loss criterion for evaluating a spatial dimension of an input to the machine learning model, wherein the plurality of model portions respectively correspond to a plurality of spatial regions of the input to the machine learning model.

[0097] In some implementations, at least one criterion includes a channel-by-channel loss criterion for evaluating a channel dimension of the machine learning model. Determining a plurality of model portions from the machine learning model based on the at least one criterion includes determining a plurality of model portions from the machine learning model based on a channel-by-channel loss criterion for evaluating a channel dimension of an input to the machine learning model, wherein the machine learning model includes a plurality of channels, and wherein each of the plurality of model portions includes a channel grouping of the plurality of channels.

[0098] Additional public content

[0099] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitioning of tasks and functions between and within components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0100] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, variations, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, variations, and / or additions to the subject matter that would be readily apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, variations, and equivalents.

Claims

1. A computer-implemented method comprising: determining, by a computing system including one or more computing devices, a plurality of model portions from the machine learning model based on at least one criterion; The computing system determines a plurality of local optimization functions for the plurality of model parts respectively; performing, by the computing system, forward mode differentiation for each of the plurality of model portions, wherein performing the forward mode differentiation comprises: applying, by the computing system, a perturbation to the output of one or more model units of the model portion; determining, by the computing system, a gradient of the local optimization function for the model portion based at least in part on the perturbation; and One or more parameters of the model portion are modified by the computing system based on the gradient.

2. The computer-implemented method of claim 1, wherein a first model portion of the plurality of model portions comprises a plurality of model cells; and Wherein applying the disturbance to the output of one or more model units of the first model part comprises: Identifying, by the computing system, a first portion of the plurality of model units as rectified linear unit (ReLU) activation units, and identifying a second portion of the plurality of model units as units other than ReLU activation units; as well as The perturbation is applied, by the computing system, to outputs of the second portion of the plurality of model units.

3. The computer-implemented method of claim 2, wherein the plurality of model portions comprises a first model portion and a second model portion; and Wherein performing the forward mode differentiation for each model part of the plurality of model parts comprises: applying, by the computing system, a first perturbation to the outputs of the one or more model elements of the first model portion; and A second perturbation is applied, by the computing system, to the outputs of the one or more model elements of the second model portion, wherein the second perturbation is different from the first perturbation.

4. The computer-implemented method of claim 1 , wherein the right Each parameter in is modified, where x i is presynaptic activity, is the rate of change of postsynaptic activity, and is the rate of change of the reward associated with the optimization function.

5. The computer-implemented method of claim 1 , wherein the at least one criterion comprises a block-wise loss criterion that evaluates a depth dimension of the machine learning model; wherein determining the plurality of model portions from the machine learning model based on the at least one criterion comprises determining, by the computing system, the plurality of model portions from the machine learning model based on the block-wise loss criterion evaluating the depth dimension of the machine learning model; and Wherein determining the multiple local optimization functions for the multiple model parts respectively includes determining, by the computing system, multiple localized greedy loss functions for the multiple model parts respectively.

6. The computer-implemented method of claim 1 , wherein the at least one criterion comprises a patch-wise loss criterion that evaluates a spatial dimension of an input to the machine learning model; and Wherein determining the multiple model parts from the machine learning model based on the at least one criterion includes determining, by the computing system, the multiple model parts from the machine learning model based on the block-by-block loss criterion of evaluating the spatial dimension of the input of the machine learning model, wherein the multiple model parts respectively correspond to multiple spatial regions of the input of the machine learning model.

7. The computer-implemented method of claim 1 , wherein the at least one criterion comprises a channel loss criterion that evaluates a channel dimension of the machine learning model; and wherein determining the plurality of model parts from the machine learning model based on the at least one criterion comprises determining, by the computing system, the plurality of model parts from the machine learning model based on the channel loss criterion evaluating the channel dimension of the input to the machine learning model, wherein the machine learning model comprises a plurality of channels, and wherein each of the plurality of model parts comprises a channel grouping of the plurality of channels.

8. A computing system comprising: one or more processors; as well as one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: determining a plurality of model parts from the machine learning model based on at least one criterion; determining a plurality of local optimization functions for the plurality of model parts respectively; Performing forward mode differentiation for each of the plurality of model portions, wherein performing the forward mode differentiation comprises: applying a perturbation to the output of one or more model units of the model portion; determining a gradient of the local optimization function for the model portion based at least in part on the perturbation; and One or more parameters of the model portion are modified based on the gradient.

9. The computing system of claim 8, wherein a first model portion of the plurality of model portions comprises a plurality of model units; and Wherein applying the disturbance to the output of one or more model units of the first model part comprises: Identifying a first portion of the plurality of model units as rectified linear unit (ReLU) activation units, and identifying a second portion of the plurality of model units as units other than ReLU activation units; as well as The perturbation is applied to outputs of the second portion of the plurality of model units.

10. The computing system of claim 8, wherein the plurality of model portions include a first model portion and a second model portion; and Wherein performing the forward mode differentiation for each model part of the plurality of model parts comprises: applying a first perturbation to the outputs of the one or more model elements of the first model portion; and A second perturbation is applied to the outputs of the one or more model elements of the second model portion, wherein the second perturbation is different from the first perturbation.

11. The computing system of claim 8, wherein according to the one or more parameters right Each parameter in is modified, where x i is presynaptic activity, is the rate of change of postsynaptic activity, and is the rate of change of the reward associated with the optimization function.

12. The computing system of claim 8, wherein the at least one criterion comprises a block-wise loss criterion that evaluates a depth dimension of the machine learning model; wherein determining the plurality of model portions from the machine learning model based on the at least one criterion comprises determining the plurality of model portions from the machine learning model based on the block-wise loss criterion evaluating the depth dimension of the machine learning model; and Wherein determining the multiple local optimization functions for the multiple model parts respectively includes determining multiple localized greedy loss functions for the multiple model parts respectively.

13. The computing system of claim 8, wherein the at least one criterion comprises a patch-wise loss criterion that evaluates a spatial dimension of an input to the machine learning model; and Wherein determining the multiple model parts from the machine learning model based on the at least one criterion includes determining the multiple model parts from the machine learning model based on the block-by-block loss criterion for evaluating the spatial dimension of the input of the machine learning model, wherein the multiple model parts respectively correspond to multiple spatial regions of the input of the machine learning model.

14. The computing system of claim 8, wherein the at least one criterion comprises a channel loss criterion that evaluates a channel dimension of the machine learning model; and wherein determining the plurality of model parts from the machine learning model based on the at least one criterion comprises determining the plurality of model parts from the machine learning model based on the channel loss criterion evaluating the channel dimension of the input to the machine learning model, wherein the machine learning model comprises a plurality of channels, and wherein each of the plurality of model parts comprises a channel grouping of the plurality of channels.

15. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: determining a plurality of model parts from the machine learning model based on at least one criterion; determining a plurality of local optimization functions for the plurality of model parts respectively; Performing forward mode differentiation for each of the plurality of model portions, wherein performing the forward mode differentiation comprises: applying a perturbation to the output of one or more model units of the model portion; determining a gradient of the local optimization function for the model portion based at least in part on the perturbation; and One or more parameters of the model portion are modified based on the gradient.

16. The one or more non-transitory computer-readable media of claim 15, wherein a first model portion of the plurality of model portions comprises a plurality of model units; and Wherein applying the disturbance to the output of one or more model units of the first model part comprises: Identifying a first portion of the plurality of model units as rectified linear unit (ReLU) activation units, and identifying a second portion of the plurality of model units as units other than ReLU activation units; as well as The perturbation is applied to outputs of the second portion of the plurality of model units.

17. The one or more non-transitory computer-readable media of claim 15, wherein the plurality of model portions comprises a first model portion and a second model portion; and Wherein performing the forward mode differentiation for each model part of the plurality of model parts comprises: applying a first perturbation to the outputs of the one or more model elements of the first model portion; and A second perturbation is applied to the outputs of the one or more model elements of the second model portion, wherein the second perturbation is different from the first perturbation.

18. The one or more non-transitory computer readable media of claim 15, wherein the one or more parameters right Each parameter in is modified, where x i is presynaptic activity, is the rate of change of postsynaptic activity, and is the rate of change of the reward associated with the optimization function.

19. The one or more non-transitory computer-readable media of claim 15, wherein the at least one criterion comprises a block-wise loss criterion that evaluates a depth dimension of the machine learning model; wherein determining the plurality of model portions from the machine learning model based on the at least one criterion comprises determining the plurality of model portions from the machine learning model based on the block-wise loss criterion evaluating the depth dimension of the machine learning model; and Wherein determining the multiple local optimization functions for the multiple model parts respectively includes determining multiple localized greedy loss functions for the multiple model parts respectively.

20. The one or more non-transitory computer-readable media of claim 15, wherein the at least one criterion comprises a patch-wise loss criterion that evaluates a spatial dimension of an input to the machine learning model; and Wherein determining the multiple model parts from the machine learning model based on the at least one criterion includes determining the multiple model parts from the machine learning model based on the block-by-block loss criterion for evaluating the spatial dimension of the input of the machine learning model, wherein the multiple model parts respectively correspond to multiple spatial regions of the input of the machine learning model.

Images