Torchdeq balancer for depth balanced models

Abstract

The invention relates to a library of TORCHDEQ for depth balance models. Methods and systems are disclosed that allow a user to define, train, and deploy a depth balance model. Decoupling and structured interfaces allow a user to easily customize the depth balance model. The disclosed system supports many different forward and backward resolvers, normalization and regularization methods.

Description

Technical Field

[0001] The present disclosure relates generally to the field of artificial intelligence. More specifically, disclosed embodiments relate to the design, training, and use of machine learning models including neural networks. Background Art

[0002] Deep Equilibrium Models ("DEQs") are a recently developed class of implicit neural networks. DEQs have an emerging community. DEQs are receiving increasing attention in theory. Stability and acceleration are active research topics in the DEQ community. DEQs show attractive generalization performance, interpretability, and robustness on semantic segmentation, optical flow, detection, inverse problems, meta-learning, object-centric learning, ensemble prediction, control, spiking neural networks, machine translation, normalized flows, and graph learning.

[0003] Scientific software has helped deep learning become more sophisticated, modular, and scalable. From foundational deep learning libraries like PyTorch, Tensorflow, and JAX to comprehensive model collections like huggingface (zoo), and domain-specific libraries like fairseq for language models and timm for visual backbones, the contribution of open source software cannot be underestimated. Summary of the invention

[0004] Recently, many deep learning libraries for neural dynamics have emerged, like differentiable optimization or differential equations, e.g., theseus, torchopt, torchdiffeq, torchdyn, betty, and pypose. But none of them are specifically designed for DEQ and validated to upgrade to modern DEQ with good stability. Furthermore, none of them are specifically designed for DEQ and validated to host model collections of implicit models. The disclosed embodiments move towards this and widely support state-of-the-art deep balanced models.

[0005] Some disclosed embodiments include methods comprising: receiving user input identifying a deep balanced model and a training dataset; and training the deep balanced model on the training dataset, wherein the training comprises performing a normalization method according to: Wherein f is a deep balanced model, W is a weight matrix, g is a learnable scaling factor, ° is a row-by-row multiplication, t is a threshold for clipping the scaling factor g, and N is a computation of a norm of the weight matrix W. Some disclosed embodiments include a method comprising: receiving a user input identifying a deep balanced model and identifying a training dataset; and training the deep balanced model on the training dataset, wherein the training comprises executing forward and backward solvers to perform forward and backward passes through the deep balanced model, and wherein the forward and backward solvers are identified in the user input. Some disclosed embodiments include a system comprising: one or more processors; and a non-volatile memory comprising processor-executable instructions, which, when executed by the one or more processors, cause the system to perform operations comprising: receiving user input identifying a deep balanced model and identifying a training data set; and training the deep balanced model on the training data set, wherein the training comprises: executing forward and backward solvers to perform forward and backward passes through the deep balanced model, wherein the forward and backward solvers are identified in the user input; and performing one or more of: automatic normalization of weight tensors; Jacobian regularization; and fixed point correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 The figure illustrates example code according to the disclosed embodiment.

[0007] Figure 2 FIG. 1 is a flowchart illustrating an example method for training a DEQ according to a disclosed embodiment.

[0008] Figure 3 The figure shows a block diagram of a system according to the disclosed embodiment.

[0009] Figure 4 The diagram illustrates an example embodiment of a general purpose computer system according to the present disclosure. DETAILED DESCRIPTION

[0010] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to use the present invention in different ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any of the figures may be combined with the features illustrated in one or more of the other figures to produce embodiments that are not explicitly illustrated or described. The combination of the illustrated features provides a representative embodiment of a typical application. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of the present disclosure may be desired.

[0011] Currently, training and applying DEQ is done in an ad hoc manner, with a variety of techniques scattered in the literature. The present disclosure systematically revisits DEQ, and example embodiments include a PyTorch-based library, referred to herein as TorchDEQ, that allows users to define, train, and infer using DEQ on multiple domains with minimal code and best practices. The disclosed embodiments, which may be referred to herein as the "DEQ Collection," support six disclosed implicit models across different domains. The disclosed embodiments include a joint framework that combines best practices across all models and is used to significantly improve the performance, training stability, and efficiency of DEQ on ten datasets across all six models in the DEQ Collection.

[0012] 1.0 Introduction

[0013] Unlike traditional feedforward models that use a fixed-size computational graph to compute its output, DEQ defines its output as a fixed point of a nonlinear system, i.e.,

[0014] z ＊ =f θ (z ＊ , x).

[0015] Where x represents the input of the network, z ★ is its output. This formula has several significant benefits: DEQ can be interpreted as a fixed-point iteration z 1+1 =f θ (z 1, x) and thus provide rich representations using relatively few parameters; they only require specifying a “single” layer in their architecture design; they can be trained with substantially less memory, since only the final fixed point z★ needs to be stored for backpropagation; they (usually) recover path-independent solutions, where the final output z★ is independent of its initialization; finally, they intuitively allow a separation between the “definition” of the network and the “solver” that computes the fixed points, a separation that mirrors many settings in, for example, differential equation solvers or optimizers.

[0016] Unfortunately, DEQs are often difficult to train and challenging to deploy in practice. Training DEQs often leads to unstable systems, and methods for addressing these stability challenges are scattered across different papers in the literature; similarly, backpropagation in these networks can be done in many different ways, i.e., by unrolling, by implicit differentiation, or via inexact "phantom" gradients; finally, architectures and balance solvers often have to be re-selected for different applications. We believe that these challenges fundamentally limit the broad impact of DEQs in deep learning.

[0017] To this end, we developed a modular library in this paper, which can be referred to as TorchDEQ in this paper. TorchDEQ is a well-designed, full-featured, and PyTorch-based library for building and deploying DEQs. It provides a decoupled and structured interface that allows users to customize their own general DEQs for arbitrary tasks with a minimal amount of code. The library supports many different forward solvers, backward pass methods, normalization and regularization methods, implementing best practices across the board.

[0018] The disclosed embodiments of the modular library (i.e., TorchDEQ) may include a collection of models for DEQ, which may be referred to herein as the "DEQ collection". The disclosed embodiments have implemented six disclosed implicit models via TorchDEQ, including the DEQ transformer, the multi-scale deep balanced model ("MDEQ"), the implicit graph neural network ("IGNN"), the deep balanced optical flow estimator ("DEQ-Flow"), the implicit layer of implicit representation ("DEQ-INR"), and the deep balanced method of diffusion model (DEQ-DDIM). In terms of performance, stability, and efficiency, all of these models have achieved consistently better results than those reported in the original papers and released code.

[0019] 2.0TorchDEQ

[0020] Embodiments of TorchDEQ and example features supported in TorchDEQ are disclosed in more detail below. The example embodiments disclosed below may include a description of DEQ; an example interface of TorchDEQ, including example code; and a computational graph design of TorchDEQ, which may highlight different methods for approximating backward pass and other DEQ strategies.

[0021] 2.1DEQ

[0022] Given an input data pair (x, y) and a loss function L, DEQ is obtained from the neural network f θ The input injection u(x) is an implicit mapping to a fixed point z. The training objectives are as follows:

[0023]

[0024] z * =f θ (z * ,u(x))

[0025] where u(x) is the injection function and y(z * ) is the decoder that produces the model predictions. In the forward pass, the “infinite depth” balanced representation z * It can be solved by black-box solvers, for example, fixed-point iteration, Anderson acceleration, or Broyden's method. Despite these "infinite layers", differentiation through this fixed-point system has an elegant solution.

[0026] Theorem 2.1 By the implicit function theorem (“IFT”), under mild conditions, the gradient of DEQ can be expressed as

[0027]

[0028] in is the gradient g T This solution requires solving another “mirror” linear fixed-point system in the backward pass to obtain the gradient g T .

[0029]

[0030] This backward balancing system is itself a (linear) fixed-point operation, so similar (or even simpler) techniques as the forward pass can be used. Thus, we can do this without storing the function f θ In the case of a computation graph of an activation or black-box solver, differentiation is performed via DEQ using O(1) memory complexity (i.e., independent of the number of solver steps).

[0031] 2.2 Sample Code and Interface

[0032] The aspects listed above have one thing in common: in all cases, the main properties of DEQ are θ The specific choice of DEQ is agnostic. That is, a unified and modular interface for implementing DEQ can be constructed for different functional designs, univariate, multivariate, or even multiresolution equilibrium systems. However, implementing DEQ remains challenging because all the components shown above and further extensions require skilled design and validation. As discussed below, differences in implementation can significantly affect downstream performance, stability, and efficiency.

[0033] Figure 1 The diagram illustrates sample code for making and using the disclosed embodiments, including using the TorchDEQ library and methods for building and training DEQ.

[0034] The get_deq operation can return a DEQ solver as a Pytorch module. The user needs to pass a functor f that defines a function for f θ The function call, for the fixed-point solver, has input injection x and initialization z 0 Fixed point reuse can be done with a previous fixed point selected by the user. For multivariable equilibrium systems with different tensor shapes, like Z * =[h * , c * ], the functor can be rewritten with only minor tweaks; TorchDEQ can do the rest of the tuning of the gradients and solvers.

[0035] f=lambda h, c: self.dq_func((h, c), x)

[0036] The operations apply_norm and reset_norm can automatically apply normalization to the balancing module f θ The weight tensors in and the values ​​used for each weight tensor can be recomputed before the next training step. More details are disclosed in Section 2.5.

[0037] The add_deq_args operation can provide a decorator for the commonly used Python argument parser. Users of the disclosed embodiments can simply call add_deq_args(parser) and customize the behavior of DEQ via the command line. This design has been adopted by community-trusted libraries like fairseq and timm.

[0038] python train.py --ift --f_solver anderson --b_solver broyden

[0039] For example, the command above starts training using implicit differentiation as the backward solver, Anderson acceleration as the forward solver, and Broyden's method as the backward solver.

[0040] TorchDEQ’s compact and modular interface enables users to focus on how to abstract, formulate, and define their requirements as a balanced model f θ And design its interaction with other explicit layers (like injection and decoder). The modular design of TorchDEQ creates an "abstract" for DEQ and minimizes the cost of learning, implementing, and tuning DEQ. In the following sections, we will introduce the features of TorchDEQ and its control commands.

[0041] 2.3 Backward Pass

[0042] TorchDEQ can create computational graphs for solvers and gradients internally. Users can receive a set of tensors registered with gradients. Users can process the output of implicit models just as they do with explicit layers and tensors. However, when computing gradients, TorchDEQ can transparently compute the backward pass using specialized methods. TorchDEQ can support two types of backward passes, implicit differentiation ("IFT") and phantom gradients ("PG"). In practice, both types and / or their combinations may be sufficient to provide empirically appealing results in a reasonable time frame.

[0043] Implicit differentiation (i.e., IFT). Implicit differentiation is a standard method for differentiation through fixed points. As shown in Equation (3), the implicit gradient can be solved from another linear fixed point system in the backward pass. The user can declare IFT with --ift, set the backward solver with --b_solver broyden, and set the solver configuration, such as the maximum solver step size --b_thres30 and the stopping criterion --b_eps 0.001.

[0044] Phantom Gradient (PG). Phantom Gradient is a structured approximation of IFT that preserves the descent direction.

[0045]

[0046] in Valid gradient updates are retained, and A is the approximate Jacobian defined below.

[0047] Phantom gradients can be applied to the computation graphs of various solvers, similar to IFT for differentiation fixed points. The previous view was that IFT was an exact gradient and PG was an inexact gradient. However, there may be numerical errors in solving the forward pass and the IFT. Therefore, distinguishing between exact and inexact gradients may be unnecessary, and instead they can be called together as the backward pass.

[0048] The instantiation of PG used in different implicit models consists of unfolding the balancing module f θ At the (approximate) fixed point z of the solution p On, with damping factor τ

[0049] z p+1 =τf θ (z p )+(1-τ)z p , (5)

[0050] It defines the following matrix A,

[0051]

[0052] Users can invoke PG, for example, with --grad 5 --tau 0.6 in conjunction with Broyden's method as a forward solver.

[0053] In the disclosed embodiments, no separate support for Backpropagation Through Time (“BPTT”) and its truncated versions is defined, since they can be expressed as special cases of PG given τ=1.0 and with the forward solver removed, i.e., the solver and gradients are determined only by f θ The expansion process definition of . The computational graph of BPTT-12 is defined using the command --f_thres 0 --grad 12 --tau 1.0.

[0054] The disclosed embodiments may include an interface mem_gc to reduce the memory usage of any unfolded computation graph. Users can trade 1.5 times the training time for a much lower constant memory overhead via gradient checkpointing.

[0055] 2.4 Solver

[0056] Prior art DEQ projects typically write their own fixed-point solvers. These task-dependent solvers can be inefficient, or even sometimes unreliable, when applied to different domains. To address this problem, the disclosed embodiments implement, verify, and improve solver implementations for TorchDEQ. Batch processing and memory access for multivariable systems can be optimized. In the disclosed embodiments, the solver can be reliable in a variety of tasks, robust in different settings, and agnostic to the scale of the fixed-point equations and their tensor shapes. Therefore, the disclosed embodiments can lead to significant efficiency improvements in multivariable and multi-scale equilibrium systems.

[0057] In TorchDEQ, the following solvers may be supported. To call these solvers, users can type --f_solver or --b_solver in the command line along with the solver name and their maximum number of iterations --f_thres 20 and stopping criteria --f_eps 1e-2. In addition, keyword arguments may be passed to the DEQ class to tune the solver. For example, an example of custom Anderson acceleration may be implemented as:

[0058] z_out, info = self.deq_solver(f, z, solver_kwargs = {'tau': 0.8, 'm': 6}). Naive solver ( Solver). Fixed point iteration is to solve the fixed point z * The classical solver for is described by the following numerical scheme

[0059] z k+1＝ f θ (z k ). (7)

[0060] Its convergence can be verified by f θ The bounded Jacobian radius is guaranteed. The user can call fixed-point iterations by typing --f_solvernaive_solver.

[0061] Anderson Solver. Anderson speedup, or Anderson hybrid (i.e., type I Anderson speedup) is an acceleration technique for fixed-point iteration using a linear combination of the past m+1 fixed-point estimates. Its update uses this numerical scheme,

[0062]

[0063] where τ is the damping factor, which defaults to 1.0. Given g θ (z) = f θ (z)-z,G k=[g θ (z k-m ), ..., g θ (z k )], solve from the following formula

[0064]

[0065] Users can call Anderson acceleration via --f_solver anderson and refer to the above examples for adjustments.

[0066] Broyden solver. Broyden's method is a quasi-Newton solver for fixed-point equations. By maintaining a buffer, Broyden's method approximates the Jacobian inverse in Newton's method through a low-rank update.

[0067] z k+1 =z k -α·B k g θ (z k ), (10)

[0068] Among them B k is to use Δz k =z k -z k-1 and Δg k =g θ (z k )-g θ (z k-1 ) Approximation of

[0069]

[0070] Equation (10) can be written as a matrix-vector product, which further avoids storing B in memory. k .

[0071]

[0072] Among them U k and V k denotes the m past estimates for low rank approximation via the Sherman-Morrison formula. The user can invoke Broyden's method by setting solver_kwargs = {'l_thres': m}, for example via --f_solverbroyden or its limited memory version.

[0073] 2.5 Normalization

[0074] Normalization techniques are crucial to modern deep balancing models. Unlike popular normalization methods applied to representations, DEQ additionally relies on normalization of weight tensors, such as weight normalization, spectral normalization. In the disclosed embodiments, a DEQ version of the normalization technique is supported in TorchDEQ.

[0075] Normalization significantly smoothes the fixed point landscape of z given input data x and makes the fixed points easier to solve in practice. Previous literature seems to underestimate this effect.

[0076] For the weight matrix W∈R mxn , weight normalization (“WN”) parameterizes the weights as:

[0077]

[0078] where ||·|| denotes the vector L2 norm, g is a learnable scaling factor, and spectral normalization (“SN”) specifies

[0079]

[0080] where ‖·‖2 represents the spectral norm, which can be calculated by the power iteration method.

[0081] In the disclosed embodiment, TorchDEQ supports the above two normalization methods via the formalism,

[0082]

[0083] where ° is a row-wise multiplication, and N represents the associated “norm” used to compute the weight matrix. After WN, a learnable scaling g can be added to the DEQ SN, which can enable the SN to match the performance and generalization of WN on DEQ flows.

[0084] Inspired by gradient clipping, the disclosed embodiments introduce an operation that significantly stabilizes the training of implicit graph neural networks on node classification, namely, by clipping the rescaling factor to a threshold t,

[0085]

[0086] This can be enabled in TorchDEQ via --norm_clip and --norm_clip_value t.

[0087] The classic implementation of WN and SN resets the weights in each forward call. However, this is a visible waste for DEQ because the balancing module f θ It will be called multiple times until convergence. Then by the number f θThe same weight parameterization is applied to all function calls. In addition, the decoration normalization must be manually encoded for each module using the state-of-the-art PyTorch implementation.

[0088] Alternatively, in the public embodiment of TorchDEQ, a unified interface is provided to automatically decorate the entire balancing module f through apply_norm θ (with keyword argument filter_out to skip some modules) and reset_norm for resetting without wasting computation. After training, the normalization decorations can be removed by remove_norm, as they do not change the model but ease its training.

[0089] TorchDEQ users can specify --norm_type weight_norm for WN, --norm_type spectral_norm for SN, and additionally --norm_no_scale to remove the learnable scaling g.

[0090] 2.6 Regularization

[0091] In most modern interpretations of DEQ learning, a conceptual change is taking place. Rather than viewing the DEQ model as merely learning a well-behaved (implicit) fixed-point mapping x→z * , they are considered to learn a smooth and flat equilibrium landscape x→B(z) that contains a unique and well-behaved fixed point z * .

[0092] Balancing module θ The regularization of ensures fast convergence to a fixed point z * , despite the use of a simple solver. The correspondence between the equilibrium landscape B(z) and the loss landscape shows that the fixed point error ||f θ (z)-z|| and loss||f θ (z)-z|| has a strong correlation. Path independence, i.e. convergence to a stable state regardless of initialization, can allow better utilization of test time computation. Therefore, the prior art approach treats DEQ as a dynamic implicit neural network, which can obtain strong results in the early equilibrium solution process and as the fixed point z is approached * The disclosed embodiment of TorchDEQ supports regularization techniques that exploit these DEQ properties and balance the landscape.

[0093] Jacobian Regularization ("JR"). Jacobian regularization penalizes the Jacobian spectral radius according to the following formula: Upper limit

[0094]

[0095] Computationally, this is achieved by adding a loss term using a randomized Hutchinson trajectory estimator, e.g., sampling from a standard Gaussian ∈,

[0096]

[0097] The disclosed embodiment of TorchDEQ may provide an interface jac_reg, which takes f θ (z) and z are used to calculate the JR loss.

[0098] Fixed-point correction (“FC”). Fixed-point correction helps learn a smooth equilibrium landscape by regularizing intermediate states during the solution from a fixed point. Given convergence to z * sequence Correction can both decode the state and supervise the prediction

[0099]

[0100] Or apply Jacobian regularization to this sequence,

[0101]

[0102] Figure 2 The figure illustrates an example method 200 for training and using DEQ according to a disclosed embodiment. In some disclosed embodiments, method 200 is performed by TorchDEQ. At operation 202, the method receives input data from a user. The input data may identify a training data set and identify a deep balanced model. For the first example, the training data set may be identified by the user selecting an item from a displayed menu. The training data set may have been preloaded into TorchDEQ, or it may be loaded into TorchDEQ in response to user input. Optionally, the user input may also identify an injection module and / or a decoder module. In a disclosed embodiment, TorchDEQ allows a user to customize a forward and / or backward solver from the TorchDEQ library. For example, the input data from the user may include parameters for modifying one or more forward and / or backward solvers.

[0103] At operation 204, TorchDEQ trains the deep balanced model. To perform forward and backward passes through non-DEQ modules, TorchDEQ may use default PyTorch functionality. To perform forward and backward passes through DEQ modules, TorchDEQ may use forward and / or backward DEQ solvers from the TorchDEQ library. The forward and / or backward solvers used by TorchDEQ may be forward and / or backward solvers customized by modifying parameters according to parameters provided by a user in input data from the user. During training of the deep balanced model, TorchDEQ may perform one or more of automatic normalization of weight tensors, Jacobian regularization, and fixed point correction.

[0104] At operation 206, TorchDEQ can perform a forward pass through the trained deep balanced model with the DEQ solver to generate predictions. For example, the trained deep balanced model can perform predictions on the test data.

[0105] Figure 3 FIG. 3 shows a block diagram of a system 300 according to a disclosed embodiment. For example, the system 300 can be an implementation of TorchDEQ. Input data 310 can be input into a model 320. The input data 310 can be, for example, the Figure 2 The model 320 includes an optional injection module 322, a DEQ 324, and an optional decoder 326. The model 320 outputs a prediction 330.

[0106] Figure 4 A block diagram of an example embodiment of a general purpose computer system 400 is shown. The computer system 400 may include a set of instructions that may be executed to cause the computer system 400 to perform any one or more of the methods or computer-based functions disclosed herein. For example, the computer system 400 may include executable instructions to perform TorchDEQ functions. The computer system 400 may be connected to other computer systems or peripheral devices via a network. Additionally, the computer system 400 may include or be included in other computing devices.

[0107] like Figure 4As shown, the computer system 400 may include one or more processors 402. The one or more processors 402 may include, for example, one or more central processing units (CPUs), one or more graphics processing units (GPUs), or both. The computer system 400 may include a main memory 404 and a static memory 406, which may communicate with each other via a bus 408. As shown, the computer system 400 may also include a video display unit 410, such as a liquid crystal display (LCD), a projection television display, a flat panel display, a plasma display, or a solid-state display. In addition, the computer system 400 may include an input device 412, such as a remote control device with a wireless keypad, a keyboard, a microphone coupled to a speech recognition engine, a camera such as a video camera or a still camera, or a cursor control device 414 such as a mouse device. The computer system 400 may also include a disk drive unit 416, a signal generating device 418 such as a speaker, and a network interface device 420. The network interface 420 may enable the computer system 400 to communicate with other systems via a network 428. For example, the network interface 420 may enable the machine learning system 120 to communicate with a database server (not shown) or a controller (not shown) in a manufacturing system.

[0108] In some embodiments, Figure 4 As depicted, disk drive unit 416 may include one or more computer-readable media 422 in which one or more sets of instructions 424, such as software, may be embedded. For example, instructions 424 may embody one or more methods or functions, such as those disclosed herein. In certain embodiments, instructions 424 may reside completely or at least partially within main memory 404, static memory 406, and / or within processor 402 during execution by computer system 400. Main memory 404 and processor 402 may also include computer-readable media.

[0109] In some embodiments, dedicated hardware implementations, such as application specific integrated circuits, programmable logic arrays, and other hardware devices, can be constructed to implement one or more methods or functions described herein. Applications that may include the apparatus and systems of various embodiments may broadly include various electronic and computer systems. One or more embodiments described herein may use two or more specific interconnected hardware modules or devices to implement functionality, wherein related control and data signals may be communicated between and through the modules, or as part of an application specific integrated circuit. Therefore, the present system includes software, firmware, and hardware implementations, or a combination thereof.

[0110] Although the computer-readable medium is shown as a single medium, the term "computer-readable medium" includes a single medium or multiple media, such as a centralized or distributed database, and / or associated caches and servers that store one or more sets of instructions. The term "computer-readable medium" should also include any medium that can store or encode a set of instructions executed by a processor or cause a computer system to perform any one or more of the methods or functions disclosed herein.

[0111] In some embodiments, some or all of the computer-readable media will be non-transitory media. In a specific non-limiting exemplary embodiment, the computer-readable medium may include a solid-state memory, such as a memory card or other package that accommodates one or more non-volatile read-only memories. In addition, the computer-readable medium may be a random access memory or other volatile rewritable memory. In addition, the computer-readable medium may include a magneto-optical or optical medium, such as a disk or tape or other storage device, to capture a carrier signal, such as a signal transmitted through a transmission medium.

[0112] Although exemplary embodiments are described above, this does not mean that these embodiments describe all possible forms included in the claims. The words used in the specification are descriptive words, not restrictive, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As mentioned above, the features of the various embodiments can be combined to form further embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations with respect to one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, strength, durability, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. Therefore, embodiments described as being less desirable than other embodiments or prior art implementations with respect to one or more features are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A method comprising: receiving user input identifying a deep balanced model and identifying a training dataset; and The deep balanced model is trained on the training dataset, wherein the training comprises performing a normalization method according to the following formula: where f is the deep balanced model, W is the weight matrix, g is the learnable scaling factor, ° is the row-wise multiplication, t is the threshold for clipping the scaling factor g, and N is the calculation for the norm of the weight matrix W. The method of claim 1 , wherein the user input identifies an injection module.

3. The method of claim 1, wherein the user input identifies the decoder module.

4. The method of claim 1, wherein the training comprises executing a forward and backward solver to perform forward and backward passes through the deep balanced model.

5. The method of claim 4, wherein one or more of the forward and backward solvers are modified by parameters included in user input.

6. The method of claim 1, wherein the training comprises performing one or more of: Automatic normalization of weight tensors; Jacobian regularization; and Fixed point correction.

7. The method of claim 4, wherein the training comprises performing one or more of: Automatic normalization of weight tensors; Jacobian regularization; and Fixed point correction.

8. A method comprising: receiving user input identifying a deep balanced model and identifying a training dataset; and The deep balanced model is trained on the training dataset, wherein the training comprises executing forward and backward solvers to perform forward and backward passes through the deep balanced model, and wherein the forward and backward solvers are identified in the user input.

9. The method according to claim 8, wherein: One or more forward and backward solvers are modified by parameters included in the user input.

10. The method of claim 8, wherein the training comprises performing one or more of the following: Automatic normalization of weight tensors; Jacobian regularization; and Fixed point correction. The method of claim 8 , wherein the user input identifies an injection module.

12. The method of claim 8, wherein the user input identifies a decoder module.

13. The method of claim 8, wherein the training comprises performing a normalization method according to the following formula: where f is the deep balanced model, W is the weight matrix, g is the learnable scaling factor, ° is the row-wise multiplication, t is the threshold for clipping the scaling factor g, and N is the calculation for the norm of the weight matrix W.

14. The method of claim 10, wherein the training comprises performing a normalization method according to the following formula: where f is the deep balanced model, W is the weight matrix, g is the learnable scaling factor, ° is the row-wise multiplication, t is the threshold for clipping the scaling factor g, and N is the calculation for the norm of the weight matrix W.

15. A system comprising: one or more processors; and A non-transitory memory including processor-executable instructions that, when executed by one or more processors, cause the system to perform operations comprising: receiving user input identifying a deep balanced model and identifying a training dataset; and Training the deep balanced model on the training dataset, wherein the training comprises: executing forward and backward solvers to perform forward and backward passes through the deep balanced model, wherein the forward and backward solvers are identified in a user input; and Do one or more of the following: Automatic normalization of weight tensors; Jacobian regularization; and Fixed point correction.

16. The system of claim 15, wherein the user input identifies an injection module.

17. The system of claim 15, wherein the user input identifies the decoder module.

18. The system of claim 15, wherein one or more of the forward and backward solvers are modified by parameters included in user input.

19. The system of claim 15, wherein: The training includes performing a normalization method according to the following formula: where f is the deep balanced model, W is the weight matrix, g is the learnable scaling factor, ° is the row-wise multiplication, t is the threshold for clipping the scaling factor g, and N is the calculation for the norm of the weight matrix W.

20. The system of claim 18, wherein: The training includes performing a normalization method according to the following formula: where f is the deep balanced model, W is the weight matrix, g is the learnable scaling factor, ° is the row-wise multiplication, t is the threshold for clipping the scaling factor g, and N is the calculation for the norm of the weight matrix W.

Images