Torchdeq: library for deep equilibrium models

Abstract

To provide methods and systems for allowing users to define, train, and deploy deep equilibrium models.SOLUTION: Decoupled and structured interfaces allow users to easily customize deep equilibrium models. Disclosed systems support a number of different forward and backward solvers, normalization, and regularization approaches.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates generally to the field of artificial intelligence. More specifically, the disclosed embodiments relate to the design, training and use of machine learning models, including neural networks. [Background technology]

[0002] Background technology Deep Equilibrium Models (DEQs) are a recently developed class of implicit neural networks. DEQs have an emerging community. DEQs are gaining theoretical attention. Stability and speedup are active research topics in the DEQ community. DEQs have shown compelling generalization performance, interpretability, and robustness compared to semantic segmentation, optical flow, detection, inverse problems, meta-learning, object-centric learning, set prediction, control, spiking neural networks, machine translation, regularized flow, and graph learning.

[0003] Scientific software supports the increasing complexity, modularity and scale of deep learning. Open source software contributions range from foundational deep learning libraries such as PyTorch, Tensorflow and JAX, to comprehensive model zoos such as huggingface, fairseq for language models and timm for vision backbones. Summary of the Invention [Problem to be solved by the invention]

[0004] overview In recent years, many deep learning libraries for neural dynamics such as differential optimization and differential equations have been developed, e.g., theseus, torchopt, torchdiffeq, torchdyn, betty, pypose, etc. However, none of these were specifically designed for DEQs, nor have they been validated to scale up to state-of-the-art DEQs with good stability. Moreover, none of these were specifically designed for DEQs, nor have they been validated to host model zoos for implicit models. The disclosed embodiments are a step towards this, providing broad support for state-of-the-art deep equilibrium models. [Means for solving the problem]

[0005] Some disclosed embodiments include a method that includes receiving user input identifying a deep equilibrium model and a training dataset, and training the deep equilibrium model on the training dataset, the training step comprising:

number

number

[0006] [Figure 1] FIG. 2 illustrates an exemplary code according to the disclosed embodiments. [Diagram 2] 1 is a flowchart illustrating an exemplary method for training a DEQ according to disclosed embodiments. [Diagram 3] FIG. 1 is a block diagram illustrating a system according to disclosed embodiments. [Figure 4] FIG. 1 illustrates an exemplary embodiment of a general computer system according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Detailed Description An embodiment of the present disclosure is described herein. However, it should be understood that the disclosed embodiment is merely an example, and other embodiments may take various alternative forms. The drawings are not necessarily to scale, and some features may be exaggerated or minimized to show details of a particular component. Therefore, specific structural and functional details disclosed herein should not be interpreted as limiting, but rather as a representative basis for teaching those skilled in the art to variously utilize the present invention. As will be understood by those skilled in the art, various features shown and described in connection with any one of the drawings can be combined with features shown in one or more other drawings to produce an embodiment not explicitly shown or described. The combination of the illustrated features provides a representative embodiment for a typical application. However, various combinations and modifications of features consistent with the teachings of the present disclosure may be desirable for a particular application or implementation.

[0008] Training and application of DEQ is currently done in an ad-hoc fashion with a variety of techniques spread across the literature. This disclosure systematically reviews DEQ and example embodiments include a PyTorch-based library, referred to herein as TorchDEQ, that allows users to define, train, and infer using DEQ across multiple domains with minimal code and best practices. The disclosed embodiments, sometimes referred to herein as the "DEQ Zoo," support six published implicit models across different domains. The disclosed embodiments include a collaborative framework that incorporates best practices across all models and have been used to significantly improve the performance, training stability, and efficiency of DEQ on ten datasets across all six models in the DEQ Zoo.

[0009] 1.0 Introduction Unlike traditional feedforward models, which use a fixed-size computational graph to compute their output, DEQ defines its output as a fixed-point representation of a nonlinear system, i.e. z * =f θ (z * ,x) where x is the input to the network, z * This formulation has several notable advantages: DEQ overcomes the "infinite depth" limitation of fixed-point iterations, z l+1 =f θ (z l ,x), it provides a rich representation using relatively few parameters, requiring only a single layer to be specified in the architecture design, and all that needs to be stored for backpropagation is the final fixed-point z * This allows training with significantly less memory and (in many cases) the final output z * recovers a path-independent solution that is independent of its initialization, and finally, intuitively allows for a separation between the "definition" of the network and the "solver" that computes the fixed points, a separation that reflects many settings, e.g., differential equation solvers or optimizers.

[0010] Unfortunately, DEQs are often difficult to train and difficult to deploy in practice. Training DEQs can often result in unstable systems, and methods to address these stability challenges are scattered across various papers in the literature; similarly, backpropagation in these networks can be done in many different ways, i.e., by unrolling, implicit differentiation, or inaccurate "phantom" gradients; and finally, the choice of architecture and equilibrium solver often needs to be made anew for each different application. These challenges are believed to have significantly limited the impact of DEQs within deep learning.

[0011] To this end, we have developed a modular library, sometimes referred to herein as TorchDEQ. TorchDEQ is a carefully designed and fully functional PyTorch-based library for building and deploying DEQs. It provides a decoupled and structured interface that allows users to customize their own general-purpose DEQ for any task with a minimal amount of code. The library supports several different forward solvers, backward paths, regularization, and regularization approaches, implementing best practices from all fields.

[0012] Disclosed embodiments of the module library (i.e., TorchDEQ) may include a model zoo for DEQ, which may be referred to herein as the "DEQ Zoo." Disclosed embodiments implement six published implicit models via TorchDEQ, including the DEQ transformer, the multi-scale deep balanced model ("MDEQ"), the implicit graph neural network ("IGNN"), the deep balanced optical flow estimator ("DEQFlow"), the implicit layer for implicit representations ("DEQ-INR"), and the deep balanced approach to diffusion models ("DEQ-DDIM"). All of these models perform uniformly better in terms of performance, stability, and efficiency than those reported in the original papers and published codes.

[0013] 2.0 TorchDEQ In the following, embodiments of TorchDEQ and exemplary supported features in TorchDEQ are disclosed in more detail. The exemplary embodiments disclosed below may include a description of the DEQ, an exemplary interface for TorchDEQ including exemplary code, and a computation graph design for TorchDEQ that can highlight different approaches for approximating the reverse path, as well as other DEQ strategies.

[0014] 2.1 DEQ Given an input data pair (x, y) and a loss function L, DEQ calculates the fixed-point z of a neural network fθ from the input injection u(x). * The training objective is to implicitly map

number

[0015] Theorem 2.1. By the Implicit Function Theorem ("IFT"), under mild conditions, the gradient of DEQ is given by:

number

number

number

[0016] 2.2 Sample Code and Interfaces The above aspects have in common that in all cases the main attribute of DEQ is the function f θ The DEQ is tolerant to the specific choice of the quantization method. That is, a unified modular interface can be constructed to implement DEQ for univariate, multivariate, and even multiresolution equilibrium systems of different functional designs. However, the implementation of DEQ remains challenging, as all the components shown above and further extensions require expert design and verification. Differences in implementation can have significant impacts on downstream performance, stability, and efficiency, as explained below.

[0017] FIG. 1 shows sample code for making and using embodiments of the present disclosure, including using the TorchDEQ library and methods for building and training a DEQ.

[0018] The operation get_deq can return a DEQ solver as a Pytorch module. The user can specify the input injection x and the initialization z for the fixed-point solver. 0 Using f θ You must pass in a function f that defines the function call for z. Fixed-point reuse may be achieved by a user-selected preceding fixed point. * =[h * ,c * For multivariate equilibrium systems with different tensor shapes such as f=lambda h,c:→self.deq_func((h,c),x) By simply rewriting the function symbols with minor adjustments, TorchDEQ can accomplish the remaining adjustments to the gradients and solvers.

[0019] The apply_norm and reset_norm operations are performed on the balanced module f θ Normalization can be automatically applied to the weight tensors in and the values ​​for each weight tensor can be recomputed before the next training step. More details are provided in Section 2.5.

[0020] The add_deq_args operation can provide decorators for commonly used Python argument parsers. Users of the disclosed embodiments can simply call add_deq_args(parser) to customize DEQ behavior via command line. This design has been adopted by community-trusted libraries such as fairseq and timm. python train.py--ift--f_solver anderson--b_solver brolyden For example, the above command will start training using implicit differentiation as the inverse, Anderson acceleration as the forward solver, and Broyden's method as the backward solver.

[0021] TorchDEQ's compact modular interface allows users to express their requirements as an equilibrium model θ We can focus on how to abstract, formalize, and define the DEQ as a DEQ and consider its interaction with other explicit layers such as injection and decoder. TorchDEQ's modular design creates an "abstraction" of the DEQ and minimizes the cost of learning, implementing, and tuning the DEQ. In the following sections, we introduce the features of TorchDEQ and its control commands.

[0022] 2.3 Reverse Route TorchDEQ can internally create a computation graph for the solver and gradients. Users can receive a group of tensors registered with gradients. Users can work with the output of implicit models in the same way as with explicit layers and tensors. However, when computing gradients, TorchDEQ may transparently compute the backward pass using a special method. TorchDEQ can support two types of backward passes, specifically, implicit derivatives ("IFT") and phantom gradients ("PG"). In practice, both types and / or their combinations may be sufficient to provide empirically compelling results within a reasonable time frame.

[0023] Implicit differentiation (i.e., IFT). Implicit differentiation is the standard approach for differentiating with fixed-point numbers. As shown in equation (3), the implicit gradient can be solved from an otherwise linear fixed-point system in the backward pass. Users can declare IFT with --ift, configure the backward solver with --b_solver broyden, and set up the solver configuration, e.g., maximum solver step --b_thres 30 and stopping criterion --b_eps 0.001.

[0024] Phantom gradient (i.e., PG). The phantom gradient is given by:

number

number

[0025] Phantom gradients may be applied to the computation graph of various solvers, similar to IFT for fixed-point differentiation. In the conventional view, IFT is considered as strict gradient and PG is considered as inexact gradient. However, they may lead to numerical errors in solving IFT in the forward direction. Therefore, it may not be necessary to distinguish between strict and inexact gradients, and they can both be alternatively referred to as backward paths.

[0026] The instantiations of PG used in the different implicit models consist of (approximate) fixed-point z solved with a damping factor τ. p Over the balance module f θ This includes deploying z p+1 = τf θ (z p )+(1-τ)z p (5) This is the following A matrix:

number

[0027] In the disclosed embodiment, separate support for backpropagation through time ("BPTT") and its truncated versions is not defined because they are given τ=1.0 and the forward solver has been removed, i.e., the solver and gradients are f θ This is because it can be expressed as a special case of PG that is defined only by the unfolded process of . By using the commands --f_thres0 --grad12 --tau1.0, the computation graph of BPTT-12 is defined.

[0028] The disclosed embodiments may include an interface memory, mem_gc, to reduce memory usage of an arbitrarily unfolded computation graph. Users can trade 1.5x training time for significantly lower, constant memory overhead via gradient checkpointing.

[0029] 2.4 Solvers Prior art DEQ projects typically wrote their own fixed-point solvers. These task-dependent solvers can be inefficient and sometimes unreliable when applied to different domains. To address this issue, disclosed embodiments implement, verify, and polish a solver implementation for TorchDEQ. Batch processing and memory access for multivariate systems may be optimized. In disclosed embodiments, the solver is reliable for a variety of tasks, robust across different settings, and may be agnostic to the scale of fixed-point equations and their tensor shapes. Thus, disclosed embodiments can provide significant improved efficiency over multivariate and multi-scale equilibrium systems.

[0030] In TorchDEQ, subsequent solvers may be supported. To invoke these solvers, the user can type --f_solver or --b_solver on the command line with the solver name, the maximum number of iterations --f_thres20, and the stopping criterion --f_eps1e-2. Additionally, keyword arguments may be passed to the DEQ class to tune the solver. For example, an example of customizing the Anderson acceleration is as follows: z_out,info=self.deq_solver(f,z,solver_kwargs={'tau':0.8,'m':6}) It can be carried out.

[0031] Naive solver. The fixed-point iterations are described in the following numerical scheme: * This is a classical solver for solving z k+1 =f θ (z k ) (7) The convergence is f θ This can be guaranteed by the bounded Jacobian spectral radius of . To invoke the fixed-point iteration, the user must You can enter -f_solver naive_solver.

[0032] Anderson solver. Anderson acceleration or Anderson mixing (i.e., Type-I Anderson Acceleration) is an acceleration technique for fixed-point iterations that uses a linear combination of the past m+1 fixed-point estimates. In the update, this numerical scheme,

number

number

number

[0033] Broyden solver. The Broyden method is a quasi-Newton solver for fixed-point equations. By maintaining a buffer, the Broyden method solves z k+1 =z k -α·B k g θ (z k ) (10) We approximate the Jacobian inverse in Newton's method by a low-rank update, where B k will be written as follows:

number

number

number

[0034] 2.5 Normalization Regularization techniques are essential for modern deep equilibrium models. Unlike common regularization methods applied to representations, DEQ additionally relies on regularization on the weight tensor, e.g., weight normalization, spectral normalization. In the disclosed embodiment, a DEQ version of the regularization technique is supported in TorchDEQ.

[0035] Normalization significantly smooths the fixed-point landscape of z given input data x, making this fixed-point representation easier to solve in practice, an effect that we believe has been underestimated in prior literature.

[0036] Weight matrix W∈R m×nFor , weight normalization ("WN") is done by normalizing the weights as follows: W i: =W i: (g i / ||W i: ||) (13) where || || represents the vector L2 norm and g is a learnable scale factor, while the spectral normalization ("SN") is denoted as W=W / ||W||2=W(1 / ||W||2) (14) where ||·||2 is the spectral norm, which may be computed by power iteration.

[0037] In the disclosed embodiment, TorchDEQ is as follows:

number

number

[0038] Inspired by gradient clipping, in the disclosed embodiment, an operation is introduced that significantly stabilizes the training of implicit graph neural networks on node classification, namely by clipping the rescaling coefficients to a threshold t,

number

[0039] In classical implementations of WN and SN, the weights are reset at every forward call. However, this is obviously wasteful for DEQ because the balancing module f θ is called many times until it converges. Then, the equivalent weight parameterization is θ It is applied by the number of calls of the function. Furthermore, decorated normalization must be manually coded for each module using traditional PyTorch implementations.

[0040] Instead, in the disclosed embodiment of TorchDEQ, we filter out the balancing module f via apply_norm (with keyword arguments filter_out to skip some modules) and reset_norm to reset it without wasting computation. θ A unified interface is provided for automatic global decoration. After training, the normalization decoration may be removed by remove_norm to facilitate training without modifying the model.

[0041] Users of TorchDEQ can specify --norm_type weight_norm for WN, --norm_type spectral_norm for SN, and --norm_no_scale to remove the learnable scaling g.

[0042] 2.6 Regularization Most modern interpretations of DEQ learning follow a conceptual shift, where we consider a DEQ model to simply be a well-performing (implicit) fixed-point mapping x → z * Instead of thinking of learning a unique, high-performance fixed-point z * We consider the smooth and flat equilibrium landscape x → B(z) to be learned, which includes

[0043] Balance module f θThe regularity of the fixed-point z * The correspondence between the equilibrium landscape B(z) and the loss landscape is governed by the fixed-point error ||f θ The results show a strong correlation between ||(z)-z|| and the loss L(y(z)). The path independence, i.e., convergence to a steady state regardless of initialization, allows for better utilization of test time calculations. Therefore, in the conventional approach, DEQ can obtain strong results in the early equilibrium solving process, and is a fixed-point z * We consider TorchDEQ to be a dynamic implicit neural network that can gradually improve its predictions as we approach Eq. The disclosed embodiments of TorchDEQ support these DEQ properties and techniques that promote regularity in the equilibrium landscape.

[0044] Jacobian Regularization ("JR"). Jacobian regularization is the

number

number

number

[0045] Fixed-point correction ("FC"). Fixed-point correction helps to learn a smooth equilibrium landscape by regularizing intermediate states from the fixed-point solution process. * A sequence that converges to

number

number

number

[0046] FIG. 2 illustrates an exemplary method 200 for training and using a DEQ according to disclosed embodiments. In some disclosed embodiments, the method 200 is performed by TorchDEQ. In an operational step 202, the method receives input data from a user. The input data can identify a training data set and identify a deep equilibrium model. In a first example, the training data set can be identified by the user by selecting an item from a displayed menu. The training data set can be already preloaded in TorchDEQ or can be loaded into TorchDEQ in response to user input. Optionally, the user input can also identify an injection module and / or a decoder module. In disclosed embodiments, TorchDEQ can allow a user to customize a forward solver and / or a backward solver from a TorchDEQ library. For example, the input data from the user can include parameters for modifying one or more forward solvers and / or backward solvers.

[0047] In the operation step 204, TorchDEQ trains the deep balanced model. For forward and backward passes through the non-DEQ modules, TorchDEQ can use default PyTorch functionality. For forward and backward passes through the DEQ modules, TorchDEQ can use forward and / or backward DEQ solvers from the TorchDEQ library. In the forward and / or backward solvers used by TorchDEQ, the forward and / or backward solvers may be customized by modifying parameters according to parameters provided by the user in the input data from the user. During the training of the deep balanced model, TorchDEQ can perform one or more of auto-normalization of weight tensors, Jacobian regularization, and fixed-point correction.

[0048] In a computation step 206, TorchDEQ can make a forward pass through the trained deep equilibrium model with the DEQ solver to generate predictions. For example, the trained deep equilibrium model can perform predictions on test data.

[0049] 3 illustrates a block diagram of a system 300 according to disclosed embodiments. For example, the system 300 may be an implementation of TorchDEQ. Input data 310 may be input to a model 320. The input data 310 may be, for example, the input data described above with respect to FIG. 2. The model 320 includes an optional injection module 322, a DEQ 324, and an optional decoder 326. The model 320 outputs a prediction 330.

[0050] 4 illustrates a block diagram of an exemplary embodiment of a general-purpose computer system 400. The computer system 400 may include a set of instructions that may cause the computer system 400 to perform any one or more of the methods or computer-based functions disclosed herein. For example, the computer system 400 may include executable instructions to perform the functions of TorchDEQ. The computer system 400 may be connected to other computer systems or peripheral devices via a network. Additionally, the computer system 400 may include or be included within other computing devices.

[0051] As shown in FIG. 4, computer system 400 may include one or more processors 402. The one or more processors 402 may include, for example, one or more central processing units (CPUs), one or more graphics processing units (GPUs), or both. Computer system 400 may include a main memory 404 and a static memory 406 that may communicate with each other via a bus 408. As shown, computer system 400 may further include a video display unit 410, such as a liquid crystal display (LCD), a projection television display, a flat panel display, a plasma display, or a solid-state display. Additionally, computer system 400 may include input devices 412, such as a wireless keypad, a keyboard, a microphone coupled to a voice recognition engine, a remote control device having a camera, such as a video camera or a still camera, or a cursor control device 414, such as a mouse device. Computer system 400 may also include a disk drive unit 416, a signal generating device 418, such as a speaker, and a network interface device 420. Network interface 420 may enable computer system 400 to communicate with other systems over network 428. For example, network interface 420 may enable machine learning system 120 to communicate with a database server (not shown) or a controller (not shown) in a manufacturing system.

[0052] 4, disk drive unit 416 may include one or more computer-readable media 422 upon which one or more sets of instructions 424, e.g., software, may be embedded. For example, instructions 424 may implement one or more methods or functionalities, such as the methods or functionalities disclosed herein. In particular embodiments, instructions 424 may reside, completely or at least partially, within main memory 404, static memory 406, and / or processor 402 during execution by computer system 400. Main memory 404 and processor 402 may also include computer-readable media.

[0053] In some embodiments, dedicated hardware implementations, such as, for example, application specific integrated circuits, programmable logic arrays, and other hardware devices, can be constructed to implement one or more of the methods or functionalities described herein. Applications that may include the devices and systems of the various embodiments may broadly include a variety of electronic and computer systems. One or more embodiments described herein may implement functionality using two or more specific interconnected hardware modules or devices with associated control and data signals that can be communicated between the modules, through the modules, or as part of an application specific integrated circuit. Thus, the system encompasses software, firmware, and hardware embodiments or combinations thereof.

[0054] Although the computer-readable medium is depicted as being a single medium, the term "computer-readable medium" includes a single medium or multiple media, such as a centralized or distributed database and / or associated caches and servers that store one or more sets of instructions. The term "computer-readable medium" also includes any medium that may store or encode a set of instructions for execution by a processor or for causing a computer system to perform any one or more of the methods or functionality disclosed herein.

[0055] In some embodiments, some or all of the computer readable medium is non-transitory. In certain non-limiting exemplary embodiments, the computer readable medium may include solid-state memory, such as a memory card or other package that contains one or more non-volatile read-only memories. Furthermore, the computer readable medium may be a random access memory or other volatile re-writable memory. Additionally, the computer readable medium may include magneto-optical or optical media, such as a disk or tape or other storage device for capturing a carrier signal, such as a signal communicated over a transmission medium.

[0056] Although exemplary embodiments have been described above, these embodiments are not intended to describe all possible forms encompassed by the claims. It will be understood that the terms used herein are terms of description rather than limitation, and that various modifications are possible without departing from the spirit and scope of the present disclosure. As previously described, features of various embodiments can be combined to form further embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may be described as offering advantages or being more preferred over other embodiments or prior implementations with respect to one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be traded off to achieve the desired overall system attributes, depending on the particular application and implementation. These attributes may include, but are not limited to, strength, durability, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, and the like. Thus, an embodiment described as less desirable than other embodiments or prior implementations with respect to one or more characteristics does not depart from the scope of the present disclosure, and may be desirable for a particular application.

Claims

1. receiving user input identifying a deep equilibrium model and identifying a training data set; training the deep equilibrium model on the training data set; A method comprising: The training step may include: [0010] where f is a deep equilibrium model, W is a weight matrix, g is a learnable scaling factor, [0025] is a row-wise multiplication, t is a threshold for clipping the scaling factor g, and N is the norm calculation for the weight matrix W.

2. The method of claim 1 , wherein the user input identifies an infusion module.

3. The method of claim 1 , wherein the user input identifies a decoder module.

4. The method of claim 1 , wherein the training step includes executing a forward solver and a backward solver for forward and backward passes through the deep equilibrium model.

5. The method of claim 4 , wherein one or more of the forward solver and the backward solver are modified by parameters included in the user input.

6. The training step includes the steps of: Automatic normalization of weight tensors, Jacobian regularization, and Fixed decimal point correction The method of claim 1 , comprising performing one or more of the following:

7. The training step includes the steps of: Automatic normalization of weight tensors, Jacobian regularization, and Fixed decimal point correction The method of claim 4 , comprising performing one or more of the following:

8. receiving user input identifying a deep equilibrium model and identifying a training data set; training the deep equilibrium model on the training data set; A method comprising: The method, wherein the training step includes executing a forward solver and a backward solver for passing forward and backward through the deep equilibrium model, the forward solver and the backward solver being identified in the user input.

9. The method of claim 8 , wherein one or more of the forward solver and the backward solver are modified by parameters included in the user input.

10. The training step includes the steps of: Automatic normalization of weight tensors, Jacobian regularization, and fixed decimal point correction, The method of claim 8 , comprising performing one or more of the following:

11. The method of claim 8 , wherein the user input identifies an infusion module.

12. The method of claim 8 , wherein the user input identifies a decoder module.

13. The training step may include: [0030] where f is a deep equilibrium model, W is a weight matrix, g is a learnable scaling factor, [0045] 9. The method of claim 8, wherein x is a row-wise multiplication, t is a threshold for clipping the scaling factor g, and N is a norm calculation for the weight matrix W.

14. The training step may include: [0050] where f is a deep equilibrium model, W is a weight matrix, g is a learnable scaling factor, [006] 11. The method of claim 10, wherein: t is a row-wise multiplication; t is a threshold for clipping the scaling factor g; and N is a norm calculation for the weight matrix W.

15. one or more processors; a non-transitory memory containing processor-executable instructions; A system comprising: The processor-executable instructions, when executed by the one or more processors, cause the system to perform the following operations: receiving user input identifying a deep equilibrium model and identifying a training data set; training the deep equilibrium model on the training data set; and The training step includes: executing a forward solver and a backward solver for making a forward and backward pass through the deep equilibrium model, the forward solver and the backward solver being identified in the user input; below: Automatic normalization of weight tensors, Jacobian regularization, and Fixed decimal point correction and performing one or more of: Including, the system.

16. The system of claim 15 , wherein the user input identifies an infusion module.

17. The system of claim 15 , wherein the user input identifies a decoder module.

18. The system of claim 15 , wherein one or more of the forward solver and the backward solver are modified by parameters included in the user input.

19. The training step may include: [0070] where f is a deep equilibrium model, W is a weight matrix, g is a learnable scaling factor, [0080] 16. The system of claim 15, wherein x is a row-wise multiplication, t is a threshold for clipping the scaling factor g, and N is a norm calculation for the weight matrix W.

20. The training step may include: [0097] where f is a deep equilibrium model, W is a weight matrix, g is a learnable scaling factor, [0089] 20. The system of claim 18, wherein x is a row-wise multiplication, t is a threshold for clipping the scaling factor g, and N is a norm calculation for the weight matrix W.

Images