Methods and systems for regularizing a neural network, including uses thereof

Abstract

Embodiments provided herein present a fast algorithm that provides all such solutions for any differentiable function f and loss L, and any constraint P that is an increasing monotone function of the absolute value of each parameter. Applications involving sparsity inducing regularization of arbitrary Neural Networks are discussed. Empirical results indicate that these sparse solutions are usually superior to their dense counterparts in both accuracy and interpretability. This improvement in accuracy can often make Neural Networks competitive with, and sometimes superior to, state-of-the-art methods in the analysis of tabular data.

Description

[0001] Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 METHODS AND SYSTEMS FOR REGULARIZING A NEURAL NETWORK, INCLUDING USES THEREOF CROSS-REFERENCETORELATEDAPPLICATIONSThis application claims the benefit of priority to United States Provisional Application Serial Number 63 / 542,992, filed October 6, 2023, and United States Provisional Application Serial Number 63 / 546,894, filed November 1, 2023, the contents which are hereby incorporated by reference in their entirety. INTRODUCTION Artificial Intelligence (AI) and Machine Learning (ML) are growing fields that are positioned to revolutionize many aspects of life. A common model used in such technologies is the neural network (NN). However, as NNs grow, they demand additional computing power, which can limit their ability to function on consumer devices. Thus, there is a need in the art to optimize NNs, so they demand less computing power while maintaining the power and function of the NN. SUMMARY Embodiments provided herein present a fast algorithm that provides all such solutions for any differentiable function f and loss L, and any constraint P that is an increasing monotone function of the absolute value of each parameter. Applications involving sparsity inducing regularization of arbitrary Neural Networks are discussed. Empirical results indicate that these sparse solutions are usually superior to their dense counterparts in both accuracy and interpretability. This improvement in accuracy can often make Neural Networks competitive with, and sometimes superior to, state-of-the-art methods in the analysis of tabular data. BRIEFDESCRIPTION OF THEDRAWINGS FIG. 1 provides an exemplary flowchart of neural network regularization in various embodiments. FIG.2 provides exemplary pseudocode of an algorithm used in various embodiments. FIG.3 provides a schematic of an apparatus for an exemplary computer system used in various embodiments. Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 FIG.4 illustrates exemplary data showing feature importance, computed as |^^|, for the model found by Lockout at the validation minimum. True values of ^^^^^^are ^^= 0.5, ^^= 0.75, and ^^= 1.0 FIG.5 illustrates exemplary data showing first-layer vs. number of iterations when the NN is trained without regularization (left) and with Lockout (right). For the latter, the vertical line represents the position of the validation minimum. FIG.6 illustrates exemplary data showing training and validation mean squared errors vs. number of iterations when the NN is trained without regularization (left) and with Lockout (right). Values of the validation minimums found in each case is also shown in the figure. FIG.7 illustrates exemplary data showing relative root mean squared error vs. number of selected features for the testing subset. In addition to the two NN architectures previously described are also shown. Results shown are over one run, after selecting the best NN using the validation set. 2 fully connected layers with 10 nodes in the hidden layers with or without the residual connection were used. FIG. 8 illustrates exemplary data showing feature importance, computed as∑^^ / ^^^^^^^^^ where ^^=∑^^^^|^^^| and K is the number of nodes in the layer. Early and to one of the partitions on dataset 2. FIG.9 illustrates exemplary data showing Validation Error vs. non-zero features in the model. This plot shows the validation error over the regularization path with Lockout on the dataset 6. FIG.10 illustrates exemplary data showing a comparison of feature selection provided by Lockout with L1 or Logarithmic penalties. Results shown here are for one of the partitions of the corresponding dataset, using a 2-layer fully connected architecture with 10 nodes in the hidden layer followed by ReLU activation functions. As it can be seen, the Logarithmic penalty provided more aggressive feature selection than L1 which resulted in higher accuracy for the problems analyzed. FIG.11A illustrates an exemplary schematic of a Multi Target NN. Lockout is applied to the first layer to provide L1 regularization. The Hidden Layer vector is learned is common to all the problems. FIG.11B illustrates exemplary data showing Lockout applied to 585 different problems to provide regularization in the first layer. Point A represents the early stopping solution from where Lockout starts and point B is the minimum found. Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 DETAILED DESCRIPTION Before the systems, methods, and computer-readable media of the present disclosure are described in greater detail, it is to be understood that the systems, methods, and computer- readable media are not limited to particular embodiments described, as such may, of course, vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the systems, methods, and computer-readable media will be limited only by the appended claims. Where a range of values is provided, it is understood that each intervening value, to the tenth of the unit of the lower limit unless the context clearly dictates otherwise, between the upper and lower limit of that range and any other stated or intervening value in that stated range, is encompassed within the systems, methods, and computer-readable media. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges and are also encompassed within the systems, methods, and computer-readable media, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the systems, methods, and computer-readable media. Certain ranges are presented herein with numerical values being preceded by the term “about.” The term “about” is used herein to provide literal support for the exact number that it precedes, as well as a number that is near to or approximately the number that the term precedes. In determining whether a number is near to or approximately a specifically recited number, the near or approximating unrecited number may be a number which, in the context in which it is presented, provides the substantial equivalent of the specifically recited number. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the systems, methods, and computer-readable media belong. Although any systems, methods, and computer- readable media similar or equivalent to those described herein can also be used in the practice or testing of the systems, methods, and computer-readable media, representative illustrative systems, methods, and computer-readable media are now described. All publications and patents cited in this specification are herein incorporated by reference as if each individual publication or patent were specifically and individually indicated to be incorporated by reference and are incorporated herein by reference to disclose and describe the materials and / or methods in connection with which the publications are cited. The citation of any publication is for its disclosure prior to the filing date and should not be construed as an admission Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 that the present systems, methods, and computer-readable media are not entitled to antedate such publication, as the date of publication provided may be different from the actual publication date which may need to be independently confirmed. It is noted that, as used herein and in the appended claims, the singular forms “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise. It is further noted that the claims may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely,” “only” and the like in connection with the recitation of claim elements, or use of a “negative” limitation. It is appreciated that certain features of the systems, methods, and computer-readable media, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the systems, methods, and computer-readable media, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable sub-combination. All combinations of the embodiments are specifically embraced by the present disclosure and are disclosed herein just as if each and every combination was individually and explicitly disclosed, to the extent that such combinations embrace operable processes and / or compositions. In addition, all sub-combinations listed in the embodiments describing such variables are also specifically embraced by the present systems, methods, and computer-readable media and are disclosed herein just as if each and every such sub-combination was individually and explicitly disclosed herein. As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present methods. Any recited method can be carried out in the order of events recited or in any other order that is logically possible. NEURALNETWORKREGULARIZATIONThe present disclosure provides systems, methods, and computer-readable media that are capable of regularizing Neural Networks (NNs). NNs have become popular in recent years for problems involving image classification and natural language processing. In the analysis of tabular data, however, they still lag behind other competitors. NNs usually have many hidden units and layers that make them very expressive models but prone to overfitting in problems that involve substantial amount of noise, many variables, or limited data. Many important problems, Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 especially those in medicine, are of this latter type. For instance, genome wide studies usually involve thousands of genes but the number of patients with outcomes is in the hundreds. Thus, they are not amenable to the advantages of representational learning and feature design that NNs provide. Numerous embodiments provide for methods for regularizing a NN. Figure 1 illustrates an exemplary flow chart providing various features that can be used for such a process. It should be noted that the features illustrated are merely exemplary and may be used in a different combination, order, or with greater or fewer features as illustrated. At 102, various implementations obtain a pre-trained NN architecture. Such obtention can include defining the NN architecture, pretraining an NN architecture de novo, and / or any other method to obtain a pretrained NN architecture. In many instances, the NN architecture can include one or more hidden layers. The number of hidden layers can include 1 hidden layer, 2 hidden layers, 3 hidden layers, 4 hidden layers, 5 hidden layers, 6 hidden layers, 7 hidden layers, 8 hidden layers, 9 hidden layers, 10 hidden layers, or more hidden layers. In some preferred embodiments, the NN architecture includes at least two hidden layers—e.g., 2 hidden layers, 3 hidden layers, 4 hidden layers, 5 hidden layers, 6 hidden layers, 7 hidden layers, 8 hidden layers, 9 hidden layers, 10 hidden layers, or more hidden layers. Each layer may include one or more nodes, and the number of nodes in each layer may be 1 node, 2 nodes, 5 nodes, 10 nodes, 25 nodes, 50 nodes, 75 nodes, 100 nodes, 150 nodes, 250 nodes, or more nodes. In various preferred embodiments, each hidden layer includes at least two nodes—e.g., 2 nodes, 5 nodes, 10 nodes, 25 nodes, 50 nodes, 75 nodes, 100 nodes, 150 nodes, 250 nodes, or more nodes. Some architectures include one or more residual layers, such as for use in a ResNet style machine learning model. Moving to 104, one or more parameters are selected to be considered for pruning. Such parameters. Such parameters may include weights at a particular node and / or biases. In various instances, all parameters can be selected to be considered for pruning. At 106, various instances assess whether a gradient scale is independent of location. If the gradient scale is not independent, then it can be normalized at 107. This process can be repeated for multiple gradient scales, as appropriate. To encourage all parameters at a particular node going to 0, a group inducing sparsity penalization is selected at 108. Such penalization can be selected from lasso, L2, group lasso, sparse group lasso, grouped elastic net, hierarchical lasso, fused lasso, tree-structures sparsity, Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 block sparsity, graph-induced sparsity, sparse grouped feature selection, structured sparsity with constraints, L1, L2, and / or any other relevant penalization. Grouping methodologies can be created at 110 to bring parameters into the same set. This grouping process assists in combining nodes and / or reducing the amount of connections within the NN architecture, thus assists in finding nodes that can be removed such as with the sparsity penalization. An optimizer can be selected at 112, in many instances an optimizer is used in training of the NN architecture. Optimizers assist in identifying an optimal set of parameters that result in the best performance of the model (e.g., the NN) based on the training data. Various optimizers are known in the art and can be selected from Stochastic Gradient Descent (SGD), Mini-Batch Gradient Descent, Adam (Adaptive Moment Estimation), RMSprop (Root Mean Square Propagation), Adagrad (Adaptive Gradient Algorithm), Adadelta, Nadam (Nesterov-accelerated Adaptive Moment Estimation), FTRL-Proximal (Follow-the-regularized-Leader Proximal), L-BFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno), Conjugate Gradient, SGD with Momentum, Proximal Gradient Descent, LBFGS-B (Limited-memory Broyden-Fletcher-Goldfarb- Shanno with Box Constraints), Frost, and Rprop (Resilient backpropagation), and / or any other relevant and applicable optimizer. Additional embodiments select a data set with which to design the architecture at 114. The dataset can be selected for training based on the desired use of the NN. The dataset can be selected from synthetic data, real data, or a mix of synthetic and real data. The data can include biological and / or medical data, such as medical imaging (e.g., X-ray, MRI, CT, PET, etc.), medical diagnostics, gene expression (e.g., microarray and / or RNA sequencing), genetic disease, and / or any other form of biological data. Additional data types can include more practical based data, such as voice recognition, image recognition, and / or any other form of data. At 116, many embodiments apply a heuristic to find a path of solutions through the NN architecture. In certain instances, the heuristic takes the form of a “Lockout” algorithm to find a regularization path of solutions. In many instances the Lockout process involves 1) linearizing a loss function and constraint, 2) solving the corresponding linear programming problem, and 3) applying these steps iteratively. In many instances, this process involves identifying an unconstrained regularization solution. In a stepwise manner, Lockout can include: 1. Updating parameters for which penalization is 0 as these do not affect the constraint. 2. Updating parameters for which decreasing their absolute value (and as such their penalty) decreases the loss. Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 3. Updating the most important parameters (as defined by Lockout) in the direction provided by the optimizer, which will increase their absolute value but decrease the loss function. 4. Updating the rest of the parameters contrary to the direction provided by the optimizer, which will decrease their absolute value but increase the loss. An example of how to implement the “Lockout” algorithm can be found in Figure 2. Specifically, Figure 2 provides some pseudocode. The Lockout algorithm provides a series of decision points and parameters to identify one or more nodes that can be removed from a neural network. This removal, or “trimming,” allows a neural network to operate without necessitating some amount of computation at nodes that have minimal or limited impact on the power of the NN. At 118, additional embodiments select the best solution based on a tradeoff between sparsity and model performance. While sparsity may allow the model to have better processor performance, such advantage may come at the sacrifice of the model’s predictive capacity. Thus, various embodiments, select a solution based on this tradeoff. Based on the selected path, nodes where all parameters are equal to zero can be removed at 120. This process reduces the computational requirements of the resultant model, since fewer nodes require less computational execution—a node where all parameters are zero still requires some computational effort. At 122, embodiments may further adjust the forward function to still compute after node removal. This process allows for the model to still compute a result, in situations where a normal path may be disrupted because of node removal. Finally, the trimmed NN is output at 124. As noted herein, the resultant NN is trimmed to allow for more efficient computational power while retaining as much predictive power as possible over a non-regularized NN. As such, many embodiments are directed to NNs regularized by the methods described herein. Additionally, such NNs may be contained within a computing device. Such computing devices can be consumer-grade or consumer-level devices including (but not limited to) a mobile phone, a tablet, a laptop computer, and / or a desktop computer. Some such embodiments can also be integrated into diagnostic devices or diagnostic equipment themselves. Such implementations allow for the power of a trained NN to be deployed to locations absent an internet connection or without sufficient power delivery to power industrial computer systems, such as servers, server farms, or other highly power computing systems. This can further allow for field deployment of such devices, empowering remote locations the power of artificial intelligence or machine learning enabled devices that were not previously capable of such access. Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 Implementation on GPUs For regularizing NNs, the most effective computer architecture today contains one or more graphical processing units (GPUs), each of which contains many separate parallel processing units. A block diagram of a computer architecture containing a GPU is shown in FIG.3. Because of its massive parallelism, the GPU can instantiate the many processing units of an NN much more efficiently than computer architectures based only on a central processing unit (CPU). This results in regularization speeds for a GPU-based system that can be orders of magnitude faster than a system containing one or a handful of CPUs. The GPU-based computing system can either be local in the form of a dedicated GPU workstation or GPU chip(s) incorporated into a personal computer, or it can be remote in the form of cloud-based GPU instances such as those available from the Amazon Web Services (AWS) Elastic Computing Cloud (EC2). . In FIG.3, a computer system is illustrated that can be configured to perform various of the steps described herein. Such computer system may be local (i.e., a single, fully contained device), or it can be another workstation, or can be a remote server in the “cloud” accessible via computer network. As shown in FIG.3, this computer system may generally include at least one input (such as a user interface) 302, at least one processor (such as a central processing unit) 304, at least one type of memory 306 which will typically include both high speed random access memory as well as non-volatile memory (such as one or more magnetic disk drives), at least one output 308 (such as a monitor or other display device), a disk (not shown) for storage of image and other data (not shown), and a device or devices for reading computer media. The computer system may preferably comprise a GPU (graphics processing unit) 310 in conjunction with GPU memory 312, as further described herein. The computer system 300 may be, for example, a workstation, network server, laptop, tablet, or notebook computer, personal digital assistant, smartphone, or mainframe computer. Additionally, and not shown in FIG.3, the computer system may have at least one network connection or other communication interface for communicating with other computers over a network, including the Internet, as well as other devices, such as via a high speed networking cable, or a wireless connection. The network connection can be used for a hospital to share data with another party, where the other party has a second Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 computer system (not shown) having similar capability to that of computer system 300 and is able to receive data to be shared from computer 300. There may optionally be a firewall (not shown) between computer 300 and a network such as the Internet. At least one or more processors 304, memory 308, and user interface or input 302, communicate with one another via at least one communication bus (not shown). Memory 308 stores procedures and data, typically including some or all of: an operating system for providing basic system services; one or more application programs, such as a parser, and a compiler, a file system, one or more databases that store image and other medical data, and optionally a floating point coprocessor where necessary for carrying out high level mathematical operations such as for carrying out image analysis. The methods of the present technology may also draw upon functions contained in one or more dynamically linked libraries stored either in memory 308, or on disk. The data and other routines that are stored in memory may instead, optionally, be stored on a disk where the amount of data in the database is too great to be efficiently stored in memory 308. The data and routines may also instead, or in part, be stored on one or more remote computers that communicate with computer system 300 through network interface, according to networking methods understood by those skilled in the art. Various implementations of the technology herein can be contemplated, particularly as performed on one or more computing apparatuses (machines that can be programmed to perform arithmetic) of varying complexity, including, without limitation, workstations, PC’s, laptops, notebooks, tablets, netbooks, and other mobile computing devices, including cell-phones, mobile phones, wearable devices, and personal digital assistants. The methods herein may further be susceptible to performance on quantum computers. The computing devices can have suitably configured processors, including, without limitation, graphics processors and math coprocessors, for running software that carries out the methods herein. In addition, certain computing functions are typically distributed across more than one computer so that, for example, one computer accepts Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 input and instructions, and a second or additional computers receive the instructions via a network connection and carry out the processing at a remote location, and optionally communicate results or output back to the first computer. Control of the computing apparatuses can be via a user interface 302, which may comprise one or more of: a display, mouse, keyboard, and / or other items not shown in FIG.3, such as a track-pad, track-ball, touch-screen, stylus, speech-recognition device, gesture-recognition technology, human fingerprint reader, or other input such as based on a user’s eye-movement, or any subcombination or combination of inputs thereof. Exemplary Computational Implementations The methods described herein are preferably implemented as instructions run as software on one or more computer systems, and the implementation is within the capability of those skilled in the art of computer image processing. In particular, the computer functions can be developed by a programmer skilled in the art of data processing. The functions can be implemented in a number and variety of programming languages including, in some cases, mixed implementations (i.e., relying on separate portions written in more than one computing language suitably configured to communicate with one another). For example, the functions, as well as any required scripting functions, can be programmed in C, C++, Java, JavaScript, VisualBasic, Tcl / Tk, Python, Perl, .Net languages such as C#, and other equivalent languages. Certain mathematical functions for image processing can be written in scientific programming languages and are preferably implemented in a form that takes advantage of a GPU. The capability of the technology is not limited by or dependent on the underlying programming language used for implementation or control of access to the basic functions. Alternatively, the functionality can be implemented from higher level functions such as toolkits that rely on previously developed functions for manipulating three-dimensional image data. The technology herein can be developed to run with any of the well-known computer operating systems in use today, as well as others not listed herein. Those operating systems include, but are not limited to: Windows (including variants such as Windows XP, Windows95, Windows2000, Windows Vista, Windows 7, Windows 8 Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 (including various updates known as Windows 8.1, etc.), and Windows 10, available from Microsoft Corporation); Apple iOS (including variants such as iOS3, iOS4, and iOS5, iOS6, iOS7, iOS8, iOS9, iOS10, and intervening updates to the same); Apple Macintosh operating systems such as OS9, OS 10.x, OS X (including variants known as “Leopard”, “Snow Leopard”, “Mountain Lion”, “Lion”, “Tiger”, “Panther”, “Jaguar”, “Puma”, “Cheetah”, “Mavericks”, “Yosemite” and “El Capitan”; the UNIX operating system (e.g., Berkeley Standard version) and variants such as IRIX, ULTRIX, and AIX; Google Chrome, and Android OS versions; and the Linux operating system (e.g., available from Red Hat Computing as well as open source distributions such as Ubuntu). To the extent that a given implementation relies on other software components, already implemented, such as functions for manipulating three dimensional image data, and functions for calculating aspects of fitting mathematical forms to the same, as well as functions for implementing aspects of neural networks and of deep learning methods, those functions can be assumed to be accessible to a programmer of skill in the art. Furthermore, it is to be understood that the executable instructions that cause a suitably-programmed computer to execute methods for regularizing an NN, as described herein, can be stored and delivered in any suitable computer-readable format. This can include, but is not limited to, a portable readable drive, such as a large capacity (for example, 0.5 TB, 1 TB, 2 TB or more) “hard-drive”, or a “pen-drive”, such as can be connected to a computer’s USB port, an internal drive to a computer, a CD- ROM, a DVD-ROM, or an optical disk. It is further to be understood that while the executable instructions can be stored on a portable computer-readable medium and delivered in such tangible form to a purchaser or user, the executable instructions can be downloaded from a remote location to the user’s computer, such as via an Internet connection which itself may rely in part on a wireless technology such as Wi-Fi. Such an aspect of the technology does not imply that the executable instructions take the form of a signal or other non-tangible embodiment. The executable instructions may also be executed as part of a “virtual machine” implementation, which may be local or in a remote location such as a data center that is part of a public, private or hybrid cloud. Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 Various embodiments are directed to NNs that have been regularized via methods described herein. With the ability of many embodiments to be stored on and / or run on consumer- level devices (e.g., laptops, tablets, smartphones, portable drives, etc.), many embodiments allow for an NN to operate on a consumer-level computing devices. Such NNs can be downloaded or provided as a single instance and optionally updated periodically (e.g., manually and / or automatically). Such NNs are thus field-deployable (e.g., can be carried into remote situations, where it is not feasible to carry larger devices (due to size, power limitations, etc.) and / or without a data connection. Furthermore, additional embodiments include such computing devices (e.g., consumer-level computing devices) comprising a storage device comprising a trained and trimmed NN. Once a trimmed NN is output, it can be used in accordance with its initial process or use, including uses in optical recognition, speech recognition, medical processing (including image recognition, risk stratification, and / or any other relevant process). COMPUTER-READABLE MEDIA AND SYSTEMS Aspects of the present disclosure further include systems and non-transitory computer-readable media. In certain aspects, provided are one or more computer-readable media comprising instructions stored thereon, which when executed by one or more processors, cause the one or more processors to perform the subject methods. According to some embodiments, the operations performed comprise receiving initial magnitude image data and producing phase image data from the initial magnitude image data using a generative model to synthesize the raw multichannel data. In some cases, the computer readable media and systems are configured to reconstruct an image based on the synthesized raw multichannel data. While such is not required, some embodiments of the systems include one or more signal receivers, such as radiofrequency coil receivers. In some such embodiments, systems include an MRI machine configured to collect magnitude image data and / or data that may be used to train the subject machine learning models. Embodiments of the systems additionally include one or more input / output devices, such as a screen display for displaying reconstructed images. The following examples are offered by way of illustration and not by way of limitation. Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 EXAMPLES Example 1 – Synthetic Data In this example, an embodiment of “Lockout” is applied with L1 regularization to four different synthetic data sets ^^^, ^^^^^. The objective is to show that Lockout can provide feature selection while improving the accuracy of the NN. The data sets for each experiment contained 500 observations with 100 features. They were each partitioned into separate training, validation, and testing subsets. Input variables ^^^^^^^^were randomly drawn from a uniform distribution ^^~^[0,1). Targets ^^^^^^^^were generated as ^^= "($^), where $^= ^^^^^− ^^^^^+ ^^^^^3 out of the 100 features and constants ^^^^^^were respectively set to 0.5, 0.75, different activation function "($^) was used for each data set. In particular, "($^) = $^, "($^)= '()^($^), "($^)= *^+ℎ($^), and "($^)= -."^ / .0($^) were used to generate the four datasets. Gaussian noise was added to both the training and validation subsets, with a signal-to- noise ratio of 1. No noise was added to the testing subset. In the first experiment, one hidden layer with a single node architecture was used, followed by the same activation "($^) used to generate the corresponding targets. More complex architectures and realistic scenarios are used in other experiments. The NN is first trained, and both the unconstrained (training data minimum) and the early stopping (validation minimum) solutions are obtained. The unconstrained solution was assumed to be reached if changes in the training loss function were smaller than 10-5for 20 consecutive iterations. This solution is then used as the starting point to train the NN with Lockout. The learning rates used in the unregularized and Lockout training were 5 x 10-3and 10-2, respectively. Stochastic gradient descent (SGD) was used as the optimization algorithm in both experiments. Figures 4 and 5 show that Lockout provides strong feature selection. For instance, at the validation minimum, only 7 out of the 100 features are non-zero. Among the seven non-zero features, the three that generated the output have the biggest importance. A similar result was found for the other examples. Lockout also provided significantly improved accuracy over the usual early stopping solutions for all activation functions used, as can be seen in Figure 6 and Table 1. Example 2 – Synthetic Data L1 based Lockout is applied to two different nonlinear synthetic data sets. The goal is to illustrate its performance in the presence of strong nonlinear dependence of ^ on Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 ^, with or without a corresponding linear dependence, and compare it to Lasso or the more recent LassoNet. For each of the datasets, 1000 observations and 200 features were generated for training, validation, and testing. Input variables ^^^^^^^^^were drawn from a uniform distribution ^^~^[0,1). The target values^^^^^^^^^for the first experiment were generated using the Friedman function. ^ = 10 ( ) ^ ^ sin (5^^^^^^) + 20 ^^^− 0.5 +10^^7+ 5^^^. For the second noise was added to the training and noise was added to the testing subset. Two NN architectures were explored in these experiments. The first one was a 2-layer fully connected architecture with 10 nodes in each hidden layer involving ReLU activation functions, and one node in the output layer. The second architecture was the same as the first, except for the addition of an extra residual linear connection between the input features and the output node (ResNet). The latter architecture was included because the method LassoNet requires this extra residual connection to provide regularization. The unconstrained solution in each case was found using the Adam optimizer with a learning rate of 10-3. The same convergence condition as in the previous experiments was used. For Lockout, SGD with a learning rate of 10-2was employed. Figure 7 shows relative root mean squared error versus number of selected features for the testing subset for Lasso (linear model), LassoNet (ResNet model), Lockout with the fully connected model and Lockout with the residual model. The results show that the strong non-linear components in the data generating function, in the absence of corresponding strong linear counterparts, cause Lockout to significantly outperform Lasso or LassoNet in these examples. Lasso is only able to model linear functions, whereas LassoNet couples the regularization of the linear part with that of the non-linear dependence. As such, they both are at a disadvantage when the variables responsible for the linear terms are not the same as those responsible for the non- linearity. By not coupling the regularization of non-linear dependencies to those of the corresponding linear ones, Lockout avoids this limitation. Example 3 – Real Datasets In this example, L1 based Lockout was applied to various classification and regression problems involving real-world datasets. The goal is to show that Lockout can often improve NN's performance, making it competitive with and sometimes superior to traditional state-of-the-art methods for tabular data such as Lasso or Gradient Boosting (GB). Additionally, this example Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 presents a novel application of Lockout to cancer research, where representational learning is a must. Experiment One Three microarray DNA datasets related to different types of cancer were analyzed. Historically, NNs have been substantially outperformed by other state-of-the-art methods in these settings. The problems are summarized in Table 2. Each example is a binary classification problem where the target variable ^^= 1 if the patient has cancer and ^^= 0 otherwise. Given the small number of observations, datasets were partitioned into 100 different training, validation, and testing subsets randomly drawn with 60%, 20% and 20%, of the data, respectively. Reported metrics for each method are the average and standard deviation over all the partitions. A paired T-test was also performed to compare performance of the algorithms against Lockout with L1 regularization. A grid search over relevant hyperparameter values was performed for each algorithm and the validation subset used for model selection. In particular, we tuned 1) the regularization strength for Lasso, 2) the number of trees, maximum depth of a tree, and shrinkage coefficient for Gradient Boosting, and 3) the learning rate for early stopping and Lockout(L1) NNs. In all examples a 2-layer fully connected architecture with 5 nodes in the hidden layer using ReLU activation functions was used. There were two nodes in the output layer. The traditional early stopping and Lockout(L1) solutions (at their respective validation minimums) were found using Adam and SGD, respectively, with cross-validated learning rate values. Lasso calculations were computed with its Scikit-Learn implementation while GB's with XGBoost package. Table 3 shows the Mean error and standard deviation over 100 different partitions for each method. The paired t-test p-values between Lockout and the different methods are shown in parentheses in each case. Lockout(L1) is able to significantly improve the performance of the NN, making it comparable with state-of-the-art algorithms for tabular data such as Lasso or GB. Even more remarkable is the fact that this increase in performance came in connection with a dramatic reduction in the number of non-zero valued features as compared to the traditional early stopping solution. This can be appreciated in Figure 8, where feature importance graphs corresponding to early stopping and Lockout(L1) models for one of the partitions of dataset 2 are shown. Experiment 2 Three datsets from the UCI repository were analyzed, including two regression and one classification problem. They are summarized in Table 4. All experiments were performed following the same procedure and NN architecture described for Experiment 1. As seen in Table 5, Lockout(L1) is able to improve the performance of the NN for the first two examples. In the last Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 one (dataset 6), Lockout performs almost as well while reducing the number of non-zero features by roughly 81%. In this case, starting the algorithm at the early stopping rather than the unconstrained solution was explored to investigate whether there exist solutions other than the validation error minimum that might be more desirable. The dependence of the validation error with number of selected features along this path is shown in Figure 9. Point A represents the model chosen by early stopping at the validation minimum. As more regularization is applied, the NN performance initially degrades but then gradually improves both in terms of accuracy and feature selection, reaching a plateau from around 4000 to 600 features. In this case, a solution (Point B) with better estimated performance than that of the early stopping solution was uncovered. However, selecting the model represented at point C might be preferred because it has similar estimated accuracy while involving only 6% of the features. This phenomenon often occurs with smaller data sets that are more prone to noisy validation error estimates. Example 4 – Non-Convex Logarithmic Regularization One of the intrinsic advantages of Lockout is that one can implement different regularization types by simply changing the constraint function and providing the new derivative, p(w). This constraint need not be a convex function of the parameters. The only restriction is that it must be monotone, increasing with the absolute value of each parameter. Here, we illustrate this by employing a non-convex (logarithmic) constraint that provides more aggressive variable selection than the Lasso. This constraint is given by 89(^) = ∑ > ^^^log ((1 − =)|^^|+=); 0 < = < 1, where J is the number This regularization has the property that parameters with smaller absolute values are more aggressively penalized than larger ones as the strength of the constraint is increased. This can be seen by examining the derivative of P respect to |^|: 1 − = 9(^^) = One sees that the rate of of smaller valued parameters is increased relative to that of the larger ones, thereby emphasizing sparser solutions. This is illustrated in Figure 10 by showing validation error versus the number of selected features in two different problems and contrasting it with the corresponding behavior of L1 regularization. Results shown in Figure 10 are for one of the partitions of the corresponding dataset, using a 2-layer fully connected architecture with 10 nodes in the hidden layer followed by ReLU activation functions. Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 There were two nodes in the output layer. In both of these cases, more accurate and sparse solutions are obtained with the non-convex logarithmic regularization. Example 5 – Common Representation of Cancer In this section, we explore a practical application of Lockout to cancer research. Integration of Gene Expression datasets (meta study) has proven extremely valuable as these datasets tend to have some observations with many genes, as described below. Meta studies to date have focused on finding a gene, single set of genes or pathways that are over expressed in a single phenotype (e.g. breast cancer). Specifically, the studies of over expressed pathways have been shown to be more robust than the analysis of single genes because the individual genes seldom act alone. Today, however, these pathways are predefined by the investigators in the majority of cases. There was an interest in designing an algorithm capable of learning which genes to choose and how to combine them (synthetic pathways) in order to explain a wide range of cancer problems and phenotypes. Combining representational learning with Lockout were taken advantage of to design a multi-output Neural Network with a common hidden layer, as shown in Figure 11A. The function of the hidden layer is to transform the original inputs into a space where many problems are easily solvable (the common representation). It was expected that by borrowing strength from multiple problems, a better representation is obtained that can improve the accuracy of each individual problem. In order to implement our architecture, we require datasets with the same sets of genes. Datasets belonging to the L1000 platform in the GEO BioBank repository that contain expressions of at least 100 of the 978 genes were chosen. The genes in the L1000 Platform were chosen because Subramanian et al defined them in a data driven approach. They maximized the information gained about the whole transcriptome while minimizing costs. They showed that this set of landmark genes can explain up to 81% of all genes that are not measured. As outcomes for each of these expression profiles, the phenotype columns that can be used for prediction (e.g. cancer vs no cancer, metastasis vs no metastasis) were needed. Since the GEO Biobank was not created with the purpose of being analyzed with Machine Learning methods, a pipeline was created to extract automatic phenotypes from these datasets. This approach was able to effectively do so by combining NLP analysis, data preprocessing pipelines and manual curation (assisted by a trained group of data science master students).553 different datasets across 23 different cancer types were assembled (The 5 Major groups are Breast = 208, Colorectal = 89, Lung = 54, Prostate = 52, Ovarian Cancer = 36). The phenotypes range from Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 different type of cells, aggressiveness of the tumors, cancer types, metastasis status, etc. All of these datasets have at least 5 observations per category and can be downloaded at lifelong- ml.github.io / madna.github.io / . Finally, each selected study uses only one platform to facilitate internal normalization and batch correction using ranks before the multitask learning analysis. In total, 39516 observations were used to learn relationships among 978 genes for 23 major classes of cancer. The data was divided in 70% for training, 10% for validation and 20% for testing. Only datasets with at least 20 observations in the test set were used for testing. There were 500 nodes in the hidden layer using Relu as activation function and the learning rate was cross validated in every case between the values [0:0001; 0:001; 0:01; 0:1]. The Adam optimizer was used to find the minimum in the validation set in the forward step. Lockout with L1 regularization was applied to the first layer, starting at the minimum of the validation solution. The sum of the absolute values of the coefficients for each of the output layers was not allowed to change. As it can be seen in Figure 11B, the use of Lockout improved the relative performance of the forward network (initial point) by approximately 14%. Additionally, to test whether the combined representation learned had meaning, an experiment using all datasets was performed with more than 100 observations (67 datasets). From the combined network 1) the coefficients of the hidden layer were extracted and 2) for each problem, the 67 set of coefficients of the output layers were extracted. These were then connected, creating 67 distinct NNs. Each of these NNs was used as starting point to find the minimum at validation for their specific problem, and the performance on the test set was recorded. Additionally, NNs were also optimized for each problem with the same architecture, but the parameters were started randomly. The NNs starting from the common representation significantly improved performance in 74% of the datasets. The median error improvement across all dataset was of 18%. The null hypothesis that starting with the representation obtained by Lockout offers the same performance as initializing the parameters randomly was rejected with a p = 7X10-11using a paired t-test. Equally relevant to the improvement in performance is the use of Lockout to identify the most important genes that help explain the many cancer problems. Table 6 shows the 5 most important genes selected by Lockout. Interestingly, all of these genes are related to hallmarks of cancer: cell proliferation, survival mechanism, and energy production. An interesting analysis, although beyond the scope of project, would be to check the genes with similar strength at the different nodes to discover unknown connections and interactions. Equally interesting would be to detect which specific genes are related to the different cancers. Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 Accordingly, the preceding merely illustrates the principles of the present disclosure. It will be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope. Furthermore, all examples and conditional language recited herein are principally intended to aid the reader in understanding the principles of the invention and the concepts contributed by the inventors to furthering the art and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the invention as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents and equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure. The scope of the present invention, therefore, is not intended to be limited to the exemplary embodiments shown and described herein.

[0002] Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 Table 1: Relative root mean squared error, defined as √1 − '^, for the validation and testing subsets corresponding to the early stopping and Lockout models. In all cases, the NN used had only one node with the same activation function that generated the data. Lockout considerably improves the early stopping solution. Note that testing subsets were generated with no noise added. Table 2: Microarray DNA datasets used in Experiment 1. Dataset size is given as N x p, where N is the number of observations and p the number of features.

[0003] Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 Table 3: Mean error and standard deviation over 100 different partitions for each method. The paired t-test p-values between Lockout and the different methods are shown in parentheses in each case. Table 4: Datasets from the UCI repository used in Experiment 2. Dataset size nomenclature is the same as in Table 2. Table 5: Mean error rate and standard deviation over 100 different partitions for each method. Nomenclature is the same as in Figure 3.

Claims

Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 WHAT IS CLAIMED IS:

1. A method for regularizing a neural network, comprising: obtaining a pretrained neural network (NN) architecture comprising at least two hidden layers, wherein each hidden layer comprises at least two nodes; applying an heuristic to create a path of solutions through the NN architecture, wherein the heuristic comprises: updating a first parameter, wherein the first parameter possesses a penalization of zero; updating a second parameter, wherein decreasing an absolute value of the second parameter decreases a loss function; updating a third parameter in a direction provided by an optimizer, wherein the third parameter is identified as important and wherein updating the third parameter increases the absolute value of the third parameter and decreases the loss function; and updating all remaining parameters in a direction contrary to the direction provided by the optimizer, wherein updating all remaining parameters decreases the absolute value of all remaining parameters and increases the loss function; removing at least one node from the NN architecture where all parameters at the at least one node equal zero; and outputting a trimmed NN.

2. The method of claim 1, applying the heuristic comprises applying the heuristic for each node in the NN architecture.

3. The method of claim 1 or 2, wherein removing the at least one node comprises removing at least one node from each hidden layer.

4. The method of any one of claims 1-3, wherein each hidden layer comprises at least 50 nodes.

5. The method of any one of claims 1-4, further comprising selecting at least one parameter to be considered for removal by the heuristic.Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 6. The method of claim 5, wherein the at least one parameter is all parameters.

7. The method of any one of claims 1-6, further comprising normalizing a gradient scale to be independent of its location.

8. The method of any one of claims 1-6, further comprising determining if a gradient scale is independent of its location.

9. The method of claim 8, further comprising, when the gradient scale is not independent of its location, normalizing the gradient scale to be independent of its location.

10. The method of any one of claims 1-9, further comprising selecting an optimizer and a dataset to use during application of the heuristic.

11. The method of claim 10, wherein the optimizer is selected from Stochastic Gradient Descent (SGD), Mini-Batch Gradient Descent, Adam (Adaptive Moment Estimation), RMSprop (Root Mean Square Propagation), Adagrad (Adaptive Gradient Algorithm), Adadelta, Nadam (Nesterov-accelerated Adaptive Moment Estimation), FTRL-Proximal (Follow-the-regularized- Leader Proximal), L-BFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno), Conjugate Gradient, SGD with Momentum, Proximal Gradient Descent, LBFGS-B (Limited-memory Broyden-Fletcher-Goldfarb-Shanno with Box Constraints), Frost, and Rprop (Resilient backpropagation).

12. The method of claim 10 or 11, wherein the dataset is a medical dataset.

13. The method of claim 12, wherein the dataset is selected from gene expression data and medical imaging data.

14. The method of claim 12 or 13, wherein the dataset is selected from microarray data, X- ray images, MRI images, and CT images.

15. The method of any one of claims 1-14, further comprising adjusting a forward function to compute after removing the nodes.Atty. Docket: UCSF-761WO SF-2024-025-3-PCT-0 16. The method of any one of claims 1-15, further comprising: inducing a sparsity penalization on a node to encourage all parameters of the node to reduce to zero; and creating grouping methodology to bring all parameters of the node into a single set.

17. The method of any one of claims 1-16, further comprising selecting one or more solutions from the heuristic based on sparsity of the NN architecture and performance of the NN architecture.

18. A neural network regularized by the method of any one of claims 1-17.

19. A non-transitory machine-readable medium comprising instructions that when executed by a processor direct the processor to execute the instructions, the instructions comprising the methods of any of claims 1-18.

20. A system for regularizing a neural network, comprising a processor and a memory, wherein the memory comprises instructions that when executed direct the processor to perform the method of any of claims 1-18.

21. The system of claim 20, wherein the processor and memory are housed in a consumer- level computing device.

22. The system of claim 21, wherein the consumer-level computing device is selected from a laptop computer, a desktop computer, a mobile phone, and a tablet.

Images