Training neural networks

Abstract

A computer-implemented method is disclosed for training a neural network, the method comprising formulating the neural network as a weighted sum of a plurality of sign pattern diagonal matrices, and training the neural network using a cost function. A neural network trained by the computer-implemented method, and a non-transitory computer-readable storage medium, are also disclosed.

Description

[0001] TRAINING NEURAL NETWORKS

[0002] Technical Field

[0003] The present application relates to neural networks, in particular training neural networks in an efficient manner.

[0004] Background

[0005] Training neural networks efficiently to build Al models and system is a rapidly evolving field, incorporating various technologies and approaches. A number of different approaches exist, including hardware acceleration, software optimization, distributed training, neural architecture search (NAS), edge computing, federated learning, and quantum computing. These different approaches aim to fulfil different needs, such as speed, efficiency, cost, and the specific nature of neural network tasks (e.g., image recognition, natural language processing).

[0006] The choice of approach often depends on the specific requirements and constraints of the project at hand. However, each approach comes with associated drawbacks and limitations. For example, hardware acceleration involves high energy consumption and is hardware dependent. Software optimization is complex to implement and can result in overfitting. NAS has a high computational cost and a lack of generalization. Edge computing has limited computational resources and has inherent security risks. The choice of an appropriate training approach for a given neural network often involves a trade-off between efficiency, accuracy, cost, and practical feasibility.

[0007] It is therefore desired to provide a method for training neural networks that attempts to mitigate at least some of these issues.

[0008] Summary

[0009] The present disclosure attempts to solve at least some of the aforementioned issues by providing a method for training a neural network in which the neural network is formulated as a weighted sum of a plurality of sign pattern diagonal matrices, and the weights of the formulated neural network are trained by solving an optimization problem. The method can be extended to both deep reinforcement learning (DRL) and recurrent neural networks (RNNs). According to a first aspect of the disclosure, there is provided a computer-implemented method for training a neural network, the method comprising formulating the neural network having an output y as: where: where hW represents a hidden layer I of the neural network having L layers, ptis a node p in layer I, 2Mtis the number of nodes in layer I, D® is a sign pattern diagonal matrix for the node p of layer I, X is a feature matrix, and w are weight parameters to be trained, and training the neural network using a cost function.

[0010] Optionally, the computer-implemented method comprises determining the sign pattern diagonal matrices recursively. Optionally, the computer-implemented method comprises determining the plurality of sign pattern diagonal matrices using a polynomial-time algorithm.

[0011] Optionally, the cost function is a convex cost function given by: subject to:

[0012]

[0013] Optionally, the output y of the neural network for a time index k is given by: where ckis a cost, xkis a measured state, ukis an input to a dynamic system, y is a discount factor, Q is a value function, and w are weight parameters to be trained.

[0014] Optionally, the cost function is a convex cost function given by: where 6 is a loss, ft is a design parameter, R is a regularisation term, and the method comprises training the neural network sequentially for the time indices k.

[0015] Optionally, the output y of the neural network for a time step t is given by: where h(t + 1) is a state of a dynamic system for timestep t + 1, and V / xk(t), , t / y(t), Uh(t), Why(t), bh(t), and by(t) are weights to be trained. Optionally, the cost function is a convex cost function given by: where ayis a ReLLI activation function, and the method comprises training the neural network forward in time by minimising the convex cost function. Optionally, the cost function is a convex cost function given by: and the method comprises training the neural network backward in time by minimising the convex cost function.

[0016] According to a second aspect of the disclosure, there is provided a neural network trained by the computer-implemented method of the first aspect.

[0017] According to a second aspect of the disclosure, there is provided a non-transitory computer-readable storage medium comprising instructions, which when executed by processing circuitry, cause the processing circuitry to perform the computer- implemented method of the first aspect.

[0018] The disclosed aspects, examples (including any preferred examples), and / or accompanying claims may be suitably combined with each other as would be apparent to anyone of ordinary skill in the art. Additional features and advantages are disclosed in the following description, claims, and drawings, and in part will be readily apparent therefrom to those skilled in the art or recognized by practicing the disclosure as described herein.

[0019] Brief Description of the Drawings

[0020] Examples are described in more detail below with reference to the appended drawings.

[0021] FIG. 1 is a flow chart of a computer-implemented method according to an example.

[0022] FIG. 2 is a schematic diagram of a computer system for implementing examples disclosed herein, according to an example

[0023] Like reference numerals refer to like elements throughout the description. Detailed Description

[0024] The detailed description set forth below provides information and examples of the disclosed technology with sufficient detail to enable those skilled in the art to practice the disclosure.

[0025] Training neural networks efficiently to build Al models and system is a rapidly evolving field, incorporating various technologies and approaches. These different approaches aim to fulfil different needs, such as speed, efficiency, cost, and the specific nature of neural network tasks (e.g., image recognition, natural language processing). However, each approach comes with associated drawbacks and limitations.

[0026] One approach is hardware acceleration. In one example, graphics processing units (GPUs) are used for training neural networks due to their ability to handle parallel tasks. This speeds up the training process by distributing computations across many cores. Tensor processing units (TPUs) are specifically designed for machine learning tasks. They are optimized for TensorFlow operations and can provide faster and more efficient processing for neural network training. Field-programmable gate arrays (FPGAs): FPGAs can be reprogrammed to suit specific tasks, including neural network training. They offer flexibility and can be more energy-efficient than GPUs. However, GPUs and TPUs consume significant amounts of energy, contributing to higher operational costs and environmental concerns. Furthermore, neural network models that are optimized for specific hardware (such as TPUs for TensorFlow) are limited in their flexibility and portability. High-end GPUs and TPUs are expensive, making them less accessible for individual researchers or small organizations.

[0027] Another approach is software optimization. Algorithms such as stochastic gradient descent (SGD) and its variants (e.g. Adam and RMSprop) are capable of training neural networks efficiently. Improvements in these algorithms can lead to faster convergence and better performance. Computing frameworks such as TensorFlow, PyTorch, and others enable parallel processing and distribution of tasks across multiple hardware units (like GPUs and TPUs). Techniques such as pruning (removing unnecessary weights) and compression (reducing the size of the model) help in making models more efficient without significant loss in accuracy. However, these approaches are complex in their implementation: Efficient algorithms and parallel computing frameworks often require sophisticated coding and deep understanding, which can be a barrier for some developers. Model pruning and compression can lead to overfitting or loss of accuracy if not carefully managed.

[0028] Another approach is distributed training. One example is data parallelism, which involves splitting the training data across multiple processors, where each processor trains a copy of the model on a subset of the data. Another example is model parallelism, where different parts of a neural network are trained on different processors. These approaches can be combined to optimize improve efficiency. However, in distributed training, significant communication overhead can occur, especially in model parallelism, where different parts of the model need to synchronize frequently. Distributing data across multiple nodes can also raise privacy concerns, especially when sensitive data is involved.

[0029] Another approach is NAS. This automates the design of neural networks, finding the most efficient architecture for a specific task. This can lead to more efficient models than those designed by humans. However, NAS can be computationally intensive and time-consuming, as it involves evaluating numerous architectures. Furthermore, architectures found by NAS can be highly specialized and not generalize well to different tasks.

[0030] Another approach is edge computing. This involves moving part of the computation to edge devices (such as smartphones and loT devices) in order to reduce the load on central servers and speed up the training by processing data where it is generated. However, edge devices often have limited processing power and memory, which can constrain the complexity of models that can be run. Distributing computation to edge devices also increases the attack surface for potential cybersecurity threats.

[0031] Another approach is federated learning. This is a technique where model training is distributed across many devices, which then share their updates. It is particularly useful for privacy-preserving applications. However, the data can be highly variable across different devices, leading to challenges in model convergence and performance. Federated learning relies on consistent network connectivity, and latency can be a significant issue, especially with large numbers of nodes.

[0032] Another approach is quantum computing. Quantum computers could potentially solve complex optimization problems faster than classical computers. However, quantum computing is still in its infancy. The choice of approach often depends on the specific requirements and constraints of the project at hand. As will be appreciated from the summary above, the choice of an appropriate training approach for a given neural network often involves a trade-off between efficiency, accuracy, cost, and practical feasibility.

[0033] Training a neural network is often posed as a non-convex optimization problem. Consider the input to a neural network training algorithm to be an n x d feature matrix X, and the vector y of labels to be predicted by a neural network with L layers given by: where y are the predicted labels and W(r), W^2\ ..., WLare weight parameters for each layer of the neural network to be learned by the neural network training algorithm. The training can be performed using any suitable optimization or cost function, for example using a non-convex, concave, or convex optimization problem or cost function. In some examples, the mean squared error can be used. For example, the training can be posed as a non- convex optimization problem given by: where p is a is regularization constant that can be set according to the particular task. Finding an optimal solution to non-convex optimization problems is difficult in general, any many cases are known to be so-called “NP-hard”.

[0034] To remedy this, systems, methods and other approaches are provided for training a neural network in which the neural network is formulated as in a particular way that allows the weights of the formulated neural network to be trained by solving a convex optimization problem. The method can be also applied to both DRL and RNNs in the manner described below.

[0035] To do this, a neural network should be formulated in a particular way. In particular, the neural network can be formulated having an output y as follows:

[0036] where represents a hidden layer I of the neural network having L layers, ptis a node p in layer I, 2Mtis the number of nodes in layer I, Dp1^ is a sign pattern diagonal matrix for the node p of layer I, X is a feature matrix, and w are weight parameters to be trained.

[0037] The sign pattern diagonal matrices Dpifor layers 1 to L can be found recursively with respect to the feature matrices using a polynomial-time algorithm. An example of such an algorithm is described in “Constructing arrangements of lines and hyperplanes with applications’’ by Edelsbrunner, H., ORourke, J., and Seidel, R., SIAM Journal on Computing, 15(2):341-363, 1986. In this example, X(o)= X, and DPl, 1 = are sign matrices obtained from over all possible vectors u, where = 0 and Da= 1 otherwise. This provides a computationally efficient way to determine the sign pattern diagonal matrices.

[0038] Training the neural network can be performed using any suitable optimization or cost function, for example using a non-convex, concave, or convex optimization problem or cost function. In some examples, the mean squared error can be used. For example, for static neural networks, a convex cost function can be formulated using the mean square error as follows: subject to

[0039] 1 = 0, 1, -,L - 1

[0040] The weights w can be obtained by solving the convex optimization problem of equation (4) in order to train the neural network. As the optimization problem is convex, it can be solved using known efficient convex optimization techniques. For example, such techniques as described in “Convex Optimization" by Stephen Boyd and Lieven Vandenberghe, or tools such as cvxpy in Python, may be used. This ensures that neural networks can be trained with the high accuracy while demanding substantially fewer computational resources. The training process can therefore be performed more quickly than currently available techniques, and does not require any form of distributed computing, thus negating any concerns over data privacy. Furthermore, the training process can be generalised for multiple (e.g. beyond 3) layers, meaning deeper neural networks can be trained with convex optimization. It will be appreciated that the weights w can also be obtained by solving an optimization problem comprising convex and non- convex terms.

[0041] The approaches disclosed herein can also be applied to DRL. In DRL, a dynamic system with an unknown model can be modelled as: where xkis a measured state and ukis an input to the dynamic system for the time indices k = 1, It is desired to find the optimal input it to minimize a cost c (or maximize a reward r = — c) given by: where y is a discount factor that can be set according to the particular task. One approach to finding the input u is Q-learning, where a so-called “value function” Q is modelled with a neural network. The neural network can be formulated to model the value function Q as follows: where w are weight parameters to be trained. The neural network formulation of equation (7) is in the form of equation (3), meaning that the weights w can be obtained by solving a convex optimization problem.

[0042] Training the neural network can be performed using any suitable optimization or cost function, for example using a non-convex, concave, or convex optimization problem or cost function. In some examples, the mean squared error can be used. For example, a convex cost function can be formulated using the mean square error as follows: where 6 is a loss, ft is a design parameter, and R is a regularisation term (for example a convex cost function.

[0043] The training of Q(x,u, wk) can be performed sequentially for the time indices k. By using this approach, building DRL models will be more efficient and accurate since the model parameters will always be optimal.

[0044] The approaches disclosed herein can also be applied to RNNs. RNNs are an extension of neural networks where dynamics are allowed for through a state in a model. Consider a time series given by n data samples (%i,yi), (x2,y2), - - -, {xt,yt),... , (ycn,yn). Suppose that a dynamic system behind the time series is given by: where ht+1is a state of the dynamic system for timestep t + 1, and t is a timestep. In some examples, equation (10) may be replaced by y = g^x^ h.^. A corresponding RNN can be modelled with a neural network according to: where h(t + 1) is a state of the dynamic system for timestep t + 1 (the same as ht+1above), and , t / y(t), Uh(t), Why(t), bh(t), by(t), chand cyare weights to be trained. One or more of the weights may be constant. The neural network formulation of equation (12) is in the form of equation (3), meaning that the weights can be obtained by solving a convex optimization problem.

[0045] Training the neural network can be performed using any suitable optimization or cost function, for example using a non-convex, concave, or convex optimization problem or cost function. In some examples, the mean squared error can be used. For example, the neural network can be trained forward in time by minimising a convex cost function formulated using the mean square error as follows: where ayis a ReLLI activation function. A similar approach can be used for a gated ReLLI or threshold activation function given by / (%) = w for x > 0, and / (%) = 0 otherwise, where w are trainable weights. The neural network can be trained backward in time by minimising a convex cost function formulated using the mean square error as follows:

[0046] Both of equations (13) and (14) represent training of a 2-layer neural network with respect to the relevant weights. This can be performed sequentially. By using this approach to train neural networks for RNNs, the training becomes more efficient and accurate. The training can be done forward and then backward in time, and vice versa.

[0047] The approaches disclosed above can be summarised in the form of the computer- implemented method 100 shown in FIG. 1. The method is for training a neural network using convex optimisation.

[0048] In a first step 110, the neural network is formulated as defined in equation (3). In particular, the neural network is formulated as a weighted sum of a plurality of sign pattern diagonal matrices. In some examples, the sign pattern diagonal matrices can be found recursively. This can be achieved using a polynomial-time algorithm.

[0049] In some examples, the neural network is formulated for DRL using equation (7). In some examples, the neural network is formulated for RNNs using equation (12).

[0050] In a second step 120, the neural network is trained using a cost function. Any cost function can be used. In some embodiments, a convex cost function can be used. In some embodiments, a cost function can be formulated using the mean square error. This can be achieved for static neural networks using the convex cost function of equation (4). For the DRL formulation of equation (7), the convex cost function of equation (8) can be used. For the RNN formulation of equation (12), the convex cost functions of equation (13) and (14) can be used. The optimization problem can be solved using known efficient convex optimization techniques to train the neural network.

[0051] The approaches disclosed herein ensure that neural networks can be trained with the high accuracy while demanding substantially fewer computational resources. The training process can therefore be performed more quickly than currently available techniques, and does not require any form of distributed computing, thus negating any concerns over data privacy. Furthermore, the training process can be generalised for multiple (e.g. beyond 3) layers, meaning deeper neural networks can be trained with convex optimization. FIG. 2 is a block diagram illustrating an exemplary computer system 200 in which examples of the present disclosure may be implemented. In particular, the computer system 200 may, according to some examples, be configured to cause performance of the method 100 of FIG. 1. This example illustrates a computer system 200 such as may be used, in whole, in part, or with various modifications, to provide the functions of the disclosed system. For example, various functions may be controlled by the computer system 200, including, merely by way of example, generating, determining, identifying, receiving, etc. The computer system 200 is shown comprising hardware elements that may be electrically coupled via a bus 290. The hardware elements may include processing circuitry 210, one or more input devices 220 (e.g., a mouse, a keyboard, etc.), and one or more output devices 230 (e.g., a display device, a printer, etc.). The computer system 200 may also include one or more memories 240.

[0052] The processing circuitry 210 may include any number of hardware components for conducting data or signal processing or for executing computer code stored in memories 240. The processing circuitry 210 may, for example, include a general- purpose processor, an application specific processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a circuit containing processing components, a group of distributed processing components, a group of distributed computers configured for processing, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The processing circuitry 210 may further include computer executable code that controls operation of the programmable device.

[0053] The bus 290 provides an interface for system components including, but not limited to, the memories 240 and the processing circuitry 210. The bus 290 may be any of several types of bus structures that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and / or a local bus using any of a variety of bus architectures.

[0054] The memories 240 may be one or more devices for storing data and / or computer code for completing or facilitating methods described herein. The memories 240 may include database components, object code components, script components, or other types of information structure for supporting the various activities herein. The memories 240 may include non-volatile memory (e.g., read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read- only memory (EEPROM), etc.), and volatile memory (e.g., random-access memory (RAM)), or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a computer or other machine with processing circuitry 210. The one or more input devices 220 are configured to receive input and selections to be communicated to the computer system 200 when executing instructions. The one or more output devices 230 are configured to forward output, such as to a display, a video display unit.

[0055] The computer system 200 may additionally include a computer-readable storage media reader 250, a communications system 260 (e.g., a modem, a network card (wireless or wired), an infrared communication device, Bluetooth™ device, cellular communication device, etc.), and a working memory 280, which may include RAM and ROM devices as described above. In some embodiments, the computer system 200 may also include a processing acceleration unit 270, which can include a digital signal processor, a special-purpose processor and / or the like.

[0056] The computer-readable storage media reader 250 can further be connected to a non- transitory computer-readable storage medium, together (and, optionally, in combination with the storage devices 240) comprehensively representing remote, local, fixed, and / or removable storage devices plus storage media for temporarily and / or more permanently containing computer-readable information. The communications system 260 may permit data to be exchanged with a network, system, computer and / or other component described above.

[0057] The computer system 200 may also comprise software elements, shown as being currently located within the working memory 280, including an operating system 288 and / or other code 284. Software of the computer system 200 may include code 284 for implementing any or all of the functions of the various elements of the architecture as described herein. For example, software, stored on and / or executed by a computer system such as the system 200, can provide the functions of the disclosed system. All or a portion of the examples disclosed herein may be implemented as a computer program stored on a transitory or non-transitory computer-usable or computer-readable storage medium (e.g., single medium or multiple media) which includes complex programming instructions (e.g., complex computer-readable program code) to cause the processing circuitry to carry out actions described herein. Thus, the computer- readable program code of the computer program can comprise software instructions for implementing the functionality of the examples described herein when executed by the processing circuitry 210.

[0058] It should be appreciated that alternative embodiments of a computer system 200 may have numerous variations from that described above. For example, customised hardware might also be used and / or particular elements might be implemented in hardware, software (including portable software, such as applets), or both. The computer system 200 may be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, or the Internet. While only a single device is illustrated, the computer system 200 may include any collection of devices that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. Furthermore, connection to other computing devices such as network input / output and data acquisition devices may also occur.

[0059] The operational actions described in any of the exemplary aspects herein are described to provide examples and discussion. The actions may be performed by hardware components, may be embodied in machine-executable instructions to cause a processor to perform the actions, or may be performed by a combination of hardware and software. Although a specific order of method actions may be shown or described, the order of the actions may differ. In addition, two or more actions may be performed concurrently or with partial concurrence.

[0060] The terminology used herein is for the purpose of describing particular aspects only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. It will be further understood that the terms "comprises," "comprising," "includes," and / or "including" when used herein specify the presence of stated features, integers, actions, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, actions, steps, operations, elements, components, and / or groups thereof.

[0061] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0062] It is to be understood that the present disclosure is not limited to the aspects described above and illustrated in the drawings; rather, the skilled person will recognize that many changes and modifications may be made within the scope of the present disclosure and appended claims. In the drawings and specification, there have been disclosed aspects for purposes of illustration only and not for purposes of limitation, the scope of the disclosure being set forth in the following claims.

Claims

Claims1. A computer-implemented method (100) for training a neural network, the method comprising: formulating (110) the neural network having an output y as: where:where represents a hidden layer I of the neural network having L layers, ptis a node p in layer I, 2Mtis the number of nodes in layer I, D® is a sign pattern diagonal matrix for the node p of layer I, X is a feature matrix, and w are weight parameters to be trained; and training (120) the neural network using a cost function.

2. The computer-implemented method (100) of claim 1 , comprising determining the sign pattern diagonal matrices recursively.

3. The computer-implemented method (100) of claim 2, comprising determining the plurality of sign pattern diagonal matrices using a polynomial-time algorithm.

4. The computer-implemented method (100) of any preceding claim, wherein the cost function is a convex cost function given by:subject to:

5. The computer-implemented method (100) of any claims 1 to 3, wherein the output y of the neural network for a time index k is given by:where ckis a cost, xkis a measured state, ukis an input to a dynamic system, y is a discount factor, Q is a value function, and w are weight parameters to be trained.

6. The computer-implemented method (100) of claim 5, wherein the cost function is a convex cost function given by:where 0 is a loss, ft is a design parameter, R is a regularisation term, and the method comprises training (130) the neural network sequentially for the time indices k.

7. The computer-implemented method (100) of any claims 1 to 3 wherein the output y of the neural network for a time step t is given by:where h(t + 1) is a state of a dynamic system for timestep t + 1, and Wxh(t), , Uy(t), Uh(t), Why(t), bh(t), and by(t) are weights to be trained.

8. The computer-implemented method (100) of claim 7, wherein the cost function is a convex cost function given by:where ayis a ReLLI activation function, and the method comprises training (130) the neural network forward in time by minimising the convex cost function.

9. The computer-implemented method (100) of claim 8, wherein the cost function is a convex cost function given by:and the method comprises training (130) the neural network backward in time by minimising the convex cost function.

10. A neural network trained by the computer-implemented method (100) of any of claims 1 to 9.

11. A non-transitory computer-readable storage medium comprising instructions, which when executed by processing circuitry, cause the processing circuitry to perform the computer-implemented method (100) of any of claims 1 to 9.

Images