Parallel residual neural network architechture and system and method for trainning a residual neural network

Abstract

Systems and methods for training a residual neural network are described. One of the methods includes arranging a plurality of residual units into one or more subsets corresponding to a plurality of warped layers; configuring a parallelizable warp operator to compute an output of a warped layer from an input to the warped layer using a first-order Taylor series approximation; and determining a final parameter setting for a plurality of parameters of the residual neural network by training the residual neural network on a training set, wherein training the residual network comprises applying the parallelizable warp operator to each of the plurality of warped layers.

Description

1 RESIDUAL NEURAL NETWORK ARCHITECTURE AND SYSTEM AND METHOD FOR 2 TRAINING A RESIDUAL NEURAL NETWORK 3 TECHNICAL FIELD 4

[0001] The following relates generally to artificial neural networks and more specifically to 5 training a residual neural network. 6 BACKGROUND 7

[0002] Deep Convolution Neural Networks (CNN) have been used in image recognition tasks 8 with great success. Some conventional approaches include AlexNet, VGG, Inception, and 9 Residual Networks (ResNet). 10

[0003] Training a deep neural network is not an easy task. As the gradient at each layer is 11 dependent upon those in higher layers multiplicatively, the gradients in earlier layers can vanish 12 or explode, ceasing the training process. The gradient vanishing problem is significant for neuron 13 activation functions such as the sigmoid, where the gradient approaches zero exponentially away 14 from the origin on both sides. The standard approach to combat vanishing gradient is to apply 15 Batch Normalization (BN) followed by the Rectified Linear Unit (ReLU) activation. More recently, 16 skip connections have been proposed to allow previous layers to propagate relatively unchanged. 17 Using this approach, it may be possible to train extremely deep networks of hundreds of layers; 18 for example, ResNets have been trained with about one thousand layers. 19

[0004] As the number of layers grows large, so does the training time. In operation of a neural 20 network in a data processing task, to evaluate the neural network's output, one needs to 21 propagate the input of the network layer by layer in a procedure known as forward propagation. 22 Similarly, during training of the neural network, one needs to propagate the gradient of the loss 23 function from the end of the network to update the model parameters, or weights, in each layer of 24 the network using gradient descent. The complexity of forward and backward propagation is O(K), 25 where K is the number of layers in the network. There is a need to speed up the training process 26 in deep neural networks. 27 SUMMARY 28

[0005] In one aspect, a system for training an artificial residual neural network having a plurality 29 of residual units, each residual unit comprising a subset of a plurality of weight layers and a 30 shortcut connection mapping an input of the residual unit to be added to an output of a final layer 31 of the subset of the plurality of weight layers, and having a plurality of parameters associated with CA 3020884 2018-10-15 1 the plurality of weight layers, is provided, the system comprising a plurality of warp units linked in 2 series, each warp unit comprising: a subset of units being an ordered and non-overlapping subset 3 of the plurality of residual units decoupled from each other; at least one derivative unit, each 4 derivative unit computed based on one of the residual units in the subset of units; a warp unit 5 input being the input of a first residual unit in the subset of units prior to decoupling; and a warp 6 unit output being the output of a last residual unit in the subset of units prior to decoupling, the 7 warp unit output being computed from the sum of the warp unit input and each operation upon 8 the input by each residual unit in the subset of units and the at least one derivative unit; and one 9 or more non-transitory computer storage media for storing the plurality of parameters computed 10 when applying the system to a training set of data. 11

[0006] These and other aspects are contemplated and described herein. It will be appreciated 12 that the foregoing summary sets out representative aspects of a system and method for training 13 a residual neural network and assists skilled readers in understanding the following detailed 14 description. 15 DESCRIPTION OF THE DRAWINGS 16

[0007] A greater understanding of the embodiments will be had with reference to the Figures, 17 in which: 18

[0008] Fig. 1 is an architecture diagram of a residual unit in a conventional residual neural 19 network; 20

[0009] Fig. 2 is a block diagram of a conventional residual neural network; 21

[0010] Fig. 3 is a block diagram of an embodiment of a system for training a residual neural 22 network; 23

[0011] Fig. 4 is a further embodiment of a system for training a residual neural network; 24

[0012] Fig. 5 is a table illustrating an exemplary structure of a system for training a residual 25 neural network, in accordance with an embodiment of the presently disclosed system; 26

[0013] Fig. 6 and Fig. 7 are tables illustrating differences in test error percentages and 27 improvement percentages in training speed between a conventional residual neural network and 28 a residual neural network trained using an embodiment of the presently disclosed system on the 29 CIFAR10 and CIFAR100 benchmark datasets; 30

[0014] Fig. 8 is a flowchart illustrating a method for training a residual neural network, in 31 accordance with an embodiment of the presently disclosed system; 2 CA 3020884 2018-10-15 1

[0015] Fig. 9 is a table illustrating differences in validation error percentages between a 2 conventional residual neural network and a residual neural network trained using an embodiment 3 of the presently disclosed system on the lmageNet benchmark dataset; and 4

[0016] Fig. 10 is a chart illustrating differences in validation error between a conventional 5 residual neural network and a residual neural network trained using an embodiment of the 6 presently disclosed system on the lmageNet benchmark dataset. 7 DETAILED DESCRIPTION 8

[0017] Embodiments will now be described with reference to the figures. For simplicity and 9 clarity of illustration, where considered appropriate, reference numerals may be repeated among 10 the figures to indicate corresponding or analogous elements. In addition, numerous specific 11 details are set forth in order to provide a thorough understanding of the embodiments described 12 herein. However, it will be understood by those of ordinary skill in the art that the embodiments 13 described herein may be practiced without these specific details. In other instances, well-known 14 methods, procedures and components have not been described in detail so as not to obscure the 15 embodiments described herein. Also, the description is not to be considered as limiting the scope 16 of the embodiments described herein. 17

[0018] Any module, unit, component, server, computer, terminal or device exemplified herein 18 that executes instructions may include or otherwise have access to computer readable media 19 such as storage media, computer storage media, or data storage devices (removable and / or non- 20 removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media 21 may include volatile and non-volatile, removable and non-removable media implemented in any 22 method or technology for storage of information, such as computer readable instructions, data 23 structures, program modules, or other data. Examples of computer storage media include RAM, 24 ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks 25 (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other 26 magnetic storage devices, or any other medium which can be used to store the desired 27 information and which can be accessed by an application, module, or both. Any such computer 28 storage media may be part of the device or accessible or connectable thereto. Any application or 29 module herein described may be implemented using computer readable / executable instructions 30 that may be stored or otherwise held by such computer readable media. 31

[0019] To speed up the training process for a deep neural network, it may be advantageous to 32 have a shallower network that accurately approximates the deep network. Applicant has found 3 CA 3020884 2018-10-15 1 that residual neural network (ResNet) architectures permit such an approximation. However, 2 Applicant has further established a faster approach to training K layer ResNets in 0(1) time as 3 opposed to O(K) time with forward and backward propagation. 4

[0020] In one aspect, an artificial neural network architecture is provided for training a ResNet. 5 In another aspect, a system and method for training a ResNet are provided. The system may 6 comprise a parallelizable warp operator for training the Res Net by restructuring residual units of 7 the ResNet using a first-order Taylor series approximation. 8

[0021] The following can be used to significantly reduce the training times for a ResNet. This 9 speedup in training may be achieved by decoupling the ResNet into individual residual units, or 1 O layers, and training them in parallel using the parallelizable warp operator. The number of 11 parallelizable computations grows exponentially with the number of ResNet layers skipped; 12 therefore, it is a significant improvement that most, or even all, of the Res Net layers can be trained 13 in the time of just a few. 14

[0022] Referring now to Fig. 1, a single residual unit 100 in a conventional ResNet is shown. 15 The residual unit 100 is composed of a stack of weight layers 102 and a shortcut connection 104. 16 The number of weight layers 102 can vary, e.g. Fig. 1 shows two weight layers 102a, 102b but 17 there may be more; single-layer residual units are uncommon, however, due to a lack of 18 advantages. The weight layers 102 can also vary in size and type, e.g. they may be fully- 19 connected layers or convolutional layers. 20

[0023] Shortcut connection 104 skips one or more weight layers 102. The purpose of shortcut 21 connections is to perform an identity mapping, such that the shortcut connection 104 maps an 22 input 106 that is given to the residual unit 100 to be added to an output 108 of a final layer 102b 23 of the weight layers 102. A person of skill will readily understand operation of a residual unit of a 24 conventional ResNet as shown. 25

[0024] Fig. 2 illustrates a conventional ResNet 200, which typically consists of a long chain of 26 residual units 100, each generally having the structure shown in Fig. 1, connected in sequence 27 by connections 202. Previous studies suggest that ResNets behave as an ensemble of shallow 28 networks and that randomly deactivating residual units during training (similar to the drop-out 29 technique) appears to improve performance. Applicant has found based on these observations 30 that the output of a residual unit can be considered as a small perturbation of the input. 31

[0025] ResNets appear to exhibit an iterative estimation behavior as opposed to representation 32 learning. Within one ResNet stage (Fig. 2 illustrates a single stage, where a stage is a chain of 4 CA 3020884 2018-10-15 1 residual units having outputs of the same dimension), which has shortcut connections in all but 2 the first residual units, the feature map representation appears to steadily converge to one that 3 minimizes the loss function at the end of the stage. ResNets appear to perform representation 4 learning in the first few layers and iterative estimation in the rest within the same stage. 5

[0026] ResNets may be explained by applying the perturbation theory. Consider the input xi of 6 the Hh residual unit of the ResNet 200 and its output xi+i, where 7 ( 1) 8

[0027] Typically, h(xi) is taken to be an identity mapping, hi(xi) = xi, which represents the 9 shortcut connection. When the feature maps are down sampled, h may be a 1x1 convolution layer 10 with a stride of 2. The function Fi may a combination of convolution, normalization and non- 11 linearity layers, so that Wi collectively represents the weights of all layers in h 12

[0028] Applicant has determined that the interpretation of ResNets as an ensemble of 13 subnetworks is accurate up to the first order in F with identity mapping. The identity mapping 14 causes F to be small compared to the input x so one can approximate the output of a chain of 15 residual units by a series expansion. For instance, the output of two residual units x3 is related to 16 the input of the first unit. When xk is expressed in terms of xk+i, which is referred to as an iteration. 17 The following equations show two iterations. 19 (2) 20 where F2 (x1, Wi) denotes the partial derivative of F2 with respect to x1 . A Taylor series expansion 21 in powers of F1 was performed on F2 in the second line above. The Taylor expansion for 22 multivariate functions is f(x + a) = x + aVxf (x) + O(E2). The first-order Taylor series expansion 23 is almost exact when ReLU activations are used. Equation (2) can be interpreted as an ensemble 24 sum of subnetworks. Although equation (2) relates to the expansion for two iterations, additional 25 iterations could be applied for additional residual units. 26

[0029] Extending equation (2) to apply across K residual units: 27 (3) 28 where Sr;={l,2,·--,K}, 0 c isanysubsetofthepowerset P(SiJ andthelasttermarisesfrom 29 higher order terms from the Taylor expansion. The indices c are ordered such that 5 CA 3020884 2018-10-15 1 2 3 (K) (K) c(i)>c(i-J)> ••• >c(l) .Forafixedcardinalitylt1cl=k,thereare k terms,where k isthe corresponding binomial coefficient. The binomial nature of subnetworks is evident from this formulation. 4

[0030] In Equation (3), the sum over all subsets of SK, <Ye, can be rewritten as a sum over the 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 (K) cardinality k. Each k has a binomial number of terms k . The binomial nature of subnetworks is evident from this formulation. For each k, there are k factors of F in the sum. Thus k can be interpreted as the subnetwork depth.

[0031] It may be observed that the gradient in a ResNet is the sum from all subnetwork contributions, including the identity network. Skilled persons are familiar with suitable magnitudes of subnetwork gradients for an 110-layer ResNet. If one takes F to have ReLU non-linearity. Then F"(x)= O . The non-trivial gradient can be expressed as just a product of F and ( F 't- 1 . Thus, it can be seen that the gradient norm decreases exponentially with subnetwork depth k.

[0032] Applicant has further determined that the first order perturbations are sufficient for approximating the ResNet and that higher order perturbations can be discarded. For ReLU nonlinearity, the second-order perturbations vanish almost exactly. The second-order perturbation terms all contain the Hessian F' '(x)= O . However, after the network is trained, the only nonlinear function in F , ReLU, is non-linear only at the origin; batch normalization layers are nonlinear during training, due to the scaling by the sample variance. Therefore, all second-order perturbations vanish with almost exactly. The same argument applies to higher orders.

[0033] Fig. 3 illustrates the presently disclosed warped residual neural network (also referred to herein as WarpNet). The WarpNet 300 comprises a plurality of warp units 304 linked in series, each warp unit 304 comprising a non-overlapping subset of the plurality of ordered residual units 302 which have been decoupled, such that the connections 202 between residual units 100 in Fig. 1 have been broken or removed. The residual units of any particular warp unit 304 are ordered, that is sequential, in the ResNet. Further, the residual units 302 are connected in parallel within each warp unit and each residual unit is independent of other residual units. A link from the input to the output of the warp unit 304, and at least one F' , each a derivative unit computed based on the residual unit F , are also connected in parallel with the residual units 302. The 6 CA 3020884 2018-10-15 1 included derivative units are determined based on a Taylor Series expansion to compute the 2 output from the input. 3

[0034] The input to the warp unit 304 is selected from the first input among the included residual 4 units 302, and the output to the warp unit 304 is selected from the last output among the included 5 residual units 302. For example, in Fig. 2, a first residual unit F 1 accepts input x I and produces 6 output x2 = x1 + F1 and a second residual unit F2 accepts input x2 (which was the output of 7 the first unit) and produces output x3 = x2 + F2 . By contrast, in the WarpNet 300, a first warp unit 8 304a, comprising residual units 302a consisting of F 1 and F 2 accepts input X1 and directly 9 produces output x, , bypassing X2 altogether. Skilled persons will appreciate the processing 10 savings of such an approach. 11

[0035] More specifically, in Fig. 3, warp unit 302a comprises residual units F 1 , F2 , derivative 12 unit F '2 , and a link from the input x I to the output X_i , connected in parallel. The output of the 13 warp unit 304 is the summation of the outputs of the included residual units, derivative units and 14 the input to the warp unit 304. It is contemplated that a particular warp unit could comprise 15 additional residual units, as described below, for bypassing additional points in the ResNet. If 16 there are additional residual units, then there may be additional derivative units as well. 17

[0036] In Fig. 3, additional warp units 304 are shown for various parallel sets of residual units 18 302. It is contemplated that a particular WarpNet need not have uniformity amongst warp units 19 304; namely, it is possible in specific implementations for one particular warp unit to comprise a 20 particular number of residual units, and another warp unit to comprise a different number of 21 residual units. 22

[0037] The computation in the warp unit 304 is different from that in the conventional ResNet 23 200. As illustrated in Fig. 4, a system 400 is provided for training a ResNet using a warp operator 24 402 to compute the output (i.e., x K+ 1 ) of the warp unit 304 directly from the input (i.e., x 1 ), as 25 shown in Equation (3). The number of weights in the warp unit 304 is the same as the one in the 26 original ResNet 200 for K consecutive residual units. For instance, the weights W 1 , H72 and 27 WK may be present in the warp unit 304. However, these weights can be used and updated in 28 parallel due to the use of the warp operator 402. Also note that, in Fig. 3, F2 shares the same 29 weights with F '~ , as they come from the same residual unit in the original ResNet 200. 30

[0038] The warp operator 402 may be first applied for forward propagation. A modified forward 7 CA 3020884 2018-10-15 1 propagation rule for the WarpNet 300 can be described using warp operator 402, 'f;rarp, which 2 may, for example, be implemented by a graphics processing unit (GPU). The expression for T!arp 3 is derived from Equation (3) by dropping the negligible 0(£ 2 ) term, that is, by using the Taylor 4 series expansion to the first order: 5 6 7 8 9 10 11 12 13 14 15 16 17 Note that T / Sarp can be calculated in a parallelizable manner for all K. For example, where K=2, where Wi corresponds to the weights in the i-th residual unit in the original ResNet 200.

[0039] The warp operator 402 may be applied for backpropagation as well. A modified backpropagation rule for the WarpNet 300 can also be described using the warp operator TJ;arp· If the upstream gradient OU O X 5 is known and in order to compute au a W1 for gradient descent, the gradient is first backpropagated down from X 5 to x,. With x5 = T1{arp(x3 ), the backpropagated gradient can be derived as: d L = d L [I+ a F3(X1, W3) + a Fix], W4) + a F4(X;, W4) a Fi(X;, W1)] ox, OX5 ax.,, axl ax, ax, where 1 is the identity matrix and the derivative of F;' is set to zero for ReLU non-linearities. Applicant has shown that the same kind of parallelism in the warp operator 402 is present for backpropagation as for forward propagation. 18

[0040] The weight gradient for updates can be evaluated as follows: 19 20 21 22 23 ~= a L [ a F 1 ( x 1. w 1} + a F 2 ( x 1. w 2) a F 1 ( x 1• w 1)] a ,,,, 1 ax 7. a w I ax I a w1 The update rule is similar for U\ . Rules for the all other weights in the WarpNet 300 can be obtained in the same way, Theweights W1 and w" can be updated in parallel independently. The derivatives oF)ox1 8 CA 3020884 2018-10-15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 in a ua JJ / 1 and other derivatives needed in a ua X; are already computed in the forward pass which could be saved and reused, for example by storing it in memory 404. Furthermore, all derivatives needed in a LI ax, can also be computed in the forward pass.

[0041] In an aspect, as previously mentioned, the warp operator 402 may be implemented on one or more processing units such as CPUs or GPUs. In another aspect, the warp operator 402 may be part of another module, such as a learning module, that is responsible for training the WarpNet 300.

[0042] Memory 404 may be provided for storing activations and learned weights for each weight layer 102. The memory 404 may further store a training set comprising training data. The training data may, for example, be used for image classification in which case the training data may comprise images with known classifications. The memory 404 may further store a validation set comprising validation data.

[0043] During the training stage, the Warp Net 300 optimizes weights for each weight layer 102. After learning, the optimized weight configuration can then be applied to test data. Exemplary applications of such a neural network include image classification, machine translation, object recognition, speech recognition and genomic-oriented applications.

[0044] During the training stage, a plurality of training cases are presented to the WarpNet 300. In order to determine a final parameter setting for the parameters of the WarpNet 300, the warp operator 402 can be applied to each warp unit 304 to perform a computation of the output of the residual units 302 using just the input to the warp unit 304, as illustrated in Fig. 4. Each independent residual unit within the warp unit 304 is activated in parallel for the purpose of this computation. Advantageously, this produces comparable results to the original ResNet 200 while having a significant reduction in training time due to the parallelism.

[0045] Referring now to Fig. 5, an exemplary structure of a system for training a ResNet is shown. In order to ensure the validity of the series expansion, the 1x1 convolution layers on shortcut connections may be replaced by an average pooling layer and a concatenate layer to reduce the spatial dimensions of feature maps and multiply their channels. In an aspect, a wide residual architecture may be adopted. The average pooling and concatenate layers may be used so that the identity mapping is preserved across dimensionality changing units. The convolution blocks may be comprised of the following layers, from input to output, BN-Conv-BN-ReLU-Conv- T(K) X N V BN, as illustrated in Fig. 5. The layers [ '""'P "'"'P ] represent forward propagating ' warp times, 9 CA 3020884 2018-10-15 1 2 3 4 5 6 7 8 9 such that Xi+K= r~~J7,(xi' W;, Wi+ 1' ••• , Wi+x- i) and the indices i correspond to the indices in the original ResNet 200.

[0046] Experimental results indicate that a computation bottleneck arises from the BN layers in F '2 . The reason being the gradient of BN layers contains an averaging operation that is expensive to compute. In an aspect, all BN layers in F '2 may be removed from the system. This results in a departure from the series approximation but experimental results indicate that the network still trains well. This is because the normalizing layers are still being trained in Fu . To further improve the speedup, it may be beneficial to omit the Fi block in the derivative term F'2F1 10

[0047] In experimentation conducted using Tensorflow, a network was implemented with 11 parameters k,., , K and N .,,,.p , where k,. is the widening factor, and K is the warp factor. 12 Tensorflow's automatic differentiation for backpropagation may be employed, where the gradients 13 are calculated by sweeping through the network through the chain rule. 14

[0048] Referring now to Fig. 6 and Fig. 7, differences in test error percentages between a 15 conventional residual neural network and a residual neural network trained using an embodiment 16 of a system for training a residua! neural network are shown. The corresponding relative speed- 17 up with respect to residual neural network is shown in parentheses. Fig. 6 corresponds to K==2 18 and Fig 7. corresponds to K=3. The conventional ResNet may have the same pooling and 19 concatenate structure as the architecture shown in Fig. 5, but with each warp operator replaced 20 by two residual units. Common data augmentation techniques, such as whitening, flipping and 21 cropping may be used. Experiments may be performed for 80000 iterations, or 204 epochs; a 22 training batch size of 128; and an initial learning rate is 0.1, which drops by a factor of 0.1 at 23 epochs 60, 120, and 160, with a weight decay of 0.0005. 24

[0049] To exploit the parallelism of the WarpNet architecture and reduce the memory footprint 25 at the same time, it may be advantageous to compute F, on one GPU, F 2 and F ': at the 26 same time on another GPU because they share the same weights. Experimental results indicate 27 significant speed up in both forward and backward propagation. 28

[0050] To further speed up WarpNet, the modification where F1 is replaced by Xi was used 29 and experiments showed that this modification does not affect the predictive performance of 30 WarpNet. 10 CA 3020884 2018-10-15 1

[0051] Experimental results suggest that the computation time is relatively constant with 2 respect to the network depth. The speedup appears to scale with increasing widening factor k", 3 and sizable with just a warp factor of K=2. 4

[0052] Fig. 6 and Fig. 7 shows the predictive accuracy of WarpNets and their speed-up in 5 parentheses for K=2 and K=3, respectively. As illustrated in Fig. 6 and Fig. 7, a WarpNet may 6 outperform a conventional ResNet in CIFAR-10 and CIFAR-100. Fig. 6 and Fig. 7 also illustrates 7 exemplary relative speed-up of a WarpNet with respect to a conventional ResNet. The relative 8 speed-up is defined as ( 1,e,-1 wm11 )It,.,, , where 1 "" is the time taken for Res Net to process a batch 9 and 1 •,mi, is the corresponding time for WarpNet. 10

[0053] The Warp Net shown in Fig. 3 is a good approximation of the ResNet of Fig. 2 and results 11 in a significant savings in processing during training of the network. 12

[0054] Another example is shown in Fig. 9 with reference to a down-sampled (32x32) lmageNet 13 data set. The data set contains 1000 classes with 1281167 training images and 50000 validation 14 images with 50 images in each class. The training batch size is 512, initial learning rate is 0.4 and 15 drops by a factor of 0.1 at every 30 epochs. The weight decay is set to be 0.0001. We use the 16 overall best performing warp operator in the CIFAR experiments, namely, the one containing F' x. 17 Exemplary results are shown ln Flg. 9 and Fig. 10. It can be seen that for a given ResNet there is 18 exists a WarpNet that obtains a higher validation accuracy with shorter training time. An increase 19 in K from 2 to 3 with all other parameters fixed corresponds to WarpNet-109-2. The network has 20 more residual units than WRN-73-2. It is observed that WarpNet-109-2 trains 12% faster than 21 WRN-73-2 while resulting in a better validation accuracy. Second, WarpNet can achieve close to 22 a benchmark validation error of 18.9% with WRN-28-10. It is notable that in the implementation 23 used, it was not reasonable to train the corresponding WRN-73-4 on the dataset as the model 24 requires too much memory on a single GPU. This shows that the weight distribution of WarpNet 25 across GPUs allows a bigger network to be trained. Remarkably, the validation error curve for 26 WRN-73-2 and its approximation WarpNet 73-2 (K = 2; Nwarp = 6) lie almost exactly on top of 27 each other. This suggests that the utilized implementation of WarpNet is a good approximation of 28 the corresponding WRN throughout training. 29

[0055] Fig. 8 illustrates a method for training a conventional ResNet using a WarpNet. At block 30 802, a plurality of residual units 302 are arranged into one or more subsets corresponding to a 31 plurality of warped units 304, wherein each residual unit 100 comprises a subset of the plurality 32 of weight layers 102 and a shortcut connection 104 mapping an input 106 of the residual unit 100 11 CA 3020884 2018-10-15 1 to be added to an output 108 of a final layer 102b of the subset of the plurality of weight layers 2 102. The residual units 302 are decoupled from one another, such that each residual unit in a 3 subset of the plurality of residual units is independent from other residual units. 4

[0056] At block 804, a warp operator 402 is configured to perform a computation of an output of 5 a warped unit 304 from an input to the warped unit using a first-order Taylor series approximation. 6 Each independent residual unit in the subset of the plurality of residual units is activated in parallel 7 for the computation. In other words, the output of each residual unit in the warp unit is calculated 8 in parallel, not in sequence as in a conventional ResNet, which greatly speeds up the computation. 9

[0057] At block 806, a final parameter setting for the plurality of parameters associated with the 10 plurality of weight layers 102 is determined by training the residual neural network on a training 11 set, wherein training the residual network comprises applying the warp operator 402 to each of 12 the plurality of warped units 304. At block 808, the final parameter setting is stored on one or more 13 non-transitory computer storage media, such as memory 404. 14

[0058] Although the invention has been described with reference to certain specific 15 embodiments, various modifications thereof will be apparent to those skilled in the art without 16 departing from the spirit and scope of the invention as outlined in the claims appended hereto. 12 CA 3020884 2018-10-15

Claims

3,020,884 2025-04-11 13 CLAIMS 1. A system for training an artificial residual neural network having a plurality of residual units, each residual unit comprising a subset of a plurality of weight layers and a shortcut connection mapping an input of the residual unit to be added to an output of a final layer of the subset of the plurality of weight layers, and having a plurality of parameters associated with the plurality of weight layers, the system comprising: a plurality of warp units linked in series, each warp unit comprising: a subset of units being an ordered and non-overlapping subset of the plurality of residual units decoupled from each other and connected in parallel, the subset of units comprising two or more of the residual units; at least one derivative unit, each derivative unit computed based on one of the residual units in the subset of units; a warp unit input being the input of a first residual unit in the subset of units prior to decoupling; and a warp unit output being the output of a last residual unit in the subset of units prior to decoupling, the warp unit output being computed from the sum of the warp unit input and each operation upon the input by each residual unit in the subset of units and the at least one derivative unit; and one or more non-transitory computer storage media for storing the plurality of parameters computed when applying the system to a training set of data.

2. The system of claim 1, wherein the weights for each residual unit in a warp unit is determined on a separate processing unit.

3. The system of claim 1, wherein the training of the residual neural network comprising; initializing the one or more weights of each residual unit; inputting a plurality of training cases to the first warp unit in the series; using each training case to optimize the one or more weights for each residual unit in parallel in the first warp unit in the series; starting with the output of the first warp unit in the series, iteratively propagating the output of each warp unit to the input of the next respective warp unit in the series, for each respective warp unit, using each training case to optimize the one or more weights for CA 3020884 Date reçue / Received date 2025-04-11 3,020,884 2025-04-11 14 each residual unit in parallel in the respective warp unit; and storing the output of the last warp unit in the series and the weights for each residual unit in the one or more non-transitory computer storage media.

4. The system of claim 3, wherein the at least one derivative unit is determined using at least one multiplication of one of the residual units to a derivative of another one of the residual units.

5. The system of claim 3, wherein the at least one derivative unit is determined as a derivative of at least one of the residual units.

6. The system of claim 3, wherein the at least one derivative unit is determined as a derivative of at least one of the residual units multiplied by the input.

7. The system of claim 3, wherein the output is determined from the input using first-order Taylor Series Expansion of the residual units and the at least one derivative unit.

8. The system of claim 3, the warp operator further performing, with the warp units, back propagation, the back propagation comprising propagating a weight gradient of a loss function from the last warp unit in the series to the first warp unit, in each warp unit, the one or more weights of each residual unit are independently updated in parallel.

9. The system of claim 8, wherein the weight gradient for each warp unit is determined as a gradient of the output of the warp unit multiplied by a sum comprising the gradients of the residual units and gradients of the at least one derivative unit.

10. The system of claim 9, wherein the gradient of each of the at least one derivative units comprising at least one multiplication of the gradient of one of the residual units to the gradient of another one of the residual units.

11. A method for training a residual neural network executed on one or more processing units, the residual neural network comprising a plurality of warp units, each warp unit comprising a subset of a plurality of weight layers and a shortcut connection mapping an input of the residual unit to be added to an output of a final layer of the subset of the plurality of weight layers, and having a plurality of parameters associated with the plurality of weight layers, the method comprising: for each warp unit of a plurality of warp units linked in series, performing: ordering a subset of units as a non-overlapping subset of the plurality of residual units decoupled from each other and connected in parallel, the subset of units CA 3020884 Date reçue / Received date 2025-04-11 3,020,884 2025-04-11 15 comprising two or more of the residual units; determining at least one derivative unit based on one of the residual units in the subset of units; determining a warp unit input being the input of a first residual unit in the subset of units prior to decoupling; and determining a warp unit output being the output of a last residual unit in the subset of units prior to decoupling, the warp unit output computed from the sum of the warp unit input and each operation upon the input by each residual unit in the subset of units and the at least one derivative unit; storing the plurality of parameters computed when applying the system to a training set of data.

12. The method of claim 11, wherein the at least one derivative unit is determined using at least one multiplication of one of the residual units to a derivative of another one of the residual units.

13. The method of claim 11, wherein the at least one derivative unit is determined as a derivative of at least one of the residual units.

14. The method of claim 11, wherein the at least one derivative unit is determined as a derivative of at least one of the residual units multiplied by the input.

15. The method of claim 11, wherein the output is determined from the input using first-order Taylor Series Expansion of the residual units and the at least one derivative unit.

16. The method of claim 11, further comprising performing back propagation comprising propagating a weight gradient of a loss function from the last warp unit in the series to the first warp unit, in each warp unit, the one or more weights of each residual unit are independently updated in parallel.

17. The method of claim 16, wherein the weight gradient for each warp unit is determined as a gradient of the output of the warp unit multiplied by a sum comprising the gradients of the residual units and gradients of the at least one derivative unit.

18. The method of claim 17, wherein the gradient of each of the at least one derivative units comprising at least one multiplication of the gradient of one of the residual units to the gradient of another one of the residual units. CA 3020884 Date reçue / Received date 2025-04-11 3,020,884 2025-04-11 16 19. The method of claim 18, wherein the sum further comprises the identity matrix.

20. The method of claim 11, wherein at least two of the warp units comprising a differing quantity of residual units from each other. CA 3020884 Date reçue / Received date 2025-04-11