Learning the truncated rank of the singular value decomposition matrix representing the weight tensor in a neural network
By learning the initialization and updating the rank selection parameters, the weight tensor rank in the deep neural network is truncated, which solves the problems of computational complexity and high resource consumption, maintains network performance, and realizes the reduction of network size.
Patent Information
- Application Number
- CN201980027217.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-04-25
- Filing Date
- 2019-04-19
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-04-19
AI Technical Summary
When training deep neural networks, it is difficult to determine the rank N of the truncated singular value decomposition matrix, resulting in excessive computational complexity and resource consumption, while it is difficult to maintain the original performance of the network.
By learning to initialize parameter offsets that are monotonically decreasing in order of singular value sizes, apply the rank selection parameters to truncate the rank of the weight tensor and regularize these parameters in the forward pass, updating them to maintain network performance.
Effectively reduces the network size, reduces the computational complexity and resource consumption, while maintaining the original performance of deep neural networks.
Smart Images

Figure CN112005250B_ABST
Abstract
Description
[0001] Priority claim
[0002] The priority claimed by this patent application is U.S. Non-Provisional Patent Application No. 15 / 962,996, entitled “Learning Truncated Rank of Singular Value Decomposition Matrices Representing Weight Tensors in Neural Networks,” filed on April 25, 2018, which is assigned to the present assignee and is expressly incorporated herein by reference. Technical Field
[0003] Certain aspects of the present disclosure relate generally to machine learning, and more particularly to systems and methods for improving the truncated rank of a singular value decomposition matrix for learning to represent a weight tensor in a neural network. Background Art
[0004] An artificial neural network is a computing device or represents a method to be performed by a computing device, and an artificial neural network may include a group of interconnected artificial neurons (eg, a neuron model).
[0005] A convolutional neural network is a type of feed-forward artificial neural network. A convolutional neural network may include a collection of neurons, each of which has a receptive field and the neurons collectively tile the input space. Convolutional neural networks (CNNs) have many applications. In particular, CNNs have been widely used in the fields of pattern recognition and classification.
[0006] Deep learning architectures, such as deep belief networks and deep convolutional networks, are layered neural network architectures in which the outputs of the first layer of neurons become the inputs of the second layer of neurons, the outputs of the second layer of neurons become the inputs of the third layer of neurons, and so on. Deep neural networks can be trained to recognize hierarchies of features, and therefore deep neural networks have been increasingly used in object recognition applications. Similar to convolutional neural networks, computations in these deep learning architectures can be distributed across a population of processing nodes that can be configured in one or more computational chains. Such multi-layer architectures can be trained one layer at a time and can be fine-tuned using backpropagation.
[0007] Other models can also be used for object recognition. For example, support vector machines (SVM) are learning tools that can be applied to classification. Support vector machines include a separating hyperplane (e.g., a decision boundary) for categorical data. The hyperplane is defined by supervised learning. The desired hyperplane increases the boundaries of the training data. In other words, the hyperplane should have the largest minimum distance to the training examples.
[0008] Deep neural networks can be trained with more parameters than required to solve a specific task to provide convergence and reduce overfitting. This training process can be used to explore large error surfaces associated with many tasks. However, at the end of training, the network may be over-parameterized. The network size can be reduced by applying compression techniques. One such technique is to perform dimensionality reduction by decomposing the weight tensors in each layer using singular value decomposition (SVD). Each weight tensor is a 2-dimensional or 4-dimensional tensor, and by applying SVD, each tensor can be divided into two 2-dimensional tensors. If full-rank SVD is used, this will increase the computational cost of the layer. However, if the rank is truncated to retain only the top N most significant ranks (the highest N singular values), the computational cost of the layer can be reduced compared to the initial combined tensor. Unfortunately, it is difficult to determine the Nth rank at which the decomposed matrix is truncated and the original performance of the deep neural network is still maintained. Therefore, it is necessary to find a technique for rank N. Summary of the invention
[0009] A simplified summary of one or more aspects is presented below to provide a basic understanding of these aspects. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify key or important elements of all aspects, nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description presented later.
[0010] It will be appreciated by those skilled in the art that the present disclosure can be easily used as a basis for modifying or designing other structures for the same purpose of performing the present disclosure. It will also be appreciated by those skilled in the art that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. When considered in conjunction with the accompanying drawings, features that are considered to be characteristics of the present disclosure, both the organization and the method of operation of the present disclosure, will be better understood from the following description. However, it should be clearly understood that each of the drawings provided in the accompanying drawings is for the purpose of illustration and description only and is not intended to be a definition of limitations to the present disclosure.
[0011] Artificial neural networks such as deep convolutional networks can be used for image classification. However, such networks can be computationally complex and consume a lot of computing resources.
[0012] In order to solve the problems of computational complexity and computing resource consumption, the truncated rank of the singular value decomposition matrix representing the weight tensor in the network can be learned to reduce the size of the network.
[0013] In one aspect of the present disclosure, a method, a computer-readable medium, and an apparatus for learning the rank of an artificial neural network are presented. The apparatus includes a memory and at least one processor coupled to the memory. The (one or more) processors are configured to decompose a weight tensor into a first weight tensor and a second weight tensor. The (one or more) processors are also configured to apply a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the ranks of the first weight tensor and the second weight tensor. In addition, the (one or more) processors are configured to update the set of rank selection parameters by averaging the updates calculated for each rank selection parameter in the set of rank selection parameters.
[0014] Additional features and potential advantages of aspects of the present disclosure will be described below. It will be appreciated by those skilled in the art that the present disclosure can be easily used as a basis for modifying or designing other structures for the same purpose of performing the present disclosure. It will also be appreciated by those skilled in the art that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. These features will be better understood from the following description when considered in conjunction with the accompanying drawings. However, it should be expressly understood that each of the figures in the accompanying drawings is provided for the purpose of illustration and description only and is not intended to be a definition of limitations to the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The features, nature and advantages of the present disclosure will become more apparent from the detailed description set forth below when considered in conjunction with the accompanying drawings, in which like reference numerals are identified correspondingly throughout.
[0016] Figure 1 An example implementation of designing a neural network using a system on a chip (SOC) including a general-purpose processor in accordance with certain aspects of the present disclosure is shown.
[0017] Figure 2 An example implementation of a system according to some aspects of the present disclosure is shown.
[0018] Figure 3A is a diagram illustrating a neural network according to some aspects of the present disclosure.
[0019] Figure 3B is a block diagram illustrating an exemplary deep convolutional network (DCN) according to some aspects of the present disclosure.
[0020] Figure 4A -C is a block diagram illustrating rank learning of an exemplary artificial neural network according to some aspects of the present disclosure.
[0021] Figure 5 is a block diagram illustrating updating of an exemplary artificial neural network according to some aspects of the present disclosure.
[0022] Figure 6is a graph showing a rank selection parameter as a function of rank / number of filters according to some aspects of the present disclosure.
[0023] Figure 7 A method for learning the rank of an artificial neural network according to some aspects of the present disclosure is shown. DETAILED DESCRIPTION
[0024] The specific embodiments set forth below in conjunction with the accompanying drawings are intended as descriptions of various configurations and are not intended to represent the only configurations in which the concepts described herein may be practiced. The specific embodiments include specific details for providing a thorough understanding of the various concepts. However, it is apparent to those skilled in the art that these concepts may be practiced without these specific details. In some examples, well-known structures and components are shown in block diagram form in order to avoid confusion with such concepts.
[0025] Based on these teachings, it will be appreciated by those skilled in the art that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether implemented independently of any other aspect of the present disclosure or implemented in combination with any other aspect of the present disclosure. For example, an apparatus or a method of practice may be implemented using any number of aspects described. In addition, the scope of the present disclosure is intended to cover such apparatus or methods that are practiced using other structures, functions, or structures and functions other than or different from some aspects of the present disclosure described. It should be understood that any aspect of the present disclosure disclosed may be embodied by one or more elements of a claim.
[0026] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.
[0027] Although specific aspects are described herein, many variations and permutations of specific aspects fall within the scope of the present disclosure. Although some benefits and advantages of aspects of the present disclosure are mentioned, the scope of the present disclosure is not intended to be limited to specific benefits, uses or purposes. On the contrary, some aspects of the present disclosure are intended to be widely applicable to different technologies, system configurations, networks and protocols, some of which are shown by way of example in the accompanying drawings and the following description of these aspects. The specific embodiments and drawings are merely illustrative of the present disclosure and are not limiting, and the scope of the present disclosure is defined by the appended claims and their equivalents.
[0028] Learning the truncated rank of the SVD matrix representing the weight tensor in a neural network
[0029] Some aspects of the present disclosure relate to learning a truncated rank of a singular value decomposition matrix. The rank of a matrix is the number of linearly independent rows or columns of the matrix (e.g., the number of non-zero rows or columns). The truncated rank is a rank approximation of the rank of the matrix. The truncated rank can be learned by learning an offset of a set of bounded parameters that are initialized to monotonically decrease in the order of the size of the singular values. In other words, the rank of the weight tensor can be truncated by learning to set a set of parameters β (which can be referred to as rank selection parameters) to 0 for successively higher singular values.
[0030] According to some aspects of the present disclosure, a weight tensor of a neural network may be decomposed into a first weight tensor and a second weight tensor via singular value decomposition (SVD). A filter with a rank selection parameter β may be applied to the first weight tensor and the second weight tensor.
[0031] The rank selection parameter may include a real value and may be initialized to a random value. In some aspects, the rank selection parameter may be initialized to a linear function of a rank number between 0.5 and 1. The rank selection parameter may be applied to the weight tensor to generate an output given as follows:
[0032] y p =β i x p , (1)
[0033] where x p is the filter input (weights of the weight tensor), and y p is the filter output.
[0034] In some aspects, the rank selection parameters can be regularized in the forward pass of the network (e.g., when computing inference). That is, a threshold can be applied to each rank selection parameter β i Each rank selection parameter β can be i The actual value of is compared with a threshold. For example, the rank selection parameter can be regularized:
[0035]
[0036] in is the actual value of the rank selection parameter, β i is the applied value of the rank selection parameter.
[0037] If the actual value of the rank selection parameter If it is less than a threshold value (e.g., 0.5), then the application value β of the rank selection parameter can be i Setting it to 0 removes the filter. On the other hand, if the actual value of the rank selection parameter is greater than or equal to a threshold (e.g., 0.5), then the application value β of the rank selection parameter can be i Set to 1 to leave the filter unchanged.
[0038] During a backward pass through the network (e.g., via backpropagation fine-tuning), each rank selection parameter may be updated. For example, each rank selection parameter may be updated by averaging the updates calculated for each parameter in the set. Thus, the same update may be applied to all rank selection parameters in the set of parameters. By applying the same update (e.g., the average of the updates to all rank selection parameters), the rank selection parameters may be configured to move consistently relative to a static threshold.
[0039] In addition, by updating all rank selection parameters using the same update (e.g., offset), correlations can be enforced between the rank selection parameters so that the first N rank selection parameters can be 1 and the remaining MN rank selection parameters can be 0. Therefore, the first N singular value dimensions can be retained. Because the method can use standard back propagation to update the rank selection parameters, learning the rank selection parameters can be combined with fine-tuning or updating the weights to achieve a smaller model. That is, in some aspects, the truncated rank of the weight tensor of the singular value decomposition can be learned while fine-tuning the weight values to reduce the model size. For example, in rank selection, the rank selection can be updated in the backward pass of the network according to the following:
[0040]
[0041]
[0042] Figure 1 An example embodiment of a method for learning the rank of an artificial neural network using a system on a chip (SOC) 100 according to certain aspects of the present disclosure is shown, which system on a chip may include a general purpose processor (CPU) and / or a multi-core general purpose processor (CPU) 102. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a dedicated memory block 118, or may be distributed across multiple blocks. Instructions executed at the general purpose processor 102 may be loaded from a program memory associated with the CPU 102, or may be loaded from a dedicated memory block 118.
[0043] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connection block 110, which may include a fourth generation long term evolution (4G LTE) connection, an unlicensed Wi-Fi connection, a USB connection, a Bluetooth connection, etc., and a multimedia processor 112 that may, for example, detect and recognize gestures. In one embodiment, the NPU is implemented in the CPU, DSP, and / or GPU. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or navigation 120, which may include a global positioning system.
[0044] SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into the general purpose processor 102 may include code for decomposing the weight tensor into a first weight tensor and a second weight tensor. The instructions loaded into the general purpose processor 102 may also include code for applying a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the ranks of the first weight tensor and the second weight tensor. The instructions loaded into the general purpose processor 102 may also include code for updating the set of rank selection parameters by averaging the updates calculated for each rank selection parameter in the set of rank selection parameters.
[0045] Figure 2 2 shows an example implementation of a system 200 according to certain aspects of the present disclosure. Figure 2 As shown, the system 200 may have a plurality of local processing units 202, which may perform various operations of the methods described herein. Each local processing unit 202 may include a local state memory 204 and a local parameter memory 206 that may store parameters of a neural network. In addition, the local processing unit 202 may have a local (neuron) model program (LMP) memory 208 for storing a local model program, a local learning program (LLP) memory 210 for storing a local learning program, and a local connection memory 212. In addition, as shown in FIG. Figure 2 As shown, each local processing unit 202 may interface with a configuration processor unit 214 for providing configuration for a local memory of the local processing unit, and with a routing connection processor unit 216 for providing routing between local processing units 202 .
[0046] Deep learning architectures can perform object recognition tasks by learning to represent the input at successively higher levels of abstraction in each layer, thereby building useful feature representations of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Before the advent of deep learning, machine learning approaches to object recognition problems may have relied heavily on human-engineered features, perhaps in combination with shallow classifiers. A shallow classifier can be a two-class linear classifier, for example, where a weighted sum of the components of a feature vector can be compared to a threshold to predict which class the input belongs to. Human-engineered features can be templates or kernels that are customized for a specific problem domain by engineers with domain expertise. In contrast, deep learning architectures can learn to represent features similar to human-engineered features learned through training. In addition, deep networks can learn to represent and recognize new types of features that humans may not have considered.
[0047] Deep learning architectures can learn hierarchies of features. For example, if visual data is presented, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if auditory data is presented, the first layer can learn to recognize spectral power in specific frequencies. A second layer, which takes the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Higher layers can learn to recognize common visual objects or spoken phrases.
[0048] Deep learning architectures can perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motor vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to recognize cars, trucks, and airplanes.
[0049] Neural networks can be designed to have various connection patterns. In a feedforward network, information is passed from a lower layer to a higher layer, and each neuron in a given layer communicates with a neuron in a higher layer. As described above, hierarchical representations can be established in successive layers of a feedforward network. Neural networks can also have loops or feedback (also called top-down) connections. In a loop connection, the output from a neuron in a given layer can be transmitted to another neuron in the same layer. The loop architecture can help identify some patterns that span more than one input data block in the input data blocks delivered sequentially to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of high-level concepts can help distinguish specific low-level features of the input, a network with many feedback connections may be helpful.
[0050] refer to Figure 3A, the connections between the layers of the neural network can be fully connected 302 or locally connected 304. In the fully connected network 302, a neuron in the first layer can transmit the output of the neuron to each neuron in the second layer so that each neuron in the second layer receives input from each neuron in the first layer. Alternatively, in the locally connected network 304, the neurons in the first layer can be connected to a limited number of neurons in the second layer. The convolutional network 306 can be locally connected and can be further configured so that the connection strengths associated with the inputs of each neuron in the second layer are shared (e.g., 308). More generally, the locally connected layers of the network can be configured so that each neuron in the layer will have the same or similar connection pattern, but with connection strengths that can have different values (e.g., 310, 312, 314, and 316). The connection patterns of the local connections can give rise to spatially different receptive fields in higher layers because higher layer neurons in a given area can receive inputs that are tuned to the nature of a limited portion of the total input to the network through training.
[0051] Locally connected neural networks can be well suited for problems where the spatial location of the input is meaningful. For example, a network 300 designed to recognize visual features from an onboard camera can develop high-level neurons with different properties based on their association with the upper portion of the image and the lower portion. Neurons associated with the lower portion of the image can learn to recognize, for example, lane markings, while neurons associated with the upper portion of the image can learn to recognize traffic lights, traffic signs, etc.
[0052] The DCN may be trained using supervised learning. During training, the DCN may be presented with an image, such as a cropped image of a speed limit sign 326, and then a "forward pass" may be calculated to produce output 322. Output 322 may be a vector of values corresponding to features such as "sign", "60", and "100". The network designer may want the DCN to output high scores for some neurons in the output feature vector, for example, neurons corresponding to "sign" and "60" as shown in output 322 of the already trained network 300. Prior to training, the output produced by the DCN is likely to be incorrect, so the error between the actual output and the target output may be calculated. The weights of the DCN may then be adjusted so that the DCN's output scores are more closely aligned with the target.
[0053] To adjust the weights, the learning algorithm may calculate a gradient vector of the weights. The gradient may indicate the amount by which the error would increase or decrease if the weights were adjusted slightly. At the top layer, the gradient may correspond directly to the value of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradients of the higher layers. The weights may then be adjusted in order to reduce the error. This manner of adjusting the weights may be referred to as "backward propagation" because a "backward pass" through the neural network is involved during weight adjustment.
[0054] In practice, the error gradient of the weights can be calculated on a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method may be called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system has stopped decreasing or until the error rate has reached a target level.
[0055] After learning, the DCN may be presented with new images 326, and a forward pass through the network may produce an output 322, which may be considered an inference or prediction of the DCN.
[0056] Deep belief network (DBN) is a probabilistic model including multiple layers of hidden nodes. DBN can be used to extract hierarchical representations of training data sets. DBN can be obtained by stacking layers of restricted Boltzmann machines (RBM). RBM is a type of artificial neural network that can learn probability distributions on a set of inputs. Because RBM can learn probability distributions in the absence of information about the categories to which each input should be classified, RBM is often used for unsupervised learning. Using a mixed unsupervised and supervised paradigm, the bottom RBM of DBN can be trained in an unsupervised manner and can act as a feature extractor, and the top RBM can be trained in a supervised manner (on the joint distribution of inputs from the previous layer and the target category) and can act as a classifier.
[0057] A deep convolutional network (DCN) is a network that is a convolutional network with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where the input and output targets are known for many examples and are used to modify the weights of the network by using a gradient descent method.
[0058] The DCN can be a feed-forward network. In addition, as described above, the connections from neurons in the first layer of the DCN to a set of neurons in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of the DCN can be used for fast processing. The computational burden of the DCN can be much smaller than that of a similarly sized neural network including loops or feedback connections, for example.
[0059] The processing of each layer of the convolutional network can be thought of as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, the convolutional network trained on the input can be thought of as three-dimensional, with two spatial dimensions along the axes of the image [e.g., the x-axis and the y-axis], and the third dimension captures color information [e.g., the z-axis]. The output of the convolutional connection can be thought of as forming a feature map in subsequent layers 318 and 320, where each element of the feature map (e.g., 320) receives input from a series of neurons in the previous layer (e.g., 318) and from each of the multiple channels. The values in the feature map can be further processed using a nonlinearity such as a correction max(0, x). Values from neighboring neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction. Normalization corresponding to whitening can also be applied by lateral inhibition between neurons in the feature map.
[0060] The performance of deep learning architectures can increase as more labeled data points become available or as computing power increases. Modern deep neural networks are routinely trained with computing resources thousands of times greater than those available to a typical researcher just fifteen years ago. New architectures and training paradigms can further improve the performance of deep learning. Rectified linear units can reduce the training problem known as vanishing gradients. New training techniques can reduce overfitting and thus enable larger models to achieve better generalization. Encapsulation techniques can abstract data within a given receptive field and further improve overall performance.
[0061] Figure 3B is a block diagram illustrating an exemplary deep convolutional network 350. The deep convolutional network 350 may include multiple different types of layers based on connectivity and weight sharing. Figure 3B As shown, the exemplary deep convolutional network 350 includes multiple convolutional blocks (e.g., C1 and C2). Each convolutional block can be configured with a convolutional layer, a normalization layer (LNorm), and a pooling layer. The convolutional layer may include one or more convolutional filters, which may be applied to input data to generate a feature map. Although only two convolutional blocks are shown, the present disclosure is not so limited, but on the contrary, any number of convolutional blocks may be included in the deep convolutional network 350 according to design preferences. The normalization layer can be used to normalize the output of the convolutional filter. For example, the normalization layer can provide whitening or lateral suppression. The pooling layer can provide spatial downsampling aggregation for local invariance and dimensionality reduction.
[0062] For example, the parallel filter bank of the deep convolutional network can be optionally loaded on the CPU 102 or GPU 104 of the SOC 100 based on the ARM instruction set to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank can be loaded on the DSP 106 or ISP 116 of the SOC 100. In addition, the DCN can access other processing blocks that may exist on the SOC, such as processing blocks dedicated to sensors 114 and navigation 120.
[0063] The deep convolutional network 350 may also include one or more fully connected layers (e.g., FC1 and FC2). The deep convolutional network 350 may also include a logistic regression (LR) layer. Between each layer of the deep convolutional network 350 are weights (not shown) to be updated. The output of each layer may be used as an input to subsequent layers in the deep convolutional network 350 to learn a hierarchical feature representation from the input data (e.g., image, audio, video, sensor data, and / or other input data) provided at the first convolutional block C1.
[0064] Figure 4A -C is a block diagram illustrating the rank of an exemplary artificial neural network for learning according to some aspects of the present disclosure. Figure 4A , a layer of an artificial neural network 400. The artificial neural network includes a layer 406 having a set of nodes. In the artificial neural network 400, a set of inputs 402 are received at a layer of the artificial neural network 400. The set of inputs 402 are convolved with a set of weights of a weight tensor 404. A bias term b1 may be added to the result of the convolution, and a sum may be output and provided as an input to the next layer 406 of the artificial neural network 400.
[0065] Singular value decomposition (SVD) may be applied to weights 404. Applying SVD, weight tensor 404 may be represented by two matrices (eg, 424, 426), as represented by Given, where U is an M×M matrix of input filters and V is an M×N matrix of output filters, each matrix having rank=M. In this way, the weight tensor 404 of the artificial neural network 400 can be decomposed into a first weight tensor U1 424 and a second weight tensor V1 426. A filter with a rank selection parameter β1 can be inserted for each output 428. The rank selection parameter β1 can be initialized with a specified value (e.g., 0.5). Therefore, the input filter U1 can be convolved with the input 422 to produce a filtered input. The rank selection parameter can be applied to the filtered input to produce a set of activations of layer_a 428. The activations of layer_a can be arranged according to singular values, where the activations of the largest singular values (e.g., 430) are sorted before the activations of the lowest singular values (e.g., 432). Then, the output filter V1 426 can be applied to the activations of layer_a 428 to produce a filtered output. A bias term b1 may be added to the filter outputs, the sum of which may be output and provided to the next layer (layer_b 434).
[0066] According to some aspects of the present disclosure, a learning technique such as back propagation can be used to, for example, transform the rank selection parameter β i is set to zero for successively larger singular values. In doing so, the rank selection parameter β is applied i This may result in some lower singular activations (e.g., 432) of layer_a 428 having zero values. Using SVD, the activation function may be linear between layers, and the rank selection parameter may be monotonically decreasing. Therefore, fewer calculations may be performed when applying output filter V1 426 to calculate the output (e.g., layer_b 434).
[0067] refer to Figure 4C , the rank selection parameter has been applied to the artificial neural network 440. A set of lowest singular value activations of layer_a 448 (see, for example, Figure 4B 432) may have a zero value and may be removed. Thus, input filter U1 444 may be applied to input 442. Rank selection parameter β i can be applied to filter the input. However, because for the lowest singular value, the rank selection parameter β i Can be zero, so a set of lowest singular value activations can in turn be zero and can be removed. Thus, the rank of the activations can be reduced, thereby reducing the computations used to produce the output of layer_b 450 (e.g., fewer multiplication and addition operations can be performed in the convolution of the activations and output filter V1 446).
[0068] Figure 5 is a block diagram illustrating an update of an exemplary artificial neural network according to some aspects of the present disclosure. Figure 5, the exemplary artificial neural network 500 at stage A may include a data input layer, convolutional layers (conv1, conv2), fully connected layers (fc3 and fc4), and a loss layer (e.g., a softmax layer). In the exemplary artificial neural network 500, the convolutional layer conv1 may have 128 parameters, the convolutional layer conv2 may have 256 parameters, and the fully connected layer fc3 may have 4096 parameters, all of which may be used to identify a category among the 100 categories in the fully connected layer fc4.
[0069] According to some aspects of the present disclosure, SVD can be applied to the artificial neural network 500. As shown in stage B, optional filters 522a, 522b, and 522c can be inserted for the outputs of the convolutional layer conv1, the convolutional layer conv2, and the fully connected layer fc3, respectively. In some aspects, the optional filters (e.g., 522a, 522b, and 522c) can each include, for example, a three-state rectifier linear unit. Each optional filter (e.g., 522a, 522b, and 522c) can have a rank selection parameter β i . The rank selection parameter β i It can be initialized with a specified value (e.g., 1). The rank selection parameter β i β may have a real value and may be initialized to a randomly determined value (e.g., using a pseudo-random number generator). In some aspects, the rank selection parameter β may be initialized to a linear function of the rank number between 0.5 and 1. i The weight tensor that can be applied to a layer (e.g., conv1) to generate the output as indicated above in Equation 1.
[0070] In some aspects, the rank selection parameter β for each optional filter can be regularized in the forward pass of the network (e.g., when computing inference or classification) i That is, a threshold can be applied to each rank selection parameter β i Each rank selection parameter β i The actual value of can be compared with a threshold value (e.g., 0.5). In one example, the rank selection parameter β i When it is less than 0.5, the optional filter can be applied with zero, so that the output to which the optional filter is applied can be removed, and the rank selection parameter β i When α is greater than or equal to 0.5, the optional filter can be applied so that the output to which the optional filter is applied can be passed to the subsequent layer. In turn, the inference can be determined and output via the fully connected layer fc4. The output can be provided to the loss layer to calculate the error.
[0071] During a backward pass through the network (e.g., via back-propagation fine-tuning), the rank selection parameters β of the optional filters 522a, 522b, and 522c may be updated. i For example, we can select a parameter β for each rank in i The gradient is determined for each rank selection parameter in . The average of the gradients can be calculated and used to determine the update for all rank selection parameters. That is, the same update can be applied to all rank selection parameters β of the optional filters 522a, 522b, and 522c i By applying the same update (eg, the average of the updates to all rank selection parameters), the rank selection parameters may be configured to move consistently relative to a static threshold.
[0072] In addition, by updating all rank selection parameters using the same update (e.g., offset), the rank selection parameter β i The correlation is enforced between them so that the first N rank selection parameters can be 1 and the remaining MN rank selection parameters can be 0. Therefore, the first N singular value dimensions can be preserved. Because standard backpropagation can be used to update the rank selection parameters β i , so the learning rank selection parameter β i This can be combined with fine-tuning or updating the weights of each layer of the artificial network 520 to achieve a smaller model. That is, in some aspects, the truncated rank of the weight tensor of the singular value decomposition can be learned while fine-tuning the weight values to reduce the model size.
[0073] In some aspects, the rank selection parameter β is updated or fine-tuned based on the performance metric of the artificial neural network 520. i For example, the performance metric may include processing speed, memory size, memory bandwidth, or the number of operations (e.g., multiply-accumulate operations (MACs)). Thus, the artificial neural network (e.g., 520) may be configured differently to run on different computing systems with different types of processors (e.g., CPU, GPU, DSP, NPU, etc.). In one example, the rank selection parameter β may be updated using i The cost function of the network (and thus whether to turn on or off the corresponding filter) is used to determine this configuration of the network, which cost function can be defined as follows:
[0074]
[0075]
[0076] Among them C l =max{t compute t memory}, λ3 is the width limit regularization factor, λ oare weight and bias regularization factors, w is a weight term applied to an optional filter (e.g., 522a), is the calculated activation, is the computed output, and y is the determined output (e.g., inferred or classified).
[0077] In stage C, after fine-tuning the artificial neural network 520, the optional filters 522a, 522b, and 522c may be removed to generate the artificial neural network 540. The artificial neural network 540 may have a reduced model size than the artificial neural network 500. For example, the convolutional layer conv1 of the artificial neural network 540 may be operated with 67 parameters, compared to 128 parameters used in the convolutional layer conv1 of the artificial neural network 500. Similarly, the conv2 of the artificial neural network 540 may be operated with 131 parameters, compared to 256 parameters used in the conv2 of the artificial neural network 500. In addition, the fully connected layer fc3 of the artificial neural network 540 may be operated with 25 parameters, compared to 4096 parameters used in the fully connected layer fc3 of the artificial neural network 500. Therefore, the artificial neural network 540 may be operated to calculate inferences using reduced computations, thereby reducing the time to calculate inferences (e.g., classification).
[0078] Figure 6 600 is a graph showing the rank selection parameter related to the rank / number of filters. i can vary linearly with respect to the rank number. In some aspects, the rank selection parameter β of each layer can be i Initialized as a linear function of the rank value between 0.5 and 1. As the rank or number of filters increases, the value of the rank selection parameter can decrease (e.g., monotonically). Each point on the line (e.g., 602) can represent the rank selection parameter of the optional filter. When updating the rank selection parameter β i , the derivatives can be calculated for each point on the line (e.g., 602). The derivatives can be averaged, and the average can be applied to each point. In this way, the rank selection parameter can be consistently moved up or down, as shown by arrow 604. Thus, when applied to a weight tensor of a layer to which SVD has been applied (e.g., Figure 4B 442), the lowest singular value (e.g., Figure 4B element 432).
[0079] Figure 7A method 700 for learning the rank of an artificial neural network is shown. In block 702, the process decomposes a weight tensor into a first weight tensor and a second weight tensor. In some aspects, the weight tensor can be decomposed by applying a singular value decomposition to the weight tensor. In block 704, a set of rank selection parameters can be applied to the first weight tensor and the second weight tensor to truncate the ranks of the first weight tensor and the second weight tensor. In some aspects, for example, the rank selection parameters can be initialized to random values.
[0080] Furthermore, in block 706, the set of rank selection parameters may be updated by averaging the updates calculated for each of the set of rank selection parameters. In some aspects, the actual value of the rank selection parameter may be updated by using Back propagation is performed to update the set of rank selection parameters. The actual value of each rank selection parameter may include a real number. A gradient may be determined for each of the rank selection parameters. An average value of the gradients of all rank selection parameters may be determined. The average value may be applied to each rank selection parameter.
[0081] In block 708, the rank selection parameter may optionally be updated simultaneously (e.g., in the same time period) or in parallel with updating the first weight tensor and the second weight tensor. Figure 5 As discussed, standard backpropagation can be used to update the rank selection parameter β i , so the learning rank selection parameter β i This can be combined with fine-tuning or updating the weights of each layer of the artificial network 520 to obtain a smaller model.
[0082] In block 710, during the calculation of the inference, a threshold may optionally be applied to each rank selection parameter in the set of rank selection parameters. In one exemplary aspect, the threshold may be applied to the rank selection parameter such that if the actual value of the rank selection parameter is greater than or equal to a threshold value (e.g., 0.5), the applied value of the rank selection parameter is set to 1. On the other hand, if the actual value of the rank selection parameter If is less than the threshold, the application value of the rank selection parameter is set to 0.
[0083] Additionally, in block 712, an inference may optionally be calculated based at least in part on the thresholded rank selection parameter. Figure 4B -C, after applying the rank selection parameter β i Afterwards, the activation with the lowest singular value is removed, and the activation with the highest singular value is used to compute the output, which can be used as the input to layer_B. Such operations can be performed in subsequent layers to compute the output layer (e.g., Figure 5 The inference in fc4).
[0084] In one configuration, the machine learning model is configured to decompose the weight tensor into a first weight tensor and a second weight tensor. The model is also configured to apply a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the ranks of the first weight tensor and the second weight tensor. The model is also configured to update the set of rank selection parameters by averaging the updates calculated for each rank selection parameter in the set of rank selection parameters. The model includes a decomposition device, an application device, and / or an update device. In one aspect, the decomposition device, the application device, and / or the update device can be a general-purpose processor 102 configured to perform the function, a program memory associated with the general-purpose processor 102, a memory block 118, a local processing unit 202, and / or a routing connection processing unit 216. In another configuration, the aforementioned device can be any module or any device configured to perform the function described by the aforementioned device.
[0085] According to certain aspects of the present disclosure, each local processing unit 202 may be configured to determine parameters of a model based on one or more desired functional characteristics of the model, and develop one or more functional characteristics toward the desired functional characteristics as the determined parameters are further adapted, tuned, and updated.
[0086] In some aspects, method 700 may be performed by SOC 100 ( Figure 1 ) or System 200( Figure 2 That is, each of the elements of method 700 may be performed, for example (but not limited to), by SOC 100 or system 200 or one or more processors (e.g., CPU 102 and local processing unit 202), and / or other components included therein.
[0087] The various operations of the above method can be performed by any suitable device capable of performing the corresponding functions. The device may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Generally, where there are operations shown in the figure, those operations may have corresponding corresponding device-plus-function components with similar numbers.
[0088] As used herein, the term "determining" includes a variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, etc. In addition, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Furthermore, "determining" may include resolving, selecting, choosing, establishing, etc.
[0089] As used herein, a phrase referring to "at least one" of a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.
[0090] The various illustrative logical blocks, modules, and circuits described in conjunction with the present disclosure may be implemented or executed using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration.
[0091] The frame of the method or algorithm described in conjunction with the present disclosure can be directly embodied in hardware, in a software component executed by a processor, or in a combination of the two. The software component can reside in any form of storage medium known in the art. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, etc. The software component can include a single instruction or many instructions, and can be distributed in several different code segments, in different programs, and across multiple storage media. The storage medium can be coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium can be integrated with the processor.
[0092] The methods disclosed herein include one or more operations or actions for implementing the described methods. Method operations and / or actions may be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of operations or actions is specified, the order and / or use of specific operations and / or actions may be modified without departing from the scope of the claims.
[0093] The functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system in a device. The processing system may be implemented with a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnecting buses and bridges. The bus may link together various circuits including a processor, a machine-readable medium, and a bus interface. The bus interface may be used to connect a network adapter, etc., to the processing system via the bus. The network adapter may be used to implement signal processing functions. For some aspects, a user interface (e.g., a keypad, a display, a mouse, a joystick, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and therefore will not be further described.
[0094] The processor may be responsible for managing the bus and routine processing, including executing software stored on a machine-readable medium. The processor may be implemented using one or more general-purpose processors and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuits capable of executing software. Software should be broadly interpreted as meaning instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or other. As an example, a machine-readable medium may include a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a disk, an optical disk, a hard disk drive, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be embodied in a computer program product. The computer program product may include packaging materials.
[0095] In hardware embodiments, the machine-readable medium can be a part of a processing system separated from the processor. However, as will be readily appreciated by those skilled in the art, the machine-readable medium or any part thereof can be outside the processing system. As an example, the machine-readable medium can include a transmission line, a carrier wave modulated by data, and / or a computer product separated from the device, all of which can be accessed by the processor through a bus interface. Alternatively or additionally, the machine-readable medium or any part thereof can be integrated into the processor, for example, with a cache and / or a general register stack. Although the various components discussed can be described as having a specific location, such as a local component, these components can also be configured in various ways, such as some components are configured as a part of a distributed computing system.
[0096] The processing system can be configured as a general processing system having one or more microprocessors providing processor functions and an external memory providing at least a portion of a machine-readable medium, all of which are linked together with other support circuits through an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the neuron model and the neural system model described herein. As another alternative, the processing system may be implemented using an application specific integrated circuit (ASIC) (wherein at least a portion of a processor, a bus interface, a user interface, a support circuit, and a machine-readable medium integrated into a single chip is integrated into a single chip) or using one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, or any other suitable circuits or circuits that can perform the various functions described throughout this disclosure. Those skilled in the art will recognize how to best implement the functions of the processing system according to the specific application and the overall design constraints imposed on the entire system.
[0097] The machine-readable medium may include multiple software components. The software components include instructions that, when executed by a processor, cause a processing system to perform various functions. The software components may include a transmission component and a reception component. Each software component may reside in a single storage device or be distributed across multiple storage devices. As an example, when a triggering event occurs, a software component may be loaded from a hard drive into a RAM. During the execution of the software component, the processor may load some instructions into a cache to increase access speed. One or more cache lines may then be loaded into a general register stack for execution by the processor. When the functions of a software component are mentioned below, it will be understood that such functions are implemented by the processor when executing instructions from the software component. In addition, it should be understood that various aspects of the present disclosure result in improvements to the functions of processors, computers, machines, or other systems that implement these aspects.
[0098] If implemented with software, the function can be stored on a computer-readable medium as one or more instructions or codes or transmitted via a computer-readable medium. Computer-readable media includes both computer storage media and communication media, and the communication media includes any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or can be used to carry or store the desired program code in the form of an instruction or data structure and any other medium that can be accessed by a computer. In addition, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, optical fiber cable, twisted pair, digital subscriber line (DSL) or wireless technologies such as infrared (IR), radio and microwave are used to transmit software from a website, server or other remote source, then coaxial cable, optical fiber cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically with lasers. Thus, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Additionally, for other aspects, computer-readable media may include transitory computer-readable media (e.g., signals). Combinations of the above should also be included within the scope of computer-readable media.
[0099] Thus, some aspects may include a computer program product for performing the operations presented herein. For example, such a computer program product may include a computer-readable medium having stored thereon (and / or encoded thereon) instructions, which may be executed by one or more processors to perform the operations described herein. For some aspects, the computer program product may include packaging materials.
[0100] In addition, it should be understood that the components and / or other appropriate devices for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when applicable. For example, such a device can be coupled to a server to facilitate the transmission of the device for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage device (e.g., RAM, ROM, a physical storage medium such as a compact disc (CD) or a floppy disk, etc.), so that the user terminal and / or base station can obtain the various methods immediately after the storage device is coupled or provided to the device. In addition, any other suitable technology for providing the methods and techniques described herein to the device can be utilized.
[0101] It is to be understood, however, that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.
Claims
1. A method for learning the rank of an artificial neural network for image classification, comprising: The image is fed to an artificial neural network to produce a classification output; decomposing a weight tensor of the artificial neural network into a first weight tensor and a second weight tensor, wherein the decomposing comprises applying a singular value decomposition to the weight tensor; Applying a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the ranks of the first weight tensor and the second weight tensor; updating the set of rank selection parameters by averaging the updates calculated for each of the set of rank selection parameters in a backward pass of the artificial neural network; as well as During the calculation of inference in the forward pass of the artificial neural network, a threshold is applied to each rank selection parameter in the set of rank selection parameters, wherein the threshold is applied to the rank selection parameter so that if the actual value of the rank selection parameter is greater than or equal to the threshold, the applied value of the rank selection parameter is set to 1, or if the actual value of the rank selection parameter is less than the threshold, the applied value of the rank selection parameter is set to 0.
2. The method of claim 1, wherein each rank selection parameter in the set of rank selection parameters is initialized to a random value.
3. The method of claim 1, wherein the updating comprises performing back-propagation to determine an error for each rank selection parameter in the set of rank selection parameters based on an actual value of the rank selection parameter, the actual value comprising a real number.
4. The method of claim 1, wherein the rank selection parameter is updated simultaneously with the first weight tensor and the second weight tensor.
5. A device for image classification for learning the rank of an artificial neural network, comprising: Memory; as well as at least one processor coupled to the memory, the at least one processor being configured to: The image is fed to an artificial neural network to produce a classification output; decomposing a weight tensor of the artificial neural network into a first weight tensor and a second weight tensor, wherein the decomposing comprises applying a singular value decomposition to the weight tensor; Applying a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the ranks of the first weight tensor and the second weight tensor; updating the set of rank selection parameters by averaging the updates calculated for each of the set of rank selection parameters in a backward pass of the artificial neural network; as well as During the calculation of inference in the forward pass of the artificial neural network, a threshold is applied to each rank selection parameter in the set of rank selection parameters, wherein the threshold is applied to the rank selection parameter so that if the actual value of the rank selection parameter is greater than or equal to the threshold, the applied value of the rank selection parameter is set to 1, or if the actual value of the rank selection parameter is less than the threshold, the applied value of the rank selection parameter is set to 0.
6. The apparatus of claim 5, wherein each rank selection parameter in the set of rank selection parameters is initialized to a random value.
7. An apparatus according to claim 5, wherein the at least one processor is further configured to: perform back propagation to determine the error of each rank selection parameter in the set of rank selection parameters based on the actual value of the rank selection parameter, wherein the actual value comprises a real number.
8. The apparatus of claim 5, wherein the rank selection parameter is updated simultaneously with the first weight tensor and the second weight tensor.
9. An apparatus for operating an artificial neural network for image classification, comprising: A device that feeds an image to an artificial neural network to produce a classified output; means for decomposing a weight tensor of the artificial neural network into a first weight tensor and a second weight tensor, wherein the decomposing comprises applying a singular value decomposition to the weight tensor; means for applying a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the ranks of the first weight tensor and the second weight tensor; means for updating the set of rank selection parameters in a backward pass of the artificial neural network by averaging the updates calculated for each of the set of rank selection parameters; as well as A device for applying a threshold to each rank selection parameter in the set of rank selection parameters during the calculation of inference in the forward pass of the artificial neural network, wherein the threshold is applied to the rank selection parameter so that if the actual value of the rank selection parameter is greater than or equal to the threshold, the applied value of the rank selection parameter is set to 1, or if the actual value of the rank selection parameter is less than the threshold, the applied value of the rank selection parameter is set to 0.
10. The apparatus of claim 9, wherein each rank selection parameter in the set of rank selection parameters is initialized to a random value.
11. The apparatus of claim 9, further comprising means for performing back-propagation to determine an error for each rank selection parameter in the set of rank selection parameters based on actual values of the rank selection parameters, the actual values comprising real numbers.
12. The apparatus of claim 9, wherein the rank selection parameter is updated simultaneously with the first weight tensor and the second weight tensor.
13. A computer readable medium storing executable code for learning the rank of an artificial neural network for image classification, comprising code for: The image is fed to an artificial neural network to produce a classification output; decomposing a weight tensor of the artificial neural network into a first weight tensor and a second weight tensor, wherein the decomposing comprises applying a singular value decomposition to the weight tensor; Applying a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the ranks of the first weight tensor and the second weight tensor; updating the set of rank selection parameters by averaging the updates calculated for each of the set of rank selection parameters in a backward pass of the artificial neural network; and During the calculation of inference in the forward pass of the artificial neural network, a threshold is applied to each rank selection parameter in the set of rank selection parameters, wherein the threshold is applied to the rank selection parameter so that if the actual value of the rank selection parameter is greater than or equal to the threshold, the applied value of the rank selection parameter is set to 1, or if the actual value of the rank selection parameter is less than the threshold, the applied value of the rank selection parameter is set to 0.
14. The computer-readable medium of claim 13, wherein each rank selection parameter in the set of rank selection parameters is initialized to a random value.
15. The computer-readable medium of claim 13, further comprising code for performing back-propagation to determine an error for each rank selection parameter in the set of rank selection parameters based on actual values of the rank selection parameters, the actual values comprising real numbers.
16. The computer-readable medium of claim 13, wherein the rank selection parameter is updated simultaneously with the first weight tensor and the second weight tensor.
