Transfer Learning with Basis Scaling and Pruning

Abstract

A method and system for performing transfer learning with basis scaling and pruning. One method includes obtaining a pre-trained deep convolutional neural network (DCNN), decomposing each weight matrix of the DCNN, and decomposing each convolution layer by applying each decomposed weight matrix to a convolution layer to form a first layer including a left matrix for convolution and a second layer including a right matrix for convolution. The method also includes providing a basis scaling convolution layer having a weight matrix derived by a function of the singular values ​​and the right singular vectors, and training basis scaling coefficients of the basis scaling convolution layer.

Description

[Technical Field]

[0001] The embodiments described herein generally relate to fine-tuning and pruning neural networks, such as for deep convolutional neural networks, as part of transfer learning. In particular, starting with a deep convolutional neural network that has been pre-trained on a labeled dataset, the embodiments described herein fine-tune and prune the neural network for classification tasks on new datasets. The fine-tuning and pruning can be performed in a transformed space where the weight features are linearly independent. Summary of the Invention

[0002] Deep convolutional neural networks are often used in the field of applied computer vision. Over the years, network architectures used for such computer vision applications (e.g., image analysis) have grown in performance along with the number of layers and parameters. The use of these networks has also been extended to resource-limited domains, such as edge computing. Edge computing is a distributed computing framework that brings enterprise applications closer to data sources, such as Internet of Things (IoT) electronic devices or local edge servers. Therefore, traditional neural network models (e.g., built on large mainframe servers), while accurate, can have size issues, especially when used in resource-limited computing environments. Therefore, optimizing network architectures to minimize computational requirements is crucial for extension to resource-limited domains. Furthermore, reducing floating-point operations (FLOPs) during inference directly impacts the power consumption of large-scale, customer-facing artificial intelligence (AI) applications. As a result, "green" AI advocates recommend using network size and the number of FLOPs, along with accuracy, as key performance evaluation metrics for neural networks.

[0003] Pruning can be used to improve architectural efficiency. Pruning is the process of discovering architectural components of a network that can be removed without significant loss of performance. Pruning algorithms can be categorized in different ways. For example, pruning can be achieved by removing unstructured weights and connections, or by removing structural content such as filters or layers. Many algorithms perform pruning directly on the convolutional weight matrix, while others attempt to reconstruct the weight matrix or its output features via low-rank approximation to reduce inference time. Also, some algorithms perform pruning without considering image data (e.g., training images), while others use image data for better pruning ratios and accuracy rates.

[0004] Although these pruning frameworks can reduce network size, they have some limitations. For example, because filters are linearly dependent in one layer, pruning in the original filter space can be ineffective. Also, low-rank approximations require further optimization apart from backpropagation to perform filter or feature reconstruction. Furthermore, fine-tuning of the entire network after pruning is required in many pruning frameworks, which may be undesirable when performing transfer learning with limited data.

[0005] Transfer learning involves transferring pre-trained network model features developed for one dataset or task to be reused as a starting point for a model for another dataset or task. For example, a pre-trained model for one use (e.g., natural image classification) can be used to generate a new model for a different use (e.g., medical image classification) by using one or more lower layers of the pre-trained model and training other layers (e.g., the final layer) to perform the desired new detection and classification. Transfer learning can be useful in areas where large, well-annotated datasets are scarce due to the cost of data acquisition and annotation, which is common in computer vision applications, particularly in the medical industry. However, the resulting network from transfer learning can be unnecessarily large and therefore inefficient, as the dataset used to train the pre-trained model usually contains features not present in the target dataset.

[0006] Thus, embodiments described herein relate to performing pruning in the context of transfer learning. Combining transfer learning and pruning as described herein provides efficient transfer learning using limited data with high accuracy rates while limiting network size.

[0007] For example, embodiments described herein provide methods and systems for fine-tuning and pruning a deep convolutional neural network pre-trained on a labeled dataset so that the network can perform classification tasks on a new dataset. Fine-tuning and pruning are performed in a transformed space where weight features are linearly independent. For example, the methods and systems described herein fine-tune and prune an orthogonal basis obtained by applying singular value decomposition (SVD) to the convolution weight matrix. In particular, the methods and systems described herein apply a pruning algorithm that prunes convolutional layers in an orthogonal subspace regardless of the network architecture. Because basis vectors are untrainable to facilitate transfer learning, the methods and systems described herein introduce basis scaling factors that are responsible for both importance estimation and fine-tuning of the basis vectors. These basis scaling factors are trainable by backpropagation during transfer learning and contribute to only a very small number of trainable parameters. Therefore, the framework provided by the methods and systems described herein is ideal for transfer learning with limited training data. Additionally, because batch normalization (BN) layers are trainable during transfer learning, the methods and systems described herein can use a dual pruning algorithm that combines basis pruning and network slimming for better flexibility and higher pruning ratios.

[0008] As described in more detail below, the embodiments described herein were tested by transferring features from four ImageNet pre-trained models to classify the CIFAR-10, MNIST, and Fashion-MNIST datasets. The results described below demonstrate the desirable properties of fine-tuning and pruning in orthogonal subspaces. For example, with minimal loss in classification accuracy (e.g., less than a 1% reduction in classification accuracy), the tested embodiments achieved high pruning ratios (e.g., pruning ratios up to 99.5% in parameters and 95.4% in FLOPs).

[0009] Therefore, the embodiments described herein provide a computer-implemented method for transfer learning, which can be implemented by an electronic processor. The computer-implemented method includes obtaining a pre-trained deep convolutional neural network (DCNN) including multiple convolutional layers. Each convolutional layer includes a weight matrix for convolution. The computer-implemented method further includes decomposing each weight matrix of the DCNN (e.g., by compact singular value decomposition (SVD)) into a left matrix whose columns are left singular vectors, a diagonal matrix of singular values, and a right matrix whose columns are right singular vectors. According to various embodiments, each of the left singular vectors and the right singular vectors is an orthonormal basis. According to one embodiment, the number of left singular vectors is the same as the number of right singular vectors, which is the same as the number of singular values.

[0010] The computer-implemented method further includes decomposing each convolutional layer of the DCNN into two successive layers using the decomposed matrix. According to one embodiment, the two successive layers include a first layer that is a convolutional layer having a left matrix as a weight matrix of the first layer, and a second layer that uses a basis-scaling convolutional layer having a weight matrix derived by a function of the singular values ​​and the right singular vectors as a weight matrix of the second layer. The computer-implemented method includes training basis scaling coefficients of the basis-scaling convolutional layer.

[0011] According to various embodiments, the computer-implemented method may further include iteratively removing basis scaling coefficients from each second layer after each training and removing corresponding matrix elements in the left and right matrices until a convergence criterion is reached. According to various embodiments, the computer-implemented method may further include adding a batch normalization layer after each convolutional layer if a batch normalization layer is not present after each convolutional layer.

[0012] The left matrix contains the left singular vectors, the diagonal matrix contains the singular values, and the right matrix contains the right singular vectors. The left singular vectors are orthogonal to each other, and the right singular vectors are orthogonal to each other. The left singular vectors and the right singular vectors do not have any explicit relationship.

[0013] The computer-implemented method may include iteratively removing scaling coefficients from each batch normalization layer after each training and removing corresponding matrix elements in the left and right matrices. The computer-implemented method may further include performing computer vision processing using the pruned neural network to detect objects in captured images or in an image dataset.

[0014] According to various embodiments described herein, a system comprising a memory and an electronic processor may be configured to perform the functions of the computer-implemented methods described above. According to various embodiments described herein, a non-transitory computer-readable medium provides computer-executable instructions that, when executed by a processor, cause the processor to perform the functions of one or more computer-implemented methods as described in this disclosure.

[0015] Other aspects of the embodiments will become apparent by consideration of the detailed description and accompanying drawings. [Brief explanation of the drawings]

[0016] [Figure 1]FIG. 1 illustrates a system for performing transfer learning with pruning, according to various embodiments.

[0017] [Figure 2A] 2 is a flowchart illustrating a method for performing transfer learning with pruning using the system of FIG. 1 , according to various embodiments.

[0018] [Figure 2B] FIG. 1 illustrates a decomposition of a convolutional layer, according to various embodiments.

[0019] [Figure 3A] FIG. 1 illustrates a basis pruning operation, according to various embodiments. [Figure 3B] FIG. 1 illustrates a double pruning operation, according to various embodiments.

[0020] [Figure 4A] FIG. 10 illustrates histograms of basis scaling coefficients in a basis scaling convolutional layer of a convolutional neural network with different L1 regularization parameters after transfer learning, according to various embodiments. [Figure 4B] FIG. 10 illustrates histograms of basis scaling coefficients in a basis scaling convolutional layer of a convolutional neural network with different L1 regularization parameters after transfer learning, according to various embodiments. [Figure 4C] FIG. 10 illustrates histograms of basis scaling coefficients in a basis scaling convolutional layer of a convolutional neural network with different L1 regularization parameters after transfer learning, according to various embodiments.

[0021] [Figure 5A] FIG. 10 illustrates basis scaling coefficients corresponding to different values ​​of the L1 regularization parameter for a basis-scaling convolutional layer, according to various embodiments.

[0022] [Figure 5B]FIG. 10 illustrates accuracy rate versus number of basis scaling coefficients below the pruning threshold with different L1 regularization parameters, according to various embodiments.

[0023] [Figure 6] FIG. 1 illustrates a comparison between network slimming and basis pruning, according to various embodiments.

[0024] [Figure 7A] This is a diagram including Table 1. [Figure 7B] This is a diagram including Table 2. [Figure 7C] This is a diagram including Table 3. [Figure 7D] This is a diagram including Table 4. [Figure 7E] This is a diagram including Table 5. DETAILED DESCRIPTION OF THE INVENTION

[0025] Before any embodiments are described in detail, it is to be understood that the embodiments are not limited in their application to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. Other embodiments are capable of being practiced or carried out in various ways.

[0026] It should also be understood that the phraseology and terminology used herein is for purposes of description and should not be regarded as limiting. The use of "including," "comprising," or "having," and variations thereof, herein is meant to encompass those previously listed items and equivalents thereof, as well as additional items. The terms "mounted," "connected," and "coupled" are used broadly and encompass both direct and indirect mounting, connections, and couplings. Furthermore, "connected" and "coupled" are not limited to physical or mechanical connections or couplings, but may include electrical connections or couplings, whether direct or indirect. Additionally, electronic communication and notification may be performed using any known means, including direct connections, wireless connections, etc.

[0027] A number of hardware- and software-based devices and a number of different structural components may be utilized to implement the embodiments described herein. Additionally, the embodiments may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, upon reading and understanding the detailed description, those skilled in the art will recognize that, in at least one embodiment, electronic-based aspects of the embodiments may be implemented in software (e.g., stored on a non-transitory computer-readable medium) executable by one or more processors. Accordingly, it should be noted that a number of hardware- and software-based devices and a number of different structural components may be utilized to implement the embodiments. For example, a “mobile device,” “smartphone,” “electronic device,” “computing device,” and “server” as described herein may include one or more electronic processors, one or more memory modules including a non-transitory computer-readable medium, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the components.

[0028] FIG. 1 illustrates a system 100 for performing transfer learning according to various embodiments. As illustrated in FIG. 1, the system 100 includes a server 110 and one or more image repositories 120. The server 110 communicates with the image repositories 120 via one or more wired or wireless communication networks 150. Portions of the wireless communication network 150 may be implemented using a wide area network (WAN) such as the Internet, a local area network (LAN) such as a Bluetooth® network or Wi-Fi®, and combinations or derivatives thereof. The system 100 may include more or fewer servers, and the server 110 and image repositories 120 shown in FIG. 1 are purely for illustrative purposes. For example, in some embodiments, functionality described herein as being performed by the server 110 is performed across multiple servers in a distributed or cloud computing environment. Also, in some embodiments, the image repository 120 may be combined with the server 110 or may communicate with the server 110 via a dedicated communication channel (as compared to a network). Additionally, in some embodiments, the components shown in system 100 may communicate through one or more intervening devices not shown in FIG.

[0029] In some embodiments, image repository 120 stores image data that can be used to train a neural network during transfer learning, as described above. In some embodiments, image repository 120 stores a large number of two-dimensional (2D) images, three-dimensional (3D) images, videos, or a combination thereof, such that some embodiments described herein can be used with computer vision applications, such as applications in the medical industry. Image repository 120 may be, for example, a picture archiving and communication system (PACS), a cloud storage environment, or the like. Image data stored in image repository 120 may be generated by one or more different types of imaging modalities, such as an X-ray computed tomography (CT) scanner, a magnetic resonance imaging (MRI) scanner, or the like. It should be understood that the embodiments described herein can be used with various types of images and are not limited to medical imaging applications.

[0030] 1, server 110 includes electronic processor 112, memory 114, and communication interface 116. Electronic processor 112, memory 114, and communication interface 116 communicate wirelessly, via a wired communication channel or bus, or a combination thereof. Server 110 may, in various configurations, include more components than those shown in FIG. 1. For example, in some embodiments, server 110 includes multiple electronic processors, multiple memory modules, multiple communication interfaces, or a combination thereof. Also, as described above, the functions described herein as being performed by server 110 may be performed in a distributed manner by multiple computers or servers located at various geographic locations.

[0031] The electronic processor 112 may be, for example, a microprocessor, an application specific integrated circuit (ASIC), or another suitable central processing unit (CPU). The electronic processor 112 is generally configured to execute software instructions to perform a set of functions, including those described herein. The memory 114 includes a non-transitory computer-readable medium such as a random access memory (RAM), a read-only memory (ROM), etc. The memory 114 stores data, including instructions executable by the electronic processor 112. The communication interface 116 communicates with other electronic devices outside the server 110. For example, the communication interface may include a wired or wireless transceiver or port that communicates via the communication network 150 and, optionally, one or more additional communication networks or connections.

[0032] 1, the memory 114 of the server 110 includes instructions 114a, a neural network 114b, and a training set 114c. The neural network 114b may be, for example, a two-dimensional (2D) U-net architecture, a 3D convolutional neural network (CNN), etc. The neural network 114b may be a pre-trained neural network (e.g., trained via a source dataset), and as described in more detail below, the server 110 uses transfer learning with basis scaling and pruning to generate a new model (referred to herein as neural network 114b′) that performs a new task using the neural network 114b, which was developed to perform a different task, as a starting point for the new model. As part of performing transfer learning, the server 110 uses the training set 114c, which may represent a set of annotated images, where the annotations (labels) pertain to the new (different) classification task. In some embodiments, the training set 114c is retrieved by the server 110 from the image repository 120. In some embodiments, after performing transfer learning with basis scaling and pruning, the resulting generated neural network 114b' can be used by the server 110 to perform a desired classification (e.g., applied to one or more images). Alternatively, or in addition, the generated neural network 114b' can be transmitted to one or more other devices. For example, as shown in FIG. 1, the neural network 114b' can be transmitted to or shared with (e.g., via the communication network 150) the edge server 140, the IoT device 130a, the smartphone 130c, or a combination thereof.

[0033] Although the transfer learning process with basis scaling and pruning (e.g., method 200) is described as being performed by server 110 and sent to edge server 140, according to one or more embodiments, edge server 140, which communicates with edge / IoT devices, may perform the method of transfer learning with basis scaling and pruning described herein. In this embodiment, information about the pre-trained neural network may be sent by server 110 to edge server 140, and edge server 140 may perform the method of transfer learning with basis scaling and pruning (e.g., method 200).

[0034] As described in more detail below, the server 110 is configured to perform transfer learning using basis scaling and simultaneous double pruning. Network pruning can be achieved by pruning individual weights or entire channels / filters. Pruning individual weights or connections can achieve high compression ratios due to its flexibility, but practical speedups may be limited given the sparsity of irregular weights unless dedicated software or hardware is utilized. In contrast, channel pruning exploits structured sparsity. While channel pruning is less flexible than weight-level pruning, the dense matrix structure is maintained after pruning, and significant practical speedups can be achieved using commercial libraries. Given the advantages described above, the embodiments described herein use channel pruning. However, to provide efficient transfer learning from one dataset to another, potentially much smaller dataset, it is desirable to minimize the number of trainable parameters during importance estimation and fine-tuning. To that end, because scaling factors enable filter-based fine-tuning that requires far fewer trainable parameters, embodiments described herein may use scaling factors in batch normalization (BN) layers as part of channel pruning (e.g., using backpropagation, an additional optimizer that updates the scaling factors during training, or a combination thereof). Furthermore, embodiments described herein can prune linearly independent filters obtained by applying singular value decomposition (SVD) or principal component analysis (PCA) to provide further improvements in efficiency. Matrix decomposition techniques such as SVD and PCA factorize convolution weight matrices or feature tensors into specified standard forms that reveal properties not observable in the original space. This transformation therefore enables specialized operations that lead to higher computational efficiency or accuracy rates. Embodiments described herein combine these advantages of SVD through rescaling and pruning of basis vectors, and in particular can perform double pruning (i.e., pruning in both the transformed and original spaces) to improve pruning ratios.The basis vectors described herein are untrainable in the transfer learning framework described, so orthogonality is preserved.

[0035] 2A is a flowchart illustrating a method 200 for performing transfer learning with pruning, according to various embodiments. Method 200 may be implemented via computer-executable instructions that cause various operations of method 200 to be performed by a processor. For example, method 200 is described herein as being performed via server 110 (executing instructions 114a via electronic processor 112). However, as noted above, different hardware and computing environments (such as a distributed computing environment) may be used.

[0036] 2A, computer-implemented method 200 includes obtaining (at block 202) a pre-trained deep convolutional neural network (DCNN) including multiple convolutional layers, such as network 114b. Method 200 also includes decomposing (at block 204) each convolutional weight matrix of the pre-trained DCNN (e.g., by compact singular value decomposition (SVD)) into a left matrix whose columns are the left singular vectors of the weight matrix, a diagonal matrix of singular values, and a right matrix whose columns are the right singular vectors of the weight matrix.

[0037] Method 200 also includes decomposing each convolutional layer of the pre-trained DCNN into two layers (at block 206), for example, by applying the respective decomposed weight matrices to the convolutional layer to form a first layer including a left matrix for convolution and a second layer including a right matrix for convolution. In particular, decomposing into two layers may include decomposing each convolutional layer of the DCNN into two consecutive layers using the decomposed weight matrices. The two consecutive layers include a first layer that is a convolutional layer having a left matrix as a weight matrix and a second layer that is a basis-scaled convolutional layer having a weight matrix derived by a function of the singular values ​​and the right singular vectors. For example, each convolutional layer may include a weight matrix for convolution. Each of the left singular vectors and the right singular vectors may be an orthonormal basis. The number of left singular vectors may be the same as the number of right singular vectors and may be the same as the number of singular values. The second layer may include a plurality of basis scaling coefficients corresponding to matrix elements in the left and right matrices and may be trainable by backpropagation.

[0038] 2A, the computer-implemented method 200 also includes training (at block 208) the basis scaling coefficients of the basis-scaling convolutional layer and the scaling coefficients of the BN layer. After each basis-scaling convolutional layer is trained, basis pruning is performed (at block 210) using the basis scaling coefficients to remove less important basis vectors. For example, the basis scaling coefficients from each second layer may be removed (pruned) along with the corresponding matrix elements in the left and right matrices.

[0039] 2A , the computer-implemented method 200 also performs (at block 212) double pruning (pruning using scaling factors of the BN layer) as described below and uses (at block 214) the transfer-trained and basis-pruned DCNN or the doubly-pruned DCNN. For example, the server 110 (or a different server, an edge server, an IoT device, a smartphone, or a combination thereof) may use the transfer-trained and basis-pruned or doubly-pruned DCNN for a computer vision application (or other applications, such as, for example, natural language processing applications). For example, a processor may perform computer vision processing using the basis-pruned or doubly-pruned neural network. The computer vision processing (e.g., application processing) may include performing an action (e.g., updating a user interface, motion control of a robotic electronic device, or another suitable action) based on performing computer vision processing using the basis-pruned or doubly-pruned neural network to detect objects (e.g., in captured images or videos, or in an image dataset).

[0040] For example, assume that the pre-trained deep neural network 114b includes multiple convolutional layers L (L1, L2, ..., Ln), where each convolutional layer Lj includes a weight matrix Mj for convolution. Thus, in this example, decomposing a layer includes decomposing each weight matrix Mj into a left matrix MLj, a diagonal matrix MDj, and a right matrix MRj. Using these matrices, the decomposed weight matrix Mj can be applied to a layer Lj to form a first layer L'j including MLj for convolution and a second layer L''j including MRj for convolution. Here, the second layer L''j includes multiple basis scaling coefficients corresponding to matrix elements in MLj and MRj and is trainable by backpropagation. Continuing with this example, the basis scaling coefficients from each second layer L''j are iteratively removed after each training, and the corresponding matrix elements in MLj and MRj are removed until a convergence criterion is reached.

[0041] If there is no batch normalization layer after each Lj, a batch normalization layer Bj is also added after each Lj. The left matrix MLj contains the left singular vectors LSVj, the diagonal matrix MDj contains the singular values ​​SVj, and the right matrix MRj contains the right singular vectors RSVj. The LSVj are orthogonal to each other, and the RSVj are orthogonal to each other, but the LSVj and RSVj do not have any explicit relationship. Using these BN layers, the scaling coefficients from each Bj are iteratively removed after each training, and the corresponding matrix elements are removed in MLj and MRj.

[0042] Further details regarding method 200 are provided below. As explained below, the embodiments described herein present a convolutional weight matrix with an orthogonal basis that enables more effective network pruning for transfer learning. In particular, features of a convolutional layer are distributed among linearly dependent filters, and feature representations differ with different initializations. By representing features using an orthogonal basis (e.g., obtained by SVD or PCA), fewer channels are needed to represent useful features, and network pruning in such a subspace can be more effective. In addition, using an orthogonal basis (e.g., via SVD or PCA) for network pruning allows weights to be approximated using low-rank tensor approximations, which also reduces computational complexity. Filter pruning in a transformed space (e.g., using an orthogonal basis) provides improved effectiveness. As discussed in detail below, the weight matrix may be decomposed into an orthogonal basis, and basis scaling may be used for importance estimation and fine-tuning. More filters can be pruned with minimal loss in accuracy. For example, as shown in Table 2 (FIG. 7B), converting an ImageNet-trained model (e.g., a ResNet-50 model) to an MNIST-trained model according to embodiments described herein resulted in a pruning ratio of 99.5% in parameters and 95.4% in FLOPs, with roughly a 1% decrease in classification accuracy.

[0043] Convolution weight representation in orthogonal subspaces

[0044] The weight matrix of a convolutional layer (e.g., a 4D convolution weight matrix) is

number

number

number

number

[0045] According to one embodiment, one or more of SVD and PCA may be used to represent the weights in an orthogonal basis, for example, compact SVD may be used for the representation.

[0046] The matrix W may be factorized by compact SVD as follows: W=UΣV T [Equation 3], where:

number

number

number

[0047] To transform the weight matrix W using PCA, the rows and columns of W may be viewed as samples and features, respectively. To use PCA, the symmetric covariance matrix

number

[0048] The relative scale between channels is important, and therefore the columns of W are not normalized. T Since U=I and Σ is diagonal, substituting equation 1 into equation 4 gives C=VΣ 2 V T This leads to Equation 5. Therefore, the columns of V are the eigenvectors of C corresponding to the non-zero eigenvalues. Using PCA, W can be projected onto an orthonormal basis of V. T Since V=I, using equation 3, the projection becomes:

number

number

[0049] Convolutional Layer Decomposition

[0050] Using SVD or PCA, the convolution weights can be represented by an orthonormal basis in U and V. The contribution of a basis vector is proportional to the corresponding singular value, but most singular values ​​are of similar magnitude, and selecting which ones to remove is nontrivial, especially without considering image data. The use of image data can help determine the importance of filters through metrics such as the rank of feature maps or the gradient of network weights. Because one goal of transfer learning is to perform transfer using limited data, it is desirable to preserve the original weights as much as possible while pruning. Therefore, similar to a framework in which the relative importance of a filter is indicated by the scaling coefficient of a BN layer, a basis scaling convolution (referred to herein as "BasisScalingConv") layer may be used to measure the importance of basis vectors.

[0051] For convolutional layer decomposition, the untrainable convolution weights W and bias

number

number

number

number

number

number

number

number

[0052] Figure 2B shows the decomposition of a convolutional layer. With respect to Figure 2B, only the vector of basis scaling coefficients, s, is trainable during transfer learning.

[0053] The first layer is a normal convolutional layer with U as the convolution weight without bias. The second layer is a convolutional layer with s,

number

number

number

number

[0054] Transfer Learning with Basis Pruning

[0055] As described above, transfer learning converts neural network features trained from one dataset (e.g., a pre-trained network) to be applied to another dataset. Given a pre-trained neural network model, all layers up to and including the final convolutional layer and associated BN and activation layers may be retained, and a global average pooling layer and a final fully connected layer for classification may be added. In the case of transfer learning with basis pruning, all convolutional layers are decomposed as discussed above. Transfer learning may include additional BN layers if they are not present for better domain adaptation. Because BN layers are important for domain adaptation, they may be trainable during transfer learning or introduced after each convolutional layer if not present (e.g., VGGNet). Thus, only the BN layers, the vector s in each BasisScalingConv layer, and the final fully connected layer are trainable in some embodiments.

[0056] To improve sparsity, L1 regularization is applied to the basis scaling coefficient s. L1 regularization is important not only for improving sparsity due to a larger pruning ratio, but also for a more precise ranking of the importance of the basis vectors. Figure 6, described below, is based on a study of the effect of different L1 parameters.

[0057] Figures 3A and 3B illustrate pruning operations according to various embodiments: Figure 3A illustrates base pruning, and Figure 3B illustrates double pruning.

[0058] Base pruning

[0059] Base pruning may include training the transformed network against a target dataset and removing basis vectors having corresponding basis scaling coefficients lower than a given threshold from the weight matrix. x o and x i Since the sizes of and x are not affected, base pruning can be applied to any architecture. As shown in FIG. 3A, base pruning can be performed by removing the same number of basis vectors from matrix U (the hatched column on the right) and matrix

Number

[0060] After training for a sufficient number of epochs for the desired classification accuracy, the basis vectors corresponding to small scaling coefficients are pruned (FIG. 3A). Let r be the number of scalars remaining after pruning. In that case, U, S, and p <in Equation 8, and

Number

Number

Number

[0061] Double Pruning with Batch Normalization

[0062] Double pruning may involve scaling factors in the BN layer being used simultaneously for further pruning. Simultaneously may mean during the same operating timeframe. For example, as discussed further below, experiments were conducted using transfer learning (according to the described embodiments) from a model pre-trained on ImageNet to other datasets (models—VGG-16, DenseNet-121, ResNet-50, MobileNetV2, and datasets—CIFAR-10, MNIST, Fashion-MNIST), which resulted in high pruning ratios with minimal loss in accuracy.

[0063] FIG. 3B shows the input pruning (lower grey row of matrix U) and output pruning (matrix U) using double pruning (e.g., the basis pruning of FIG. 3A and the immediately preceding BN layer and the immediately following BN layer, respectively).

number

[0064] Pruning Procedure

[0065] Thus, as an overview of the above functionality, the pruning procedure (performed as part of transfer learning) involves the following stages:

[0066] Step 1. Given a pre-trained model, keep all layers up to and including the final convolutional layer and associated BN and activation layers. Insert BN layers if necessary.

[0067] Step 2. Decompose each convolutional layer into a convolutional layer and a BasisScalingConv layer as described above. A global average pooling layer and a final fully connected layer for classification may be added.

[0068] Step 3. Train the model. Only the BN layer, the scaling coefficients in the BasisScalingConv layer, and the fully connected layer are trainable.

[0069] Step 4. Prune the trained model. Remove basis vectors whose scaling coefficients in the BasisScalingConv layer are lower than a given threshold. In the case of double pruning, also remove filters whose scaling coefficients in the BN layer are lower than a given threshold.

[0070] Step 5. Train the pruned model as in step 3 above.

[0071] If necessary, further iterations starting from stage 4 can be performed, but one iteration is sufficient, especially for relatively simple problems (e.g., MNIST). Note that in stages 3 and 5, each scaling coefficient modifies the weights of the basis vectors, or filters, as a whole. This may be viewed as basis- or filter-based fine-tuning, which requires far fewer trainable parameters than the individual weights in the fine-tuning.

[0072] Experimental and Test Data

[0073] As noted above, the framework described herein has been tested. Details of these tests are provided below. These details are provided with respect to various embodiments of the methods and systems described herein and should not be viewed as limitations.

[0074] Models and Datasets

[0075] To study the characteristics of the framework described herein, we performed transfer learning experiments using four ImageNet pre-trained models on three other datasets. ImageNet was used as the source dataset due to its rich features, trained from 1.2 million images. The four models correspond to the architectures of VGG-16, DenseNet-121, ResNet-50, and MobileNetV2. VGG-16 has a relatively simple architecture. DenseNet-121 and ResNet-50 have skip connections implemented by tensor concatenation and addition, respectively. MobileNetV2 is a very compact model with lightweight deep-by-depth convolutions and skip connections. Because deep-by-depth convolutions contribute only approximately 3% of the total convolution weights, deep-by-depth convolutions may not need to be pruned in some embodiments of the framework described herein. The three datasets include CIFAR-10, MNIST, and Fashion-MNIST. The CIFAR-10 dataset consists of 32x32 color images in 10 classes of animals and vehicles, with 50k training images and 10k test images. The MNIST dataset of handwritten digits (0-9) has 60k 28x28 grayscale training images in 10 classes and 10k test images. The Fashion-MNIST dataset has a training set of 60k 28x28 grayscale training images in 10 classes of fashion categories and 10k test images, which can be used as a drop-in replacement for MNIST. Each set of training images was split into 90% for training and 10% for validation. Only results for the test images are reported.

[0076] Tested Frameworks

[0077] Given a pre-trained model, all layers up to and including the final convolutional layer and associated BN and activation layers may be retained, and a global average pooling layer and a final fully connected layer may be added. The BN layer may be trainable using the final fully connected layer while the other layers are frozen. Based on this configuration, different frameworks may be tested: (a) baseline: no layer decomposition and pruning; (b) basis pruning: all convolutional layers are decomposed with the basis scaling coefficients trainable; (c) pruning only by the basis scaling coefficients (as discussed above); and (d) double pruning: pruning also by the scaling coefficients in the BN layer (as discussed above). The framework may be applied for transfer learning from ImageNet to other tested datasets.

[0078] In the case of the network slimming framework, since the BN layer can be trained on the baseline model with L1 regularization and non-negativity constraints, pruning can be applied directly to the baseline model, followed by filter fine-tuning through the BN layer.

[0079] Training Strategies

[0080] In various training embodiments, since the network architectures of the ImageNet pre-trained models were generated for an image size of 224x224, directly applying them to target datasets with smaller image sizes would result in insufficient spatial size (e.g., feature maps of size 1x1) in deeper layers and therefore poor performance.

[0081] Therefore, in training embodiments, image sizes were scaled up by a factor of 4 in each dimension, i.e., 128x128 for CIFAR-10 and 112x112 for MNIST and Fashion-MNIST. Image augmentation was used to reduce overfitting, using approximately ±15% shifts in height and width for all datasets, random horizontal flips for CIFAR-10 and Fashion-MNIST, and ±15% rotation for MNIST. According to various embodiments, all images were zero-centered in intensity. Dropout with a rate of 0.5 was applied before the final fully connected layer. Stochastic gradient descent (SGD) with warm restarts with cosine annealing was performed for 10 iterations, respectively. -4 and 10 -2 was used as the learning rate scheduler, with minimum and maximum learning rates as

[0082] According to various embodiments, the scheduler initially restarted at the 100th epoch, which may be increased by 1.31 times at every restart. According to this embodiment, there was no learning rate decay at the restarts. The SGD optimizer may be used with a momentum of 0.9 and a batch size of 128. According to this embodiment, there were 400 epochs for each training, and the same L1 regularization parameter was imposed on the scaling coefficients of the BasisScalingConv and BN layers. According to various embodiments, all scaling coefficients were initialized to 1 and constrained to be non-negative. The thresholds for the basis scaling coefficients and the BN scaling coefficients for pruning were empirically obtained as 10. -2 and 10 -10 Only one iteration of the pruning procedure described above was performed for each experiment, except for one framework where a different pruning procedure was used.

[0083] Effect of L1 regularization

[0084] Appropriate L1 regularization imposed on the scaling coefficients suppresses the magnitude of less important ones with minimal effect on the accuracy rate. Therefore, we performed experiments to study the effect of L1 regularization on an embodiment of the framework described herein. Figures 4A-4C show histograms (100 bins) of all basis scaling coefficients (roughly 4.2k) in all BasisScalingConv layers of VGG-16 with different L1 regularization parameters (λ) after transfer learning (stage 3 above). In Figures 4A-4C, the basis scaling coefficients correspond to different values ​​of the L1 regularization parameter (λ) for VGG-16. The larger λ is, the smaller the scaling coefficients (<10 -2 , which is found to be a suitable threshold for pruning the basis vectors without a large reduction in accuracy.

[0085] Figure 5A shows the normalized basis scaling coefficients in the BasisScalingConv layer corresponding to different values ​​of the L1 regularization parameter (λ) for VGG-16. The scaling coefficients are sorted in descending order according to their corresponding singular values. Regardless of the value of λ, scaling coefficients corresponding to smaller singular values ​​tended to be more suppressed, while those corresponding to larger singular values ​​had similar normalized values. The larger the L1 regularization parameter (λ), the more the scaling coefficients of smaller singular values ​​were pushed toward zero. Nevertheless, some scaling coefficients of smaller singular values ​​were large regardless of the L1 regularization parameter (λ). Therefore, rather than simply pruning according to singular values, pruning according to an importance threshold may be performed. -2 Removing basis vectors using a scaling factor smaller than ∇ ...

[0086] Figure 5B shows the accuracy rate versus the number of basis scaling coefficients below the pruning threshold with different L1 regularization parameters for CIFAR-10. Here, Figure 5B shows the accuracy rate and potential parameter reduction using different L1 regularization parameter (λ) values ​​for CIFAR-10. λ = 2 × 10 -4 provides a significant amount of parameter reduction without a large drop in accuracy. This can also be observed in other datasets, so λ = 2 × 10 -4 was chosen to be used for further experiments under the same training strategy. Similar observations were found for the BN layer, and therefore the same value of the L1 regularization parameter (λ) was applied.

[0087] Comparison between basis pruning and pruning in the original space

[0088] Figure 6 shows a comparison between pruning in the original space using BN layers (i.e., network slimming) and basis pruning. Figure 6 shows a comparison between network slimming and basis pruning in CIFAR-10. Each convolutional layer had the same percentage of basis vectors or filters removed. Different percentages (30%, 50%, 70%, and 90%) were tested. To investigate the effect of pruning on accuracy, after step 3 above, the model may be pruned by removing a fixed percentage of the least important basis vectors or filters from all layers. Step 5 above may then be performed to fine-tune the pruned model.

[0089] For all models, the accuracy reduction of network slimming increased as the percentage of removed filters increased. Such reduction in accuracy was less severe for ResNet-50 and MobileNetV2, but relatively large for VGG-16 and DenseNet-121. In contrast, the accuracy reduction was much smaller for basis pruning. This is consistent with the notion that features have lower dispersion under orthonormal bases, and therefore pruning can be performed with a lower accuracy reduction.

[0090] Comparison between frameworks

[0091] 7A-7E show Tables 1-5, respectively. Tables 1-3 show the performance of various embodiments of the "base" pruning and "dual" pruning frameworks described herein, and comparisons with other frameworks.

[0092] In particular, Figure 7A (Table 1) shows the pruning results for CIFAR-10 using a model pre-trained on ImageNet. Figure 7B (Table 2) shows the pruning results for MNIST using a model pre-trained on ImageNet. Figure 7C (Table 3) shows the pruning results for Fashion-MNIST using a model pre-trained on ImageNet. Figure 7D (Table 4) shows the number of total parameters and the number of trainable parameters before pruning. Figure 7E (Table 5) shows the pruning results for CIFAR-10 (20% training) using a model pre-trained on ImageNet.

[0093] In these tables, PR stands for pruning ratio, and the best results after pruning are bolded. In Tables 2 and 3, images were upsampled to 112 × 112, and in Tables 1 and 5, images were upsampled to 128 × 128. Transfer learning results were proportional to the difficulty of the dataset. The best classification accuracy rates after pruning were 94.1%, 99.6%, and 94.4% for CIFAR-10, MNIST, and Fashion-MNIST, respectively. Regardless of the dataset, the baseline models DenseNet-121 and ResNet-50 performed better than the other baseline models, with DenseNet-121 slightly better than ResNet-50. For VGG-16, the accuracy rate after pruning by our algorithm was better than the baseline model. For VGG-16 and ResNet-50, which had more parameters and FLOPs among the baseline models, the basis pruning algorithm achieved better pruning ratios than pruning with the BN layer alone (i.e., network slimming) at similar accuracy rates. The double pruning algorithm achieved larger pruning ratios with less than a 0.1% reduction or even increase in accuracy rate.

[0094] Different frameworks behaved differently for different models and datasets. For DenseNet-121, network slimming produced large pruning ratios for CIFAR-10 and MNIST, but with a large reduction in accuracy, especially for MNIST (99.9% pruning ratio at 48.5% accuracy). In contrast, the basis pruning framework was more stable but with smaller pruning ratios. For MobileNetV2, the pruning ratio for basis pruning was smaller than that for network slimming. The FLOP pruning ratio had a more adverse effect on CIFAR-10 with basis pruning, since only 13.5% of the parameters were pruned. On the other hand, double pruning produced larger pruning ratios than network slimming for both MNIST and Fashion-MNIST.

[0095] Transfer learning with limited data

[0096] Table 4 shows the total number of parameters and the number of trainable parameters before pruning. Because only the BN layer, basis scaling coefficients, and final fully connected layer were trainable in each model, the number of trainable parameters was very small (<104K). To verify performance with limited data, we performed experiments using only 20% of the CIFAR-10 training images. Apart from frameworks using basis or filter-based fine-tuning, we also performed experiments on a framework using first-order Taylor expansion (Taylor-FO) for importance approximation, and fine-tuning was performed on all weights in the model. Table 5 shows that the Taylor-FO framework had the lowest accuracy rate for both DenseNet-121 and ResNet-50. Furthermore, for ResNet-50, Taylor-FO had the largest parameter pruning ratio, but it also had the second-lowest pruning ratio in FLOPs. Therefore, using basis or filter-based fine-tuning is advantageous when training data is limited.

[0097] Experimental Conclusion

[0098] As evidenced by the above experiments and test data, the framework described herein provides an efficient transfer learning framework that performs pruning and fine-tuning of trained convolution weights in a transformed space. Using singular value decomposition, a convolution layer can be decomposed into two consecutive layers with basis vectors as their convolution weights. When basis scaling factors are introduced, the basis vectors can be fine-tuned and pruned to reduce network size and inference time. Similarly, simultaneous double pruning can be achieved using scaling factors from a batch normalization layer. Experimental results show that basis vectors with smaller singular values ​​tend to be pruned more, and pruning the basis vectors leads to a smaller reduction in accuracy than pruning in the original space. When transferring pre-trained features on ImageNet to other datasets, high classification accuracy rates with pruning ratios greater than 99% can be achieved. Furthermore, as shown above, large pruning ratios and high levels of accuracy can be maintained even when only 20% of the CIFAR-10 training data is used. This is a desirable property for transfer learning in data-limited scenarios.

[0099] The method may include (i) providing a pre-trained deep neural network including convolutional layers (each convolutional layer including a weight matrix for convolution); and (ii) decomposing each weight matrix by compact singular value decomposition into three matrices: a matrix (U) whose columns are left singular vectors, a diagonal matrix (Σ) of singular values, and a matrix (V) whose columns are right singular vectors. Each of U and V is an orthonormal basis. The number of left singular vectors and the number of right singular vectors are the same, which is equal to the number of singular values. The method may include (iii) using the decomposed matrices, where each convolutional layer is decomposed into two consecutive layers. The first layer is a convolutional layer with U as a weight matrix. The second layer is a basis scaling convolutional (BasisScalingConv) layer, whose weight matrix is ​​the multiplication of Σ and the transpose of V. This BasisScalingConv layer includes basis scaling coefficients that can be trained by backpropagation. The number of basis scaling coefficients is the same as the number of singular values. Each row of the weight matrix in the BasisScalingConv layer is multiplied by the corresponding basis scaling coefficient before convolution.

[0100] The method may include (iv) training basis scaling coefficients of the BasisScalingConv layer, and (v) removing any basis scaling coefficients, singular values ​​in Σ, and singular vectors in U and V that have corresponding basis scaling coefficients lower than a given threshold. Operations (iv)-(v) may be repeated until at least one convergence criterion is met. The method may include adding a batch normalization layer after each convolutional layer, if one is not present. The method may include removing any filters in the weight matrix that have corresponding scaling coefficients in the batch normalization layer that are lower than a given threshold, if a batch normalization layer is present.

[0101] Therefore, the embodiments described herein provide a framework for fine-tuning and pruning orthogonal bases obtained by applying singular value decomposition (SVD) to convolution weight matrices. In particular, the embodiments described herein apply a basis pruning algorithm to prune any convolutional layer in an orthogonal subspace. Because the basis vectors are untrainable in some embodiments to facilitate transfer learning, basis scaling coefficients are prescribed, which are responsible for both importance estimation and fine-tuning of the basis vectors. These basis scaling coefficients are trainable by backpropagation during transfer learning and contribute only to a small number of trainable parameters. Therefore, the framework described herein is ideal for transfer learning with limited training data. In addition, because the BN layer is also trainable during transfer learning, the framework described herein uses a dual-pruning algorithm that combines basis pruning and network slimming for better flexibility and a higher pruning ratio.

[0102] Various features and advantages of the embodiments are set forth in the following claims.

Claims

1. 1. A computer-implemented method of transfer learning, comprising: obtaining a pre-trained deep convolutional neural network (DCNN) including a plurality of convolutional layers, where each convolutional layer includes a weight matrix for convolution; decomposing each weight matrix of the DCNN into a left matrix whose columns are left singular vectors, a diagonal matrix of singular values, and a right matrix whose columns are right singular vectors, where each of the left singular vectors and the right singular vectors is an orthonormal basis, and the number of the left singular vectors is the same as the number of the right singular vectors and the number of the singular values; decomposing each convolutional layer of the DCNN into two successive layers using the decomposed matrix, wherein the two successive layers include the first layer being a convolutional layer having the left matrix as the weight matrix of the first layer, and the second layer using a basis scaling convolutional layer having a weight matrix derived by a function of the singular values ​​and the right singular vectors as the weight matrix of the second layer; and training basis scaling coefficients of the basis scaling convolution layer; A computer-implemented method comprising:

2. after each training iteratively removing the basis scaling coefficients from each second layer and removing corresponding matrix elements in the left and right matrices until a convergence criterion is reached. The computer-implemented method of claim 1 further comprising:

3. adding a batch normalization layer after each convolutional layer, if one does not already exist after each convolutional layer; The computer-implemented method of claim 2 further comprising:

4. The computer-implemented method of claim 1 , wherein the left matrix contains left singular vectors, the diagonal matrix contains singular values, and the right matrix contains right singular vectors.

5. The computer-implemented method of claim 1 , wherein the left singular vectors are orthogonal to one another and the right singular vectors are orthogonal to one another, but the left singular vectors and the right singular vectors do not have any explicit relationship.

6. after each training iteratively removing the basis scaling coefficients from each batch normalization layer and removing corresponding matrix elements in the left matrix and the right matrix. The computer-implemented method of claim 1 further comprising:

7. performing computer vision processing using the pruned neural network to detect objects in the captured images or in the image dataset; The computer-implemented method of claim 1 further comprising:

8. The computer-implemented method of claim 1 , wherein the decomposition of each weight matrix is ​​by compact singular value decomposition (SVD).

9. memory; and 1. An electronic processor comprising: Obtaining a pre-trained deep convolutional neural network (DCNN) including multiple convolutional layers, where each convolutional layer includes a weight matrix for convolution; decomposing each weight matrix of the DCNN into a left matrix whose columns are left singular vectors, a diagonal matrix of singular values, and a right matrix whose columns are right singular vectors, where each of the left singular vectors and the right singular vectors is an orthonormal basis, and the number of the left singular vectors is the same as the number of the right singular vectors and the number of the singular values; decomposing each convolutional layer of the DCNN into two successive layers using the decomposed matrix, wherein the two successive layers include the first layer being a convolutional layer having the left matrix as the weight matrix of the first layer, and the second layer using a basis scaling convolutional layer having a weight matrix derived by a function of the singular values ​​and the right singular vectors as the weight matrix of the second layer; and training basis scaling coefficients of the basis scaling convolutional layer; an electronic processor configured to: A system comprising:

10. The electronic processor: after each training, iteratively remove the basis scaling coefficients from each second layer and remove corresponding matrix elements in the left matrix and the right matrix until a convergence criterion is reached. The system of claim 9 , further configured to:

11. The electronic processor: Add a batch normalization layer after each convolutional layer if one does not already exist. The system of claim 10 , further configured to:

12. The system of claim 9 , wherein the left matrix includes left singular vectors, the diagonal matrix includes singular values, and the right matrix includes right singular vectors.

13. The system of claim 9 , wherein the left singular vectors are orthogonal to one another and the right singular vectors are orthogonal to one another, but the left singular vectors and the right singular vectors do not have any explicit relationship.

14. 10. The system of claim 9, wherein the electronic processor is further configured to: iteratively remove the basis scaling coefficients from each batch normalization layer after each training and remove corresponding matrix elements in the left matrix and the right matrix.

15. 15. The system of claim 9, wherein the electronic processor is further configured to: perform computer vision processing using a pruned neural network to detect objects in captured images or in an image dataset.

16. When executed by a processor, the processor: Obtaining a pre-trained deep convolutional neural network (DCNN) including multiple convolutional layers, where each convolutional layer includes a weight matrix for convolution; decomposing each weight matrix of the DCNN into a left matrix whose columns are left singular vectors, a diagonal matrix of singular values, and a right matrix whose columns are right singular vectors, where each of the left singular vectors and the right singular vectors is an orthonormal basis, and the number of the left singular vectors is the same as the number of the right singular vectors and the number of the singular values; decomposing each convolutional layer of the DCNN into two successive layers using the decomposed matrix, wherein the two successive layers include the first layer being a convolutional layer having the left matrix as the weight matrix of the first layer, and the second layer using a basis scaling convolutional layer having a weight matrix derived by a function of the singular values ​​and the right singular vectors as the weight matrix of the second layer; and training basis scaling coefficients of the basis scaling convolutional layer; A computer program comprising computer-executable instructions for causing a computer to perform the following:

17. after each training, iteratively remove the basis scaling coefficients from each second layer and remove corresponding matrix elements in the left matrix and the right matrix until a convergence criterion is reached. The computer program of claim 16 further comprising:

18. The computer-executable instructions cause the processor to:

17. The computer program of claim 16, further configured to: add a batch normalization layer after each convolutional layer if a batch normalization layer is not present after each convolutional layer.

19. 17. The computer program of claim 16, wherein the left singular vectors are orthogonal to one another and the right singular vectors are orthogonal to one another, but the left singular vectors and the right singular vectors do not have any explicit relationship.

20. 20. The computer program of claim 16, wherein the computer-executable instructions are further configured to cause the processor to: perform computer vision processing using a pruned neural network to detect objects in captured images or in an image dataset.

Images