Training neural networks through dense connection-based knowledge extraction
By generating a teacher network based on student network and using internal and cross-network connections for knowledge refinement, the problems of high training costs and complex hyperparameter adjustment in the existing technology are solved, and efficient and accurate DNN training is achieved.
Patent Information
- Application Number
- CN202280094906.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-16
- Publication Date
- 2024-12-03
AI Technical Summary
When training deep neural networks (DNNs), existing knowledge refinement techniques rely on well-defined teacher networks, and are costly to train, and require manual adjustment of hyperparameters, resulting in waste of computing resources, human resources and time.
By generating a teacher network based on student networks, leveraging internal and cross-network connections to facilitate knowledge refinement, allowing student networks and teacher networks to train simultaneously, reducing dependence on teacher networks and training costs.
More efficient DNN training is achieved, reducing training costs, improving accuracy, and simplifying user operations so that users do not need to define the teacher network in advance.
Smart Images

Figure CN119072695A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to neural networks, and more particularly to training deep neural networks (DNNs) through densely connected based knowledge distillation. Background Art
[0002] DNNs are widely used in various artificial intelligence applications, including computer vision, speech recognition, and natural language processing, due to their ability to achieve high accuracy. However, high accuracy comes at a significant computational cost. The computational requirements of DNNs are extremely high, as each inference may require hundreds of millions of MAC (multiply-accumulate) operations, as well as hundreds of millions of weight operations to be stored for classification or detection. Therefore, techniques to improve the efficiency of DNNs are needed. BRIEF DESCRIPTION OF THE DRAWINGS
[0003] The embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To aid this description, similar reference numerals designate similar structural elements. The embodiments are illustrated in the accompanying drawings by way of example and not by way of limitation.
[0004] Figure 1 An example layer structure of a DNN is illustrated according to various embodiments.
[0005] Figure 2 is a block diagram of a DNN system according to various embodiments.
[0006] Figure 3 An example teacher network formed based on a student network is illustrated according to various embodiments.
[0007] Figure 4 Merging a student network with a teacher network is illustrated according to various embodiments.
[0008] Figure 5 Training the merged network is illustrated according to various embodiments.
[0009] Figure 6 A deep learning (DL) environment is illustrated according to various embodiments.
[0010] Figure 7 is a flowchart illustrating a method for training a DNN through densely connected based knowledge distillation according to various embodiments.
[0011] Figure 8 is a block diagram of an example computing device in accordance with various embodiments. DETAILED DESCRIPTION
[0012] Overview
[0013] DNNs are widely used in the fields of computer vision, speech recognition, image and video processing, mainly due to their ability to achieve super-human level accuracy. However, the improvement in accuracy comes at a significant computational cost. The underlying DNN has extremely high computational requirements, as each input requires at least hundreds of millions of MAC operations and hundreds of millions of weights to be processed for classification or detection. Energy-constrained mobile and embedded systems with extremely limited energy and area budgets often use area- and energy-efficient DNN accelerators as the underlying hardware for executing ML applications.
[0014] Knowledge distillation is a solution that provides a teacher-learning training framework to train a compact, computationally efficient DNN model that achieves higher prediction accuracy than standard training. Most knowledge distillation techniques require two training phases. The first training phase is to train a teacher network. The second training phase is to train a student network using a pre-trained teacher network. In the second phase, the training of the student network is guided by the predictions made by the teacher model. For example, the internal parameters (e.g., weights) of one or more layers of the student network are adjusted so that the features output from these layers can match the features output from the corresponding layers of the teacher network. The student network is typically smaller in size than the teacher network and therefore requires fewer computational resources for inference. Some other knowledge distillation techniques may be one-stage solutions and can train the teacher and student networks collaboratively.
[0015] However, these knowledge distillation techniques have several drawbacks. For example, these knowledge distillation techniques rely on a well-defined teacher network to train any given student network. In some applications, no such teacher network may be available. In addition, because these knowledge distillation techniques require additional training of more complex teacher models, the training cost may be significantly higher than the cost of training a single student network. In some scenarios, the training cost may be 3 to 20 times higher. In addition, these knowledge distillation techniques typically require adjustment of hyperparameters to train the student network. Examples of such hyperparameters include learning rate, temperature, weighting coefficients for different loss function terms, and so on. This adjustment sometimes has to be done manually. The requirement for adjustment may result in additional consumption of computing resources, human resources, energy, and time. Therefore, improved knowledge distillation techniques are needed.
[0016] Embodiments of the present disclosure can improve at least some of the above challenges and problems by providing methods and devices that promote knowledge distillation based on dense connections. In various embodiments of the present disclosure, a teacher network is generated based on the structure of a student network. In one example, each layer of part or all of the teacher network ("teacher layer") is generated based on a layer of the student network ("student layer"). The structure of the teacher layer may be a mirror image of the structure of the corresponding student layer. For example, the teacher layer may include the same number of processing elements as the student layer, and the arrangement of the processing elements in the two layers may be the same. The generation of the teacher network also includes forming connections ("internal connections") within the teacher network. Internal connections connect from one layer of the teacher network to another layer of the teacher network.
[0017] After forming the teacher network, the teacher network and the student network are merged, which forms a merged network. The merging of the two networks may include forming a connection (cross-network connection) between the two networks. The cross-network connection is connected from a layer of the teacher network to a layer of the student network. The connection (e.g., an internal connection or a cross-network connection) can facilitate data transfer between the two layers (e.g., the transfer of features). In addition, the merged network is trained. For example, a training sample is provided to the merged network, for example, to both the teacher network and the student network. Each network provides an output. The output of the training sample and the ground truth label can be used to adjust the parameters of the merged network based on a loss function. After the merged network is trained, the student network can be separated from the teacher network and used in one or more applications.
[0018] Unlike traditional knowledge extraction techniques, the present disclosure provides a technique for promoting knowledge extraction by merging connections in the networks of the teacher network and the student network. This knowledge extraction technique is suitable for training DNNs for various applications, such as image classification, face recognition, action recognition, person re-identification, machine translation, and voice recognition, etc. In addition, compared with traditional knowledge extraction techniques, the techniques in the present disclosure are more user-friendly. It enables users to develop teacher networks using student networks. Users do not have to define teacher networks in advance. In addition, it allows users to train student networks and teacher networks simultaneously, for example, through one training process.
[0019] In addition, the present disclosure provides a better accuracy-efficiency trade-off. Since the student network and the teacher network are trained together, the training cost can be significantly reduced. For some applications, the training cost can be several times lower than the training cost of traditional knowledge extraction techniques. The reduction in training cost does not sacrifice the accuracy of the network. On the contrary, the accuracy of the network trained by the technology in the present disclosure can be better than the accuracy of the network trained by many traditional knowledge extraction techniques. Through the present disclosure, computationally intensive DNNs can be converted into lighter-weight DNNs with similar accuracy. The present disclosure can also achieve the replacement of deep sequential processing with parallel distributed processing. This type of structural conversion can utilize general-purpose processors (GPPs), such as multi-core CPUs (central processing units) and GPUs (image processing units), to promote the acceleration of DNN training and reasoning. By utilizing additional approximations, further benefits can be achieved given the flexibility of custom hardware.
[0020] For purposes of illustration, specific numbers, materials, and configurations are described to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be implemented without the specific details, or / and may be implemented with only some of the described aspects. In other cases, well-known features are omitted or simplified so as not to obscure the illustrative implementations.
[0021] In addition, reference is made to the accompanying drawings, which form a part of this document, in which embodiments that can be implemented are illustrated in a graphical manner. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be understood in a limiting sense.
[0022] Various operations may be described in turn as a plurality of discrete actions or operations in a manner that is most helpful for understanding the claimed subject matter. However, the order of description should not be interpreted as meaning that these operations must be dependent on the order. Specifically, these operations may not be performed in the order presented. The operations described may be performed in an order different from the described embodiments. Various additional operations may be performed or the operations described may be omitted in additional embodiments.
[0023] For purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). The term "between" when used in reference to a measurement range includes both ends of the measurement range.
[0024] The description uses the phrases "in one embodiment" or "in an embodiment", each of which refers to one or more identical or different embodiments. The terms "including", "comprising", "having", etc., used for the embodiments of the present disclosure, are synonymous. The present disclosure may use perspective-based descriptions, such as "above", "below", "top", "bottom", and "side", to illustrate various features of the drawings, but these terms are only for ease of discussion and do not imply a desired or required orientation. The drawings are not necessarily drawn to scale. Unless otherwise indicated, the use of ordinal adjectives "first", "second", and "third", etc. to describe common objects simply indicates that different instances of similar objects are referenced, and is not intended to imply that the objects described in this way must be in a given sequence in time, space, ranking, or in any other way.
[0025] In the detailed description that follows, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.
[0026] The terms "substantially," "close," "roughly," "near," and "approximately" generally refer to input operating objects based on specific values as described herein or as known in the art, within + / -20% of target values. Similarly, terms indicating the orientation of various elements, such as "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements, generally refer to input operating objects based on specific values as described herein or as known in the art, within + / -5-20% of target values.
[0027] Furthermore, the terms "includes," "comprising," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such method, process, apparatus, or DNN accelerator. Additionally, the term "or" refers to an inclusive "or" rather than an exclusive "or."
[0028] The DNN systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is entirely responsible for all of the desirable properties disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the following description and accompanying drawings.
[0029] Example DNN layer structure
[0030] Figure 1 An example layer structure of DNN 100 is illustrated according to various embodiments. For illustration, Figure 1The DNN 100 in FIG. 1 is a convolutional neural network (CNN). In other embodiments, the DNN 100 may be another type of DNN. The DNN 100 is trained to receive an image and output a classification of an object in the image. Figure 1 In an embodiment of the present invention, DNN 100 receives an input image 105 including objects 115, 125, and 135. DNN 100 includes a sequence of layers including a plurality of convolutional layers 110 (individually referred to as "convolutional layers 110"), a plurality of pooling layers 120 (individually referred to as "pooling layers 120"), and a plurality of fully connected layers 130 (individually referred to as "fully connected layers 130"). In other embodiments, DNN 100 may include fewer, more, or different layers.
[0031] The convolutional layer 110 summarizes the presence of features in the input image 105. Figure 1 In an embodiment of the present invention, the first layer of the DNN 100 is a convolutional layer 110. The convolutional layer 110 functions as a feature extractor. The convolutional layer 110 may receive an input and output features extracted from the input. In one example, the convolutional layer 110 performs a convolution on the IFM (input feature map) 140 using the filter 150, generates an output OFM (output feature map) 160 from the convolution, and passes the OFM 160 to the next layer in the sequence. The IFM 140 may include multiple IFM matrices. The filter 150 may include multiple weight matrices. The OFM 160 may include multiple OFM matrices. For the first convolutional layer 110, which is also the first layer of the DNN 100, the IFM 140 is the input image 105. For other convolutional layers, the IFM 140 may be the output of another convolutional layer 110 or the output of the pooling layer 120.
[0032] Convolution can be a linear operation that involves multiplying a weight operation object in filter 150 with a patch of the size of the weight operation object of IFM 140. The weight operation object can be a weight matrix in filter 150, such as a 2-dimensional array of weights, where the weights are arranged in columns and rows. The weights can be initialized and updated by backpropagation using gradient descent. The magnitude of the weights can indicate the importance of filter 150 in extracting features from IFM 140. The weight operation object can be smaller than IFM 140. Multiplication can be an element-wise multiplication between a patch of the size of the weight operation object of IFM 140 and the corresponding weight operation object, and then added, always producing a single value. Because its result is a single value, this operation is often called a "scalar product".
[0033] In some embodiments, using a weight operation object that is smaller than the IFM 140 is intentional because it allows the same weight operation object (a set of weights) to be multiplied with the IFM 140 multiple times at different points on the IFM 140. Specifically, the weight operation object is systematically applied to each overlapping portion of the IFM 140 or patch of the weight operation object size, from left to right and from top to bottom. The result of multiplying the weight operation object with the IFM 140 once is a single value. Since the weight operation object is applied to the IFM 140 multiple times, the result of the multiplication is a two-dimensional array representing the output values of the weight operation of the IFM 140. Therefore, the 2-dimensional output array from this operation is called a "feature map."
[0034] In some embodiments, OFM 160 is passed through an activation function. An example activation function is a rectified linear activation function (ReLU). ReLU is a calculation that directly returns the value provided as an input, or returns a value of zero if the input is zero or less. Convolutional layer 110 can receive several images as input and calculate the convolution of each of them with each weight operation object. This process can be repeated several times. For example, OFM 160 is passed to a subsequent convolutional layer 110 (i.e., a convolutional layer 110 after the convolutional layer 110 that generates OFM 160 in the sequence). The subsequent convolutional layer 110 performs convolution on OFM 160 with the new weight operation object and generates a new feature map. The new feature map can also be normalized and resized. The new feature map can be weighted again by a further subsequent convolutional layer 110, and so on and so forth.
[0035] In some embodiments, the convolution layer 110 has four hyperparameters: the number of weight operation objects, the size F of the weight operation object (e.g., the size of the weight operation object is F×F×D pixels), the step size S of dragging the window corresponding to the weight operation object on the image (e.g., a step size of one means moving the window one pixel at a time), and zero padding P (e.g., adding a black outline of P pixels thick to the input image of the convolution layer 110). The convolution layer 110 can perform various types of convolutions, such as 2-dimensional convolutions, dilated or hollow convolutions, spatially separable convolutions, depthwise separable convolutions, transposed convolutions, and the like. The DNN 100 includes 16 convolution layers 110. In other embodiments, the DNN 100 may include a different number of convolution layers.
[0036] The pooling layer 120 downsamples the feature map generated by the convolution layer, for example, by summarizing the presence of features in patches of the feature map. The pooling layer 120 is placed between two convolutional layers 110: the preceding convolutional layer 110 (the convolutional layer 110 before the pooling layer 120 in the sequence of layers) and the subsequent convolutional layer 110 (the convolutional layer 110 after the pooling layer 120 in the sequence of layers). In some embodiments, for example, after an activation function (e.g., ReLU) is applied to the OFM 160, the pooling layer 120 is added after the convolutional layer 110.
[0037] The pooling layer 120 receives the feature map generated by the previous convolutional layer 110 and applies a pooling operation to the feature map. The pooling operation reduces the size of the feature map while retaining their important characteristics. Therefore, the pooling operation improves the efficiency of the DNN and avoids over-learning. The pooling layer 120 can perform a pooling operation by average pooling (calculating the average value for each patch on the feature map), maximum pooling (calculating the maximum value for each patch of the feature map), or a combination of the two. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is 2×2 pixels applied with a span of two pixels, so that the pooling operation reduces the size of the feature map by 2 times, for example, the number of pixels or values in the feature map is reduced to a quarter of the size. In one example, the pooling layer 120 applied to a 6×6 feature map results in a 3×3 output pooled feature map. The output of the pooling layer 120 is input into the subsequent convolutional layer 110 to further extract features. In some embodiments, the pooling layer 120 operates on each feature map separately to create a new set of pooled feature maps of the same number.
[0038] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 can be convolutional or non-convolutional. The fully connected layer 130 receives an input operation object. The input operation object defines the output of the convolution layer 110 and the pooling layer 120, and includes the value of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies linear combinations and activation functions to the input operation objects and generates individual partial sums. The individual partial sums can contain as many elements as categories: element i represents the probability that the image belongs to category i. Therefore, each element is between 0 and 1, and the total value of all is one. These probabilities are calculated by the last fully connected layer 130 using a logistic function (binary classification) or a softmax function (multi-class classification) as an activation function.
[0039] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. Figure 1In the embodiment of , N is equal to 3 because there are three objects 115, 125 and 135 in the input image. Each element of the operation object indicates the probability that the input image 105 belongs to a certain category. To calculate the probability, the fully connected layer 130 multiplies each input element by the weight, obtains the sum, and then applies an activation function (e.g., if N=2, it is logical, if N>2, it is softmax). This is equivalent to multiplying the input operation object with the matrix containing the weights. In one example, the individual partial sums include three probabilities: the first probability indicates that the object 115 is a tree, the second probability indicates that the object 125 is a car, and the third probability indicates that the object 135 is a person. In other embodiments where the input image 105 includes different objects or a different number of objects, the individual partial sums can be different.
[0040] Example DNN System
[0041] Figure 2 is a block diagram of a DNN system 200 according to various embodiments. The DNN system 200 trains a DNN by using densely connected based knowledge distillation. The DNN can be used to perform one or more machine learning tasks. A machine learning task is a task of performing reasoning. Reasoning is the process of running available data into a DNN to generate an output, and the output provides a solution to the question or query posed. An example of an output is one or more numerical scores that can indicate the probability that an object in an image belongs to a certain category. The DNN system 200 can train DNNs that can be used to solve various problems, such as image classification, learning relationships between biological cells (e.g., DNA, proteins, etc.), controlling the behavior of devices (e.g., robots, machines, etc.), and the like.
[0042] The DNN system 200 includes an interface module 210, a training set generator 220, a student network generator 230, a teacher network generator 240, a merging module 250, a training module 260, and a verification module 270. In other embodiments, alternative configurations, different or additional components may be included in the DNN system 200. In addition, the functions attributed to the components of the DNN system 200 may be performed by different components included in the DNN system 200 or by different systems.
[0043] The interface module 210 facilitates communication between the DNN system 200 and other systems. For example, the interface module 210 establishes communication between the DNN system 200 and an external database to receive data that can be used to train the DNN or data that can be input into the DNN to perform a machine learning task. As another example, the interface module 210 supports the DNN system 200 to distribute the DNN to other systems, such as computing devices configured to apply the DNN to perform tasks. The computing device can be an edge device, a client device, and the like.
[0044] The training set generator 220 forms a training data set to be used to train the DNN. The training data set includes training samples and ground truth labels. The training data set may include one or more ground truth labels for each training sample. The ground truth labels of the training samples may be known or verified labels that answer the questions or queries that the DNN will use to answer. In the example of training the DNN to recognize objects in images, the training data set includes training images and ground truth labels indicating the classification of the objects in the training images. The ground truth labels in the example may be numbers indicating the probability that the object belongs to a certain category. The object may be associated with other ground truth labels that indicate the probability that the object belongs to other categories.
[0045] In some embodiments, the training set generator 220 may also form a validation data set for validating the performance of the trained DNN by the validation module 270. The validation data set may include validation samples and ground truth labels for the validation samples. The validation data set for the DNN may include different samples from the training data set used to train the DNN. In one embodiment, a portion of the training data set may be used to initially train the DNN, while the remainder of the training data set may be retained as a validation subset for the validation module 270 to validate the performance of the trained DNN. The portion of the training data set that does not include the validation subset may be used to train the DNN.
[0046] The student network generator 230 generates a student network. A student network is a DNN that can be used to perform machine learning tasks after training. The student network generator 230 can generate a student network based on parameters that define the architecture of the DNN. Examples of parameters include the number of layers, the type of layers, the sequence of layers, the number of processing elements (PEs) in a layer, the type of PEs, the arrangement of PEs in a layer (e.g., interconnections between PEs, the number of columns in a PE array, the number of rows in a PE array, etc.), activation functions, pooling functions, or other types of parameters. The processing elements perform MAC operations.
[0047] In some embodiments, the student network generator 230 determines some or all of the parameters, for example, based on the question or query that the DNN is to answer, the resources available for training, the resources available for reasoning, some other factors that may be critical to the architecture of the DNN, or some combination of these. In other embodiments, the student network generator 230 may receive some or all of the parameters from a different system (e.g., from a computing device that will run the DNN for reasoning, a system that manages such a computing device, etc.) or from a user (e.g., through a user interface that allows the user to provide information for the DNN).
[0048] The architecture of a DNN includes an input layer, an output layer, and multiple hidden layers. The input layer of a DNN may include tensors (e.g., multidimensional arrays) that specify attributes of an input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits that specify the color of a pixel in the input image). The output layer includes labels of objects in the input layer. The hidden layer is a layer between the input layer and the output layer. The hidden layer includes one or more convolutional layers and one or more other types of layers, such as rectified linear unit (ReLU) layers, pooling layers, fully connected layers, regularization layers, softmax or logical layers, and the like. The convolutional layer of a DNN abstracts the input image into a feature map represented by a tensor that specifies the feature map height, feature map width, and feature map channels (e.g., red, green, and blue images include three channels). The pooling layer is used to reduce the spatial volume of the input image after convolution. It is used between two convolutional layers. The fully connected layer involves weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories through training. An example DNN is the above combined with Figure 1 Described DNN 100.
[0049] The teacher network generator 240 generates a teacher network based on the student network. The teacher network will be used to train the student network through knowledge extraction. The teacher network can be a DNN. In some embodiments, the teacher network generator 240 determines the structure of the teacher network based on the structure of the student network. For example, the teacher network generator 240 can generate a teacher network including the same number and / or type of layers as the student network. The arrangement of the layers ("teacher layers") in the teacher network can be the same as the arrangement of the layers ("student layers") in the student network. In addition, for individual teacher layers, the teacher network generator 240 can design the teacher layer based on the corresponding student layer. The teacher network generator 240 can make the teacher layer mirror the student layer. For example, the teacher layer can have the same number and / or type of PEs as the student layer. The arrangement of PEs in the two layers can also be the same. Since the teacher network can be automatically generated based on the student network, it is not necessary to have a well-defined teacher network available in advance. In addition, the generation of the teacher network does not introduce additional parameters. The dimensions of the IFM and OFM of the teacher layer and the student layer can be the same, which can facilitate feature transfer and knowledge extraction during training, which will be described below.
[0050] The teacher network generator 240 also generates internal connections within the teacher network. The internal connection can connect two teacher layers, for example, from a first teacher layer to a second teacher layer. The second teacher layer can be arranged after the first layer in the teacher network. The internal connection facilitates data transfer between the two teacher layers. For example, the first layer can send features (e.g., OFM 160) to the second layer via the internal connection. The second layer receives the features and can aggregate the features from the first layer with the features generated in the second layer to output the aggregated features. The internal connection can be bidirectional, for example, the second layer can also send data to the first layer.
[0051] In some embodiments, the teacher network generator 240 forms multiple internal connections for the teacher layer. In one example, the teacher network generator 240 identifies one or more layers that are located after the target layer. For each layer in the one or more layers, the teacher network generator 240 forms an internal connection from the target layer to the corresponding layer. The one or more layers can be of the same type as the target layer. For example, the one or more layers and the target layer are convolutional layers. The teacher layer can receive data (e.g., features) from multiple other teacher layers and can aggregate the data with data generated in the teacher layer itself.
[0052] The merging module 250 merges the student network and the teacher network, for example, the teacher network generated based on the student network. The merging module 250 can receive the student network from the student network generator 230, and receive the teacher network from the teacher network generator 240. In some embodiments, the merging module 250 builds the cross-network connection between the student network and the teacher network during the merging process. Data can be transmitted between the two networks by the cross-network connection. In one embodiment, the merging module 250 builds the cross-network connection from the student network to the teacher network. The cross-network connection can be connected to the teacher layer from the student layer, and allows the student layer to send data to the teacher layer. In other embodiments, the cross-network connection can be bidirectional, for example, the teacher layer can also send data to the student layer.
[0053] The merging module 250 can form multiple cross-network connections for the student layer. In one example, the teacher network generator 240 identifies one or more teacher layers for the student layer. One or more teacher layers can include a teacher layer corresponding to the student layer (e.g., a teacher layer generated based on the student layer, a position of the teacher layer in the teacher network matches the position of the student layer in the student network, etc.). One or more teacher layers can also include an additional teacher layer after the teacher layer in the teacher network. In one example, for the student layer, the teacher network generator 240 identifies the corresponding teacher layer and one or more other teacher layers after the corresponding teacher layer. The identified teacher layers can be of the same type, for example, they are all convolutional layers. In addition, the identified teacher layers can be of the same type as the student layer. For each identified teacher layer, the merging module 250 can form a cross-network connection from the student layer to the identified teacher layer.
[0054] A teacher layer that receives data (e.g., features) from a student layer can aggregate that data with data generated in the teacher layer itself. A teacher layer can be connected to multiple student layers and receive data from some or all of these student layers. A student layer can be connected to multiple teacher layers and send data to some or all of these teacher layers. The merging of the two networks produces a merged network that can be trained as a whole, for example, like training a single network. Since the student network is merged with the teacher network during training, the two networks can share with each other the "knowledge" they learned during training. Due to the dense backward gradient flow from the teacher network to the student network, the feature similarity between the teacher network and the student network can be enhanced. During the training of the merged model, the knowledge learned by the teacher network can be transmitted to the student network through dense teacher-to-student gradient propagation. The merge can promote knowledge distillation without requiring two training processes (e.g., one for training the teacher network and the other for training the student network) required by many other knowledge distillation techniques. In addition, the need to design feature distillation losses and adjust weight factors to balance loss terms can be avoided.
[0055] The training module 260 trains a DNN, such as the merged network provided by the merge module 250. The training module 260 may also receive a training data set from the training set generator 220. The training module 260 may also determine hyperparameters for the training process. Hyperparameters may be different from parameters (e.g., weights) inside the network. In some embodiments, hyperparameters include variables that determine how to train the DNN, such as batch size, number of rounds, and the like. The batch size defines the number of training samples to be worked through before updating the parameters of the DNN. The batch size is the same as or smaller than the number of samples in the training data set. The training data set may be divided into one or more batches. The number of rounds defines how many times the entire training data set is passed forward and backward throughout the network. The number of rounds defines the number of times the DL algorithm works through the entire training data set. One round means that each training sample in the training data set has a chance to update the parameters inside the network. A round may include one or more batches. The number of rounds may be 10, 100, 500, 1000, or even larger.
[0056] The training module 260 sends the training samples in the training data set to the merged network, for example, the training module 260 inputs the training samples into both the teacher network and the student network. The training module 260 modifies the parameters inside the merged network (for example, the weights of the convolutional layers of the teacher network, the student network, or both) to minimize the error between the labels of the training samples generated by the merged network ("generated labels") and the ground truth labels in the data set. The generated labels may include one or more labels generated by the student network and one or more labels generated by the teacher network. The training module 260 can use a loss function, for example, a cross entropy loss function, to minimize the error.
[0057] After the threshold condition is met, the training module 260 can stop adjusting the parameters in the merged network. The threshold condition can be a condition of completing a predetermined number of rounds, satisfying the target performance (e.g., accuracy) of the merged network, the student network, or the teacher network, or other types of conditions. The network with updated parameters is referred to as a trained network. After the merged network is trained, the training module 260 separates the student network from the teacher network, for example, by removing cross-network connections. The student network can be used to process machine learning tasks. In some embodiments, the student network or the parameters of the student network can be sent to another system or device (e.g., edge device, client device, etc.) for reasoning.
[0058] The verification module 270 verifies the performance (e.g., accuracy) of a trained DNN, such as a trained student network separated from its corresponding teacher network. The verification module 270 may determine the accuracy of the trained student network and determine whether the accuracy meets a threshold (e.g., a requirement for model accuracy). In response to determining that the accuracy of the student network meets the threshold, the verification module 270 may deploy the student network to another system or device, for example, through the interface module 210. In some embodiments, the verification module 270 may also verify the performance of the merged network or the teacher network. For example, the verification module 270 determines whether the accuracy of the merged network meets the threshold. In response to determining that the accuracy does not meet the threshold, the verification module 270 may instruct the training module 260 to further train the merged network. In response to determining that the accuracy meets the threshold, the verification module 270 may notify the training module 260 that the merged network has been fully trained, or instruct the training module 260 to separate the student network from the teacher network.
[0059] In some embodiments, the verification module 270 inputs samples in the verification data set into the DNN and uses the output of the DNN to determine the model accuracy. In some embodiments, the verification data set may be formed by some or all samples in the training data set. Additionally or alternatively, the verification data set includes additional samples in addition to the samples in the training set. In some embodiments, the verification module 270 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. The verification module 270 may use the following metrics to determine the accuracy score: precision = TP / (TP+FP) and recall = TP / (TP+FN), where precision may be the number of correctly predicted (TP or true positives) by the reference classification model in the total number of its predictions (TP+FP or false positives), and recall may be the number of correctly predicted (TP) by the reference classification model in the total number of objects that do have the relevant attributes (TP+FN or false negatives). F-score (F-score = 2*PR / (P+R)) unifies precision and recall into a single metric.
[0060] Example teacher network formed from the student network
[0061] Figure 3An example teacher network 320 formed based on the student network 310 is illustrated according to various embodiments. The student network 310 may be provided by the student network generator 230. The formation of the teacher network 320 may be performed by the teacher network generator 240. For simplicity and for purposes of illustration, the student network 310 includes four layers 315A-315D (collectively referred to as "student layers 315"), and the teacher network 320 includes four layers 325A-325D (collectively referred to as "teacher layers 325"). The student layers 315 and the teacher layers 325 may be convolutional layers, self-attention layers, linear layers, or some combination of these. In other embodiments, the student network 310 or the teacher network 320 may include more, fewer, or different layers. For example, the student network 310 or the teacher network 320 may include layers arranged in Figure 3 Additional layers between, before, or after the layers shown.
[0062] exist Figure 3 In an embodiment, the teacher layer 325 is aligned with the student layer 315. Each teacher layer 325 corresponds to one student layer 315. For example, the teacher layer 325A corresponds to the student layer 315A, the teacher layer 325B corresponds to the student layer 315B, the teacher layer 325C corresponds to the student layer 315C, and the teacher layer 325D corresponds to the student layer 315D. The teacher layer 325 can be generated based on its corresponding student layer 315. For example, the teacher layer 325 can have the same structural units as the student layer 315. The arrangement of the structural units in the teacher layer 325 can be the same as that in the student layer 315. The structural units may include PEs that perform MAC operations, data storage units (e.g., memories, register files, etc.), or other types of units.
[0063] Internal connections 327 (individually referred to as "internal connections 327") between teacher layers 325 are added to the teacher network 320 to form a tightly connected architecture. A teacher layer 325 can be connected to each other teacher layer 325 in the teacher network 320. For illustration, the teacher network 320 includes six internal connections 327: one from teacher layer 325A to teacher layer 325B, one from teacher layer 325A to teacher layer 325C, one from teacher layer 325A to teacher layer 325D, one from layer 325B to layer 325C, one from teacher layer 325B to teacher layer 325D, and one from teacher layer 325C to teacher layer 325D. In other embodiments, the teacher network 320 may have more, fewer, or different internal connections.
[0064] The internal connection 327 facilitates data transfer between two teacher layers 325 (a previous teacher layer 325 and a subsequent teacher layer 325), where the subsequent teacher layer 325 is the layer arranged after the previous teacher layer 325 in the teacher network 320. There may be zero, one, or more other teacher layers 325 between the previous teacher layer 325 and the subsequent teacher layer 325. Alternatively, the internal connection 327 may be bidirectional so that data can be transferred from the previous teacher layer 325 to the subsequent teacher layer 325, or from the subsequent teacher layer 325 to the previous teacher layer 325.
[0065] The teacher layer 325 may receive features from one or more other teacher layers 325 via internal connections 327. After receiving the features, the teacher layer 325 may aggregate the features with the features generated in the teacher layer 325 itself. The features from the teacher layer 325 may be an aggregation of the features generated in the teacher layer 325 and the features received by the teacher layer 325 from one or more other teacher layers 325. For example, the teacher layer 325D may generate OFMs by itself, for example, through MAC operations of PEs in the teacher layer 325D. The teacher layer 325D also receives features from the teacher layers 324A-325C: the teacher layer 325D may receive OFMs from the teacher layer 325A, aggregated feature maps (AFMs) from the teacher layer 325B, and AFMs from the teacher layer 325C. The AFMs from the teacher layer 325B may be the result of aggregating the OFMs from the teacher layer 325A and the OFMs generated in the teacher layer 325B. Similarly, the AFM from teacher layer 325C may be the result of aggregating the AFM from teacher layer 325B and the OFM generated in teacher layer 325C. In some embodiments, teacher layers 325 have the same feature dimensions, e.g., the same OFM dimensions. This can enable internal connections to be applied to any layer of teacher network 320, and the generation of teacher network 320 can be parameter-free.
[0066] Given L layers and their corresponding output features AFM of the Lth layer (L>1) of the stack can be calculated as:
[0067]
[0068] in, is the OFM generated in the Lth teacher layer 325 (eg, the Lth teacher layer 325 applies a filter to the IFM and generates the OFM), and is the AFM received from the i-th teacher layer 325 (i.e., through the internal connection 327 from the i-th teacher layer 325 to the L-th teacher layer 325). For a teacher layer 325A that does not receive features from other teacher layers 325:
[0069]
[0070] Example Merge Network
[0071] Figure 4 According to various embodiments, the student network 310 is illustrated as being merged with the teacher network 320. The merge can be performed by Figure 2 The merging module 250 in is completed. Figure 4 As shown, a series of cross-network connections 410 (individually referred to as "cross-network connections 410") are formed between the student network 310 and the teacher network 320. The cross-network connections 410 connect the student network 310 to the teacher network 320 and can facilitate data transmission from the student network 310 to the teacher network 320. Figure 4 As shown, each cross-network connection 410 provides a data transmission path from the student layer 315 to the teacher layer 325. The student layer 315 is connected to its corresponding teacher layer 325 and each teacher layer 325 after its corresponding teacher layer 325. For example, the student layer 315A is connected to the teacher layer 325A corresponding to the student layer 315A, and is connected to the teacher layers 325B-325D, which are the teacher layers 325 after the teacher layer 325A. Similarly, the student layer 315B is connected to the teacher layers 325B-325D, but not to the teacher layer 325A. The student layer 315C is connected to the teacher layers 325C and 325D. The student layer 315C is connected to the teacher layer 325D, but not to the other teacher layers 325A-325C.
[0072] The teacher layer 325 may receive the OFM of the student layer 315 via the cross-network connection 410A between the teacher layer 325 and the student layer 315. As described above, the teacher layer 325 may also receive (one or more) AFMs (or OFMs) from other teacher layers 325 via the internal connection 327. In addition, the teacher layer 325 itself may generate an OFM. The teacher layer 325 may aggregate its own OFM, the AFMs (or OFMs) received from other teacher layers 325, and the OFMs received from the student layer 315, and generate an AFM of the teacher layer 325. The AFM may be transmitted to another teacher layer 325, for example, a subsequent teacher layer 325, and the subsequent teacher layer 325 may perform further aggregation of features.
[0073] The AFM of the Lth teacher layer 325 (L>1) after merging can be calculated as:
[0074]
[0075] in, It is the OFM of the Lth teacher level 325, is the AFM of the ith teacher level received via internal connection 327, and is the OFM of the L-th student layer 315 received via the cross-network connection 410. With the cross-network connection 410, feature similarity (e.g., layer-level feature similarity) between the student network 310 and the teacher network 320 is enhanced, e.g., due to the dense backward gradient flow from the teacher network 320 to the student network 310.
[0076] Example training of the merged network
[0077] Figure 5 According to various embodiments, the training of the merged network is illustrated. The merged network includes a student network 310, a teacher network 320, and cross-network connections 410. The training can be performed by Figure 2 The training module 260 in is completed. Figure 5 As shown, training samples 510 are input into student network 310 and teacher network 320. Student network 310 generates output 515. Teacher network 320 generates output 525. In some embodiments, output 515 or 525 can be a prediction, classification, determination, or other type of solution to a problem or question. Parameters in the combined network are adjusted during training to optimize outputs 515 and 525 by minimizing the error in outputs 515 and 525. The error can be determined by comparing outputs 515 and 525 to ground truth labels 520 of training samples 510. Ground truth labels 520 and training samples 510 can be included in a network composed of Figure 2 The training data set is formed by the training set generator 220 in .
[0078] A loss function, such as a cross entropy loss function, can be used to adjust parameters in the merged network. The training module 260 can minimize the error based on a joint optimization objective, such as an objective for optimizing both the output of the teacher network and the output of the student network. An example of a joint optimization objective can be represented by the following algorithm:
[0079] L=L CE (θ S ,x)+L CE ({θ S ,θ T},x)
[0080] Among them, x is the training data, θ S represents the parameters of the student network, θ T represents the parameters of the teacher network, L CEis the cross entropy loss calculated on the training data x, and L represents the joint loss of the merged network. The training module 260 can adjust the parameters (e.g., θ S and θ T ), to minimize the joint loss L.
[0081] In other embodiments, the training module 260 can facilitate bidirectional refinement, for example, the student network can benefit from knowledge refined from the teacher network, and the teacher network can benefit from knowledge refined from the student network. The optimization objective of training with bidirectional refinement can be defined as:
[0082]
[0083] Among them, x is the training data, θ S represents the parameters of the student network, θ T represents the parameters of the teacher network, L is the joint loss in the above algorithm, p represents the probability distribution, and D kl represents the Kullback-Leibler divergence, and represents the joint loss of the bidirectional refinement training. The training module 260 can adjust the parameters (e.g., θ S and θ T ), to minimize the joint loss
[0084] Example DL Environment
[0085] Figure 6 A DL environment 600 according to various embodiments is illustrated. The DL environment 600 includes a DL server 610 and a plurality of client devices 620 (individually referred to as client devices 620). The DL server 610 is connected to the client devices 620 via a network 630. In other embodiments, the DL environment 600 may include fewer, more, or different components.
[0086] DL server 610 uses neural network to train DL model. The structure of neural network is like human brain, and is composed of artificial neurons, also known as nodes. These nodes are stacked on each other in three types of layers: input layer, (one or more) hidden layer, and output layer. Data provides information to each node in the form of input. The node multiplies the input with random weights, performs calculations, and adds biases. Finally, a nonlinear function, also known as an activation function, is applied to determine which neuron to start. DL server 610 can use various types of neural networks, such as DNN, recurrent neural network (RNN), generative adversarial network (GAN), long short-term memory network (LSTMN), etc. During the process of training DL model, neural network uses unknown elements in input distribution to extract features, group objects, and find useful data patterns. DL model can be used to solve various problems, such as making predictions, classifying images, etc. DL server 610 can build DL model specifically for specific types of problems that need to be solved. DL model is trained to receive input and output solutions to specific problems.
[0087] exist Figure 6 In the example, DL server 610 includes DNN system 640, database 650, and distributor 660. DNN system 640 trains DNN. DNN can be used to process images, such as images captured by autonomous vehicles, medical equipment, satellites, etc. In one embodiment, DNN receives input images and outputs classifications of objects in the input images. Examples of DNN are described above in conjunction with Figure 1 DNN 100 described above or in combination Figure 3-Figure 5 The student network 310 described above. In some embodiments, the DNN system 640 trains the DNN by knowledge distillation (e.g., knowledge distillation based on dense connections). The trained DNN can be used on low-memory systems, such as mobile phones, IOT edge devices, etc. An embodiment of the DNN system 640 is the above combined Figure 2 Described DNN system 200.
[0088] The database 650 stores data received, used, generated, or otherwise associated with the DL server 610. For example, the database 650 stores a training data set used by the DNN system 640 to train the DNN. In one embodiment, the training data set is an image library that can be used to train a DNN for classifying images. The training data set may include data received from the client device 620. As another example, the database 650 stores hyperparameters of a neural network constructed by the DL server 610.
[0089] The distributor 660 distributes the DL model generated by the DL server 610 to the client device 620. In some embodiments, the distributor 660 receives a request for a DNN from the client device 620 via the network 630. The request may include a description of a problem that the client device 620 needs to solve. The request may also include information about the client device 620, such as information describing the available computing resources on the client device. The information describing the available computing resources on the client device 620 may be information indicating the network bandwidth, information indicating the available memory size, information indicating the processing power of the client device 620, and the like. In one embodiment, the distributor may instruct the DNN system 640 to generate a DNN in accordance with the request. The DNN system 640 may generate a DNN based on the information in the request. For example, the DNN system 640 may determine the structure of the DNN and / or train the DNN according to the request.
[0090] In another embodiment, the distributor 660 may select a DNN from a set of pre-existing DNNs based on the request. The distributor 660 may select a DNN for a particular client device 630 based on the size of the DNN and the available resources of the client device 630. In an embodiment where the distributor 660 determines that the memory or processing power of the client device 630 is limited, the distributor 660 may select a compressed DNN for the client device 630 instead of an uncompressed DNN with a larger size. The distributor 660 then transmits the DNN generated or selected for the client device 620 to the client device 620.
[0091] In some embodiments, the distributor 660 can receive feedback from the client device 620. For example, the distributor 660 receives new training data from the client device 620 and can send the new training data to the DNN system 640 to further train the DNN. As another example, the feedback includes an update of available computer resources on the client device 620. The distributor 660 can send a different DNN to the client device 620 based on the update. For example, after receiving feedback indicating that the computing resources of the client device 620 have been reduced, the distributor 660 sends a smaller DNN to the client device 620.
[0092] The client device 620 receives the DNN from the distributor 660 and applies the DNN to perform machine learning tasks, for example, to solve a problem or answer a question. In various embodiments, the client device 620 inputs an image into the DNN and uses the output of the DNN for various applications, for example, visual reconstruction, augmented reality, robot positioning and navigation, medical diagnosis, weather prediction, and the like. The client device 620 may be one or more computing devices that are capable of receiving user input and sending and / or receiving data via the network 630. In one embodiment, the client device 620 is a traditional computer system, such as a desktop or laptop computer. Alternatively, the client device 620 may be a device with computer functionality, such as a personal digital assistant (PDA), a mobile phone, a smart phone, an autonomous vehicle, or other appropriate device. The client device 620 is configured to communicate via the network 630. In one embodiment, the client device 620 executes an application that allows a user of the client device 620 to interact with the DL server 610 (e.g., the distributor 660 of the DL server 610). The client device 620 may request the DNN or send feedback to the distributor 660 through the application. For example, the client device 620 executes a browser application to enable interaction between the client device 620 and the DL server 610 via the network 630. In another embodiment, the client device 620 executes a browser application on the native operating system of the client device 620 (e.g. or ANDROID TM ) to interact with the DL server 610.
[0093] In one embodiment, client device 620 is an integrated computing device, which operates as an independent device with network capabilities. For example, client device 620 includes a display, a speaker, a microphone, a camera, and an input device. In another embodiment, client device 620 is a computing device for coupling to an external media device, such as a television or other external display and / or an audio output system. In this embodiment, client device 620 can be coupled to an external media device via a wireless interface or a wired interface (e.g., an HDMI cable), and various functions of the external media device can be utilized, such as its display, speaker, microphone, camera, and input device. Here, client device 620 can be configured to be compatible with a general external media device, which does not have specialized software, firmware, or hardware specifically for interacting with client device 620.
[0094] The network 630 supports communication between the DL server 610 and the client device 620. The network 630 may include any combination of a local area network and / or a wide area network, using wired and / or wireless communication systems. In one embodiment, the network 630 may use standard communication technologies and / or protocols. For example, the network 630 may include communication links using technologies such as Ethernet, 8010.11, worldwide interoperability for microwave access (WiMAX), 3G, 4G, code division multiple access (CDMA), digital subscriber line (DSL), etc. Examples of networking protocols for communicating via the network 630 may include multiprotocol label switching (MPLS), transmission control protocol / Internet protocol (TCP / IP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), and file transfer protocol (FTP). Data exchanged through network 630 may be represented in any suitable format, such as hypertext markup language (HTML) or extensible markup language (XML). In some embodiments, all or some of the communication links of network 630 may be encrypted using any one or more suitable techniques.
[0095] Example method for training a neural network
[0096] Figure 7 7 is a flow chart showing a method 700 for training a DNN by densely connected knowledge distillation according to various embodiments. The method 700 may be Figure 2 The DNN system 200 in FIG. Figure 7 The flowchart shown in describes method 700, but many other methods for training DNNs by densely connected knowledge distillation may be used instead. For example, Figure 7 The execution order of the steps in the method can be changed. As another example, some steps can be changed, eliminated or combined.
[0097] The DNN system 200 generates 710 a supporting neural network based on the target neural network. The supporting neural network can be a teacher network, such as Figure 3 The target neural network can be a student network, such as Figure 3 The student network 310 in the example embodiment of the present invention is a student network 310 in the example embodiment of the present invention. The support neural network includes a plurality of support layers. The target neural network includes a plurality of target layers. The DNN system 200 may generate respective corresponding support layers in the plurality of support layers based on respective target layers in the plurality of target layers. Each support layer and each target layer may include the same number of PEs arranged in the same structure. The PEs are configured to perform multiplication-accumulation operations. The plurality of support layers may be aligned with the plurality of target layers.
[0098] The DNN system 200 may also generate internal connections within the supporting neural network. The internal connection is from the first supporting layer to the second supporting layer. The second supporting layer may be configured to receive a first feature map from the first supporting layer, generate a second feature map, and aggregate the first feature map and the second feature map. In some embodiments, the DNN system 200 generates multiple internal connections that connect a supporting layer to multiple other supporting layers. For example, the DNN system 200 generates additional internal connections within the supporting neural network, where the additional internal connection is from the third supporting layer to the second supporting layer. The second supporting layer may receive the first feature map from the first supporting layer, receive the third feature map from the third supporting layer, generate the second feature map, and aggregate the first feature map, the second feature map, and the third feature map.
[0099] The DNN system 200 merges 720 the target neural network and the supporting neural network to form a merged model. Merging the target neural network and the supporting neural network includes establishing 725 a connection between a target layer in the plurality of target layers and a supporting layer in the plurality of supporting layers. The connection is used to transfer data between the target layer and the supporting layer. In some embodiments, the DNN system 200 forms a plurality of connections (e.g., Figure 4 The cross-network connection 410 in the DNN system 200 may be from a supporting layer to a target layer. The supporting layer may receive a first feature map from the target layer, generate a second feature map, and aggregate the first feature map with the second feature map. Alternatively, the connection may be bidirectional. The DNN system 200 may establish another connection between the target layer and another supporting layer in the plurality of supporting layers.
[0100] The DNN system 200 trains 730 the merged model by using the training data set. In some embodiments, the DNN system 200 inputs the training samples in the training data set into the target neural network, and the target neural network generates the target output. The DNN system 200 also inputs the training samples into the support neural network, and the support neural network generates the support output. The DNN system 200 adjusts the parameters of the target neural network and the support neural network based on the ground truth labels in the training data set, the target output, and the support output.
[0101] After the merged model is trained, the DNN system 200 separates the target neural network from the supporting neural network. For example, the DNN system 200 can remove the connection connecting the target neural network to the supporting neural network. In some embodiments, the supporting neural network may be inseparable. After separation, the target neural network can be used to perform machine learning tasks, for example, by running available data through reasoning to solve problems or answer questions.
[0102] Example computing device
[0103] Figure 8 is a block diagram of an example computing device 800 in accordance with various embodiments. Figure 8 Several components are illustrated as being included in computing device 800, but any one or more of these components may be omitted or repeated as appropriate for the application. In some embodiments, some or all of the components included in computing device 800 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-a-chip (SoC) die. Additionally, in various embodiments, computing device 800 may not include Figure 8 800 may include one or more components shown in , but the computing device 800 may include interface circuitry for coupling to the one or more components. For example, the computing device 800 may not include the display device 806, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which the display device 806 may be coupled. In another set of examples, the computing device 800 may not include the audio input device 818 or the audio output device 808, but may include audio input or output device interface circuitry (e.g., a connector and supporting circuitry) to which the audio input device 818 or the audio output device 808 may be coupled.
[0104] The computing device 800 may include a processing device 802 (e.g., one or more processing devices). As used herein, the term "processing device" or "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory to transform the electronic data into other electronic data that can be stored in registers and / or memory. The processing device 802 may include one or more digital signal processors (DSPs), application-specific ICs (ASICs), CPUs, GPUs, cryptographic processors (specialized processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices. The computing device 800 may include a memory 804, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), flash memory, solid-state memory, and / or a hard disk. In some embodiments, the memory 804 may include a memory that shares a die with the processing device 802. In some embodiments, the memory 804 includes one or more non-transitory computer-readable media storing instructions that are executable to perform operations for training a DNN through densely connected knowledge distillation, for example, in conjunction with Figure 7 The method 700 described above or in combination Figure 2 The operations performed by the described DNN system 200. The instructions stored in one or more non-transitory computer-readable media may be executed by the processing device 802.
[0105] In some embodiments, computing device 800 may include a communication chip 812 (e.g., one or more communication chips). For example, communication chip 812 may be configured to manage wireless communications for transmitting data to and from computing device 800. The term "wireless" and its derivatives may be used to describe circuits, devices, DNN accelerators, methods, techniques, communication channels, and the like that can communicate data through a non-solid medium using modulated electromagnetic radiation. The term does not mean that the associated devices do not contain any wires, although in some embodiments they may not.
[0106] The communication chip 812 can implement any of several wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards, including Wi-Fi (IEEE 802.13 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), Long-Term Evolution (LTE) project and any amendments, updates and / or revisions (e.g., Advanced LTE project, ultramobile broadband (UMB) project (also known as "3GPP2"), etc.). Broadband Wireless Access (BWA) networks compatible with IEEE802.16 are generally referred to as WiMAX networks. This abbreviation stands for Worldwide Interoperability for Microwave Access, which is a certification mark for products that have passed the compliance and interoperability tests of the IEEE 802.16 standard. The communication chip 812 can operate according to the Global system for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications system (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA) or LTE network. The communication chip 812 can operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN) or Evolved UTRAN (E-UTRAN).The communication chip 812 may operate according to CDMA, Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO) and its derivatives, and any other wireless protocols designated as 3G, 4G, 5G and above. In other embodiments, the communication chip 812 may operate according to other wireless protocols. The computing device 800 may include an antenna 822 to facilitate wireless communications and / or receive other wireless communications (e.g., AM or FM radio transmissions).
[0107] In some embodiments, the communication chip 812 can manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 812 may include multiple communication chips. For example, the first communication chip 812 may be dedicated to shorter-range wireless communications, such as Wi-Fi or Bluetooth, while the second communication chip 812 may be dedicated to longer-range wireless communications, such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 812 may be dedicated to wireless communications, while the second communication chip 812 may be dedicated to wired communications.
[0108] The computing device 800 may include a battery / power circuit 814. The battery / power circuit 814 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuits for coupling components of the computing device 800 to an energy source separate from the computing device 800 (e.g., AC line power).
[0109] Computing device 800 may include display device 806 (or corresponding interface circuitry, as described above). Display device 806 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.
[0110] Computing device 800 may include audio output device 808 (or corresponding interface circuitry, as described above). Audio output device 808 may include any device that generates audible indications, such as a speaker, headphones, or earbuds.
[0111] Computing device 800 may include audio input device 818 (or corresponding interface circuitry, as described above). Audio input device 818 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital musical instrument (e.g., a musical instrument with a musical instrument digital interface (MIDI) output).
[0112] Computing device 800 may include GPS device 816 (or corresponding interface circuitry, as described above). GPS device 816 may communicate with a satellite-based system and may receive the location of computing device 800, as is known in the art.
[0113] Computing device 800 may include other output devices 813 (or corresponding interface circuitry, as described above). Examples of other output devices 813 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or additional storage devices.
[0114] The computing device 800 may include other input devices 820 (or corresponding interface circuits, as described above). Examples of other input devices 820 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (e.g., a mouse), a stylus, a touchpad, a barcode reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.
[0115] The computing device 800 may have any desired form factor, such as a handheld or mobile computing system (e.g., a cellular phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA, an ultra-mobile personal computer, etc.), a desktop computing system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computing system. In some embodiments, the computing device 800 may be any other electronic device that processes data.
[0116] Selected Examples
[0117] The following paragraphs provide various examples of the embodiments disclosed herein.
[0118] Example 1 provides a method for training a target neural network, the method comprising generating a supporting neural network based on the target neural network, wherein the supporting neural network comprises a plurality of supporting layers, and the target neural network comprises a plurality of target layers; merging the target neural network and the supporting neural network to form a merged network, wherein merging the target neural network and the supporting neural network comprises establishing a connection between a target layer among the plurality of target layers and a supporting layer among the plurality of supporting layers, the connection being used to transfer data between the target layer and the supporting layer; training the merged network by using a training data set; and after the merged network is trained, separating the target neural network from the supporting neural network.
[0119] Example 2 provides the method of Example 1, wherein generating the supporting neural network based on the target neural network includes generating respective ones of the plurality of supporting layers based on respective ones of the plurality of target layers.
[0120] Example 3 provides the method of Example 2, wherein each supporting layer and each target layer each includes PEs arranged in a same structure, and the PEs are configured to perform multiply-accumulate operations.
[0121] Example 4 provides the method of Example 1, wherein generating the supporting neural network includes generating internal connections within the supporting neural network, wherein the internal connections are from a first supporting layer to a second supporting layer.
[0122] Example 5 provides a method as described in Example 4, wherein the second supporting layer is configured to receive the first feature map from the first supporting layer through the internal connection; generate a second feature map; and aggregate the first feature map and the second feature map.
[0123] Example 6 provides the method of Example 4, wherein generating the supporting neural network further comprises generating an additional internal connection within the supporting neural network, wherein the additional internal connection is from a third supporting layer to the second supporting layer.
[0124] Example 7 provides a method as described in Example 6, wherein the second supporting layer is configured to receive the first feature map from the first supporting layer through the internal connection; receive the third feature map from the third supporting layer through the additional internal connection; generate a second feature map; and aggregate the first feature map, the second feature map, and the third feature map.
[0125] Example 8 provides a method as described in Example 1, wherein the connection is from the supporting layer to the target layer, and the supporting layer is configured to receive a first feature map from the target layer through the connection, generate a second feature map, and aggregate the first feature map with the second feature map.
[0126] Example 9 provides the method of Example 1, wherein merging the target neural network and the supporting neural network further comprises establishing another connection between the target layer and another supporting layer of the plurality of supporting layers.
[0127] Example 10 provides a method as described in Example 1, wherein training the merged network by using the training data set includes inputting training samples in the training data set into the target neural network, the target neural network generates a target output; inputting the training samples into the supporting neural network, the supporting neural network generates a supporting output; and adjusting parameters of the target neural network and the supporting neural network based on the ground truth labels in the training data set, the target output and the supporting output.
[0128] Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations for training a target neural network, the operations comprising generating a supporting neural network based on the target neural network, wherein the supporting neural network comprises a plurality of supporting layers and the target neural network comprises a plurality of target layers; merging the target neural network and the supporting neural network to form a merged network, wherein merging the target neural network and the supporting neural network comprises establishing a connection between a target layer of the plurality of target layers and a supporting layer of the plurality of supporting layers, the connection being used to transfer data between the target layer and the supporting layer; training the merged network by using a training data set; and separating the target neural network from the supporting neural network after the merged network is trained.
[0129] Example 12 provides one or more non-transitory computer-readable media as described in Example 11, wherein generating the supporting neural network based on the target neural network includes generating respective ones of the plurality of supporting layers based on respective ones of the plurality of target layers.
[0130] Example 13 provides one or more non-transitory computer-readable media as described in Example 12, wherein each supporting layer and each target layer each includes PEs arranged in a same structure, and the PEs are configured to perform multiply-accumulate operations.
[0131] Example 14 provides one or more non-transitory computer-readable media as described in Example 11, wherein generating the supporting neural network includes generating an internal connection within the supporting neural network, wherein the internal connection is from a first supporting layer to a second supporting layer.
[0132] Example 15 provides one or more non-transitory computer-readable media as described in Example 14, wherein the second supporting layer is configured to receive the first feature map from the first supporting layer through the internal connection; generate a second feature map; and aggregate the first feature map and the second feature map.
[0133] Example 16 provides one or more non-transitory computer-readable media as described in Example 14, wherein generating the supporting neural network further comprises generating an additional internal connection within the supporting neural network, wherein the additional internal connection is from a third supporting layer to the second supporting layer.
[0134] Example 17 provides one or more non-transitory computer-readable media as described in Example 16, wherein the second supporting layer is configured to receive the first feature map from the first supporting layer through the internal connection; receive the third feature map from the third supporting layer through the additional internal connection; generate a second feature map; and aggregate the first feature map, the second feature map, and the third feature map.
[0135] Example 18 provides one or more non-transitory computer-readable media as described in Example 11, wherein the connection is from the supporting layer to the target layer, and the supporting layer is configured to receive a first feature map from the target layer through the connection, generate a second feature map, and aggregate the first feature map with the second feature map.
[0136] Example 19 provides one or more non-transitory computer-readable media as described in Example 11, wherein merging the target neural network and the supporting neural network further comprises establishing another connection between the target layer and another supporting layer of the plurality of supporting layers.
[0137] Example 20 provides one or more non-transitory computer-readable media as described in Example 11, wherein training the merged network by using the training data set includes inputting training samples in the training data set into the target neural network, the target neural network generates a target output; inputting the training samples into the supporting neural network, the supporting neural network generates a supporting output; and adjusting parameters of the target neural network and the supporting neural network based on the ground truth labels in the training data set, the target output, and the supporting output.
[0138] Example 21 provides an apparatus for training a target neural network, the apparatus comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions, the instructions being executable by the computer processor to perform operations, the operations comprising generating a supporting neural network based on the target neural network, wherein the supporting neural network comprises a plurality of supporting layers, and the target neural network comprises a plurality of target layers, merging the target neural network and the supporting neural network to form a merged network, wherein merging the target neural network and the supporting neural network comprises establishing a connection between a target layer of the plurality of target layers and a support layer of the plurality of support layers, the connection being used to transfer data between the target layer and the support layer, training the merged network by using a training data set, and separating the target neural network from the supporting neural network after the merged network is trained.
[0139] Example 22 provides the apparatus of Example 21, wherein generating the supporting neural network based on the target neural network comprises generating respective ones of the plurality of supporting layers based on respective ones of the plurality of target layers.
[0140] Example 23 provides the apparatus of Example 21, wherein generating the supporting neural network comprises generating an internal connection within the supporting neural network, wherein the internal connection is from a first supporting layer to a second supporting layer.
[0141] Example 24 provides an apparatus as described in Example 21, wherein the connection is from the supporting layer to the target layer, and the supporting layer is configured to receive a first feature map from the target layer through the connection, generate a second feature map, and aggregate the first feature map with the second feature map.
[0142] Example 25 provides an apparatus as described in Example 21, wherein training the merged network by using the training data set includes inputting training samples in the training data set into the target neural network, the target neural network generates a target output; inputting the training samples into the supporting neural network, the supporting neural network generates a supporting output; and adjusting parameters of the target neural network and the supporting neural network based on the ground truth labels in the training data set, the target output and the supporting output.
[0143] The above description of the illustrated implementations of the present disclosure, including those described in the Abstract, is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Although specific implementations of the present disclosure and examples thereof are described herein for illustrative purposes, as will be appreciated by those skilled in the relevant art, various equivalent modifications are possible within the scope of the present disclosure. These modifications may be made to the present disclosure in accordance with the above detailed description.
Claims
1. A method for training a target neural network, the method comprising: generating a supporting neural network based on the target neural network, wherein the supporting neural network includes a plurality of supporting layers and the target neural network includes a plurality of target layers; Merging the target neural network and the supporting neural network to form a merged network, wherein merging the target neural network and the supporting neural network comprises: Establishing a connection between a target layer of the plurality of target layers and a support layer of the plurality of support layers, the connection being used to transmit data between the target layer and the support layer; training the merged network by using a training data set; and After the merged network is trained, the target neural network is separated from the supporting neural networks.
2. The method of claim 1, wherein: Generating the supporting neural network based on the target neural network includes: Respective ones of the plurality of support layers are generated based on respective ones of the plurality of target layers.
3. The method of claim 2, wherein: Each supporting layer and each target layer each includes processing elements arranged in an identical structure, the processing elements being configured to perform multiply-accumulate operations.
4. The method of claim 1, wherein: Generating the supporting neural network comprises: An internal connection is generated within the supporting neural network, wherein the internal connection is from a first supporting layer to a second supporting layer.
5. The method of claim 4, wherein: The second support layer is configured as: receiving a first feature map from the first support layer; generating a second feature map; and The first feature map and the second feature map are aggregated.
6. The method of claim 4, wherein: Generating the supporting neural network further comprises: An additional internal connection is generated within the supporting neural network, wherein the additional internal connection is from a third supporting layer to the second supporting layer.
7. The method of claim 6, wherein: The second support layer is configured as: receiving a first feature map from the first support layer; receiving a third feature map from the third support layer; generating a second feature map; and The first feature map, the second feature map, and the third feature map are aggregated.
8. The method of claim 1, wherein: The connection is from the supporting layer to the target layer, and the supporting layer is configured to: Receiving a first feature map from the target layer; Generate a second feature map, and Aggregate the first feature map with the second feature map.
9. The method of claim 1, wherein: Merging the target neural network and the supporting neural network further includes establishing another connection between the target layer and another supporting layer of the plurality of supporting layers.
10. The method of claim 1, wherein: Training the merged network by using the training data set includes: Inputting the training samples in the training data set into the target neural network, and the target neural network generates a target output; inputting the training samples into the supporting neural network, the supporting neural network generating a supporting output; and Parameters of the target neural network and the support neural network are adjusted based on the ground truth labels in the training dataset, the target output, and the support output.
11. One or more non-transitory computer-readable media storing instructions executable to perform operations for training a target neural network, the operations comprising: generating a supporting neural network based on the target neural network, wherein the supporting neural network includes a plurality of supporting layers and the target neural network includes a plurality of target layers; Merging the target neural network and the supporting neural network to form a merged network, wherein merging the target neural network and the supporting neural network comprises: Establishing a connection between a target layer of the plurality of target layers and a support layer of the plurality of support layers, the connection being used to transmit data between the target layer and the support layer; training the merged network by using a training data set; and After the merged network is trained, the target neural network is separated from the supporting neural networks.
12. The one or more non-transitory computer readable media of claim 11, wherein: Generating the supporting neural network based on the target neural network includes: Respective ones of the plurality of support layers are generated based on respective ones of the plurality of target layers.
13. One or more non-transitory computer readable media as recited in claim 12, wherein: Each supporting layer and each target layer each includes processing elements arranged in an identical structure, the processing elements being configured to perform multiply-accumulate operations.
14. The one or more non-transitory computer readable media of claim 11, wherein: Generating the supporting neural network comprises: An internal connection is generated within the supporting neural network, wherein the internal connection is from a first supporting layer to a second supporting layer.
15. The one or more non-transitory computer readable media of claim 14, wherein: The second support layer is configured as: receiving a first feature map from the first support layer; generating a second feature map; and The first feature map and the second feature map are aggregated.
16. The one or more non-transitory computer readable media of claim 14, wherein: Generating the supporting neural network further comprises: An additional internal connection is generated within the supporting neural network, wherein the additional internal connection is from a third supporting layer to the second supporting layer.
17. The one or more non-transitory computer readable media of claim 16, wherein: The second support layer is configured as: receiving a first feature map from the first support layer; receiving a third feature map from the third support layer; generating a second feature map; and The first feature map, the second feature map, and the third feature map are aggregated.
18. The one or more non-transitory computer readable media of claim 11, wherein: The connection is from the supporting layer to the target layer, and the supporting layer is configured to: Receiving a first feature map from the target layer; Generate a second feature map, and Aggregate the first feature map with the second feature map.
19. The one or more non-transitory computer readable media of claim 11, wherein: Merging the target neural network and the supporting neural network further includes establishing another connection between the target layer and another supporting layer of the plurality of supporting layers.
20. The one or more non-transitory computer readable media of claim 11, wherein: Training the merged network by using the training data set includes: Inputting the training samples in the training data set into the target neural network, and the target neural network generates a target output; inputting the training samples into the supporting neural network, the supporting neural network generating a supporting output; and Parameters of the target neural network and the support neural network are adjusted based on the ground truth labels in the training dataset, the target output, and the support output.
21. A device for training a target neural network, the device comprising: A computer processor for executing computer program instructions; as well as a non-transitory computer readable memory storing computer program instructions executable by the computer processor to perform operations comprising: generating a supporting neural network based on the target neural network, wherein the supporting neural network includes a plurality of supporting layers and the target neural network includes a plurality of target layers, Merging the target neural network and the supporting neural network to form a merged network, wherein merging the target neural network and the supporting neural network comprises: establishing a connection between a target layer of the plurality of target layers and a support layer of the plurality of support layers, the connection being used to transfer data between the target layer and the support layer, By training the merged network using a training dataset, and After the merged network is trained, the target neural network is separated from the supporting neural networks.
22. The device of claim 21, wherein: Generating the supporting neural network based on the target neural network includes: Respective ones of the plurality of support layers are generated based on respective ones of the plurality of target layers.
23. The device of claim 21, wherein: Generating the supporting neural network comprises: An internal connection is generated within the supporting neural network, wherein the internal connection is from a first supporting layer to a second supporting layer.
24. The device of claim 21, wherein: The connection is from the supporting layer to the target layer, and the supporting layer is configured to: Receiving a first feature map from the target layer; Generate a second feature map, and Aggregate the first feature map with the second feature map.
25. The apparatus of claim 21, wherein: Training the merged network by using the training data set includes: Inputting the training samples in the training data set into the target neural network, and the target neural network generates a target output; inputting the training samples into the supporting neural network, the supporting neural network generating a supporting output; and Parameters of the target neural network and the support neural network are adjusted based on the ground truth labels in the training dataset, the target output, and the support output.
