Training neural networks through many-to-one knowledge injection

Abstract

The target neural network may be trained by many-to-one knowledge injection using a support neural network. Many-to-one knowledge injection is facilitated by two layers inserted into the target neural network. The first layer converts a target OFM in a target neural network into an extended feature map with more channels. The second layer converts the extended feature map into a new feature map having the same dimension as the target OFM. The extended feature map may be divided into segments, each of which has the same number of channels as the support OFM in the support neural network, such that knowledge in the support OFM may be injected into each of the segments by many-to-one injection. In order to train the target neural network, parameters within the target neural network are modified to minimize the feature distance between the extended feature map and the support OFM.

Description

Technical Field

[0001] The present disclosure relates generally to neural networks, and more particularly to training deep neural networks (DNNs) through many-to-one knowledge injection. Background Art

[0002] DNNs are widely used in various artificial intelligence applications, ranging from computer vision to speech recognition and natural language processing, due to their ability to achieve high accuracy. However, high accuracy comes at a significant computational cost. DNNs have extremely high computational requirements, as each inference may require the storage of hundreds of millions of MAC (multiply-accumulate) operations and hundreds of millions of weight operands (weight operands) weights to be stored for classification or detection. Therefore, techniques to improve the efficiency of DNNs are needed. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The embodiments will be easily understood by the following detailed description in conjunction with the accompanying drawings. For ease of description, the same reference numerals represent the same structural elements. In the figures of the accompanying drawings, the embodiments are shown by way of example and not limitation.

[0004] Figure 1 An example layer structure of a DNN according to various embodiments is shown.

[0005] Figure 2 is a block diagram of a DNN system according to various embodiments.

[0006] Figure 3 is a block diagram of a training module according to various embodiments.

[0007] Figure 4 An example process of training a student network using a teacher network through many-to-one knowledge injection according to various embodiments is shown.

[0008] Figure 5 An example process of training a student network using a teacher network through many-to-one knowledge injection according to various embodiments is shown.

[0009] Figure 6 A deep learning (DL) environment according to various embodiments is shown.

[0010] Figure 7 is a flowchart illustrating a method of training a DNN through knowledge distillation according to various embodiments.

[0011] Figure 8 is a block diagram of an example computing device in accordance with various embodiments. DETAILED DESCRIPTION

[0012] Overview

[0013] DNNs are widely used in the fields of computer vision, speech recognition, image and video processing, mainly because they can achieve super-human level accuracy. However, the improvement in accuracy comes at a significant computational cost. The underlying DNN has extremely high computational requirements, as each input requires at least hundreds of millions of MAC operations and processing hundreds of millions of weight operand weights for classification or detection. Energy-constrained mobile systems and embedded systems, where energy and area budgets are extremely limited, typically use area- and energy-efficient DNN accelerators as the underlying hardware for executing machine learning applications.

[0014] Knowledge distillation is one of the solutions that provides a teacher-student training framework to train a compact, computationally efficient DNN model that has improved prediction accuracy compared to standard training. However, many existing knowledge distillation solutions have various limitations. For example, these solutions typically use feature maps, attention maps, and abstract feature forms at multiple hidden layers as knowledge representations. Due to different network depths and layer widths, the output feature maps of the teacher-student layer pairs typically have different dimensions. In order to align the feature dimensions, these solutions perform various teacher / student transformations. However, due to the reduction in feature dimensions, this transformation design results in different levels of information loss. In addition, many existing knowledge distillation methods (in both the two-stage family and the one-stage family) typically adopt a one-to-one representation match between each pre-selected teacher-student layer pair. That is, for any teacher-student layer pair, there is a knowledge transfer inlet that sometimes cannot efficiently transfer knowledge from the teacher to the student. Therefore, improved knowledge distillation techniques are needed.

[0015] Embodiments of the present disclosure may improve at least some of the above challenges and problems by providing methods and apparatus for facilitating knowledge distillation through many-to-one knowledge injection, which is also referred to as N-to-1 knowledge injection, where N represents an integer greater than 1.

[0016] In some embodiments of the present disclosure, a target neural network is trained by acquiring knowledge from a supporting neural network. The target neural network may be referred to as a student neural network, a student network, or a student. The supporting neural network may be referred to as a teacher neural network, a teacher network, or a teacher. The supporting neural network has been trained. The knowledge learned by the supporting neural network may be represented by one or more feature maps within the supporting neural network (e.g., an output feature map (OFM) of a convolutional layer in the supporting neural network). The convolutional layer in the supporting neural network may correspond to the convolutional layer in the target neural network, e.g., the two layers are aligned, or the stages in which the two layers are included are aligned. The knowledge learned by the convolutional layer in the supporting neural network may be transferred to the convolutional layer in the target neural network through many-to-one knowledge injection. The convolutional layer in the supporting neural network may be referred to as a teacher layer. The convolutional layer in the target neural network may be referred to as a student layer.

[0017] Many-to-one knowledge injection can be facilitated by two layers inserted into the target neural network. The two layers can be placed after the student layer. The first layer can convert the OFM of the student layer into an extended feature map with more channels. The OFM of the student layer can be called a student feature map. The second layer can convert the extended feature map into a new feature map with the same dimension as the student feature map. The extended feature map can be divided into segments, each of which has the same number of channels as the OFM of the teacher layer ("teacher feature map"), so that the knowledge in the teacher feature map can be injected into each of the segments through many-to-one injection. The target neural network can be trained by modifying the parameters within the target neural network to minimize the feature distance between the extended feature map and the teacher feature map. After training the target neural network, the two layers can be merged into a layer arranged after the student layer in the target neural network, for example, a fully connected layer.

[0018] Compared with existing knowledge distillation solutions using one-to-one knowledge injection, many-to-one knowledge injection can be used to train various types of DNNs with better accuracy and efficiency trade-offs. These DNNs can be used in various AI (artificial intelligence) applications, such as image classification, facial recognition, action recognition, person re-identification, machine translation, and speech recognition. The present disclosure provides a knowledge distillation solution that can retain the complete information learned by a pre-trained teacher network and convert computationally intensive DNNs into lighter-weight DNNs with similar accuracy. From a hardware perspective, this can facilitate the use of parallel distributed processing to replace deep sequential processing. This structural conversion can enable the use of general-purpose processors (GPPs) (e.g., multi-core central processing units (CPUs) and graphics processing units (GPUs)) to accelerate DNN training and reasoning.

[0019] For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without the specific details and / or may be practiced with only some of the described aspects. In other instances, well-known features are omitted or simplified so as not to obscure the illustrative implementations.

[0020] In addition, reference is made to the accompanying drawings which form a part thereof, and in the accompanying drawings, embodiments that can be practiced are shown by way of illustration. It should be understood that other embodiments can be utilized and structural or logical changes can be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be considered to have a limiting meaning.

[0021] Various operations may be described in turn as multiple discrete actions or operations in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be interpreted as implying that these operations are necessarily order-dependent. In particular, these operations may not be performed in the order presented. The described operations may be performed in an order different from the described embodiments. In additional embodiments, various additional operations may be performed, or the described operations may be omitted.

[0022] For purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used with reference to a measurement range, the term "between" includes the ends of the measurement range.

[0023] The specification uses the phrases "in an embodiment" or "in multiple embodiments", which may each refer to one or more embodiments in the same or different embodiments. The terms "including", "comprising", "having", etc. used with respect to the embodiments of the present disclosure are synonymous. The present disclosure may use perspective-based descriptions such as "above", "below", "top", "bottom", and "side" to explain various features of the drawings, but these terms are merely for ease of discussion and do not imply a desired or required orientation. The drawings are not necessarily drawn to scale. Unless otherwise stated, the ordinal adjectives "first", "second", and "third", etc., are used to describe common objects to indicate only that different instances of the same object are being referenced, and are not intended to imply that the objects so described must be in a given sequence in time, space, order, or in any other way.

[0024] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0025] The terms "substantially," "close," "approximately," "near," and "about" generally refer to input operands being within + / - 20% of target values ​​based on specific values ​​as described herein or known in the art. Similarly, terms indicating the orientation of various elements (e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements) generally refer to input operands being within + / - 5-20% of target values ​​based on specific values ​​as described herein or known in the art.

[0026] Furthermore, the terms "comprise," "comprising," "include," "including," "have," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, apparatus, or DNN accelerator. Moreover, the term "or" refers to an inclusive "or" rather than an exclusive "or."

[0027] The DNN systems, methods, and devices of the present disclosure each have several innovative aspects, no single aspect of which is solely responsible for all of the desirable properties disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the following description and the accompanying drawings.

[0028] Example DNN layer structure

[0029] Figure 1 An example layer structure of DNN 100 according to various embodiments is shown. For illustration purposes, Figure 1 The DNN 100 in is a convolutional neural network (CNN). In other embodiments, the DNN 100 may be another type of DNN. The DNN 100 is trained to receive an image and output a classification of an object in the image. Figure 1 In an embodiment of the invention, DNN 100 receives an input image 105 including objects 115, 125, and 135. DNN 100 includes a sequence of layers including a plurality of convolutional layers 110 (each referred to as “convolutional layer 110”), a plurality of pooling layers 120 (each referred to as “pooling layer 120”), and a plurality of fully connected layers 130 (each referred to as “fully connected layer 130”). In other embodiments, DNN 100 may include fewer, more, or different layers.

[0030] The convolutional layer 110 summarizes the presence of features in the input image 105. Figure 1In an embodiment of the present invention, the first layer of the DNN 100 is a convolutional layer 110. The convolutional layer 110 is used as a feature extractor. The convolutional layer 110 may receive an input and output features extracted from the input. In an example, the convolutional layer 110 performs a convolution on an IFM (input feature map) 140 using a filter 150, generates an OFM 160 according to the convolution, and passes the OFM 160 to the next layer in the sequence. The IFM 140 may include a plurality of IFM matrices. The filter 150 may include a plurality of weight matrices. The OFM 160 may include a plurality of OFM matrices. For the first convolutional layer 110 (which is also the first layer of the DNN 100), the IFM 140 is the input image 105. For other convolutional layers, the IFM 140 may be the output of another convolutional layer 110 or the output of the pooling layer 120.

[0031] The convolution may be a linear operation involving the multiplication of a weight operand in the filter 150 and a patch of the IFM 140 having the size of the weight operand. The weight operand may be a weight matrix (e.g., a 2-dimensional weight array) in the filter 150, where the weights are arranged in columns and rows. The weights may be initialized and updated by back propagation using gradient descent. The magnitude of the weight may indicate the importance of the filter 150 in extracting features from the IFM 140. The weight operand may be smaller than the IFM 140. The multiplication may be an element-by-element multiplication between a patch of the IFM 140 having the size of the weight operand and the corresponding weight operand, which is then summed, which always produces a single value. Because this produces a single value, the operation is often referred to as a "scalar product".

[0032] In some embodiments, using a weight operand that is smaller than the IFM 140 is intentional because it allows the same weight operand (set of weights) to be multiplied with the IFM 140 multiple times at different points on the IFM 140. Specifically, the weight operand is systematically applied to each overlapping portion of the IFM 140 or tile of the size of the weight operand, in order from left to right and from top to bottom. The result of multiplying the weight operand with the IFM 140 once is a single value. When the weight operand is applied to the IFM 140 multiple times, the result of the multiplication is a two-dimensional array of output values ​​representing the weighted operation of the IFM 140. Therefore, the two-dimensional output array from this operation is called a "feature map."

[0033] In some embodiments, OFM 160 is passed through an activation function. An example activation function is a rectified linear activation function (ReLU). ReLU is a calculation that directly returns the value provided as an input, or a calculation that returns the value zero if the input is zero or less. Convolutional layer 110 can receive several images as input and calculate the convolution of each of these images with each weight operand in the weight operand. This process can be repeated several times. For example, OFM 160 is passed to a subsequent convolutional layer 110 (i.e., a convolutional layer 110 after the convolutional layer 110 that generates OFM 160 in the sequence). The subsequent convolutional layer 110 performs convolution on OFM 160 using the new weight operand and generates a new feature map. The new feature map can also be normalized and resized. The new feature map can be weighted again by another subsequent convolutional layer 110, and so on.

[0034] In some embodiments, the convolutional layer 110 has four hyperparameters: the number of weight operands; the size F of the weight operands (e.g., the dimensions of the weight operands are F×F×D pixels); the step S with which the window corresponding to the weight operands is dragged on the image (e.g., a step of one means moving the window one pixel at a time); and zero padding P (e.g., adding a black outline of pixel thickness P to the input image of the convolutional layer 110). The convolutional layer 110 can perform various types of convolutions, such as 2-dimensional convolutions, dilated or atrous convolutions, spatially separable convolutions, depthwise separable convolutions, transposed convolutions, etc. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 can include a different number of convolutional layers.

[0035] The pooling layer 120 downsamples the feature map generated by the convolution layer, for example, by summarizing the presence of features in the tiles of the feature map. The pooling layer 120 is placed between the following two convolutional layers 110: the previous convolutional layer 110 (the convolutional layer 110 before the pooling layer 120 in the sequence of layers) and the subsequent convolutional layer 110 (the convolutional layer 110 after the pooling layer 120 in the sequence of layers). In some embodiments, the pooling layer 120 is added after the convolutional layer 110 (for example, after the activation function (e.g., ReLU) has been applied to the OFM 160).

[0036] The pooling layer 120 receives the feature map generated by the previous convolution layer 110 and applies a pooling operation to the feature map. The pooling operation reduces the size of the feature map while retaining the important characteristics of the feature map. Therefore, the pooling operation improves the efficiency of the DNN and avoids over-learning. The pooling layer 120 can perform a pooling operation by average pooling (calculating the average value for each tile on the feature map), maximum pooling (calculating the maximum value for each tile of the feature map), or a combination of the two. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is 2×2 pixels applied with a stride of two pixels, so that the pooling operation reduces the size of the feature map by 2 times, for example, the number of pixels or values ​​in the feature map is reduced to one-quarter of the size. In the example, the pooling layer 120 applied to a 6×6 feature map produces a 3×3 output pooled feature map. The output of the pooling layer 120 is input into the subsequent convolution layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates on each feature map separately to create a new set with the same number of pooled feature maps.

[0037] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 can be convolutional or non-convolutional. The fully connected layer 130 receives input operands. The input operands define the outputs of the convolutional layer 110 and the pooling layer 120, and include the values ​​of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies linear combinations and activation functions to the input operands and generates individual partial sums. The individual partial sums can contain as many elements as categories: element i represents the probability that the image belongs to category i. Therefore, each element is between 0 and 1, and the sum of all elements is 1. These probabilities are calculated by the last fully connected layer 130 using a logistic function (binary classification) or a softmax function (multi-class classification) as an activation function.

[0038] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of categories in the image classification problem. Figure 1In the embodiment of , N is equal to 3 because there are three objects 115, 125 and 135 in the input image. Each element of the operand indicates the probability that the input image 105 belongs to a category. To calculate the probability, the fully connected layer 130 multiplies each input element by a weight, sums, and then applies an activation function (e.g., a logistic function if N=2, and a softmax function if N>2). This is equivalent to multiplying the input operand by a matrix containing weights. In the example, the individual partial sums include three probabilities: a first probability indicating that the object 115 is a tree, a second probability indicating that the object 125 is a car, and a third probability indicating that the object 135 is a person. In other embodiments in which the input image 105 includes different objects or different numbers of objects, the individual partial sums may be different.

[0039] Example DNN System

[0040] Figure 2 is a block diagram of a DNN system 200 according to various embodiments. The DNN system 200 trains a DNN by using knowledge distillation (e.g., knowledge distillation with many-to-one knowledge injection). A DNN can be used to perform one or more machine learning tasks. A machine learning task is a task of performing reasoning. Reasoning is the process of running available data into a DNN to generate an output, and the output provides a solution to the question or query being asked. An example of an output is one or more numerical scores that indicate the probability that an object in an image belongs to a class. The DNN system 200 can train a DNN that can be used to solve various problems (e.g., image classification, learning relationships between biological cells (e.g., DNA, proteins, etc.), control behavior for devices (e.g., robots, machines, etc.), etc.).

[0041] The DNN system 200 includes an interface module 210, a training set generator 220, a student network generator 230, a teacher network generator 240, a training module 250, a verification module 260, and a memory 270. In other embodiments, alternative configurations, different components, or additional components may be included in the DNN system 200. In addition, the functionality attributed to the components of the DNN system 200 may be implemented by different components included in the DNN system 200 or by different systems.

[0042] The interface module 210 facilitates communication between the DNN system 200 and other systems. For example, the interface module 210 establishes communication between the DNN system 200 and an external database to receive data that can be used to train the DNN or data that can be input into the DNN to perform a machine learning task. As another example, the interface module 210 supports the DNN system 200 to distribute the DNN to other systems, such as computing devices configured to apply the DNN to perform tasks. The computing device can be an edge device, a client device, etc.

[0043] The training set generator 220 forms a training data set to be used to train the DNN. The training data set includes training samples and ground-truth labels. The training data set may include one or more ground-truth labels for each training sample. The ground-truth labels of the training samples may be known or verified labels that answer questions or queries that the DNN will use to answer. In an example in which the DNN is trained to recognize objects in an image, the training data set includes training images and ground-truth labels indicating the classification of the objects in the training images. The ground-truth labels in the example may be numbers indicating the probability that an object belongs to a category. Objects may be associated with other ground-truth labels indicating the probability that the object belongs to other categories.

[0044] In some embodiments, the training set generator 220 may also form a validation data set, which is used to verify the performance of the trained DNN by the verification module 260. The validation data set may include validation samples and ground truth labels of the validation samples. The validation data set for the DNN may include samples different from the training data set used to train the DNN. In an embodiment, a portion of the training data set may be used to initially train the DNN, and the remainder of the training data set may be retained as a validation subset used by the verification module 260 to verify the performance of the trained DNN. The portion of the training data set that does not include the validation subset may be used to train the DNN.

[0045] The student network generator 230 generates a student network. The student network is a DNN that can be used to perform machine learning tasks after training. The student network generator 230 can generate a student network based on parameters that define the architecture of the DNN. Examples of parameters include the number of layers, the type of layers, the sequence of layers, the number of processing elements (PEs) in a layer, the type of PEs, the arrangement of PEs in a layer (e.g., interconnections between PEs, the number of columns in a PE array, the number of rows in a PE array, etc.), activation functions, pooling functions, or other types of parameters. The processing elements perform MAC operations.

[0046] In some embodiments, the student network generator 230 determines some or all of the parameters, for example, based on the questions or queries that the DNN is to answer, the resources available for training, the resources available for reasoning, some other factors that may be critical to the architecture of the DNN, or some combination thereof. In other embodiments, the student network generator 230 may receive some or all of the parameters from a different system (e.g., from a computing device that will run the DNN for reasoning, a system that manages such a computing device, etc.) or from a user (e.g., through a user interface that allows the user to provide information for the DNN).

[0047] The architecture of a DNN includes an input layer, an output layer, and multiple hidden layers. The input layer of a DNN may include tensors (e.g., multidimensional arrays) that specify attributes of an input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits that specify the color of a pixel in the input image). The output layer includes labels of objects in the input layer. The hidden layer is a layer between the input layer and the output layer. The hidden layer includes one or more convolutional layers and one or more other types of layers, such as rectified linear unit (ReLU) layers, pooling layers, fully connected layers, normalization layers, softmax layers, or logical layers. The convolutional layer of a DNN abstracts the input image into a feature map, which is represented by a tensor that specifies the feature map height, feature map width, and feature map channels (e.g., red, green, and blue images include three channels). The pooling layer is used to reduce the spatial volume of the input image after convolution. The pooling layer is used between two convolutional layers. The fully connected layer involves weights, biases, and neurons. The fully connected layer connects neurons in one layer to neurons in another layer. The fully connected layers are used to classify images between different categories through training. The example DNN is a combination of the above Figure 1 DNN 100 described.

[0048] The teacher network generator 240 generates a teacher network for training a student network by knowledge distillation. In some embodiments, the teacher network generator 240 may generate a single teacher network for training a single student network or multiple student networks, or may generate multiple teacher networks for training a single student network. The teacher network may be a DNN. The teacher network may have a different architecture from the student network trained using the teacher network.

[0049] In some embodiments, the teacher network generator 240 determines the structure of the teacher network based on the structure of the student network. For example, the teacher network generator 240 can generate a teacher network including the same number and / or type of layers as the student network. The arrangement of the layers in the teacher network ("teacher layer") can be the same as the arrangement of the layers in the student network ("student layer"). Moreover, for a single teacher layer, the teacher network generator 240 can design the teacher layer based on the corresponding student layer. The teacher network generator 240 can make the teacher layer mirror the student layer. For example, the teacher layer can have the same number and / or type of PEs as the student layer. The arrangement of PEs can also be the same in the two layers.

[0050] The teacher network generator 240 also generates internal connections within the teacher network. The internal connection can connect two teacher layers, for example, from a first teacher layer to a second teacher layer. The second teacher layer can be arranged after the first layer in the teacher network. The internal connection facilitates data transfer between the two teacher layers. For example, the first layer can send features (e.g., OFM 160) to the second layer via the internal connection. The second layer receives the features and can aggregate the features from the first layer with the features generated in the second layer to output aggregated features. The internal connection can be bidirectional, for example, the second layer can also send data to the first layer.

[0051] The training module 250 trains the DNN, e.g., the student network, through many-to-one knowledge injection from the student network. The training module 250 can generate student transform layers and insert these layers into the student network. In some embodiments, the training module 250 places the student transform layer after the convolutional layer in the student network (e.g., the last convolutional layer in the student network). During the training of the student network, the student transform layer facilitates many-to-one knowledge injection from the teacher network (e.g., from the feature map in the teacher network).

[0052] A student transform layer in the student transform layer can convert the feature map output from the convolution layer in the student network into an extended feature map. The extended feature map includes more channels than the student feature map, but the pixels in each channel in the extended feature map can be the same as the pixels in each channel in the student feature map. The extended feature map includes multiple segments, each segment has the same number of channels as the teacher feature map. The training module 250 can modify the internal parameters in the student network so that each segment of the extended feature map can be similar or identical to the teacher feature map. Since there are many segments in the extended feature map and one teacher feature map in the process, the knowledge injection in the process is many-to-one knowledge injection.

[0053] Another layer in the student transform layer can transform the expanded feature map into a new feature map including the same number of channels as the student feature map, so that the new feature map can be fed to and processed by the next layer in the student network without interrupting the operation in the next layer.

[0054] In addition to injecting knowledge from the teacher feature map into the student feature map, the training module 250 can further train the student network based on the training set. The training module 250 can also receive a training data set from the training set generator 220. The training module 250 can send the training samples in the training data set to the student network. The training module 250 modifies the parameters in the student network to minimize the error between the labels of the training samples generated by the student network and the ground truth labels in the data set. The training module 250 can use a loss function (e.g., a cross entropy loss function) to minimize the error. In some embodiments, the training module 250 modifies the parameters in the student network to minimize the combination of the difference between the teacher feature map and the extended feature map and the difference between the label generated by the student network and the ground truth label.

[0055] The training module 250 can stop adjusting the parameters in the merged network after a threshold condition is met. The threshold condition can be completing a predetermined number of epochs, meeting the target performance (e.g., accuracy) of the merged network, the student network, or the teacher network, or other types of conditions. The trained student network can be used to process machine learning tasks. In some embodiments, the student network or the parameters of the student network can be sent to another system or device (e.g., an edge device, a client device, etc.) for reasoning.

[0056] The training module 250 may also determine hyperparameters for the training process. Hyperparameters may be different from parameters within the network (e.g., weights). In some embodiments, hyperparameters include variables that determine how the DNN is trained, such as batch size, the number of rounds, and the like. The batch size defines the number of training samples to be processed before updating the parameters of the DNN. The batch size is equal to or less than the number of samples in the training data set. The training data set may be divided into one or more batches. The number of rounds defines how many times the entire training data set is passed forward and backward through the entire network. The number of rounds defines the number of times the DL algorithm processes the entire training data set. One round means that each training sample in the training data set has a chance to update the parameters within the network. Rounds may include one or more batches. The number of rounds may be 10, 100, 500, 1000, or even larger.

[0057] The verification module 260 verifies the performance (e.g., accuracy) of a trained DNN (e.g., a trained student network separated from its corresponding teacher network). The verification module 260 may determine the accuracy of the trained student network and determine whether the accuracy meets a threshold (e.g., a requirement for model accuracy). In response to determining that the accuracy of the student network meets the threshold, the verification module 260 may deploy the student network to another system or device, for example, through the interface module 210. In some embodiments, the verification module 260 may also verify the performance of the merged network or the teacher network. For example, the verification module 260 determines whether the accuracy of the merged network meets the threshold. In response to determining that the accuracy does not meet the threshold, the verification module 260 may instruct the training module 250 to further train the merged network. In response to determining that the accuracy meets the threshold, the verification module 260 may notify the training module 250 that the merged network has been fully trained, or instruct the training module 250 to separate the student network from the teacher network.

[0058] In some embodiments, the verification module 260 inputs samples in the verification data set into the DNN and uses the output of the DNN to determine the model accuracy. In some embodiments, the verification data set may be formed by some or all samples in the training data set. Additionally or alternatively, the verification data set includes additional samples in addition to the samples in the training set. In some embodiments, the verification module 260 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. The verification module 260 may use the following metrics to determine the accuracy score: precision = TP / (TP+FP), and recall = TP / (TP+FN), where precision can be the number of correct predictions (TP or true positives) of the reference classification model in its total number of predictions (TP+FP or false positives), and recall can be the number of correct predictions (TP) of the reference classification model in the total number of objects that do have the attribute in question (TP+FN or false negatives). The F score (F score = 2*PR / (P+R)) unifies precision and recall into a single measurement.

[0059] The memory 270 stores data associated with the DNN system 200, for example, data received, generated, or used by the DNN system 200. For example, the memory 270 may store parameters (e.g., internal parameters, hyperparameters, etc.) of a student network or a teacher network generated by the student network generator 230, the teacher network generator 240, or the training module 250. The memory 270 may also store training sets and validation sets for training networks and validating networks. In some embodiments, the DNN system 200 may be associated with multiple memories. The memory 270 may include a random access memory (RAM) (e.g., a static RAM (SRAM)), a disk storage device, a near-line storage device, an online storage device, an offline storage device, and the like.

[0060] Figure 3 is a block diagram of a training module 250 according to various embodiments. The training module 250 includes a layer generator 310, a student transformation module 330 including an expansion layer 340 and a contraction layer 350, an insertion module 360, a knowledge injection module 370, and a merging module 380. In other embodiments, alternative configurations, different or additional components may be included in the training module 250. For example, the training module 250 may include multiple student transformation modules. In addition, the functionality attributed to the components of the training module 250 may be implemented by different components included in the training module 250 or by different systems.

[0061] The layer generator 310 generates the expansion layer 340 and the contraction layer 350 in the student transformation module 330. The expansion layer 340 can expand the feature map F in the student network s And generate an extended feature map. Extended feature map F se It can include a ratio feature map F s More channels, but expanded feature map F se The spatial size in each channel can be related to the feature map F s The spatial size in each channel is the same. For example, the feature map F s The extended feature map F can be included in each channel se Same number of pixels.

[0062] In some embodiments, the layer generator 310 defines a convolution kernel W for the extended layer 340. se To generate the extended layer 340. The extended layer 340 can be based on the feature map F s and the convolution kernel W se Perform a convolution operation to generate an expanded feature map F se , for example, F se =W se *F s , where * represents a convolution operation. The layer generator 310 may determine the convolution kernel based on the feature map in the student network and the corresponding feature map in the teacher network. Where the number of channels in the student feature map is C s (For example, ) and the number of channels in the teacher feature map is C t (For example, ), the convolution kernel W se It can be a 1×1 convolution kernel along the channel dimension F se Each pixel in is projected to the desired channel dimension NC t, thus producing an extended student representation with N times the feature channels of the teacher feature map

[0063] The layer generator 310 can also generate the contraction layer 350 in the student transform module 330. The layer generator 310 can be configured by defining another convolution kernel W sc To generate the contraction layer 350. Convolution kernel W sc It can be another 1×1 convolution kernel, for example, The shrinking layer 350 can be used to expand the feature map F se and the convolution kernel W sc Perform a convolution operation to expand the feature map F se Convert back to the original channel dimension C s , thus generating This can be expressed as F se =W se *F s , F sc =W sc *F se , where * represents the convolution operation.

[0064] The insertion module 360 ​​inserts the student transform module 330 into the student network. In some embodiments, the insertion module 360 ​​inserts the student transform module 330 after the convolutional layer in the student network. s is the output of the convolutional layer. The convolutional layer can be the last convolutional layer in the student network. Similarly, the teacher feature map F t It can be the output of the last convolutional layer in the teacher network, so that t By training the student transform module 330, the student network can obtain the knowledge in the last convolutional layer in the teacher network, which can incorporate the knowledge into all previous convolutional layers in the teacher network. In other embodiments, the student transform module 330 or an additional student transform module 330 can be inserted after a convolutional layer that is not the last convolutional layer in the student network. In the process of inserting the student transform module 330 into the student network, the insertion module 360 ​​places the expansion layer 340 before the contraction layer 350.

[0065] The knowledge injection module 370 uses many-to-one knowledge injection to train the student network. se Split into N non-overlapping segments with the same number of feature channels In this way, each fragment The number of channels in is equal to the teacher feature map F t Then, the knowledge injection module 370 can force each individual segment Approximately or even equal to the teacher feature map F t For example, the knowledge injection module 370 can modify the internal parameters of the student network (e.g., the values ​​in the filters of the convolutional layers in the student network) to reduce the and teacher characteristics graph F t The difference between the fragments and teacher characteristics graph F t The L2 norm feature distance between them.

[0066] Alternatively, the knowledge injection module 370 can modify the internal parameters of the student network to reduce the overall expanded feature map F se and teacher characteristics graph F t The overall difference between the segments. Each of the teacher features F t The aggregation of feature distances between. Aggregation can be average or cumulative. In some embodiments, the knowledge injection module 370 defines a feature distance function:

[0067]

[0068] in, Representation fragment and teacher characteristics graph F t The characteristic distance between .

[0069] The knowledge injection module 370 can change the internal parameters of the student network to make L norm Minimize, for example, until L norm The value of reaches the threshold. As the internal parameters of the student network change, the student feature map F s The pixels in the extended feature map F will have different values ​​and se and each fragment The pixels in will have different values. And you can change L norm When the knowledge injection module 370 converts the teacher feature map F t The knowledge in is injected into N fragments In the process of knowledge distillation, N to 1 knowledge injection is involved.

[0070] The knowledge injection module 370 can further modify the internal parameters of the student network based on the training samples and the ground truth labels of the training samples during the training of the student network. In some embodiments, the knowledge injection module 370 can modify the internal parameters of the student network so that L norm Minimize, L norm is the loss between the ground truth label and the determination made by the student network based on the training samples, or a combination of the two. For example, the knowledge injection module 370 can express the matching loss L by making N to 1norm With the cross entropy loss L supervised by the ground truth labels ce Joint minimization to train the student network:

[0071] L=L norm +L ce

[0072] In some embodiments, the expansion layers 340 and the contraction layers 350 can be trained simultaneously with other layers (e.g., existing layers) in the student network.

[0073] After the student transform module 330 is trained, the merging module 380 merges the student transform module 330 into the student network. In some embodiments, the merging module 380 may merge the student transform module 330 into the student network before outputting the student feature map F. s The layer after the convolutional layer of . This layer can be a fully connected layer or another convolutional layer. In some embodiments, this layer can be the final fully connected layer that outputs the determination of the student network. The learning parameters of the fully connected layer can be expressed as The merging module 380 can transform the student transformation module 330 (which can be expressed as T s (W se , W sc ))Merge into a fully connected layer: W fc =W fc (W sc W se ).

[0074] Example process of training a student network

[0075] Figure 4 An example process of training a student network 410 using a teacher network 420 through many-to-one knowledge injection according to various embodiments is shown. The student network 410 can be provided by the student network generator 230. The formation of the teacher network 420 can be completed by the teacher network generator 240. For the purpose of brevity and illustration, the student network 410 includes four layers 415A-415N (collectively referred to as "multiple student layers 415" or "student layers 415"), and the teacher network 420 includes four layers 425A-425N (collectively referred to as "multiple teacher layers 425" or "teacher layers 425"). The student layer 415 and the teacher layer 425 can be convolutional layers, self-attention layers, linear layers, or some combination thereof. In other embodiments, the student network 410 or the teacher network 420 may include more, fewer, or different layers. For example, the student network 410 or the teacher network 420 may include layers arranged in Figure 4 Additional layers between, before, or after the layers shown.

[0076] In some embodiments, some or all of the teacher layers 425 may be aligned with some or all of the student layers 415, or the stages in the teacher network 420 may be aligned with the stages in the student network 410. A stage may include one or more layers. The teacher network 420 may include more or different layers than the student network 410. The teacher network 420 has been trained. The knowledge learned by the teacher network 420 during training may be injected into the student network 410 through the student transformation module 430.

[0077] like Figure 4 As shown, the student transformation module 430 is inserted into the student network 410. The student transformation module 430 may be Figure 3 An embodiment of the student transformation module 330 in FIG. Figure 4 In the embodiment of the present invention, the student transform module 430 is placed immediately after the layer 415N. The layer 415N is a convolutional layer having the output of the OFM 417. The layer 415N is the last convolutional layer in the student network 410. The student transform module 430 includes an expansion layer 433 and a contraction layer 437. The expansion layer 433 converts the OFM 417 into an expanded feature map 435, which includes a set of segments 439 (referred to as "segments 439" respectively). Each segment 439 has the same number of channels as the OFM 427 in the teacher network 420. The OFM 427 is the output of the layer 425N in the teacher network 420.

[0078] Layer 425N may be aligned with layer 415N, or a stage including layer 425N in teacher network 420 may be aligned with a stage including layer 415N in student network 410. In some embodiments (e.g., embodiments in which layer 415N is the last convolutional layer in student network 410), layer 425N is the last convolutional layer in teacher network 420. Figure 4 In an embodiment of the invention, layer 425N precedes the fully connected layer 423 which outputs labels 429. Labels 429 may represent the determination of the teacher network 420. OFM 427 may include different values ​​than OFM 417 or have different dimensions than OFM 417. OFM 427 is considered to have "better knowledge" than OFM 417, and the "knowledge" in OFM 427 may be injected into OFM 417 during the process of training the student network 410 using the teacher network 420.

[0079] like Figure 4As shown, the "knowledge" in OFM 427 is injected into each segment 439 in the expanded feature map 435. Since the segment 439 includes the same number of channels as OFM 427, knowledge injection can be performed by forcing each segment 439 to approximate OFM 427. This knowledge injection is performed on an N-to-1 basis and is referred to as N-to-1 knowledge injection. In some embodiments, the internal parameters of the student network 410 (e.g., some or all filters in layers 415A-415N) can be modified during knowledge injection, for example, based on a feature map function that aggregates the feature distance between OFM 427 and the segment 439.

[0080] The shrinking layer 437 may convert the expanded feature map 435 back to the dimensions of the OFM 417. The shrinking layer 437 may generate a new feature map having the same dimensions as the OFM 417 but with different pixels. The new feature map may be fed into the fully connected layer 413, which processes the new feature map and outputs a label 419. The label 419 may be a determination made by the student network 410. In some embodiments, the training of the student network 410 also includes a process of minimizing the difference between the label 419 and the ground truth labels of the training samples, based on which the student network 410 generates the label 419. For example, the internal parameters of the modified student network 410 are used to minimize the combination of the aggregated feature distance and the difference between the label 419 and the ground truth label.

[0081] Figure 5 Another example process of training a student network 510 with a teacher network 520 through many-to-one knowledge injection according to various embodiments is shown. The student network 510 can be provided by the student network generator 230. The formation of the teacher network 520 can be completed by the teacher network generator 240. For the purpose of brevity and illustration, the student network 510 includes four layers 515A-415N (collectively referred to as "multiple student layers 515" or "student layers 515"), and the teacher network 520 includes four layers 525A-425N (collectively referred to as "multiple teacher layers 525" or "teacher layers 525"). The student layer 515 and the teacher layer 525 can be convolutional layers, self-attention layers, linear layers, or some combination thereof. In other embodiments, the student network 510 or the teacher network 520 may include more, fewer, or different layers. For example, the student network 510 or the teacher network 520 may include layers arranged in Figure 5 Additional layers between, before, or after the layers shown.

[0082] In some embodiments, some or all of the teacher layers 525 may be aligned with some or all of the student layers 515, or the stages in the teacher network 520 may be aligned with the stages in the student network 510. A stage may include one or more layers. The teacher network 520 may include more or different layers than the student network 510. The teacher network 520 has been trained. The knowledge learned by the teacher network 520 during training can be injected into the student network 510 through the student transformation modules 530 and 540.

[0083] like Figure 5 As shown, the student transformation module 530 is inserted into the student network 510. The student transformation module 530 may be Figure 3 An embodiment of the student transformation module 330 in FIG. Figure 5 In an embodiment of the invention, the student transform module 530 is placed immediately after layer 515B. Layer 515B is a convolutional layer with the output of OFM 514. Layer 515B is not the last convolutional layer in the student network 510. The student transform module 530 can convert OFM 514 into an extended feature map, which includes a set of segments into which the knowledge from OFM 524 in the teacher network 520 can be injected. Each segment has the same number of channels as OFM 524 in the teacher network 520. OFM 527 is the output of layer 525B in the teacher network 520. Layer 525B can be aligned with layer 515B, or a stage including layer 525B in the teacher network 520 is aligned with a stage including layer 515B in the student network 510. The student transform module 530 can convert the extended feature map into a new feature map, which can be fed to the next layer (e.g., a convolutional layer) in the student network 510 for further processing.

[0084] The student transformation module 540 is also inserted into the student network 510. The student transformation module 540 may be Figure 3 An embodiment of the student transformation module 330 in FIG. Figure 5 In the embodiment of the present invention, the student transform module 540 is placed immediately after the layer 515N. The layer 525N is a convolutional layer with the output of the OFM 527. The layer 515N is the last convolutional layer in the student network 510. The student transform module 530 can convert the OFM 517 into an extended feature map, which includes a set of segments into which the knowledge from the OFM 527 in the teacher network 520 can be injected. Each segment can have the same number of channels as the OFM 527 in the teacher network 520.

[0085] OFM 527 is the output of layer 525N in teacher network 520. Layer 525N may be aligned with layer 515N, or a stage including layer 525N in teacher network 520 may be aligned with a stage including layer 515N in student network 510. In some embodiments, layer 525N is the last convolutional layer in teacher network 520. OFM 527 may be fed into a fully connected layer 523, which processes OFM 527 and outputs a label 529 representing the determination of teacher network 520. Student transformation module 540 may convert the expanded feature map into a new feature map, which may be fed into a fully connected layer 513, which processes the new feature map and outputs a label 519. Label 519 may represent the determination of student network 510. It is believed that OFM 524 and OFM 527 have “better knowledge” than OFM 514 and OFM 517 , respectively, and the “knowledge” in OFM 524 and OFM 527 can be injected into OFM 514 and OFM 517 in the process of training student network 510 using teacher network 520 .

[0086] In some embodiments, the training of the student network 510 also includes a process of minimizing the difference between the label 519 and the ground truth label of the training sample, based on which the student network 510 generates the label 519. For example, the internal parameters of the modified student network 510 are used to minimize the combination of the aggregated feature distance and the difference between the label 519 and the ground truth label.

[0087] Example DL Environment

[0088] Figure 6 A DL environment 600 according to various embodiments is shown. The DL environment 600 includes a DL server 610 and a plurality of client devices 620 (referred to as client devices 620, respectively). The DL server 610 is connected to the client devices 620 via a network 630. In other embodiments, the DL environment 600 may include fewer, more, or different components.

[0089] The DL server 610 uses a neural network to train a DL model. The construction of a neural network is similar to that of a human brain and is composed of artificial neurons (also referred to as nodes). These nodes are stacked adjacent to each other in the following three types of layers: an input layer, a hidden layer, and an output layer. Data provides information to each node in the form of input. The node multiplies the input with a random weight, calculates it, and adds a bias. Finally, a nonlinear function (also referred to as an activation function) is applied to determine which neuron to excite. The DL server 610 can use various types of neural networks, such as DNN, recursive neural network (RNN), generative adversarial network (GAN), long short-term memory network (LSTMN), etc. During the process of training a DL model, the neural network uses unknown elements in the input distribution to extract features, group objects, and discover useful data patterns. DL models can be used to solve various problems, such as making predictions, classifying images, etc. The DL server 610 can build a DL model specific to a specific type of problem that needs to be solved. The DL model is trained to receive input and output a solution to a specific problem.

[0090] exist Figure 6 In the embodiment, the DL server 610 includes a DNN system 640, a database 650, and a distributor 660. The DNN system 640 trains the DNN. The DNN can be used to process images, such as images captured by autonomous vehicles, medical devices, satellites, etc. In an embodiment, the DNN receives an input image and outputs a classification of an object in the input image. An example of a DNN is the above combined Figure 1 DNN100 described above or combined Figure 4 or Figure 5 The student network 410 or the student network 510 described above. In some embodiments, the DNN system 640 trains the DNN by knowledge distillation (e.g., knowledge distillation based on dense connections). The trained DNN can be used in low-memory systems, such as mobile phones, IOT edge devices, etc. An embodiment of the DNN system 640 is a combination of the above Figure 2 Described DNN system 200.

[0091] The database 650 stores data received, used, generated, or otherwise associated with the DL server 610 by the DL server 610. For example, the database 650 stores a training data set used by the DNN system 640 to train the DNN. In an embodiment, the training data set is an image library that can be used to train the DNN to classify images. The training data set may include data received from the client device 620. As another example, the database 650 stores hyperparameters of a neural network constructed by the DL server 610.

[0092] The distributor 660 distributes the DL model generated by the DL server 610 to the client device 620. In some embodiments, the distributor 660 receives a request for a DNN from the client device 620 via the network 630. The request may include a description of a problem that the client device 620 needs to solve. The request may also include information about the client device 620, for example, information describing the available computing resources on the client device. The information describing the available computing resources on the client device 620 may be information indicating network bandwidth, information indicating available memory size, information indicating the processing power of the client device 620, and the like. In an embodiment, the distributor may instruct the DNN system 640 to generate a DNN according to the request. The DNN system 640 may generate a DNN based on the information in the request. For example, the DNN system 640 may determine the structure of the DNN and / or train the DNN according to the request.

[0093] In another embodiment, the distributor 660 may select a DNN from a set of pre-existing DNNs based on the request. The distributor 660 may select a DNN for a particular client device 630 based on the size of the DNN and the available resources of the client device 620. In embodiments where the distributor 660 determines that the client device 620 has limited memory or processing power, the distributor 660 may select a compressed DNN for the client device 620 instead of an uncompressed DNN having a larger size. The distributor 660 then sends the DNN generated or selected for the client device 620 to the client device 620.

[0094] In some embodiments, the distributor 660 can receive feedback from the client device 620. For example, the distributor 660 receives new training data from the client device 620 and can send the new training data to the DNN system 640 to further train the DNN. As another example, the feedback includes an update of available computer resources on the client device 620. The distributor 660 can send a different DNN to the client device 620 based on the update. For example, after receiving feedback indicating that the computing resources of the client device 620 have been reduced, the distributor 660 sends a DNN with a smaller size to the client device 620.

[0095] The client device 620 receives the DNN from the distributor 660 and applies the DNN to perform machine learning tasks, such as solving problems or answering questions. In various embodiments, the client device 620 inputs images into the DNN and uses the output of the DNN for various applications, such as visual reconstruction, augmented reality, robot positioning and navigation, medical diagnosis, weather prediction, etc. The client device 620 can be one or more computing devices capable of receiving user input and sending and / or receiving data via the network 630. In one embodiment, the client device 620 is a conventional computer system, such as a desktop computer or a laptop computer. Alternatively, the client device 620 can be a device with computer functions, such as a personal digital assistant (PDA), a mobile phone, a smart phone, an autonomous vehicle, or another suitable device. The client device 620 is configured to communicate via the network 630. In one embodiment, the client device 620 executes an application that allows a user of the client device 620 to interact with the DL server 610 (e.g., the distributor 660 of the DL server 610). The client device 620 can request the DNN or send feedback to the distributor 660 through the application. For example, the client device 620 executes a browser application to enable interaction between the client device 620 and the DL server 610 via the network 630. In another embodiment, the client device 620 executes a browser application on a local operating system of the client device 620 (e.g., or ANDROID TM ) interacts with the DL server 610 through an application programming interface (API) running on the DL server 610.

[0096] In an embodiment, client device 620 is an integrated computing device that operates as an independent network-enabled device. For example, client device 620 includes a display, a speaker, a microphone, a camera, and an input device. In another embodiment, client device 620 is a computing device for coupling to an external media device such as a television or other external display and / or an audio output system. In this embodiment, client device 620 can be coupled to the external media device via a wireless interface or a wired interface (e.g., an HDMI cable), and various functions of the external media device can be utilized, for example, the display, speaker, microphone, camera, and input device of the external media device. Here, client device 620 can be configured to be compatible with a general-purpose external media device that does not have dedicated software, firmware, or hardware specifically for interacting with client device 620.

[0097] The network 630 supports communication between the DL server 610 and the client device 620. The network 630 may include any combination of a local area network and / or a wide area network using both wired and / or wireless communication systems. In one embodiment, the network 630 may use standard communication technologies and / or protocols. For example, the network 630 may include communication links using technologies such as Ethernet, 8010.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, Code Division Multiple Access (CDMA), Digital Subscriber Line (DSL), etc. Examples of network protocols for communicating via the network 630 may include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). The data exchanged by the network 630 may be represented using any suitable format (e.g., Hypertext Markup Language (HTML) or Extensible Markup Language (XML)). In some embodiments, any suitable one or more technologies may be used to encrypt all or some of the communication links in the communication links of the network 630.

[0098] Example method for training a neural network

[0099] Figure 7 is a flowchart illustrating a method 700 for training a DNN by knowledge distillation according to various embodiments. The method 700 may be Figure 2 Although reference Figure 7 The flowchart shown describes method 700, but many other methods for training DNNs by densely connected knowledge distillation may be used instead. For example, one may change Figure 7 As another example, some of the steps may be changed, eliminated, or combined.

[0100] The training module 250 inserts 710 the first layer into the target neural network by placing the first layer after the convolutional layer in the target neural network. The first layer is configured to convert the OFM of the convolutional layer into an extended OFM. In some embodiments, the target neural network includes a sequence of convolutional layers, the sequence of convolutional layers including the convolutional layer. The convolutional layer is the last convolutional layer in the sequence.

[0101] The training module 250 inserts 720 a second layer into the target neural network by placing the second layer after the first layer in the target neural network. The second layer is configured to convert the extended OFM into a new OFM. The extended OFM includes more channels than the OFM and the new OFM.

[0102] The training module 250 trains 730 the first layer based on a support feature map from a support neural network, wherein the support neural network is separate from the target neural network. In some embodiments, the training module 250 divides the extended OFM into a plurality of segments. The number of channels in the segment is the same as the number of channels in the support feature map. The first layer may be configured to convert the OFM of the convolutional layer into an extended OFM by performing a convolution operation on the OFM and the convolution kernel. The training module 250 may train the first layer by modifying one or more filters in the target neural network. For example, the training module 250 may adjust the convolution kernel to minimize the feature distance between the support feature map and the plurality of segments of the extended feature map. In some embodiments, the training module 250 may determine, for each respective segment in the plurality of segments, a segment feature distance between the respective segment and the support feature map. The feature distance may be an aggregation of the segment feature distances of the plurality of segments.

[0103] After training the first layer, the training module 250 merges 740 the first layer and the second layer into a layer in the target neural network. The layer is after the convolutional layer in the target neural network. The layer can be a fully connected layer or another convolutional layer. In some embodiments, the training module 250 inputs the training sample into the target neural network. The layer outputs a determination based on the training sample. The training module 250 can further train the first layer based on the determination and the ground truth label associated with the training sample.

[0104] Example computing device

[0105] Figure 8 is a block diagram of an example computing device 800 according to various embodiments. Figure 8 800, but any one or more of these components may be omitted or repeated as appropriate for the application. In some embodiments, some or all of the components included in the computing device 800 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-chip (SoC) die. Additionally, in various embodiments, the computing device 800 may not include Figure 8800 may include one or more of the components shown in , but the computing device 800 may include interface circuitry for coupling to one or more of the components. For example, the computing device 800 may not include the display device 806, but may include display device interface circuitry (e.g., connector and driver circuitry), to which the display device 806 may be coupled. In another set of examples, the computing device 800 may not include the audio input device 818 or the audio output device 808, but may include audio input or output device interface circuitry (e.g., connector and support circuitry), to which the audio input device 818 or the audio output device 808 may be coupled.

[0106] The computing device 800 may include a processing device 802 (e.g., one or more processing devices). As used herein, the term "processing device" or "processor" may refer to any device or portion of a device that processes electronic data from registers and / or memory to transform the electronic data into other electronic data that can be stored in registers and / or memory. The processing device 802 may include one or more digital signal processors (DSPs), application specific ICs (ASICs), CPUs, GPUs, cryptographic processors (specialized processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices. The computing device 800 may include a memory 804, which may itself include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), flash memory, solid-state memory, and / or a hard drive. In some embodiments, the memory 804 may include a memory that shares a die with the processing device 802. In some embodiments, the memory 804 includes one or more non-transitory computer-readable media storing instructions that are executable to implement operations for training a DNN, such as the above in conjunction with Figure 7 The method 700 described above or in combination with Figure 2 The operations performed by the described DNN system 200 (eg, operations performed by the training module 250 ) may be performed by the processing device 802 . Instructions stored in one or more non-transitory computer-readable media may be executed by the processing device 802 .

[0107] In some embodiments, the computing device 800 may include a communication chip 812 (e.g., one or more communication chips). For example, the communication chip 812 may be configured to manage wireless communications to transmit data to and from the computing device 800. The term "wireless" and its derivatives may be used to describe circuits, devices, DNN accelerators, methods, techniques, communication channels, etc., which can transmit data through a non-solid medium using modulated electromagnetic radiation. The term does not imply that the associated devices do not contain any wires, although in some embodiments these devices may not contain wires.

[0108] The communication chip 812 can implement any of a plurality of wireless standards or protocols, including but not limited to the Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.13 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), Long Term Evolution (LTE) projects, and any amendments, updates, and / or revisions (e.g., Advanced LTE projects, Ultra Mobile Broadband (UMB) projects (also known as "3GPP2"), etc.). IEEE 802.16-compatible broadband wireless access (BWA) networks are commonly referred to as WiMAX (the abbreviation stands for Worldwide Interoperability for Microwave Access) networks, which are certification marks for products that have passed the IEEE 802.16 standard conformance and interoperability tests. The communication chip 812 can operate according to a Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication chip 812 may operate according to Enhanced Data GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 812 may operate according to CDMA, Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution Data Optimized (EV-DO) and its derivatives, and any other wireless protocol designated as 3G, 4G, 5G and higher. In other embodiments, the communication chip 812 may operate according to other wireless protocols. The computing device 800 may include an antenna 822 for facilitating wireless communications and / or receiving other wireless communications (e.g., AM or FM radio transmissions).

[0109] In some embodiments, the communication chip 812 can manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 812 may include multiple communication chips. For example, the first communication chip 812 may be dedicated to shorter-range wireless communications (e.g., Wi-Fi or Bluetooth), and the second communication chip 812 may be dedicated to longer-range wireless communications (e.g., Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or other). In some embodiments, the first communication chip 812 may be dedicated to wireless communications, and the second communication chip 812 may be dedicated to wired communications.

[0110] Computing device 800 may include battery / power circuitry 814. Battery / power circuitry 814 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of computing device 800 to an energy source separate from computing device 800 (e.g., AC line power).

[0111] Computing device 800 may include display device 806 (or corresponding interface circuitry, as described above). Display device 806 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.

[0112] Computing device 800 may include audio output device 808 (or corresponding interface circuitry, as described above). Audio output device 808 may include any device that generates an audible indicator, such as a speaker, a headset, or earphones.

[0113] Computing device 800 may include audio input device 818 (or corresponding interface circuitry, as described above). Audio input device 818 may include any device that generates a signal representative of sound, such as a microphone, a microphone array, or a digital musical instrument (e.g., a musical instrument with a Musical Instrument Digital Interface (MIDI) output).

[0114] Computing device 800 may include GPS device 816 (or corresponding interface circuitry, as described above). GPS device 816 may communicate with a satellite-based system and may receive the location of computing device 800, as is known in the art.

[0115] Computing device 800 may include another output device 813 (or corresponding interface circuitry, as described above). Examples of other output devices 813 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0116] The computing device 800 may include another input device 820 (or corresponding interface circuitry, as described above). Examples of other input devices 820 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (e.g., a mouse), a stylus, a touchpad, a barcode reader, a quick response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0117] The computing device 800 may have any desired form factor, such as a handheld or mobile computer system (e.g., a mobile phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA, an ultra-mobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 800 may be any other electronic device that processes data.

[0118] Selected Examples

[0119] The following paragraphs provide various examples of the embodiments disclosed herein.

[0120] Example 1 provides a method for training a target neural network, the method comprising: inserting a first layer into the target neural network by placing the first layer after a convolutional layer in the target neural network, the first layer being configured to convert the OFM of the convolutional layer into an extended OFM; inserting a second layer into the target neural network by placing the second layer after the first layer in the target neural network, the second layer being configured to convert the extended OFM into a new OFM, wherein the extended OFM includes more channels than the OFM and the new OFM; training the target neural network based on a support feature map from a support neural network, wherein the support neural network is separate from the target neural network; and after training the first layer, merging the first layer and the second layer into a layer in the target neural network, wherein the layer is after the convolutional layer in the target neural network.

[0121] Example 2 provides the method of Example 1, wherein training the target neural network based on the supporting feature map from the supporting neural network includes: dividing the extended OFM into multiple segments, wherein the number of channels in the segments is the same as the number of channels in the supporting feature map.

[0122] Example 3 provides the method described in Example 1 or 2, wherein the first layer is configured to: convert the OFM of the convolution layer into the extended OFM by performing a convolution operation on the OFM and a convolution kernel.

[0123] Example 4 provides the method of Example 3, wherein training the target neural network based on the supporting feature map from the supporting neural network includes: modifying one or more filters in the target neural network.

[0124] Example 5 provides the method of Example 4, wherein modifying the one or more filters in the target neural network includes: adjusting values ​​in the one or more filters to minimize feature distances between multiple segments of the support feature map and the extended feature map.

[0125] Example 6 provides the method described in Example 5, wherein adjusting the convolution kernel based on the extended OFM and a supporting feature map from a supporting neural network also includes: determining, for each corresponding fragment among the multiple fragments, a fragment feature distance between the corresponding fragment and the supporting feature map, wherein the feature distance is an aggregation of the fragment feature distances of the multiple fragments.

[0126] Example 7 provides the method of any of the preceding examples, wherein the target neural network comprises a sequence of convolutional layers, the sequence of convolutional layers comprises the convolutional layer, and the convolutional layer is the last convolutional layer in the sequence.

[0127] Example 8 provides the method of any of the preceding examples, wherein the layer is a fully connected layer.

[0128] Example 9 provides the method of any of the preceding examples, wherein the layer is another convolutional layer.

[0129] Example 10 provides the method of any of the preceding examples, further comprising: inputting a training sample into the target neural network, the layer output being a determination made based on the training sample; and further training the first layer based on the determination and a ground truth label associated with the training sample.

[0130] Example 11 provides one or more non-transitory computer-readable media storing instructions that are executable to implement operations for training a target neural network, the operations comprising: inserting a first layer into the target neural network by placing the first layer after a convolutional layer in the target neural network, the first layer being configured to convert an OFM of the convolutional layer into an extended OFM; inserting a second layer into the target neural network by placing the second layer after the first layer in the target neural network, the second layer being configured to convert the extended OFM into a new OFM, wherein the extended OFM includes more channels than the OFM and the new OFM; training the target neural network based on a support feature map from a support neural network, wherein the support neural network is separate from the target neural network; and after training the first layer, merging the first layer and the second layer into a layer in the target neural network, wherein the layer is after the convolutional layer in the target neural network.

[0131] Example 12 provides one or more non-transitory computer-readable media of Example 11, wherein training the target neural network based on the supporting feature map from the supporting neural network includes: dividing the extended OFM into a plurality of segments, wherein the number of channels in the segments is the same as the number of channels in the supporting feature map.

[0132] Example 13 provides one or more non-transitory computer-readable media of Example 12, wherein the first layer is configured to convert the OFM of the convolutional layer into the extended OFM by performing a convolution operation on the OFM and a convolution kernel.

[0133] Example 14 provides the one or more non-transitory computer-readable media of Example 13, wherein training the target neural network based on the supporting feature map from the supporting neural network comprises modifying one or more filters in the target neural network.

[0134] Example 15 provides one or more non-transitory computer-readable media of Example 14, wherein modifying the one or more filters in the target neural network comprises adjusting values ​​in the one or more filters to minimize feature distances between multiple segments of the support feature map and the extended feature map.

[0135] Example 16 provides one or more non-transitory computer-readable media described in Example 15, wherein adjusting the convolution kernel based on the extended OFM and a supporting feature map from a supporting neural network also includes: determining, for each corresponding fragment of the multiple fragments, a fragment feature distance between the corresponding fragment and the supporting feature map, wherein the feature distance is an aggregation of the fragment feature distances of the multiple fragments.

[0136] Example 17 provides one or more non-transitory computer-readable media of any of Examples 11-16, wherein the target neural network includes a sequence of convolutional layers, the sequence of convolutional layers includes the convolutional layer, and the convolutional layer is the last convolutional layer in the sequence.

[0137] Example 18 provides one or more non-transitory computer-readable media of any of Examples 11-17, wherein the layer is a fully connected layer.

[0138] Example 19 provides one or more non-transitory computer-readable media of any of Examples 11-18, wherein the layer is another convolutional layer.

[0139] Example 20 provides one or more non-transitory computer-readable media of any of Examples 11-19, wherein the operation further includes: inputting training samples into the target neural network, the layer output being a determination made based on the training samples; and further training the first layer based on the determination and ground truth labels associated with the training samples.

[0140] Example 21 provides an apparatus for training a target neural network, the apparatus comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions, the computer program instructions executable by the computer processor to implement operations including: inserting a first layer into the target neural network by placing the first layer after a convolutional layer in the target neural network, the first layer being configured to convert an OFM of the convolutional layer into an extended OFM; inserting a second layer into the target neural network by placing the second layer after the first layer in the target neural network, the second layer being configured to convert the extended OFM into a new OFM, wherein the extended OFM includes more channels than the OFM and the new OFM; training the target neural network based on a support feature map from a support neural network, wherein the support neural network is separate from the target neural network; and after training the first layer, merging the first layer and the second layer into a layer in the target neural network, wherein the layer is after the convolutional layer in the target neural network.

[0141] Example 22 provides the apparatus of Example 21, wherein training the target neural network based on the supporting feature map from the supporting neural network includes: dividing the extended OFM into a plurality of segments, wherein the number of channels in the segments is the same as the number of channels in the supporting feature map.

[0142] Example 23 provides the apparatus of Example 21 or 22, wherein the first layer is configured to convert the OFM of the convolutional layer into the extended OFM by performing a convolution operation on the OFM and a convolution kernel.

[0143] Example 24 provides the apparatus of any of Examples 21-23, wherein the target neural network comprises a sequence of convolutional layers, the sequence of convolutional layers includes the convolutional layer, and the convolutional layer is the last convolutional layer in the sequence.

[0144] Example 25 provides an apparatus as described in any of Examples 21-24, wherein the operation further includes: inputting a training sample into the target neural network, the layer output being a determination made based on the training sample; and further training the first layer based on the determination and a ground truth label associated with the training sample.

[0145] The above description of the illustrative implementation of the present disclosure (including the content described in the abstract) is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Although the specific implementation and examples of the present disclosure are described herein for illustrative purposes, as will be appreciated by those skilled in the relevant art, various equivalent modifications are possible within the scope of the present disclosure. These modifications may be made to the present disclosure in view of the above detailed description.

Claims

1. A method for training a target neural network, the method comprising: inserting the first layer into the target neural network by placing the first layer after a convolutional layer in the target neural network, the first layer being configured to convert an output feature map of the convolutional layer into an expanded output feature map; inserting the second layer into the target neural network by placing the second layer after the first layer in the target neural network, the second layer being configured to convert the extended output feature map into a new output feature map, wherein the extended output feature map includes more channels than the output feature map and the new output feature map; training the target neural network based on support feature maps from a support neural network, wherein the support neural network is separate from the target neural network; and After training the first layer, merging the first layer and the second layer into a layer in the target neural network, wherein the layer is after the convolutional layer in the target neural network.

2. The method according to claim 1, wherein: Training the target neural network based on the supporting feature map from the supporting neural network includes: Dividing the extended output feature map into a plurality of segments, The number of channels in the segment is the same as the number of channels in the support feature map.

3. The method according to claim 1, wherein: The first layer is configured as: The output feature map of the convolution layer is converted into the expanded output feature map by performing a convolution operation on the output feature map and a convolution kernel.

4. The method according to claim 3, wherein: Training the target neural network based on the supporting feature map from the supporting neural network includes: One or more filters in the target neural network are modified.

5. The method according to claim 4, wherein: Modifying the one or more filters in the target neural network includes: Values ​​in the one or more filters are adjusted to minimize feature distances between the support feature map and a plurality of segments of the extended feature map.

6. The method according to claim 5, wherein: Adjusting the convolution kernel based on the expanded output feature map and a supporting feature map from a supporting neural network further includes: For each corresponding segment of the plurality of segments, determining a segment feature distance between the corresponding segment and the supporting feature map, The characteristic distance is an aggregation of the fragment characteristic distances of the multiple fragments.

7. The method according to claim 1, wherein: The target neural network comprises a sequence of convolutional layers, the sequence of convolutional layers comprises the convolutional layer, and the convolutional layer is the last convolutional layer in the sequence.

8. The method according to claim 1, wherein: The layer is a fully connected layer.

9. The method according to claim 1, wherein: The layer is another convolutional layer.

10. The method according to claim 1, further comprising: Inputting a training sample into the target neural network, the layer outputs a determination based on the training sample; as well as The first layer is further trained based on the determining and ground truth labels associated with the training samples.

11. One or more non-transitory computer-readable media storing instructions executable to perform operations for training a target neural network, the operations comprising: inserting the first layer into the target neural network by placing the first layer after a convolutional layer in the target neural network, the first layer being configured to convert an output feature map of the convolutional layer into an expanded output feature map; inserting the second layer into the target neural network by placing the second layer after the first layer in the target neural network, the second layer being configured to convert the extended output feature map into a new output feature map, wherein the extended output feature map includes more channels than the output feature map and the new output feature map; training the target neural network based on support feature maps from a support neural network, wherein the support neural network is separate from the target neural network; and After training the first layer, merging the first layer and the second layer into a layer in the target neural network, wherein the layer is after the convolutional layer in the target neural network.

12. The one or more non-transitory computer-readable media of claim 11, wherein: Training the target neural network based on the supporting feature map from the supporting neural network includes: Dividing the extended output feature map into a plurality of segments, The number of channels in the segment is the same as the number of channels in the support feature map.

13. The one or more non-transitory computer-readable media of claim 12, wherein: The first layer is configured as: The output feature map of the convolution layer is converted into the expanded output feature map by performing a convolution operation on the output feature map and a convolution kernel.

14. The one or more non-transitory computer-readable media of claim 13, wherein: Training the target neural network based on the supporting feature map from the supporting neural network includes: One or more filters in the target neural network are modified.

15. The one or more non-transitory computer-readable media of claim 14, wherein: Modifying the one or more filters in the target neural network includes: Values ​​in the one or more filters are adjusted to minimize feature distances between the support feature map and a plurality of segments of the extended feature map.

16. The one or more non-transitory computer-readable media of claim 15, wherein: Adjusting the convolution kernel based on the expanded output feature map and a supporting feature map from a supporting neural network further includes: For each corresponding segment of the plurality of segments, determining a segment feature distance between the corresponding segment and the supporting feature map, The characteristic distance is an aggregation of the fragment characteristic distances of the multiple fragments.

17. The one or more non-transitory computer-readable media of claim 11, wherein: The target neural network comprises a sequence of convolutional layers, the sequence of convolutional layers comprises the convolutional layer, and the convolutional layer is the last convolutional layer in the sequence.

18. The one or more non-transitory computer-readable media of claim 11, wherein: The layer is a fully connected layer.

19. The one or more non-transitory computer-readable media of claim 11, wherein: The layer is another convolutional layer.

20. The one or more non-transitory computer-readable media of claim 11, wherein: The operations also include: Inputting a training sample into the target neural network, the layer outputs a determination based on the training sample; and The first layer is further trained based on the determining and ground truth labels associated with the training samples.

21. A device for training a target neural network, the device comprising: a computer processor for executing instructions of a computer program; as well as a non-transitory computer readable memory storing computer program instructions executable by the computer processor to perform operations including: inserting the first layer into the target neural network by placing the first layer after a convolutional layer in the target neural network, the first layer being configured to convert an output feature map of the convolutional layer into an expanded output feature map; inserting the second layer into the target neural network by placing the second layer after the first layer in the target neural network, the second layer being configured to convert the extended output feature map into a new output feature map, wherein the extended output feature map includes more channels than the output feature map and the new output feature map; training the target neural network based on support feature maps from a support neural network, wherein the support neural network is separate from the target neural network; and After training the first layer, merging the first layer and the second layer into a layer in the target neural network, wherein the layer is after the convolutional layer in the target neural network.

22. The device according to claim 21, wherein Training the target neural network based on the supporting feature map from the supporting neural network includes: Dividing the extended output feature map into a plurality of segments, The number of channels in the segment is the same as the number of channels in the support feature map.

23. The device according to claim 21, wherein The first layer is configured as: The output feature map of the convolution layer is converted into the expanded output feature map by performing a convolution operation on the output feature map and a convolution kernel.

24. The device according to claim 21, wherein The target neural network comprises a sequence of convolutional layers, the sequence of convolutional layers comprises the convolutional layer, and the convolutional layer is the last convolutional layer in the sequence.

25. The device according to claim 21, wherein The operations also include: Inputting a training sample into the target neural network, the layer outputs a determination based on the training sample; and The first layer is further trained based on the determining and ground truth labels associated with the training samples.

Images