Extended neural network training layer for convolution
By introducing an extended training layer in the convolutional neural network, combining the K×K filter and 1×1 filter, the problem of increasing inference cost when improving performance of the CNN model is solved, and higher model accuracy and the same The inference cost is suitable for training multiple models.
Patent Information
- Application Number
- CN202280094892.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-16
- Publication Date
- 2024-12-06
AI Technical Summary
Existing convolutional neural network (CNN) models often increase inference costs when improving performance, making it less attractive when performing inference on low-performance processors, and techniques that improve model performance increase computational load, making it difficult to Improve model performance without increasing inference costs.
By introducing an extended training layer, replacing the filter in the convolution layer, combining the K×K filter and 1×1 filter to form a new training model, using the parameter combination of the extended training layer to determine the parameters of the trained model, Keep the inference cost unchanged while improving model performance.
Significantly improves model accuracy while maintaining the same inference cost, it is suitable for training various CNN models, compatible with multiple model training schemes, and provides significant benefits for shallow models.
Smart Images

Figure CN119096266A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to computer modeling and, more particularly, to training neural network models with convolutional filters. Background Art
[0002] Convolutional Neural Network (CNN) has become the main learning model for processing various artificial intelligence (AI) applications such as image classification, face recognition, scene understanding and Go game. Current technology trends show that CNN architectures are becoming more and more complex, with increasing depth and complexity. Although the most advanced CNN architectures may include various techniques for improving model training and accuracy, such models usually become more and more complex and are used for reasoning (e.g., applying a trained model to an input to generate an output) at the expense of increasing the cost of executing the model. Therefore, many techniques for improving CNN performance also increase the runtime reasoning cost, which may be less attractive when reasoning is performed on a lower performance processor or when the increased computational load after the improvement needs to be compromised with other processes competing for processing capacity. Techniques that improve model performance without increasing the cost of reasoning may therefore provide substantial benefits. BRIEF DESCRIPTION OF THE DRAWINGS
[0003] The embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To aid this description, similar reference numerals designate similar structural elements. The embodiments are illustrated in the accompanying drawings by way of example and not by way of limitation.
[0004] Figure 1 is an example process for training parameters of a model architecture using extended training layers according to one embodiment.
[0005] Figure 2 Another example of extending the training layer according to one embodiment is shown.
[0006] Figure 3 An example training model architecture of convolutional layers according to one embodiment is shown.
[0007] Figure 4 Example computer model inference and computer model training are shown.
[0008] Figure 5 An example neural network architecture is illustrated.
[0009] Figure 6is a block diagram of an example computing device that may include one or more components for training, analyzing, or implementing a computer model according to any embodiment disclosed herein. DETAILED DESCRIPTION
[0010] Overview
[0011] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for all of the desirable attributes disclosed herein. The details of one or more implementations of the subject matter described in this specification are set forth in the following description and drawings.
[0012] The present disclosure provides a scheme for training convolutional layers that has significantly better model accuracy while not adding additional computational cost to inference (i.e., maintaining the same type of target model at inference time). The inference model architecture can be a convolutional layer of several K×K convolutional filters (also called convolutional kernels) that produce several output channels in the output data of the layer. K can be any appropriate value, for example, 3, 5, 7, and so on. The inference model architecture is extended to a training model architecture, and the convolutional layer (or at least some of the convolutional filters) is replaced by an extended training layer. The extended training layer includes a layer of K×K filters and a layer of 1×1 filters. For the inference model architecture, the number of K×K filters in the extended training layer may exceed the number of convolutional filters in the convolutional layer. The number of 1×1 filters in the 1×1 layer may match the number of convolutional filters in the training layer so that the output of the 1×1 filter is a number of channels that matches the number of output channels. The K×K layer can be viewed as learning a number of different convolutional filters, which may exceed the number of convolutional filters of the inference model, while the 1×1 layer can be viewed as learning a weighted combination of the resulting data from the expanded K×K layers. In order to apply the learned values to the inference model architecture, the values of the expanded training layers are "absorbed" so that the parameters from the expanded training layers are combined into mathematically equivalent values as the K×K filters of the trained inference model for inference. In other words, due to the structure of the expanded layers, the parameters of a particular output channel in the trained convolutional layer can be determined directly from the trained parameters of the expanded training layer without the expected loss of mathematical accuracy.
[0013] Given a CNN architecture built with regular convolutions or their variants, this scheme transforms the regular convolution kernel of any convolution layer into two (or more) sequential convolution layers of K×K convolution kernels and 1×1 convolution kernels during training, while transforming it back to a single convolution layer with regular K×K convolution kernels, thereby enjoying improved final model performance while also maintaining the same inference cost. This scheme is also compatible with many other model training schemes.
[0014] Therefore, this "absorbable" convolution is a general design for improving the training of convolutional filters of neural networks. It can be easily used to train these models with better model accuracy while adding no additional computational cost to inference (i.e., keeping the same type of target CNN model at inference time).
[0015] For purposes of illustration, specific numbers, materials, and configurations are described to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be implemented without the specific details, or / and may be implemented with only some of the described aspects. In other cases, well-known features are omitted or simplified so as not to obscure the illustrative implementations.
[0016] In the following detailed description, reference is made to the accompanying drawings, which form a part of this document, in which embodiments that can be implemented are shown by way of illustration. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be understood in a limiting sense.
[0017] Various operations can be described in turn as multiple discrete actions or operations in a manner that is most helpful for understanding the claimed subject matter. However, the order of description should not be interpreted as meaning that these operations must be dependent on the order. Specifically, these operations may not be performed in the order presented. The operations described may be performed in an order different from the described embodiments. Various additional operations may be performed and / or the operations described may be omitted in additional embodiments.
[0018] For purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). The term "between" when used in reference to a measurement range includes both ends of the measurement range. The meanings of "a", "an", and "the" include a plurality of references. The meaning of "in" includes "in" and "on".
[0019] The description uses the phrases "in one embodiment" or "in an embodiment", each of which refers to one or more of the same or different embodiments. In addition, the terms "including", "comprising", "having", etc. used with respect to the embodiments of the present disclosure are synonymous. The present disclosure may use perspective-based descriptions, such as "above", "below", "top", "bottom", and "side"; such descriptions are used to facilitate discussion and are not intended to limit the application of the disclosed embodiments. The drawings are not necessarily drawn to scale. The terms "substantially", "close", "roughly", "approximately", and "approximately" generally refer to within + / -20% of the target value. Unless otherwise specified, the use of ordinal adjectives "first", "second", and "third", etc. to describe common objects simply indicates that different instances of similar objects are referenced, and is not intended to imply that the objects described in this way must be in a given sequence in time, space, ranking, or in any other way.
[0020] In the detailed description that follows, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.
[0021] Dilated convolutions in training layers
[0022] Figure 1 is an example process for training parameters of a model architecture using an extended training layer according to one embodiment. The target model architecture 100 includes a convolutional layer including convolutional filters 120 that process layer input data 110 into layer output data 130. When the model applies the learned parameters to the input data to generate an output, the target model architecture 100 is the model architecture to be used in the deployment of the model. As described below for Figure 4-Figure 5 As discussed, computer models typically include parameters for processing inputs to predict outputs. Such computer models can be iteratively trained to learn specific parameters, including weights, for predicting various outputs based on input data. Figure 5 As further discussed in, individual layers in a neural network can receive input activations and process the input activations to generate output activations for each layer. Computer models that can be trained using the disclosed schemes may include the types discussed below, including various types of neural networks that include at least one convolutional layer.
[0023] The layer input data 110 represents the data input to the model, which may be the activations from a previous layer (e.g., the output of a previous layer). Similarly, the layer output data 130 represents the output of the convolutional filters 120 applied to the layer input data 110, and is the output of the next layer in the target model architecture 100. The layer input data 110 is referred to as a matrix F, having a height and width H×W, and a depth representing the number of channels of the layer input data 110. The number of channels in the layer input data 110 is designated as c0. The layer input data 110 applied to each convolutional filter 120 generates a respective channel for the received input. The convolutional filters 120 may be applied to different portions of the input to generate different corresponding outputs, which together form the output of the layer output data 130, represented here as the matrix F B .
[0024] The number of convolutional filters 120 is denoted as c2, resulting in a corresponding number of channels c2 of the layer output data 130. A convolutional layer also has a size that indicates the size of the input processed by each filter. Figure 1 In the example of , the size of the convolution filter 120 is 3×3, indicating that it receives channels of a 3×3 region of the layer input data 110. The size of the filter may represent a portion of the height and width of the matrix received by the convolution filter. Thus, a filter of size 3×3 may receive and process a 3×3×c0 portion of the input from the layer input data 110. Since the layer input data 110 has a number of channels c0, the convolution filter 120 may also have parameters to be learned for each channel c0 on the input size. The size of the convolution filter 120 may be 3×3, 5×5, 7×7, and so on, indicating different sizes of the layer input data 110 to be processed by the layer when a given convolution layer is applied to a portion of the layer input data 110. The size is typically square and may be generally represented as K×K, so that the number of convolution filters (c2) each processes a K×K×c0 matrix using the corresponding parameters of the filter. The convolution filter 120 in the convolution layer of the target model architecture 100 may also be referred to as D. Thus, the layer output data 130 may be defined as the layer input data 110 convolved according to the following convolution filter: B =F*D, where * represents the convolution operation. For one channel of the output, the channel can be given by the following formula:
[0025]
[0026] As shown in Equation 1, each output channel is the sum of each element in the convolutional filter multiplied by the corresponding element of the layer input data.
[0027] In various embodiments, rather than directly training the parameters of the target model architecture 100, the convolutional filters 120 of the convolutional layers are expanded into two or more expanded training layers 140 in the training model architecture 102. After the parameters of the training model architecture 102 are trained, the parameters of the convolutional layers are determined by combining the parameters of the expanded training layers 140. Thus, the training model architecture 102 provides additional layers relative to the target model architecture, and after training, the parameters of the expanded training layers 140 are "absorbed" to determine the parameters of the trained inference model 104. As described below, the combined parameters of the trained inference model 104 are equivalent to the expanded training layers 140. This allows the training model architecture 102 to use the additional expanded training layers 140 (and additional parameters) to provide additional "space", which can enable the training model architecture 102 to more accurately learn the training target while still being effectively combined for the convolutional filters 120 without the expected mathematical loss.
[0028] like Figure 1 As shown, in the training model architecture 102, the convolution layer D is replaced by extended training layers 140A and 140B, including filters of size 3×3 (e.g., K×K) and 1×1, respectively. The extended training layer 140B can have the same number of filters as the convolution filter 120 (i.e., c2), so that the output of the extended training layer 140B is the same layer output data 130 with channels c2. The extended training layer 140B can thus be regarded as an output extension layer, so that its output will be the learned output of the convolution layer in the inference network. The input of the extended training layer 140B is the extended data layer 150, which is generated as the output of the extended training layer 140A. That is, the extended training layer 140A receives the layer input data 110, generates channels for the extended data layer 150, and the extended training layer 140B processes these channels to generate the layer output data 130. The extended data layer 150 thus becomes an intermediate data layer, which represents the channels from the output of the extended training layer 140B. As Figure 1 As shown, the number of filters c1 of the extended training layer 140B produces an extended data layer 150 having the same number of channels c1. The extended training layer 140A can receive as input a set of c1 channels produced in the 1×1 convolution. In a sense, each filter in the 1×1 convolution of the extended training layer 140A thus provides a weighted combination of the respective K×K convolutions of the extended training layer 140B.
[0029] In one embodiment, the number of filters c1 of the extended training layer 140A is the same as the number of filters of the convolution filter 120 and the extended training layer 140B (e.g., c1=c2). In other embodiments, the number of filters c1 of the extended training layer 140A may be significantly higher, such as two or three times the number of filters (c1>c2). Therefore, when the number of filters c1 increases, many additional filters can be learned, and then these additional filters can be combined with the learned convolution weights according to c2 to generate layer output data 130. When the parameters are combined into the trained inference model 104 after training, the parameters of the resulting convolution filter 120 can benefit from the additional filters in the extended training layer 140A. In this example, the extended training layer 140A can thus have the same dimensions (K×K) as the convolution filter 120, while allowing the number of such filters to vary and be integrated by the extended training layer 140B.
[0030] In addition, although Figure 1 The expansion of the convolution filter 120 is shown for one convolution layer in , but any number (or all) of the convolution layers in the target model architecture 100 can be similarly expanded for training in the training model architecture 102 and converted into parameters of the trained inference model 104.
[0031] After generating the training model architecture 102 with the extended training layer 140, the training model architecture 102 can be trained with relevant training data, and the extended layer can be included in the training model architecture 102 as an inline replacement for the convolution filter 120. Therefore, although the extended training layer 140 can add an additional layer to the trained inference model 104, it does not otherwise affect the training process of the trained inference model 104, and normal model training schemes can be applied. The training model architecture 102 can be trained, as shown in reference Figure 4-Figure 5 described.
[0032] After training, the parameters learned for the extended training layers 140A-B are combined so that the corresponding parameters are "absorbed" to form the parameters of the convolution filter 120. That is, since the 1×1 convolution of the extended training layer 140B combines the K×K convolutions of the extended training layer 140A, a mathematically equivalent result can be generated by combining the inputs of the filters in the extended training layer 140A and the weights of the c0 channels according to the c1 weight channels in the extended training layer 140B. In the following example formula for determining the weights of the channels of the matrix D of the convolution filter 120, the filter of the extended training layer 140A is designated as the matrix A, and the filter of the extended training layer 140B is designated as the matrix B:
[0033]
[0034] in:
[0036] A corresponding version of Formula 2 may be applied to determine the weights for the additional channels of the convolutional filters 120 of the trained inference model 104. Thus, the convolutional layers may be expanded for training, trained with the expanded training layers 140, and transformed back to the original model architecture for inference without loss of information between the expanded training model architecture and the trained inference model 104.
[0037] Figure 2 Another example of an extended training layer according to one embodiment is shown. Figure 1 As in the example of , the convolution filter 120 can be extended to the extended training layer 140 for training, and the resulting parameters can be combined for inference. In this example, the inference model architecture includes a normalization layer, such as batch normalization (BN) 200 in the inference model architecture. To include this layer in the training model architecture, multiple batch normalization layers can be included, namely batch normalization 210A after the extended training layer 140A, and batch normalization 210B after the extended training layer 140B. Figure 1 Similarly, the training model architecture is trained and the parameters of batch normalization 210A-B are combined to generate the parameters of batch normalization 220 for inference. In addition to batch normalization, other types of normalization can also be used similarly, such as instance normalization (IN), group normalization (GN), etc. These different types of normalization can modify the dimensions in which normalization is performed.
[0038] To further illustrate batch normalization, Equation 3 shows the formal representation of BN200 applied in the target model architecture 100:
[0039] F B =BN(D*F)
[0040] Formula 3
[0041] Therefore, batch normalization can be applied to the result (D*F) of the convolution operation between the layer input data 110 and the convolution filter 120 to generate the layer output data 130 (F B ).
[0042] Equation 4 illustrates the operation of batch normalization applied to one channel in the target model architecture 100:
[0043]
[0044] After training the training model architecture with the BN layers 210A, 210B, the combination of the BN layers and the expanded training layers can be formally given by Formula 5, which is similar to Formula 2 but modified to include the batch normalization layer:
[0045]
[0046] in and
[0047] O is an all-one matrix.
[0048] Formula 5
[0049] Figure 3 An example training model architecture of convolutional layers according to one embodiment is shown. Figure 1 The example of shows all filters of the expanded convolutional layer in the training model architecture 102, while Figure 3 In the example of , some convolution filters are not expanded in the training model architecture. In this example, the first part of the filters of the convolution layer are unexpanded filters 330, and a first set of output channels 320A can be output. The filters used to generate additional output channels 320B can be expanded to produce expanded training layers 340A-B.
[0050] The expansion of the convolutional filters can be performed more than once - in this example, the output channel 320B can be generated in the training model architecture by summing the results of the expanded training layers on different "branches". In this example, the first expanded training branch includes the expanded training layers 340A-B and the expanded data layer 350A, which is summed with the second expanded training branch, which includes the expanded training layers 340C-D and the expanded data layer 350B. When the training layers are "absorbed", the corresponding value of the model can be determined for each output channel in the set of output channels 320B by following the following operations: tracking the respective contributions of the layer input data 310 through the expanded training layers 340A-D, and forming the equivalent value of the filter for that output channel in the inference model.
[0051] Experimental Results
[0052] The extended training process is applied to various datasets for comparison. Specifically, the ImageNet-1k classification dataset is evaluated with ResNet and EfficientNet backbones. In these experiments, during training, all regular 3×3 convolutions are extended with c1=2c2, and during inference, the trained parameters are combined to obtain 3×3 parameters.
[0053]
[0054]
[0055] Table 1
[0056] As shown in Table 1, adding expansion layers and subsequently combining the trained parameters (“absorbable convolutions”) leads to significant improvements, especially for shallower models with fewer layers (e.g., ResNet18).
[0057] Similar experiments were performed with the EfficientNet backbone:
[0058]
[0059] Table 2
[0060] Table 2 confirms that expanding training layers provides similar benefits to EfficientNet and, like ResNet, provides the most significant benefits for smaller model architectures.
[0061] Example Computer Modeling
[0062] Figure 4 Example computer model inference and computer model training are shown. Computer model inference refers to applying a computer model 410 to a set of input data 400 to generate an output or model output 420. The computer model 410 determines the model output 420 based on parameters of the model, also referred to as model parameters. The parameters of the model can be determined based on a training process that finds an optimization of the model parameters, typically using training data and the expected output of the model for each training data, as described below. The output of the computer model can be referred to as "inference" because it is a predicted value based on the input data 400 and based on previous example data used in model training.
[0063] The input data 400 and the model output 420 vary depending on the specific use case. For example, for computer vision and image analysis, the input data 400 may be an image with a specific resolution, such as 75×75 pixels, or a point cloud describing a volume. In other applications, the input data 400 may include a vector, such as a sparse vector, representing information about an object. For example, in a recommendation system, such a vector may represent a user-object interaction, so that the sparse vector indicates an individual item that is given a positive rating by a user. In addition, the input data 400 may be a processed version of another type of input object, such as representing various features of the input object or representing the preprocessing of the input object before the object is input to the computer model 410. As an example, a 1024×1024 resolution image may be processed and subdivided into 64×64 individual image portions, which are the input data 400 processed by the computer model 410. As another example, an input object, such as the sparse vector discussed above, may be processed to determine an embedding or another compact representation of the input object, which may be used to represent the object as input data 400 in the computer model 410. This additional processing of the input objects may itself be a learned representation of the data, whereby another computer model processes the input objects to generate an output that is used as input data 400 to computer model 410. Although not discussed further herein, this further computer model may be trained independently or in conjunction with computer model 410.
[0064] As described above, model output 420 may depend on the specific application of computer model 410 and represent a recommendation system, a computer vision system, a classification system, a labeling system, weather prediction, autonomous control, and any other type of modeling output / prediction.
[0065] The computer model 410 includes various model parameters, which, as described above, describe the characteristics and functions of generating the model output 420 from the input data 400. Specifically, the model parameters may include the model structure, model weights, and the model execution environment. The model structure may include, for example, the specific type of computer model 410 and its structure and organization. For example, the model structure may specify a neural network, which may be composed of multiple layers, and the model parameters may describe the individual types of layers included in the neural network and the connections between the layers (e.g., which layers' outputs constitute which other layers' inputs). Such a network may include, for example, feature extraction layers, convolutional layers, pooling / dimensionality reduction layers, activation layers, output / prediction layers, and the like. While in some cases the model structure may be determined by the designer of the computer model, in other examples, the model structure itself may be learned via a training process, thereby forming certain "model parameters" of the model.
[0066] Model weights may represent values that the computer model 410 uses to process the input data 400 into the model output 420. Each portion or layer of the computer model 410 may have such weights. For example, weights may be used to determine values used to process inputs at a particular portion of the model to determine outputs. In other words, for example, model weights may describe how to combine or manipulate the values of the input data 400 or thresholds used to determine activation as outputs of the model. As an example, a convolutional layer typically includes a set of convolution "weights," also referred to as convolution kernels, to be applied to a set of inputs to the layer. These are then combined, typically along with "bias" parameters and other transformed weights, to generate an output for the convolutional layer.
[0067] Model execution parameters represent parameters describing the execution conditions of the model. Specifically, various aspects of the model can be implemented on various types of hardware or circuits for executing computer models. For example, parts of the model can be implemented in various types of circuits, such as general circuits (e.g., general CPUs), circuits specifically used for certain computer model functions (e.g., GPUs or programmable multiplication and accumulation circuits), or circuits specifically designed for specific computer model applications. In some configurations, different parts of the computer model 410 can be implemented on different types of circuits. As described below, the training of the model may include optimizing the type of hardware used for certain aspects of the computer model (e.g., co-training), or may be determined after determining other parameters of the computer model without considering the configuration of the execution model. In another example, the execution parameters may also determine or limit the types of processes or functions available in different parts of the model, such as the range of values available at certain points in the process, operations that can be used to perform tasks, and the like.
[0068] Computer model training can thus be used to determine or "train" values of model parameters of the computer model 440. During training, the model parameters are optimized to "learn" values of the model parameters (e.g., individual weights, activation values, model execution environment, etc.) based on an optimization function that seeks to improve a cost function (sometimes also referred to as a loss function) to improve the model parameters. Prior to training, the computer model 440 has model parameters that have initial values that can be selected in a variety of ways, such as by random initialization, initial values selected based on other or similar computer models, or by other means. During training, the model parameters are modified based on the optimization function to improve the cost / loss function relative to previous model parameters.
[0069] In many applications, training data 430 includes a data set used to train computer model 440. The data set varies depending on the specific application and purpose of computer model 440. In supervised learning tasks, training data typically includes a set of training data labels that describe the training data and the expected output of the model with respect to the training data. For example, for an object classification task, the training data may include individual images where individual parts, regions, or pixels in the image are labeled with the classification of the object. For this task, the training data may include training data images depicting dogs and people, and training data labels that mark the areas of the image that include dogs and people, so that the computer model is intended to learn to also label the same parts of the image as dogs and people, respectively.
[0070] To train the computer model, a training module (not shown) applies training data 430 to computer model 440 to determine the output predicted by the model for a given input of training data 430. Although not shown, a training module is a computational module for performing training of a computer model by executing the computer model according to its inputs and outputs given the parameters of the model and modifying the model parameters based on the results. The training module may apply an actual execution environment of computer model 440, or may simulate the results of an execution environment to, for example, estimate the performance, run time, memory, or circuit area of the computer model (e.g., if dedicated hardware is used). The training module, along with the training data and model evaluation, may be processed by one or more processing devices (e.g., Figure 6 The example computing device 600 shown is instantiated in software and / or hardware. In various examples, the training process can also be performed by multiple computing systems in coordination with each other, such as distributed / cloud computing systems.
[0071] After processing the training input according to the current model parameters of the computer model 440, the predicted output of the model is evaluated 450, and the computer model is evaluated for the cost function and optimized using the optimization function of the training model. Depending on the optimization function, the training parameters after the specific training process and model evaluation are updated to improve the optimization function of the computer model. In supervised training (i.e., training data labels are available), the cost function can evaluate the predicted output of the model relative to the training data labels, and evaluate the relative cost or loss of the prediction relative to the "known" labels of the data. This provides a measure of the correct prediction frequency of the computer model, and can be measured in various ways, such as accuracy (the frequency of false positives) and recall (the frequency of false negatives). The cost function can also evaluate other characteristics of the model in some cases, such as model complexity, processing speed, memory requirements, physical circuit characteristics (e.g., power requirements, circuit throughput) and other characteristics of the computer model structure and execution environment (e.g., to evaluate or modify these model parameters).
[0072] After determining the result of the cost function, the optimization function determines the modification of the model parameters to improve the cost function of the training data. Many such optimization functions are known to those skilled in the art. Many such schemes distinguish the cost function from the parameters of the model and determine the modification of the model parameters to improve the cost function. The parameters of the optimization function, including the algorithm for modifying the model parameters, are the training parameters of the optimization function. For example, the optimization algorithm can use gradient descent (or its variant), momentum-based optimization, or other optimization methods used in the art that are suitable for the specific purpose of the model. The optimization algorithm thus determines the parameter update of the model parameters. In some implementations, the training data is batched, and the parameter update is iteratively applied to the batch of training data. For example, the model parameters can be initialized and then applied to the first batch of data to determine the first modification of the model parameters. Then, the second batch of data can be evaluated with the modified model parameters to determine the second modification of the model parameters, and so on and so forth, until the stopping point, usually based on the amount of available training data or the incremental improvement in the model parameters is below a threshold (for example, additional training data no longer continues to improve the model parameters). Additional training parameters may describe the batch size of the training data, the portion of the training data used as validation data, the step size for parameter updates, the learning rate of the model, etc. Additional techniques may also be used to determine the global optimum or to cope with indifferentiable model parameter spaces.
[0073] Figure 5 An example neural network architecture is illustrated. In general, a neural network includes an input layer 510, one or more hidden layers 520, and an output layer 530. The value of the data in each layer of the network is generally determined based on one or more previous layers of the network. Each layer of the network generates a set of values, called "activations," which represent the output values of that layer of the network and can serve as input to the next layer of the network. For the input layer 510, the activations are typically the values of the input data, although the input layer 510 can represent the input data modified by one or more transformations to generate a representation of the input data. For example, in a recommendation system, the interactions between users and objects can be represented as a sparse matrix. Then, individual users or objects can be represented as input layer 510 as transformations of the data in the sparse matrix associated with the user or object. The neural network can also receive the output of another (or several) computer models as its input layer 510, so that, Figure 5 The input layer 510 of the neural network shown is the output of another computer model. Thus, each layer can receive a set of inputs, also called "input activations," representing the activations of one or more previous layers of the network, and generate a set of outputs, also called "output activations," representing the activations of that layer of the network. In other words, the output activations of one layer become the input activations of another layer of the network (except for the final output layer 530 of the network).
[0074] Each layer of a neural network typically represents its output activations (i.e., also called its outputs) in a matrix, which can be 1-, 2-, 3-, or n-dimensional, depending on the specific structure of the network. Figure 5 As shown, the dimensions of each layer can be different depending on the design of each layer. The dimensions of the output layer 530 depend on the characteristics of the predictions made by the model. For example, a computer model for multi-object classification can generate an output layer 530 with a one-dimensional array, where each position in the array represents the possibility of a different classification of the input layer 510. In another example for classification of image portions, the input layer 510 can be an image with a resolution of, for example, 512×512, and the output layer can be a matrix of 512×512×n, where the output layer 530 provides n classification predictions for each input pixel, so that the corresponding position of each pixel in the input layer 510 in the output layer 530 is an n-dimensional array corresponding to the classification prediction of the pixel.
[0075] Hidden layer 520 provides output activations that differently characterize input layer 510 in various ways that assist in efficiently generating output layer 530. Hidden layers can thus be viewed as providing additional features or characteristics of input layer 510. Although Figure 5 Two hidden layers are shown in FIG, but in practice any number of hidden layers may be provided in various neural network architectures.
[0076] Each layer generally determines the output activation value of its position in the activation matrix based on the output activation of one or more previous layers of the neural network (which can be regarded as the input activation of the evaluated layer). Each layer applies a function to the input activation to generate its activation. Such layers may include fully connected layers (e.g., each input is connected to each output of a layer), convolutional layers, deconvolutional layers, pooling layers, and recursive layers. Layers can apply various types of functions, including linear combinations, convolution kernels, activation functions, pooling, and so on. The parameters of the function of a layer are used to determine the output activation of the layer from the activation input of the layer, and are usually modified during the model training process. Parameters that describe the contribution of specific parts of previous layers are often called weights. For example, in some layers, the function is that each input is multiplied by a respective weight to determine the activation of the layer. For neural networks, the parameters of the entire model may thus include the parameters of each individual layer, and in large-scale networks may include hundreds of thousands, millions, or more different parameters.
[0077] As an example of training a neural network, a cost function is evaluated at the output layer 530. In order to determine the modification of the parameters for each layer, the parameters of each previous layer can be evaluated to determine the respective modification. In one example, the cost function (or "error") is back-propagated so that the parameters are evaluated by the optimization algorithm for each layer in turn until the input layer 510 is reached.
[0078] Example Device
[0079] Figure 6 600, which may include one or more components for computer model training and reasoning according to any of the embodiments disclosed herein. For example, computing device 600 may include a training module for training models and performing the functions of computing device 600, and in some cases may include specialized hardware and / or software for computer model computations.
[0080] Figure 6 Several components are illustrated as being included in computing device 600, but any one or more of these components may be omitted or repeated where appropriate for the application. In some embodiments, some or all of the components included in computing device 600 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system-on-a-chip (SoC) die.
[0081] Additionally, in various embodiments, computing device 600 may not include Figure 6 , but computing device 600 may include interface circuitry for coupling with the one or more components. For example, computing device 600 may not include display device 606, but may include display device interface circuitry (e.g., connectors and driver circuitry) to which display device 606 may be coupled. In another set of examples, computing device 600 may not include audio input device 618 or audio output device 608, but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which audio input device 618 or audio output device 608 may be coupled.
[0082] The computing device 600 may include a processing device 602 (e.g., one or more processing devices). As used herein, the term "processing device" or "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory to transform the electronic data into other electronic data that can be stored in registers and / or memory. The processing device 602 may include one or more digital signal processors (DSPs), application-specific ICs (ASICs), central processing units (CPUs), graphics processing units (GPUs), cryptographic processors (specialized processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices. The computing device 600 may include a memory 604, which itself may include one or more memory devices, such as volatile memory (e.g., dynamic random access memory (DRAM)), non-volatile memory (e.g., read-only memory (ROM)), flash memory, solid-state memory, and / or a hard disk. The memory 604 may include instructions that can be executed by the processing device to perform the methods and functions described herein. Such instructions may be instantiated in various types of memory, which may include non-volatile memory, and stored on one or more non-transitory media. In some embodiments, memory 604 may include memory that shares a die with processing device 602. This memory may be used as cache memory and may include embedded dynamic random-access memory (eDRAM) or spin transfer torque magnetic random-access memory (STT-MRAM).
[0083] In some embodiments, computing device 600 may include communication chip 612 (e.g., one or more communication chips). For example, communication chip 612 may be configured to manage wireless communications for transmitting data to and from computing device 600. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. that can communicate data through a non-solid medium using modulated electromagnetic radiation. The term does not mean that the associated devices do not contain any wires, although in some embodiments they may not.
[0084] The communication chip 612 can implement any of several wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards, including Wi-Fi (IEEE 802.11 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), Long-Term Evolution (LTE) project and any amendments, updates and / or revisions (e.g., Advanced LTE project, ultramobile broadband (UMB) project (also known as "3GPP2"), etc.). Broadband Wireless Access (BWA) networks compatible with IEEE802.16 are generally referred to as WiMAX networks. This abbreviation stands for Worldwide Interoperability for Microwave Access, which is a certification mark for products that have passed the compliance and interoperability tests of the IEEE 802.16 standard. The communication chip 612 can operate according to the Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications Syste (UMTS), High-Speed Packet Access (HSPA), Evolved HSPA (E-HSPA) or LTE network. The communication chip 612 can operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN) or Evolved UTRAN (E-UTRAN).The communication chip 612 may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO) and its derivatives, and any other wireless protocols designated as 3G, 4G, 5G and above. In other embodiments, the communication chip 612 may operate according to other wireless protocols. The computing device 600 may include an antenna 622 to facilitate wireless communications and / or receive other wireless communications (e.g., AM or FM radio transmissions).
[0085] In some embodiments, the communication chip 612 can manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 612 may include multiple communication chips. For example, the first communication chip 612 may be dedicated to shorter-range wireless communications, such as Wi-Fi or Bluetooth, while the second communication chip 612 may be dedicated to longer-range wireless communications, such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 612 may be dedicated to wireless communications, while the second communication chip 612 may be dedicated to wired communications.
[0086] The computing device 600 may include a battery / power circuit 614. The battery / power circuit 614 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuits for coupling components of the computing device 600 to an energy source separate from the computing device 600 (e.g., AC line power).
[0087] Computing device 600 may include display device 606 (or corresponding interface circuitry, as described above). Display device 606 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.
[0088] Computing device 600 may include audio output device 608 (or corresponding interface circuitry, as described above). Audio output device 608 may include any device that generates audible indications, such as a speaker, headphones, or earbuds.
[0089] Computing device 600 may include audio input device 618 (or corresponding interface circuitry, as described above). Audio input device 618 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital musical instrument (e.g., a musical instrument with a musical instrument digital interface (MIDI) output).
[0090] Computing device 600 may include GPS device 616 (or corresponding interface circuitry, as described above). GPS device 616 may communicate with a satellite-based system and may receive the location of computing device 600, as is known in the art.
[0091] Computing device 600 may include other output devices 610 (or corresponding interface circuitry, as described above). Examples of other output devices 610 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or additional storage devices.
[0092] The computing device 600 may include other input devices 620 (or corresponding interface circuits, as described above). Examples of other input devices 620 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (e.g., a mouse), a stylus, a touchpad, a barcode reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.
[0093] The computing device 600 may have any desired form factor, such as a handheld or mobile computing device (e.g., a cellular phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultra-mobile personal computer, etc.), a desktop computing device, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computing device. In some embodiments, the computing device 600 may be any other electronic device that processes data.
[0094] Selected Examples
[0095] The following paragraphs provide various examples of the embodiments disclosed herein.
[0096] Example 1 provides a method, which includes: identifying a convolutional layer having multiple convolutional filters of a target model architecture; generating a training model architecture by replacing a first number of convolutional filters among the multiple convolutional filters in the target model architecture with multiple extended training layers; training parameters of the training model architecture; and determining parameters for a trained inference model having the target model architecture based on the parameters of the training model architecture, wherein the parameters of the first number of convolutional filters among the multiple convolutional filters are determined by combining the parameters of the multiple extended training layers.
[0097] Example 2 provides a method as described in Example 1, wherein the multiple expansion layers include an output expansion layer that outputs the result of the expanded training layer and has a dimension different from the dimension of the multiple convolution filters in the target model architecture.
[0098] Example 3 provides the method of Example 2, wherein the output extension layer has a dimension of 1×1.
[0099] Example 4 provides a method as described in Example 2, wherein the multiple extended training layers include a training layer having a dimension that matches the dimension of the multiple convolutional filters in the target model architecture.
[0100] Example 5 provides a method as described in any of Examples 1-4, wherein at least one of the extended training layers has a second number of convolutional filters that is greater than the first number.
[0101] Example 6 provides a method as described in any of Examples 1-5, wherein a first number of convolution filters in the plurality of convolution filters are part of the plurality of convolution filters.
[0102] Example 7 provides a method as described in any of Examples 1-6, wherein the multiple extended training layers include a normalization layer.
[0103] Example 8 provides a system comprising a processor; and a non-transitory computer-readable storage medium containing computer program code for execution by the processor to: identify a convolutional layer having multiple convolutional filters of a target model architecture; generate a training model architecture by replacing a first number of the multiple convolutional filters in the target model architecture with multiple extended training layers; train parameters of the training model architecture; and determine parameters for a trained inference model having the target model architecture based on the parameters of the training model architecture, wherein the parameters of the first number of the multiple convolutional filters are determined by combining the parameters of the multiple extended training layers.
[0104] Example 9 provides a system as described in Example 8, wherein the multiple expansion layers include an output expansion layer that outputs the result of the expanded training layer and has a dimension different from the dimension of the multiple convolution filters in the target model architecture.
[0105] Example 10 provides the system of Example 9, wherein the output extension layer has a dimension of 1×1.
[0106] Example 11 provides a system as described in Example 9, wherein the multiple extended training layers include a training layer having a dimension that matches the dimension of the multiple convolutional filters in the target model architecture.
[0107] Example 12 provides a system as described in any of Examples 8-11, wherein at least one of the extended training layers has a second number of convolutional filters greater than the first number.
[0108] Example 13 provides a system as described in any of Examples 8-12, wherein a first number of convolution filters in the plurality of convolution filters are part of the plurality of convolution filters.
[0109] Example 14 provides a system as described in any of Examples 8-13, wherein the plurality of extended training layers includes a normalization layer.
[0110] Example 15 provides a non-transitory computer-readable storage medium containing instructions that can be executed by a processor to: identify a convolutional layer having multiple convolutional filters of a target model architecture; generate a training model architecture by replacing a first number of convolutional filters in the target model architecture with multiple extended training layers; train parameters of the training model architecture; and determine parameters for a trained inference model having the target model architecture based on the parameters of the training model architecture, wherein the parameters of the first number of convolutional filters in the multiple convolutional filters are determined by combining the parameters of the multiple extended training layers.
[0111] Example 16 provides a non-transitory computer-readable storage medium as described in Example 15, wherein the multiple expansion layers include an output expansion layer that outputs the result of the expanded training layer and has a dimension different from the dimension of the multiple convolution filters in the target model architecture.
[0112] Example 17 provides the non-transitory computer-readable storage medium of Example 16, wherein the output extension layer has a dimension of 1×1.
[0113] Example 18 provides a non-transitory computer-readable medium as described in Example 16, wherein the multiple extended training layers include a training layer having dimensions that match the dimensions of the multiple convolutional filters in the target model architecture.
[0114] Example 19 provides a non-transitory computer-readable medium as described in any of Examples 15-18, wherein at least one of the extended training layers has a second number of convolutional filters greater than the first number.
[0115] Example 20 provides the non-transitory computer-readable medium of any of Examples 15-19, wherein a first number of convolution filters in the plurality of convolution filters are a portion of the plurality of convolution filters.
[0116] Example 21 provides a non-transitory computer-readable medium as described in any of Examples 15-20, wherein the plurality of extended training layers includes a normalization layer.
[0117] The above description of the illustrated implementations of the present disclosure, including those described in the Abstract, is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Although specific implementations of the present disclosure and examples thereof are described herein for illustrative purposes, as will be appreciated by those skilled in the relevant art, various equivalent modifications are possible within the scope of the present disclosure. These modifications may be made to the present disclosure in accordance with the above detailed description.
Claims
1. A method comprising: Recognize the convolutional layer with multiple convolutional filters of the target model architecture; generating a training model architecture by replacing a first number of convolutional filters in the plurality of convolutional filters in the target model architecture with a plurality of expanded training layers; Training parameters of the training model architecture; and Determining parameters for a trained inference model having the target model architecture based on the parameters of the training model architecture, wherein parameters of a first number of the plurality of convolutional filters are determined by combining parameters of the plurality of extended training layers.
2. The method of claim 1, wherein: The multiple expansion layers include an output expansion layer that outputs a result of the expanded training layer and has a dimension different from a dimension of the multiple convolutional filters in the target model architecture.
3. The method of claim 2, wherein: The output expansion layer has a dimension of 1×1.
4. The method of claim 2, wherein: The plurality of extended training layers include training layers having dimensions that match dimensions of the plurality of convolutional filters in the target model architecture.
5. The method of claim 1, wherein: At least one of the extended training layers has a second number of convolutional filters greater than the first number.
6. The method of claim 1, wherein: A first number of convolutional filters in the plurality of convolutional filters are part of the plurality of convolutional filters.
7. The method of claim 1, wherein: The plurality of extended training layers include a normalization layer.
8. A system comprising: processor; as well as A non-transitory computer readable storage medium containing computer program code for execution by the processor to: Recognize the convolutional layer with multiple convolutional filters of the target model architecture; generating a training model architecture by replacing a first number of convolutional filters in the plurality of convolutional filters in the target model architecture with a plurality of expanded training layers; Training parameters of the training model architecture; and Determining parameters for a trained inference model having the target model architecture based on the parameters of the training model architecture, wherein parameters of a first number of the plurality of convolutional filters are determined by combining parameters of the plurality of extended training layers.
9. The system of claim 8, wherein: The multiple expansion layers include an output expansion layer that outputs a result of the expanded training layer and has a dimension different from a dimension of the multiple convolutional filters in the target model architecture.
10. The system of claim 9, wherein: The output expansion layer has a dimension of 1×1.
11. The system of claim 9, wherein: The plurality of extended training layers include training layers having dimensions that match dimensions of the plurality of convolutional filters in the target model architecture.
12. The system of claim 8, wherein: At least one of the extended training layers has a second number of convolutional filters greater than the first number.
13. The system of claim 8, wherein: A first number of convolutional filters in the plurality of convolutional filters are part of the plurality of convolutional filters.
14. The system of claim 8, wherein: The plurality of extended training layers include a normalization layer.
15. A non-transitory computer-readable storage medium comprising instructions executable by a processor to: Recognize the convolutional layer with multiple convolutional filters of the target model architecture; generating a training model architecture by replacing a first number of convolutional filters in the plurality of convolutional filters in the target model architecture with a plurality of expanded training layers; Training parameters of the training model architecture; and Determining parameters for a trained inference model having the target model architecture based on the parameters of the training model architecture, wherein parameters of a first number of the plurality of convolutional filters are determined by combining parameters of the plurality of extended training layers.
16. The non-transitory computer readable medium of claim 15, wherein: The multiple expansion layers include an output expansion layer that outputs a result of the expanded training layer and has a dimension different from a dimension of the multiple convolutional filters in the target model architecture.
17. The non-transitory computer readable medium of claim 16, wherein: The output expansion layer has a dimension of 1×1.
18. The non-transitory computer readable medium of claim 16, wherein: The plurality of extended training layers include training layers having dimensions that match dimensions of the plurality of convolutional filters in the target model architecture.
19. The non-transitory computer readable medium of claim 15, wherein: At least one of the extended training layers has a second number of convolutional filters greater than the first number.
20. The non-transitory computer readable medium of claim 15, wherein: A first number of convolutional filters in the plurality of convolutional filters are part of the plurality of convolutional filters.
21. The non-transitory computer readable medium of claim 15, wherein: The plurality of extended training layers include a normalization layer.
