Method for establishing neural network and readable storage medium

Abstract

The present application relates to a method for establishing a neural network and a readable storage medium, first setting a super network, the super network comprising multiple stages, each stage comprising multiple layers, and each layer comprising multiple units; setting an input control gate for each unit to reflect the probability of selecting the unit; setting an output control gate for each layer to determine the number of channels of the layer; inputting image data into the super network for forward propagation to calculate a loss function; updating the input control gate and the output control gate according to the loss function; and determining the structure of the neural network according to the updated input control gate and the output control gate. The model size is compressed as much as possible while ensuring no loss of accuracy.

Description

Technical Field

[0001] This invention generally relates to the field of neural networks. More specifically, this invention relates to methods for creating neural networks and readable storage media. Background Technology

[0002] With the development of artificial intelligence, neural networks are widely used for image classification and detection. Traditionally, the backbone networks for classification and detection tasks are designed by developers based on their own experience, selecting a network with acceptable performance after numerous trial-and-error experiments. Furthermore, most networks are initially designed for classification tasks, then the networks and parameters suitable for classification are transferred to detection tasks, and finally, accuracy is achieved through retraining. Networks obtained in this way often require extensive trial and error and have less than ideal general applicability.

[0003] When designing networks, developers often prioritize task accuracy as the primary design metric, while giving less consideration to performance metrics such as network latency and frames per second (FPS).

[0004] Therefore, there is an urgent need for an automated neural network construction scheme that can comprehensively evaluate multiple performance indicators. Summary of the Invention

[0005] In order to at least partially solve the technical problems mentioned in the background art, the present invention provides a method for establishing a neural network and a readable storage medium.

[0006] In one aspect, the present invention discloses a method for establishing a neural network, comprising: setting a supernet, the supernet comprising multiple levels, each level comprising multiple layers, and each layer comprising multiple units; setting an input control gate for each unit to reflect the probability of selecting the unit; setting an output control gate for each layer to determine the number of channels of the layer; inputting image data into the supernet for forward propagation to calculate a loss function; updating the input control gate and the output control gate according to the loss function; and determining the structure of the neural network according to the updated input control gate and the output control gate.

[0007] In another aspect, the present invention discloses a computer-readable storage medium having stored thereon computer program code for establishing a neural network, which, when run by a processing device, performs the aforementioned method.

[0008] This invention utilizes automated machine learning, and by setting the search space for channel number and latency, it makes the searched network structure more suitable for classification and detection tasks, while compressing the model size as much as possible without sacrificing accuracy. Attached Figure Description

[0009] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts wherein:

[0010] Figure 1 This is a schematic diagram illustrating multiple units pre-configured for specific hardware in an embodiment of the present invention;

[0011] Figure 2 This is a flowchart illustrating the establishment of a neural network according to an embodiment of the present invention;

[0012] Figure 3 This is a schematic diagram illustrating a supernet according to an embodiment of the present invention;

[0013] Figure 4 This is a schematic diagram illustrating a neural network model according to an embodiment of the present invention;

[0014] Figure 5 This is a flowchart illustrating another embodiment of the present invention for establishing a neural network; and

[0015] Figure 6 This is a flowchart illustrating the establishment of a neural network according to another embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the specification and claims of this invention indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0018] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0019] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0020] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] Building neural network models in the field of machine learning is an extremely complex undertaking. A 10-layer network model can generate 10^10 possible network combinations, let alone a network model with hundreds of layers. In the past, these models were manually tuned by developers, consuming a great deal of time and resources. To automate the process of designing machine learning models, an automated machine learning (AutoML) system has emerged.

[0022] For AutoML, the primary task is to architect the neural network model. This is because if the neural network architecture has problems, no amount of parameter tuning can significantly improve the model's performance. The process of architecting a neural network in AutoML can be divided into three stages: the search space stage, the search strategy stage, and the evaluation strategy stage.

[0023] The search space phase is based on certain premises and assumptions, and pre-defines some units based on existing experience. These units are like building blocks, and the search space is constructed by stacking these blocks. Given the search space, AutoML follows the algorithm designed by the developers, uses certain evaluation metrics, and generates a relatively perfect network structure through continuous testing. In other words, the neural network model is selected based on this search space.

[0024] AutoML views neural network structures as being composed of many small, repetitive units. When building the entire neural network, it is only necessary to search and arrange these units, rather than searching the entire network structure.

[0025] The purpose of the search strategy phase is to determine which algorithm can quickly and accurately find the optimal network structure parameter configuration. Common search methods include: random search, Bayesian optimization, evolutionary algorithms, reinforcement learning, gradient-based algorithms, etc.

[0026] The evaluation strategy phase is used to assess the performance of the generated network structure, similar to an agent model in engineering optimization. Because the performance of deep learning models is highly dependent on the scale of the training data, training models on large-scale data is extremely time-consuming; therefore, an approximate evaluation of the design method is necessary. Several commonly used evaluation strategies are as follows.

[0027] The simplest and most intuitive strategy is to train and validate the network architecture using a dataset to evaluate its performance. Low-fidelity strategies consider only a few evaluation criteria, such as shorter training time, smaller training sets, lower image resolution, or fewer filters per layer, to reduce training overhead. While this can lead to bias, it is generally acceptable. The core idea of ​​inference methods is to initially establish multiple learning curves and terminate those that perform poorly, thus accelerating the architecture search process. Weight parameter migration strategies directly apply the parameters of a pre-trained model to the current network architecture to shorten training time, provided there is a similar pre-trained model available for transfer.

[0028] This invention utilizes AutoML to model neural networks. In the search space stage, units are defined. In the search strategy stage, coefficients are set for the number of channels in each unit and layer, and parameter update methods are planned. In the evaluation strategy stage, the network structure is trained and validated using a dataset to evaluate its performance and finally, an ideal neural network model is obtained.

[0029] An embodiment of the present invention is a method for building a neural network using AutoML. This neural network includes a backbone network and a classifier. The backbone network is used to extract features, and the classifier is connected after the backbone network. The classifier includes a classifier and / or a detector, enabling the neural network model to perform classification and / or detection tasks.

[0030] In the search space phase, this embodiment first pre-defines some units. These units can be designed according to the artificial intelligence chip running the neural network model, making them more compatible with the specific artificial intelligence chip and faster in operation. Figure 1A plurality of units pre-configured for a specific hardware are shown, including a first unit 101, a second unit 102, a third unit 103, a fourth unit 104, and a fifth unit 105.

[0031] Unit 101 includes four network operators: a 1x1 convolution-batch normalization-ReLU activation operator, a 3x3 convolution-batch normalization-ReLU activation operator, a 1x1 convolution-batch normalization operator, and a ReLU activation operator. Figure 1 The specific order shown is used.

[0032] Unit 2 102 includes four network operators: a 1*1 convolution-batch normalization-ReLU activation operator (1*1 Conv-BN-ReLU), a 3*3 convolution-batch normalization-ReLU activation operator with a dilation of 2 (3*3 dilation = 2 Conv-BN-ReLU), a 1*1 convolution-batch normalization operator (1*1 Conv-BN), and a ReLU activation operator. Figure 1 The specific order shown is used.

[0033] Unit 3 (103) includes four network operators: a 1x1 convolution-batch normalization-ReLU activation operator (1x1 Conv-BN-ReLU), a 3x3 convolution-batch normalization-ReLU activation operator with a dilation of 4 (3x3 dilation = 4 Conv-BN-ReLU), a 1x1 convolution-batch normalization operator (1x1 Conv-BN), and a ReLU activation operator. Figure 1 The specific order shown is used.

[0034] Unit 4, 104, includes three network operators: a 3x3 convolution-batch normalization-ReLU activation operator (3x3Conv-BN-ReLU), a 3x3 convolution-batch normalization operator (3x3Conv-BN), and a ReLU activation operator. Figure 1 The specific order shown is used.

[0035] Unit 5, 105, is an empty operator, meaning it has no operator, performs no calculations, and its output equals its input. The purpose of Unit 5, 105, is to allow the number of layers in the neural network model to be variable. When a layer selects Unit 5, 105, it means that the layer does not perform any calculations for any operator, which is equivalent to the layer not existing.

[0036] These five units act as "building blocks" in the search space stage, constructing the search space. In other words, the framework of the search space will only utilize these five units in combination; no other undefined units will appear in the search space. The method for building the neural network in this embodiment is as follows:Figure 2 As shown.

[0037] In step 201, the delay of each unit is measured.

[0038] In step 202, all delays are made into a lookup table, that is, the delays of the first unit 101 to the fifth unit 105 are made into a lookup table.

[0039] In step 203, a supernet is defined. A supernet is a large network composed of multiple smaller networks. Figure 3 The diagram illustrates a hypernet of this embodiment. The hypernet comprises any number of levels; the diagram shows i levels (level 0 to level i-1), each level containing multiple layers, and each level containing j layers (layer 0 to layer j-1). Each layer includes parallel units 101 to 105, where i and j are positive integers. For example, assuming i = 4 and j = 10, this hypernet has 4 levels, each with 10 layers, resulting in a total of 4 × 10 = 40 layers. Each layer has 5 selectable units (unit 101 to unit 105). This hypernet is the aforementioned backbone network, also known as the search space.

[0040] The search space phase is now complete; we will now proceed to the search strategy phase.

[0041] In step 204, an input control gate is configured for each unit to reflect the probability of selecting that unit. Each unit is configured with one input control gate. This is used to represent the input control gate of the h-th cell in the j-th layer of the i-th level in a supernet. The input control gate reflects the probability of selecting the corresponding cell as an operator in that layer.

[0042] For example, This represents the probability that the first unit 101 of the fifth layer of the first level is an operator of the fifth layer of the first level. This represents the probability that unit 102 of level 1, layer 5, is an operator of level 1, layer 5. This represents the probability that unit 103 of level 1, layer 5, is an operator of level 1, layer 5. This represents the probability that unit 104 of level 1, layer 5, is an operator of level 1, layer 5. This represents the probability that unit 105 of level 1, layer 5, is an operator of level 1, layer 5. The sum of the values ​​of the input control gates of each layer is 1, i.e.

[0043] In this step, the initial values ​​of the input control gates can be set randomly or evenly distributed, for example, the initial values ​​of all 5 input control gates can be set to 0.2. However, the input control gate of the fifth unit 105 (empty operator) in the starting layer (layer 0) of each level is always 0 and is not updated as the model is trained, in order to ensure that there is at least one layer in each level during the policy evaluation phase.

[0044] In step 205, an output control gate is set for each layer to determine the number of channels for that layer. The number of channels in this step refers to the number of output channels for each layer, and the output control gate includes multiple channel coefficients. This represents the output control gate for the c-th channel coefficient in the i-th level and j-th layer. Each channel coefficient corresponds to one channel number. The output control gate reflects the probability of selecting a specific number of channels in that layer.

[0045] In this embodiment, each layer has a maximum of 256 channels, and the output control gates (channel coefficients) have four values: for example, value 0 corresponds to 64 channels, value 1 corresponds to 128 channels, value 2 corresponds to 192 channels, and value 3 corresponds to 256 channels. Therefore, each layer in the supernet has four selectable channel counts. The sum of the values ​​of the output control gates for each layer is 1.

[0046] The output control gate of the fifth unit 105 in each layer is set to 9 and is not updated with model training. In this embodiment, an output control gate of 9 indicates that the empty operator has no channel.

[0047] In step 206, a classifier is added after the supernet. The classifier includes a classifier and a detector. The classifier is used to classify the images. Image classification refers to using a neural network to predict and classify the category of a new set of test images given a set of images, each labeled as a single category. The detector is used to detect images. Image detection refers to identifying objects in an image and outputting bounding boxes and labels for them.

[0048] The supernet is merely the backbone network used to filter image features; the specific function of the neural network model is determined by the classifier. If a classifier is added after the supernet, the neural network model then has image classification capabilities (e.g., ...). Figure 4 The neural network model 401); if a detector is added after the supernet, the neural network model has the function of image detection (e.g., Figure 4 The neural network model 402); if a classifier and a detector are added simultaneously after the supernet, the neural network model has the function of image classification and detection (e.g., Figure 4 The neural network model 403).

[0049] In step 207, the output feature map of each layer in the supernet is calculated. In this embodiment, the output feature map is:

[0050]

[0051] Among them, I i,j Let i be the input feature map of the i-th level and j-th layer. This is the input control gate for the h-th unit in the j-th layer of the i-th level. Let h be the output function of the i-th level, j-th layer, and h-th unit. This is the output control gate for the c-th channel coefficient corresponding to the j-th layer of the i-th level.

[0052] The evaluation strategy phase is divided into in-search evaluation and post-search evaluation. In-search evaluation is conducted simultaneously during the search strategy phase by inferring the model on the training set to obtain the loss function or model accuracy (based on reinforcement learning) for evaluation. Post-search evaluation involves selecting input / output control gates to obtain the network structure, retraining the model on the training set, and finally evaluating the model accuracy on the validation set. This embodiment uses in-search evaluation; therefore, the following steps are simultaneously the search strategy phase and the evaluation strategy phase.

[0053] As mentioned earlier, various methods can be used to evaluate the performance of supernets during the evaluation strategy phase. This embodiment chooses to directly train the supernet using the dataset to find the optimal input control gate, output control gate, and other network parameters.

[0054] Training a neural network involves adjusting the parameters of each layer by inputting training samples, aiming to make the network's calculations as close as possible to the actual results. Neural network training includes forward propagation and backpropagation. Forward propagation, based on the existing model, uses the input training samples to process the network's layers, progressively extracting abstract features from the input feature maps. Backpropagation, on the other hand, calculates a loss function based on the forward propagation results and the actual values. Using gradient descent and the chain rule, it calculates the partial derivative of the loss function with respect to each parameter to update the parameters. These updated parameters are then used for training, and this process is repeated multiple times until the forward propagation results meet expectations.

[0055] In practice, the training of a neural network model may involve multiple epochs. An epoch refers to the process of training once using all the training samples. The set of these training samples is called the training set, and each batch of training samples is called an iteration. For example, if the training set has 1000 training samples and the batch size is set to 10, then each iteration requires 10 training samples to participate in the training, and there are a total of 100 iterations in one epoch.

[0056] In step 208, the input image data is fed into the supernet for forward propagation to calculate the loss function. The training set contains a large number of image data training samples, which are imported from the input end of the supernet. Each layer of the supernet generates an output feature map as shown in step 207. The output feature map of the last layer of the supernet is input into the sorting unit. The loss function is the deviation between the actual output value and the predicted output value of the sorting unit; in other words, the loss function is derived based on the output feature map of the supernet and the output feature map of the sorting unit.

[0057] In this embodiment, the loss function includes a control gate loss term, a time loss term, and a sorter loss term.

[0058] The control gate loss term reflects the L1 regularization loss of both the input and output control gates. Regularization is an effective way in machine learning to avoid overfitting and ensure generalization ability by explicitly controlling model complexity. L1 regularization, besides constraining the order of magnitude, also makes the parameters more sparse. The result of sparsity is that some of the optimized parameters are 0, while the rest are non-zero real values. These non-zero real values ​​help select important parameters or feature dimensions and also remove noise. Since the input control gate corresponds to the probability of each unit becoming an operator in that layer, the control gate loss term effectively selects operators for each layer. Furthermore, since the output control gate corresponds to the number of channels in each layer, the control gate loss term also helps select the number of channels in each layer.

[0059] The time loss term is calculated based on the lookup table obtained in step 202 to determine the delay of each unit. Therefore, the time loss term allows the neural network model to take unit delay into account and select appropriate units.

[0060] The classifier loss term includes the loss of the classifier and the loss of the detector, which is the bias caused by the classifier. As mentioned earlier, the classifier may include only a classifier or a detector, or both. If it includes only a classifier or a detector, the classifier loss term reflects the bias of the classifier or the bias of the detector; if it includes both, the classifier loss term reflects the bias of both the classifier and the detector.

[0061] Specifically, the loss function in this embodiment is:

[0062] λ0×L cls +λ1×L det +λ2×L gate +λ3×L lat

[0063] Where λ0 is the hyperparameter of the classifier's loss, L cls Let λ be the loss parameter of the classifier, λ1 be the hyperparameter of the loss parameter of the detector, and L be the loss parameter of the classifier. detλ is the detector loss, λ2 is the hyperparameter of the control gate loss, and L gate To control the gate loss, λ3 is a hyperparameter for the time loss, L lat The time loss is defined by λ0, λ1, λ2, and λ3, which are greater than or equal to 0 and are set by the developers based on experience, not updated during training. If the sorting unit only includes a classifier, then λ1 is 0; if the sorting unit only includes a detector, then λ0 is 0. The time loss L... lat for:

[0064]

[0065] Where T0 is the preset target delay, set by the developer based on experience, and T is:

[0066]

[0067] in, For the input control gate of the h-th unit in the j-th layer of the i-th level, t h,c The delay of the coefficient of the c-th channel corresponding to the h-th unit can be obtained through the lookup table established in step 202. This is the output control gate for the c-th channel coefficient corresponding to the j-th layer of the i-th level.

[0068] In step 209, all parameters, including the input and output control gates, are updated according to the loss function. Gradient descent is used during backpropagation, and the partial derivatives of the loss function with respect to each network parameter are calculated using the chain rule. Optionally, the learning rates for the input and output control gates are different from the learning rates for the other network parameters.

[0069] In step 210, the structure of the neural network is determined based on the updated input and output control gates. After training, the values ​​of the input control gates for each unit have been updated. At this point, the unit corresponding to the maximum value of the input control gate in each layer is selected, and this unit is designated as the operator for that layer. For example, suppose the input control gates of the i-th level and j-th layer are as follows: The maximum value of the input control gate is If this corresponds to the third unit 103, then the operator of the i-th level and j-th layer is set to the third unit 103, and all other units in the same layer are discarded. If the input control gate of the fifth unit 105 of a certain layer is the largest, it means that there are no operators in that layer, and that layer can be directly deleted. If there are multiple maximum values ​​for the input control gates, one is randomly selected. In this step, the operator of each layer is selected by the input control gate.

[0070] In this step, the number of channels for each layer is also set to the number of channels in that layer whose output control gates are greater than a channel threshold. This channel threshold is between 0 and 1, for example, 0.25. Assuming that the four channel coefficients of a certain layer are 0.2, 0.4, 0.2, and 0.2 respectively, and the second channel coefficient is greater than the channel threshold, then the number of output channels for that layer is set to 128. If multiple channel coefficients are greater than the channel threshold, the one with the larger number of channels is selected. In this step, the number of channels for each layer is selected through the output control gates.

[0071] At this point, after the strategy evaluation phase, the supernet generates the backbone network of the neural network model. Combined with the subsequent classifier, a complete neural network model with classification / detection capabilities is established. This neural network model can be retrained to improve accuracy.

[0072] Another embodiment of the present invention is a method for building a neural network using AutoML. This neural network includes a backbone network and a classifier. The backbone network is used to extract features, and the classifier is connected after the backbone network. The classifier includes a classifier and / or a detector, so that the neural network model can perform classification and / or detection tasks.

[0073] This embodiment also pre-sets the following during the search space phase: Figure 1 The units shown are described. The method for establishing the neural network in this embodiment is as follows: Figure 5 As shown.

[0074] In step 501, the delay of each unit is measured.

[0075] In step 502, all delays are created into a lookup table.

[0076] In step 503, set as follows Figure 3 The supernet shown is the backbone network and also the search space.

[0077] The search space phase is now complete; we will now proceed to the search strategy phase.

[0078] In step 504, an input control gate is configured for each unit to reflect the probability of selecting the unit. Each unit is configured with one input control gate. This represents the input control gate of the h-th unit in the j-th layer of the i-th level in the supernet. The input control gate reflects the probability of selecting the corresponding unit as the operator in that layer. The sum of the values ​​of the input control gates in each layer is 1. In this step, the initial values ​​of the input control gates can be set randomly or evenly. The input control gate of the fifth unit 105 (empty operator) in the starting layer (layer 0) of each level is always 0 and is not updated with model training. This is to ensure that there is at least one layer in each level during the policy evaluation phase.

[0079] In step 505, an output control gate is set for each layer to determine the number of channels for that layer. The number of channels in this step refers to the number of output channels for each layer. The output control gate includes multiple channel coefficients, each channel coefficient corresponding to a channel number, and the sum of the values ​​of the output control gates for each layer is 1.

[0080] The output control gate of the fifth unit 105 in each layer is set to 9 and is not updated with model training. In this embodiment, an output control gate of 9 indicates that the empty operator has no channel.

[0081] In step 506, a sorting device is added after the supernet. The sorting device in this embodiment is the same as that in the previous embodiment, and will not be described again.

[0082] In step 507, the output feature map of each layer in the supernet is calculated. In this embodiment, the output feature map is:

[0083]

[0084] Among them, I i,j Let i be the input feature map of the i-th level and j-th layer. This is the input control gate for the h-th unit in the j-th layer of the i-th level. Let h be the output function of the i-th level, j-th layer, and h-th unit. This is the output control gate for the c-th channel coefficient corresponding to the j-th layer of the i-th level.

[0085] This embodiment also uses evaluation during the search, so the next step is both the search strategy phase and the evaluation strategy phase.

[0086] In step 508, the input image data is fed into the supernet for forward propagation to calculate the loss function. The training set contains a large number of image data training samples, which are imported from the input end of the supernet. Each layer of the supernet generates an output feature map as shown in step 507. The output feature map of the last layer of the supernet is input into the sorting unit. The loss function is the deviation between the actual output value and the predicted output value of the sorting unit; in other words, the loss function is derived based on the output feature map of the supernet and the output feature map of the sorting unit.

[0087] In this embodiment, the loss function also includes a control gate loss term, a time loss term, and a sorter loss term, which are the same as the loss function in the previous embodiment, and will not be described again.

[0088] In step 509, the network parameters of non-input control gates and output control gates are updated according to the loss function. Gradient descent is used in backpropagation, and the partial derivative of the loss function with respect to each network parameter is calculated using the chain rule. The values ​​of all network parameters except the input and output control gates are then updated based on a certain learning rate. The reason for not updating the input and output control gates in this step is to allow other parameters to be updated first. Once these parameters have been trained for a period and begin to approach reasonable values, the input and output control gates are then updated, resulting in faster convergence of the entire network.

[0089] To suppress the updates of the input and output control gates, the specific operation method is to set the control gate probability P1 for the input control gate and the control gate probability P2 for the output control gate in the loss function. In this step, P1 = P2 = 0, thereby hiding the input and output control gates.

[0090] In step 510, it is determined whether the training progress has reached a progress threshold. The progress threshold can be set to 10%, 20%, 25%, or 30%, and in this embodiment, it is 25%. If the training progress has not yet reached the progress threshold, step 509 is executed to update the network parameters of the non-input control gate and the output control gate. When the training progress reaches the progress threshold, step 511 is executed.

[0091] In step 511, the input and output control gates are updated according to the loss function. The initial values ​​of the input and output control gates are randomly set. During backpropagation, gradient descent is used, and the partial derivatives of the loss function with respect to the input and output control gates are calculated using the chain rule to update their values. The updated input and output control gates are then used for training until training is complete. In this step, P1 = P2 = 1, thus introducing the input and output control gates into the loss function.

[0092] Optionally, the learning rates for the input and output control gates are different from those for other network parameters. It should be noted that the network parameters for the non-input and output control gates continue to be updated during this step.

[0093] In step 512, the structure of the neural network is determined based on the updated input and output control gates. After training, the values ​​of the input control gates for each unit have been updated. At this point, the unit corresponding to the maximum value of the input control gate in each layer is selected, and this unit is designated as the operator for that layer; other units in the same layer are discarded. If the fifth unit (105) of a certain layer has the maximum input control gate value, it means that there are no operators in that layer, and that layer can be directly deleted. If there are multiple maximum values ​​for the input control gates, one is randomly selected. In this step, the operators for each layer are selected based on the input control gates.

[0094] Simultaneously, the number of channels in each layer is set to the number of channels corresponding to output control gates in that layer exceeding a channel threshold. This channel threshold is between 0 and 1. If multiple output control gates exceed the channel threshold, the one with the most channels is selected. In this step, the number of channels in each layer is selected through the output control gates.

[0095] At this point, the supernet has completed the strategy evaluation phase, generating the backbone network of the neural network model. Combined with the subsequent classifier, a complete neural network model with classification / detection capabilities is established. This neural network model can be retrained to improve accuracy.

[0096] Another embodiment of the present invention is a method for building a neural network using AutoML. This neural network includes a backbone network and a classifier. The backbone network is used to extract features, and the classifier is connected after the backbone network. The classifier includes a classifier and a detector, so that the neural network model can perform classification and / or detection tasks.

[0097] This embodiment also pre-sets the following during the search space phase: Figure 1 The units shown are described. The method for establishing the neural network in this embodiment is as follows: Figure 6 As shown.

[0098] In step 601, the delay of each unit is measured.

[0099] In step 602, all delays are created into a lookup table.

[0100] In step 603, set as follows Figure 3 The supernet shown is the backbone network and also the search space.

[0101] The search space phase is now complete; we will now proceed to the search strategy phase.

[0102] In step 604, an input control gate is configured for each unit to reflect the probability of selecting the unit. Each unit is configured with one input control gate. This represents the input control gate of the h-th unit in the j-th layer of the i-th level in the supernet. The input control gate reflects the probability of selecting the corresponding unit as the operator in that layer. The sum of the values ​​of the input control gates in each layer is 1. In this step, the initial values ​​of the input control gates can be set randomly or evenly. The input control gate of the fifth unit 105 (empty operator) in the starting layer (layer 0) of each level is always 0 and is not updated with model training. This is to ensure that there is at least one layer in each level during the policy evaluation phase.

[0103] In step 605, an output control gate is set for each layer to determine the number of channels for that layer. The number of channels in this step refers to the number of output channels for each layer. The output control gate includes multiple channel coefficients, each channel coefficient corresponding to a channel number, and the sum of the values ​​of the output control gates for each layer is 1.

[0104] The output control gate of the fifth unit 105 in each layer is set to 9 and is not updated with model training. In this embodiment, an output control gate of 9 indicates that the empty operator has no channel.

[0105] In step 606, a sorting device is added after the supernet. The sorting device in this embodiment is the same as the sorting device in the previous embodiment, and will not be described again.

[0106] In step 607, the output feature map of each layer in the supernet is calculated. In this embodiment, the output feature map is:

[0107]

[0108] Among them, I i,j Let i be the input feature map of the i-th level and j-th layer. This is the input control gate for the h-th unit in the j-th layer of the i-th level. Let h be the output function of the i-th level, j-th layer, and h-th unit. This is the output control gate for the c-th channel coefficient corresponding to the j-th layer of the i-th level.

[0109] This embodiment also uses evaluation during the search, so the next step is both the search strategy phase and the evaluation strategy phase.

[0110] In step 608, the input image data is fed into the supernet for forward propagation to calculate the loss function. The training set contains a large number of image data training samples, which are imported from the input end of the supernet. Each layer of the supernet generates an output feature map as shown in step 607. The output feature map of the last layer of the supernet is input into the sorting unit. The loss function is the deviation between the actual output value and the predicted output value of the sorting unit; in other words, the loss function is derived based on the output feature map of the supernet and the output feature map of the sorting unit.

[0111] In this embodiment, the loss function includes a control gate loss term, a time loss term, and a sorter accuracy term.

[0112] The control gate loss term reflects the L1 regularization loss of both the input and output control gates. The time loss term calculates the delay of each unit based on the lookup table obtained in step 602, thus allowing the neural network model to consider unit delays and select appropriate units.

[0113] The classifier accuracy rate includes the accuracy of the classifier and / or the accuracy of the detector. As mentioned earlier, the classifier may include only a classifier or a detector, or both. If it includes only a classifier or a detector, the classifier accuracy rate reflects either the accuracy of the classifier or the accuracy of the detector; if it includes both, the classifier accuracy rate reflects both the accuracy of the classifier and the accuracy of the detector.

[0114] Specifically, the loss function in this embodiment is:

[0115] λ0×Q cls +λ1×Q det +λ2×L gate +λ3×L lat

[0116] Where λ0 is the hyperparameter of the classifier's accuracy, Q cls Let λ be the accuracy of the classifier, λ1 be the hyperparameter of the detector's accuracy, and Q be the accuracy of the detector. det Let L be the detector accuracy, λ2 be the hyperparameter of the control gate loss, and L be the value of L. gate To control the gate loss, λ3 is a hyperparameter for the time loss, L lat The time loss is defined as λ0 and λ1 being greater than or equal to 0, and λ2 and λ3 being less than 0. These values ​​are set by the developers based on experience and are not updated during training. If the sorting unit only includes a classifier, then λ1 is 0; if the sorting unit only includes a detector, then λ0 is 0. The time loss L... lat for:

[0117]

[0118] Where T0 is the preset target delay, set by the developer based on experience, and T is:

[0119]

[0120] in, For the input control gate of the h-th unit in the j-th layer of the i-th level, t h,c The delay of the coefficient of the c-th channel corresponding to the h-th unit can be obtained through the lookup table established in step 202. This is the output control gate for the c-th channel coefficient corresponding to the j-th layer of the i-th level.

[0121] In step 609, all parameters, including the input and output control gates, are updated according to the loss function. This embodiment uses reinforcement learning to update the parameters. Reinforcement learning refers to treating the supernet as an iterative training of a reinforcement learning surrogate model, using the surrogate model's actions in the space to build a search neural network architecture space. Then, based on the generated model's performance on the test set, reinforcement learning techniques are used to reward or penalize the generated model, allowing the surrogate model to adjust its framework generation direction accordingly. In this embodiment, the network structure parameters and input / output control gates are updated alternately. Traditional gradient descent is used to update the network structure parameters, while reinforcement learning is used to update the input / output control gates.

[0122] In step 610, the structure of the neural network is determined based on the updated input and output control gates. After training, the unit corresponding to the maximum value of the input control gate in each layer is selected and designated as the operator for that layer; other units in the same layer are discarded. If there are multiple maximum values ​​for the input control gate, one is randomly selected. The number of channels in each layer is then set to the number of channels corresponding to output control gate values ​​greater than a channel threshold, which is between 0 and 1. If multiple channel coefficients are greater than the channel threshold, the one with the larger number of channels is selected.

[0123] At this point, the supernet has completed the strategy evaluation phase, generating the backbone network of the neural network model. Combined with the subsequent classifier, a complete neural network model with classification / detection capabilities is established. This neural network model can be retrained to improve accuracy.

[0124] Once the aforementioned embodiments establish a neural network model, the model can be used for image classification, where image data is input into the neural network for inference to perform prediction and classification. It can also be used for image detection, where image data is input into the neural network for inference to identify objects in the image and output bounding boxes and labels.

[0125] Another embodiment of the present invention is a computer-readable storage medium storing computer program code for establishing a neural network. When the computer program code is run by a processor, it performs the methods described in the preceding embodiments. When the solutions of the present invention are embodied in the form of a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, a server, or a network device) to perform some or all of the steps of the methods described in the embodiments of the present invention. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0126] This invention utilizes automated machine learning, and by setting the search space for the number of channels and latency, it makes the searched network structure more suitable for classification and detection tasks, while compressing the model size as much as possible without sacrificing accuracy.

[0127] It should be noted that, for the sake of brevity, this invention describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions of this invention are not limited to the order of the described actions. Therefore, based on the disclosure or teachings of this invention, those skilled in the art will understand that some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this invention can be considered as optional embodiments, that is, the actions or units involved are not necessarily essential for the implementation of one or more solutions of this invention. In addition, depending on the solution, the description of some embodiments of this invention also has different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this invention can also be referred to the relevant descriptions of other embodiments.

[0128] In terms of specific implementation, based on the disclosure and teachings of this invention, those skilled in the art will understand that the several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or device embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection utilizing an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0129] The foregoing can be better understood in accordance with the following terms:

[0130] Clause A1. A method for establishing a neural network, comprising: setting a supernet, the supernet comprising multiple levels, each level comprising multiple layers, each layer comprising multiple units; setting an input control gate for each unit to reflect the probability of selecting the unit; setting an output control gate for each layer to determine the number of channels of the layer; inputting image data into the supernet for forward propagation to calculate a loss function; updating the input control gate and the output control gate according to the loss function; and determining the structure of the neural network according to the updated input control gate and the output control gate.

[0131] Clause A2, the method according to Clause A1, wherein the plurality of units comprises: a plurality of network operators arranged in a specific order; and an empty operator.

[0132] Clause A3, the method according to Clause A2, further includes: setting the input control gate of the empty operator corresponding to the starting layer of each level to 0.

[0133] Clause A4, the method according to Clause A2, further includes: setting the output control gate corresponding to the empty operator to 1.

[0134] Clause A5, the method described in Clause A1, further includes: measuring the delay of each unit; and creating a lookup table of all delays.

[0135] Clause A6, the method according to Clause A5, further includes: adding a sorting unit after the supernet, wherein the sorting unit includes at least one of a classifier and a detector; wherein the input step further calculates the loss function based on the output feature map of the sorting unit.

[0136] Clause A7. The method according to Clause A6, wherein the loss function comprises: a classifier loss, including a loss amount corresponding to the classifier and a loss amount corresponding to the detector; a control gate loss, corresponding to an L1 regularization loss amount corresponding to the input control gate and the output control gate; and a time loss, wherein the delay of each unit is calculated according to the lookup table.

[0137] Clause A8, the method described in Clause A7, wherein the loss function is:

[0138] λ0×L cls +λ1×L det +λ2×L gate +λ3×L lat

[0139] Where λ0 is the hyperparameter of the loss of the classifier, L cls Let λ be the loss of the classifier, λ1 be the hyperparameter of the loss of the detector, and L be the loss of the detector. det λ1 is the loss of the detector, λ2 is the hyperparameter of the control gate loss, and L is the loss of the detector. gate Let λ3 be the control gate loss, λ3 be the hyperparameter of the time loss, and L be the time loss. lat This refers to the time loss.

[0140] Clause A9, the method described in Clause A8, wherein λ0, λ1, λ2 and λ3 are greater than or equal to 0.

[0141] Clause A10, the method according to Clause A6, wherein the loss function comprises: classifier accuracy, including the accuracy corresponding to the classifier and the accuracy of the detector; control gate loss, corresponding to the L1 regularization loss of the input control gate and the output control gate; and time loss, calculated for the delay of each unit according to the lookup table.

[0142] Clause A11, the method described in Clause A10, wherein the loss function is:

[0143] λ0×Q cls +λ1×Q det +λ2×L gate +λ3×L lat

[0144] Where λ0 is the hyperparameter of the accuracy of the classifier, L cls Let λ be the accuracy of the classifier, λ1 be the hyperparameter of the accuracy of the detector, and L be the accuracy of the detector. det Let λ be the accuracy of the detector, λ2 be the hyperparameter of the control gate loss, and L be the value of the detector. gate Let λ3 be the control gate loss, λ3 be the hyperparameter of the time loss, and L be the time loss. lat This refers to the time loss.

[0145] Clause A12, the method described in Clause A11, wherein λ0 and λ1 are greater than or equal to 0, and λ2 and λ3 are less than 0.

[0146] Clause A13, the method described pursuant to Clause A7 or 10, wherein the time loss is:

[0147]

[0148] Where T0 is the preset target delay, and T is:

[0149]

[0150] in, For the input control gate of the h-th unit in the j-th layer of the i-th level, t h,c The delay is the timeout of the c-th channel coefficient corresponding to the h-th cell in the lookup table. This is the output control gate for the c-th channel coefficient corresponding to the i-th level and j-th layer;

[0151] Clause A14, the method described in Clause A1, wherein the sum of the values ​​of the input control gates at each level is 1.

[0152] Clause A15. The method according to Clause A1, wherein the output control gate includes multiple channel coefficients, each channel coefficient corresponding to a number of channels.

[0153] Clause A16, the method described in Clause A15, wherein the channel coefficients are four.

[0154] Clause A17. The method according to Clause A15 further includes: calculating an output feature map for each layer, said output feature map being:

[0155]

[0156] Among them, I i,j Let i be the input feature map of the i-th level and j-th layer. This is the input control gate for the h-th unit in the j-th layer of the i-th level. Let h be the output function of the i-th level, j-th layer, and h-th unit. This is the output control gate for the c-th channel coefficient corresponding to the i-th level and j-th layer;

[0157] The input step is based on the derivative of the output feature map.

[0158] Clause A18. The method described in Clause A1 further includes: determining whether the training progress has reached a progress threshold; and if so, performing the update step.

[0159] Clause A19, the method according to Clause A18, wherein the progress threshold is 25%.

[0160] Clause A20, the method according to Clause A1, wherein the decision step includes: setting the operator of each layer as the unit corresponding to the maximum value of the input control gate in the layer.

[0161] Clause A21, the method according to Clause A1, wherein the determination step includes: setting the number of channels per layer to the number of channels in the layer whose output control gate is greater than a channel threshold.

[0162] Clause A22, the method according to Clause A21, wherein the channel threshold is between 0 and 1.

[0163] Clause A23. A computer-readable storage medium having stored thereon computer program code for establishing a neural network, which, when run by a processing device, performs the method described in any one of Clauses A1-22.

[0164] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for establishing a neural network, comprising: A supernet is defined, which includes multiple levels, each level includes multiple layers, and each layer includes multiple units, wherein the multiple units include multiple network operators and an empty operator; An input control gate is set for each unit to reflect the probability of selecting the unit; An output control gate is set for each layer to determine the number of channels for that layer; The input image data is forward propagated into the supernet to calculate the loss function; The input control gate and the output control gate are updated according to the loss function. A classifier is added after the supernet, wherein the classifier includes a classifier and a detector. Each layer of the supernet generates an output feature map, and the output feature map of the last layer of the supernet is input into the classifier. The loss function is derived based on the output feature map of the supernet and the output feature map of the classifier. The output feature map generated by each layer of the supernet is as follows: in, Let i be the input feature map of the i-th level and j-th layer. This is the input control gate for the h-th unit in the j-th layer of the i-th level. Let h be the output function of the i-th level, j-th layer, and h-th unit. For the output control gate of the i-th level and j-th layer corresponding to the c-th channel coefficient; and The structure of the neural network is determined based on the updated input and output control gates; It also includes: measuring the delay of each unit; and creating a lookup table for all delays; wherein the loss function includes: classifier loss, including the loss amount corresponding to the classifier and the loss amount of the detector; control gate loss, corresponding to the L1 regularization loss amount of the input control gate and the output control gate; and time loss, calculating the delay of each unit according to the lookup table.

2. The method according to claim 1, further comprising: Set the input control gate of the empty operator corresponding to the starting layer of each level to 0.

3. The method according to claim 1, further comprising: Set the output control gate corresponding to the empty operator to 1.

4. The method according to claim 1, wherein the loss function is: + + in, The hyperparameter of the loss of the classifier is... The loss of the classifier. The hyperparameter for the loss of the detector is... The loss of the detector, The hyperparameter for the control gate loss is... For the control gate loss, The hyperparameter for the time loss is... This refers to the time loss.

5. The method according to claim 4, wherein λ 0、 λ 1. λ 2 and λ 3 is greater than or equal to 0.

6. The method according to claim 1, wherein the time loss is: in, T0 is the preset target delay, and T is: in, This is the input control gate for the h-th unit in the j-th layer of the i-th level. The delay is the timeout of the c-th channel coefficient corresponding to the h-th cell in the lookup table. This is the output control gate for the c-th channel coefficient corresponding to the j-th layer of the i-th level.

7. The method according to claim 1, wherein the sum of the values ​​of the input control gates in each layer is 1.

8. The method according to claim 1, wherein the output control gate includes a plurality of channel coefficients, each channel coefficient corresponding to a number of channels.

9. The method according to claim 8, wherein the channel coefficients are four.

10. The method according to claim 1, further comprising: Determine whether the training progress has reached the progress threshold; as well as If so, proceed with the update steps.

11. The method of claim 10, wherein the progress threshold is 25%.

12. The method of claim 1, wherein the determining step comprises: The operator for each layer is defined as the unit corresponding to the maximum value of the input control gate in that layer.

13. The method of claim 1, wherein the determining step comprises: The number of channels in each layer is set to the number of channels in that layer whose output control gate is greater than the channel threshold.

14. The method of claim 13, wherein the channel threshold is between 0 and 1.

15. A computer-readable storage medium having stored thereon computer program code for establishing a neural network, wherein when the computer program code is run by a processing device, it performs the method of any one of claims 1-14.

Images