A Method for Automatically Searching Neural Network Structures

Abstract

The present invention discloses a method for automatically searching a neural network structure. The network model of this method is implemented in a differentiable parameterized form, getting rid of the process of repeatedly verifying the search structure and greatly reducing the search cost. This method comprehensively considers the role of channel performance measurement of channel attention, measures and sorts channels before the input channels are sent to candidate operations, only samples some important channels for calculation to reduce the video memory occupation, and avoids the problems of unstable structure search and degradation of the image classification performance of the search structure caused by the loss of important information, thus improving the search efficiency. This method designs a temperature regularization coefficient in the process of continuousization of structure weights, expands the weight gap between different candidate operations, reduces the discretization error, improves the stability and accuracy of the method, and thus improves the relevance in the search and evaluation stages.

Description

Technical Field

[0001] The present invention relates to a method for searching neural network structures. Background Art

[0002] With the continuous deepening of machine learning theory and the improvement of computing hardware performance, Convolutional Neural Networks (CNNs) have been increasingly widely used in the field of computer vision. Traditional convolutional neural network structures are usually designed manually. These designed structures may be relatively complex, such as classic network structures like VGGNet, ResNet, DenseNet, etc., and there are also some lightweight structures, such as MobileNet and ShuffleNet, etc. Although these classic structures can be adjusted and optimized according to different visual tasks, they still have certain limitations in terms of flexibility. However, the design of network structures requires sufficient professional knowledge and repeated trials, and has high requirements for computing hardware, which to a certain extent limits the application of deep neural networks. Therefore, the emergence of Neural Architecture Search (NAS) is of great significance for promoting the development of deep learning.

[0003] NAS is a method for automatically designing network structures. Since convolutional neural networks play an important role in deep learning, current research work mainly focuses on the structure search of convolutional neural networks, and automatically searches for the optimal network structure through specific search strategies. In 2016, Zoph et al. proposed a NAS method based on reinforcement learning, and the accuracy of this method on the CIFAR10 dataset exceeded that of classic network structures designed manually such as VGGNet and ResNet. Since then, many NAS methods based on search strategies such as evolutionary learning, reinforcement learning, and gradient descent have been proposed. Although these methods have achieved good performance, their computational overhead is very large. For example, the method proposed by Zoph et al. requires 22,400 GPU days to complete the search task on CIFAR10.

[0004] Differentiable Architecture Search (DARTS) integrates structure search into model training and reduces the search cost by continuousizing discrete problems, but it has a bottleneck in terms of video memory occupancy. For this reason, researchers proposed the Partial Channel Connections for Memory-Efficient Architecture Search (PC-DARTS) method, which reduces video memory occupancy by randomly sampling partial channels, thereby improving the efficiency and stability of the method.

[0005] The research found that there are still two key problems with PC-DARTS: 1) The selection of some channels is randomly selected, which easily leads to the loss of effective information during the training process of the method. When the number of selected channels is reduced to a certain extent, the performance of the method drops significantly; 2) To maximize the information flow, the weights of candidate operations tend to converge to a concentrated interval, which will cause a large discretization error when the final discretization operation is performed. Summary of the Invention

[0006] Object of the Invention: Aiming at the information loss caused by PC-DARTS using fewer channels with random sampling, and aiming at the problem of larger discretization error when selecting the optimal structure, a method for automatically searching neural network structures (MCP-DARTS) is proposed. The method measures the importance of each channel, selects some important channels for sampling, and makes the weight distribution for judging the pros and cons of each candidate operation more discriminative during the structure search process.

[0007] Technical Solution: A method for automatically searching neural network structures includes the following steps:

[0008] Step 1: Obtain data and preprocess the data;

[0009] Step 2: Construct a unit network;

[0010] Step 3: Channel sampling;

[0011] Step 4: Expand the structural weight difference;

[0012] Step 5: Secondary optimization of structure search;

[0013] Step 6: Repeat Step 3 to Step 5 until the network iteration times reach the set iteration times;

[0014] Step 7: The training ends, and the network parameters are fixed and no longer updated;

[0015] Step 8: Obtain the final unit network structure through the learned structure weights, and the one with the largest probability calculated by softmax is the optimal candidate operation;

[0016] Step 9: Expand and stack the final unit network structure into an M-layer unit network, and retrain and verify the structure performance.

[0017] Further, the Step 1 includes: reading the input pictures, the dataset is CIFAR10, which consists of 60,000 pictures with a resolution of 32×32, including 50,000 training pictures and 10,000 test pictures; each picture is converted into three RGB channels and converted into a three-dimensional matrix of w×h×c, where w and h are the width and height of the image respectively, and c is the number of channels; the dataset is converted into n SFour-dimensional matrix data of ×w×h×c, n S represents the number of samples; the picture categories are encoded in one-hot. If there are N categories of pictures in total, the label of the first category is represented as [1, 0, 0, ..., 0] 1×N and the label of the second category is represented as [0, 1, 0, ..., 0] 1×N …, and the label of the Nth category of pictures is represented as [0, 0, 0, ..., 1] 1×N .

[0018] Further, step 2 includes: The unit network is a directed acyclic graph composed of C sequence nodes. There are 8 candidate operations between every two nodes, namely zero operation, 3×3 max pooling, 3×3 average pooling, skip connection, 5×5 depthwise convolution, 3×3 depthwise convolution, 3×3 dilated depthwise convolution, and 5×5 dilated depthwise convolution. The unit network obtained by searching is repeatedly stacked to form the final neural network structure. The softmax function can be used to continuously represent the selection probabilities of all candidate operations in the unit network, so as to realize the optimization of the continuous space during the search process. The process of node x i transmitting to node x j is as follows:

[0019]

[0020] where O represents the set of candidate operations, o(·) is a specific candidate operation, represents the candidate operation weight between node x i and node x j . Therefore, the structure search task can be simplified to learning a set of continuous variables

[0021] Further, step 3 includes:

[0022] 3.1 Channel sorting: Measuring the channel performance, this method uses global average pooling and global max pooling to compress the information of each channel into a numerical feature, and sends these two features into a two-layer fully connected neural network with shared weights for calculation. Finally, the channel attention coefficient Mc(F) between 0 and 1 is obtained through sigmoid function activation as follows:

[0023] M c (F) = σ(MLP(Avgpool(F)) + MLP(Maxpool(F))) (2)

[0024] Among them, F represents the input features of each channel, σ represents the sigmoid function, MLP represents the fully connected layer, Avgpool and Maxpool represent global average pooling and global max pooling respectively; the channels are sorted in descending order according to the obtained channel attention coefficients;

[0025] 3.2 Partial channel connection: Select the first 1 / K channels for sampling, and perform masking operations on the remaining channels. The final output f i transmitted to node x j is: (i,j) (x i ; K) is:

[0026]

[0027] Among them, and represent the selected and masked channel ratios, and K is the channel selection ratio.

[0028] Furthermore, the step 4 includes: After the differentiable architecture search method finishes searching, select the candidate operation corresponding to the maximum softmax value; Add a temperature regularization coefficient to the softmax function in Equation (3), and the propagation process from node i to node j becomes:

[0029]

[0030] Among them, T is a hyperparameter set manually between 0 and 1.

[0031] Furthermore, the step 5 includes:

[0032] 5.1 Alternately optimize the training loss and the validation loss:

[0033]

[0034]

[0035] Among them, α and ω represent the structure weight and the network weight respectively, ω * (α) represents the fixed network weight, L val and L train represent the validation set loss and the training set loss respectively, and alternately optimize the two by fixing the structure weight and the fixed network weight;

[0036] 5.2 Network parameter update: Update the learning rates of the network weight and the structure weight by the gradient descent method:

[0037]

[0038]

[0039] Among them, η1 and η2 respectively represent the learning rate of network weights and the learning rate of structural weights, and α k is the structural weight at the k-th iteration, and ω k is the network weight at the k-th iteration.

[0040] Beneficial effects: The present invention provides a new research idea for further improving the search efficiency of differentiable neural network structure search. The proposed mechanism for measuring and sorting channel performance using the channel attention mechanism can avoid the loss of a large amount of effective image information on the premise of reducing the redundancy of the search space, and can further select fewer channels for the calculation of structure search, saving a large amount of search time. At the same time, the present invention adds a temperature regularization coefficient when continuousizing the search space, increasing the distribution of structural weights, making the performance advantages and disadvantages of each candidate operation more distinguishable, ensuring the discriminability of different candidate operations when the final structure is selected, so as to find the most suitable network structure for the current task and improve the search stability of the method. Brief Description of the Drawings

[0041] Figure 1 is a flowchart of the method for automatically searching for neural network structure construction according to the present invention;

[0042] Figure 2 is the unit network structure obtained by searching on CIFAR10 of the present invention. Detailed Embodiments

[0043] The following further explains the present invention with reference to the accompanying drawings.

[0044] Aiming at the problems that the network is prone to losing effective information during training and generating large discretization errors when selecting the optimal structure, the present invention provides a method for automatically searching for neural network structures. This method can obtain a faster search speed on the basis of ensuring good search performance, solves the problem of serious loss of image information caused by randomly sampling channels, and further improves the structure search efficiency. In addition, in traditional differentiable neural network structure search, the convergence interval of structural weights is relatively concentrated when selecting the optimal structure after the search. This method adds a temperature regularization coefficient when continuousizing the search space, which can expand the weight difference of each candidate operation during training, thereby reducing the discretization error generated by this method.

[0045] The channel sorting process is a typical process of using the attention mechanism to focus on local information, generally consisting of four stages: compression, learning, weighting, and sorting. Specifically, the image information contained in each channel is different. After the weighting stage, according to the attention coefficients obtained by learning, each channel is sorted from large to small. By focusing on the information useful for the current task, some channels can be selectively selected for the structure search calculation. The fundamental motivation is to select fewer channel connections without reducing the search performance. As Figure 1 shown, a method for automatically searching for a neural network structure according to the present invention includes the following steps:

[0046] (1) Obtain data: Read the input pictures; the dataset is CIFAR10, which consists of 60,000 pictures with a resolution of 32×32, including 50,000 training pictures and 10,000 test pictures; each picture is converted into three RGB channels and converted into a three-dimensional matrix of w×h×c, where w and h are the width and height of the image respectively, and c is the number of channels; in this way, the pictures can be converted into four-dimensional matrix data of n S ×w×h×c, and n S represents the number of samples; the picture categories are encoded in one-hot. If there are N categories of pictures in total, the label of the first category is [1,0,0,...,0] 1×N , the label of the second category is represented as [0,1,0,...,0] 1×N , …, the label of the Nth category of pictures is represented as [0,0,0,...,1] 1×N .

[0047] (2) Construct the unit network: The unit network is a directed acyclic graph composed of C sequence nodes. There are 8 candidate operations between every two nodes, namely zero operation, 3×3 max pooling, 3×3 average pooling, skip connection, 5×5 separable convolution, 3×3 separable convolution, 3×3 dilated separable convolution, and 5×5 dilated separable convolution. The unit networks obtained by searching are repeatedly stacked to form the final neural network structure. The softmax function can be used to continuously represent the selection probabilities of all candidate operations in the unit network, so as to realize the optimization of the continuous space during the search process. The process of node x i transmitting to node x j is as follows:

[0048]

[0049] Among them, O represents the set of candidate operations, o(·) is a candidate operation such as convolution and pooling, represents the candidate operation weight between node x i and node x j . Therefore, the structure search task can be simplified to learning a set of continuous variables

[0050] (3) Channel sampling, which is implemented by the following sub-steps:

[0051] (3.1) Channel sorting: Measure the channel performance. In this method, global average pooling and global max pooling are used to compress the information of each channel into a numerical feature, and these two features are fed into a two-layer fully connected neural network with shared weights for calculation. Finally, the channel attention coefficient Mc(F) between 0 and 1 is obtained through sigmoid function activation as follows:

[0052] M c (F) = σ(MLP(Avgpool(F)) + MLP(Maxpool(F))) (2)

[0053] where F represents the input features of each channel, σ represents the sigmoid function, MLP represents the fully connected layer, and Avgpool and Maxpool represent global average pooling and global max pooling respectively. The channels are sorted in descending order according to the obtained attention coefficients.

[0054] (3.2) Partial channel connection: Select the top 1 / K channels for sampling, and perform masking operations on the remaining channels. The final output f i transmitted to node x j is: (i,j) (x i ; K) is:

[0055]

[0056] where, and represent the proportions of the selected and masked channels, K is the channel selection ratio. Increasing the value of K will result in a decrease in the number of selected channels, thereby reducing the computational overhead, allowing a larger batch size to be used to improve the running efficiency, sampling more data in each iteration, reducing the uncertainty brought by weight updates and structure updates, and thus improving the stability of the structure search. In addition, since the masked channel part can alleviate the vanishing gradient and select fewer channels for sampling, a better skip connection regularization effect can be achieved for the differentiable structure search method, thereby improving the stability of the search structure. In short, increasing the value of K can improve the stability and efficiency of the method by reducing the computational overhead and improving the regularization effect.

[0057] (4) Expand the structural weight difference: After the differentiable structure search method finishes searching, select the one with the maximum softmax Candidate operations corresponding to the values. During the search process, the input-output transfer between nodes is obtained by weighted summation of all candidate operations. The structural weights of the candidate operations tend to converge to a concentrated interval to improve the information flow of the network, resulting in weak correlation of the search structure. To make the method more discriminative in selecting the optimal candidate operation, a temperature regularization coefficient is added to the softmax function in Equation (3). The propagation process from node i to node j becomes:

[0058]

[0059] where T is a hyperparameter set manually between 0 and 1. Introducing the temperature regularization coefficient makes the weights of different operations diverge, becoming more discriminative, thereby expanding the range for determining excellent operations, and further achieving the purpose of reducing discretization errors and narrowing the performance gap of the network between the search stage and the evaluation stage.

[0060] (5) Secondary optimization of the structure search, which is implemented by the following sub-steps:

[0061] (5.1) Alternately optimize the training loss and the validation loss:

[0062]

[0063]

[0064] where α and ω represent the structural weight and the network weight respectively, ω * (α) represents fixing the network weight, L val and L train represent the validation set loss and the training set loss respectively, and they are alternately optimized by fixing the structural weight pair and the network weight pair.

[0065] (5.2) Update network parameters: η1 and η2 represent the learning rates of the network weight and the structural weight respectively, and their parameters are updated by the gradient descent method:

[0066]

[0067]

[0068] where η1 and η2 represent the learning rates of the network weight and the structural weight respectively, α k is the structural weight at the k-th iteration, and ω k is the network weight at the k-th iteration.

[0069] (6) Repeat steps (3), (4), and (5) until the number of network iterations reaches the set number of iterations.

[0070] (7) Training is completed, and the network parameters are fixed and no longer updated.

[0071] (8) Obtain the final cell network structure through the learned structure weights, and the one with the highest probability calculated by softmax is the optimal candidate operation.

[0072] (9) Expand and stack the final cell network structure into an M-layer cell network, and retrain to verify the structure performance.

[0073] The network model of this method is implemented in a differentiable parameterized form, getting rid of the process of repeatedly verifying the search structure, and greatly reducing the search cost. This method comprehensively considers the role of channel performance measurement of channel attention, measures and sorts channels before the input channels are sent to the candidate operations, only samples some important channels for calculation to reduce the video memory occupancy, and avoids the problems of unstable structure search and decreased image classification performance of the search structure caused by the loss of important information, improving the search efficiency. This method designs a temperature regularization coefficient during the process of continuous structure weights, expanding the weight gap between different candidate operations, reducing the discretization error, enhancing the stability and accuracy of the method, and thus improving the relevance between the search and evaluation stages.

[0074] The present invention can be further illustrated by the following experiments:

[0075] To verify the effectiveness of the present invention, experiments were conducted on CIFAR10 and ImageNet respectively. Among them, ImageNet was only used to verify the structure performance. CIFAR10 consists of 60,000 pictures with a resolution of 32×32, including 50,000 training pictures and 10,000 test pictures. These pictures in CIFAR10 are evenly distributed among 10 classes, and the training pictures are evenly divided into two subsets, which are respectively used to train the network weight structure weights. The ImageNet dataset includes 1.3 million training pictures and 50,000 validation pictures of 1000 object categories, and the picture size is 224×224.

[0076] To make the effect of the present invention more intuitive, 5 independent experiments were conducted respectively, and the performance was compared with PC-DARTS (K = 4) and PC-DARTS (K = 8), as shown in Table 1. The visualization results of the searched normal cell and reduction cell network structures are respectively as Figure 2 shown in (a) and (b).

[0077] Table 1 Comparison experiment results of PC-DARTS with different K values and MCP-DARTS on CIFAR10

[0078]

[0079] On the CIFAR10 dataset, by comparing the results of PC-DARTS (K = 8) and MCP-DARTS, it is found that the accuracy of PC-DARTS (K = 8) is lower, while MCP-DARTS improves the average accuracy by about 0.6 percentage points on the basis of only increasing 0.01 GPU days. Moreover, in 5 independent experiments, the accuracy of the optimal structure has also been significantly improved. Compared with PC-DARTS (K = 4), MCP-DARTS can save 29.6% of the search time, and the average accuracy is higher than 97.33%. The highest accuracy of the obtained results reaches 97.44%. MCP-DARTS uses the channel attention mechanism to measure the performance of each channel during the training process, thus reducing the loss of image information generated during the structure search when selecting fewer channel connections, and using the same batch size during the search phase to reduce the video memory occupation between nodes and improve the video memory utilization rate. Therefore, compared with PC-DARTS, MCP-DARTS can improve the search efficiency while ensuring the same accuracy.

[0080] The comparison of the verification performance of the structures obtained by searching with different T values is shown in Table 2. It can be seen that when T = 1, due to the concentrated distribution range of the weights of each candidate operation, the discretization error is relatively large, resulting in poor model classification accuracy and stability. When T = 0.2, the differences between the candidate operations on each edge are obvious, the accuracy of the method has been greatly improved, and the stability of the method has also been improved. When the T value is too small, the information flow of the neural network is restricted, and it is easy to fall into the local optimal solution, and the exploration of the structure search becomes weaker, resulting in the influence on the stability of the method and large fluctuations in accuracy. Therefore, selecting a reasonable temperature regularization coefficient can significantly improve the accuracy and stability of the method.

[0081] Table 2 Influence of Regularization Temperature Coefficient on Accuracy

[0082] T value Accuracy 1 97.20±0.1 0.6 97.31±0.08 0.2 97.35±0.07 0.05 97.07±0.22

[0083] Table 3 and Table 4 respectively compare the performance results of this method and several other structure search methods on the CIFAR10 and ImageNet datasets. In the method comparison, the error rate, the number of model parameters, and the GPU time required for the search of the search structure on the two datasets are used as the indicators to measure the method performance. Avg in Table 3 represents the average value of the results of 5 independent experiments, and Best represents the optimal experimental result.

[0084] Compared with the early manually designed CNN structures, NAS methods can automatically search for better network structures, with a significant improvement in accuracy, fewer parameters, and easier to train. Among the NAS methods based on reinforcement learning, NASNet-A has good accuracy performance, but the search process takes a lot of time and is difficult to apply to actual scenarios. The search time of BlockQNN is shortened, but the number of model parameters is too large and difficult to train. ENAS further shortens the search time through weight sharing, but the accuracy decreases compared with NASNet-A. Evolution-based NAS methods generally have the problem of large computational overhead.

[0085] The gradient-based NAS method shows unique superiority compared with other design methods due to its efficient structure search and lightweight search model. This method is a further improvement based on its research and can search for a structure with a minimum error rate of 2.56% on CIFAR10 and top-1 / 5 error rates of 25.9% / 8.4% on ImageNet in only 0.09 GPU days. Compared with PC-DARTS and other gradient-based NAS methods, MCP-DARTS has better search efficiency with less search time while ensuring search accuracy. Moreover, the method not only performs well on small datasets but also can be extended to large-scale image classification tasks.

[0086] Table 3 Performance of manually designed network structures and NAS method structures on CIFAR10

[0087] Structure Error rate (%) Number of parameters (MB) GPU (days) Design method DesNet (k = 12) 5.2 1.0 - Manual VGG 6.7 20.4 - Manual DesNet-BC 3.8 25.6 - Manual ResNet (depth = 101) 6.4 25.3 - Manual NASNet-A + cutout 2.65 3.3 2000 Reinforcement-based BlockQNN 3.54 39.8 96 Reinforcement-based ENAS 2.89 4.6 0.5 Reinforcement-based LargeScaleEvolution 5.4 5.4 2750 Evolution-based AmoebaNet-A + cutout 3.12 3.1 3150 Evolution-based Hierarchicalevolution 3.6 15.7 300 Evolution-based ProxylessNAS + cutout 2.08 5.7 4 Gradient-based BayesNAS + cutout 2.81±0.04 3.4 0.2 Gradient-based DARTS (firstorder) + cutout 3.00±0.14 3.3 0.4 Gradient-based DARTS (secondorder) + cutout 2.76±0.09 3.3 1 Gradient-based SNAS (moderate) + cutout 2.85±0.02 2.8 1.5 Gradient-based P-DARTS + cutout 2.5 3.4 0.3 Gradient-based SGAS + cutout 2.66±0.24 3.7 0.25 Gradient-based Fair-DARTS + cutout 2.54±0.05 3.32 0.4 Gradient-based PC-DARTS (avg) 2.67±0.06 3.6 0.125 Gradient-based PC-DARTS (best) 2.57 3.6 0.125 Gradient-based MCP-DARTS (avg) 2.65±0.07 3.6 0.09 Gradient-based MCP-DARTS (best) 2.56 3.4 0.09 Gradient-based

[0088] Table 4 Performance of manually designed network structures and NAS method structures on ImageNet

[0089]

[0090]

[0091] The present invention is a method for automatically searching neural network structures. Experimental results on the CIFAR10 dataset prove that this method can search for a CNN structure with a classification error rate of only 2.56% in only 0.09 GPU days, showing an obvious search efficiency advantage compared with mainstream methods. It should be noted that the current gradient-based structure search is mainly applicable to small datasets. For large datasets such as ImageNet, the structure search on CIFAR10 is still used and network stacking is performed. How to directly perform efficient structure search on large datasets remains the research focus of subsequent work.

[0092] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for automatically searching neural network structures, characterized in that, It includes the following steps: Step 1: Obtain image data and preprocess the data; Step 2: Construct a cell network; the cell network is a directed acyclic graph composed of C sequence nodes, and there are 8 candidate operations between every two nodes, namely zero operation, 3×3 max pooling, 3×3 average pooling, skip connection, 5×5 separable convolution, 3×3 separable convolution, 3×3 dilated separable convolution, and 5×5 dilated separable convolution; Step 3: Channel sampling; It includes channel sorting and partial channel connection; Step 5: Use the differentiable architecture search method to enlarge the architecture weight difference; Step 6: Second-level optimization of architecture search; it includes alternately optimizing the training loss and the validation loss, and updating the network parameters; Step 7: Repeat Step 3 to Step 5 until the network iteration times reach the set iteration times; Step 8: The training ends, and the network parameters are fixed and no longer updated; Step 9: Obtain the final cell network architecture through the learned architecture weights, and the one with the largest probability calculated by softmax is the optimal candidate operation; Step 10: Stack the final cell network architecture into an M-layer cell network and retrain to verify the architecture performance.

2. The method for automatically searching a neural network structure according to claim 1, wherein Step 1 includes: reading the input pictures. The dataset is CIFAR10, which consists of 60,000 pictures with a resolution of 32×32, including 50,000 training pictures and 10,000 test pictures; each picture is converted into three RGB channels and then into a three-dimensional matrix of w×h×c, where w and h are the width and height of the image respectively, and c is the number of channels; the dataset is converted into a four-dimensional matrix data of n S ×w×h×c, where n S represents the number of samples; the picture categories are encoded using one-hot encoding. If there are N categories of pictures in total, the label of the first category is represented as [1,0,0,...,0] 1×N , the label of the second category is represented as [0,1,0,...,0] 1×N , …, the label of the Nth category of pictures is represented as [0,0,0,...,1] 1×N .

3. The method for automatically searching a neural network structure according to claim 2, wherein Step 2 includes: repeatedly stacking the obtained unit networks to form the final neural network structure; the softmax function can be used to continuously represent the selection probabilities of all candidate operations in the unit network, so as to achieve optimization in the continuous space during the search process, node x i to node x j The transfer process is as follows: where $O$ represents the set of candidate operations, and $o(\cdot)$ is a specific candidate operation, represents node $x$ i and node $x$ j the candidate operation weight between them. Therefore, the architecture search task can be simplified to learning a set of continuous variables 4. The method for automatically searching a neural network structure according to claim 3, wherein The said Step 3 includes: 3.1 Channel sorting: Measure the channel performance. In this method, global average pooling and global max pooling are used to compress the information of each channel into a numerical feature, and these two features are sent into a two-layer fully connected neural network with shared weights for calculation. Finally, the channel attention coefficient Mc(F) between 0 and 1 is obtained through sigmoid function activation as follows: M c (F) = σ(MLP(Avgpool(F)) + MLP(Maxpool(F))) (2) where F represents the input features of each channel, σ represents the sigmoid function, MLP represents the fully connected layer, Avgpool and Maxpool represent global average pooling and global max pooling respectively; the channels are sorted from large to small using the obtained channel attention coefficients; 3.2 Partial channel connection: Select the first 1 / K channels for sampling, and perform masking operations on the remaining channels, node x i to node x j The final output f (i,j) (x i ; K) is: Among them, and represent the selected and masked channel ratios, and K is the channel selection ratio.

5. The method for automatically searching a neural network structure according to claim 4, wherein Step 4 includes: after the differentiable structure search method finishes searching, selecting the candidate operation corresponding to the maximum value of softmax ; adding a temperature regularization coefficient to the softmax function in Equation (3), and the propagation process from node i to node j becomes: where T is a hyperparameter set between 0 and 1.

6. The method for automatically searching a neural network structure according to claim 5, wherein The said Step 5 includes: 5.1 Alternately optimize the training loss and the validation loss: Among them, α and ω represent the structure weight and the network weight respectively, and ω * (α) represents the fixed network weight, L val and L train represent the validation set loss and the training set loss respectively, and the two are alternately optimized by fixing the structure weight and the fixed network weight respectively; 5.2 Network parameter update: Update the parameters of the learning rate of the network weights and the learning rate of the architecture weights through the gradient descent method: Among them, η1 and η2 represent the learning rate of the network weights and the learning rate of the structure weights, respectively, and α k is the structure weight of the k-th iteration, and ω k is the network weight of the k-th iteration.

Images