A Neural Architecture Search Approach for Long-tail Datasets

Abstract

The present invention relates to the field of long-tail learning technology, and discloses a neural architecture search method for long-tail data sets. The differentiable architecture search method is improved from two aspects, namely, the search space and the search strategy, to obtain a neural network suitable for long-tail medical data sets. The improvement of the differentiable architecture search method includes: constructing a new search space: replacing the dilated convolution and the separable convolution of the search space in the differentiable architecture search method with the long-tail aggregate convolution and the long-tail hierarchical convolution in sequence to obtain a new search space; the search strategy is improved, including introducing the tail-perceived loss in the differentiable architecture search method, and increasing the regularization strength of the architecture weight in the differentiable architecture search method. The present invention defines a new search space suitable for long-tail data, and implements a new search strategy on the new search space, providing a reliable solution for architecture search in a long-tail data environment.

Description

Technical Field

[0001] The present invention relates to the field of long-tail learning technology, and in particular to a neural architecture search method for long-tail data sets. Background Art

[0002] With the vigorous development of artificial intelligence and the continuous advancement of medical informatization, medical technology based on artificial intelligence has become one of the hottest research directions at present. The introduction of deep learning for medical auxiliary diagnosis has great development prospects. It is well known that the performance of a deep learning model depends largely on the design of its network architecture. However, due to the large number of samples in medical scenarios and the variety of types, it is obviously unrealistic to design a unified neural network architecture to perform well in a variety of different tasks. Designing a network architecture for each task has the following disadvantages: (1) Designing the architecture requires professional knowledge, which is often difficult for medical staff to complete. They need to rely on external forces, which may lead to the leakage of highly private medical data. (2) Manually designing the network architecture is time-consuming, labor-intensive and inefficient.

[0003] In the current technological context, the Neural Architecture Search (NAS) algorithm provides a potential solution to the above problems. It can automatically search in a predefined architecture space and design architectures for specific tasks. However, in real medical environments, the number of samples of different categories of diseases varies significantly, resulting in a long-tail problem, which poses a huge challenge to traditional neural architecture search methods. After conducting relevant experiments, it was found that the existing NAS algorithm was used to search for architectures on long-tail medical datasets. The experimental results show that the performance of the searched architecture on long-tail medical datasets cannot reach the baseline standard (using the general network architecture designed by experts as the baseline).

[0004] Therefore, the present invention proposes a neural architecture search method for long-tail datasets. This method can search on an unbalanced dataset to obtain a network architecture that is more suitable for long-tail medical datasets. Its performance is better than that of the general architecture on long-tail data, providing a complementary angle for solving the long-tail problem. Summary of the invention

[0005] In response to the problem that existing neural architecture search methods perform poorly on long-tail medical datasets, the present invention proposes a neural architecture search method for long-tail datasets. Different from traditional neural architecture search methods, the present invention defines a new search space suitable for long-tail data and implements a new search strategy on the new search space. Based on these improvements, the present invention can search for architectures that perform well under long-tail medical data, providing a reliable solution for architecture search in a long-tail medical data environment.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A neural architecture search method for long-tail data sets is proposed. The differentiable architecture search method is improved from two aspects: search space and search strategy, so as to obtain a neural network suitable for long-tail medical data sets. The improvements to the differentiable architecture search method include:

[0008] A. Constructing a new search space : Differentiable architecture search method search space The dilated convolution and separable convolution are removed, and the long-tail aggregate convolution and long-tail hierarchical convolution are newly added to obtain a new search space ; Among them, the long-tail aggregate convolution is to enhance the recognition ability of the neural network for different disease data categories in the long-tail medical data set by aggregating the features of different regions; specifically, the long-tail aggregate convolution includes the first convolution kernel, the ReLU activation function, N1 convolution paths, the batch normalization operation and the second convolution kernel in sequence; the feature vectors output by the ReLU activation function are divided into N1 groups, and the feature vectors after the first convolution are obtained through N1 convolution paths respectively. ,right After batch normalization, the data is input to the second convolution kernel. The feature vector input to the long-tail aggregate convolution is concatenated with the feature vector output by the second convolution kernel as the output of the long-tail aggregate convolution. The number of disease data of different categories in the long-tail medical data set is significantly different. For example, there are at least disease data of category S1 and category S2 in the long-tail medical data set, and the number of disease data of category S1 and category S2 differs by more than 10 times.

[0009] The long-tailed hierarchical convolution includes the third convolution kernel, N2 convolution paths, batch normalization operation, ReLU activation function and the fourth convolution kernel in sequence; the feature vectors obtained after the third convolution kernel are divided into N2 groups, and N2 feature vectors after the second convolution are obtained through N2 convolution paths respectively. ,right After batch normalization, the vectors are input into the ReLU activation function and the fourth convolution kernel in sequence. The feature vectors input into the long-tailed hierarchical convolution are concatenated with the feature vectors output by the fourth convolution kernel as the output of the long-tailed hierarchical convolution.

[0010] B. Search strategy improvements, including the introduction of tail-aware loss in the differentiable architecture search method and the increase of architecture weights in the differentiable architecture search method The regularization strength of

[0011] Introducing tail-aware loss , specifically including: optimizing architecture weights When , the following sampling strategy is used to extract disease data: a part of the disease data is extracted by uniform distribution, and the other part of the disease data is extracted by reverse long-tail distribution; increase the weight of the architecture The regularization strength of , including: Regularization updates architecture weights ;

[0012] Weighting the architecture Updated total loss function for: ;in, is the regularization strength, Represents the architecture weight The L2 norm of the algorithm is obtained; an architecture search is performed on a specific long-tail medical dataset through an improved differentiable architecture search method, and a neural network suitable for the specific long-tail medical dataset is obtained.

[0013] Furthermore, the feature vectors output by the ReLU activation function are divided into N1 groups and passed through N1 convolution paths to obtain the first convolution feature vector , including:

[0014] ;

[0015] ;

[0016] Input to the long-tailed aggregation convolution The characteristic components of the convolutional paths, , Represents the long-tailed aggregation convolution The convolution kernel of the convolution path, represents the concatenation operation of feature vectors, Represents the i-th component of the eigenvector after the first convolution.

[0017] Furthermore, the feature vectors obtained after the third convolution kernel are divided into N2 groups, and N2 feature vectors after the second convolution are obtained through N2 convolution paths respectively. , including:

[0018] ;

[0019] in, is the feature component input to the jth convolution path of the long-tailed convolution layer, is the convolution kernel of the jth convolution path of the long-tailed convolution. Represents the feature vector output by the jth convolution path of the long-tail convolution.

[0020] Furthermore, the tail-perceived loss for:

[0021] ;

[0022] in, and They are the disease data extracted by using the inverse long-tail distribution method and the disease data extracted by using the uniform distribution method; is a hyperparameter; represents the neural network parameters.

[0023] Compared with the prior art, the beneficial technical effects of the present invention are:

[0024] The present invention can effectively run on long-tail medical datasets, providing a solution for the application of neural architecture search technology on unbalanced datasets. In addition, the present invention also provides a rarely discussed entry point for long-tail learning, that is, improving the performance of deep models on unbalanced datasets from the perspective of network architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of updating the architecture weights according to the present invention.

[0026] Figure 2 It is a schematic diagram of the structure of the long-tail aggregate convolution in the present invention; wherein 1×1 represents a convolution kernel of size 1, and 3×3 represents a convolution kernel of size 3.

[0027] Figure 3 It is a schematic diagram of the structure of the long-tail hierarchical convolution in the present invention; wherein 1×1 represents a convolution kernel of size 1, and 3×3 represents a convolution kernel of size 3. DETAILED DESCRIPTION

[0028] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0029] 1. Differentiable Architecture Search Method

[0030] The present invention is based on the mainstream neural network architecture search method - Differentiable Architecture Search (DARTS) for improvement. DARTS uses a differentiable method to represent the neural network architecture, and represents the different operations of each layer (such as convolution, pooling, skip connection, etc.) as trainable mixing coefficients. These mixing coefficients are learned through gradient descent to construct an optimal connection method between layers, called a cell. DARTS forms the final neural network architecture by stacking cells.

[0031] A cell is defined as a directed acyclic graph consisting of nodes and edges. The i-th ordered node in the directed acyclic graph is represents the feature graph; and connects the ordered nodes and ordered nodes Edge Represents the search space The selected Convert to Therefore, the intermediate node is calculated based on the ordered nodes of all its predecessors:

[0032] ; (1)

[0033] In order to make the search space continuous, DARTS adopts a continuous relaxation of the following form:

[0034] ; (2)

[0035] in, indicates a mixed operation, which will be replaced by .

[0036] are trainable architecture weights, which can be optimized in the following ways:

[0037] ; (3)

[0038] ;

[0039] in, Indicated by The architecture weights of represents the neural network parameters, The value is equal to the loss The smallest hour value. and Represent the training loss and validation loss respectively. The search process is based on and Gradient optimization is performed to find the best unit architecture and then the best neural network architecture.

[0040] 2. Technical Overview

[0041] like Figure 1 As shown, the present invention improves the original algorithm from two aspects, namely, the search space and the search strategy, so that DARTS can be better used on long-tail medical data sets.

[0042] First, the present invention defines a new search space, which contains candidate operations including: pooling operation, long-tailed aggregate convolution with a convolution kernel size of 3×3, long-tailed aggregate convolution with a convolution kernel size of 5×5, long-tailed hierarchical convolution with a convolution kernel size of 3×3, long-tailed hierarchical convolution with a convolution kernel size of 5×5, jump connection (that is, two nodes are directly connected, and the vector is directly transferred from one node to another without transformation) and zero operation (that is, there is no edge connecting the two nodes). The present invention first constructs a supernet, which consists of two input nodes, four intermediate nodes and one output node. There is an edge connecting every two nodes. The algorithm will calculate the vector based on the weights of each operation on the edge, and input the calculated weighted sum to the next node. Its formal representation is shown in formula (2).

[0043] The network parameters of the supernet include the architecture weights and neural network parameters The architecture weights control the probability of selecting each operation, while the neural network parameters are the weights of the neural network under a specific architecture. During initialization, the architecture weights are usually set to zero or close to zero, while the neural network parameters use standard initialization methods.

[0044] During the training process, the present invention first performs forward propagation, selects specific operations through the current architecture weights and applies them to the training data set. , and then calculate the cross entropy loss function Next, we use back propagation and gradient descent to calculate the neural network parameters .

[0045] In the neural network parameters After completing the single-step update, the architecture weight We will also use the validation dataset Specifically, the present invention uses a uniformly distributed validation data set. And the validation dataset of the inverse long-tail distribution Two batches of data are used for verification, and the loss function is tail-perceived loss , whose formal definition is as above. In addition, the present invention also proposes to use a stronger regularization coefficient μ to help improve the search performance. Through the back propagation of the tail-aware loss, the architecture weight Will be updated.

[0046] Through continuous iteration of the above two optimization processes, the network performance gradually improved, and finally the The largest operation is the final operation.

[0047] 3. Search space design

[0048] The present invention finds that the original search space Although the dilated convolution and separable convolution in the paper perform well on balanced datasets, they do not perform well on long-tailed medical datasets. To solve this problem, the two new convolution structures proposed in this paper, namely long-tailed aggregate convolution and long-tailed hierarchical convolution, construct a new search space. .

[0049] The core idea of ​​long-tailed aggregate convolution is to enhance the neural network's ability to recognize minority categories in long-tail distributions by aggregating features from different regions. Specifically, long-tailed aggregate convolution adopts a design paradigm with a pre-activation function, placing the ReLU activation function before the convolution operation. This design helps introduce nonlinearity before the convolution operation, thereby enhancing the neural network's ability to express different features. In addition, the use of the pre-activation function can also promote the propagation of gradients, which helps with optimization during training. The formal representation of the ReLU activation function is as follows:

[0050] ; (4)

[0051] Specifically, the internal design of the long-tailed aggregate convolution divides the feature vector into Groups, each group uses a convolution kernel for convolution operation, formally expressed as follows:

[0052] ; (5)

[0053] Input to the long-tailed aggregation convolution The characteristic components of the convolutional paths, , Represents the long-tailed aggregation convolution The convolution kernel of the convolution path. represents the first post-convolution feature vector obtained after convolution, Represents a vector concatenation operation.

[0054] The long-tailed aggregate convolution also combines batch normalization technology to reduce internal variable offset, accelerate the convergence speed of the neural network, and improve its generalization ability. Furthermore, the long-tailed aggregate convolution adopts a bottleneck topology similar to the residual network (ResNet), which effectively improves the feature extraction ability of the convolution structure. The specific design of the long-tailed aggregate convolution is as follows: Figure 2 shown.

[0055] The design of long-tailed hierarchical convolution uses layered convolution with post-activation function. In terms of topology and normalization method, it uses the bottleneck topology and batch normalization method respectively, just like long-tailed aggregate convolution. In addition, long-tailed hierarchical convolution uses ReLU post-activation function, which can reduce the number of parameters when implementing nonlinear transformation, which helps to reduce the amount of calculation and the size of neural network. In terms of specific detailed design, long-tailed hierarchical convolution uses hierarchical convolution. Assume that the feature vector is divided equally into Group, long-tailed hierarchical convolution uses The formal representation of convolution with convolution kernels is as follows:

[0056] ; (6)

[0057] in, is the feature component input to the jth convolution path of the long-tailed convolution layer, The convolution kernel of the jth convolution path of the long-tailed layer convolution, Represents the feature vector output by the jth convolution path of the long-tailed hierarchical convolution. The specific design of the long-tailed hierarchical convolution is as follows: Figure 3 shown.

[0058] The present invention replaces the proposed convolution operation with The original convolution operation in , thus establishing a new search space The present invention compares these two convolutions with the original dilated convolution and separable convolution, and the results are shown in Table 1. The proposed aggregate convolution (Agg.) and hierarchical convolution (Hier.) perform better than the original dilated convolution (Dil.) and separable convolution (Spe.).

[0059] Table 1 Performance comparison of different convolutional structures

[0060]

[0061] 4. Search strategy improvement

[0062] In the long-tail learning paradigm, neural networks are trained on imbalanced source distributions. training on a uniform target distribution The test is performed on , where there is a label distribution shift, i.e. The present invention improves the architecture weight Specifically, the present invention introduces a loss function for long-tail medical datasets and strengthens the The regularization strength of is set to search for architectures with better generalization ability.

[0063] In the original DARTS, the architecture weights The optimization of is shown in formula (3) and is performed on the validation set. However, in the case of long tail, following such an approach makes it difficult to evaluate the performance of the current neural network on the tail category, because the performance of the neural network on the tail category is necessarily lower than that on the head category, which brings about the trap of architecture optimization and makes the neural network constantly tend to the head category. To solve this problem, the present invention introduces tail-aware loss, which is an auxiliary loss that is sensitive to the performance of the tail category. Specifically, in the optimization Two different sampling strategies are used to extract validation disease data: one part of the disease data is extracted using a uniform distribution, and the other part of the disease data is extracted using an inverse long-tail distribution. Therefore, the tail-aware loss is defined as follows:

[0064] . (7)

[0065] in, and They are disease data extracted using the inverse long-tail distribution method and disease data extracted using the uniform distribution method. is a hyperparameter that determines the perception strength of the tail categories. Through this tail-aware loss, the present invention increases the sensitivity to the performance of the tail categories in the process of updating the architecture weights. While ensuring excellent performance on all categories, it also effectively captures the performance of the neural network on the tail categories. This approach significantly alleviates the tendency of the architecture weight update to continue to be biased towards the head category, making it possible to explore architectures that perform well on both the head and tail categories.

[0066] In addition, the long-tail problem involves the distribution shift challenge between the training set and the test set. The present invention solves it from the perspective of generalization. Regularization updates architecture weights , a neural network architecture with stronger generalization ability can be searched. Specifically, the present invention uses a stronger Regularization to constrain architecture weights The experimental results show that this method can search for neural network architectures with stronger generalization ability.

[0067] In summary, the present invention has a significant impact on the architecture weight. The updated loss function is as follows:

[0068] ; (8)

[0069] in, is the tail-aware loss, is the regularization strength, express The L2 norm of , which is the sum of the squares of all parameters.

[0070] Example

[0071] The present invention utilizes 4 cards of 24GB NVIDIA RTX3090 GPU, based on the 1.13.0 version of PyTorch and the 11.7 version of CUDA. The present invention uses long-tail data sets such as CIFAR-10-LT, CIFAR-100-LT, Places365-LT to simulate the data imbalance in medical scenarios, and verifies the effectiveness of the method, where the imbalance rate is 200, 100, and 50. For the hyperparameter settings in the operation of the neural network, the present invention mostly follows the default settings of the original DARTS, and only modifies some of the parameters. Specifically, the present invention uses fewer proxy layers (6 layers) to speed up the search process. The present invention sets a new hyperparameter , to help the architecture focus on samples of the tail category. In addition, the present invention applies a larger weight decay (L2 regularization), and the regularization strength =6×10e-4, to enhance the generalization ability of the neural network.

[0072] As shown in Table 2, the neural network searched by the method of the present invention (LT-DARTS) performs better than the general neural network architecture under the long-tail dataset.

[0073] Table 2

[0074]

[0075] The optimal neural network architecture searched by the present invention on the CIFAR-10-LT dataset and the CIFAR-100-LT dataset achieved better results than the general network architecture. Compared with baselines such as ResNet, ResNext and Res2Net, the neural network architecture searched by the method of the present invention (LT-DARTS) showed higher accuracy under equal neural network parameters. This superiority has been verified on different datasets and different balance coefficients (i.e., the ratio of the number of samples of the category with the largest number of samples to the number of samples of the category with the smallest number of samples), confirming the universality of the method of the present invention.

[0076] In Table 3, all comparison objects use ResNet32 as the backbone network, while LT-DARTS indicates that a neural network of similar size searched by the present invention is used as the backbone network.

[0077] Table 3

[0078]

[0079] In addition to the comparison with the baseline, the present invention is also compared with some mainstream state-of-the-art methods for solving the long-tail problem. Among them, CE+Mixup uses cross entropy as the loss function and Mixup data enhancement technology. LDAM+DRW uses weighted loss to increase the loss weight of the tail class. CE+Mixup+cRT uses a uniformly distributed data set to retrain the classifier based on CE+Mixup, which effectively alleviates the classifier's offset problem. BBN proposes to use two network branches, namely the traditional learning branch and the rebalancing branch, to handle long-tail recognition. The rebalancing branch applies a reverse sampler to sample more tail class samples in each small batch to improve the tail class performance. MiSLAS enhances representation learning through data mixing in the first stage, applies a label-aware smoothing strategy in the second stage to achieve better model generalization, and further solves the distribution offset problem between the two training stages by continuously updating the running mean and variance in the batch normalization layer. GCL proposes a Gaussian cloud-like logit adjustment method to improve the performance of the tail class by applying Gaussian perturbations of different magnitudes to logits of different categories.

[0080] Since the method in the present invention mainly focuses on backbone enhancement from the perspective of architecture, it may not be able to achieve state-of-the-art performance when used alone. However, by seamlessly integrating existing techniques for solving long-tail problems, the present invention can easily achieve significant performance improvements. For example, by integrating Mixup data enhancement and cRT classifier retraining technology into the neural network architecture searched by the present invention, superior performance can be achieved on the CIFAR-10-LT dataset and the CIFAR-100-LT dataset. In addition, the architecture searched by the present invention can obtain competitive results with fewer parameters.

[0081] Simply increasing the depth of the neural network searched by the present invention can significantly improve the performance of the neural network. However, in the context of ResNet, this principle does not hold true. As the number of parameters increases, the performance of ResNet's network is not significantly improved, or even decreases, while the performance of the architecture searched by the present invention gradually improves.

[0082] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0083] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A neural architecture search method for long-tail data sets, which improves the differentiable architecture search method from two aspects: search space and search strategy, to obtain a neural network suitable for long-tail medical data sets, characterized in that: Improvements to the differentiable architecture search method include: A. Constructing a new search space : Differentiable architecture search method search space The dilated convolution and separable convolution are removed, and the long-tail aggregate convolution and long-tail hierarchical convolution are newly added to obtain a new search space ; Among them, the long-tail aggregate convolution is to enhance the recognition ability of the neural network for different disease data categories in the long-tail medical data set by aggregating the features of different regions; specifically, the long-tail aggregate convolution includes the first convolution kernel, the ReLU activation function, N1 convolution paths, the batch normalization operation and the second convolution kernel in sequence; the feature vectors output by the ReLU activation function are divided into N1 groups, and the feature vectors after the first convolution are obtained through N1 convolution paths respectively. ,right After batch normalization, the data is input into the second convolution kernel; the feature vector input into the long-tail aggregate convolution is concatenated with the feature vector output by the second convolution kernel as the output of the long-tail aggregate convolution; the long-tail medical data set contains at least disease data of category S1 and category S2, and the number of disease data of category S1 and disease data of category S2 differs by more than 10 times; The long-tailed hierarchical convolution includes the third convolution kernel, N2 convolution paths, batch normalization operation, ReLU activation function and the fourth convolution kernel in sequence; the feature vectors obtained after the third convolution kernel are divided into N2 groups, and N2 feature vectors after the second convolution are obtained through N2 convolution paths respectively. ,right After batch normalization, the vectors are input into the ReLU activation function and the fourth convolution kernel in sequence. The feature vectors input into the long-tailed hierarchical convolution are concatenated with the feature vectors output by the fourth convolution kernel as the output of the long-tailed hierarchical convolution. B. Search strategy improvements, including the introduction of tail-aware loss in the differentiable architecture search method and the increase of architecture weights in the differentiable architecture search method The regularization strength of Introducing tail-aware loss , specifically including: optimizing architecture weights When , the following sampling strategy is used to extract disease data: a part of the disease data is extracted by uniform distribution, and the other part of the disease data is extracted by reverse long-tail distribution; increase the weight of the architecture The regularization strength of , including: Regularization updates architecture weights ; Weighting the architecture Updated total loss function for: ;in, is the regularization strength, Represents the architecture weight The L2 norm of the algorithm is obtained; an architecture search is performed on a specific long-tail medical dataset through an improved differentiable architecture search method, and a neural network suitable for the specific long-tail medical dataset is obtained.

2. The neural architecture search method for long-tail data sets according to claim 1, characterized in that: The feature vectors output by the ReLU activation function are divided into N1 groups and passed through N1 convolution paths to obtain the feature vector after the first convolution. , specifically including: ； ； Input to the long-tailed aggregation convolution The characteristic components of the convolutional paths, , Represents the long-tailed aggregation convolution The convolution kernel of the convolution path, represents the concatenation operation of feature vectors, Represents the i-th component of the eigenvector after the first convolution.

3. The neural architecture search method for long-tail datasets according to claim 1, characterized in that The feature vectors obtained after the third convolution kernel are divided into N2 groups, and N2 feature vectors after the second convolution are obtained through N2 convolution paths respectively. , specifically including: ； in, is the feature component input to the jth convolution path of the long-tailed convolution layer, is the convolution kernel of the jth convolution path of the long-tailed convolution. Represents the feature vector output by the jth convolution path of the long-tail convolution.

4. The neural architecture search method for long-tail datasets according to claim 1, characterized in that The tail-perceived loss for: ； in, and They are the disease data extracted by using the inverse long-tail distribution method and the disease data extracted by using the uniform distribution method; is a hyperparameter; represents the neural network parameters.

Images