A flatness-based lightweight neural network architecture search method

Abstract

The application provides a lightweight neural network architecture search method based on flatness, which is used for reducing the time cost and computing resources of searching the neural network architecture. From the perspective of predicting the network generalization, the application provides a method for comparing the networks by taking the flatness of the candidate network at the initial time as an evaluation index. The method comprises the following steps: determining an image processing task and determining a data set; determining a network search space; selecting a certain number of neural network architectures in the search space; verifying the effectiveness of the evaluation index; calculating the flatness of the neural network architecture, and sorting the selected architectures according to the flatness; selecting the architecture with the maximum flatness as the optimal architecture; and training the optimal architecture to complete the image classification task. The application can select an architecture with high accuracy in a small time consumption.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and more specifically, it is a flatness-based neural network architecture search method in the field of deep learning. It can be applied to tasks such as image classification, object detection, and semantic segmentation. Background Technology

[0002] Deep learning, a crucial branch of machine learning, has achieved tremendous success in many fields in recent years due to its powerful automatic representation capabilities. It has been proven that the design of the network architecture is crucial for the feature representation of data and the final performance. To obtain good feature representations of data, researchers have designed various complex network architectures. However, the design of network architectures largely depends on researchers' prior knowledge and experience, significantly limiting the development of deep learning. Due to the inherent limitations of human knowledge, it is difficult to break free from existing thought paradigms and design an optimal model. Even data scientists need to spend a significant amount of time and effort, constantly experimenting and adjusting parameters to achieve the best performance of the designed network model on a specific dataset. As the field of deep learning continues to develop and deepen, the amount of data processed is increasing, and the networks are becoming increasingly deep. For different tasks, different neural networks are often required to extract the best feature information. These problems bring many inconveniences to the design and application of neural networks. Therefore, neural architecture search has emerged.

[0003] Neural architecture search utilizes search algorithms to find the optimal neural network architecture within a search space, which refers to the set of candidate neural network architectures. Early neural network architecture search techniques employed reinforcement learning and evolutionary algorithms. These two methods consumed enormous computational resources and had excessively long search times, making it impossible to complete search tasks on large datasets. This significantly limited the development and practical application of neural network architecture search.

[0004] If we could find a heuristic metric that allows for lightweight training or even no training required to directly measure the final performance of a current network architecture, or to obtain a relative ranking of the final performance of different network architectures, we could significantly reduce costs and time. This invention proposes a more efficient and convenient neural network architecture search method based on the flatness of the neural network and uses this to evaluate the initial architecture to measure its final performance. A good example of this method, using image classification, demonstrates its effectiveness. Summary of the Invention

[0005] The problem this invention addresses is that for machine learning tasks, such as image classification, researchers often need to conduct extensive experiments on existing network architectures to select the optimal neural network architecture model when using network architecture search techniques. However, our goal is not only to find the optimal neural network architecture, but also to reduce the time and cost required during the search process, achieving a perfect balance between efficiency and optimization. Based on this, this invention proposes a lightweight training-based flatness-based neural network architecture search method. This method measures the generalization error during network convergence by calculating the initial flatness value of all networks in the selected network search space, and uses this value to select the optimal network architecture, significantly reducing the time and resources required to search for the optimal architecture.

[0006] This invention is generally divided into four main parts:

[0007] Select a dataset, set up a search space for neural network architectures that contains all desired network architectures, and search for the optimal network architecture within this space.

[0008] For a specific image classification task, samples are taken from different network architectures in the search space, and the initial flatness of each network is calculated. This flatness is used as an indicator to measure the generalization error of the network.

[0009] Calculate the correlation coefficient between flatness and final generalization error for different network architectures to verify the correctness of the metrics.

[0010] The optimal network architecture is selected by searching the entire network space based on the indicators, and the network is trained to achieve the image classification task.

[0011] The technical solution adopted by the present invention to achieve the above-mentioned objectives is as follows:

[0012] 1. We chose a unified neural network architecture search and comparison platform, NAS-Bench-201, as our search space. This space is based on the cell stacking construction method, providing a unified benchmark for most state-of-the-art NAS algorithms. Within this space, we adopted a scheme of repeated cell stacking, using relaxation techniques to make discrete network architectures continuous, thus creating a continuous search space that includes all candidate network models. This search space is modeled as a directed acyclic graph (DAG), which represents the cell structure in a convolutional neural network. In this graph, nodes represent information from feature maps, while edges represent various operations. To control the size of the search space, we adjust it by setting the number of nodes and the set of operations in the DAG. Notably, apart from the cells, other structural settings are identical across different architectures. Therefore, the diversity of cell structures directly corresponds to the variety of candidate architectures. Specifically, this cell structure contains four nodes and five associated operations. The connections between nodes are sequentially numbered, with smaller-numbered nodes connected to larger-numbered nodes, forming six connecting edges. These connecting edges can perform five different operations, including: convd 1x1 convolution, convd 3x3 convolution, avg_pool, skip-connect, and zeroing. In the directed acyclic graph (DAG), each intermediate node has five directed edges connecting it to its successor intermediate nodes, with each edge representing one operation from the set of operations. The feature information of an intermediate node is composed of the feature extraction results obtained by all its predecessor nodes through the corresponding directed edge operations. In summary, this search space can be represented as a directed acyclic graph that can generate a total of 15,625 cell candidate objects, meaning there are 15,625 possible candidate architectures in the search space.

[0013] 2. After setting up the specific search space, select a specific dataset for the image classification task to verify the performance of the specific architecture in the classification task.

[0014] 3. Due to the large number of candidate architectures, we adopted a sampling strategy to evaluate the flatness of different neural network architectures, and then calculated the correlation coefficient between flatness and generalization error. In this experiment, we sampled 100 different architectures to analyze the correlation between flatness and generalization error. The generalization error of a neural network is a key concept for measuring the model's performance on new, unseen data. It is an important part of understanding and improving the performance of machine learning models. The following are some important concepts related to neural network generalization error and their mathematical expressions:

[0015] 1. Empirical Error

[0016] Empirical error, or training error, is the error rate of a model on the training data. If we have a dataset {(x1,y1),(x2,y2),(x3,y3),...,(x...}... n ,y n Given a model f (e.g., a neural network), then the empirical error R0 f This can be expressed as:

[0017]

[0018] Where L(y) i ,f(x i ) is a loss function, such as mean squared error or cross-entropy loss.

[0019] 2. Expected Error

[0020] Generalization error refers to the average error rate of a model on new, unseen data. Mathematically, generalization error can be expressed as the expected loss of model f:

[0021] R′ f =E (x,y) ~[L(y,f(x)]p

[0022] Here, (x, y) is drawn from the true distribution P of the data. This is a theoretical quantity because the true distribution is usually unknown.

[0023] 3. Generalization Error

[0024] Gap = R′ f -R f

[0025] Specific data and analysis results regarding this correlation are provided in the appendix. Figure 3 Detailed information is provided below.

[0026] 4. After verifying the accuracy of our chosen evaluation metrics, we begin a comprehensive exploration of the entire neural network architecture search space. In each iteration, we select different candidate network architectures and calculate their flatness to rank them. If the flatness of a candidate architecture is greater than that of an architecture we previously retained, we consider the architecture with higher flatness to outperform the architecture with lower flatness. Therefore, we update the current candidate architecture. This process continues until we have traversed the entire search space. Ultimately, we find the architecture with the minimum upper bound on the generalization error; this architecture is the optimal solution for this round of search. We repeat this process multiple times, minimizing the error in each round, until we obtain the optimal neural network architecture after multiple iterations.

[0027] 5. The selected architecture model will be chosen as the deep neural network model for this image classification dataset. Next, we will utilize some common optimization techniques in deep learning to train this chosen network architecture to achieve effective image classification.

[0028] The present invention has the following beneficial effects:

[0029] This invention proposes a flatness-based neural network architecture search method to identify superior neural network architecture models. Compared to current techniques, this method, after verifying the accuracy of the selected metrics, utilizes flatness to evaluate the performance of different network architecture models, ultimately identifying a high-quality neural network architecture. This approach reduces training time, significantly decreasing computational load and time consumption, thus alleviating the heavy burden of neural network architecture search.

[0030] This not only greatly reduces the waste of computing resources and saves technical experts a lot of time, but also provides convenience for non-technical personnel or those with lower professional skills, enabling them to apply this method to image classification tasks and the broader field of artificial intelligence more easily and conveniently.

[0031] It is worth noting that the above general description and the following detailed description are for illustrative purposes only and do not imply that the invention is subject to any limitation. Attached Figure Description

[0032] Figure 1 The complete flowchart of this invention.

[0033] Figure 2 Search space diagram and network framework

[0034] Figure 3 Experimental graph showing the correlation between flatness index and generalization error

[0035] Figure 4 Flatness and final generalization error trend chart

[0036] Figure 5 Cell structure diagram of the selected optimal network architecture Detailed Implementation

[0037] This invention proposes a neural network architecture search method based on flatness. The experimental environment resources used are as follows: it runs on a Windows 10 system and is implemented using Python 3.8.3 and PyTorch 1.7.0 frameworks. To accelerate the optimization of deep neural networks, this invention uses an NVIDIA GTX 3080 (16G) GPU with CUDA version 11.4.

[0038] Step 1: Image Dataset Preparation

[0039] This paper uses the CIFAR-10 public dataset, with images sourced from the real world. It contains 60,000 32x32 color images across 10 categories: airplane, car, bird, cat, deer, dog, frog, horse, boat, and truck, with 6,000 images in each category. The dataset includes 50,000 images for training and 10,000 images for testing.

[0040] Step 2: Set up the search space for the neural network architecture

[0041] This invention targets image classification tasks in everyday life, excluding images from specialized fields such as remote sensing and medical images. Therefore, this invention uniformly adopts NAS-Bench-201 as our network architecture search space. Since neural networks in NAS-Bench-201 are composed of stacked cells, the difference between different neural networks lies in the different cells that make up the network. The specific neural network structure is shown in the attached figure. Figure 2 As shown, the initial input consists of 256 32x32 pixel images with 3 channels. This layer uses a 3x3 convolutional kernel with a stride of 1, padding of 1, and no bias term, expanding the 3 channels to 16 channels. A batch normalization layer normalizes the output of the convolutional layer using standard parameters (eps = 1e-05, momentum = 0.1). The output image is (32x32, 16 channels). Then, we move to the cell: which consists of several sub-layers, including ReLUConvBN (containing ReLU activation, a 3x3 Conv2d convolutional layer, and a BatchNorm2d batch normalization layer) and a Zero layer (for resizing). The image size and number of channels remain unchanged (32x32 pixels, 16 channels). After N cells, we reach the first ResNetBasicblock (inC = 16, outC = 32, stride = 2): the image size is halved to 16x16 pixels, and the number of channels increases to 32.

[0042] After passing through N cells, the second ResNet Basic block (e.g., inC=32, outC=64, stride=2) is reached: the image size is halved (e.g., from 16x16 to 8x8 pixels) and the number of channels is increased (e.g., from 32 to 64 channels). Finally, after passing through N cells, the last part of the network is reached, which includes a global average pooling layer to convert the feature map into a single vector, and a fully connected (Linear) layer to map this vector to the output of 10 classes (the number of classes in the CIFAR-10 dataset). The search space has a total of 56 =15625 network architectures. The search space is set to Searchspace, with the parameter args.dataset='cifar10'. NAS-Bench-201-v1_1-096897.pth is loaded, indicating that we are loading a search space based on the CIFAR10 dataset. The args.dataset parameter can be changed later to switch the search space.

[0043] Step 3. Sample the networks in the search space, calculate the flatness of the networks, and sort them.

[0044] In implementing this method, we sampled various architectures for image recognition from the search space, calculated the generalization boundaries of different candidate architectures, and verified the relationship between the generalization boundaries and performance of the neural network. We chose flatness as the criterion for evaluating neural networks because the number of candidate neural network architectures in the selected search space is very large. Therefore, we selected 436 neural network architectures for sampling and calculated their flatness according to the calculation method described below. This flatness also serves as the evaluation criterion provided by this invention for ranking these architectures. (See attached...) Figure 3 The results shown demonstrate that the mentioned evaluation criteria are highly correlated with the final accuracy of the network on our selected dataset (CIFAR-10), which verifies the effectiveness of the evaluation criteria provided in this invention.

[0045] Specifically, for different architectures, we calculate the flatness of each architecture separately. Here's a brief introduction on how to calculate the flatness of an architecture and why it measures the network's generalization ability. For flatness, we calculate the second derivative of the network architecture's loss function with respect to the network parameters and combine them into a matrix, which is the Hessian matrix.

[0046] Let the loss function L be:

[0047]

[0048] Let R be... m Let θ be an m-dimensional parameter space, and R be a parameter space. m A set of m weight parameters for a model, i.e., θ = (θ1, θ2, θ3, ..., θ4). m ), l(M(x i ),y i (x, θ) is the loss function loss, M is the neural network used for image recognition, (x, θ) i ,y i ) represents a single image and its corresponding label from a batch of input images, where N represents the batch size of the training data. Next, we consider the i-th parameter θ in θ.i Find the derivative, let the derivative be g:

[0049]

[0050] Since θ has m parameters,

[0051]

[0052] After that, we discussed... Taking the first derivative is equivalent to finding the loss function L with respect to θ. i The second derivative:

[0053]

[0054] Similarly, the second derivative of the entire loss function L with respect to θ can be obtained:

[0055]

[0056] We call the above matrix the Hessian matrix of a neural network. From the form of the Hessian matrix, we know that its diagonal elements are the second derivatives of the loss function L with respect to each parameter θ. We assume the trace of the Hessian matrix is ​​the sum of the diagonal elements, and let T... r (H) indicates

[0057]

[0058] The trace of the Hessian matrix (i.e., the sum of its diagonal elements) represents the flatness of the network because it captures the local curvature of the loss function in the parameter space. In neural network training, flat regions generally imply better generalization performance. Specifically: a small trace of the Hessian matrix indicates that the loss function changes relatively smoothly in the multidimensional parameter space, i.e., it is "flat." Such regions are often sought by gradient descent optimization algorithms because they tend to have better generalization ability, resulting in higher accuracy for image recognition. Conversely, a large trace of the Hessian matrix indicates that the loss function has a large curvature in that region, i.e., it is "steep," which may lead to overfitting and lower image recognition accuracy, as the network may only perform well on training data and poorly on unseen data. Therefore, when T... r The larger (H) is, the less flat the surface.

[0059] After calculating the flatness of each network architecture, we sort each network, denoted as rank, where rank... iLet represent the rank of the i-th network. Based on the API, the generalization error gap after 200 epochs of training for each network is obtained. A graph of the two is plotted to verify the effectiveness of the metric. The results are attached. Figure 4 As shown.

[0060] Step 4: Search for the optimal architecture in the neural network architecture search space.

[0061] After confirming the reliability of the selected evaluation metrics, we begin traversing the entire neural network architecture search space. In each round of the search, we select different candidate network architectures and evaluate and rank them based on their flatness. If we find that the flatness of a candidate architecture exceeds that of previously recorded architectures, this indicates that the architecture has better performance, so we set it as a new candidate architecture.

[0062] Let uid be the ID of each schema i in the search space S. i The range is 0 to 15625. We randomly select a batch of networks, setting the number of networks to 1000. For each network i, we first load the network weights and initialize them randomly. Then, we use the SGD optimizer to train the selected networks, setting the learning rate to 0.001, the weight decay to 0.0005, the batch size to 128, and the number of training epochs to 3. After training the network uid... i After training, we calculate the network UID. i The flatness value, i.e., flat i Next, for the next network UID j Perform the same operation, train for the same number of epochs, and then calculate the flat value. j If flat j <flat i Update the selected optimal network network = uid j

[0063] This process continues until we have traversed all networks and found the architecture with the highest flatness, which is the optimal solution network for that round of search. Through multiple rounds of such searches and by reducing errors, we will eventually identify the best neural network architecture after multiple rounds of search.

[0064] Step 5: Obtain and train the optimal neural network architecture model to complete the image classification task. The cells of the optimal architecture are shown in the attached image. Figure 5 As shown. Specifically, the network architecture is as follows:

[0065] First, there's a convolutional layer and a batch normalization (BN) layer, which we call the initial layer.

[0066] 1. Conv2d layer parameters:

[0067] (1) Number of input channels (in_channels): 3;

[0068] (2) Number of output channels (out_channels): 16;

[0069] (3) Kernel size: (3,3);

[0070] (4) stride: (1,1);

[0071] (5) Padding: (1,1);

[0072] (6) Bias: False.

[0073] 2. BatchNorm2d layer parameters:

[0074] (1) Number of features (num_features): 16; This matches the number of output channels of the Conv2d layer, indicating that batch normalization is performed on 16 feature maps.

[0075] (2) eps(epsilon): 1e-05, which is a small constant added to the denominator for numerical stability;

[0076] (3) Momentum: 0.1. The momentum used to calculate the runtime mean and variance;

[0077] (4) Affine: True. This indicates that the batch normalization layer has learnable affine parameters;

[0078] (5) track_running_stats: True. This indicates that the layer will track the running averages and variances during training for use at test time.

[0079] Next is the appendix Figure 5 The cell shown

[0080] 3. The internal structure and operations of a cell are as follows:

[0081] The first layer is a ReLUConvBN layer, which combines ReLU, convolution, and BN layers. The ReLUConvBN parameters are as follows:

[0082] Conv2d layer parameters:

[0083] (1) Number of input channels (in_channels): 3;

[0084] (2) Number of output channels (out_channels): 16;

[0085] (3) Kernel size: (3,3);

[0086] (4) stride: (1,1);

[0087] (5) Padding: (1,1);

[0088] (6) Bias: False.

[0089] BatchNorm2d layer parameters:

[0090] (1) Number of features (num_features): 16;

[0091] (2)eps(epsilon):1e-05;

[0092] (3) Momentum: 0.1;

[0093] (4) Affine: True;

[0094] (5)track_running_stats: True.

[0095] The second layer is the POOLING layer, with the following parameters:

[0096] (1) kernel_size: 3. This is the size of the pooling window.

[0097] (2) stride: 1. This is the stride of the pooling window.

[0098] (3) padding: 1. This is the amount of padding added to the edges of the input data, used to control the spatial dimension of the output feature map.

[0099] The third and fourth layers are both ReLUConvBN layers, marking the end of one cell. Our network then stacks 10 cells consecutively according to the above cell structure and parameter settings, adding ResNetBasicblock residual blocks in the 5th and 6th cells. The specific structure is as follows: two ReLUConvBN layers and one downsample layer.

[0100] The ReLUConvBN parameters are as follows:

[0101] Conv2d layer parameters:

[0102] (1) Number of input channels (in_channels): 16;

[0103] (2) Number of output channels (out_channels): 32;

[0104] (3) Kernel size: (3,3);

[0105] (4) stride: (2,2);

[0106] (5) Padding: (1,1);

[0107] (6) Bias: False.

[0108] BatchNorm2d layer parameters:

[0109] (1) Number of features (num_features): 32;

[0110] (2)eps(epsilon):1e-05;

[0111] (3) Momentum: 0.1;

[0112] (4) Affine: True;

[0113] (5) track_running_stats: True

[0114] The downsample parameters are as follows:

[0115] AvgPool2d layer parameters:

[0116] (4) kernel_size:

[0117] (5) stride:2.

[0118] (6)padding:0.

[0119] Conv2d layer parameters:

[0120] (1) Number of input channels (in_channels): 16;

[0121] (2) Number of output channels (out_channels): 32;

[0122] (3) Kernel size: (1,1);

[0123] (4) stride: (1,1);

[0124] (5) Padding: (1,1);

[0125] (6) Bias: False.

[0126] Then, ResNetBasicblock residual blocks are added to the 10th and 11th cells, with 32 input channels and 64 output channels. Subsequent cells also have 64 input and output channels, with other parameters remaining unchanged. The last layer is an AdaptiveAvgPool2d layer with parameters: output_size: 1; Linear (fully connected) layer: in_features: 64; out_features: 10; bias: True. This is the optimal architecture found. In this invention, we focus on the CIFAR-10 dataset and use the optimal neural network cell (as shown in the attached diagram) to... Figure 5 As shown, we construct a neural network architecture model for image classification tasks. At the end of the model, we perform global average pooling and linear classification, and use the softmax function to convert the output into class probabilities, thus obtaining the image classification result.

[0127] During the training phase, we employed various data augmentation techniques, including random pruning, random horizontal flipping, and random occlusion. To minimize the cross-entropy loss function for the classification task, we used batch stochastic gradient descent with 256 samples per batch, a learning rate of 0.001, weight decay of 0.0005, and 600 training epochs to ensure the neural network architecture converged. The optimal network selected had a training loss of 0.223, and the training loss remained relatively stable over the last 20 epochs, indicating that the network had converged. After training, the optimal neural network architecture model found in this invention achieved a classification accuracy of 93.1% on the CIFAR-10 dataset.

[0128] Notably, compared to current methods, which range from several days to hundreds of days for search time, our method reduces the neural network architecture search time to within minutes. In summary, we propose a flatness-based neural network architecture search method. This method uses the flatness of each candidate network as an evaluation metric, reflecting the classification ability of the neural network. Ultimately, we find the optimal neural network architecture without requiring time-consuming training; the average selection time per network is only 3-5 minutes, significantly reducing computational burden and time consumption.

Claims

1. A lightweight neural network architecture search method based on flatness, characterized in that, Includes the following steps: Includes the following steps: Step 1: Image Dataset Preparation The CIFAR-10 public dataset was used, and the image data came from the real world; Step 2: Set up the search space for the neural network architecture The NAS-Bench-201 network architecture search space is used. The initial input is a batch of 256 32x32 pixel images with 3 channels. This layer uses a 3x3 convolutional kernel with a stride of 1, padding of 1, and no bias term, expanding the 3 channels to 16 channels. A batch normalization layer is then applied, using standard parameters eps=1e-05, momentum=0.1 to normalize the output of the convolutional layer. The output image is (32x32, 16 channels). Next, we reach the cell: which consists of several sub-layers, including ReLUConvBN and Zero layers. ReLUConvBN includes ReLU activation, a 3x3 Conv2d convolutional layer, and a BatchNorm2d batch normalization layer. The image size and number of channels remain 32x32 pixels, with 16 channels remaining constant. After N cells, we reach the first ResNetBasicblock, with inC=16 and outC=32. stride=2; the image size is halved to 16x16 pixels, and the number of channels is increased to 32. After passing through N cells, the second ResNet Basic block is reached, with inC=32, outC=64, and stride=2. The image size is halved from 16x16 to 8x8 pixels, and the number of channels increases from 32 to 64. Finally, after passing through N cells, the network reaches its final part, which includes a global average pooling layer to convert the feature map into a single vector, and a fully connected layer to map this vector to the output of 10 classes, i.e., the number of classes in the CIFAR-10 dataset. The search space has a total of 5... 6 =15625 network architectures; set the space to Searchspace, with the parameter args.dataset ='cifar10', and load NAS-Bench-201-v1_1-096897.pth, indicating that the search space based on the CIFAR10 dataset is loaded; Step 3. Sample the networks in the search space, calculate the flatness of the networks, and sort them. For flatness, calculate the second derivative of the network architecture's loss function with respect to the network parameters and combine them into a matrix, which is the Hessian matrix. Let the loss function L be: ； Let R m Let θ be an m-dimensional parameter space, and R be a parameter space. m A set of m weight parameters for a model, i.e. , The loss function is M, which is a neural network used for image recognition. , () is a single image and its corresponding label from a batch of input images, where N represents the batch size of the training data; then... The i-th parameter Find the derivative, let the derivative be g: ； because There are m parameters, so ； Afterwards, on Taking the first derivative is equivalent to finding the loss function L for... The second derivative: ； The entire loss function L is obtained The second derivative: ； The above matrix is ​​called the Hessian matrix of a neural network. As can be seen from the form of the Hessian matrix, its diagonal elements represent the loss function L for each parameter. The second derivative of the Hessian matrix, let the trace of the Hessian matrix be the sum of the diagonals of the matrix, and use... express ； The trace of the Hessian matrix, which is the sum of its diagonal elements, represents the flatness of the network. After calculating the flatness of each network architecture, each network is sorted and set as follows: ,in This represents the ranking of the i-th network, and the generalization error gap after training each network for 200 epochs is obtained from the API. Step 4: Search for the optimal architecture in the neural network architecture search space; After confirming the reliability of the selected evaluation metrics, the entire neural network architecture search space is traversed. In each round of search, different candidate network architectures are selected and evaluated and ranked according to their flatness. If a candidate architecture is found to have a flatness that exceeds that of the previously recorded architectures, it indicates that the architecture has better performance and is therefore set as a new candidate architecture. Let the id of each schema i in the search space S be... The range is 0~15625. A batch of networks is randomly selected, with the number of networks set to 1000. For each network i, the network weights are first loaded and initialized randomly. Then, the selected network is trained using the SGD optimizer with a learning rate of 0.001, a weight decay of 0.0005, a batch size of 128, and a training epoch of 3. After the network is trained... After training, the computational network The flatness value, i.e. Then proceed to the next network Perform the same operation, train for the same number of epochs, and then perform the calculation. ,like Update the selected optimal network = ; This process continues until all networks have been traversed and the architecture with the highest flatness is found, which is the optimal solution network for that round of search. Through multiple rounds of such searches and by reducing errors, the best neural network architecture after multiple rounds of search will eventually be locked in. Step 5: Obtain and train the optimal neural network architecture model to complete the image classification task; the network architecture is as follows: First, there is a convolutional layer and a batch normalization (BN) layer, called the initial layer. 1) Conv2d layer parameters: (1) Number of input channels (in_channels): 3; (2) Number of output channels (out_channels): 16; (3) Kernel size: (3, 3); (4) Step size: (1, 1); (5) Padding: (1, 1); (6) Bias: False; 2) BatchNorm2d layer parameters: (1) Number of features (num_features): 16; This matches the number of output channels of the Conv2d layer, indicating that batch normalization is performed on 16 feature maps; (2) eps (epsilon): 1e-05, which is a small constant added to the denominator for numerical stability; (3) Momentum: 0.1; used to calculate the runtime average and variance. (4) Affine: True; This indicates that the batch normalization layer has learnable affine parameters; (5) track_running_stats: True; This indicates that the layer will track the running average and variance during training for use at test time; Next is the cell shown in Figure 5. 3) The internal structure and operations of a cell are as follows: The first layer is a ReLUConvBN layer, which combines ReLU, convolution, and BN layers. The ReLUConvBN parameters are as follows: Conv2d layer parameters: (1) Number of input channels (in_channels): 3; (2) Number of output channels (out_channels): 16; (3) Kernel size: (3, 3); (4) Step size: (1, 1); (5) Padding: (1, 1); (6) Bias: False; BatchNorm2d layer parameters: (1) Number of features (num_features): 16; (2) eps (epsilon): 1e-05; (3) Momentum: 0.1; (4) Affine: True; (5)track_running_stats: True; The second layer is the POOLING pooling layer, with the following parameters: (1) kernel_size: 3; This is the size of the pooling window; (2) stride: 1; This is the stride of the pooling window. (3) padding: 1; This is the amount of padding added to the edges of the input data, used to control the spatial dimension of the output feature map. The third and fourth layers are both ReLUConvBN layers, and this concludes one cell. After that, the network stacks 10 cells consecutively according to the structure and parameter settings of the cells mentioned above, and adds ResNetBasicblock residual blocks in the 5th and 6th cells. The specific structure is as follows: two ReLUConvBN layers and one downsample layer. The ReLUConvBN parameters are as follows: Conv2d layer parameters: (1) Number of input channels (in_channels): 16; (2) Number of output channels (out_channels): 32; (3) Kernel size: (3, 3); (4) Step size: (2, 2); (5) Padding: (1, 1); (6) Bias: False; BatchNorm2d layer parameters: (1) Number of features (num_features): 32; (2) eps (epsilon): 1e-05; (3) Momentum: 0.1; (4) Affine: True; (5) track_running_stats: True The downsample parameters are as follows: AvgPool2d layer parameters: (4) kernel_size: (5) stride: 2; (6)padding: 0; Conv2d layer parameters: (1) Number of input channels (in_channels): 16; (2) Number of output channels (out_channels): 32; (3) Kernel size: (1, 1); (4) Step size: (1, 1); (5) Padding: (1, 1); (6) Bias: False; Then, ResNetBasicblock residual blocks are added to the 10th and 11th cells, with 32 input channels and 64 output channels. Subsequent cells also have 64 input and output channels, with other parameters remaining unchanged. The last layer is an AdaptiveAvgPool2d adaptive average pooling layer with parameters: output_size: 1; Linear fully connected layer: in_features: 64; out_features: 10; bias: True. This is the optimal architecture found. At the end of the model, global average pooling is performed, followed by linear classification. The softmax function is used to convert the output into class probabilities, thus obtaining the image classification result. To minimize the cross-entropy loss function for the classification task, a batch stochastic gradient descent algorithm was used, with each batch containing 256 samples, a learning rate of 0.001, a weight decay of 0.0005, and 600 training epochs.

2. The lightweight neural network architecture search method based on flatness as described in claim 1, characterized in that: The flatness of the candidate network during initialization is used as an indicator to evaluate the network's generalization ability and performance; including: A network is selected from the network search space, and the trace of the Hessian matrix of the loss function with respect to the weight parameters is calculated during network initialization to measure the flatness of the network at initialization. This value is then used as a calculation value to measure the generalization ability of the network after convergence.

3. The lightweight neural network architecture search method based on flatness according to claim 1, characterized in that, The optimal neural network architecture model is selected and trained within the neural network architecture search space to achieve image classification tasks; including: This process begins by extracting one architecture at a time from the candidate architecture library and calculating its flatness, i.e., the trace of the Hessian matrix, and then ranking the performance of each architecture based on its flatness. Then, the optimal neural network architecture is determined through multiple fixed-cycle searches and error reduction. Subsequently, the determined optimal architecture is used to create a neural network architecture model suitable for image classification tasks. Finally, global average pooling and linear classification are performed on each feature map, and the output values ​​are converted into class probabilities using a softmax operation to obtain the image classification result.

Images