A neural network macro-architecture search method and system based on attention mechanism

Abstract

The present invention proposes a neural network macro-architecture search method and system based on an attention mechanism, comprising: obtaining a search space for a neural network architecture, constructing a macro-architecture super-network in the search space, and adding an attention module to the output of all candidate modules in each layer of the macro-architecture super-network to obtain a network to be searched; obtaining a labeled target dataset, dividing the target dataset into a training set and a validation set, training the network to be searched based on the training set to obtain an intermediate search network, inputting the validation set into the intermediate search network, and obtaining the attention weights of each candidate module in each layer through the attention module of each layer in the intermediate search network; and obtaining the search results of the target dataset by retaining only the candidate modules in each layer of the intermediate search network except for the ones with the largest attention weights. Thus, the present invention can increase the diversity of the neural network macro-networks searched and improve the performance of the network structure.

Description

Technical Field

[0001] The invention relates to neural network architecture search in the field of automated machine learning, and in particular to a neural network macro-architecture search method and system based on an attention mechanism. Background Art

[0002] Automated machine learning (Auto-ML) automates steps in machine learning, such as data preprocessing, feature selection, and algorithm selection, as well as neural network architecture design, hyperparameter optimization, and neural network model training in deep learning, achieving desired results without human intervention. Neural Architecture Search (NAS), a network design technique within automated machine learning, automatically searches for the optimal neural network architecture within a predefined search space. In many tasks, such as classification, detection, and segmentation, the performance of networks obtained through NAS has surpassed that of manually designed network structures.

[0003] Neural network architecture search can be divided into a task process consisting of a search space, a search strategy, and a performance evaluation strategy. The entire process can be described as starting from a search space containing multiple candidate operations (such as convolution, pooling, and skip connections), combining these operations according to the search strategy to obtain a neural network architecture. Then, under a specified evaluation strategy, the performance of this neural network architecture on the corresponding computer vision task is measured. Generally speaking, the search space can be divided into two types: those based on stacking MobileNet modules (MBConv) and those based on stacking cells (Cell). Furthermore, the search strategy and performance evaluation strategy are generally designed in conjunction with each other. Early neural network architecture search techniques used reinforcement learning or evolutionary algorithms for search strategies. Performance evaluation required retraining each resulting network architecture to assess its performance. However, this process typically requires sampling, training, and evaluating thousands of network architectures, making it extremely time-consuming. Even searching for the target architecture on a small dataset such as CIFAR-10 can take hundreds or thousands of days to complete. In recent years, developers have structured the search space into a supernetwork and, using weight sharing techniques, have enabled subnetwork performance evaluation by directly inheriting the supernetwork's weights, significantly reducing the overall search process time. Representative macroarchitecture searches are based on stacking MobileNet modules (MBConv) or cells (Cell). Both integrate architectures in the search space, alternately optimize network weights based on gradients, and finally select the optimal target network based on different performance evaluation strategies.

[0004] The existing DARTS method manually stacks the searched units repeatedly to form the target network, resulting in a lack of diversity in the final target structure. On the other hand, existing neural network macroarchitecture search based on the MobileNet module (MBConv) typically trains a supernetwork containing all subnetworks to convergence, then uses an evolutionary algorithm to select a large number of subnetworks. Inference is then performed on a validation set, and the optimal target subnetwork is selected based on performance on the validation set. However, verifying a large number of subnetworks using an evolutionary algorithm is both time-consuming and resource-intensive. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the present invention proposes a neural network macro-architecture search method based on the attention mechanism, which includes:

[0006] Step 1: Obtain a search space for a neural network architecture, construct a macro-architecture super-network in the search space, and add an attention module to the output of all candidate modules in each layer of the macro-architecture super-network to obtain a network to be searched.

[0007] Step 2: Obtain a labeled target dataset, divide the target dataset into a training set and a validation set, train the network to be searched based on the training set to obtain an intermediate search network, input the validation set into the intermediate search network, and obtain the attention weights of each candidate module in each layer through the attention module of each layer in the intermediate search network;

[0008] Step 3: Obtain the search results of the target dataset by retaining only the candidate modules with the largest attention weight in each layer of the intermediate search network.

[0009] The neural network macro-architecture search method based on the attention mechanism, wherein the target data set includes multiple samples, each sample has a corresponding label, the sample is a picture, and the label is the picture category; the search space is the DARTS search space or the MobileNet search space.

[0010] The neural network macro-architecture search method based on the attention mechanism further includes:

[0011] Step 4: Use the target data set to train the search results to obtain an image search model, input the image to be classified into the image search model, and obtain the image category of the image to be classified.

[0012] The neural network macro-architecture search method based on the attention mechanism, wherein each layer of the network to be searched has multiple candidate modules, and the output feature maps of the multiple candidate modules are spliced ​​to obtain the spliced ​​features;

[0013] The splicing feature F conThe features after pooling are obtained through global average pooling The specific process is shown in the following formula:

[0014]

[0015] Then the feature Pass the attention module with two layers of full connection and Sigmoid layer to obtain the attention weight of each candidate module;

[0016] The first fully connected layer in the attention module is used for dimensionality reduction, and the second fully connected layer restores the dimension to the original dimension. The process is expressed as follows:

[0017]

[0018] in and is the weight of the fully connected layer, r is the hyperparameter coefficient for dimensionality reduction, σ(·) is the Sigmoid activation function, and the output is the attention weight;

[0019] The attention weights of each candidate module in each layer of the network to be searched are:

[0020]

[0021] The activation value of each channel is multiplied by the original concatenated feature map Get the attention-weighted feature map And the feature map is calculated by the following formula Perform point-add operation:

[0022]

[0023] The present invention also proposes a neural network macro-architecture search system based on an attention mechanism, which includes:

[0024] An initialization module is used to obtain a search space for a neural network architecture, construct a macro-architecture super-network in the search space, and add an attention module to the output of all candidate modules in each layer of the macro-architecture super-network to obtain a network to be searched.

[0025] The training module is used to obtain a labeled target dataset, divide the target dataset into a training set and a validation set, train the network to be searched based on the training set to obtain an intermediate search network, input the validation set into the intermediate search network, and obtain the attention weights of each candidate module in each layer through the attention module of each layer in the intermediate search network;

[0026] The elimination module is used to retain only the candidate modules with the largest attention weight in each layer of the intermediate search network to obtain the search results of the target dataset.

[0027] The neural network macro-architecture search system based on the attention mechanism, wherein the target data set includes multiple samples, each sample has a corresponding label, the sample is a picture, and the label is the picture category; the search space is the DARTS search space or the MobileNet search space.

[0028] The neural network macro-architecture search system based on the attention mechanism further includes:

[0029] The image classification module is used to use the target data set to train the search results to obtain an image search model, input the image to be classified into the image search model, and obtain the image category of the image to be classified.

[0030] The neural network macro-architecture search system based on the attention mechanism, wherein each layer of the network to be searched has multiple candidate modules, and the output feature maps of the multiple candidate modules are spliced ​​to obtain the spliced ​​features;

[0031] The splicing feature F con The features after pooling are obtained through global average pooling The specific process is shown in the following formula:

[0032]

[0033] Then the feature Pass the attention module with two layers of full connection and Sigmoid layer to obtain the attention weight of each candidate module;

[0034] The first fully connected layer in the attention module is used for dimensionality reduction, and the second fully connected layer restores the dimension to the original dimension. The process is expressed as follows:

[0035]

[0036] in and is the weight of the fully connected layer, r is the hyperparameter coefficient for dimensionality reduction, σ(·) is the Sigmoid activation function, and the output is the attention weight;

[0037] The attention weights of each candidate module in each layer of the network to be searched are:

[0038]

[0039] The activation value of each channel is multiplied by the original concatenated feature map Get the attention-weighted feature map And the feature map is calculated by the following formula Perform point-add operation:

[0040]

[0041] The present invention also proposes a storage medium for storing a program for executing any one of the neural network macro-architecture search methods based on the attention mechanism.

[0042] The present invention also proposes a client for use in any of the aforementioned attention mechanism-based neural network macro-architecture search systems.

[0043] From the above scheme, it can be seen that the advantages of the present invention are:

[0044] The present invention proposes using an attention mechanism to evaluate the performance of each candidate sub-network. That is, after adding an attention module to each layer of candidate modules in the neural network macro structure, the candidate modules can be individual candidate cells in DARTS or individual candidate blocks in the MobileNet space. In this way, the attention weights of each candidate module and the weight of the super network can be optimized simultaneously. Finally, after the super network training converges, the optimal modules of each layer in the neural network macro structure can be immediately selected according to the attention weights, thereby forming the target network structure. In this way, the present invention can increase the diversity of the searched neural network macro network and improve the performance of the network structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of a neural network macro-architecture search method based on an attention mechanism according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of a macro-architecture super network according to an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of an attention module according to an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of a device for searching a neural network macro-architecture based on an attention mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention aims to address the issues in neural network macro-architecture search, such as the manual and repeated stacking of identical cells in the prior art, which results in a lack of target structure diversity or the time-consuming selection of sub-networks using evolutionary algorithms. The present invention provides a method and apparatus for neural network macro-architecture search based on an attention mechanism. Specifically, the present invention primarily addresses neural network macro-architecture search in the DARTS search space or the MobileNet search space, employing an attention mechanism to measure the importance of candidate modules at each layer in the neural network macro-architecture.

[0050] In a first aspect, the present invention provides a neural network macro-architecture search method based on an attention mechanism, which specifically includes the following steps:

[0051] Step 1. Concatenate the output feature maps of all candidate modules on the same layer of the neural network macro architecture.

[0052] The purpose of neural network macro-architecture search is to select the best module from all candidate modules in each layer. Therefore, when training the super network, the output information of the previous layer must pass through all candidate modules on this layer. Assuming that there are n candidate modules in this layer, the corresponding n feature maps will be obtained. The size of each feature map is c is the number of channels, h and w represent the length and width of the feature map respectively, and b represents the size of the convolution kernel. In order to keep the dimension of the output feature map unchanged, the final output of this layer needs to add these n feature maps to get the output feature, so the dimension of the output feature map is also Before adding multiple feature maps to get the final output of the layer, we need to splice these n feature maps according to the channel dimension to get the spliced ​​features. Then it is sent to the attention mechanism module we proposed to calculate the attention weights of each candidate module.

[0053] Step 2. The feature map passes through the attention module to extract the attention weight.

[0054] Splicing features of all candidate modules First, the pooled features are obtained through global average pooling. The specific process is shown in the following formula:

[0055]

[0056] In the above formula, h and w represent the length and width of the feature map respectively, and p and q are summation variables. This operation performs global average pooling on each feature map in the spatial dimension to obtain

[0057] Then the feature After two layers of full connection and Sigmoid layer, the importance of each candidate module is obtained. In order to reduce the complexity of the model and improve generalization, the present invention uses a bottleneck structure. This structure has two fully connected layers. The first fully connected layer is responsible for dimensionality reduction, and the second fully connected layer restores the dimension to the original dimension. This process is expressed by the following formula:

[0058]

[0059] In the above formula and is the weight of the fully connected layer, r is the hyperparameter coefficient for dimensionality reduction, Relu(.) is the linear rectification activation function, σ(·) is the Sigmoid activation function, and its output The attention weight can be considered as the importance of each channel. Therefore, the importance of the first candidate module A1 in this layer of the macro structure can be expressed as the sum of the activation values ​​of the first c channels. Similarly, the importance of the second candidate module A2 is expressed as the sum of the activation values ​​of the cth to 2cth channels. In summary, the importance of each candidate module in each layer of the neural network macro structure is expressed as:

[0060]

[0061] In the above formula, Atten is the attention weight of the concatenated feature map after passing through the attention module, c is the number of channels, then the importance A corresponding to each candidate module i (i=1,2,···,n) is the sum of the attention weights of the corresponding channels.

[0062] Step 3. Perform point addition on the attention-weighted feature map to obtain the output features.

[0063] In order to update the weights of the attention module, the activation value of each channel is multiplied by the original concatenated feature map Get the attention-weighted feature map In order to keep the same dimension as the original input feature map, the feature map after attention weighting The point addition operation is also required. However, the point addition operation here must be performed separately according to the channel of each candidate operation, that is, as follows:

[0064]

[0065] In the above formula, c is the number of channels, Represents the first c feature maps, that is, the feature map of the first candidate operation after attention weighting. Similarly, the feature maps of all candidate operations after attention weighting are added together to obtain F out .

[0066] This ensures that the feature maps of all candidate modules on each layer in the neural network macro architecture continue to be gradient updated through the super network after attention weighting, thereby updating the weight of the attention module, that is, the importance of each candidate module.

[0067] In a second aspect, the present invention provides a neural network macro-architecture search device based on an attention mechanism, which specifically includes the following modules:

[0068] A. Data Reading and Preprocessing Module: This module divides the original dataset into a training set and a validation set based on the corresponding hyperparameter settings. The training set is used to update the weights of the hypernetwork and attention module, while the validation set is used to select the target macro architecture based on the attention weights.

[0069] B. Macro-architecture super-network construction module: This module builds a macro-architecture super-network containing all candidate modules according to the settings of the search space, which is used for the neural network macro-architecture search process and the optimization of the weights of each part.

[0070] C. Neural network macro-architecture search and optimization module: This module mainly sends the training set into the macro-architecture super-network for forward propagation, and optimizes the weight parameters of different candidate modules in each layer of the macro-architecture super-network and the weight parameters of the attention module through backpropagation.

[0071] D. Importance evaluation module: This module is used to evaluate the importance of each sub-model in the macro-architecture super network, that is, each candidate module in each layer.

[0072] E. Single model acquisition module: This module is used to obtain the optimal single model based on the importance of each candidate module.

[0073] F. Single model training and validation model: This module uses the entire training set to retrain the optimal single model to optimize the weight parameters in the single model, and performs forward propagation on the test set to compare the output with the label to verify the performance of the searched single model.

[0074] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0075] Example 1

[0076] Figure 1 The present invention provides a neural network macro-architecture search method based on the attention mechanism, which comprises the following steps:

[0077] S11: Define the target dataset and search space.

[0078] In this step, the present invention is aimed at neural network macro-architecture search, that is, it is used to search and determine the optimal candidate modules at each layer in the neural network macro-architecture. For example, in the DARTS search space, the target structure in this space is composed of stacked cells, and the purpose of the macro-architecture search is to select the optimal unit from several candidate cells in each layer to form the final target macro-structure; or in the MobileNet search space, each layer of the macro-structure in this space is a chain structure composed of several stacked candidate blocks, and the purpose of the macro-architecture search is to determine the optimal selected blocks in each layer to form the target macro-structure.

[0079] S12: In the search space, a macro-architecture super-network containing an attention module is constructed for search.

[0080] In this step, if Figure 2 As shown, the super network under this search space has L layers, each layer has several candidate units or candidate blocks. Candidate units are unit structures that have been searched or predefined in advance, and candidate blocks are convolution modules with different convolution kernel sizes and expansion rates. At the same time, the present invention adds an attention module after all candidate modules in each layer of the neural network macro architecture output feature maps, such as Figure 3 As shown in Figure 2, the attention weight of the attention module can be expressed as the importance of each candidate module.

[0081] S13: After randomly cropping, flipping, and normalizing the training set images, they are sent to the macro-architecture super network. After forward propagation, the cross entropy loss is calculated based on the label, and backpropagation is performed to calculate the gradient. The stochastic gradient descent (SGD) optimizer is used to optimize the macro-architecture super network and attention module weights according to the gradient direction.

[0082] In this step, the main work is to train the macro-architecture super-network containing the attention module on the training set. During the training, the weights of the super-network are updated, and the weights of the attention module are also updated accordingly to learn the importance of each candidate module in each layer of the macro-architecture super-network in the super-network.

[0083] S14: Whether the algorithm reaches the specified number of iterations.

[0084] In this step, it is used to determine whether the super network containing the attention module has been trained and converged. If the training has not converged, step S13 is continued to train. If the specified number of iterations is reached, that is, after the training converges, the next step is performed.

[0085] S15: After randomly cropping, flipping, and normalizing the validation set images, they are sent to the trained macro-architecture super network. After forward propagation, the attention weights of the candidate modules in each layer are obtained.

[0086] In this step, the verification set images are fed into the macro-architecture supernet for forward propagation, and the attention weights of all candidate modules in each layer of the macro-architecture supernet are extracted. The attention weights can be used as an indicator to measure the importance of each module in each layer.

[0087] S16: Select the optimal target structure based on the attention weight.

[0088] In this step, according to the attention weight obtained in step S15, there are n candidate modules in each layer. The module with the largest weight is selected from these n attention weights and retained as the optimal module of the layer. The other candidate modules and all attention modules are removed, and the entire macro structure is determined in sequence.

[0089] S17: Retrain the target structure on the entire dataset and verify the performance.

[0090] In this step, the target structure obtained in the above step S16 is retrained on the entire training set until convergence, and its performance index is tested in the test set.

[0091] Example 2

[0092] The embodiment of the present invention also provides a device for searching a neural network macro-architecture based on an attention mechanism, such as Figure 4 As shown, the device includes: a data reading and preprocessing module 21, a super network construction module 22, a neural network architecture search and optimization module 23, a single model acquisition module 24, and a single model retraining and verification module 25.

[0093] Among them, the data reading and preprocessing module 21 divides the training data into training sets and verification sets according to the corresponding hyperparameter settings for neural network architecture search, reads in and preprocesses the training sets and test sets for training and verification of the searched single model; the super network construction module 22 constructs a macro network chain structure with an added attention module; the neural network architecture search and optimization module 23 uses the training set to train the network weights of the macro architecture super network and the weights of the attention module; the single model acquisition module 24 uses the verification set image for forward propagation to extract the attention weights of each candidate module in each layer of the macro architecture. The attention weights can be used as an indicator to measure the importance of each module, and then the optimal modules of each layer are selected according to the attention weights to form the final target macro architecture; the single model retraining and verification module 25 retrains the target structure on the entire training set until convergence, and then tests its performance indicators in the test set.

[0094] In the device for searching the macro-architecture of the Shenjiang network based on the attention mechanism provided by the example of the present invention, the working process of each module has the same technical characteristics as the method for searching the macro-architecture of the neural network based on the attention mechanism. Therefore, the aforementioned functions can also be realized and will not be repeated here.

[0095] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0096] The present invention also proposes a neural network macro-architecture search system based on an attention mechanism, which includes:

[0097] An initialization module is used to obtain a search space for a neural network architecture, construct a macro-architecture super-network in the search space, and add an attention module to the output of all candidate modules in each layer of the macro-architecture super-network to obtain a network to be searched.

[0098] The training module is used to obtain a labeled target dataset, divide the target dataset into a training set and a validation set, train the network to be searched based on the training set to obtain an intermediate search network, input the validation set into the intermediate search network, and obtain the attention weights of each candidate module in each layer through the attention module of each layer in the intermediate search network;

[0099] The elimination module is used to retain only the candidate modules with the largest attention weight in each layer of the intermediate search network to obtain the search results of the target dataset.

[0100] The neural network macro-architecture search system based on the attention mechanism, wherein the target data set includes multiple samples, each sample has a corresponding label, the sample is a picture, and the label is the picture category; the search space is the DARTS search space or the MobileNet search space.

[0101] The neural network macro-architecture search system based on the attention mechanism further includes:

[0102] The image classification module is used to use the target data set to train the search results to obtain an image search model, input the image to be classified into the image search model, and obtain the image category of the image to be classified.

[0103] The neural network macro-architecture search system based on the attention mechanism, wherein each layer of the network to be searched has multiple candidate modules, and the output feature maps of the multiple candidate modules are spliced ​​to obtain the spliced ​​features;

[0104] The splicing feature Fcon The features after pooling are obtained through global average pooling The specific process is shown in the following formula:

[0105]

[0106] Then the feature Pass the attention module with two layers of full connection and Sigmoid layer to obtain the attention weight of each candidate module;

[0107] The first fully connected layer in the attention module is used for dimensionality reduction, and the second fully connected layer restores the dimension to the original dimension. The process is expressed as follows:

[0108]

[0109] in and is the weight of the fully connected layer, r is the hyperparameter coefficient for dimensionality reduction, σ(·) is the Sigmoid activation function, and the output is the attention weight;

[0110] The attention weights of each candidate module in each layer of the network to be searched are:

[0111]

[0112] The activation value of each channel is multiplied by the original concatenated feature map Get the attention-weighted feature map And the feature map is calculated by the following formula Perform point-add operation:

[0113]

[0114] The present invention also proposes a storage medium for storing a program for executing any one of the neural network macro-architecture search methods based on the attention mechanism.

[0115] The present invention also proposes a client for use in any of the aforementioned attention mechanism-based neural network macro-architecture search systems.

Claims

1. A neural network macro-architecture search method based on attention mechanism, characterized in that: include: Step 1: Obtain a search space for a neural network architecture, construct a macro-architecture super-network in the search space, and add an attention module to the output of all candidate modules in each layer of the macro-architecture super-network to obtain a network to be searched. Step 2: Obtain a labeled target dataset, divide the target dataset into a training set and a validation set, train the network to be searched based on the training set to obtain an intermediate search network, input the validation set into the intermediate search network, and obtain the attention weights of each candidate module in each layer through the attention module of each layer in the intermediate search network; Step 3: Remove the candidate modules except the candidate module with the largest attention weight in each layer of the intermediate search network, and remove all attention modules in the intermediate search network to obtain the search results of the target dataset; The target dataset includes multiple samples, each sample has a corresponding label, the sample is an image, and the label is the image category; the search space is the DARTS search space or the MobileNet search space.

2. The neural network macro-architecture search method based on the attention mechanism according to claim 1, characterized in that: Also includes: Step 4: Use the target data set to train the search results to obtain an image search model, input the image to be classified into the image search model, and obtain the image category of the image to be classified.

3. The neural network macro-architecture search method based on the attention mechanism according to claim 1, characterized in that: Each layer of the network to be searched has multiple candidate modules, and the output feature maps of multiple candidate modules are spliced ​​to obtain spliced ​​features; The splicing feature F con The features after pooling are obtained through global average pooling The specific process is shown in the following formula: Then the feature Pass the attention module with two layers of full connection and Sigmoid layer to obtain the attention weight of each candidate module; The first fully connected layer in the attention module is used for dimensionality reduction, and the second fully connected layer restores the dimension to the original dimension. The process is expressed as follows: in and is the weight of the fully connected layer, r is the hyperparameter coefficient for dimensionality reduction, σ(·) is the Sigmoid activation function, and the output is the attention weight; The attention weights of each candidate module in each layer of the network to be searched are: The activation value of each channel is multiplied by the original concatenated feature map Get the attention-weighted feature map And the feature map is calculated by the following formula Perform point-add operation:

4. A neural network macro-architecture search system based on attention mechanism, characterized in that: include: An initialization module is used to obtain a search space for a neural network architecture, construct a macro-architecture super-network in the search space, and add an attention module to the output of all candidate modules in each layer of the macro-architecture super-network to obtain a network to be searched. The training module is used to obtain a labeled target dataset, divide the target dataset into a training set and a validation set, train the network to be searched based on the training set to obtain an intermediate search network, input the validation set into the intermediate search network, and obtain the attention weights of each candidate module in each layer through the attention module of each layer in the intermediate search network; A removal module is used to remove candidate modules except the candidate module with the largest attention weight in each layer of the intermediate search network, and remove all attention modules in the intermediate search network to obtain the search results of the target dataset; The target dataset includes multiple samples, each sample has a corresponding label, the sample is an image, and the label is the image category; The search space is the DARTS search space or the MobileNet search space.

5. The attention-based neural network macro-architecture search system according to claim 4, wherein: Also includes: The image classification module is used to use the target data set to train the search results to obtain an image search model, input the image to be classified into the image search model, and obtain the image category of the image to be classified.

6. The attention-based neural network macro-architecture search system according to claim 4, wherein: Each layer of the network to be searched has multiple candidate modules, and the output feature maps of multiple candidate modules are spliced ​​to obtain spliced ​​features; The splicing feature F con The features after pooling are obtained through global average pooling The specific process is shown in the following formula: Then the feature Pass the attention module with two layers of full connection and Sigmoid layer to obtain the attention weight of each candidate module; The first fully connected layer in the attention module is used for dimensionality reduction, and the second fully connected layer restores the dimension to the original dimension. The process is expressed as follows: in and is the weight of the fully connected layer, r is the hyperparameter coefficient for dimensionality reduction, σ(·) is the Sigmoid activation function, and the output is the attention weight; The attention weights of each candidate module in each layer of the network to be searched are: The activation value of each channel is multiplied by the original concatenated feature map Get the attention-weighted feature map And the feature map is calculated by the following formula Perform point-add operation:

7. A storage medium for storing a program for executing any one of the attention mechanism-based neural network macro-architecture search methods as described in claims 1 to 3.

8. A client for use in any one of the attention mechanism-based neural network macro-architecture search systems of claims 4 to 6.

Images