A multi-objective neural architecture search method integrating convolution and self-attention

Abstract

This invention belongs to the field of multi-objective evolutionary neural architecture search, proposing a multi-objective evolutionary neural architecture search method that integrates convolution and self-attention. By constructing an improved multi-objective evolutionary algorithm, the efficiency of neural architecture search is increased, allowing for faster discovery and preservation of individual neural architectures during the search process, while balancing convergence and diversity. By constructing a search space that integrates convolution and attention operations, the processing capability of global and local information in image processing tasks is improved, enhancing the plasticity of the search space and the generalization performance of individuals across different task sets. Decision variables such as convolution kernel size and number of layers in the search space are encoded as individuals in the population of the improved multi-objective evolutionary algorithm. The optimal individual is searched through the operation of the improved multi-objective evolutionary algorithm and decoded into the optimal network architecture. This invention effectively improves search efficiency and obtains the best-performing neural architecture.

Description

Technical Field

[0001] This invention relates to the field of multi-objective evolutionary neural architecture search, and more particularly to a multi-objective evolutionary neural architecture search method that integrates convolution and self-attention. Background Technology

[0002] With the increasing number of tasks involving large-scale datasets, the structure of convolutional neural networks has become increasingly complex, with more and more parameters. This means that more resources and time are required. Therefore, Neural Architecture Search (NAS) is a method proposed in recent years for specific task design and datasets, enabling the system to automatically find and generate the optimal architecture within a given search space. Its core idea is that the network automatically finds the optimal network architecture through techniques such as reinforcement learning, thus replacing the manual design of the network by the designer. Neural architecture search methods are generally divided into those based on random search, evolutionary algorithms, and reinforcement learning. To improve the efficiency of NAS algorithms, many methods have been proposed, from top to bottom, with most focusing on lower levels, including weight sharing. This allows genetic algorithms to be well applied to the evolution and selection of architectures.

[0003] Self-attention and convolution are two major techniques in computer vision. Self-attention emerged later than convolution, but it achieved results that matched or even surpassed traditional convolutional neural networks upon its introduction. The main difference lies in that convolution focuses primarily on local features, while self-attention makes excellent use of global information. Recently, researchers have combined convolution and attention to create novel models, such as CoAtNet proposed by Z. Dai et al. in "CoAtNet: Marrying Convolution and Attention for All Data Sizes" and the CvT series models proposed by H. Wu et al. in "CvT: Introducing Convolutions to Vision Transformers." These models combine the advantages of both and achieve results exceeding previous advancements, providing a good starting point for the fusion of convolution and self-attention. Previous search spaces were built by extending existing convolutional neural network models or by combining convolutional operations.

[0004] The multi-objective evolutionary neural architecture search that integrates convolution and self-attention mainly considers the following issues: (1) Low search efficiency: Neural architecture search usually requires sampling in a vast search space and evaluation of the collected subnets, which requires a lot of practical and computational resources. Some search tasks take tens of days or even months on multiple GPUs, which is unbearable for most researchers and ordinary laboratories. (2) Low plasticity of the search space: In the process of neural architecture search, if the search space does not have enough depth and complete connections, it is difficult to find excellent individuals even if the sampling and evaluation process is good. Therefore, it is necessary to design a search space with strong plasticity. (3) Poor generalization of search results: The ideal individuals found should have good generalization, that is, they can be adapted to different datasets by simply fine-tuning the individuals found. However, the individuals found by most current search methods do not perform well after being fully trained on other datasets.

[0005] The structure and parameters of the neural network are optimized using an improved evolutionary algorithm. This improved algorithm maximizes the reduction of the neural network's size while maintaining its performance. It also better balances diversity and convergence when searching for individuals, thereby improving search efficiency. A network structure search space is proposed by fusing attention and convolutional modules, and the best-performing structure is selected from this space. The searched model architecture demonstrates excellent performance in both accuracy and model complexity on a specific dataset. Summary of the Invention

[0006] The goal of this invention is to improve the multi-objective evolutionary algorithm, enabling it to more rationally select the optimal individual during the search process and leveraging the advantages of multi-objective evolutionary algorithms to improve the efficiency of architecture search. A novel search space integrating convolutional operations and self-attention mechanisms is established, and fusion parameters are introduced to better balance the complexity and accuracy of the searched models. The fusion module of convolutional operations and self-attention mechanisms can more comprehensively consider global and local information during model execution, thereby improving the plasticity of the search space and increasing the generalization performance of the searched individuals.

[0007] The technical solution of this invention is as follows: A multi-objective evolutionary neural architecture search method integrating convolution and self-attention, which improves the search efficiency of neural architecture by constructing an improved multi-objective evolutionary algorithm, and finds and saves individual neural architectures faster during the search process, taking into account both convergence and diversity; by constructing a search space integrating convolution and self-attention operations, the method improves the processing capability of global and local information in image processing tasks, and improves the plasticity of the search space and the generalization performance of individuals on different task sets; the decision variables such as the convolution kernel size and the number of layers in the search space are encoded as individuals in the population of the improved multi-objective evolutionary algorithm, and the best-performing individuals are searched by running the improved multi-objective evolutionary algorithm, and then decoded into the optimal network architecture.

[0008] Regarding the evolutionary algorithm, the traditional multi-objective evolutionary algorithm NSGA-II was improved, resulting in significant enhancements in both convergence and optimization performance. The improved algorithm flow is as follows: Figure 2 As shown. Regarding search space construction, a search space integrating convolution and self-attention mechanisms is proposed. To better consider the processing of local and global feature information in image classification tasks, convolution operations and self-attention mechanisms are integrated to form a novel model backbone, such as... Figure 3 As shown, the improved multi-objective evolutionary algorithm based on NSGA-II performs architecture search on the number of layers and their arrangement order in the convolutional and attention hybrid regions, finds the best-performing structure, and trains it to the optimal performance on a specific dataset.

[0009] S1. Construct a neural architecture search framework for a multi-objective evolutionary algorithm based on NSGA-II;

[0010] S1.1. Uniform population initialization based on Euclidean distance;

[0011] S1.1.1: Randomly initialize the initial population to generate T times the preset number of individuals;

[0012] S1.1.2: Set up replicas of the initial population, and normalize the decision variables of individuals in the replicas according to the range of values ​​of the decision variables; the decision variable of any individual in the population is... Its value range is [ The normalized decision variables are as follows:

[0013] = (1)

[0014] S1.1.3: Set the parameter ρ to represent the Euclidean distance of each individual, and n to be the number of decision variables;

[0015] ρ= (2)

[0016] S1.1.4: Arrange individuals according to their ρ value, and select the required number of individuals from the initial population evenly based on their ρ value;

[0017] S1.2 Adaptive selection strategy based on convergence distance and crowding;

[0018] A distance-based convergence index α is proposed: After the population individuals are ordered by dominance in step S1.1.4, the convergence direction of all individuals in the first layer of the population is determined as the convergence direction of the entire population, and α is the average distance from each individual in the last layer of the population to all individuals in the first layer of the population; after fast non-dominated sorting, the target values ​​of the j-th individual in the first layer are expressed as: = ( , ..., The target values ​​for the k-th individual in the last layer are represented as follows: = ( , ..., );

[0019] Let the first layer have m individuals, y represent the number of objective functions, z represent the objective function, and the convergence distance of the k-th individual in the last layer be expressed as:

[0020] α(k) = (3)

[0021] β represents the crowding level of an individual. After adaptively fusing crowding level and convergence distance, the final individual selection criterion is as follows:

[0022] = (1-eta)× + eta× (4)

[0023] eta = (5)

[0024] Gen represents the current generation, and maxGen represents the maximum generation. In each iteration, individuals are selected from the last layer of individuals. The individual with the largest value;

[0025] S1.3 Adaptive crossover and mutation operator;

[0026] Use the maximum fitness value of the population and the average fitness value of the population The difference is used as a convergence index. The smaller the difference, the more the population converges towards the center, and the better the convergence. In order to further preserve the superior individuals in the population and change the inferior individuals, the crossover probability and mutation probability used for superior individuals are lower than those used for inferior individuals.

[0027] The crossover and mutation probabilities are ultimately expressed as:

[0028] = (6)

[0029] = (7)

[0030] The fitness value of an individual with higher fitness during the crossover operation. For individual target values;

[0031] S1.4 introduces the local gradient operator;

[0032] To better balance local and global search, a gradient operator is introduced into the genetic algorithm to perform local search on individuals, and the objective function value of the searched individual is compared with the original value. If the objective function value of an individual is better than the original value, it replaces the original value and the update is completed, based on the population convergence index. and The difference is used to determine whether to perform gradient search. - < Gradient calculation is performed at that time.

[0033] A random perturbation is generated for each decision variable of the individual.

[0034] σX = { }

[0035] The gradient vectors of each decision variable are represented as follows:

[0036] = (8)

[0037] The updates for each decision variable are represented as follows:

[0038]

[0039] i∈[1,2,..,n], λ is the step size; set the termination value £, when - When < £, gradient update terminates;

[0040] S2. The optimal architecture is output by a multi-objective evolutionary algorithm based on NSGA-II:

[0041] Step 2.1: Individually encode the decision variables;

[0042] Step 2.2: Initialize the first generation population uniformly within the range of decision variables;

[0043] Step 2.3: Perform non-dominated sorting, use the tournament selection algorithm to select the parent generation, and use adaptive crossover and mutation operators to generate the offspring population;

[0044] Step 2.4: Merge the offspring and parents to generate a new population, calculate the objective function value of all individuals in the population, and determine whether the conditions for local search of the population are met. If the conditions are met, proceed to Step 2.5; otherwise, proceed to Step 2.6.

[0045] Step 2.5: Use the gradient operator to perform a local search on each individual in the population, and update the population individuals according to the target value after the search until the search termination condition is met. After the population update is completed, proceed to Step 2.6.

[0046] Step 2.6: Perform a fast non-dominated sort on the population, and calculate the crowding degree and convergence distance for all individuals in the last layer of the selection phase. Based on the β value, perform the final individual selection to generate a new generation of parent population.

[0047] Step 2.7: Determine if the current iteration count is less than or equal to the maximum iteration count. If so, increment the current iteration count by 1 and jump to Step 2.3; otherwise, output the optimal solution.

[0048] Step 2.8: Decode the optimal architecture into a neural network architecture and train it on the task dataset until convergence;

[0049] S3. Construct a search space that integrates convolution and self-attention mechanisms;

[0050] The convolution operation and self-attention mechanism are combined, and the backbone of the search space region is in the form of Mobilenetv3, which includes a self-attention and convolution hybrid module. After the input feature map is convolved by 1×1, convolution operation and self-attention operation are performed respectively, and finally fused by two fusion parameters of the self-attention and convolution hybrid module.

[0051] The main structure of the search space consists of four concatenated blocks. A 3×3 convolution is used to process the input image to obtain feature maps. Each block contains 1-4 layers. As the number of block layers increases, the dimension of the feature map increases while the length and width decrease. A downsampling operation is set at the first layer of each block, with the sampling set to 2. The input and output resolutions of the remaining layers remain unchanged.

[0052] Based on the MobileNetv3 network, the original 3×3 depthwise convolutional kernel is replaced by a self-attention and convolution hybrid module; the hybrid module is as follows:

[0053] The input and output feature maps are represented as A∈ B∈ The feature tensors at pixel (i,j) in the input and output feature maps are respectively ∈ , ∈ ,definition ∈ Let p,s∈[0,1,2,...,g-1] be the weights of the convolution kernel at (p,s). , , and Let represent the query, key, value, and transformation matrix at pixel (i,j) in the corresponding input feature map, and define the Shift operation. ;

[0054] The convolution operation consists of two steps;

[0055] The first step of the convolution operation: = (10)

[0056] The second step of the convolution operation: = shift( , (11)

[0057] = (12)

[0058] Attention mechanisms consist of two steps;

[0059] The first step of the attention mechanism: = , = , = (13)

[0060] The second step of the attention mechanism: = (14)

[0061] The first part of both the convolution operation and the self-attention mechanism maps features to a deeper space and accounts for the majority of floating-point operations and parameters; the second part of the convolution operation and the self-attention mechanism respectively performs feature aggregation for the two operations; after sharing the first part of the convolution operation and the self-attention mechanism, they each perform their respective second part operations.

[0062] The steps for operating the self-attention and convolution hybrid module are as follows:

[0063] S3.1 Feature Projection

[0064] The input feature map is projected using three 1×1 convolutions, and the three projected feature maps are then divided into R groups according to the number of channels C.

[0065] S3.2 Feature Aggregation

[0066] S3.2.1 Self-attention mechanism branch

[0067] S3.2.1.1 Self-Attention Linearization

[0068] The original attention mechanism formula is:

[0069] Attention(Q,K,V) = softmax( V (15)

[0070] The space complexity is O( The time complexity is O( ). (), where N is the length of the input sequence and d is the dimension of the input. N is greater than d. The original attention mechanism formula is decomposed into a function.

[0071] (16)

[0072] make It is an identity function. For softmax, the linearized self-attention mechanism formula is:

[0073] Attention(Q,K,V) = (softmax V) (17)

[0074] The linearized version has a space complexity of O(Nd) and a time complexity of O(N). );

[0075] S3.2.1.2 Relative Position Encoding

[0076] Using a linearized self-attention mechanism, when the input... and When local features are identical, differences cannot be obtained. Positional encoding is added to introduce positional information. Let the positional encoding matrix be P, which represents the relationship between Q and V, and then depthwise convolution is introduced.

[0077] PositionAtten(Q,V) = Q°Depthwiseconv(P,V) (18)

[0078] ° represents the product of Hadamard matrices;

[0079] The final attention branch output feature map is

[0080] Atten = PositionAtten. , (19)

[0081] S3.2.2 Convolution Operation Branch

[0082] S3.2.2.1 Fully Connected Layer

[0083] The three feature maps obtained from projection are transformed into N sets of individual features using a fully connected layer. Each feature map;

[0084] S3.2.2.2 Shift Operation

[0085] For each group Shift each feature map, and let a set of matrices be... n∈{0,1,..., In each matrix, starting from the first value in the first row and counting sequentially along the row, the nth element is set to 1, and all other elements are set to 0. The feature map group is represented as follows: The final output of the convolution branch is:

[0086] Conv = (20)

[0087] S3.3 Final Output

[0088] Set two fusion parameters and , and The values ​​of are all in the range of [0, 0.25, 0.5, 0.75, 1]. The two fusion parameters represent the fusion ratio of the self-attention mechanism and the convolution operation during the fusion process, respectively. The final output is expressed as:

[0089] (twenty one).

[0090] The beneficial effects of this invention are:

[0091] 1. To better consider the processing of local and global feature information in image classification tasks, this invention fuses convolution and self-attention to form a new model search backbone. A hybrid module is set up to explore the relationship between the convolution module and the self-attention module.

[0092] 2. The neural architecture search method of the improved multi-objective evolution algorithm for neural networks in this invention determines the number of regions and mixing parameters of the convolution and self-attention hybrid module, and searches for the neural architecture with the best performance, effectively improving the search efficiency.

[0093] 3. This invention employs a multi-objective evolutionary algorithm based on NSGA-II to improve the efficiency of neural architecture search, significantly increasing the convergence speed and the selection rate of superior individuals during the search process. It constructs a search space that integrates convolution and self-attention mechanisms, effectively balancing the processing of local and global information. This enhances the generalization ability of search individuals, while achieving a general improvement in both performance and complexity. Attached Figure Description

[0094] Figure 1 This is a flowchart illustrating the multi-objective evolutionary neural architecture search method that integrates convolution and self-attention according to the present invention.

[0095] Figure 2 Here is a flowchart of the multi-objective evolution algorithm based on NSGA-II;

[0096] Figure 3 This is a schematic diagram of a hybrid module;

[0097] Figure 4 This is a schematic diagram of the search space backbone. Detailed Implementation

[0098] The core idea of ​​this invention is to improve upon the NSGA-II algorithm, resulting in significant enhancements in both convergence and optimization performance. An improved multi-objective evolutionary algorithm is introduced for neural architecture search of the convolutional module and the self-attention-based module. Firstly, to better consider the processing of local and global feature information in image classification tasks, convolution and self-attention are fused together to form a novel model backbone. A neural architecture search method based on a genetic algorithm is then used to find the best-performing structure.

[0099] See Figure 1 The present invention includes the following steps:

[0100] S1. Construct a structure search framework based on the improved multi-objective evolutionary algorithm of NSGA-II, including:

[0101] S1.1 Population Uniform Initialization Based on Euclidean Distance

[0102] The distribution of individuals during the initial run of the algorithm is crucial to its subsequent progress. Randomly initializing the population is detrimental to the development of population diversity and convergence. Therefore, a uniform population initialization method based on Euclidean distance is proposed:

[0103] Step 1.1.1: Randomly initialize the initial population to generate 20 times the preset number of individuals.

[0104] Step 1.1.2: Set up replicas of the initial population. Based on the range of values ​​for the decision variables, normalize the decision variables of the individuals in the replicas. The decision variable for any individual in the population is... Its value range is [ The normalized decision variables are as follows:

[0105] = (1)

[0106] Step 1.1.3: Set the parameter ρ to represent the Euclidean distance of each individual, and m to represent the total number of decision vectors.

[0107] ρ= (2)

[0108] Step 1.1.4: Arrange the individuals according to their ρ values, and select the required number of individuals from the initial population evenly based on their ρ values.

[0109] S1.2 Adaptive selection strategy based on convergence distance and crowding

[0110] During algorithm execution, both convergence and diversity of the population should be considered. However, in the early stages, individuals are scattered throughout the target space, so the focus should be on convergence. In the later stages, most individuals are clustered near the optimal solution, so diversity should be emphasized. Therefore, in the final layer of individual selection, in addition to using crowding to ensure diversity, a convergence metric should also be introduced.

[0111] Here, a distance-based convergence index α is proposed: After the population individuals are ordered by dominance in step S1.1.4, the convergence direction of all individuals in the first layer of the population is determined as the convergence direction of the entire population, and α is the average distance from each individual in the last layer of the population to all individuals in the first layer of the population. After fast non-dominated sorting, the objective values ​​of the j-th individual are expressed as: = ( , ..., The target values ​​for the k-th individual in the last layer are represented as follows: = ( , ..., ).

[0112] Let the first layer have m individuals, y represent the number of objective functions, z represent the objective function, and the convergence distance of the k-th individual in the last layer be expressed as:

[0113] α(k) = (3)

[0114] β represents the crowding level of an individual. After adaptively fusing crowding level and convergence distance, the final individual selection criterion is as follows:

[0115] = (1-eta)× + eta× (4)

[0116] eta = (5)

[0117] Gen represents the current generation, and maxGen represents the maximum generation. In each iteration, individuals in the last layer are selected first. The individual with the largest value.

[0118] S1.3 Adaptive Crossover Mutation Operator

[0119] We use the maximum fitness value of the population and the average fitness value of the population The difference is used as a convergence indicator. The smaller the difference, the more the population converges towards the center, and the better the convergence. In this case, the population should tend to develop towards diversity, and the probability of crossover and mutation should be increased; conversely, it should be decreased. In order to further preserve the superior individuals in the population and change the inferior individuals, a lower crossover and mutation probability is used for the superior individuals to promote their better preservation; a higher crossover and mutation probability is used for the inferior individuals.

[0120] The adaptive crossover and mutation probabilities are expressed as follows:

[0121] = (6)

[0122] = (7)

[0123] The fitness value of an individual with higher fitness during the crossover operation. For individual target values.

[0124] S1.4 Introduces the local gradient operator

[0125] To better balance local and global search, a gradient operator is introduced into the genetic algorithm to perform a local search on each individual. The objective function value of the searched individual is compared with the original value; if it is better, the original value is replaced and the update is completed. The population convergence index is then used as the basis for the final update. and The difference is used to determine whether to perform gradient search. - < Gradient calculation is performed at that time.

[0126] A random perturbation is generated for each decision variable of the individual.

[0127] σX = { }

[0128] The gradient vectors of each decision variable can be approximated as follows:

[0129] = (8)

[0130] The updates for each decision variable are represented as follows:

[0131] = -λ* (9)

[0132] i∈[1,2,..,n], λ is the step size. Set a termination value £, when... - When the gradient update is less than £, the gradient update terminates.

[0133] The flowchart of the multi-objective evolution algorithm based on NSGA-II is as follows: Figure 2 As shown.

[0134] S2. The optimal architecture is output by a multi-objective evolutionary algorithm based on NSGA-II:

[0135] Step 2.1: Individually encode the decision variables;

[0136] Step 2.2: Initialize the first generation population uniformly within the range of decision variables;

[0137] Step 2.3: Perform non-dominated sorting, use the tournament selection algorithm to select the parent generation, and use adaptive crossover and mutation operators to generate the offspring population;

[0138] Step 2.4: Merge the offspring and parents to generate a new population, calculate the objective function value of all individuals in the population, and determine whether the conditions are met to perform a local search on the population. If the conditions are met, proceed to Step 2.5; otherwise, proceed to Step 2.6.

[0139] Step 2.5: Use the gradient operator to perform a local search on each individual in the population, and update the population individuals according to the target value after the search until the search termination condition is met. After the population update is completed, proceed to Step 2.6.

[0140] Step 2.6: Perform a fast non-dominated sort on the population, and calculate the crowding degree and convergence distance for all individuals in the last layer of the selection phase. Based on the β value, perform the final individual selection to generate a new generation of parent population.

[0141] Step 2.7: Check if the current iteration count is less than or equal to the maximum iteration count. If so, increment the current iteration count by 1 and jump to Step 2.3; otherwise, output the optimal solution.

[0142] Step 2.8: Decode the optimal architecture into a neural network architecture and train it on the task dataset until convergence.

[0143] S3. Construct a search space that integrates convolution and attention mechanisms, including:

[0144] Most neural network models are designed with multiple regions, each containing different modules and undergoing a varying number of iterations. In image recognition tasks, convolutional operations are suitable for local processing of shallow features in the early stages, while attention mechanisms are used for global feature processing in deeper layers later on. However, this approach treats convolution and attention mechanisms as two distinct operations and fails to explore the connection between them at a deeper level.

[0145] To this end, convolutional operations and attention mechanisms are organically integrated. The regional backbone is in the form of Mobilenetv3, which includes a hybrid module of self-attention and convolution. After the input feature map is convolved by 1×1, convolutional operations and self-attention operations are performed respectively. Finally, the two fusion parameters of the hybrid module of self-attention and convolution are fused.

[0146] The main structure of the search space consists of four concatenated blocks. A 3×3 convolution is used to process the input image to obtain feature maps. Each block contains 1-4 layers. As the number of block layers increases, the dimension of the feature map increases while the length and width decrease. A downsampling operation is set at the first layer of each block, with the sampling set to 2. The input and output resolutions of the remaining layers remain unchanged.

[0147] A schematic diagram of the main structure of the search space is shown below. Figure 3 As shown.

[0148] Hybrid module structure diagram as follows Figure 4 As shown, the operation steps are as follows:

[0149] S3.1 Feature Projection

[0150] The input feature map is projected using three 1×1 convolutions, and the three projected feature maps are then divided into R groups according to the number of channels C.

[0151] S3.2 Feature Aggregation

[0152] S3.2.1 Attention Mechanism Branch

[0153] S3.2.1.1 Attention Linearization

[0154] The original attention mechanism formula is:

[0155] Attention(Q,K,V) = softmax( V (15)

[0156] The space complexity is O( The time complexity is O( ). (), where N is the length of the input sequence and d is the dimension of the input. N is greater than d. The original attention mechanism formula is decomposed into a function.

[0157] (16)

[0158] make It is an identity function. For softmax, the linearized attention mechanism formula is:

[0159] Attention(Q,K,V) = (softmax V) (17)

[0160] The linearized version has a space complexity of O(Nd) and a time complexity of O(N). );

[0161] S3.2.1.2 Relative Position Encoding

[0162] Using a linearized self-attention mechanism, when the input... and When local features are identical, differences cannot be obtained. Positional encoding is added to introduce positional information. Let the positional encoding matrix be P, which represents the relationship between Q and V, and then depthwise convolution is introduced.

[0163] PositionAtten(Q,V) = Q°Depthwiseconv(P,V) (18)

[0164] ° represents the product of Hadamard matrices;

[0165] The final attention branch output feature map is

[0166] Atten = PositionAtten. , (19)

[0167] S3.2.2 Convolution Operation Branch

[0168] S3.2.2.1 Fully Connected Layer

[0169] The three feature maps obtained from projection are transformed into N sets of individual features using a fully connected layer. Each feature map;

[0170] S3.2.2.2 Shift Operation

[0171] For each group Shift each feature map, and let a set of matrices be... n∈{0,1,..., In each matrix, starting from the first value in the first row and counting sequentially along the row, the nth element is set to 1, and all other elements are set to 0. The feature map group is represented as follows: The final output of the convolution branch is:

[0172] Conv = (20)

[0173] S3.3 Final Output

[0174] Set two fusion parameters and , and The values ​​of are all in the range of [0, 0.25, 0.5, 0.75, 1]. The two fusion parameters represent the fusion ratio of the self-attention mechanism and the convolution operation during the fusion process, respectively. The final output is expressed as:

[0175] (twenty one).

[0176] Example

[0177] First, based on the specific environment and conditions, subnet sampling was performed using a multi-objective evolutionary algorithm improved from NSGA-II. With accuracy and model complexity as optimization objectives, the initial population was set to 50 individuals, the initial mutation probability of the mutation operator was 0.2, the initial crossover probability of the crossover operator was 0.9, and the maximum number of evolutions was set to 7. The collected subnets were incompletely trained using the Oxford Flowers 102 and Cifar-10 datasets, and their performance was evaluated and validated on the test set. Then, the individuals selected after structure search were fully trained on the dataset, with an initial learning rate of 0.1×b / 256, where b is the batch size. The SGD optimizer was used, and the learning rate could be dynamically adjusted adaptively with the number of epochs. The final training results were then compared with the performance of other semi-automatically or automatically designed models.

[0178] The TACNetM series of architecture models we searched for, after being fully trained on the Cifar-10 and Flowers-102 datasets, have shown their final performance and comparison in Table 1:

[0179] Table 1. Model Performance and Comparison on Datasets

[0180]

[0181] The comparison results show that the four searched models maintain a low number of parameters while ensuring a high level of accuracy, and are also faster in computation.

Claims

1. A multi-objective evolutionary neural architecture search method integrating convolution and self-attention, characterized in that, By constructing an improved multi-objective evolutionary algorithm, the efficiency of neural architecture search is enhanced, allowing for faster discovery and preservation of individual neural architectures during the search process, while balancing convergence and diversity. Furthermore, by constructing a search space that integrates convolution and self-attention operations, the processing capability of global and local information in image processing tasks is improved, enhancing the plasticity of the search space and the generalization performance of individuals across different task sets. The decision variables of the search space are encoded as population individuals in the improved multi-objective evolutionary algorithm, and the optimal individual is searched through the operation of the improved algorithm and decoded into the optimal network architecture. S1. Construct a neural architecture search framework for a multi-objective evolutionary algorithm based on NSGA-II; S1.

1. Uniform population initialization based on Euclidean distance; S1.1.1: Randomly initialize the initial population to generate T times the preset number of individuals; S1.1.2: Set up replicas of the initial population, and normalize the decision variables of individuals in the replicas according to the range of values ​​of the decision variables; the decision variable of any individual in the population is... Its value range is [ The normalized decision variables are as follows: = (1) S1.1.3: Set the parameter ρ to represent the Euclidean distance of each individual, and n to be the number of decision variables; p= (2) S1.1.4: Arrange individuals according to their ρ value, and select the required number of individuals from the initial population evenly based on their ρ value; S1.2 Adaptive selection strategy based on convergence distance and crowding; A distance-based convergence index α is proposed: After the population individuals are ordered by dominance in step S1.1.4, the convergence direction of all individuals in the first layer of the population is determined as the convergence direction of the entire population, and α is the average distance from each individual in the last layer of the population to all individuals in the first layer of the population; after fast non-dominated sorting, the target values ​​of the j-th individual in the first layer are expressed as: = ( , ..., The target values ​​for the k-th individual in the last layer are represented as follows: = ( , ..., ); Let the first layer have m individuals, y represent the number of objective functions, z represent the objective function, and the convergence distance of the k-th individual in the last layer be expressed as: α(k) = (3) β represents the crowding level of an individual. After adaptively fusing crowding level and convergence distance, the final individual selection criterion is as follows: = (1-and)× + and× (4) and = (5) Gen represents the current generation, and maxGen represents the maximum generation. In each iteration, individuals are selected from the last layer of individuals. The individual with the largest value; S1.3 Adaptive crossover and mutation operator; Use the maximum fitness value of the population and the average fitness value of the population The difference is used as a convergence index. The smaller the difference, the more the population converges towards the center, and the better the convergence. In order to further preserve the superior individuals in the population and change the inferior individuals, the crossover probability and mutation probability used for superior individuals are lower than those used for inferior individuals. The adaptive crossover probability and mutation probability are expressed as: = (6) = (7) The fitness value of an individual with higher fitness during the crossover operation. For individual target values; S1.4 introduces the local gradient operator; To better balance local and global search, a gradient operator is introduced into the genetic algorithm to perform local search on individuals, and the objective function value of the searched individual is compared with the original value. If the objective function value of an individual is better than the original value, it replaces the original value and the update is completed, based on the population convergence index. and The difference is used to determine whether to perform gradient search. - < Gradient calculation is performed at that time. A random perturbation is generated for each decision variable of the individual. σX = { } The gradient vectors of each decision variable are represented as follows: = (8) The updates for each decision variable are represented as follows: = -λ (9) i∈[1,2,..,n], where λ is the step size; Set the termination value £, when - When < £, gradient update terminates; S2. The optimal architecture is output by a multi-objective evolutionary algorithm based on NSGA-II: Step 2.1: Individually encode the decision variables; Step 2.2: Initialize the first generation population uniformly within the range of decision variables; Step 2.3: Perform non-dominated sorting, use the tournament selection algorithm to select the parent generation, and use adaptive crossover and mutation operators to generate the offspring population; Step 2.4: Merge the offspring and parents to generate a new population, calculate the objective function value of all individuals in the population, and determine whether the conditions for local search of the population are met. If the conditions are met, proceed to Step 2.5; otherwise, proceed to Step 2.

6. Step 2.5: Use the gradient operator to perform a local search on each individual in the population, and update the population individuals according to the target value after the search until the search termination condition is met. After the population update is completed, proceed to Step 2.

6. Step 2.6: Perform fast non-dominated sorting on the population, and calculate the crowding degree and convergence distance for all individuals in the last layer of the selection phase. Based on the β value, perform the final individual selection to generate a new generation of parent population. Step 2.7: Determine if the current iteration count is less than or equal to the maximum iteration count. If so, increment the current iteration count by 1 and jump to Step 2.3; otherwise, output the optimal solution. Step 2.8: Decode the optimal architecture into a neural network architecture and train it on the task dataset until convergence; S3. Construct a search space that integrates convolution and attention mechanisms; The convolution operation and self-attention mechanism are combined, and the backbone of the search space region is in the form of Mobilenetv3, which includes a self-attention and convolution hybrid module. After the input feature map is convolved by 1×1, convolution operation and self-attention operation are performed respectively, and finally fused by two fusion parameters of the self-attention and convolution hybrid module. The main structure of the search space consists of four concatenated blocks. A 3×3 convolution is used to process the input image to obtain feature maps. Each block contains 1-4 layers. As the number of block layers increases, the dimension of the feature map increases while the length and width decrease. A downsampling operation is set at the first layer of each block, with the sampling set to 2. The input and output resolutions of the remaining layers remain unchanged. Based on the MobileNetv3 network, the self-attention and convolution hybrid module replaces the 3×3 depthwise convolution; the hybrid module is as follows: The input and output feature maps are represented as A∈ B∈ The feature tensors at pixel (i,j) in the input and output feature maps are respectively ∈ , ∈ ,definition ∈ p,s∈[0,1,2,...,g-1] are the weights of the convolution kernel at (p,s). , , and Let represent the query, key, value, and transformation matrix at pixel (i,j) in the corresponding input feature map, and define the Shift operation. ; The convolution operation is broken down into two steps; The first step of the convolution operation: = (10) The second step of the convolution operation: = shift( , (11) = (12) The attention mechanism can be broken down into two steps; The first step of the attention mechanism: = , = , = (13) The second step of the attention mechanism: = (14) The first part of both the convolution operation and the self-attention mechanism maps features to a deeper space and accounts for the majority of floating-point operations and parameters; the second part of the convolution operation and the self-attention mechanism respectively performs feature aggregation for the two operations; after sharing the first part of the convolution operation and the self-attention mechanism, they each perform their respective second part operations. The steps for operating the self-attention and convolution hybrid module are as follows: S3.1 Feature Projection The input feature map is projected using three 1×1 convolutions, and the three projected feature maps are then divided into R groups according to the number of channels C. S3.2 Feature Aggregation S3.2.1 Attention Mechanism Branch S3.2.1.1 Attention Linearization The original attention mechanism formula is: Attention(Q,K,V) = softmax( )V (15) The space complexity is O( The time complexity is O( ). (), where N is the length of the input sequence and d is the dimension of the input. N is greater than d. The original attention mechanism formula is decomposed into a function. (16) make It is an identity function. For softmax, the linearized attention mechanism formula is: Attention(Q,K,V) = (softmax V) (17) The linearized version has a space complexity of O(Nd) and a time complexity of O(N). ); S3.2.1.2 Relative Position Encoding Using a linearized self-attention mechanism, when the input... and When local features are identical, differences cannot be obtained. Positional encoding is added to introduce positional information. Let the positional encoding matrix be P, which represents the relationship between Q and V, and then depthwise convolution is introduced. PositionAtten(Q,V) = Q°Depthwiseconv(P,V) (18) ° represents the product of Hadamard matrices; The final attention branch output feature map is Atten = PositionAtten( , ) (19) S3.2.2 Convolution Operation Branch S3.2.2.1 Fully Connected Layer The three feature maps obtained from projection are transformed into N sets of individual feature maps using a fully connected layer. Each feature map; S3.2.2.2 Shift Operation For each group Shift each feature map, and let a set of matrices be... n∈{0,1,..., In each matrix, starting from the first value in the first row and counting sequentially along the row, the nth element is set to 1, and all other elements are set to 0. The feature map group is represented as follows: The final output of the convolution branch is: Conv = (20) S3.3 Final Output Set two fusion parameters and , and The values ​​of are all in the range of [0, 0.25, 0.5, 0.75, 1]. The two fusion parameters represent the fusion ratio of the self-attention mechanism and the convolution operation during the fusion process, respectively. The final output is expressed as: Output = ×Atten + × Conv (21)。

Images