Multi-target deep neural network architecture search method oriented to hybrid CNN-Transform architecture

Abstract

The invention discloses a multi-target deep neural network architecture search method oriented to a hybrid CNN-Transform architecture, and the method comprises the steps: carrying out the parallel generation and iterative optimization of a candidate network at a population level through a multi-target evolution algorithm, and carrying out the system exploration of different module combinations and parameter configurations of a network structure. Meanwhile, the model precision and the reasoning speed are used as objective functions, and an efficient framework most suitable for a specific application scene is approximated step by step. According to the method, different network architectures are continuously tested and evaluated, so that an efficient model structure is quickly found, and the labor cost and low efficiency in a trial and error process are avoided. According to the method, the fusion mode of the CNN and the Transform is adjusted in a self-adaptive manner, so that the model keeps relatively high accuracy and relatively low calculation overhead in diversified image features and complex scenes. The innovative hybrid architecture and evolution optimization method provide a new thought for the automatic design of the deep neural network.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural architecture search based on evolutionary learning and hybrid CNN-Transformer supernet and search space design, and relates to a multi-objective deep neural network architecture search method for a hybrid CNN-Transformer architecture. Background Art

[0002] Deep neural networks have become a core technology in the field of computer vision, with outstanding performance in tasks such as image classification, object detection, and image segmentation. At the same time, these tasks also place higher demands on multi-scale fusion of feature representations and global context modeling. Traditional convolutional neural networks (CNNs) are good at capturing local texture and edge information, but their ability to model long-range dependencies is insufficient. The Transformer structure, with its self-attention mechanism, demonstrates unique advantages in integrating global image information and has been widely introduced into computer vision tasks. Representative models such as ViT and Swin Transformer have achieved remarkable results in many visual applications. In order to balance the local feature extraction capabilities of CNN and the global perception capabilities of Transformer, the hybrid CNN-Transformer architecture came into being. By fusing convolution and self-attention modules in the backbone network, it maintains high-resolution detail expression while enhancing the understanding of global context information.

[0003] In order to improve model performance, many complex network structures have been proposed. However, these emerging structural designs greatly increase the difficulty of manually building models, making the process not only cumbersome but also time-consuming and energy-consuming. To meet this challenge, Neural Architecture Search (NAS) has emerged as an automated method. Its core goal is to automatically search for efficient and high-performance network architectures through algorithms, thereby greatly improving the efficiency of model design. Many researchers have proposed NAS methods for image classification and achieved good performance. However, most existing NAS methods choose to use continuous relaxation technology to jointly optimize the architecture parameters and weights, thereby searching the entire architecture at one time. Although continuous relaxation provides efficiency for the search of network architectures, it requires a reformulation of the problem and tends to favor architectures that converge faster, resulting in the search of potential suboptimal architectures.

[0004] To address these issues, a multi-objective evolutionary algorithm is used to evolve neural architectures. This algorithm approaches the optimal architecture by processing a set of solutions in parallel, sampling based on the relative differences between solutions, thus avoiding the need for gradient estimation from continuous relaxation. By optimizing model accuracy and inference speed across multiple objectives, the algorithm searches for efficient networks with high-resolution representations and attention, achieving superior model performance. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-objective deep neural network architecture search method for a hybrid CNN-Transformer architecture, which combines the advantages of CNN and Transformer and uses a multi-objective evolutionary neural architecture search algorithm to optimize the network architecture to improve performance.

[0006] The technical solution of the present invention:

[0007] A multi-objective deep neural network architecture search method for a hybrid CNN-Transformer architecture, with the following steps:

[0008] Step 1: Build a dual-branch supernet of hybrid CNN and Transformer and its search space

[0009] The dual-branch supernet is mainly composed of STEM blocks, context branches, spatial branches, fusion modules and classification heads. The dual-branch supernet consists of three stages, each of which is connected in sequence. The first stage is as follows: the input image x is preliminarily extracted through the STEM block to obtain the feature x stem , then the feature x stem Input into the context branch and spatial branch respectively;

[0010] In the spatial branch, we first transform the feature x stem Input to the multi-scale hybrid scaling convolution module; on the one hand, this module extracts high-resolution detail features x through 1×1 convolution res On the other hand, x stem The low-resolution features obtained by downsampling by 2 times are fed into three depth convolution kernels of different sizes (3×3, 5×5, and 7×7) for feature extraction; the convolution outputs of each branch are concatenated and fused, and then upsampled to restore their original spatial size to obtain the feature map. Finally, the feature is combined with x stem Add element by element to generate spatial feature representation x s ′.

[0011] In the context branch, we first transform the feature map x stem Input to the multi-scale hybrid scaling convolution module and downsample it by 2 times to obtain a low-resolution feature map Will Input to the lightweight Transformer module to capture global context dependencies and improve the global representation ability of features; in the lightweight Transformer module, the query Q is first generated through the linear projection layer, and The input is fed into the deep convolution for calculation, and the output is passed through the linear projection layer to obtain the key K and value V. The multi-head self-attention mechanism is used to process the query (Q), key (K), and value (V) and calculate the attention score to better capture global information. The specific attention function calculation process is as follows:

[0012]

[0013] Conv(·) represents a 1×1 convolution operation. The outputs of all attention heads are concatenated to obtain x sa ; x sa with x stem After addition normalization, it is input into the feedforward neural network for feature transformation and nonlinear mapping to obtain the final output x c ';

[0014] Then the processed feature x s ′ and x c ′ is input into the feature fusion module, and the low-resolution feature map x obtained by the context branch is c ′ performs a series of processing, including feature transformation through 1×1 convolution layer, standardization through batch normalization layer, and activation operation through Sigmoid layer; the processed feature map is upsampled using bilinear interpolation to increase its resolution to 1 / 8 of the original image; the high-resolution feature map x obtained by the upsampled feature map and the spatial branch is s 'Perform element-by-element multiplication to achieve feature fusion; after three stages of processing, the features obtained in the last stage are input into the classification head for upsampling to obtain the final output;

[0015] After the dual-branch supernet is constructed, the search space is constructed; the starting STEM block, fusion module and classification head are fixed; the search space is represented by real number coding and converted into an operational form; for each multi-scale hybrid scaling convolution, the search is concentrated on the channel dimension; the three stages of the dual-branch supernet include a total of six multi-scale hybrid scaling convolution modules; in each multi-scale hybrid scaling convolution, the search unit consists of three depth convolution layers with different kernel sizes, namely 3×3, 5×5 and 7×7. The number of channels of these convolution blocks constitutes the basic unit of the search space, which is encoded as {x i ,y i ,z i}, i = [1, 6]; for each lightweight Transformer module, denote the number of Transformer blocks, the number of attention heads, the embedding dimension, and the MLP ratio in the i-th Transformer module, respectively.

[0016] Step 2: Weight-sharing-based adaptive genetic multi-objective neural architecture search

[0017] For the target dataset D = {D train ,D valid ,D test}, construct a multi-objective optimization problem to maximize both accuracy and inference speed:

[0018]

[0019] Among them, Ω x is the neural architecture search space, x={x1,...,x n} is a candidate neural network, the inner variable w∈Ω w is the weight associated with it; the target vector F is composed of the accuracy f1 and the inference speed f2; L(w;x) is the architecture x in the training dataset D train The training loss on

[0020] First, the training dataset D train Pre-train the two-branch supernet for 100 rounds; then, randomly generate 100 initial individuals to initialize the population in the search space, each individual represents a candidate neural network architecture; then perform performance-aware hybrid sampling on the trained two-branch supernet; let the set of subnets sampled in history be S = {s1,s2,...,s t}, where each subnet s i Performance P(s i ) By using the validation dataset D valid Evaluate on the obtained; suppose subnet s t It is the performance of the subnet that performs better in the t-th sampling, and its local neighborhood space H(s t ) is a set of subnetworks generated by fine-tuning its structural parameters, expressed as:

[0021] H(s t )={s′ t |||s′ t -s t ||≤ε} (3)

[0022] Among them, ε is a hyperparameter that controls the magnitude of the subnet neighborhood transformation, s′ t is the performance of the currently sampled subnet; at each sampling, a probability β is set to decide whether to perform global random sampling or local sampling based on performance; during the training process, the subnet is randomly sampled with a probability of β, and the local neighborhood space H(s) of the excellent historical subnet is selected with a probability of 1-β. t) in the sampled subnet; then, the importance of the channels of the multi-scale hybrid scaled convolution and the input tokens of the lightweight Transformer module are scored based on the L1 norm, and the features ranked in the top 50% of importance are retained; by selecting parameter configurations with high-quality features from the two-branch supernet, including the number of convolution channels, the number of Transformer blocks, the number of attention heads, the embedding dimension and the MLP ratio, a population with sorted features is generated; each architecture needs to be evaluated for fitness to determine its performance in a specific task; this evaluation process is based on two main objectives, namely accuracy and inference speed; on the validation set D valid The accuracy and inference speed of each subnet are evaluated, and the fitness value of the subnet architecture is calculated. The generated population is subjected to adaptive crossover and mutation operations, and the adaptive crossover and mutation probabilities are calculated using the following formula:

[0023]

[0024] Calculate the fitness value of each individual in the population and determine the highest fitness value F in the population max And the average fitness value F avg , while considering the fitness F of the individuals currently participating in the genetic operation; set the adjustment parameters α1 and γ1 to 0.3, α2 and γ2 to 0.7; when the individual fitness value is lower than the average fitness value, select α2 and γ2 as the probability of individual crossover and mutation operations; if the individual fitness value is higher than the average fitness value, select and To adjust the probability of crossover and mutation operations; perform crossover operations, each individual represents a subnetwork architecture, each subnetwork architecture has 3 stages, each stage consists of context branch, space branch and fusion module; randomly generate a random number r in the range of [0,1], if r is less than the crossover probability P c , perform fine-grained crossover within each stage; otherwise, perform coarse-grained crossover across stages and randomly select the target stage to perform gene exchange between modules; set and is the i-th stage of individuals p1 and p2, where the fine-grained crossover operation is as follows: first sort the search parameter encoding values ​​n1 and n2 of individuals p1 and p2 in descending order, assuming n1>n2, if the performance of individual p1 is better than that of individual p2, then The code bit with the larger value is Perform crossover operation on the corresponding coding bits; if the performance of individual p2 is better than that of individual p1, it indicates that The operator in is more superior, and The coded value at the intersection is assigned The corresponding bit improves the p1 performance, thereby increasing the probability of producing better offspring; then a random number between 0 and 1 is generated for each gene bit in sequence, and the number is added to the mutation probability P m Compare; if the random number is greater than the mutation probability P m , then randomly select a different value in the coding space corresponding to the gene position to replace it; if the random number is less than or equal to the mutation probability P m , then the gene position will not be mutated;

[0025] Next, a multi-objective sorting method based on the Pareto front is used to guide environmental selection, and solutions that are not dominated by other individuals are retained in the Pareto front set. To prevent the loss of population diversity, the crowding distance is calculated, and individuals with larger distances represent greater diversity in the multi-objective space. Based on the crowding distance, individuals with high fitness are given priority in the environmental selection process, and some individuals with strong diversity are retained. Then, the individuals after environmental selection are judged. If the iteration termination condition is met, the final architecture is output. Otherwise, the fitness is evaluated and the next round of evolution is carried out.

[0026] Beneficial effects of the present invention: In traditional deep neural network designs, CNN has powerful local feature extraction capabilities and is widely used in image processing tasks. However, due to its weak ability to capture long-range dependencies, it is difficult to process complex image context information. Transformer has obvious advantages in processing long-range dependencies and global context information, but its computational complexity is high, especially in pixel-level tasks, which easily leads to waste of computing resources. Therefore, a hybrid architecture based on CNN and Transformer combines the advantages of both, and can avoid the limitations of a single model architecture in specific tasks while ensuring high accuracy. However, designing an efficient hybrid architecture is not easy. Traditional manual design methods are not only time-consuming and labor-intensive, but also difficult to achieve global optimization. In order to overcome this problem, the present invention proposes an optimization strategy based on a multi-objective evolutionary algorithm, which explores the optimal CNN-Transformer hybrid architecture through automated neural architecture search technology. In this method, a multi-objective evolutionary algorithm is used to generate and iteratively optimize candidate networks in parallel at the population level, and systematically explore different module combinations and parameter configurations of the network structure. In this process, accuracy and inference speed are used as objective functions at the same time, and the efficient architecture that is most suitable for specific application scenarios is gradually approached. Compared to traditional manual design methods, evolutionary algorithms can rapidly identify efficient model structures by continuously testing and evaluating different network architectures, avoiding the labor costs and inefficiencies of trial and error. Furthermore, by adaptively adjusting the fusion of CNN and Transformer, the model can maintain high accuracy and low computational overhead in diverse image features and complex scenarios. This innovative hybrid architecture and evolutionary optimization method provides new insights into the automated design of deep neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a framework diagram of the present invention.

[0028] Figure 2 This is the structure diagram of the multi-scale hybrid scaling convolution module.

[0029] Figure 3 This is the structure diagram of the lightweight Transformer module.

[0030] Figure 4 It is a double-branch supernet structure diagram.

[0031] Figure 5 This is a schematic diagram of the Transformer searchable block.

[0032] Figure 6 It is a schematic diagram of the dual-branch supernet search space. DETAILED DESCRIPTION

[0033] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0034] like Figure 1 As shown, the present invention provides a multi-objective deep neural network architecture search method for a hybrid CNN-Transformer architecture. First, the data set is divided into a training set, a validation set, and a test set in a ratio of 6:2:2, and the divided data is subjected to tensor conversion and normalization. Then, a multi-scale hybrid scaling convolution module and a lightweight Transformer module are constructed as search units. Then, a dual-branch supernet and a search space containing a context branch and a spatial branch are constructed, and the supernet is trained. Finally, a multi-objective optimization problem is established to maximize the accuracy and inference speed, and the network architecture is optimized to the final architecture through an evolutionary algorithm. The specific steps are as follows:

[0035] (1) Constructing a multi-scale hybrid scaling convolution module and a lightweight Transformer module

[0036] 1) Construct a multi-scale hybrid scaling convolution module

[0037] In order to effectively maintain a larger receptive field during the feature extraction process, thereby enhancing the model's ability to capture information at different scales, a multi-scale hybrid scaling convolution module is constructed, such as Figure 2 As shown. For the input feature map F in Preserve high-resolution details through 1×1 point-by-point convolution to generate feature map F 1×1 Next, multi-scale feature extraction is performed. in Downsample to reduce its size to twice its original size and obtain a new feature map F down 。 down Input into three deep convolutional layers (kernel sizes are 3×3, 5×5, and 7×7 respectively) to generate a multi-scale feature map F 3×3 、F 5×5 、F 7×7 Then, along the channel dimension, F 3×3 、F 5×5 、F 7×7 Splicing to get a fused feature map F concat Subsequently, bilinear interpolation is applied to the fused feature map to upsample it by 2 times to obtain the upsampled feature map F up Finally, the upsampled feature map F up With F 1×1 Perform element-by-element addition to generate the required output feature map F out .

[0038] 2) Building a lightweight Transformer module

[0039] In order to capture the global features in the image and reduce the computational overhead of the Transformer, a lightweight Transformer module is constructed, such as Figure 3 As shown. For the input feature F in Compress the spatial dimension through depth convolution (kernel size 3×3, stride 2, padding 1) to generate a compressed feature map F compressed Then, F in ′ performs linear projection to generate query Q, and compressed Perform linear projection to generate key K and value V. Calculate multi-head self-attention:

[0040]

[0041] The query Q is multiplied by the transposed key K and scaled, and then the interaction between different heads is modeled by 1×1 point-by-point convolution. After softmax normalization, it is multiplied by the value V to obtain the output feature. Two linear layers are added after the attention layer. The first layer converts the input dimension from d to in Expanded to 4D in , the second layer changes the dimension from 4d in Reduce to d in Finally, the feature map F is output out With the input feature map F in The residual connections between them are added to obtain the final output feature map.

[0042] (2) Constructing a dual-branch supernet and search space based on CNN and Transformer

[0043] 1) STEM module

[0044] The input image undergoes a 3×3 convolution with a stride of 2 to generate a 1 / 2 resolution feature map. Four MobileNet blocks (kernel size 3×3) are then connected in sequence, where the first and third blocks use depthwise convolution with a stride of 2 to gradually reduce the resolution to 1 / 8.

[0045] 2) Space branch

[0046] The spatial branch consists of multiple stages, each of which contains a multi-scale hybrid convolution module for feature extraction, which is then fused with the output features of the context branch through the feature fusion module. First, the high-resolution feature map output by the STEM module is input into the multi-scale hybrid convolution module to generate a multi-scale feature map F multi-scale .

[0047] 3) Contextual branching

[0048] The context branch contains a multi-scale hybrid scaling convolution module and a lightweight Transformer module. Feature extraction is performed through the multi-scale hybrid scaling convolution module, and then 2 times downsampling is performed to obtain the feature map F context , F contex Input the lightweight Transformer module to obtain the global context feature map F global .

[0049] 4) Feature Fusion

[0050] Use 1×1 convolution operation to perform global context feature map F global After processing, it is upsampled and its resolution is restored to the same resolution as the spatial branch. The upsampled feature map is the same as the multi-scale feature map F extracted from the spatial branch. multi-scale Multiply element by element to get F fused .

[0051] 5) Search Space

[0052] The fixed STEM block, fusion module and classification head remain unchanged, and the search space is represented by real number encoding. For each convolution module, the search is concentrated on the channel dimension. For each multi-scale hybrid scaling convolution, the search is concentrated on the channel dimension; the three stages of the dual-branch supernet include a total of six multi-scale hybrid scaling convolution modules; in each multi-scale hybrid scaling convolution, the search unit consists of three depth-wise convolution layers with different kernel sizes, namely 3×3, 5×5 and 7×7. The number of channels of these convolution blocks constitutes the basic unit of the search space, encoded as {x i ,y i ,z i}, i = [1, 6]; for each lightweight Transformer module, denote the number of Transformer blocks, the number of attention heads, the embedding dimension, and the MLP ratio in the i-th Transformer module, respectively.

[0053] (3) Adaptive genetic multi-objective neural architecture search based on weight sharing

[0054] This implementation involves using an evolutionary algorithm to search for a multi-objective neural architecture based on weight sharing to optimize the network architecture and maximize accuracy and inference speed. The specific steps are as follows:

[0055] 1) Multi-objective optimization modeling

[0056] We approach image classification tasks from the perspective of multi-objective evolution and mathematically transform the search for the optimal architecture into a multi-objective two-level optimization problem to maximize both accuracy and inference speed. train ,Dvalid ,D test}, the modeling expression is:

[0057] maximizeF(x)={f1(x;w * (x)),f2(x)},

[0058]

[0059] x∈Ω x ,w∈Ω w

[0060] Among them, Ω x is the neural architecture search space, x={x1,...,x n} is a candidate neural network, the inner variable w∈Ω w is the weight associated with it. The target vector F consists of the accuracy (f1) and the inference speed (f2). L(w;x) is the architecture x in the training dataset D train The training loss on .

[0061] 2) Adaptive genetic algorithm

[0062] First, a supernet is pre-trained on the training set for 100 epochs. After training, the supernet is sampled. For the first 50 epochs, subnets are randomly sampled with a probability of 70%, and for the second 50 epochs, neighboring architectures are sampled based on previously sampled good subnets with a probability of 30%. Subnets are initialized by inheriting the weights of the corresponding modules from the supernet. Next, the importance of convolutional channels and Transformer input tokens is scored based on the L1 norm, and the top 50% of important features are retained. A feature-ranked population is generated by selecting parameter configurations from the supernet that demonstrate high quality (such as the number of convolutional channels, number of Transformer blocks, number of attention heads, embedding dimension, and MLP ratio). Each architecture undergoes a fitness evaluation to determine its performance on a specific task. The evaluation process is based on two core objectives: accuracy and inference speed. Crossover and mutation are then performed, with the crossover and mutation probabilities calculated as follows:

[0063]

[0064] Calculate the fitness value of each individual in the population and determine the highest fitness value F in the population max And the average fitness value F avg , while considering the fitness F of the individuals currently participating in the genetic operation; set the adjustment parameters α1 and γ1 to 0.3, α2 and γ2 to 0.7; when the individual fitness value is lower than the average fitness value, select α2 and γ2 as the probability of individual crossover and mutation operations; if the individual fitness value is higher than the average fitness value, select and To adjust the probability of crossover and mutation operations; perform crossover operations, each individual represents a subnetwork architecture, each subnetwork architecture has 3 stages, each stage consists of context branch, space branch and fusion module; randomly generate a random number r between [0,1], if r is less than the crossover probability P c , perform fine-grained crossover within each stage; otherwise, perform coarse-grained crossover across stages and randomly select the target stage to perform gene exchange between modules; set and is the i-th stage of p1 and p2, where the fine-grained crossover operation is as follows: first, the search parameter encoding values ​​n1 and n2 of individuals p1 and p2 are sorted in descending order. Assuming n1>n2, if the performance of individual p1 is better than that of individual p2, it is usually better than the model with larger network capacity in terms of accuracy. Therefore, The code bit with the larger value is Perform crossover operation on the corresponding coding bits; if the performance of individual p2 is better than that of individual p1, it indicates that The operator in is more superior, and The coded value at the intersection is assigned The corresponding bit improves the p1 performance, thereby increasing the probability of producing better offspring. Then generate a random number between 0 and 1 for each gene bit in sequence, and add it to the mutation probability P m Compare; if the random number is greater than the mutation probability P m , then randomly select a different value in the coding space corresponding to the gene position to replace it; if the random number is less than or equal to the mutation probability P m , then the gene position will not be mutated.

[0065] Next, a multi-objective ranking method based on the Pareto front guides environmental selection, retaining solutions that are not dominated by other individuals in the Pareto front set. To prevent loss of population diversity, a crowding distance is calculated; individuals with a larger crowding distance represent greater diversity in the multi-objective space. Based on the crowding distance, individuals with high fitness are prioritized during environmental selection, while some individuals with high diversity are retained. After environmental selection, individuals are evaluated. If the iteration termination criteria are met, the final architecture is output; otherwise, a fitness evaluation is performed and the next round of evolution is performed.

[0066] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-objective deep neural network architecture search method for a hybrid CNN-Transformer architecture, characterized by: Here are the steps: Step 1: Build a dual-branch supernet of hybrid CNN and Transformer and its search space The dual-branch supernet is mainly composed of STEM blocks, context branches, spatial branches, fusion modules and classification heads. The dual-branch supernet consists of three stages, each of which is connected in sequence. The first stage is as follows: the input image x is preliminarily extracted through the STEM block to obtain the feature x stem , then the feature x stem Input into the context branch and the spatial branch respectively; in the spatial branch, the feature x stem Input to the multi-scale mixed scaling convolution, use 1×1 convolution to retain high-resolution feature details to get x res , and at the same time for x stem The output is then concatenated and upsampled by 2 times to obtain the output. Then x stem and Add and get the output x s '; In the context branch, feature x stem Input to the multi-scale hybrid scaling convolution and then downsampled by 2 times to obtain Then Input to the lightweight Transformer module to extract global context information; Apply the linear projection layer to generate the query vector Q, and The output is fed into the deep convolution layer for operation, and the key K and value V are obtained through the linear projection layer. After that, multi-head self-attention calculation is performed, and the attention function of query Q, key K and value V is as follows: Among them, Conv(·) is a 1×1 point-by-point convolution, which models the interaction between different heads and makes the attention of each head depend on all K and V; Finally, the output values ​​of each attention head are concatenated and linearly projected to obtain the output x sa ; x sa with x stem After addition normalization, it is input into the feedforward neural network for feature transformation and nonlinear mapping to obtain the final output x c '; Then the feature x processed by the spatial branch and the context branch is s ′ and x c ′ is input to the feature fusion module, and the low-resolution feature map x obtained by the context branch is first c ′ is processed by 1×1 convolution layer, batch normalization layer, Sigmoid layer, and upsampled to 1 / 8 resolution using bilinear interpolation method, and then multiplied by the spatial branch to obtain high-resolution feature x s 'Perform feature fusion operation; after three stages of processing, the features obtained in the last stage are input into the classification head for upsampling to obtain the final feature map; After the dual-branch supernet is constructed, the search space is constructed; the starting STEM block, fusion module and classification head are fixed and unchanged; the search space is encoded by real number coding; for each multi-scale hybrid scaled convolution, the search is concentrated on the channel dimension; the three stages of the dual-branch supernet contain a total of 6 multi-scale hybrid scaled convolutions, and the search unit of each multi-scale hybrid scaled convolution consists of the channel number of 3×3, 5×5, and 7×7 depth convolution blocks, which are encoded as {x i ,y i ,z i }, i = [1, 6]; For each Transformer module, the number of search Transformer modules, the number of attention heads, the embedding dimension, and the MLP ratio are encoded as Step 2: Weight-sharing-based adaptive genetic multi-objective neural architecture search For the target dataset D = {D train ,D valid ,D test }, construct a multi-objective optimization problem to maximize both accuracy and inference speed: Among them, Ω x is the neural architecture search space, x={x1,...,x n } is a candidate neural network, the inner variable w∈Ω w is the weight associated with it; the target vector F is composed of the accuracy f1 and the inference speed f2; L(w;x) is the architecture x in the training dataset D train The training loss on .

2. The multi-objective deep neural network architecture search method for a hybrid CNN-Transformer architecture according to claim 1, characterized in that: The specific implementation process of step 2 is as follows: First, in the training dataset D train Pre-train the two-branch supernet for 100 rounds; then, randomly generate 100 initial individuals to initialize the population in the search space, each individual represents a candidate neural network architecture; then perform performance-aware hybrid sampling on the trained two-branch supernet; let the set of subnets sampled in history be S = {s1,s2,...,s t }, where each subnet s i Performance P(s i ) By using the validation dataset D valid Evaluate on the obtained; suppose subnet s t It is the performance of the subnet that performs better in the t-th sampling, and its local neighborhood space H(s t ) is a set of subnetworks generated by fine-tuning its structural parameters, expressed as: H(s t )={s t ′|||s t ′-s t ||≤ε} (3) Among them, ε is a hyperparameter that controls the magnitude of the subnet neighborhood transformation, s t ′ is the performance of the current sampled subnet; at each sampling, a probability β is set to decide whether to perform global random sampling or local sampling based on performance; during the training process, the subnet is randomly sampled with a probability of β, and the local neighborhood space H(s) of the excellent historical subnet is selected with a probability of 1-β. t ) in the sampled subnet; then, the importance of the channels of the multi-scale hybrid scaled convolution and the input tokens of the lightweight Transformer module are scored based on the L1 norm, and the features ranked in the top 50% of importance are retained; by selecting parameter configurations with high-quality features from the two-branch supernet, including the number of convolution channels, the number of Transformer blocks, the number of attention heads, the embedding dimension and the MLP ratio, a population with sorted features is generated; each architecture needs to be evaluated for fitness to determine its performance in a specific task; this evaluation process is based on two main objectives, namely accuracy and inference speed; on the validation set D valid The accuracy and inference speed of each subnet are evaluated, and the fitness value of the subnet architecture is calculated. The generated population is subjected to adaptive crossover and mutation operations, and the adaptive crossover and mutation probabilities are calculated using the following formula: Calculate the highest fitness value F in the population respectively max , average fitness value F avg And the fitness value F of the individual that currently needs to participate in the genetic operation; set the adjustment parameters α1 and γ1 to 0.3, α2 and γ2 to 0.7; when the individual fitness value is lower than the average fitness value, select α2 and γ2 as the probability of individual crossover and mutation operations; otherwise, select and As the probability of individual crossover and mutation operations; then perform crossover operations, each individual represents a subnetwork architecture, each subnetwork architecture has 3 stages, each stage consists of context branch, space branch and fusion module; randomly generate a random number r in the range of [0,1] and the crossover probability P c Compare, if r<P c , then perform fine-grained crossover within each stage; otherwise, randomly select a stage from it to perform coarse-grained crossover within the stage; let and is the i-th stage of individuals p1 and p2, where the fine-grained crossover operation is as follows: first sort the search parameter encoding values ​​n1 and n2 of individuals p1 and p2 in descending order, assuming n1>n2, if the performance of individual p1 is better than that of individual p2, then The code bit with the larger value is Perform crossover operation on the corresponding coding bits; if the performance of individual p2 is better than that of individual p1, it indicates that The operator in is more superior, and The coded value at the intersection is assigned The corresponding bit improves the p1 performance, thereby increasing the probability of producing better offspring; then traverses each gene bit sequentially, generates a random number between [0,1], and the mutation probability P m Perform mutation operation, if the random number is greater than the mutation probability P m , then randomly select other values ​​in the encoding space corresponding to the operation to replace, otherwise the mutation operation is not performed; Next, a multi-objective sorting method based on the Pareto front is used to guide environmental selection, and solutions that are not dominated by other individuals are retained in the Pareto front set; on this basis, each individual is evaluated by the fitness function; to prevent the loss of population diversity, the crowding distance is calculated, and individuals with larger distances represent greater diversity in the multi-objective space; based on the crowding distance, individuals with high fitness are given priority in the environmental selection process, and some individuals with strong diversity are retained; then, the individuals after environmental selection are judged, and if the iteration termination condition is met, the final architecture is output, otherwise a fitness evaluation is performed and the next round of evolutionary operations is carried out.

Images