A NAS neural network design method and system based on Shapley value
By using Shapley value in NAS algorithm to optimize the residual connection of Cell structure and overall structure, the deviation and insufficient information channels caused by artificial intervention in the existing NAS methods are solved, and more efficient and diversified neural networks are achieved Network design improves the performance and generalization capabilities of the model.
Patent Information
- Application Number
- CN202410385618.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-01
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-04-01
AI Technical Summary
The existing NAS algorithm relies on artificial intervention to design Cell structures, resulting in bias and limitations. The Cell-based NAS method lacks information channels across Cell in the overall structure, which reduces the performance and generalization capabilities of the model.
Using the NAS neural network design method based on Shapley values, we optimize the Cell structure and overall structure to build a more efficient and diversified neural network by performing residual connections and adjustment of Shapley values within the Cell structure and the overall structure.
The performance and generalization capabilities of the model are improved, local search optimal solutions are avoided, more flexible and diverse structure automatic design is achieved, manual participation is reduced, and network expression and generalization capabilities are improved.
Smart Images

Figure CN118378667B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning, and in particular to a NAS neural network design method and system based on Shapley value. Background Art
[0002] In the field of deep learning and neural networks, Neural Architecture Search (NAS) is a type of automated machine learning technology. Its goal is to solve highly complex neural network design problems in a fully automatic way by systematically searching, building a large number of neural network structures, and evaluating their performance. NAS evaluates the structural performance in a large set of neural networks based on the search space predefined by experts and finds the best performing network structure. NAS faces significant time and computational costs when searching and building a large number of structures and performing tasks such as training and verification. Existing NAS algorithms rely on human intervention to design cell structures to construct search spaces, which requires humans to embed prior knowledge, which may bring bias and limitations, thereby limiting the exploration of novel and diverse architectures. Most of the existing cell-based NAS methods construct networks by linearly stacking cells. Although these cells have residual connections internally, in terms of the overall structure, cells are only connected by linear connections, and there is no cross-cell information channel. This design hinders the transmission and fusion of information between different cells, thereby reducing the performance and generalization ability of the model. Summary of the invention
[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a NAS neural network design method and system based on Shapley value, which avoids the model from falling into the local search for the optimal solution by performing residual connections within the Cell structure and the overall structure and adjusting the mutation probability of different residual connections through Shapley value, thereby improving the performance and generalization ability of the model.
[0004] The first technical solution adopted by the present invention is: a NAS neural network design method based on Shapley value, comprising the following steps:
[0005] Based on the Cell unit, structural coding processing is performed to construct a Cell structure, wherein the Cell structure includes a chain backbone structure and a residual structure;
[0006] The Cell structure is optimized by using the performance contribution quantification method based on the disordered Shapley value of the residual structure to obtain the optimal Cell structure.
[0007] The optimal Cell structure is stacked and optimized through the ordered Shapley value quantization method to construct a NAS neural network.
[0008] Image classification is performed based on the NAS neural network to obtain image classification results.
[0009] Furthermore, the step of performing structural coding processing based on the Cell unit to construct the Cell structure specifically includes:
[0010] Obtaining a structural unit raw material, wherein the structural unit raw material includes a 1x1 convolution layer, a 3x3 convolution layer, a 5x5 convolution layer, a 3x3 depth-separable convolution layer, and a 5x5 depth-separable convolution layer;
[0011] Based on the raw materials of the structural unit, a structural coding representation is performed to output a preliminary Cell structure, wherein the preliminary Cell structure is a backbone single-chain structure with four layers of basic operations;
[0012] Perform residual structure insertion processing on the preliminary Cell structure to construct the Cell structure.
[0013] Further, the structure code includes a first element, a second element and a third element, wherein:
[0014] The first element represents the operation code in the Cell structure;
[0015] The second element represents the input position of the residual structure and the output position of the residual structure in the Cell structure. The input position of the residual structure is connected to the first-layer basic operation of the backbone single-chain structure, and the output position of the residual structure is connected to the third-layer basic operation of the backbone single-chain structure.
[0016] The third element represents the number of output channels of each layer of the backbone single-chain structure.
[0017] Furthermore, the step of performing residual optimization processing on the Cell structure by using the performance contribution quantification method based on the disordered Shapley value of the residual structure to obtain the optimal Cell structure specifically includes:
[0018] Obtain the mutation probability of the Cell structure, the mutation parameters of the Cell structure, the accuracy of the Cell structure, and the topological complexity of the Cell structure;
[0019] Perform Shapley value calculation on the residual structure in the Cell structure to obtain the Shapley value of the Cell residual structure;
[0020] According to the Shapley value of the Cell residual structure, the mutation probability of the Cell structure is adjusted, and the adjusted Cell structure is output;
[0021] Performing individual mutation processing on the adjusted Cell structure through the mutation parameters of the Cell structure to obtain the Cell structure after mutation update;
[0022] Obtain the accuracy of the Cell structure after mutation and update and the topological complexity of the Cell structure after mutation and update;
[0023] The performance contribution quantification method based on the disordered Shapley value of the residual structure is cyclically used to perform residual optimization processing steps on the Cell structure until the accuracy of the mutated and updated Cell structure and the topological structure complexity of the mutated and updated Cell structure are minimized, and the optimal Cell structure is output.
[0024] Further, the expression for calculating the Shapley value is specifically as follows:
[0025] ;
[0026] In the above formula, The Shapley value representing the Cell residual structure, represents the sum of all residual structures, Represents the Cell residual structure weight, Represents the Cell residual structure branch, represents the value function, Represents the Cell structure, express A subset of any residual connections in , excluding the residual connection currently being evaluated , Represents the set of all residual connections in all Cell structures.
[0027] Furthermore, the step of stacking the optimal Cell structure and optimizing it by an ordered Shapley value quantization method to construct a NAS neural network specifically includes:
[0028] The optimal Cell structure is stacked to construct a Cell structure population;
[0029] Generate a large-span residual connection structure for the Cell structure population;
[0030] The large-span residual connection structure is optimized by the ordered Shapley value quantization method, and the optimal large-span residual connection structure is output;
[0031] Insert the optimal large-span residual connection structure into the Cell structure population to construct a NAS neural network.
[0032] Furthermore, the Cell structure population is formed by stacking four optimal Cell structures in a single chain, wherein the second optimal Cell structure and the fourth optimal Cell structure are both set as Reduce Cell and perform downsampling operations.
[0033] The second technical solution adopted by the present invention is: a NAS neural network design system based on Shapley value, comprising:
[0034] A construction module, used for performing structural coding processing based on the Cell unit to construct a Cell structure, wherein the Cell structure includes a chain backbone structure and a residual structure;
[0035] The first optimization module is used to perform residual optimization processing on the Cell structure by using a performance contribution quantification method based on the disordered Shapley value of the residual structure to obtain an optimal Cell structure;
[0036] The second optimization module is used to stack the optimal Cell structure and optimize it through the ordered Shapley value quantization method to construct a NAS neural network;
[0037] The classification module is used to perform image classification processing based on the NAS neural network to obtain image classification results.
[0038] The beneficial effects of the method and system of the present invention are as follows: the present invention first constructs a Cell structure based on a Cell unit, and further performs residual optimization processing on the Cell structure based on a performance contribution quantification method of an unordered Shapley value of a residual structure, and uses an evolutionary algorithm improved by an unordered Shapley value of a residual structure to guide the search, and guides the direction of evolution of the Cell structure by reducing the mutation probability of a residual structure with a higher Shapley value and increasing the mutation probability of a residual structure with a lower Shapley value according to the marginal contribution (Shapley value) of the performance when a residual structure is evaluated in a Cell structure, thereby improving the efficiency of designing a Cell structure with excellent performance, and further stacking the optimal Cell structure, and optimizing it through an ordered Shapley value quantification method, and then constructing a NAS neural network, and using an evolutionary algorithm improved by an ordered Shapley value of a large-span residual structure to guide the search, and considering the order in which the large-span residual structure is added to the overall structure in the automatic design of the overall structure. By adding residual connections in order and calculating their Shapley values, the impact of each connection on the overall performance can be evaluated more accurately. The direction of overall structural evolution can be guided by reducing the mutation probability of residual structures with higher Shapley values and increasing the mutation probability of residual structures with lower Shapley values, thereby improving the efficiency of designing overall structures with excellent performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a flow chart of the steps of a NAS neural network design method based on Shapley value of the present invention;
[0040] Figure 2 It is a structural block diagram of a NAS neural network design system based on Shapley value of the present invention;
[0041] Figure 3 It is a schematic diagram of a process of automatic search of a neural network structure provided by a specific embodiment of the present invention;
[0042] Figure 4 It is a structural diagram of the Cell structure coding form provided by a specific embodiment of the present invention;
[0043] Figure 5 It is a schematic diagram of the principle of automatic design of Cell structure provided by a specific embodiment of the present invention;
[0044] Figure 6 It is a flow chart of automatic design of Cell structure provided by a specific embodiment of the present invention;
[0045] Figure 7 It is a search flow chart of an evolutionary algorithm guided by the residual structure Shapley value provided by a specific embodiment of the present invention;
[0046] Figure 8 It is a flowchart of automatic design of the overall structure of a NAS neural network provided by a specific embodiment of the present invention;
[0047] Fig. 9 It is a schematic diagram of the automatic design of the overall structure of a NAS neural network provided by a specific embodiment of the present invention;
[0048] Fig.10 It is a schematic diagram of a small span residual and a large span residual provided by a specific embodiment of the present invention;
[0049] Fig.11 is a schematic diagram of the Cifar10 data set provided by a specific embodiment of the present invention;
[0050] Fig.12 It is a result schematic diagram of the Shapley value of each residual structure when the automatic design of the Cell structure provided by the specific embodiment of the present invention is completed;
[0051] Fig.13 This is a schematic diagram of the existing structure of building a network by linearly stacking cells. DETAILED DESCRIPTION
[0052] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0053] In the field of deep learning and neural networks, Neural Architecture Search (NAS) is a type of automated machine learning technology. The goal of this technology is to solve highly complex neural network design problems in a fully automatic way by systematically searching, building a large number of neural network structures, and evaluating their performance. In actual industrial applications, due to factors such as different product batches and raw material sources, the data distribution faced by the task changes frequently, causing concept drift and resulting in a decrease in the recognition accuracy of the original model. In order to cope with these changes, human experts usually need to constantly adjust the structure and parameters of the neural network, thus bearing a huge workload. Using the NAS method for structure search and adjustment is expected to make it easier to cope with these challenges.
[0054] Specifically, NAS evaluates structural performance and finds the best performing network structure in a large collection of neural networks based on the search space predefined by experts. The results of NAS search can usually achieve better performance than expert manual design. The goal of this method is to solve the tedious and time-consuming problem of manually designing neural network structures, improve model performance and generalization capabilities, and adapt to different tasks. Traditional neural network structure design is usually completed by professional manual designers, which requires rich experience and a lot of time. To solve this problem, NAS uses an automatic search algorithm to find the optimal neural network structure, making the neural network design process more efficient and intelligent.
[0055] NAS faces significant time and computational costs when searching and building a large number of structures and performing tasks such as training and verification. Therefore, the cell-based NAS method came into being. The core idea of cell-based NAS is to decompose the entire network into multiple repeated units (cells), each of which consists of some basic operations (such as convolution, pooling, activation, etc.). By stacking these units according to different search options, a complete network architecture can be constructed. The advantage of cell-based NAS is that it significantly reduces the search space and improves search efficiency and generalization ability. Cell-based NAS can also flexibly adjust the depth and width of the network according to different hardware and resource constraints to adapt to different application scenarios.
[0056] However, traditional NAS algorithms rely on human intervention to design the Cell structure and thus construct the search space, which requires humans to embed prior knowledge, which may introduce biases and limitations, thereby limiting the exploration of novel and diverse architectures. This process requires sufficient understanding and analysis of the task and data to determine the appropriate search space. A search space that is too large or too small will affect NAS performance. Human involvement will have subjective factors and introduce human bias, which prevents the model from learning novel structures beyond human current cognition. In essence, these methods shift the workload of traditional manual design to search space design, so current NAS methods are not fully automated.
[0057] However, most of the existing Cell-based NAS methods construct the network by linearly stacking cells, such as Fig.13 As shown in the figure. Although these cells have residual connections internally, in terms of the overall structure, the cells are only connected by linear connections, and there is no cross-cell information channel. This design hinders the transmission and fusion of information between different cells, thereby reducing the performance and generalization ability of the model.
[0058] Based on this, Figure 3 As shown, an embodiment of the present invention proposes a NAS neural network design method and system based on Shapley value, which can realize more flexible and diverse automatic structure design. The algorithm of the present invention adopts the Shapley value of the residual structure as an evaluation index to measure the contribution of the residual branch to the performance of the neural network to guide the evolution of structural search. The algorithm of the present invention can improve the blindness of the traditional NAS based on evolutionary algorithm in the search direction by utilizing the Shapley value of the residual connection. The algorithm of the present invention can improve the search efficiency and increase the diversity of Cell structure by retaining the residual branches that are beneficial to performance improvement and eliminating the branches with low contribution or redundant branches, thereby improving the expression and generalization ability of the network, helping to discover higher quality network structures and avoid falling into local optimal solutions. The algorithm of the present invention can help the machine automatically design a network with high precision, simple structure and few parameters.
[0059] Reference Figure 1 The present invention provides a NAS neural network design method based on Shapley value, which comprises the following steps:
[0060] S1. Based on the Cell unit, structural coding processing is performed to construct a Cell structure, wherein the Cell structure includes a chain backbone structure and a residual structure;
[0061] Specifically, Figure 4 and Figure 6As shown, in the process of structural design, the evolutionary algorithm is used as a search strategy, and the way to implement structural search is in the form of structural coding. For example, when searching a Cell structure, the embodiment of the present invention uses "0" to represent 1x1 convolution, "1" to represent 3x3 convolution, "2" to represent 5x5 convolution, "3" to represent 3x3 depth-separable convolution (DW), and "4" to represent 5x5 depth-separable convolution (DW). During the evolutionary algorithm search process, a structural code such as [[4, 3, 0, 4, 3], [[1, 3]], [160, 96, 96, 64]] will be output to represent the structure of a Cell. The first element [4, 3, 0, 4, 3] of the coding array represents the operation code in the Cell structure. The second element [[1, 3]] represents the input and output positions of the residual in the Cell, such as the input position is connected to the output position of the first layer of the backbone structure, and the output position is connected to the output position of the third layer of the backbone structure. The third element [160, 96, 96, 64] represents the number of output channels of each layer of the backbone structure.
[0062] In order to expand the design space and improve the flexibility of design, each Cell is allowed to independently decide whether to add branches and how to connect the branches. These branches can be regarded as additional operations in the Cell, which can change the dimension of the input data or increase the depth of the network. In this design method, each Cell contains a backbone chain structure consisting of a series of basic operations, such as 1x1 convolution, 3x3 convolution, 5x5 convolution, 3x3 depth-separable convolution (DW) and 5x5 depth-separable convolution (DW). These operations represent the most basic and common operations in neural networks and can cover most network structures and functions.
[0063] In order to enhance the flexibility of the design, the system adopts a strategy of machine-generating branch cells with different connection methods. Before generating these cells, the main structure of each cell is clearly defined to ensure universality and scalability. The main structure of the cell uses the most basic and common operations as raw materials to build a single chain containing four layers of basic operations as the main structure of the cell, such as Figure 5 As shown. This design expands the design space and provides the machine with more choices and combinations. When designing the Cell structure, the machine can freely select and combine these basic operations, which helps to build a more complex and efficient network. This degree of freedom enables the machine to automatically adjust and optimize the network structure according to data and task requirements to obtain a network structure that is more suitable for specific tasks. At the same time, by adding blank operations (Skip), information can be directly transmitted to deeper levels, further increasing the flexibility of the design. This method helps to explore the design space of neural networks more comprehensively and achieve more outstanding performance.
[0064] In the automatic design of the Cell structure, although Normal Cell and Reduce Cell are still included, the embodiment of the present invention does not specifically search for Reduce Cell. Instead, by adjusting the padding and stride parameters in the convolution, the function of reducing the size of the output feature map by half is achieved without the need for additional Reduce Cell search.
[0065] S2, the Cell structure is optimized by using the performance contribution quantification method based on the disordered Shapley value of the residual structure to obtain the optimal Cell structure;
[0066] Specifically, Figure 7 As shown in the figure, in the process of automatic structural design of Cell, it is necessary to further design residual connections based on the design of the backbone chain structure. The residual structure can exist on the backbone chain structure in the form of a combination of multiple input positions and output positions. Different numbers of residual connections further increase the complexity of the structural design task. This method allows each Cell to add up to three branches, which can change the dimension of the input data.
[0067] The residual connection branch is a special connection method that allows a direct connection between the input and output of the neural network, thereby improving the depth and expressiveness of the neural network. In this way, the neural network can better overcome problems such as gradient vanishing and overfitting, thereby improving the generalization ability and robustness of the neural network. In order to increase the diversity and flexibility of the neural network, the method of the embodiment of the present invention allows the machine to freely design the residual connection branch when automatically designing the Cell structure. In the Cell structure, the small-span residual connection can make full use of the information flow inside the Cell and help optimize the extraction of local features. Due to the small scale of the Cell structure, the small-span residual connection can better capture and transmit local features, thereby improving the performance of the model.
[0068] The search strategy of the traditional evolutionary algorithm will lead to uncontrollable search direction. In order to improve the efficiency and quality of Cell design, it is necessary to determine which operations are more effective as residual connections, so as to better guide NAS to search in an effective direction. Therefore, the embodiment of the present invention calculates the Shapley value of the residual connection to evaluate its contribution to the improvement of network performance, and uses the Shapley value to improve the design direction of the residual connection in the evolutionary algorithm, thereby guiding the automatic design of cells with excellent performance.
[0069] In the process of automatic design of Cell structure, it is necessary to design residual connections with small span in the Cell backbone structure. This process can be mapped to a cooperative game. This paper proposes a performance contribution quantification method based on the disordered Shapley value of the residual structure to improve the NAS method of the evolutionary algorithm in the automatic generation of the Cell structure. The pseudo code of the evolutionary algorithm with improved residual structure Shapley value is shown in Algorithm 1. In the design of the Cell structure, we do not pay attention to the order in which the residual connections are added to the backbone structure, but focus on the marginal contribution of the performance when the residual structure exists in the Cell structure. By calculating the contribution of each residual structure to the overall performance, it is helpful to design a Cell structure with more effective residual connections and better guide the direction of structural optimization.
[0070] The Shapley value evaluation method regards each residual connection as a player and each structure as a cooperative game, and the structural performance comprehensively evaluated by topological complexity and accuracy is the total benefit. The Shapley value is used to evaluate the contribution of each residual branch to the improvement of Cell structural performance, and the mutation probability is adjusted according to the contribution. In a cooperative game, N participants are interconnected, and a value function Participants Every subset of is mapped to a real number , the real number Indicated by The expected performance gain that the substructure composed of the residual branches in can achieve on the task. The term represents the residual connection The marginal contribution to the performance improvement of all structural combinations of alliances, It means that we need to sum all possible structures. The term represents the weight of the structure combination.
[0071] The expression for calculating the Shapley value is as follows:
[0072] ;
[0073] In the above formula, The Shapley value representing the Cell residual structure, represents the sum of all residual structures, Represents the Cell residual structure weight, Represents the Cell residual structure branch, represents the value function, Represents the Cell structure, express A subset of any residual connections in , excluding the residual connection currently being evaluated , Represents the set of all residual connections in all Cell structures.
[0074] The Shapley value of residual connections can guide the NAS of the evolutionary algorithm to mutate residual connections during the structural design process. For residual branches with higher Shapley values, it indicates that these branches have a positive impact on performance improvement, so it is worth retaining or increasing the appearance of their structures. The mutation probability of these branches needs to be reduced to ensure that the most effective residual branches are retained. On the contrary, for residual branches with lower Shapley values, it indicates that these branches will hinder performance improvement, and the system will increase their mutation probability to reduce the structure containing residual branches that are harmful to performance improvement. In this way, the system increases innovation and diversity on the basis of making full use of existing knowledge, thereby improving the efficiency and effectiveness of the automatic design of evolutionary algorithm strategies. This guidance mechanism helps to optimize the structural design so that it more accurately reflects the contribution of residual connections to performance, thereby achieving more superior results in the automatic design process.
[0075] Furthermore, the process of the performance contribution quantification method based on the disordered Shapley value of the residual structure is specifically explained:
[0076] 1) Input: task dataset D, number of evolutions T, number of individuals per generation N, mutation probability , variation parameter ;
[0077] 2) population = generate_random_cells(N): Generate a random set of cells;
[0078] 3) performance_index = evaluate(population, D): calculates its accuracy and topological complexity;
[0079] 4) for t = 1, 2, . . . , T do;
[0080] 5) if t % == 0;
[0081] 6) shapley_values = compute_shapley_value(Library , D): the Shapley value of each residual branch in the designed structure library;
[0082] 7) adjust_mutation_prob(population, shapley_values, pM, qM): adjust the mutation probability according to the Shapley value;
[0083] 8) Selection: parents = selection(population): select the parent;
[0084] 9) Mutation: offspring = mutate(parents, pM, qM) : mutate individuals using mutation probability parameters;
[0085] 10) Storage structure: Library = Store_to_Library(population,performance_index): put the individuals in the population and the corresponding performance parameters into the library;
[0086] 11) Random Samples: new_cells = generate_random_cells(N - len(offspring));
[0087] 12) Recombination: offspring.extend(new_cells): update the population;
[0088] 13) Evaluation: performance_index = evaluate(offspring, D): Calculates the recognition accuracy and topological complexity of the next generation of individuals;
[0089] 14) end for;
[0090] 15) Output: Recognition accuracy and topological complexity of the optimal structure.
[0091] S3, stacking the optimal Cell structure, optimizing it through the ordered Shapley value quantization method, and constructing a NAS neural network;
[0092] Specifically, Figure 8 and Fig. 9 As shown, the above detailed discussion is about reducing the cost and time consumption of neural network design by automatically designing the basic unit of the neural network, the Cell structure. However, the complexity of neural network design is not only reflected in the design stage of the Cell structure, but also in how to effectively combine these well-designed Cell structures to build an overall network structure with reasonable levels and functions. The embodiments of the present invention will further explore the fully automatic design of neural networks, including methods for automatically designing the overall network structure and generating residual connections.
[0093] The construction process of the overall network structure is very similar to the automatic design process of the Cell structure in the previous section. First, in order to achieve fully automatic design of the neural network, the evolutionary algorithm is used as the search strategy, and the Cell structure with excellent performance obtained by the previous automatic design is used as the basis to build an efficient search space. After the efficient search space is automatically constructed, the topology of the overall network will be further automatically constructed. The difference from the automatic design of the Cell structure is that the search space composed of basic and general operations is evolved into a search space composed of high-quality Cells. This method constructs a single chain structure composed of four stacked Cells as the backbone of the overall network structure. In order to reduce the computational cost during the structure training process and optimize the network performance, the second and fourth Cells are set as Reduce Cells and downsampled, that is, the size of the feature map is halved after the Reduce Cell. When designing the overall structure, the machine can freely select and combine Cells to build a complex and efficient overall network.
[0094] In order to further reduce manual intervention, the overall structure can also automatically design long-span residual connections across multiple cells. In a chain-type backbone structure consisting of four different cell stacks, up to three long-span residual connections can be freely designed to further optimize model performance. Long-span residual connections help transfer information throughout the network and enhance the learning ability of global features. By integrating multiple cells and long-span residual connections, the model can better extract and learn features and improve performance. The small-span residual connection in the cell structure utilizes the information flow inside the cell to help extract local features; the long-span residual connection in the overall structure utilizes the information transfer ability of the overall network to enhance the learning ability of global features. These two residual connection methods complement each other and can promote each other in network search and design, making the overall network more flexible and adaptable, further improving model performance.
[0095] When it comes to automatically designing structures with multiple residual connections, there are significant differences in the contributions of different residual connections to structural performance according to the order in which they are added. In order to more accurately optimize the generation of large-span residual structures, the order in which large-span residual structures are added must be considered in the automatic design of the overall structure. By gradually introducing a certain residual structure and considering the impact of the order in which different residual structures are added on the performance, an embodiment of the present invention calculates the change in the structural performance improvement after each residual structure is added to calculate its Shapley value, which can more accurately evaluate the impact of each connection on the overall performance. This method helps to deeply understand and optimize the construction process of the Cell structure, so that the embodiment of the present invention can clearly determine which residual connections are critical to performance improvement and their optimal position in the structure. The application of ordered residual connection Shapley values to optimize structural design strategies to meet specific performance requirements improves the stability and reliability of the design, thereby significantly improving the success rate of the overall network.
[0096] After the automatic design of the Cell and the overall structure is completed, it is necessary to perform multi-objective optimization on the accuracy and topological complexity to evaluate the comprehensive performance of the structure. This step is intended to ensure that the embodiment of the present invention can screen out structures that perform well in all aspects of performance, thereby achieving a network with high accuracy, few parameters, and simple structure in the automatic model design work. In this way, high-performance neural networks that are suitable for different tasks and data can be automatically designed. This not only eliminates the subjective bias and errors that may be introduced in the traditional design process, but also ensures that the obtained network can operate efficiently.
[0097] In addition, in order to further enhance the performance of the neural network, an automated hyperparameter adjustment method is introduced. Hyperparameters are key parameters that control the learning and performance of the neural network during the training process, such as learning rate, regularization parameters, etc. By automatically adjusting these hyperparameters, the neural network can be better generalized and adapted to different tasks. By automatically designing the Cell structure, the overall network topology and residual connections, and automatically adjusting the hyperparameters, a complete automated neural network design framework can be constructed. This framework can greatly reduce the degree of manual participation and the complexity of the design process, while improving the design efficiency and performance of the neural network.
[0098] S4. Perform image classification processing based on NAS neural network to obtain image classification results.
[0099] Specifically, the embodiments of the present invention are mainly used in image classification tasks. The embodiments of the present invention are verified and developed in the Cifar10 dataset. The Cifar10 dataset is a benchmark database widely used to test machine learning applications and pattern recognition technologies. The CIFAR10 dataset has a total of 60,000 samples, each of which is a 32*32 pixel RGB image (color image), and each RGB image must be divided into 3 channels (R channel, G channel, B channel). These 60,000 samples are divided into 50,000 training samples and 10,000 test samples.
[0100] The CIFAR10 dataset is used for supervised learning training, so each sample must be equipped with a label value (used to distinguish what the sample is). Different categories of objects use different label values. There are 10 categories of objects in CIFAR10, and the label values are distinguished by 0~9. They are airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks.
[0101] The steps of NAS neural network training in the embodiment of the present invention are as follows:
[0102] 1) Data preparation:
[0103] Data preprocessing: The cifar10 images are preprocessed, including resizing, normalizing pixel values, data augmentation (e.g., rotation, flipping, scaling), etc., to increase the diversity of the training set and the robustness of the model.
[0104] 2) Model selection:
[0105] Choosing appropriate neural network structures, such as convolutional neural networks (CNNs), are widely used in image processing tasks because they can effectively capture the spatial structure information in images.
[0106] 3) Model initialization:
[0107] The weights and biases of the neural network are initialized using random initialization.
[0108] 4) Forward propagation:
[0109] The image data is input into the neural network and forward propagated through each layer of the network to generate the prediction results.
[0110] 5) Calculate the loss:
[0111] There is an error between the output of the model and the actual label, and the error is quantified by the loss function. This experiment uses cross entropy loss for classification tasks. The network loss function uses the cross entropy loss function Cross-Entropy Loss. The model training in this experiment is supervised training, and the sample labels have been determined. During training, the results and labels calculated in each round are used to calculate the loss value, and then the loss value is back-propagated.
[0112] 6) Back Propagation:
[0113] Through the back-propagation algorithm, the gradients of the loss with respect to the network parameters are calculated. These gradients are used to update the network parameters to reduce the loss.
[0114] 7) Optimizer update:
[0115] The optimizer uses the stochastic gradient descent SGD optimizer. After back propagation, the optimizer parameters are updated for the next round of learning. The initial learning rate is set to 0.1, and a total of 80 epochs are searched. The learning rate is attenuated by half at the 30th and 60th epochs. After each round of learning, the network outputs the learning accuracy, and the parameters can be further adjusted based on the accuracy.
[0116] 8) Repeat training:
[0117] Repeat steps 4) to 7) until the model converges or the predetermined number of training rounds is reached.
[0118] 9) Model Evaluation:
[0119] Use the validation set to evaluate the performance of the model and check whether overfitting or underfitting occurs. The model can be evaluated by indicators such as accuracy, precision, and recall.
[0120] 10) Testing:
[0121] Use the test set to evaluate the performance of the final model and check its generalization ability on unseen test set data.
[0122] In summary, if Fig.10As shown, the embodiment of the present invention mainly involves two key aspects of the automatic structural design of neural networks: the automatic design of the Cell structure and the automatic design of the overall network structure. The position and number of residual structures have an important impact on the improvement of network performance. However, the existing NAS method based on evolutionary algorithms has uncontrollable problems in the mutation direction, resulting in the possibility that the automatically designed structure contains many residual structures with poor performance, thereby reducing the search efficiency of NAS. To this end, the embodiment of the present invention proposes a NAS method that uses the Shapley value of residual connections to improve the evolutionary algorithm. The method of the embodiment of the present invention uses accuracy and topological complexity to comprehensively evaluate the marginal contribution of network performance and structure, thereby achieving efficient network capabilities with high automatic design accuracy, few parameters and simple structure. The mutation probability parameters in the evolutionary algorithm are adjusted by calculating the Shapley value of the residual connection, thereby guiding the design direction of the automatic evolution of the structure.
[0123] The main differences between the embodiments of the present invention and the prior art are:
[0124] 1) Traditional NAS algorithms still require manual design of the search space and have a low degree of automation.
[0125] Traditional NAS algorithms rely on human intervention to design the Cell structure to construct the search space, which requires humans to embed prior knowledge, which may bring bias and limitations, thereby limiting the exploration of novel and diverse architectures. This process requires sufficient understanding and analysis of the task and data to determine the appropriate search space. A search space that is too large or too small will affect the NAS performance. A search space that is too large will lead to a space explosion problem, that is, due to the limitation of computing resources, only very limited information can be used to guide the search process. The NAS process will become difficult to train and often find suboptimal architectures. A search space that is too small will result in a limited search range and the inability to find the optimal neural network structure, thus affecting the performance of the model. Since the search space is designed by humans, this process requires the introduction of prior knowledge suitable for the task to reduce the search range and simplify the search. Human participation will have subjective factors and introduce human bias, which prevents the model from learning novel structures beyond human current cognition. In essence, these methods transfer the workload of traditional manual design to search space design, so this NAS method is not fully automated.
[0126] The present invention is to further reduce the manual participation in the NAS method, so that users without relevant artificial intelligence expertise can also automatically design the optimal neural network structure for a specific task through the system of the present invention. The present invention uses the most primitive and basic operations (1x1 convolution, 3x3 convolution, 3x3DW convolution, 5x5 convolution, 5x5DW convolution) to allow the machine to automatically design Cells, thereby automatically constructing the search space and the overall structure, thereby avoiding the need for manual participation in the design of the search space in the traditional NAS method.
[0127] 2) The search direction of traditional NAS methods based on evolutionary algorithms is uncontrolled.
[0128] In the process of structure search and design by traditional NAS, it is impossible to determine the contribution of certain structures to performance improvement, and the searched structures are not used to guide the subsequent search direction (that is, traditional NAS does not use Shapley value to evaluate the contribution value of residual structure to guide the search).
[0129] Traditional NAS methods based on evolutionary algorithms have certain uncontrollability in the search direction. In the lack of fine control in structural design, the direction of structural evolution and mutation is blind and uncontrolled, resulting in a large number of structures with poor performance in the searched structures, thereby reducing the efficiency of neural network search. In the process of structural search and design by traditional NAS, it is difficult to clearly understand which structures have a substantial effect on performance improvement, and this method fails to make full use of the structures that have been searched to guide and optimize subsequent search directions. This defect leads to a limitation in the neural network design process, hindering the embodiments of the present invention from fully mining existing information to more effectively build neural network structures with excellent performance. In order to overcome this challenge, the embodiments of the present invention need to explore more intelligent and orderly methods to guide the search of evolutionary algorithms in a more targeted manner, so that they can more efficiently discover and utilize potential performance advantage structures. Such methods will bring a deeper understanding and optimization to neural network design, prompting future NAS research to pay more attention to the effectiveness and interpretability of structures.
[0130] The embodiment of the present invention uses Shapley value to evaluate the performance contribution of the residual structure within the Cell structure and the overall structure to guide the evolution of structural search. By using the Shapley value of the residual connection, the problem that the traditional NAS based on the evolutionary algorithm is relatively blind in the search direction can be improved. After evaluating a certain number of structures, the method adjusts the mutation probability of different residual connections according to the Shapley value to guide the search direction of NAS. By retaining the residual branches that are beneficial to performance improvement and eliminating branches with low contribution or redundancy, this method improves the search efficiency and increases the diversity of the Cell structure, thereby improving the expression and generalization ability of the network, helping to discover higher quality network structures and avoid falling into local optimal solutions.
[0131] 3) Most traditional cell-based NAS methods build networks by linearly stacking cells. This overall structure is manually defined. Moreover, although these cells have residual connections internally, in terms of the overall structure, cells are only connected by linear connections, and there is no cross-cell information channel. This design hinders the transmission and fusion of information between different cells, thereby reducing the performance and generalization ability of the model.
[0132] The artificially defined overall structure is affected by the subjective factors of the designer, and the designed structural form is limited. At the same time, this chain structure may cause gradient disappearance and difficulty in training as the number of network layers increases during the search process.
[0133] In the process of automatic design of cell structure, the present invention can use basic operations to construct residual connections within the cell structure to improve performance. The small-span residual connection in the cell structure utilizes the information flow inside the cell to help extract local features. In the automatic design of the overall structure, a large-span residual structure spanning multiple cells can be automatically designed to further optimize the model performance. The large-span residual connection helps to transfer information throughout the network and enhances the learning ability of global features. By integrating multiple cells and large-span residual connections, the model can better extract and learn features and improve performance. The large-span residual connection in the overall structure utilizes the information transfer ability of the overall network to enhance the learning ability of global features. These two residual connection methods complement each other and can promote each other in network search and design, making the overall network more flexible and adaptable, further improving model performance.
[0134] The main improvements of the embodiments of the present invention are:
[0135] 1) Automatic design method of neural network structure.
[0136] The embodiment of the present invention uses the most basic and common operations (structures) as the raw materials for automatic structural design. The automatic design of neural network structures mainly includes two steps: automatic design of cell structures and automatic design of overall structures. In the automatic design of cell structures, the most basic and common operations are used to automatically design cells through NAS of the improved evolutionary algorithm, and the performance (accuracy and topological complexity) of the cells are evaluated to select 10 excellent cells as efficient search spaces. In the automatic design of the overall structure, the efficient search space automatically constructed in the previous step is used to select excellent cells from it to further search the overall structure.
[0137] 2) The NAS method based on the Shapley value of the residual structure guides the automatic structure design.
[0138] The automatic design of neural network structure mainly includes two steps: automatic design of Cell structure and automatic design of overall structure.
[0139] The automatic design of the cell structure uses an evolutionary algorithm that is improved by the disordered Shapley value of the residual structure to guide the search. Based on the marginal contribution (Shapley value) of a residual structure in the cell structure, the direction of the cell structure evolution is guided by reducing the mutation probability of the residual structure with a higher Shapley value and increasing the mutation probability of the residual structure with a lower Shapley value, thereby improving the efficiency of designing a cell structure with excellent performance.
[0140] The automatic design of the overall structure is guided by the search of an evolutionary algorithm improved by the ordered Shapley value of the large-span residual structure. In order to more accurately optimize the generation of the large-span residual structure, the order in which the large-span residual structure is added to the overall structure is considered in the automatic design of the overall structure. By adding residual connections in order and calculating their Shapley values, the impact of each connection on the overall performance can be more accurately evaluated. The method of the present invention helps to deeply understand and optimize the construction process of the Cell structure, and enables the embodiment of the present invention to clearly determine which residual connections are crucial to performance improvement and their optimal positions in the structure. According to the evaluation of the marginal contribution (Shapley value) of a certain large-span residual structure in the overall structure, the direction of the overall structure evolution is guided by reducing the mutation probability of the residual structure with a higher Shapley value and increasing the mutation probability of the residual structure with a lower Shapley value, thereby improving the efficiency of designing an overall structure with excellent performance.
[0141] 3) Automatically design the small-span residual structure within the cell structure and the large-span residual structure in the overall structure.
[0142] The large-span Cell structure has important advantages and significance in the neural network architecture. Traditional Cell-based NAS methods usually adopt a chain structure, in which there is a lack of direct residual connection between each Cell. In contrast, the method of the embodiment of the present invention can not only automatically design the residual structure within a single Cell, but also introduce a large-span residual connection between multiple Cells in the overall structure.
[0143] First, the introduction of a large-span residual structure can promote more efficient information transmission. In the traditional chain structure, information needs to pass through each cell layer by layer, which may cause the information to pass through a long path in the network, increasing the risk of gradient disappearance or explosion. The large-span residual structure can directly pass information across multiple cells, shortening the path of information transmission and helping to transmit information more stably and quickly.
[0144] Secondly, the large-span residual connection also helps capture a wider range of contextual information. Since information can be transmitted more quickly throughout the network, the model can better utilize information from different parts to more fully understand the input data. This helps to improve the network's perception range and enhance the model's ability to model complex data relationships.
[0145] In addition, the large-span residual structure can also improve the robustness of the network. By introducing connections across multiple cells, the model can more easily learn feature representations with better generalization capabilities, thereby better adapting to unseen data. This is especially important for facing diverse and complex input data in practical applications.
[0146] The main beneficial effects of the embodiments of the present invention are:
[0147] The embodiment of the present invention can automatically design the Cell structure and the overall structure, and transform the original search space composed of basic operations into an advanced search space composed of high-performance Cells, thereby realizing the automatic design of the overall structure and reducing manual participation in NAS.
[0148] In the process of automatic structure design, this method enables the machine to automatically realize more diverse, free and complex network structures. In the automatic design stage of the Cell structure, the system allows the free construction of small-span residual connections inside the Cell. Small-span residual structures usually involve information transfer between adjacent operations, which helps to capture and optimize specific features in a local range. In the automatic design of the overall structure, the embodiment of the present invention can flexibly design large-span residuals spanning multiple Cell structures, successfully solving the problem of poor information flow between multiple Cells in the traditional Cell-based neural network architecture search method. This makes the automatically designed structure have a higher degree of freedom and diversity. The large-span residual structure establishes long-distance information connections between multiple Cells, allowing the network to better understand global information and complex data associations. The coupling of small-span and large-span residual structures exists in the overall structure, thereby enhancing the expression and learning capabilities of the neural network. This innovation effectively improves the design capability of neural network architecture search and enhances the ability to search and obtain structures with better performance.
[0149] In view of the problem that the direction of structural evolution guided by the evolutionary algorithm in the traditional NAS based on the evolutionary algorithm is uncertain, an embodiment of the present invention proposes an evolutionary algorithm NAS method based on the Shapley value of residual connection. This method guides the evolution direction of the NAS search network by evaluating the contribution of each residual connection to the performance improvement. Specifically, the mutation probability is controlled according to the Shapley value of the residual structure. For residual branches with higher Shapley values, it indicates that these branches have a positive impact on performance improvement, and therefore it is worth retaining or increasing the appearance of their structures. The mutation probability of these branches needs to be reduced to ensure that the most effective residual branches are retained. On the contrary, for residual branches with lower Shapley values, it indicates that these branches will hinder performance improvement. The system will increase its mutation probability to reduce the structure containing residual branches that are harmful to performance improvement, thereby increasing the chance of designing a high-performance model structure.
[0150] The embodiment of the present invention adopts an automatic design method of a Cell structure guided by Shapley value of a disordered residual structure. The Shapley value of a disordered residual structure is adopted to pay more attention to the influence of the existence of a certain residual structure on the performance improvement of the Cell structure.
[0151] In addition, the embodiment of the present invention adopts an overall structure automatic design method guided by the Shapley value of the ordered residual structure. The use of the ordered residual structure Shapley value fully considers the influence of the order in which different residual structures are added to the structure on the network performance when there are multiple residual structures. By carefully analyzing the Shapley value of the ordered residual structure, the embodiment of the present invention can more comprehensively understand the influence of the structural combination, thereby optimizing the performance of the overall neural network.
[0152] Further simulation experiments were performed according to the embodiments of the present invention to verify that:
[0153] The embodiment of the present invention uses the Cifar10 dataset to evaluate the performance of the proposed automatic structure design method. The Cifar10 dataset is a benchmark database widely used to test machine learning applications and pattern recognition technology. The dataset contains 60,000 color images from 10 different categories, with each category containing 6,000 images. The resolution of these images is 32x32 pixels, which provides a solid data foundation for neural network training.
[0154] The CIFAR10 dataset has a total of 60,000 samples, each of which is a 32*32 pixel RGB image (color image), and each RGB image must be divided into 3 channels (R channel, G channel, B channel). These 60,000 samples are divided into 50,000 training samples and 10,000 test samples.
[0155] The CIFAR10 dataset is used for supervised learning training, so each sample must be equipped with a label value (to distinguish what the sample is). Different categories of objects use different label values. There are 10 categories of objects in CIFAR10, and the label values are distinguished from 0 to 9. They are airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. The content of the CIFAR10 dataset, such as Fig.11 CIFAR-100 is similar to CIFAR-10, except that it has 100 classes, each containing 600 images. Each class has 500 training images and 100 test images.
[0156] The experimental results of automatic design of the Cell structure of the embodiment of the present invention are further described:
[0157] The Cifar10 dataset is preprocessed by random rotation and random cropping, and then input into the designed framework for automatic Cell structure design. The designed Cell backbone is a chain structure with a depth of L=4 layers, and the number and type of residual connections are freely designed by the system. The system uses basic operations to build Cells, and repeatedly stacks the searched Cells into a network in a 4-layer chain for performance verification, and uses an improved evolutionary algorithm as the design strategy. Mutation generates new Cell structure individuals, and the performance is verified in Cifar10.
[0158] In the process of automatic cell design, the residual connections inside the cell structure often have a small span. In order to better evaluate the value of these connections, the embodiment of the present invention adopts an unordered Shapley value evaluation method. This method can comprehensively consider the contribution of each residual when it exists in the entire cell structure, without considering the order relationship between them. Experiments have found that residual connections with high Shapley values are located at the edge of the cell or connect different modules, which has a great impact on performance. During optimization, these residual connections should be focused on to reduce the mutation probability of residual connections with low Shapley values. Connections with high Shapley values have a key impact on performance. When optimizing the cell structure, the embodiment of the present invention should retain these connections as much as possible.
[0159] Fig.12 The Shapley values of the residual connections of the automatically designed and verified Cell after the search are completed are shown. Among them, the residual connection at the position [3, 4] contributes the most to the performance improvement of the Cell. The final optimal Cell structure encoding is [[0, 0, 1, 3],[[2, 3]],[64, 64, 64, 64]]. This structure has a low structural complexity while maintaining a high accuracy. Its structure is shown in the form of encoding, where the first element [0, 0, 1, 3] represents the operation combination of the Cell's trunk chain structure. The second element [[2, 3]] represents the design result of the number of input and output channels of the residual connection branch in the Cell in the trunk chain structure. The third element [64, 64, 64, 64] represents the number of channels at the output position of each layer of the Cell trunk chain structure.
[0160] During the experiment, the system automatically designed 100 cells before ending the search process. For the automatically designed cells, the embodiment of the present invention comprehensively selects the cells with the best comprehensive performance according to the topological complexity and accuracy as the high-quality search space, which provides a search basis for the next step of automatically designing the overall network structure. Table 1 shows the optimal cells after comprehensive performance screening.
[0161] Table 1 Cell structure with optimal comprehensive performance obtained by automatic design
[0162]
[0163] Furthermore, after evaluating the performance of the automatically designed cells, the framework will select 10 cells with the best overall performance as the search space for the overall structural design. The backbone structure of the overall network is composed of 4 layers of cells connected in a chain. The framework can automatically design the type of cells in each layer and the connection method of the residual branches. At the same time, the system uses the evolutionary algorithm search strategy improved by the Shapley value of the residual connection to guide the direction of automatic design, which helps to obtain a network structure that is effective for performance improvement.
[0164] Table 2 shows the 10 best cell structures and their corresponding accuracy and parameter counts after 30 rounds of searching on the Cifar10 dataset. The best structure achieved an accuracy of 86% and only had 0.51M parameters. This result demonstrates the effectiveness of the framework in automatically designing cells and optimizing network structures.
[0165] Table 2 The overall network with the best overall structure performance obtained by automatic design
[0166] structure Accuracy Complex topology (the smaller the better) Comprehensive evaluation (0.6) Overall structure residual: [[4, 5], [4, 5],[1, 4]] Residual structure cell: [[[4, 3,3, 3], [[2, 4]],[128, 64, 96,256]], [[0, 4,1, 1], [[0, 3]],[64, 128, 32,64], 0.01], [[4,3, 0, 4], [[2,4], [2, 4]],[160, 96, 96,64]]] Backbone structure cell: [[[4, 3, 0,4], [[2, 4], [2,4]], [160, 96,96, 64]], [[0,3, 1, 0], [[1,3]], [32, 32,32, 128]], [[4,3, 3, 3], [[2,4]], [128, 64,96, 256]], [[3,3, 4, 3], [[2,3]], [32, 96,128, 100]]] 0.86 285867.0 0.90013 , 32, 32, 100]]]Cell of the backbone structure: [[[4, 3, 0, 4], [[2, 4], [2, 4]], [160,96, 96, 64]],[[4, 1, 0, 1],[[0, 2], [2, 4],[0, 3]], [96,64, 64, 16]],[[0, 3, 1, 0],[[1, 3]], [32,32, 32, 100]]]Cell of the backbone structure: [[[4, 3, 0, 4],[[2, 4], [2,4]], [160, 96,96, 64]], [[4,3, 3, 3], [[2,4]], [128, 64,96, 128]], [[3,3, 4, 3], [[2,3]], [32, 96,128, 256]], [[0,3, 1, 0], [[1,3]], [32, 32,32, 100]]] 0.8523 276331.0 0.89604 Overall structure residual: [[1, 3], [3, 5],[2, 3]] Residual structure Cell: [[[0, 3,1, 0], [[1, 3]],[32, 32, 32,100]], [[3, 4,2, 0], [[3, 4]],[128, 64, 32,32]], [[0, 3, 1,0], [[1, 3]],[32, 32, 32,100]]] Backbone structure Cell: [[[0, 4, 1,1], [[0, 3]],[64, 128, 32,64]], [[3, 3, 4,3], [[2, 3]],[32, 96, 128,128]], [[0, 0,1, 3], [[2, 3]],[64, 64, 64,256]], [[0, 3,1, 0], [[1, 3]],[32, 32, 32,100]]] 0.8449 263071.0 0.89234
[0167] In the process of structural design, the evolutionary algorithm is used as a search strategy, and the way to implement structural search is in the form of structural coding. For example, when searching a Cell structure, the embodiment of the present invention uses "0" to represent 1x1 convolution, "1" to represent 3x3 convolution, "2" to represent 5x5 convolution, "3" to represent 3x3 depth-separable convolution (DW), and "4" to represent 5x5 depth-separable convolution (DW). During the evolutionary algorithm search process, a structural code such as [[4, 3, 0, 4, 3], [[1, 3]], [160, 96, 96,64]] will be output to represent the structure of a Cell. The first element of the encoding array [4, 3, 0, 4,3] represents the operation code in the Cell structure. The second element [[1, 3]] represents the input and output positions of the residual in the Cell, such as the input position is connected to the output position of the first layer of the backbone structure, and the output position is connected to the output position of the third layer of the backbone structure. The third element [160, 96, 96, 64] represents the number of output channels of each layer of the backbone structure.
[0168] Furthermore, the embodiment of the present invention compares in detail the automatic structural design performance with Google's AutoML-Zero on the same Cifar10 binary classification subset. The experimental results are shown in Table 3, which clearly show the performance of the neural network framework of the embodiment of the present invention and AutoML-Zero on the common Cifar10 binary classification subset. AutoML-Zero uses a variety of training and optimization methods, while the method of the embodiment of the present invention only uses a fixed SGD optimizer and weight decay means for training. However, the framework proposed in the embodiment of the present invention adopts an improved evolutionary algorithm to jointly guide the search design from the two aspects of topological complexity and recognition accuracy to obtain an efficient and lightweight Cell. The Cell obtained by this search design can enable the overall structure to be automatically designed with an accuracy of 98.5%. In contrast, the highest accuracy of AutoML-Zero is 84.06%. Therefore, the framework proposed in the embodiment of the present invention can improve the accuracy by 14.44%, fully demonstrating its superior performance.
[0169] Table 3 Performance comparison between the proposed method and AutoML-Zero on the Cifar10 binary classification subset
[0170] structure Accuracy (%) Complex topology (the smaller the better) Comprehensive evaluation (0.6) AutoML-Zero 84.06 354711 0.84 Ours 98.50 209635 0.97
[0171] As shown in Table 4, after the algorithm of the embodiment of the present invention was experimented and evaluated on multiple public data sets, the results show that the algorithm proposed in the embodiment of the present invention can automatically design a lightweight and efficient neural network structure on different tasks and data sets, achieving performance comparable to or exceeding that of artificial expert design or existing NAS methods, while greatly reducing the complexity of the structure and the consumption of computing resources. The embodiment of the present invention also applies the algorithm to the target recognition task in the industrial field, proving that the algorithm of the embodiment of the present invention has good generalization ability and adaptability.
[0172] Reference Figure 2 , a NAS neural network design system based on Shapley value, comprising:
[0173] A construction module, used for performing structural coding processing based on the Cell unit to construct a Cell structure, wherein the Cell structure includes a chain backbone structure and a residual structure;
[0174] The first optimization module is used to perform residual optimization processing on the Cell structure by using a performance contribution quantification method based on the disordered Shapley value of the residual structure to obtain an optimal Cell structure;
[0175] The second optimization module is used to stack the optimal Cell structure and optimize it through the ordered Shapley value quantization method to construct a NAS neural network;
[0176] The classification module is used to perform image classification processing based on the NAS neural network to obtain image classification results.
[0177] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0178] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A NAS neural network design method based on Shapley value, characterized in that: The following steps are involved: Based on the Cell unit, structural coding processing is performed to construct a Cell structure, wherein the Cell structure includes a chain backbone structure and a residual structure, including: Obtaining a structural unit raw material, wherein the structural unit raw material includes a 1x1 convolution layer, a 3x3 convolution layer, a 5x5 convolution layer, a 3x3 depth-separable convolution layer, and a 5x5 depth-separable convolution layer; Based on the raw materials of the structural unit, a structural coding representation is performed to output a preliminary Cell structure, wherein the preliminary Cell structure is a backbone single-chain structure with four layers of basic operations; The structure code includes a first element, a second element and a third element, wherein the first element represents the operation code in the Cell structure; the second element represents the input position of the residual structure and the output position of the residual structure in the Cell structure, the input position of the residual structure is connected to the first layer of basic operation of the backbone single-chain structure, and the output position of the residual structure is connected to the third layer of basic operation of the backbone single-chain structure; the third element represents the number of output channels of each layer of the backbone single-chain structure; Perform residual structure insertion processing on the preliminary Cell structure to construct the Cell structure; Obtain the mutation probability of the Cell structure, the mutation parameters of the Cell structure, the accuracy of the Cell structure, and the topological complexity of the Cell structure; Perform Shapley value calculation on the residual structure in the Cell structure to obtain the Shapley value of the Cell residual structure; The expression for calculating the Shapley value is specifically as follows: ; In the above formula, The Shapley value representing the Cell residual structure, represents the sum of all residual structures, Represents the Cell residual structure weight, Represents the Cell residual structure branch, represents the value function, Represents the Cell structure, express A subset of any residual connections in , excluding the residual connection currently being evaluated , Represents the set of all residual connections in all Cell structures; According to the Shapley value of the Cell residual structure, the mutation probability of the Cell structure is adjusted, and the adjusted Cell structure is output; Performing individual mutation processing on the adjusted Cell structure through the mutation parameters of the Cell structure to obtain the Cell structure after mutation and update; Obtain the accuracy of the Cell structure after mutation and update and the topological complexity of the Cell structure after mutation and update; The performance contribution quantification method based on the disordered Shapley value of the residual structure is cyclically used to perform residual optimization processing steps on the Cell structure until the accuracy of the Cell structure after mutation and the topological structure complexity of the Cell structure after mutation and update are minimum, and the optimal Cell structure is output; The optimal Cell structure is stacked to construct a Cell structure population; The Cell structure population is formed by stacking four optimal Cell structures in a single chain, wherein the second optimal Cell structure and the fourth optimal Cell structure are both set as Reduce Cell and subjected to downsampling operation processing; Generate a large-span residual connection structure for the Cell structure population; The large-span residual connection structure is optimized by the ordered Shapley value quantization method, and the optimal large-span residual connection structure is output; Insert the optimal large-span residual connection structure into the Cell structure population to construct a NAS neural network; Image classification is performed based on the NAS neural network to obtain image classification results.
2. A NAS neural network design system based on Shapley value, characterized in that: Includes the following modules: The construction module is used to perform structural coding processing based on the Cell unit to construct a Cell structure, wherein the Cell structure includes a chain backbone structure and a residual structure, including: Obtaining a structural unit raw material, wherein the structural unit raw material includes a 1x1 convolution layer, a 3x3 convolution layer, a 5x5 convolution layer, a 3x3 depth-separable convolution layer, and a 5x5 depth-separable convolution layer; Based on the raw materials of the structural unit, a structural coding representation is performed to output a preliminary Cell structure, wherein the preliminary Cell structure is a backbone single-chain structure with four layers of basic operations; The structure code includes a first element, a second element and a third element, wherein the first element represents the operation code in the Cell structure; the second element represents the input position of the residual structure and the output position of the residual structure in the Cell structure, the input position of the residual structure is connected to the first layer of basic operation of the backbone single-chain structure, and the output position of the residual structure is connected to the third layer of basic operation of the backbone single-chain structure; the third element represents the number of output channels of each layer of the backbone single-chain structure; Perform residual structure insertion processing on the preliminary Cell structure to construct the Cell structure; The first optimization module is used to obtain the mutation probability of the Cell structure, the mutation parameter of the Cell structure, the accuracy of the Cell structure and the topological complexity of the Cell structure; Perform Shapley value calculation on the residual structure in the Cell structure to obtain the Shapley value of the Cell residual structure; The expression for calculating the Shapley value is specifically as follows: ; In the above formula, The Shapley value representing the Cell residual structure, represents the sum of all residual structures, Represents the Cell residual structure weight, Represents the Cell residual structure branch, represents the value function, Represents the Cell structure, express A subset of any residual connections in , excluding the residual connection currently being evaluated , Represents the set of all residual connections in all Cell structures; According to the Shapley value of the Cell residual structure, the mutation probability of the Cell structure is adjusted, and the adjusted Cell structure is output; Performing individual mutation processing on the adjusted Cell structure through the mutation parameters of the Cell structure to obtain the Cell structure after mutation and update; Obtain the accuracy of the Cell structure after mutation and update and the topological complexity of the Cell structure after mutation and update; The performance contribution quantification method based on the disordered Shapley value of the residual structure is cyclically used to perform residual optimization processing steps on the Cell structure until the accuracy of the Cell structure after mutation and the topological structure complexity of the Cell structure after mutation and update are minimum, and the optimal Cell structure is output; The second optimization module is used to stack the optimal Cell structure and construct a Cell structure population; The Cell structure population is formed by stacking four optimal Cell structures in a single chain, wherein the second optimal Cell structure and the fourth optimal Cell structure are both set as Reduce Cell and subjected to downsampling operation processing; Generate a large-span residual connection structure for the Cell structure population; The large-span residual connection structure is optimized by the ordered Shapley value quantization method, and the optimal large-span residual connection structure is output; Insert the optimal large-span residual connection structure into the Cell structure population to construct a NAS neural network; The classification module is used to perform image classification processing based on the NAS neural network to obtain image classification results.
