A hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search

Abstract

A hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search, comprising the following steps: initializing architecture information; generating a training set and a validation set corresponding to each tree chromosome; constructing a convolutional neural network; agent model assisted multi-objective evolutionary search: using a multi-objective selection algorithm to screen a new elite population from the combined parent population and elite population; performing a crossover operation on the new elite population with a probability to generate a crossover population; performing a mutation operation on the individuals in the crossover population to form a mutation population; using the mutation population as the next generation population; iterating the population to select the optimal convolutional neural network; calculating the classification accuracy of the convolutional neural network on all samples to be classified in the hyperspectral image, and outputting the corresponding classification accuracy as the result; the present application has the advantages of reducing network model complexity, reducing network model calculation time and improving classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image classification, and in particular relates to a hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search. Background Art

[0002] In recent years, hyperspectral images have been widely used in geological exploration, agriculture and other fields. With the large-scale production of hyperspectral images, the industry has posed new challenges to the task of land feature classification.

[0003] Patent application document with publication number CN109376804A proposes a hyperspectral remote sensing image classification method based on attention mechanism and convolutional neural network. This method uses artificially constructed convolutional neural network to solve the hyperspectral image classification problem. The steps of this method are as follows: (1) using principal component analysis to divide and sample the original hyperspectral image; (2) performing 3D convolution and pooling operations on the sample blocks to obtain feature maps; (3) passing the feature maps through spectral and spatial attention modules, and performing 3D convolution and pooling operations again; (4) inputting the final feature maps into the classifier to complete the hyperspectral image classification. This method improves the accuracy of hyperspectral image classification. However, this scheme uses artificially designed neural networks to achieve classification, which has high requirements on the designer's model design experience and hyperspectral related knowledge. For different hyperspectral images, the workload of artificially designing neural network architecture is too large, and artificially designed neural network architecture is prone to structural redundancy and human errors, affecting classification efficiency and classification accuracy.

[0004] The patent application document with publication number CN114882294A proposes a multi-objective evolutionary convolutional neural network for hyperspectral image classification. The method uses a tree-shaped multi-branch super network structure as the basic structure of the network block to realize the extraction and fusion of multi-scale features. At the same time, the size of the input image block and the spectral dimension of the input image block are encoded, and the classification accuracy and network complexity are simultaneously used as optimization targets to realize the search for a convolutional neural network with high classification accuracy and low network structure complexity. The specific implementation steps include: (1) initializing the architecture information and encoding the convolutional neural network to be constructed; (2) generating a data set of gene individuals and using principal component analysis (PCA) to reduce the input spectral dimension; (3) constructing a convolutional neural network corresponding to each gene individual in the population and training it; (4) calculating the two fitness of the convolutional neural network and performing crossover and mutation on the individuals in the population; (5) taking the population with the maximum number of iterations as the optimal population, and taking the convolutional neural network corresponding to the gene individual with the highest classification accuracy in the optimal population as the optimal convolutional neural network and performing hyperspectral image classification. However, this method requires a large amount of network model training and evaluation during the multi-objective evolution process, which leads to huge consumption of computing resources and low search efficiency.

[0005] The patent application document with application publication number CN 113780146 A proposes a lightweight neural architecture search technology for hyperspectral image classification. The technology constructs a super network composed of modular lightweight operations and optimizes it using a two-layer gradient optimization method. At the same time, it uses a sub-network weight sharing mechanism to improve the model search efficiency. The specific implementation steps include: (1) defining a unit-based search space, constructing a super network by stacking multiple repeated units, and setting a series of candidate operations; (2) training the constructed super network using a selected training sample set to obtain a lightweight deep neural network architecture suitable for hyperspectral image classification; (3) training the obtained lightweight deep neural network architecture using the training set and verifying it using the test set. Once the predicted label of the test sample meets the predetermined condition, the hyperspectral remote sensing image to be classified is input into the trained lightweight deep neural network architecture to obtain the classification result. This technology effectively solves the limitations of manually designed convolutional neural networks and further improves the classification accuracy. However, the gradient-based neural network structure search method used in this existing technology requires large computing resources and is difficult to apply in practical problems. In addition, this technology cannot search the size and spectral dimension of the model's input image blocks, which will affect the model's classification accuracy to a certain extent.

[0006] In their paper "Particle Swarm Optimization Based Deep Learning Architecture Search for Hyperspectral Image Classification" (IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium, 2020, pp. 509-512), Chaochao Zhang et al. proposed a hyperspectral image classification method based on deep learning architecture search using particle swarm optimization. This method uses a particle swarm optimization algorithm to implement a unit-based CNN architecture search method. The steps of this method are: (1) establishing a unit-based search space framework; (2) using a direct encoding method to map the architecture to particles; and (3) using the particle swarm algorithm to search for the optimal deep learning architecture in the particle swarm. This method has shown excellent performance on two commonly used hyperspectral image datasets. This existing technology is a hyperspectral image classification method based on deep learning architecture search using particle swarm optimization. This method can only extract features at a single scale, and the model's feature extraction capability is limited, which directly affects its classification accuracy. Although this method reduces search time, it has little advantage on complex data sets. During the search process, a large number of model evaluations consume huge computing resources, resulting in low search efficiency. Summary of the Invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search. By adopting classification accuracy and floating-point computational complexity as environmental selection targets at the same time, a network model that meets high classification accuracy and high computational efficiency can be automatically designed, thereby reducing the complexity of the network model and reducing the network model calculation time; through an agent-assisted scheme of a nonlinear convolutional neural network, the tree chromosome encoding is converted into graph representation data and embedded in the GIN agent model, thereby improving classification accuracy and reducing evolutionary optimization time; by using the tree subnet as the basic unit of the architecture search space, the feature extraction capability and multi-scale feature fusion capability can be enhanced; by optimizing the convolutional neural network under multi-objective indicators through an evolutionary algorithm, the problems of high cost and structural redundancy of manually designed convolutional neural networks can be solved, classification accuracy can be improved, and network structure complexity can be reduced; by designing crossover and mutation operators and multi-objective environment selection based on tree data structures, the network depth and chromosome genes can be adaptively adjusted, the algorithm search capability can be enhanced, and the classification efficiency can be improved.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search, comprising the following steps:

[0010] S1. Initialize neural architecture information:

[0011] A tree-shaped multi-branch supernet is used as the basic unit of the search space. A tree-based data structure encoding strategy is used to encode the proxy model parameters and structure into a tree chromosome. Three tree subnets are set in the initial search space, forming the high-level network architecture of the search.

[0012] S2. Generate the training set and validation set corresponding to each tree chromosome:

[0013] The principal component analysis (PCA) method is used to reduce the dimensionality of the input spectrum of the hyperspectral image to be classified and denoise it. The ground feature label of the central pixel of each image block in the hyperspectral image after dimensionality reduction and denoising is used as the input data label. All image blocks are divided into training and validation sets according to a preset ratio.

[0014] S3. Construct a convolutional neural network based on the tree chromosome generated in step S1:

[0015] Traverse all nodes of the first branch of the tree encoding, generate a convolutional layer for each node, connect the convolutional layers to generate a branch network, connect all branch subnetworks in parallel, build the first tree network block, aggregate the outputs of all branch subnetworks as the output of the first tree network block; decode other attributes contained in the network from the root node features, and complete the integration of the tree network block based on these attribute encodings; build all tree network blocks, connect the input and output of all tree network blocks, and finally obtain the convolutional neural network corresponding to the tree chromosome;

[0016] S4. Surrogate model assisted multi-objective evolutionary search:

[0017] The chromosome population P encoded by the random initialization step S1 n , initialize the maximum number of iterations, population size, the accuracy tau of the proxy model on the proxy database R and the elite population P n ′, if the conditional accuracy tau < 0.9 is met, use the construction method of the convolutional neural network in step S3 to construct the population P n The corresponding convolutional neural network uses the training set and validation set divided in step S2 to train the population P n The corresponding convolutional neural network is trained and evaluated, and the classification accuracy is obtained. The data set is added to the proxy database R, and the proxy database R is divided into proxy training sets in proportion. and proxy test set Using a Proxy Training Set Train the GIN proxy model on the proxy test set The upper test accuracy tau, if the condition is not met, the GIN proxy model is used to predict the population P n The fitness of {y1,y2,..,y N}; Calculate the population P n The floating point operation complexity fitness {q1,q2,..,q N}; Combine parent population and elite population;

[0018] S5. Using a multi-objective selection algorithm to select a new elite population for the parent population and the elite population combined in step S4;

[0019] S6, performing a crossover operation on the new elite population in step S5 according to probability to generate a crossover population;

[0020] S7, performing a mutation operation on the individuals in the crossover population obtained in step S6, and forming a mutation population from all the mutant offspring;

[0021] S8, using the mutant population obtained in step S7 as the next generation population;

[0022] S9, iterate the population obtained in step S8. If the current number of iterations is equal to the maximum number of iterations, take the current iterated population as the optimal population, and perform real training evaluation on the optimal population. Select the convolutional neural network corresponding to the individual with the highest classification accuracy as the optimal convolutional neural network;

[0023] S10: Input the hyperspectral image to be classified into the trained optimal convolutional neural network selected in step S9 to obtain the final classification result, and calculate the classification accuracy of the convolutional neural network for all samples to be classified in the hyperspectral image, and output the corresponding classification accuracy as the result.

[0024] Specifically, the hybrid encoding method in step S1 is:

[0025] First, the number of branches of each tree subnetwork is initialized. In the initial search space, each tree subnetwork has a maximum of three branches. Then, the nodes of each branch are initialized. Each branch has a maximum of three nodes at initialization. The tree chromosome includes the root node and other nodes. The characteristics of the root node include: the number of subnetwork output channels, skip connection flags, downsampling operations, and attention model information; the characteristics of other nodes include: the number of convolutional layer output channels, convolution operation type, and convolution kernel size.

[0026] Specifically, the specific steps of generating the training set and the validation set in step S2 are:

[0027] S201, using principal component analysis (PCA) to reduce the dimensionality of the input spectrum of the hyperspectral image to be classified and remove noise, so that its spectral dimension is equal to the input spectrum dimension corresponding to the gene individual;

[0028] S202. Taking each labeled pixel in the hyperspectral image after dimensionality reduction and denoising in step S201 as the center, intercept a rectangular block of the pixel. The width and height of the rectangular block are equal to the image block size corresponding to the genetic individual, and the label of the central pixel is used as the label of the rectangular block. All image blocks are divided into a training set and a validation set according to a preset ratio. The accuracy of the model classification result on the validation set is calculated as follows:

[0029]

[0030] Among them, A c The network model corresponding to the tree chromosome encoding c, |D Test | represents the size of the validation set, X i is the i-th image block data in the verification dataset, y i is the object category label of the i-th image block, F(·) is the prediction result output by the network model, and the function equal(·) is used to compare the difference between the prediction result and the true label. When the predicted value is the same as the true label, its value is equal to 1.

[0031] Specifically, the step S4 is as follows:

[0032] S401, randomly initialize the chromosome population P encoded in step S1 n , initialize the maximum number of iterations M, the population size N, the accuracy of the proxy model on the proxy database R tau = 0 and the elite population

[0033] S402: If the condition accuracy tau < 0.9 is met, use the convolutional neural network construction method in step S3 to construct the population P n The corresponding convolutional neural network uses the training set and validation set divided in step S202 to train the population P n The corresponding convolutional neural network is trained and evaluated, and the classification accuracy is obtained and then the dataset is added to the proxy database R. The tree chromosome encoding is converted into graph data that can be used by GIN. The tree chromosome encoding is obtained according to the node generation sequence number to obtain the graph node representation. The adjacency matrix and node features are generated according to the tree branch connection, root node attributes and other node attributes. For the attribute information of all nodes, binary encoding is used to convert it into node features to enrich the information embedded in the graph representation. The proxy database R is divided into proxy training sets in proportion. and proxy test set Using a Proxy Training Set Train the GIN proxy model on the proxy test set The test accuracy is tau. The GIN-based proxy model adopts a dual-end input and weight-sharing structure. In each batch of training data, the input model is combined with two pairs to learn the ranking information. In training, SME-TCNN uses a loss function that emphasizes pairwise ranking learning.

[0034]

[0035] γ(θ)=log(1+e -θ )

[0036] Where w is a learnable parameter, n is the batch size, c is the tree chromosome, y is the true classification accuracy corresponding to the chromosome, g(·) is the prediction accuracy of the proxy model output, and γ(·) is the logistic regression function. λ is used as the proportional parameter of the ranking loss and the MSE loss. Based on the output of GIN, the prediction accuracy of GIN for the fitness ranking of the current batch of data is calculated as follows:

[0037]

[0038] Among them, the function rank(·) indicates whether the ranking relationship predicted by the proxy model and the actual ranking relationship of a pair of chromosome data match. It is 1 when the relationship matches and 0 when the relationship contradicts.

[0039] S403. If the conditions are not met, use the GIN proxy model to predict the population P. n The fitness of {y1,y2,..,y N The input process of the GIN proxy model is the same as step S402;

[0040] S404. Calculate population P n The floating point operation complexity fitness {q1,q2,..,q N}, combined parent population and elite population; population P n The calculation formula of the floating-point operation complexity fitness is:

[0041]

[0042] Among them, flops c is the number of floating-point operations calculated after a forward inference of the real neural network model corresponding to the tree chromosome c, flops max and flops min They represent the maximum and minimum number of floating-point operations in all chromosomes of the current population, respectively. H and W are the width and height of the input data, and K, C inand C out are the convolution kernel size of the convolution layer, the dimensions of the input and output channels, and I and U represent the dimensions of the input and output of the fully connected layer.

[0043] Specifically, in step S6, the crossover operation performed on the population is as follows:

[0044] S601. Randomly select two chromosome individuals corresponding to the convolutional neural network to be optimized from the parent population of the current iteration;

[0045] S602: Perform a crossover operation on the two selected chromosome individuals according to the crossover probability cp. Generate a random number that satisfies a uniform distribution of 0-1 for each pair of selected chromosome individuals. If the random number is less than cp, 0≤cp≤1, for the first tree of the parent generation, randomly select and exchange branches from the two parent chromosomes. For all branches in the same tree subnetwork, the number of output channels of the convolutional layer of the leaf node needs to be reset according to the attributes of the root node.

[0046] S603. Determine whether all chromosome individuals have been selected in the parent population after the current iteration. If so, form a crossover population with all chromosome individuals obtained through the above crossover operation in the parent population after the current iteration and then execute step S7. Otherwise, execute step S601.

[0047] Specifically, in step S7, the mutation operation performed on the population is as follows:

[0048] S701, performing a mutation operation on the chromosome individuals in the crossover population in step S603 according to probability; generating a random number that satisfies a uniform distribution of 0-1 for each selected chromosome individual, and performing a mutation operation on the chromosome if the random number is less than mp, 0≤mp≤1;

[0049] S702: All chromosome individuals obtained through the above mutation operation in the crossover population of step S603 are combined into a mutation population.

[0050] In step S3, all nodes of the first branch of the tree encoding are traversed, and a convolution layer is generated for each node. The convolution operation types include 3D separable convolution and ordinary 3D convolution. The convolution kernel sizes include three sizes: {3×3, 5×5, 7×7}. The value range of the number of output channels of the convolution layer is {16, 32, 48, 64}.

[0051] The aggregation method in step S3 includes: summation, weighted summation or spectral dimension splicing.

[0052] In the root node feature decoded in step S3, the number of subnetwork output channels refers to the convolution output of all leaf nodes of the current subtree, which is used to aggregate the results of all branches, and the value range is {16, 32, 48, 64}.

[0053] The mutation operation in step S701 includes any one of the following (1)-(5):

[0054] (1) Deleting randomly selected branches. If this results in a tree without any branches, a new branch is generated by randomly selecting new nodes from the search space.

[0055] (2) Generate a new branch containing three nodes, where the nodes are randomly selected from the search space;

[0056] (3) A new node is added to the branch of the tree, and the convolution kernel size and the number of output channels of the new node are randomly selected from the search space;

[0057] (4) Delete a node from a branch of the tree;

[0058] (5) The value of the skip connection flag is inverted and a skip connection is added.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] (1) Currently, most hyperspectral image classification models are manually designed, which requires the designer to have both model design experience and hyperspectral knowledge. In addition, the errors caused by manual design are not conducive to improving the classification accuracy of the model. To address this problem, the present invention designs a neural network structure search method that automatically designs a neural network structure on hyperspectral images by encoding, evolving, and decoding the network in a defined search space, thereby reducing the requirements and errors of manual design.

[0061] (2) The existing technology only considers the classification accuracy of the model. As the industry's requirements for hyperspectral image classification models become increasingly higher, the classification efficiency of the model is receiving more and more attention. The present invention designs a multi-objective neural network structure search method, which uses classification accuracy and floating-point calculation complexity as environmental selection targets at the same time, so that individuals with both fitness levels are retained, and automatically designs a network model that meets both high classification accuracy and high calculation efficiency. It avoids the human errors caused by manually constructing convolutional neural networks, solves the problem of excessive complexity of the network model and excessive demand for computing resources caused by single-target search, and simplifies the complexity of the network model as much as possible while ensuring the high classification accuracy of the model, which has the advantages of reducing the complexity of the network model and reducing the calculation time of the network model.

[0062] (3) In the neural network structure search, it is necessary to evaluate the performance of a large number of network models, and the computing resources consumed are very huge. As the industry's requirements for hyperspectral image classification models are getting higher and higher, the search efficiency of the model is receiving more and more attention. The present invention designs an agent-assisted neural network structure search method, which converts the tree chromosome encoding into graph representation data and embeds it into the GIN agent model through the agent-assisted scheme of the nonlinear convolutional neural network, and fits the classification accuracy labels of the chromosome corresponding network based on the pairwise sorting idea. Under the premise of ensuring the high classification accuracy of the model, the model evaluation is reduced as much as possible, and the model performance is directly predicted. Finally, high classification accuracy is achieved while significantly reducing the calculation time of evolutionary optimization, which has the advantages of reducing costs and saving computing resources.

[0063] (4) Most existing models use a sequential serial structure, which lacks the ability to extract multi-scale features of hyperspectral images, thus affecting the classification accuracy of the model. The present invention uses a tree-shaped multi-branch network structure to extract multi-scale features from hyperspectral images, and uses a tree subnet as the basic unit of the architecture search space. Each tree subnet contains multiple branches, and each branch consists of multiple nodes, which are encoded as convolutional layers of a convolutional neural network. The tree network has a strong ability to extract multi-branch features. The scheme of adding attention weights to the output of each branch helps to fuse multi-scale features. The features of each branch are fused at multiple scales, thereby improving the feature extraction capability of the model.

[0064] (5) Existing neural network structure search methods for hyperspectral image classification are unable to search the input resolution of the model. The input resolution determines how much effective information the input image provides to the model. An unsuitable input resolution cannot provide the required amount of information for the model, resulting in low classification accuracy. The present invention designs a neural network structure search method based on evolutionary computation, which incorporates the input resolution in discrete space into the search space and designs crossover and mutation operators. This allows individuals to obtain neural network models of different depths through crossover and mutation operations, adaptively adjusts the network depth, and improves the network search capability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a schematic diagram of the chromosome decoding network of the present invention.

[0066] Figure 2 This is a diagram of the tree-coded graph representation embedding and GIN proxy model training of the present invention.

[0067] Figure 3 Schematic diagram of the chromosome crossover operator of the present invention.

[0068] Figure 4 is a schematic diagram of the chromosome mutation operator of the present invention, wherein, Figure 4(a) is the parent tree encoding graph, Figure 4 (b) is the deleted branch graph, Figure 4 (c) is to add a branch diagram, Figure 4 (d) is to add a node graph, Figure 4 (e) is the deleted node graph, Figure 4 (f) is the graph with added skip connections.

[0069] Figure 5 is a diagram showing the classification results of the Indian Pines hyperspectral image by the present invention and the prior art, wherein Figure 5(a) is a true annotation diagram, Figure 5(b) is a diagram showing the classification results of the present invention, Figure 5(c) is a diagram showing the classification results of AE-CNN, and Figure 5(d) is a diagram showing the classification results of 3-D-ANAS.

[0070] Figure 6 shows the classification results of the PaviaU hyperspectral image by the present invention and the prior art, where Figure 6(a) is the true annotation image, Figure 6(b) is the classification result image of the present invention, Figure 6(c) is the classification result image of AE-CNN, and Figure 6(d) is the classification result image of 3-D-ANAS.

[0071] Figure 7 shows the classification results of the Houston 2013 hyperspectral image by the present invention and the prior art, wherein Figure 7(a) is the true annotation image, Figure 7(b) is the classification result image of the present invention, Figure 7(c) is the classification result image of AE-CNN, and Figure 7(d) is the classification result image of 3-D-ANAS. DETAILED DESCRIPTION

[0072] The technical solution adopted by the present invention will be further described below with reference to the accompanying drawings.

[0073] A hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search, comprising the following steps:

[0074] like Figure 1 As shown, S1, initialize neural architecture information:

[0075] The present invention adopts a tree-shaped multi-branch supernet as the basic unit of the search space, and uses a tree data structure-based encoding strategy to hybrid encode the proxy model parameters and structure into tree-shaped chromosomes. Three tree subnets are set in the initial search space. The three tree subnets constitute the high-level network architecture of the search, and chromosomes are randomly initialized from the search space to form the initial population.

[0076] Specifically, the hybrid encoding method in step S1 is:

[0077] First, the number of branches of each tree subnetwork is initialized. In the initial search space, each tree subnetwork has at most three branches. Then, the nodes of each branch are initialized. Each branch has at most three nodes at the time of initialization. The tree chromosome includes the root node and other nodes. The root node and other nodes of the tree chromosome contain different features. The features of the root node include: the number of subnetwork output channels, the jump connection flag, the downsampling operation and the attention model information. The features of other nodes include: the number of convolutional layer output channels, the convolution operation type and the convolution kernel size. The depth and width of the tree encoding corresponding model can be adaptively adjusted according to the tree depth and the number of tree branches.

[0078] S2. Generate the training set and validation set corresponding to each tree chromosome:

[0079] The principal component analysis (PCA) method is used to reduce the dimensionality of the input spectrum of the hyperspectral image to be classified and denoise it. The ground feature label of the central pixel of each image block in the hyperspectral image after dimensionality reduction and denoising is used as the input data label, and all image blocks are divided into training set and validation set according to a preset ratio.

[0080] Specifically, the steps for generating the training set and validation set in step S2 are as follows:

[0081] S201, using principal component analysis (PCA) to reduce the dimensionality of the input spectrum of the hyperspectral image to be classified and remove noise, so that its spectral dimension is equal to the input spectrum dimension corresponding to the gene individual;

[0082] S202. Taking each labeled pixel in the hyperspectral image after dimensionality reduction and denoising in step S201 as the center, intercept a rectangular block of the pixel. The width and height of the rectangular block are equal to the image block size corresponding to the genetic individual, and the label of the central pixel is used as the label of the rectangular block. All image blocks are divided into a training set and a validation set according to a preset ratio. The accuracy of the model classification result on the validation set is calculated as follows:

[0083]

[0084] Among them, A c The network model corresponding to the tree chromosome encoding c, |D Test | represents the size of the validation set, X i is the i-th image block data in the verification dataset, y i is the object category label of the i-th image block, F(·) is the prediction result output by the network model, and the function equal(·) is used to compare the difference between the prediction result and the true label. When the predicted value is the same as the true label, its value is equal to 1.

[0085] S3. Construct a convolutional neural network based on the tree chromosome generated in step S1:

[0086] Traverse all nodes of the first branch of the tree code, generate a convolutional layer for each node, connect the convolutional layers to generate a branch network, and connect all branch subnetworks in parallel to build the first tree network block. The outputs of all branch subnetworks are aggregated as the output of the first tree network block; decode other attributes contained in the network from the root node features, and complete the integration of the tree network block based on these attribute encodings; build multiple tree network blocks, connect the input and output of all tree network blocks, and finally obtain the convolutional neural network corresponding to the tree chromosome;

[0087] Specifically, step S3 is as follows:

[0088] S301, traverse all nodes of the first branch of the tree encoding, and generate a convolution layer for each node. The convolution operation types include 3D separable convolution and ordinary 3D convolution. The convolution kernel sizes include {3×3, 5×5, 7×7}. The number of output channels of the convolution layer ranges from {16, 32, 48, 64}.

[0089] S302: Connect the convolutional layers generated in step S301 to generate a branch network. All branch subnetworks are connected in parallel to construct the first tree network block. The outputs of all branch subnetworks are aggregated as the output of the first tree network block.

[0090] Aggregation methods include: summation, weighted summation, or spectral dimension concatenation;

[0091] S303, decoding other attributes contained in the network from the root node features, and completing the integration of the tree-shaped sub-network blocks according to these attribute encodings;

[0092] Among them, in the root node features decoded in step S303, the number of subnetwork output channels refers to the convolution output of all leaf nodes in the current subtree, which is used to aggregate the results of all branches and has a value range of {16, 32, 48, 64}; the attention model information refers to the addition of attention modules to the current subnetwork according to the encoding to increase the weight of more valuable features, thereby improving the feature extraction ability of the model, including spatial attention modules, channel attention modules, and spatial-channel attention modules; downsampling operations include averaging and maximum pooling; the skip connection flag indicates whether to add a residual connection between the input and output of the network block, represented by a Boolean value. The tree-shaped multi-branch supernetwork extracts features of different scales through multiple branches, which can overcome the disadvantage of weak feature extraction ability of a single scale and has the advantages of enhancing the feature extraction ability of the network and improving classification accuracy.

[0093] S304, executing steps S301 to S303 multiple times, building all tree network blocks, connecting the inputs and outputs of all tree network blocks, and finally obtaining a hyperspectral image classification network model corresponding to the tree chromosome.

[0094] like Figure 2 As shown in S4, agent model assisted multi-objective evolutionary search:

[0095] The chromosome population P encoded by the random initialization step S1 n , initialize the maximum number of iterations, population size, the accuracy tau of the proxy model on the proxy database R and the elite population P n ′, if the conditional accuracy tau < 0.9 is met, use the construction method of the convolutional neural network in step S3 to construct the population P n The corresponding convolutional neural network uses the training set and validation set divided in step S2 to train the population P n The corresponding convolutional neural network is trained and evaluated, and the classification accuracy is obtained. The data set is added to the proxy database R, and the proxy database R is divided into proxy training sets in proportion. and proxy test set Using a Proxy Training Set Train the GIN proxy model on the proxy test set The upper test accuracy tau, if the condition is not met, the GIN proxy model is used to predict the population P n The fitness of {y1,y2,..,y N}; Calculate the population P n The floating point operation complexity fitness {q1,q2,..,q N}; Combine the parent population and the elite population.

[0096] Specifically, step S4 is as follows:

[0097] S401, randomly initialize the chromosome population P encoded in step S1 n , initialize the maximum number of iterations M, the population size N, the accuracy of the proxy model on the proxy database R tau = 0 and the elite population

[0098] S402: If the condition accuracy tau < 0.9 is met, use the convolutional neural network construction method in step S3 to construct the population P n The corresponding convolutional neural network uses the training set and validation set divided in step S202 to train the population P nThe corresponding convolutional neural network is trained and evaluated, and the classification accuracy is obtained and then the dataset is added to the proxy database R; the tree chromosome encoding is converted into graph data that can be used by GIN. Since GIN can input graphs of different sizes, the three trees are directly generated according to the node generation sequence to obtain the graph node representation. Then, the adjacency matrix and node features are generated according to the tree branch connection, root node attributes and other node attributes. For the attribute information of all nodes, binary encoding needs to be converted into node features to enrich the amount of information embedded in the graph representation and improve the feature extraction performance of GIN for tree chromosomes. The proxy database R is divided into proxy training sets in proportion. and proxy test set Using a Proxy Training Set Train the GIN proxy model on the proxy test set The test accuracy is tau. The GIN-based proxy model adopts a dual-end input and weight-sharing structure. In each batch of training data, the input model is combined with two pairs to learn the ranking information. In training, SME-TCNN uses a loss function that emphasizes pairwise ranking learning.

[0099]

[0100] γ(θ)=log(1+e -θ )

[0101] Where w is a learnable parameter, n is the batch size, c is the tree chromosome, y is the true classification accuracy corresponding to the chromosome, g(·) is the prediction accuracy of the proxy model output, and γ(·) is the logistic regression function. To smooth the output of GIN, λ is used as the proportional parameter of the ranking loss and the MSE loss. Based on the output of GIN, the prediction accuracy of GIN for the fitness ranking of the current batch of data can be calculated as follows:

[0102]

[0103] Among them, the function rank(·) indicates whether the ranking relationship predicted by the proxy model and the actual ranking relationship of a pair of chromosome data match. It is 1 when the relationship matches and 0 when the relationship contradicts.

[0104] S403. If the conditions are not met, use the GIN proxy model to predict the population P. n The fitness of {y1,y2,..,y N The input process of the GIN proxy model is the same as step S402;

[0105] S404. Calculate population P n The floating point operation complexity fitness {q1,q2,..,q N}. Combine the parent population and the elite population; population P n The calculation formula of the floating-point operation complexity fitness is:

[0106]

[0107] Among them, flops c is the number of floating-point operations calculated after a forward inference of the real neural network model corresponding to the tree chromosome c, flops max and flops min They represent the maximum and minimum number of floating-point operations in all chromosomes of the current population, respectively. H and W are the width and height of the input data, and K, C in and C out are the convolution kernel size of the convolution layer, the dimensions of the input and output channels, and I and U represent the dimensions of the input and output of the fully connected layer.

[0108] S5. Using a multi-objective selection algorithm to select a new elite population for the parent population and the elite population combined in step S4.

[0109] like Figure 3 As shown, S6, performing a crossover operation on the new elite population in step S5 according to probability to generate a crossover population;

[0110] Specifically, in step S6, the crossover operation performed on the population is as follows:

[0111] S601. Randomly select two chromosome individuals corresponding to the convolutional neural network to be optimized from the parent population of the current iteration;

[0112] S602: Perform a crossover operation on the two selected chromosome individuals according to the crossover probability cp. Generate a random number that satisfies a uniform distribution of 0-1 for each pair of selected chromosome individuals. If the random number is less than cp, 0≤cp≤1, for the first tree of the parent generation, randomly select and exchange branches from the two parent chromosomes. For all branches in the same tree subnetwork, the number of output channels of the convolutional layer of the leaf node needs to be reset according to the attributes of the root node.

[0113] S603. Determine whether all chromosome individuals have been selected in the parent population after the current iteration. If so, form a crossover population with all chromosome individuals obtained through the above crossover operation in the parent population after the current iteration and then execute step S7. Otherwise, execute step S601.

[0114] like Figure 4 As shown, S7, performing mutation operation on the individuals in the crossover population obtained in step S6, and forming a mutation population from all the mutant offspring.

[0115] Specifically, in step S7, the mutation operation performed on the population is as follows:

[0116] S701, performing a mutation operation on the chromosome individuals in the crossover population in step S603 according to probability; generating a random number that satisfies a uniform distribution of 0-1 for each selected chromosome individual, and performing a mutation operation on the chromosome if the random number is less than mp, 0≤mp≤1;

[0117] The mutation operation in step S701 includes any one of the following (1)-(5):

[0118] (1) Delete randomly selected branches. If this operation results in a tree without any branches, generate new branches by randomly selecting new nodes from the search space.

[0119] (2) Generate a new branch containing three nodes, where the nodes are randomly selected from the search space;

[0120] (3) A new node is added to the branch of the tree, and the convolution kernel size and the number of output channels of the new node are randomly selected from the search space;

[0121] (4) Delete a node from a branch of the tree;

[0122] (5) The value of the skip connection flag is inverted and a skip connection is added.

[0123] S702: All chromosome individuals obtained through the above mutation operation in the crossover population of step S603 are combined into a mutation population.

[0124] S8. Use the mutant population obtained in step S7 as the next generation population.

[0125] S9. Iterate the population obtained in step S8. If the current number of iterations is equal to the maximum number of iterations, take the current iterated population as the optimal population, and perform a real training evaluation on the optimal population. Select the convolutional neural network corresponding to the individual with the highest classification accuracy as the optimal convolutional neural network.

[0126] S10: Input the hyperspectral image to be classified into the trained optimal convolutional neural network selected in step S9 to obtain the final classification result, and calculate the classification accuracy of the convolutional neural network for all samples to be classified in the hyperspectral image, and output the corresponding classification accuracy as the result.

[0127] The advantages of the present invention are:

[0128] (1) The present invention uses three tree-shaped hybrid coding subnets to represent the convolutional neural network, and uses multi-objective selection, crossover, and mutation to iteratively update the population. It uses an evolutionary algorithm to optimize the convolutional neural network under multi-objective indicators, solving the problems of high cost and structural redundancy caused by the manual design of convolutional neural networks in the prior art. It has the advantages of improving classification accuracy and reducing the complexity of the network structure.

[0129] (2) The present invention uses both classification accuracy and floating-point computational complexity as environmental selection targets, retaining individuals with both fitness levels and automatically designing a network model that meets both high classification accuracy and high computational efficiency. This avoids the human error caused by manually constructing convolutional neural networks, and solves the problem of excessive network model complexity and excessive computational resource requirements caused by single-target search. This approach has the advantages of reducing network model complexity and computational time.

[0130] (3) The present invention designs a tree subnetwork as the basic unit of the architecture search space. Each tree subnetwork contains multiple branches, and each branch consists of multiple nodes, which are encoded as convolutional layers of a convolutional neural network. The tree network has a strong ability to extract multi-branch features, and the scheme of adding attention weights to the output of each branch helps to fuse multi-scale features.

[0131] (4) The present invention designs a proxy-assisted scheme for nonlinear convolutional neural networks, converts tree-shaped chromosome encoding into graph representation data and embeds it into the GIN proxy model, and fits the classification accuracy labels of the chromosome corresponding network based on the idea of ​​pairwise sorting, ultimately achieving high classification accuracy while significantly reducing the computational time of evolutionary optimization;

[0132] (5) The present invention designs crossover and mutation operators and multi-target environment selection based on tree data structure, so that individuals can obtain neural network models of different depths through crossover and mutation operations, adaptively adjust the network depth and chromosome genes, and thus obtain suitable multi-scale features, which has the advantages of enhancing algorithm search capabilities and improving classification efficiency.

[0133] 1. Simulation experiment conditions:

[0134] The hardware platform of the simulation experiment of the present invention is: Nvidia GeForce RTX 3090 GPU model.

[0135] The software platform for the simulation experiment of the present invention is: Ubuntu 22.10 system, Python 3.9, and PyTorch 1.13.

[0136] The input images used in the simulation experiments of the present invention are: IndianPines hyperspectral dataset, PaviaU hyperspectral dataset and Houston2013 hyperspectral dataset.

[0137] The population size N and the maximum number of iterations M in evolutionary computation are set to 100 and 30 respectively, and the probability of crossover operation p c and the probability p of mutation operation m They are set to 0.7 and 0.3 respectively. The performance threshold of the proxy model is set to 0.9, and the hidden layer dimension of the GIN network is set to 48. Due to the use of tree coding and graph representation embedding, it is difficult to add hyperparameter search for input image block size and spectral resolution under the condition of matching different tree coding conversion dimensions. Therefore, based on experience, in the experiment, they are set to 19 and 55 respectively for the IndianPines dataset; for the PaviaU dataset, they are set to 23 and 55 respectively; for the Houston2013 dataset, they are set to 29 and 90 respectively. The present invention uniformly sets the input image block size of all comparison methods on the same dataset to the above settings, and the spectral dimension is processed according to the original text of the comparison method. In addition, the model training hyperparameters of all methods are uniformly set to: the number of training rounds is set to 150, the optimizer uses Adam, the momentum is 0.9, the learning rate is 0.001, the batch size is set to 16, and the weight decay coefficient of L2 regularization is 0.0003.

[0138] 2. Simulation content and result analysis:

[0139] The simulation experiment of the present invention uses the present invention and the existing technologies AE-CNN and 3-D-ANAS to perform classification tests on the labeled samples in the IndianPines hyperspectral dataset, the PaviaU hyperspectral dataset and the Houston2013 hyperspectral dataset.

[0140] The existing technology AE-CNN refers to: Sun et al. proposed an evolution-based convolutional neural network structure search method for solving the problem of hyperspectral image object classification in "Completely automated CNN architecture design based on blocks. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(4): 1242–1254.", referred to as AE-CNN.

[0141] The existing technology 3-D-ANAS refers to: Zhang et al. proposed a neural network architecture search method based on echelon descent for solving the problem of hyperspectral image object classification in "3-d-anas: 3-d asymmetric neural architecture search for fast hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1–19.", referred to as 3-D-ANAS.

[0142] The simulation content of the present invention is:

[0143] Simulation Experiment 1 uses the present invention and the existing technologies AE-CNN and 3-D-ANAS to simulate Indian Pines hyperspectral image data. Indian Pines is the earliest test data used for hyperspectral image classification. The airborne visible infrared imaging spectrometer (AVIRIS) imaged a pine forest in Indiana, USA in 1992. The imaging wavelength range is 0.4-2.5μm. The ground features were imaged continuously in 220 consecutive bands. Because the 104th to 108th, 150th to 163rd, and 220th bands are not reflected by water, after correction, the remaining 200 bands are used as the research objects. Then, pixels of size 145×145 are cut off and annotated as hyperspectral image classification test data. The real annotation image used in the simulation experiment of the present invention is shown in Figure 5(a). The image in Figure 5(a) contains 16 types of ground objects. The areas in the image are colored from dark to light, representing categories 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, and 16, respectively. The total number of labeled samples is 10,249. Simulation Experiment 1 samples 5% of the labeled samples from the ground objects in this image to form a training set, 5% of the labeled samples to form a validation set, and 100% of the labeled samples to form a test set. The simulation is then performed to obtain the classification results for each sample in the test set using the present invention, the prior art AE-CNN, and 3-D-ANAS. The classification results obtained by the present invention, the prior art AE-CNN, and 3-D-ANAS are plotted as the classification result graphs shown in Figures 5(b), 5(c), and 5(d), respectively.

[0144] By comparing the classification result maps shown in Figures 5(b), 5(c) and 5(d) with the true annotation map shown in Figure 5(a), it can be seen that in the classification result maps obtained by the method of the present invention, although there are misclassified samples in a small number of categories, the classification results of most samples are correct and very close to the true annotation map shown in Figure 5(a). In the classification result maps obtained by the existing technologies AE-CNN and 3-D-ANAS, there are not only a large number of small area misclassification samples, but also a large number of concentrated misclassification samples of some ground objects. The errors are more obvious and widely distributed.

[0145] In order to verify the effect of the simulation experiment 1 of the present invention, the classification accuracy of each type of ground object, the overall classification accuracy and the average classification accuracy are used to quantify the classification results of the method of the present invention and the existing technologies AE-CNN and 3-D-ANAS.

[0146] The calculation formulas of the following three indicators are used to quantify various types of ground objects in the results of simulation experiment 1. The results are shown in Table 1.

[0147]

[0148] Table 1. Quantitative results of simulation experiment 1

[0149]

[0150] Table 1 lists the classification accuracy per class, average classification accuracy, and overall classification accuracy of the present invention, the existing AE-CNN, and 3-D-ANAS methods, respectively. Table 1 shows that the overall accuracy of the present invention method improves by 0.47% and 3.48% compared to the existing AE-CNN and 3-D-ANAS methods, respectively. The average accuracy improves by 3.38% compared to the existing AE-CNN method, while decreasing by 0.65% compared to the existing 3-D-ANAS method. In the per-class classification accuracy comparison, the present invention method improves the classification accuracy for classes 4, 5, 7, 8, 9, 10, and 11 compared to the existing AE-CNN method. Class 8 achieves the best classification accuracy using the present invention method, while the classification accuracy for the other classes decreases compared to the existing AE-CNN method. The present invention method does not improve the classification accuracy for classes 1, 3, 6, 7, 9, and 16 compared to the existing 3-D-ANAS method, but improves the classification accuracy for the other classes compared to the existing 3-D-ANAS method.

[0151] Simulation Experiment 2 employed the present invention and the prior art AE-CNN and 3-D-ANAS, respectively, on the PaviaU data. The PaviaU data is a subset of hyperspectral data captured by a German airborne reflectance optical spectroscopic imager in Pavia, Italy, in 2003. The original PaviaU data consists of 115 bands, with a spectral range of 430-860 nm and a spatial resolution of 1.3 m. In this experiment, the original image was preprocessed by removing 12 bands with low signal-to-noise ratios from the original 115 bands, leaving 103 bands. Certain samples containing no information were also removed from the original image, resulting in a 610×340 hyperspectral image classification dataset. The actual annotation image is shown in Figure 6(a). Figure 6(a) shows nine ground feature types. The image regions, from dark to light, represent category 1, category 2, category 3, category 4, category 5, category 6, category 7, category 8, and category 9, respectively. The total number of annotated samples is 42,776. In simulation experiment 2, 200 and 100 labeled data samples of the ground objects in the image are extracted as training sets and validation sets, respectively. The entire labeled data set is used as the test set for simulation, and the classification results of each sample in the validation set are obtained by the present invention, the prior art AE-CNN, and 3-D-ANAS. The classification results obtained by the present invention, the prior art AE-CNN, and 3-D-ANAS are plotted as classification result diagrams shown in Figures 6(b), 6(c), and 6(d), respectively.

[0152] By comparing the classification result maps shown in Figures 6(b), 6(c) and 6(d) with the true annotation map shown in Figure 6(a), it can be seen that in the classification result maps obtained by the method of the present invention, almost most samples have achieved correct classification results, and the overall effect is very close to the true annotation map shown in Figure 6(a). However, in the classification result maps obtained by the existing technologies AE-CNN and 3-D-ANAS, there are still many misclassified samples of some types of ground objects.

[0153] In order to verify the effect of simulation experiment 2 of the present invention, the classification accuracy of each type of ground object, the overall classification accuracy and the average classification accuracy are used to quantify the classification results of the method of the present invention and the existing technologies AE-CNN and 3-D-ANAS respectively.

[0154] Using the same calculation formulas for the three indicators as those in simulation experiment 1, various types of ground objects in the results of simulation experiment 2 are quantified, and the results are shown in Table 2.

[0155] Table 2. Quantitative results of simulation experiment 2

[0156]

[0157]

[0158] Table 2 lists the classification accuracy per class, average classification accuracy, and overall classification accuracy of the present invention, the prior art AE-CNN, and 3-D-ANAS. Table 2 shows that the overall accuracy of the present invention's method improves by 2.29% and 2.26% compared to the prior art AE-CNN and 3-D-ANAS, respectively, and the average accuracy improves by 6.49% and 3.44% compared to the prior art AE-CNN and 3-D-ANAS, respectively. In the per-class classification accuracy comparison between the present invention and the prior art AE-CNN, the classification accuracy of all objects improves compared to the prior art AE-CNN, with the exception of categories 1, 5, and 6. In the per-class classification accuracy comparison between the present invention and the prior art 3-D-ANAS, the classification accuracy of all objects improves compared to the prior art 3-D-ANAS, with the exception of categories 1, 4, 5, and 9.

[0159] Simulation Experiment 3 employed the present invention and the prior art AE-CNN and 3-D-ANAS, respectively, on the Houston2013 dataset. The Houston2013 dataset was acquired by the ITRES CASI-1500 sensor on the University of Houston campus and the surrounding urban area during the 2013 IEEE GRSS Data Fusion Competition. Houston2013 contains 144 bands in the spectral range of 380-1050 nm, with a spatial resolution of 2.5 m and an image size of 349 × 1905. Its actual annotation image is shown in Figure 7(a). Figure 7(a) shows 15 ground feature types. The image regions, from dark to light, represent category 1, category 2, category 3, category 4, category 5, category 6, category 7, category 8, category 9, category 10, category 11, category 12, category 13, category 14, and category 15, respectively. The total number of annotated samples is 15,029. In simulation experiment 3, 200 and 100 labeled data samples of the ground objects in the image are respectively extracted as the training set and the validation set, and the entire labeled data set is used as the test set for simulation. The classification results of each sample in the validation set by the present invention and the prior art AE-CNN and 3-D-ANAS are obtained, and the classification results obtained by the present invention and the prior art AE-CNN and 3-D-ANAS are plotted as the classification result diagrams shown in Figures 7(b), 7(c) and 7(d), respectively.

[0160] By comparing the classification result maps shown in Figures 7(b), 7(c) and 7(d) with the true annotation map shown in Figure 7(a), it can be seen that in the classification result maps obtained by the method of the present invention, except for a small number of ground object samples with classification errors, most of the sample classifications are correct and very close to the true annotation map shown in Figure 7(a). In the classification result maps obtained by the existing technologies AE-CNN and 3-D-ANAS, some types of ground objects have samples with obvious classification errors, and the number is large and the distribution is wide.

[0161] In order to verify the effect of simulation experiment 3 of the present invention, the classification accuracy of each type of ground object, the overall classification accuracy and the average classification accuracy are used to quantify the classification results of the method of the present invention and the existing technologies AE-CNN and 3-D-ANAS.

[0162] Using the same calculation formulas of the three indicators as those in simulation experiment 1, various types of ground objects in the results of simulation experiment 3 are quantified, and the results are shown in Table 3.

[0163] Table 3. Quantitative results of simulation experiment 3

[0164]

[0165] Table 3 lists the classification accuracy per class, average classification accuracy, and overall classification accuracy of the present invention, the existing AE-CNN, and 3-D-ANAS methods, respectively. As shown in Table 3, the overall accuracy of the present invention's method improves by 4.30% and 3.39% compared to the existing AE-CNN and 3-D-ANAS methods, respectively, and the average accuracy improves by 3.47% and 2.43% compared to the existing AE-CNN and 3-D-ANAS methods, respectively. In the per-class classification accuracy comparison between the present invention and the existing AE-CNN method, the classification accuracy for classes 5, 6, 10, 11, and 13 remains unchanged. Class 14 achieves the best classification accuracy using the present invention's method, while the classification accuracy of all other objects improves compared to the existing AE-CNN method. In the per-class classification accuracy comparison between the present invention and the existing 3-D-ANAS method, the classification accuracy for classes 4, 6, 12, 13, and 15 remains unchanged, while the classification accuracy of all other objects improves compared to the existing 3-D-ANAS method.

[0166] Simulation experiment 4 is a simulation experiment using the present invention and the prior art AE-CNN to search the network architecture time cost and overall accuracy on three types of data.

[0167] Table 4. Quantitative results of simulation experiment 4

[0168]

[0169] Table 4 lists the overall accuracy and GPU-day calculation results of the proposed method and the prior art AE-CNN on three datasets. As shown in Table 4, the proposed method improves overall accuracy by 0.93% on IndianPines compared to the prior art, and reduces GPU-days by 9.4981 days. The proposed method improves overall accuracy by 1.83% on PaviaU compared to the prior art, and reduces GPU-days by 7.537 days. The proposed method improves overall accuracy by 3.07% on Houston2013 compared to the prior art, and reduces GPU-days by 7.2682 days. Therefore, the proposed method outperforms the prior art AE-CNN in both search time cost and classification accuracy.

[0170] The above simulation experiments show that the method of the present invention uses a multi-objective evolutionary algorithm to automatically search for convolutional neural networks and construct a multi-branch parallel tree network structure to enhance the network's extraction of hyperspectral image features. While ensuring high classification accuracy for ground object samples, it can search for a simpler and more effective convolutional neural network architecture, reducing the complexity and number of parameters of the network model, and greatly improving the efficiency of convolutional neural networks in hyperspectral image ground object classification tasks. At the same time, the GIN proxy model is added to the evolutionary process, which reduces the burden of traditional neural architecture search, significantly reduces search time, and improves the prediction accuracy of the proxy model.

[0171] Application prospects of the present invention

[0172] As hyperspectral images are widely used in military and civilian fields, the demand for hyperspectral image classification is also increasing. Traditional manually designed neural network models are no longer sufficient to meet the model design requirements for massive hyperspectral images. In addition, as the industry's requirements for hyperspectral image classification are getting higher and higher, existing technologies can no longer meet the industry's requirements for high precision and high efficiency at the same time. The present invention: (1) uses classification accuracy and floating-point computational complexity as environmental selection targets at the same time, so that individuals with both fitness levels are retained, and automatically designs a network model that meets both high classification accuracy and high computational efficiency. It avoids the human errors caused by manually constructing convolutional neural networks, solves the problem of high complexity of network models and excessive demand for computing resources caused by single-target search, and has the advantages of reducing the complexity of network models and reducing the computing time of network models; (2) It designs an agent-assisted scheme for nonlinear convolutional neural networks, converts tree chromosome encoding into graph representation data and embeds it into the GIN agent model, and fits the classification accuracy labels of the chromosome corresponding network based on the pairwise sorting idea, ultimately achieving high classification accuracy while greatly reducing the computing time of evolutionary optimization; (3) It designs a tree subnet as the basic unit of the architecture search space, each tree subnet contains multiple branches, each branch consists of multiple nodes, and these nodes are encoded as convolutional layers of the convolutional neural network. The tree network has a strong ability to extract multi-branch features, and the scheme of adding attention weights to the output of each branch is conducive to the fusion of multi-scale features; (4) Three tree-shaped hybrid coding subnetworks are used to represent the convolutional neural network, and multi-target selection, crossover, and mutation are used to iteratively update the population. The evolutionary algorithm is used to optimize the convolutional neural network under multi-target indicators, which solves the problems of high cost and structural redundancy brought about by the manual design of convolutional neural networks in the existing technology, and has the advantages of improving classification accuracy and reducing the complexity of network structure; (5) Crossover, mutation operators and multi-target environment selection based on tree data structure are designed, so that individuals can obtain neural network models of different depths through crossover and mutation operations, adaptively adjust the network depth and chromosome genes, and thus obtain suitable multi-scale features, which has the advantages of enhancing the algorithm search capability and improving classification efficiency.

Claims

1. A hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search, characterized in that: The following steps are involved: S1. Initialize neural architecture information: A tree-shaped multi-branch supernet is used as the basic unit of the search space. A tree-based data structure encoding strategy is used to encode the proxy model parameters and structure into a tree chromosome. Three tree subnets are set in the initial search space, forming the high-level network architecture of the search. S2. Generate the training set and validation set corresponding to each tree chromosome: The principal component analysis (PCA) method is used to reduce the dimensionality of the input spectrum of the hyperspectral image to be classified and denoise it. The ground feature label of the central pixel of each image block in the hyperspectral image after dimensionality reduction and denoising is used as the input data label. All image blocks are divided into training and validation sets according to a preset ratio. S3. Construct a convolutional neural network based on the tree chromosome generated in step S1: Traverse all nodes of the first branch of the tree encoding, generate a convolutional layer for each node, connect the convolutional layers to generate a branch network, connect all branch subnetworks in parallel, build the first tree network block, aggregate the outputs of all branch subnetworks as the output of the first tree network block; decode other attributes contained in the network from the root node features, and complete the integration of the tree network block based on these attribute encodings; build all tree network blocks, connect the input and output of all tree network blocks, and finally obtain the convolutional neural network corresponding to the tree chromosome; S4. Surrogate model assisted multi-objective evolutionary search: The chromosome population P encoded by the random initialization step S1 n , initialize the maximum number of iterations, population size, the accuracy tau of the proxy model on the proxy database R and the elite population P n ′, if the conditional accuracy tau < 0.9 is met, use the construction method of the convolutional neural network in step S3 to construct the population P n The corresponding convolutional neural network uses the training set and validation set divided in step S2 to train the population P n The corresponding convolutional neural network is trained and evaluated, and the classification accuracy is obtained. The data set is added to the proxy database R, and the proxy database R is divided into proxy training sets in proportion. and proxy test set Using a Proxy Training Set Train the GIN proxy model on the proxy test set The upper test accuracy tau, if the condition is not met, the GIN proxy model is used to predict the population P n The fitness of {y1,y2,..,y N }; Calculate the population P n The floating point operation complexity fitness {q1,q2,..,q N }; Combine parent population and elite population; S5. Using a multi-objective selection algorithm to select a new elite population for the parent population and the elite population combined in step S4; S6, performing a crossover operation on the new elite population in step S5 according to probability to generate a crossover population; S7, performing a mutation operation on the individuals in the crossover population obtained in step S6, and forming a mutation population from all the mutant offspring; S8, using the mutant population obtained in step S7 as the next generation population; S9, iterate the population obtained in step S8. If the current number of iterations is equal to the maximum number of iterations, take the current iterated population as the optimal population, and perform real training evaluation on the optimal population. Select the convolutional neural network corresponding to the individual with the highest classification accuracy as the optimal convolutional neural network; S10: Input the hyperspectral image to be classified into the trained optimal convolutional neural network selected in step S9 to obtain the final classification result, and calculate the classification accuracy of the convolutional neural network for all samples to be classified in the hyperspectral image, and output the corresponding classification accuracy as the result.

2. The hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search according to claim 1 is characterized in that: The hybrid encoding method in step S1 is specifically: First, the number of branches of each tree subnetwork is initialized. In the initial search space, each tree subnetwork has a maximum of three branches. Then, the nodes of each branch are initialized. Each branch has a maximum of three nodes at initialization. The tree chromosome includes the root node and other nodes. The characteristics of the root node include: the number of subnetwork output channels, skip connection flags, downsampling operations, and attention model information; the characteristics of other nodes include: the number of convolutional layer output channels, convolution operation type, and convolution kernel size.

3. The hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search according to claim 1 is characterized in that: The specific steps for generating the training set and validation set in step S2 are: S201, using principal component analysis (PCA) to reduce the dimensionality of the input spectrum of the hyperspectral image to be classified and remove noise, so that its spectral dimension is equal to the input spectrum dimension corresponding to the gene individual; S202. Taking each labeled pixel in the hyperspectral image after dimensionality reduction and denoising in step S201 as the center, intercept a rectangular block of the pixel. The width and height of the rectangular block are equal to the image block size corresponding to the genetic individual, and the label of the central pixel is used as the label of the rectangular block. All image blocks are divided into a training set and a validation set according to a preset ratio. The accuracy of the model classification result on the validation set is calculated as follows: Among them, A c The network model corresponding to the tree chromosome encoding c, |D Test | represents the size of the validation set, X i is the i-th image block data in the verification dataset, y i is the object category label of the i-th image block, F(·) is the prediction result output by the network model, and the function equal(·) is used to compare the difference between the prediction result and the true label. When the predicted value is the same as the true label, its value is equal to 1.

4. A hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search according to claim 1, 2 or 3, characterized in that: Step S4 is specifically as follows: S401, randomly initialize the chromosome population P encoded in step S1 n , initialize the maximum number of iterations M, the population size N, the accuracy of the proxy model on the proxy database R tau = 0 and the elite population S402: If the condition accuracy tau < 0.9 is met, use the convolutional neural network construction method in step S3 to construct the population P n The corresponding convolutional neural network uses the training set and validation set divided in step S202 to train the population P n The corresponding convolutional neural network is trained and evaluated, and the classification accuracy is obtained and then the dataset is added to the proxy database R. The tree chromosome encoding is converted into graph data that can be used by GIN. The tree chromosome encoding is obtained according to the node generation sequence number to obtain the graph node representation. The adjacency matrix and node features are generated according to the tree branch connection, root node attributes and other node attributes. For the attribute information of all nodes, binary encoding is used to convert it into node features to enrich the information embedded in the graph representation. The proxy database R is divided into proxy training sets in proportion. and proxy test set Using a Proxy Training Set Train the GIN proxy model on the proxy test set The test accuracy is tau. The GIN-based proxy model adopts a dual-end input and weight-sharing structure. In each batch of training data, the input model is combined with two pairs to learn the ranking information. In training, SME-TCNN uses a loss function that emphasizes pairwise ranking learning. γ(θ)=log(1+e -θ ) Where w is a learnable parameter, n is the batch size, c is the tree chromosome, y is the true classification accuracy corresponding to the chromosome, g(·) is the prediction accuracy of the proxy model output, and γ(·) is the logistic regression function. λ is used as the proportional parameter of the ranking loss and the MSE loss. Based on the output of GIN, the prediction accuracy of GIN for the fitness ranking of the current batch of data is calculated as follows: Among them, the function rank(·) indicates whether the ranking relationship predicted by the proxy model and the actual ranking relationship of a pair of chromosome data match. It is 1 when the relationship matches and 0 when the relationship contradicts. S403. If the conditions are not met, use the GIN proxy model to predict the population P. n The fitness of {y1,y2,..,y N The input process of the GIN proxy model is the same as step S402; S404, calculate population P n The floating point operation complexity fitness {q1,q2,..,q N }, combined parent population and elite population; population P n The calculation formula of the floating-point operation complexity fitness is: Among them, flops c is the number of floating-point operations calculated after a forward inference of the real neural network model corresponding to the tree chromosome c, flops max and flops min They represent the maximum and minimum number of floating-point operations in all chromosomes of the current population, H and W are the width and height of the input data, K, C in and C out are the convolution kernel size of the convolution layer, the dimensions of the input and output channels, and I and U represent the dimensions of the input and output of the fully connected layer.

5. The hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search according to claim 1 is characterized in that: In step S6, the crossover operation is performed on the population as follows: S601. Randomly select two chromosome individuals corresponding to the convolutional neural network to be optimized from the parent population of the current iteration; S602: Perform a crossover operation on the two selected chromosome individuals according to the crossover probability cp. Generate a random number that satisfies a uniform distribution of 0-1 for each pair of selected chromosome individuals. If the random number is less than cp, 0≤cp≤1, for the first tree of the parent generation, randomly select and exchange branches from the two parent chromosomes. For all branches in the same tree subnetwork, the number of output channels of the convolutional layer of the leaf node needs to be reset according to the attributes of the root node. S603. Determine whether all chromosome individuals have been selected in the parent population after the current iteration. If so, form a crossover population with all chromosome individuals obtained through the above crossover operation in the parent population after the current iteration and then execute step S7. Otherwise, execute step S601.

6. A hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search according to claim 1 or 5, characterized in that: In step S7, the mutation operation is performed on the population as follows: S701, performing a mutation operation on the chromosome individuals in the crossover population in step S603 according to probability; generating a random number that satisfies a uniform distribution of 0-1 for each selected chromosome individual, and performing a mutation operation on the chromosome if the random number is less than mp, 0≤mp≤1; S702: All chromosome individuals obtained through the above mutation operation in the crossover population of step S603 are combined into a mutation population.

7. The hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search according to claim 1 is characterized in that: In step S3, all nodes of the first branch of the tree encoding are traversed, and a convolution layer is generated for each node. The convolution operation types include 3D separable convolution and ordinary 3D convolution. The convolution kernel sizes include three sizes: {3×3, 5×5, 7×7}. The value range of the number of output channels of the convolution layer is {16, 32, 48, 64}.

8. The hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search according to claim 1 is characterized in that: The aggregation method in step S3 includes: summation, weighted summation or spectral dimension splicing.

9. The hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search according to claim 1, characterized in that: In the root node feature decoded in step S3, the number of subnetwork output channels refers to the convolution output of all leaf nodes of the current subtree, which is used to aggregate the results of all branches, and the value range is {16, 32, 48, 64}.

10. The hyperspectral image classification method based on agent-assisted multi-objective evolutionary neural architecture search according to claim 6, characterized in that: The mutation operation in step S701 includes any one of the following (1)-(5): (1) Deleting randomly selected branches. If this results in a tree without any branches, a new branch is generated by randomly selecting new nodes from the search space. (2) Generate a new branch containing three nodes, where the nodes are randomly selected from the search space; (3) A new node is added to the branch of the tree, and the convolution kernel size and the number of output channels of the new node are randomly selected from the search space; (4) Delete a node from a branch of the tree; (5) The value of the skip connection flag is inverted and a skip connection is added.

Images