A hybrid architecture expert-driven remote sensing semantic segmentation self-learning network search method

Abstract

The application relates to a hybrid architecture expert-driven remote sensing semantic segmentation self-learning network search method, which comprises the following steps: acquiring a data set of remote sensing semantic segmentation; designing a hybrid architecture expert network as a feature encoder based on prior parameters; constructing a heterogeneous coding dynamic search space, adaptively searching and learning to fuse multi-granularity visual features of the hybrid architecture expert; constructing a coarse-fine granularity representation search space, adaptively searching and learning to fuse the multi-granularity visual features; adopting a step-by-step up-sampling strategy from bottom to top, adaptively searching and learning to restore the multi-granularity visual features from coarse to fine to the original resolution of an image; outputting a semantic segmentation mapping result, and adopting an accuracy evaluation to evaluate the performance of the network. The application fuses multi-architecture and hybrid expert visual features and multi-granularity visual features, constructs a remote sensing image semantic segmentation network with richer feature expression, and enhances the adaptability to multi-scale characteristics of remote sensing ground objects.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote sensing image semantic segmentation network design, and particularly relates to a hybrid architecture expert-driven remote sensing semantic segmentation self-learning network search method. BACKGROUND

[0002] Land use and land cover mapping is an important task in geographical research, which has been widely used in monitoring global environmental change, ecological and climate change, urban planning, resource investigation and other fields. With the development of remote sensing technology, the resolution of remote sensing images has become higher and higher, especially with its advantages of rich spatial details and potential semantic content, it has become the main means of large-scale land use and land cover monitoring.

[0003] The semantic segmentation network based on deep learning mainly relies on the following two design methods:

[0004] (1) Artificial network design based on prior knowledge

[0005] The artificial network design based on prior knowledge relies on the prior knowledge and experience of researchers, and its mainstream design methods include: single architecture design and multi-architecture hybrid design.

[0006] 1) The single architecture design method usually relies on a single network to design the corresponding network, and its mainstream networks include: convolutional neural network (CNN), Transformer, Mamba, etc. The above network frameworks have their own advantages and disadvantages in visual feature expression, such as: the convolutional neural network has strong local spatial feature learning ability; the Transformer architecture enhances its ability to capture global information in the image through the self-attention mechanism; and the visual expression of the Mamba architecture is more balanced.

[0007] 2) The multi-architecture hybrid design method fuses the above single networks to enhance the ability of visual representation. For example: the fusion of convolutional neural network and Transformer network, and its typical network architecture includes: FT-UnetFormer, etc. Through the fusion of different architectures, the advantages of different architectures can be further fused, and the defects of themselves can be made up.

[0008] (2) Automatic network design based on network architecture search

[0009] The automatic network design method based on network architecture search can effectively solve the waste of human and computing resources caused by manual network design by automatically searching for an optimal neural network architecture. Gradient-based algorithms are the mainstream algorithms, which can quickly and effectively adjust network architecture parameters, and have lower dependence on computing resources, helping to find more accurate optimal architecture.

[0010] In summary, the above two methods have certain shortcomings and defects:

[0011] (1) For manual network design based on prior knowledge, the single architecture design method is limited by its own design characteristics, and its advantages and disadvantages coexist, which further limits the further improvement of its generalization ability. The multi-architecture hybrid design method is limited by the experience of manual design, which is often manifested in the lack of feature richness of multi-architecture network fusion, and the lack of potential mining of feature expression;

[0012] (2) For automatic network design based on network architecture search, the search space needs to be increased due to multi-architecture cooperation, which often consumes a large amount of computing resources and prolongs the training time, and the current research lacks a reasonable search scheme for multi-architecture network fusion. SUMMARY

[0013] To solve the above technical problems, the application provides a hybrid architecture expert driven remote sensing semantic segmentation self-learning network search method, which enhances the generalization ability of remote sensing feature semantic segmentation by fusing hybrid architecture expert features and multi-granularity visual features.

[0014] The hybrid architecture expert driven remote sensing semantic segmentation self-learning network search method provided by the application adopts the following technical scheme:

[0015] A hybrid architecture expert driven remote sensing semantic segmentation self-learning network search method, comprising the following steps:

[0016] S1, obtaining a remote sensing semantic segmentation dataset, and dividing the dataset into a training set for network training, a validation set for network parameter tuning, and a test set for network evaluation;

[0017] S2, designing a hybrid architecture expert network as a feature encoder based on prior parameters;

[0018] S3, constructing a heterogeneous coding dynamic search space to learn and fuse multi-granularity visual features of hybrid architecture experts through adaptive search;

[0019] S4, constructing a coarse-fine granularity representation search space to learn and fuse coarse-fine granularity visual features through adaptive search;

[0020] S5, using a bottom-up stepwise up-sampling strategy, restoring the multi-granularity visual features from coarse to fine to the original resolution of the image through adaptive search learning;

[0021] S6, outputting the semantic segmentation mapping result and evaluating the performance of the network using precision evaluation.

[0022] Preferably, the S1 comprises the following:

[0023] An open source or self-made remote sensing semantic segmentation dataset is obtained, and the dataset is divided into a training set for network training, a verification set for network hyperparameter adjustment, and a test set for network precision evaluation according to a preset ratio, wherein the dataset comprises remote sensing images and label images.

[0024] Preferably, the S2 comprises the following steps:

[0025] S21, selecting a hybrid architecture expert network, wherein a network based on three semantic segmentation network architectures is selected as the hybrid architecture expert, namely: 1) a convolutional neural network expert: selecting DeepLapV3+ or PSPNet or ConvNeXt, denoted as C Expert1 , C Expert2 , …, C ExpertN ; 2) a Transformer network expert: selecting Segformer or SwinTransformer or PoolFormer, denoted as T Expert1 , T Expert2 , …, T ExpertN ; and 3) a Mamba network expert, denoted as M Expert1 , M Expert2 , …, M ExpertN .

[0026] S22, selecting a hybrid architecture expert representative based on prior evaluation, wherein for the hybrid architecture expert network, the optimal network in each architecture is selected as the hybrid architecture expert representative based on prior training knowledge:

[0027] C Expert =Ψ(C Expert1 ,C Expert2 ,…,C ExpertN );

[0028] T Expert =Ψ(T Expert1 ,T Expert2 ,…,T ExpertN );

[0029] M Expert =Ψ(M Expert1 ,M Expert2 ,…,M ExpertN );

[0030] wherein C Expert , T Expert , M Expert are the selected optimal convolutional neural network expert representative, the Transformer network expert representative and the Mamba network expert representative respectively, and Ψ(·) represents a network optimization strategy relying on prior pre-training knowledge, that is, the model parameters trained on open source or self-made data sets are obtained through network evaluation to obtain the network with the highest precision as the representative.

[0031] S23, mixed architecture expert visual feature coding, taking the selected optimal mixed architecture expert representative in S22 as the feature encoder of the architecture, extracting the visual features of the input image, and generating multi-granularity visual features one by one;

[0032] ;

[0033] ;

[0034] ;

[0035] wherein, is the input image, , , are the selected convolutional neural network representative, the Transformer network representative and the Mamba network representative in S22, , and are the obtained multi-granularity features, .

[0036] Preferably, the S3 comprises the following steps:

[0037] S31, constructing a heterogeneous coding dynamic search space, constructing a directed acyclic graph structure as a basic cell unit for the mixed architecture expert's same scale visual features, and constructing a two-layer node layer as a search space to adaptively fuse features of different expert networks;

[0038] S32, constructing a mixed architecture expert interaction module, the formula is:

[0039] ;

[0040] wherein, , and is used to represent the index value of different scale features; , , is a dynamic convolution layer, which includes 1 dynamic convolution layer and 1 normalization layer, Used to enhance features , , Adaptability to the scale of features in the is the learnable contribution weight, and the formula is:

[0041] ;

[0042] in, is the input feature, It is a softmax function used to calculate the contribution probability, and the TopK algorithm is used to select only the top k networks to give contribution weights.

[0043] Preferably, the S4 comprises the following steps:

[0044] S41, builds a coarse-to-fine-grained representation search space. For the multi-granularity features fused by hybrid architecture experts, a directed acyclic graph structure is constructed as the basic cell unit, and a multi-layer node is constructed as the search space for fusing multi-granularity features:

[0045] S42, build a coarse-fine feature dynamic enhancement module, and transform the current input feature As the main feature, and the remaining granularity features as auxiliary features, that is:

[0046] ;

[0047] in, is a coarse-fine feature dynamic enhancement module, where is the selected main feature, For other remaining features, it includes the following steps:

[0048] S421, firstly, the auxiliary features are sampled to the same scale as the main features through a dynamic learnable parameter, the formula is: ;

[0049] in, ; represents a dynamic convolutional layer, represents a normalization layer, is a downsampling convolutional layer, is a 1×1 convolutional layer, represents an upsampling convolutional layer;

[0050] S422, through self-learning search for coarse-fine feature fusion, enhance the multi-granularity representation capability of auxiliary features. The process is as follows: ;

[0051] in, The learnable weight contribution rate is:

[0052] ;

[0053] wherein, is the input feature, is a softmax function, used to calculate the contribution probability, and TopK algorithm is used to select the top k network to give the contribution weight;

[0054] S423, based on the auxiliary feature, the self-attention mechanism is used to focus on and fuse the interested region of the main feature, and the process is: ;

[0055] wherein, is a linear conversion layer, is the number of feature channels, is a softmax function, is a linear layer, is the transpose of.

[0056] Preferably, the S5 comprises:

[0057] ;

[0058] ;

[0059] ;

[0060] ;

[0061] wherein, is executed according to S42.

[0062] Preferably, the S6 comprises:

[0063] ;

[0064] wherein, includes an up-sampling convolutional layer, and a softmax layer is included to output the probability of each class.

[0065] In summary, the present application has the following beneficial technical effects:

[0066] 1、The present application constructs a remote sensing image semantic segmentation network with more rich feature expression by fusing multiple architectures and mixing expert visual features and fusing multi-granularity visual features, and enhances the adaptability to multi-scale characteristics of remote sensing features.

[0067] 2、The application further enhances the richness of feature expression by adaptively searching and learning the advantages of multi-architecture networks, thereby effectively suppressing the defects of each architecture network expert in visual feature expression, further focusing on network optimization of their respective advantages, while calculating the effective gain of each architecture through the mixed expert network to suppress noise and improve the precision of semantic segmentation and the adaptability to multi-scale characteristics of remote sensing ground objects.

[0068] 3、The application improves the scientificity of network design through adaptive search learning, avoids the limitations of manual design, further reduces invalid parameters, and thereby reduces the consumption of computing resources and improves the generalization and reasoning ability of the network. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 is a flowchart in the embodiments of the application.

[0070] Figure 2 is a schematic diagram of the remote sensing image semantic segmentation result given in the embodiments of the application. DETAILED DESCRIPTION

[0071] The application will be further described in detail below with reference to the accompanying drawings.

[0072] The embodiments of the application disclose a mixed architecture expert driven remote sensing semantic segmentation self-learning network search method.

[0073] Referring to Figure 1 The mixed architecture expert driven remote sensing semantic segmentation self-learning network search method comprises the following steps:

[0074] S1, acquiring a data set of remote sensing semantic segmentation, and dividing the data set into a training set for network training, a verification set for network parameter tuning, and a test set for network evaluation;

[0075] Specifically, it comprises:

[0076] An open source or self-made data set of remote sensing semantic segmentation is acquired, and the data set is divided into a training set, a verification set and a test set according to a preset ratio;

[0077] The data set includes remote sensing images and label images, and the size is cut according to 512*512. The training set is used for network training, the verification set is used for hyperparameter adjustment, and the test set is used for network precision evaluation. In this embodiment, LoveDA remote sensing semantic segmentation data set is adopted, which is from three different cities and covers urban and rural scenes. The data set contains 5987 high-resolution 0.3-meter images, each of which has a size of 1024*1024 pixels. The images are divided into training set, verification set and test set, including 2522, 1669 and 1796 images respectively. These images are labeled as seven land cover types: building, road, water, wasteland, forest, agricultural land and background.

[0078] S2, design a mixed architecture expert network as a feature encoder based on prior parameters;

[0079] Specifically, the following steps are included:

[0080] S21, select a mixed architecture expert network, wherein a network based on three semantic segmentation network architectures is selected as the mixed architecture expert, namely: 1) a convolutional neural network expert: DeepLapV3+ or PSPNet or ConvNeXt is selected, denoted as C Expert1 ,C Expert2 ,…,C ExpertN ; 2) a Transformer network expert: Segformer or SwinTransformer or PoolFormer is selected, denoted as T Expert1 ,T Expert2 ,…,T ExpertN ; and 3) a Mamba network expert, denoted as M Expert1 ,M Expert2 ,…,M ExpertN .

[0081] S22, select a mixed architecture expert representative based on prior evaluation, and for the mixed architecture expert network, the optimal network in each architecture is selected as the mixed architecture expert representative based on prior training knowledge: C Expert =Ψ(C Expert1 ,C Expert2 ,…,C ExpertN ); T Expert =Ψ(T Expert1 ,T Expert2 ,…,T ExpertN ); and M Expert =Ψ(M Expert1 ,M Expert2 ,…,M ExpertN ).

[0082] Wherein C Expert , T Expert , MExpert respectively are the selected optimal convolutional neural network expert representatives, the Transformer network expert representatives and the Mamba network expert representatives, and represents a network optimization strategy relying on prior pre-training knowledge, that is, the model parameters trained on open source or self-made data sets are used to obtain the network with the highest accuracy through network evaluation as the representative;

[0083] A plurality of networks of the same architecture are selected as the backbone, and a unified decoder is selected, pre-training is performed on a semantic segmentation data set, the generalization and semantic segmentation capability of the above network for the corresponding data set are determined through prior knowledge of accuracy evaluation, and the optimal network in the specified architecture is selected through a plurality of indexes. According to the above strategy, the optimal network in the network of different architectures is selected, and the parameters in the backbone are frozen as a feature extractor.

[0084] The above strategy has the following advantages:

[0085] 1) Through network optimization, the network with the optimal performance is selected, which can better play the ability of the architecture for feature expression;

[0086] 2) Through prior pre-training, the prior network parameters are obtained, the network parameter adjustment process is reduced, and the network training consumes less memory through freezing the network;

[0087] 3) By freezing the effective parameters in the backbone, the network is more inclined to adjust the gain of the architecture fusion, and the process of feature parameter adjustment is reduced.

[0088] S23, mixed architecture expert visual feature coding, the optimal mixed architecture expert representative selected in S22 is used as the feature encoder of the architecture, visual features of the input image are extracted, and multi-granularity visual features are generated one by one; ; ; ;

[0089] wherein, is the input image, , , respectively are the convolutional neural network representative, the Transformer network representative and the Mamba network representative selected in S22, , and are the obtained multi-granularity features, .

[0090] S3, constructing a heterogeneous coding dynamic search space, learning and fusing the multi-granularity visual features of the mixed architecture expert through adaptive search;

[0091] Specifically comprising the following steps:

[0092] S31, construct a heterogeneous coding dynamic search space, construct a directed acyclic graph structure as a basic cell unit for the same scale visual features of the mixed architecture expert, and construct a two-layer node layer as a search space to adaptively fuse the features of different expert networks;

[0093] S32, construct a mixed architecture expert interaction module, through the above-mentioned heterogeneous coding dynamic search space, through self-learning search to construct a robust and effective network structure to focus on and fuse the advantages of visual representation of different mixed architecture expert architectures. The formula is:

[0094] ;

[0095] Among them, , and is used to represent the index value of different scale features; , , is a dynamic convolution layer, which includes 1 dynamic convolution layer and 1 normalization layer, are respectively used to enhance the adaptability of the feature , , of the ground object scale; is a learnable contribution weight, which quantifies the importance of different mixed architecture expert features, self-adaptively learns the contribution weight parameter, and sorts based on this value, further weighted fusion of visual features of different experts, the formula is:

[0096] ;

[0097] Among them, is the input feature, is a softmax function, which is used to calculate the contribution probability, and TopK algorithm is used to select the top k network given contribution weight.

[0098] S4, construct a coarse-fine granularity representation search space, fuse coarse-fine granularity visual features through adaptive search learning, and set the main feature , through the construction of coarse-to-fine granularity representation search space, self-adaptively learn the effective gain of other granularity auxiliary features to the main feature, specifically comprising the following steps:

[0099] S41, construct a coarse-to-fine granularity representation search space, construct a directed acyclic graph structure as a basic cell unit for the mixed architecture expert fused multi-granularity features, and construct a multi-layer node as a search space for fusing multi-granularity features:

[0100] S42, a coarse-fine feature dynamic enhancement module is constructed to enhance the current input feature as the main feature, and the remaining granular features as auxiliary features, that is:

[0101] ;

[0102] wherein, is a coarse-fine feature dynamic enhancement module, wherein is the selected main feature, is the other remaining features, comprising the following steps:

[0103] In the above mixed architecture expert interactive fusion of the same scale of the mixed architecture expert's features, further through self-learning search from coarse to fine multi-granularity feature enhancement mode;

[0104] S421, first, the auxiliary feature is sampled to the same scale as the main feature through a dynamic learnable parameter, and the formula is: ;

[0105] wherein, ; denotes a dynamic convolution layer, denotes a normalization layer, is a down-sampling convolution layer, is a 1x1 convolution layer, denotes an up-sampling convolution layer;

[0106] S422, through self-learning search coarse-fine feature fusion mode, enhance the multi-granularity representation ability of auxiliary features, the process is: ;

[0107] wherein, learnable weight contribution rate, by quantifying the importance of different granular features, self-adaptive learning contribution weight parameters, and based on this value sorting, further weighted fusion of different granularity visual features, the formula is:

[0108] ;

[0109] wherein, is the input feature, is a softmax function, used to calculate the contribution probability, using TopK algorithm only selects the top k network given contribution weight.

[0110] S423, based on the auxiliary feature through self-attention mechanism to focus and fuse the main feature's interesting region, the process is: ;

[0111] wherein, is a linear transformation layer, is the number of feature channels, is a softmax function, is a linear layer, is is the transpose of.

[0112] S5, using a bottom-up step-by-step up-sampling strategy, through adaptive search learning, the multi-granularity visual features are recovered to the original resolution of the image from coarse to fine;

[0113] Specifically, the following steps are included:

[0114] Using a bottom-up step-by-step up-sampling strategy, further through the fused multi-granularity features, step-by-step up-sampling is used to enhance the expression ability of the features. Through the multi-granularity fusion of semantic information and detail information from coarse to fine, the complementarity of the features is constantly enhanced, and through adaptive search learning, the multi-granularity visual features from coarse to fine are recovered to the original resolution of the image; the process is as follows:

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] wherein, According to S42.

[0120] S6, outputting the semantic segmentation mapping result, and using precision evaluation to evaluate the performance of the network.

[0121] The output result formula is:

[0122] ;

[0123] wherein, Including an up-sampling convolution layer, including a softmax layer outputting the probability of each category.

[0124] S7, comparing the output semantic segmentation mapping result with the current deep learning network in performance.

[0125] The above are preferred embodiments of the present application, and are not intended to limit the protection scope of the present application, therefore: any equivalent changes made on the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A hybrid architecture expert-driven remote sensing semantic segmentation self-learning network search method, characterized by: The following steps are involved: S1, obtain a remote sensing semantic segmentation dataset and divide the dataset into a training set for network training, a validation set for network parameter adjustment, and a test set for network evaluation; S2, a hybrid architecture expert network is designed as a feature encoder based on prior parameters; The S2 comprises the following steps: S21. Select a hybrid architecture expert network, where the network based on three semantic segmentation network architectures is selected as the hybrid architecture expert, namely: 1) Convolutional neural network expert: Select DeepLapV3+ or PSPNet or ConvNeXt, denoted as C Expert1 , C Expert2 ,…,C ExpertN ;2) Transformer network expert: select Segformer or SwinTransformer or PoolFormer, denoted as T Expert1 , T Expert2 ,…,T ExpertN ; 3) Mamba network expert, denoted as M Expert1 , M Expert2 ,…,M ExpertN ; S22, based on prior evaluation, selects the hybrid architecture expert representative. For the hybrid architecture expert network, based on prior training knowledge, the optimal network is selected in each architecture as the hybrid architecture expert representative: C Expert =Ψ(C Expert1 ,C Expert2 ,…,C ExpertN ); T Expert =Ψ(T Expert1 ,T Expert2 ,…,T ExpertN ); M Expert =Ψ(M Expert1 ,M Expert2 ,…,M ExpertN ); Among them, C Expert 、T Expert 、M Expert The best convolutional neural network, Transformer network, and Mamba network expert representatives are selected, respectively. Ψ(·) represents the network optimization strategy based on prior pre-training knowledge, that is, the model parameters with excellent training on open source or self-made datasets are used as the representative through network evaluation to obtain the highest accuracy network. S23, hybrid architecture expert visual feature encoding, uses the optimal hybrid architecture expert representative selected in S22 as the feature encoder of the architecture, extracts the visual features of the input image, and generates multi-granularity visual features one by one, namely: ； ； ； in, is the input image, 、 、 The convolutional neural network representative, Transformer network representative and Mamba network representative selected from S22 are respectively, , and are the multi-granularity features obtained, ; S3, constructs a dynamic search space for heterogeneous encoding and learns to fuse multi-granular visual features of hybrid architecture experts through adaptive search; The S3 includes the following steps: S31, constructs a dynamic search space for heterogeneous encoding. For the same-scale visual features of hybrid architecture experts, a directed acyclic graph structure is constructed as the basic cell unit, and a two-layer node layer is constructed as the search space to adaptively fuse the features of different expert networks; S32, build a hybrid architecture expert interaction module, the formula is: ； in, ,and Index values ​​used to represent features of different scales; , , It is a dynamic convolution layer, which includes 1 dynamic convolution layer and 1 normalization layer. Used to enhance features , , Adaptability to the scale of features in the is the learnable contribution weight, and the formula is: ； in, is the input feature, It is a softmax function used to calculate the contribution probability. The TopK algorithm is used to select only the top k networks to give contribution weights. S4, constructing a coarse-fine representation search space and fusing coarse-fine visual features through adaptive search learning; S5 adopts a bottom-up step-by-step upsampling strategy to restore multi-granularity visual features from coarse to fine to the original image resolution through adaptive search learning; S6 outputs the semantic segmentation mapping results and uses accuracy evaluation to evaluate the performance of the network.

2. The hybrid architecture expert-driven remote sensing semantic segmentation self-learning network search method according to claim 1 is characterized in that: The S1 includes the following: Obtain an open source or self-made remote sensing semantic segmentation dataset. The dataset is divided into a training set for network training, a validation set for network hyperparameter adjustment, and a test set for network accuracy evaluation according to a preset ratio. The dataset includes remote sensing images and labeled images.

3. The hybrid architecture expert-driven remote sensing semantic segmentation self-learning network search method according to claim 1 is characterized in that: The S4 comprises the following steps: S41, builds a coarse-to-fine-grained representation search space. For the multi-granularity features fused by hybrid architecture experts, a directed acyclic graph structure is constructed as the basic cell unit, and a multi-layer node is constructed as the search space for fusing multi-granularity features: S42, build a coarse-fine feature dynamic enhancement module, and transform the current input feature As the main feature, and the remaining granularity features as auxiliary features, that is: ； in, is a coarse-fine feature dynamic enhancement module, where is the selected main feature, For the remaining features, it includes the following steps: S421, firstly, the auxiliary features are sampled to the same scale as the main features through a dynamic learnable parameter, the formula is: ； in, ; represents a dynamic convolutional layer, represents a normalization layer, is a downsampling convolutional layer, is a 1×1 convolutional layer, represents an upsampling convolutional layer; S422, through self-learning search for coarse-fine feature fusion, enhance the multi-granularity representation capability of auxiliary features. The process is as follows: ; in, The learnable weight contribution rate is: ； in, is the input feature, It is a softmax function used to calculate the contribution probability. The TopK algorithm is used to select only the top k networks to give contribution weights. S423, based on the auxiliary features, the self-attention mechanism is used to focus on and fuse the region of interest of the main features. The process is as follows: ; in, is a linear transformation layer, is the number of feature channels, is the softmax function, is a linear layer, for The transpose of .

4. The hybrid architecture expert-driven remote sensing semantic segmentation self-learning network search method according to claim 3 is characterized in that: The S5 includes: ； ； ； ； in, Follow S42.

5. The hybrid architecture expert-driven remote sensing semantic segmentation self-learning network search method according to claim 1, characterized in that: The S6 includes: ； in, It includes an upsampling convolution layer and a softmax layer to output the probability of each category.

Images