Multi-modal image land cover classification method based on network architecture search

Abstract

The application provides a kind of multi-modal image land cover type classification method based on network architecture search, comprising S1, using feature extraction network to the input pair of optical, SAR remote sensing image is extracted features;S2, construct the multi-modal feature fusion architecture search space based on SMBO algorithm, search the best fusion strategy;S3, according to the fusion strategy, optical, SAR features are fused, and land cover type classification result is obtained.The application proposes a double-input U-shaped feature extraction network, so that the network can obtain more sufficient information for classification, and considering that the traditional artificial design deep learning feature fusion module is difficult and lacks interpretability, a multi-modal feature fusion architecture search method based on SMBO is created to explore the best combination mode between different modal features, and the optical, SAR image feature fusion efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image land cover type classification, and particularly relates to a multi-modal image land cover type classification method based on network architecture search. BACKGROUND

[0002] Remote sensing images contain rich geographic information and are widely used in military and civilian fields. Corresponding interpretation techniques have also developed rapidly. Land cover type classification is an important branch of remote sensing image application and plays an important role in fields such as urban planning, environmental monitoring and disaster response. Land cover type classification requires assigning a corresponding class to each pixel in the image, and the image interpretation requires high accuracy. Using a single earth observation data, due to the differences in imaging mechanism and imaging characteristics, it is inevitable to encounter performance bottlenecks in identifying different pixel classifications. Optical remote sensing images have high resolution and rich semantic information such as color and texture, but imaging is easily affected by adverse weather factors such as clouds and fog. Synthetic aperture radar (SAR) has all-weather and all-day observation capability, but the special imaging mechanism makes SAR images easily disturbed by speckle noise, severe geometric distortion and complex backscattering. A large number of studies have shown that matching optical and SAR images can provide complementary information for the same area. With the rapid development of earth observation technology, it has become easier to obtain a large number of multi-modal remote sensing images for the same area, which also provides conditions for using multi-modal remote sensing images for land cover type classification.

[0003] With the continuous development of deep learning, more and more neural networks are applied to multi-modal networks, and the fusion strategy becomes the key to determining the classification performance. Existing multi-modal fusion methods are mostly manually designed fusion modules, but the imaging differences between SAR images and optical images may cause their features not to be in the same semantic space, and the neural network feature extraction has poor interpretability. Artificial design of fusion modules for these features may result in unsatisfactory fusion results.

[0004] Therefore, with the increasing amount of remote sensing images, how to improve the accuracy of multi-modal remote sensing image land cover type classification has become a problem to be faced at present. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application provides a multi-modal image land cover type classification method based on network architecture search, to solve the technical problem that the existing multi-modal fusion method is mostly manually designed fusion, resulting in unsatisfactory fusion results and affecting the classification accuracy of multi-modal remote sensing image land cover type.

[0006] The application provides a multi-modal image land cover type classification method based on network architecture search, comprising:

[0007] S1, using a feature extraction network to extract features from inputted pairs of optical and SAR remote sensing images;

[0008] S2, constructing a multi-modal feature fusion architecture search space based on an SMBO algorithm to search for an optimal fusion strategy;

[0009] S3, fusing optical and SAR features according to the fusion strategy to obtain a land cover type classification result.

[0010] Optionally, the feature extraction network is used to extract features from inputted pairs of optical and SAR remote sensing images, comprising:

[0011] The pairs of matched optical and SAR remote sensing images are simultaneously inputted into a U-shaped feature extraction network, and a convolution module is used to extract features.

[0012] Optionally, the multi-modal feature fusion architecture search space based on the SMBO algorithm is constructed, comprising:

[0013] After the features are extracted by the U-shaped feature extraction network, SAR modal features {x1, x2,..., x m} and optical modal features {y1, y2,..., y n} of different levels are obtained, and the SAR modal features and the optical modal features are selected to construct a search space for fusion, and the size of the search space is represented by (MxN) L , wherein M and N respectively represent the number of features provided by the SAR and the optical modal for fusion, and L represents the number of fusion layers.

[0014] Optionally, the optimal fusion strategy is searched, comprising:

[0015] The fusion module is sampled from the search space, the network constructed by the fusion module is evaluated one by one, and then the sampling strategy is guided according to the evaluation result, and the search process is realized through repeated iteration.

[0016] Optionally, the fusion module is sampled from the search space, the network constructed by the fusion module is evaluated one by one, and then the sampling strategy is guided according to the evaluation result, and the search process is realized through repeated iteration, comprising:

[0017] Defining the maximum number of fusion layers L, the number of search iterations E search , the training epoch E train , the number of sampled sub-networks K, the training set S strain , the verification set S val ;

[0018] Build all fusion module sets of I = 1, train and evaluate all fusion modules in the fusion module set, save the fusion module parameters and evaluation results and update the agent model;

[0019] When I = n + 1, n > 1, the n + 1 layer fusion is constructed on the K fusion modules saved under I = n, the prediction is continued, the K fusion modules are trained and evaluated after being used, and the agent model is updated, and the process is repeated until I = L, and finally the multi-modal classification model is output.

[0020] Optionally, the fusion of the optical and SAR features according to the fusion strategy comprises:

[0021] According to the obtained optimal fusion combination, a high-performance multi-modal classification model is constructed to predict the input paired optical and SAR modal images and obtain accurate land cover type classification results.

[0022] Compared with the prior art, the present application has the following beneficial effects:

[0023] 1. A double-input U-shaped feature extraction network is proposed, which can obtain more sufficient information for classification.

[0024] 2. Considering that the traditional artificially designed deep learning feature fusion module is difficult and lacks interpretability, a multi-modal feature fusion architecture search method based on SMBO is created to explore the best combination mode between different modal features and improve the optical and SAR image feature fusion efficiency.

[0025] 3. The present application applies network architecture search technology to search the features extracted from different modalities by the network on the dataset, obtains the best combination mode between the features, and effectively fuses the information of optical and SAR images for land cover type classification. BRIEF DESCRIPTION OF DRAWINGS

[0026] The drawings incorporated into the specification and forming a part thereof, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows, and obviously, other drawings can also be obtained by those skilled in the art without creative labor.

[0028] Figure 1 The method flowchart of the present application;

[0029] Figure 2A flowchart of the process of using U-shaped feature extraction in the present application is shown in the figure.

[0030] Figure 3 A schematic diagram of the optical image and the SAR image in the present application is shown in the figure.

[0031] Figure 4 A schematic diagram of the U-shaped input network and the conventional UNET network in the present application is shown in the figure. DETAILED DESCRIPTION

[0032] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. The function units with the same reference numerals in the present application have the same and similar structures and functions.

[0033] Referring to Figure 1 The present application provides a method for multi-modal image land cover type classification based on network architecture search, comprising:

[0034] S1, using a feature extraction network to extract features from inputted pairs of optical and SAR remote sensing images;

[0035] S2, constructing a multi-modal feature fusion architecture search space based on an SMBO algorithm to search for the best fusion strategy;

[0036] S3, fusing the optical and SAR features according to the fusion strategy to obtain a land cover type classification result.

[0037] The specific implementation process of the present application is as follows:

[0038] S1, using a feature extraction network to extract features from inputted pairs of optical and SAR remote sensing images.

[0039] In the present embodiment, referring to Figure 2 The optical and SAR remote sensing images of the same scene and the same time are simultaneously inputted into the U-shaped feature extraction network, and the convolution module is used for feature extraction, wherein the convolution module is composed of convolution-Batch Normalization-activation layer-pooling layer. The two branches extract optical image and SAR image features respectively; referring to Figure 3We use the convolutional module shown in (a) to extract features from the SAR image, and use the convolutional module shown in (b) to extract features from the optical image. Due to the presence of speckle noise and complex scattering information interference in the SAR image, we use more convolutional layers to extract features to obtain more representative semantic features. The extracted multi-modal feature layer is sent to the architecture search space to obtain the best fusion combination method.

[0040] S2, construct a multi-modal feature fusion architecture search space based on the SMBO algorithm to search for the best fusion strategy.

[0041] In this embodiment, the focus is on introducing a multi-modal feature fusion-based architecture search network in multi-modal image processing, which greatly improves the efficiency and effect of fusion. The steps are as follows:

[0042] a1. Construct a multi-modal feature fusion architecture search space

[0043] Let Figure 1 The input of the network architecture is (x, y, z), where x represents the input image of the SAR modality, y represents the input image of the optical modality matched with it, and z represents the land cover type classification label corresponding to it.

[0044] After extracting features through the constructed U-shaped feature extraction network, SAR modality features {x1, x2, …, xm} and optical modality features {y1, y2, …, yn} of different levels will be obtained. Select this part of the feature to construct a search space for fusion. The size of the search space can be represented by M and N, respectively. M and N represent the number of features provided by the SAR and optical modalities for fusion, and L represents the number of fusion layers.

[0045] a2. Search for fusion methods based on the SMBO-based architecture search method

[0046] The search strategy process is roughly as follows: first, sample the fusion module from the search space, evaluate the network constructed by the fusion module one by one, and then guide the sampling strategy according to the evaluation results, and repeatedly iterate to realize the search process.

[0047] First, define the maximum number of fusion layers L and the number of search iterations E search parameters, L represents the establishment of L layers of fusion relationship between SAR modality features and optical modality features. The search algorithm will iterate E search times, each iteration will start from the low complexity level of the smallest fusion module, i.e. I = 1, and gradually increase the number of layers to the maximum fusion layer L during the process, and through multiple iterations to ensure the accuracy of the search.

[0048] The following takes the first iteration process as an example to introduce the entire algorithm process. First, construct the set of all fusion modules Γ1 of I = 1, and then sample a fusion module from Γ1l All fusion modules in are trained and evaluated, and the fusion module parameters and evaluation results are saved and the surrogate model is updated. In the entire search process, all modules are evaluated and trained only at the first iteration I = 1.

[0049] When I = 2, the second fusion is constructed based on the fusion modules saved at I = 1. To speed up the search, the surrogate model is used to predict all I = 2 fusion modules, and the sampling probability of each fusion module is determined according to the performance of the prediction results. Then, according to the sampling probability, K fusion modules are sampled from the current I layer fusion module set, and the K fusion modules are trained and evaluated, and the K fusion modules and the evaluation results are saved, and the surrogate model is updated with the real evaluation results.

[0050] When I = n + 1 (n > 1), the n + 1 layer fusion is constructed based on the K fusion modules saved at I = n, and the prediction is continued. The K modules are trained and evaluated and the surrogate model is updated, and the process is repeated until I = L. In the subsequent iteration process, when I = 1, the network constructed by all fusion modules will not be trained, but the processes of prediction, adoption, training, and updating are repeated. Finally, the top K best architectures are selected from the set of all adopted architectures Β as the result output, and a multi-modal classification model is obtained.

[0051] Specifically described as:

[0052] Input: maximum fusion layer number L, search iteration number E search , training epoch E train , sampling sub-network number K, training set S strain , validation set S val , SAR modality feature s, optical modality feature o

[0053] Define the fusion module set B and the corresponding real evaluation result set A;

[0054] for e = 1…E search do;

[0055] if e = 1 || l = 1 then;

[0056] sample a fusion module S1 in the fusion module set Γ l of fusion layer number 1;

[0057] construct a network model according to the features of different modalities and the fusion module S1;

[0058] M1 = descToFusionNet(S1, s, o);

[0059] train the model M1, C1 = train(M1, Sstrain ,E search );

[0060] Evaluate network model, get real evaluation result A1 = evaluate(C1, S val );

[0061] Update fusion module set B = B U B1, real evaluation result set A = A U A1;

[0062] Optimize agent model π = update(S1, A1);

[0063] for I = 1…L do;

[0064] if e = 1 || l = 1 then;

[0065] Break;

[0066] Build fusion module S' containing I layers l = addLayer(S l-1 , Γ l );

[0067] Predict performance through agent model

[0068] Calculate sampling probability of each fusion module according to predicted performance result;

[0069]

[0070] Randomly sample K fusion modules S1 = sampleK(S1, P l , k);

[0071] Build network model M1 = descToFusionNet(S1, s, o);

[0072] Train network C1 = train(M1, S strain , E seaech );

[0073] Evaluate network

[0074] Update fusion module set Update fusion module set B = B U B l , real evaluation result set A = A U A l ;

[0075] Optimize agent model π = update(S l , A l );

[0076] End, output top K best fusion modules topK(B, A, K).

[0077] Wherein, in the model training, we will first adjust the SAR and optical image to the consistent image resolution, and then 75% of the data as the training set, the rest of the data as the validation set. At the same time, in the model optimization, the Focal Loss loss function is used to reduce the interference of sample imbalance on classification accuracy. The network performance of the proxy model prediction fusion module effectively reduces the time and resources consumed in performance evaluation. At the same time, the algorithm also allocates sampling probability according to the prediction result, trains and evaluates the sampled fusion module, so as to optimize the proxy model, and avoid using the prediction result of the proxy model blindly and causing too large error with the real result.

[0078] S3, fusing the optical and SAR features according to the fusion strategy to obtain a land cover type classification result.

[0079] In this embodiment, according to the obtained optimal fusion combination, a high-performance multi-modal classification model is constructed to predict the input paired optical and SAR modal images and obtain accurate land cover type classification results.

[0080] Reference Figure 4 Taking the data set as an example, we take the average precision MPA and the average intersection over union MIOU corresponding to each category as the main indicators, and we select the UNET network commonly used in land cover type classification tasks as a control. Among them, UNET_S and UNET_O respectively represent the single modal classification network based on UNET network, and UNET_S&O represents the ordinary dual modal input classification network. It can be seen that the method proposed by us has the best performance.

[0081] It should be noted that in this paper, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a…" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0082] The foregoing is considered as illustrative only of the principles of the application. Numerous modifications and changes will readily occur to those skilled in the art, and it is intended to embrace all such modifications and changes that fall within the scope of the application. Accordingly, the application is not to be restricted in scope to the specific embodiments disclosed herein but is to be accorded the full scope that the principles and novel features request appropriately granted.

Claims

1. A method for classifying multimodal image land cover types based on network architecture search, characterized in that, include: S1. Input the paired matched light and SAR remote sensing images into the U-shaped feature extraction network and use the convolution module to extract features. S2. After feature extraction by the U-shaped feature extraction network, SAR modal features {x1, x2, ..., x} at different levels will be obtained. m } and optical modal characteristics {y1,y2,…,y n } SAR modal features and optical modal features are selected respectively to construct a search space for fusion. The size of the search space is (M×N). L The expression is defined as follows: M and N represent the number of features provided by SAR and optical modes for fusion, respectively; L represents the number of fusion layers; and the maximum number of fusion layers L and the number of search iterations E are defined. search Training epoch E train Number of sampling subnets K, training set S strain Validation set S val ; Construct a set of all fusion modules with I=1, train and evaluate all fusion modules in the set, save the fusion module parameters and evaluation results, and update the surrogate model. When I = n+1, n>1, the (n+1)th fusion layer will be built on the K fusion modules saved under I = n, and prediction will continue. After using K fusion modules, the surrogate model will be trained, evaluated and updated. This process will be repeated until I = L, and finally the multimodal classification model will be output. S3. Based on the fusion strategy, the light and SAR features are fused to obtain the land cover type classification results.

2. The method for multimodal image land cover type classification based on network architecture search as described in claim 1, characterized in that, The process of fusing light and SAR features according to the fusion strategy to obtain land cover type classification results includes: Based on the optimal fusion combination, a high-performance multimodal classification model is constructed to predict the input paired light and SAR modal images, thereby obtaining accurate land cover type classification results.

Images