Multispectral and Panchromatic Satellite Image Land Classification Method Based on Global Collaborative Fusion

Abstract

The present invention relates to a method for classifying the surface of multi-spectral and panchromatic satellite images based on global collaborative fusion, including: obtaining multi-spectral satellite remote sensing images and panchromatic satellite remote sensing images of the study area, and performing surface element sample annotation to obtain a training sample map; constructing a globally collaborative deep convolutional neural network for classifying surface elements of multi-spectral and panchromatic satellite remote sensing images, which network includes two single-source branches and one multi-source branch; inputting the multi-spectral and panchromatic satellite remote sensing images and the training sample map into the network for training to obtain a trained network model; obtaining the multi-spectral and panchromatic satellite remote sensing images to be classified and inputting them into the network model for prediction to obtain probability classification maps of each network branch; performing decision-level fusion on each probability classification map through probability weighting to obtain the final classification map of the surface elements in the study area. Compared with the prior art, the present invention has the advantages of high classification accuracy, good result robustness, and fast prediction speed, etc.

Description

Technical Field

[0001] The present invention relates to the field of feature-level fusion and classification of multi-source remote sensing images, and particularly to a method for classifying the surface of multi-spectral and panchromatic satellite images based on global collaborative fusion. Background Art

[0002] With the continuous enrichment of remote sensing platforms and sensor types, the classification and detection of surface cover by fusing multi-modal remote sensing images (such as hyperspectral and LiDAR, multi-spectral and SAR, multi-spectral and panchromatic images) can effectively improve the classification and recognition accuracy of surface elements. Among the numerous fusions of multi-modal remote sensing images, multi-spectral and panchromatic remote sensing images are the pair of image data that are most easily acquired simultaneously. Although the spatial resolution of multi-spectral images is low, the multiple spectral information they contain can be used to distinguish different types of ground objects; conversely, panchromatic images only contain one band, but their high spatial resolution helps to accurately describe the shape boundaries of objects and their spatial relationships. Therefore, the joint use of multi-spectral and panchromatic images can make full use of the advantages of both types of data.

[0003] The current fusion classification of multi-spectral and panchromatic remote sensing images has two defects: 1) Classification method: Using a slice-based per-pixel classification method, slices generate redundancy, increasing the data processing burden, and slices destroy the integrity of ground objects. The limitations of slices prevent the deep network from freely obtaining context information. 2) Fusion strategy: Most existing fusion methods only consider the relatively rich cross-fusion features of multi-spectral and panchromatic bands. Although the cross-fusion features are relatively rich, they may destroy the representative features of single-source images. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above-mentioned defects existing in the prior art, and provide a method for classifying the surface of multi-spectral and panchromatic satellite images based on global collaborative fusion, which does not require slicing and simultaneously considers the features of single multi-spectral and panchromatic images, as well as their cross-fusion features.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for classifying the surface of multi-spectral and panchromatic satellite images based on global collaborative fusion, comprising the following steps:

[0007] S1: Obtain multi-spectral satellite remote sensing images and panchromatic satellite remote sensing images of the study area, and perform surface element sample annotation to obtain a training sample map of the study area;

[0008] S2: Construct a deep convolutional neural network for classifying surface elements of multi-spectral and panchromatic satellite remote sensing images with global collaboration, which network includes two single-source branches and one multi-source branch;

[0009] S3: Input the multi-spectral satellite remote sensing image, panchromatic satellite remote sensing image of the study area obtained in step S1, and the corresponding training sample map into the deep convolutional neural network for classifying surface elements of multi-spectral and panchromatic satellite remote sensing images with global collaboration constructed in step S2 for network training to obtain a trained network model.

[0010] S4: Obtain the multi-spectral satellite remote sensing image and panchromatic satellite remote sensing image of the study area to be classified, and input them into the trained network model in step S3 for prediction to obtain the probability classification maps of each network branch.

[0011] S5: Perform decision-level fusion on the probability classification maps of the three network branches in step S4 by means of probability weighting to obtain the final classification map of surface elements in the study area.

[0012] Further, in step S1, the size of the multi-spectral satellite remote sensing image is H×W×B, the size of the panchromatic satellite remote sensing image is nH×nW×1, and the size of the training sample map is nH×nW×1, where H, W, and B are the height, width, and number of bands of the multi-spectral image respectively, and n is the multiple of the size of the panchromatic image relative to the size of the multi-spectral image.

[0013] Further, the two single-source branches include a single-source multi-spectral deep and shallow feature fusion branch and a single-source panchromatic deep and shallow feature fusion branch, the multi-source branch is a multi-scale multi-spectral and panchromatic cross-feature fusion branch, and the deep convolutional neural network for classifying surface elements of multi-spectral and panchromatic satellite remote sensing images also includes an adaptive weighted cross-entropy loss for the three network branches.

[0014] Further, the single-source multi-spectral deep and shallow feature fusion branch includes a multi-spectral encoding module and a multi-spectral decoding module. The multi-spectral encoding module includes a multi-spectral input layer, an encoding convolutional block MS1, a spectral attention module 1, an encoding convolutional block MS2, a spectral attention module 2, an encoding downsampling layer MS1, and an encoding convolutional block MS3 connected in sequence; the multi-spectral decoding module includes a decoding convolutional layer MS1, a decoding upsampling layer MS1, a decoding convolutional layer MS2, a decoding upsampling layer MS2, a decoding convolutional layer MS3, a decoding upsampling layer MS3, a decoding convolutional layer MS4, and a multi-spectral output layer connected in sequence.

[0015] The single-source panchromatic light and dark feature fusion branch includes a panchromatic encoding module and a panchromatic decoding module. The panchromatic encoding module includes a panchromatic input layer, an encoding convolutional block PAN1, an encoding downsampling layer PAN1, an encoding convolutional block PAN2, an encoding downsampling layer PAN2, an encoding convolutional block PAN3, an encoding downsampling layer PAN3, and an encoding convolutional block PAN4 connected in sequence. The panchromatic decoding module includes a decoding convolutional layer PAN1, a decoding upsampling layer PAN1, a decoding convolutional layer PAN2, a decoding upsampling layer PAN2, a decoding convolutional layer PAN3, a decoding upsampling layer PAN3, a decoding convolutional layer PAN4, and a panchromatic output layer connected in sequence.

[0016] Further, the encoding convolutional block MS1, the decoding convolutional layer MS1, the encoding convolutional block MS2, the decoding convolutional layer MS2, the encoding convolutional block MS3, the decoding convolutional layer MS3, the encoding convolutional block PAN1, the decoding convolutional layer PAN1, the encoding convolutional block PAN2, the decoding convolutional layer PAN2, the encoding convolutional block PAN3, the decoding convolutional layer PAN3, the encoding convolutional block PAN4, and the decoding convolutional layer PAN4 all include a convolutional layer, a group normalization layer, and an activation layer.

[0017] Further, the spectral attention module 1 and the spectral attention module 2 adopt a squeeze-and-excitation attention module.

[0018] Further, in the multi-scale multi-spectral and panchromatic cross-feature fusion branch, the shallow features and deep features of the multi-spectral branch are fused by adding the encoding convolutional block MS2 to the decoding upsampling layer MS1; the shallow features and deep features of the panchromatic branch are fused by adding the encoding convolutional block PAN3 to the decoding upsampling layer PAN1, adding the encoding convolutional block PAN2 to the decoding upsampling layer PAN2, and adding the encoding convolutional block PAN1 to the decoding upsampling layer PAN3.

[0019] A side connection convolutional layer is added after the shallow features of the multi-spectral branch and the shallow features of the panchromatic branch to adjust the number of channels of the shallow features to be the same as that of the corresponding deep features.

[0020] Further, the multi-scale multi-spectral and panchromatic cross-feature fusion branch includes an addition operation layer 1, a convolutional layer Fusion1, an upsampling layer Fusion1, an addition operation layer 2, a convolutional layer Fusion2, an upsampling layer Fusion2, an addition operation layer 3, a convolutional layer Fusion3, an upsampling layer Fusion3, a convolutional layer Fusion4, and a fusion output layer connected in sequence.

[0021] The addition operation layer 1 includes a decoded convolutional layer MS1 + a decoded convolutional layer PAN1. The addition operation layer 2 includes a decoded convolutional layer MS2 + a decoded convolutional layer PAN2 + an upsampling layer Fusion1. The addition operation layer 3 includes a decoded convolutional layer MS3 + a decoded convolutional layer PAN3 + an upsampling layer Fusion2.

[0022] Further, the calculation expression of the overall loss of the global collaborative multi-spectral and panchromatic satellite remote sensing image surface element classification deep convolutional neural network is:

[0023] L total = λ1L MS + λ2L PAN + λ3L Fusion

[0024] In the formula, L total is the overall loss of the global collaborative multi-spectral and panchromatic satellite remote sensing image surface element classification deep convolutional neural network. L MS , L PAN and L Fusion are the losses of the single-source multi-spectral deep and shallow feature fusion branch, the single-source panchromatic deep and shallow feature fusion branch, and the multi-scale multi-spectral and panchromatic cross-feature fusion branch respectively. λ1, λ2, and λ3 are the weight values corresponding to the losses;

[0025] The calculation expression of the adaptive weighted cross-entropy loss of the three network branches is:

[0026]

[0027] In the formula, C is the number of surface element classes marked in the study area, y is the training sample map, is the probability that the j-th channel at the image position (u, v) of the last convolutional output layer of each branch belongs to class C, and

[0028] Further, in step S5, the process of performing decision-level fusion on the probability classification maps of the three network branches in step S4 includes:

[0029] Denote and as the surface element probability classification maps of the single-source multi-spectral deep and shallow feature fusion branch, the single-source panchromatic deep and shallow feature fusion branch, and the multi-scale multi-spectral and panchromatic cross-feature fusion branch respectively, and calculate the weighted probability value of each channel of the network model:

[0030]

[0031] Wherein, λ1, λ2, and λ3 are the final parameters obtained after training the network model in the step S3, C is the number of surface feature classes labeled in the study area, and by stacking the probability maps of each channel, the final surface feature probability classification map is obtained:

[0032]

[0033] At each pixel position of the final surface feature probability classification map, the class represented by the channel with the maximum probability score is taken as the final class label, and the final surface feature classification map is obtained:

[0034]

[0035] Wherein, is the final surface feature classification map.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] (1) Different from the existing methods that only use pixel-level or fused features of multispectral and panchromatic bands for classification, the present invention proposes a new fusion architecture. This architecture simultaneously considers the shallow and deep features of multispectral and panchromatic images and their multi-scale cross-modal features. An adaptive loss weighted fusion strategy is designed to calculate the total loss of the single-source multispectral and panchromatic shallow and deep feature fusion branches and the multi-scale multispectral and panchromatic feature cross-modal fusion branches, and a probability weighted decision-level fusion strategy is constructed to further improve the classification performance.

[0038] (2) The present invention proposes a global non-slice-based classification scheme, which is not limited by the slice size, takes into account the spatial integrity and connectivity of the image features, and can freely obtain rich context information. At the same time, it avoids the redundancy generated by the slice-based method, thereby reducing the computational amount.

[0039] (3) In the training process, the present invention uses both sample area and non-sample area data, which can more comprehensively model the features in the complex scene of the image, and at the same time meets the requirements of the deep learning network for a large amount of training data. Especially in the case of extremely small samples, the advantages of the present invention are more obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic flow chart of a method for classifying surface features of multispectral and panchromatic satellite remote sensing images based on a global collaborative fusion network provided by the present invention;

[0041] Figure 2 is a schematic diagram of the structure and parameters of the neural network of the present invention;

[0042] Figure 3It is the true color composite map of the multi-spectral image in the embodiment of the present invention;

[0043] Figure 4 It is the schematic diagram of the panchromatic image in the embodiment of the present invention;

[0044] Figure 5 It is the schematic diagram of the ground truth reference map in the embodiment of the present invention;

[0045] Figure 6 It is the schematic diagram of the classification result of surface elements of the DMIL (Deep multiple instance learning) method in the embodiment of the present invention;

[0046] Figure 7 It is the schematic diagram of the classification result of surface elements of the MultiResoLCC (Multi-Resolution land covervlassification) method in the embodiment of the present invention;

[0047] Figure 8 It is for the embodiment of the present invention, the schematic diagram of the classification result of surface elements of the CRHFF (F EMAP*PAN ) method;

[0048] Figure 9 It is the schematic diagram of the classification result of surface elements of the GAFnet (Group attention fusion network) method in the embodiment of the present invention;

[0049] Figure 10 It is the schematic diagram of the classification result of surface elements of the method proposed in the embodiment of the present invention in the embodiment of the present invention. Detailed implementation manners

[0050] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Generally, the components of the embodiments of the present invention described and illustrated herein can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0052] It should be noted that similar reference numerals and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined or explained in subsequent figures.

[0053] Embodiment 1

[0054] As Figure 1 shown, this embodiment provides a method for classifying the surface of multi-spectral and panchromatic satellite images based on global collaborative fusion, which specifically includes the following steps:

[0055] S1. Obtain multi-spectral and panchromatic satellite remote sensing images of the study area, and perform annotation of surface element samples to obtain a training sample map of the study area.

[0056] Among them, the size of the multi-spectral satellite remote sensing image is H×W×B, the size of the panchromatic satellite remote sensing image is nH×nW×1, and the size of the training sample map is nH×nW×1, where H, W, and B are the height, width, and number of bands of the multi-spectral image respectively, and n is the multiple of the size of the panchromatic image relative to the size of the multi-spectral image. The unannotated areas in the training sample map are represented by 0, and the annotated categories are numbered starting from 1 until the Cth category. As Figure 3 is the true color composite map of the multi-spectral image, Figure 4 is the schematic diagram of the panchromatic image, Figure 5 is the schematic diagram of the ground truth reference map.

[0057] S2. Construct a deep convolutional neural network for classifying surface elements of multi-spectral and panchromatic satellite remote sensing images with global collaboration, which includes two single-source and one multi-source branch fusion modules.

[0058] Among them, the two single-source branches are the single-source multi-spectral deep and shallow feature fusion branch and the single-source panchromatic deep and shallow feature fusion branch respectively, and the multi-source branch is the multi-scale multi-spectral and panchromatic cross-feature fusion branch, and an adaptive weighted cross-entropy loss for the three branches is constructed.

[0059] The schematic diagram and parameters of the deep convolutional neural network for classifying surface elements of multi-spectral and panchromatic satellite remote sensing images with global collaboration are shown in Figure 2 , and in the following description, the names of the encoding module and the decoding module in the multi-spectral encoding and decoding module and the panchromatic encoding and decoding module are not distinguished, but they can be uniquely determined according to the attachment Figure 2 for the specific object they represent.

[0060] Among them, the single-source multi-spectral shallow and deep feature fusion branch includes an encoding module and a decoding module. The encoding module successively includes an input layer, a convolutional block MS1, a spectral attention module 1, a convolutional block MS2, a spectral attention module 2, a downsampling layer MS1, and a convolutional block MS3. The decoding module successively includes a convolutional layer MS1, an upsampling layer MS1, a convolutional layer MS2, an upsampling layer MS2, a convolutional layer MS3, an upsampling layer MS3, a convolutional layer MS4, and an output layer.

[0061] The single-source panchromatic shallow and deep feature fusion branch includes an encoding module and a decoding module. The encoding module successively includes an input layer, a convolutional block PAN1, a downsampling layer PAN1, a convolutional block PAN2, a downsampling layer PAN2, a convolutional block PAN3, a downsampling layer PAN3, and a convolutional block PAN4. The decoding module successively includes a convolutional layer PAN1, an upsampling layer PAN1, a convolutional layer PAN2, an upsampling layer PAN2, a convolutional layer PAN3, an upsampling layer PAN3, a convolutional layer PAN4, and an output layer.

[0062] Among them, the convolutional block MS1, convolutional block MS2, convolutional block MS3, convolutional block PAN1, convolutional block PAN2, convolutional block PAN3, and convolutional block PAN4 all include a convolutional layer, a group normalization layer, and an activation layer. The spectral attention module 1 and spectral attention module 2 use the squeeze-and-excitation (SE) attention module.

[0063] In the single-source multi-spectral and panchromatic shallow and deep feature fusion branches, the shallow features and deep features of the multi-spectral branch are fused by addition, that is, convolutional block MS2 + upsampling layer MS1; and the shallow features and deep features of the panchromatic branch are fused by addition, that is, convolutional block PAN3 + upsampling layer PAN1, convolutional block PAN2 + upsampling layer PAN2, convolutional block PAN1 + upsampling layer PAN3. In order to enable the addition of shallow and deep features, a side connection convolutional layer is added after the multi-spectral and panchromatic shallow features to adjust the number of channels of the shallow features to be the same as that of the corresponding deep features.

[0064] The multi-scale multi-spectral and panchromatic cross-feature fusion branch successively includes an addition operation layer 1 (convolutional layer MS1 + convolutional layer PAN1), a convolutional layer Fusion1, an upsampling layer Fusion1, an addition operation layer 2 (convolutional layer MS2 + convolutional layer PAN2 + upsampling layer Fusion1), a convolutional layer Fusion2, an upsampling layer Fusion2, an addition operation layer 3 (convolutional layer MS3 + convolutional layer PAN3 + upsampling layer Fusion2), a convolutional layer Fusion3, an upsampling layer Fusion3, a convolutional layer Fusion4, and an output layer.

[0065] Figure 2Inside the dashed box are the specific parameters of the network. Taking [3×3]:64 as an example, it means that the kernel size of the convolution is 3×3 and the number of feature maps is 64. r is the compression ratio of the spectral attention module SE, and Interpolation(2) indicates the nearest neighbor interpolation upsampling method with a factor of 2. For downsampling, a convolutional layer with a stride of 2 is used instead of the max pooling layer.

[0066] After the network architecture is built, start constructing the adaptive weighted cross-entropy loss for the three network branches:

[0067] L total = λ1L MS + λ2L PAN + λ3L Fusion

[0068] where L MS , L PAN and L Fusion are the losses of the single-source multi-spectral deep and shallow feature fusion branch, the single-source panchromatic deep and shallow feature fusion branch, and the multi-scale multi-spectral and panchromatic cross-feature fusion branch respectively. λ1, λ2, and λ3 are the weight values corresponding to their losses. The losses L MS , L PAN and L Fusion are calculated as follows:

[0069]

[0070] where C is the number of surface feature classes labeled in the study area, y is the training sample map, is the probability that the j-th channel at the image position (u, v) of the last convolutional output layer of each branch belongs to class C, and

[0071] S3. Input the multi-spectral and panchromatic satellite remote sensing images of the study area and the corresponding training sample maps into the globally collaborative multi-spectral and panchromatic satellite remote sensing image surface feature classification deep convolutional neural network constructed in step S2, and obtain the trained network model through network training.

[0072] The network training uses the Adam (Adaptive Momentum Estimation) optimizer, the learning rate is set to 0.0001, the number of network training rounds is 1000, and the batch size is set to 1. The parameters λ1, λ2, and λ3 are set as trainable parameters, and their initial values are all normalized to [0, 1] and automatically adjusted during the network training process.

[0073] S4. Input the multi-spectral and panchromatic satellite remote sensing images of the study area into the network model trained in step S3 for prediction to obtain the probability classification maps of the three network branches.

[0074] S5. Perform decision-level fusion on the classification maps of the three network branches in step S4 through probability weighting to obtain the final classification map of surface features in the study area.

[0075] Specifically, perform decision-level fusion on the classification maps of the three network branches in step S4 through probability weighting. Denote and as the probability classification maps of surface features of the single-source multi-spectral deep and shallow feature fusion branch, the single-source panchromatic deep and shallow feature fusion branch, and the multi-scale multi-spectral and panchromatic cross-feature fusion branch respectively. Then the weighted probability value of each channel is expressed as:

[0076]

[0077] where λ1, λ2, and λ3 are the final parameters obtained after network training in step S3. By stacking the probability maps of each channel, obtain the final probability classification map of surface features:

[0078]

[0079] On the final probability classification map, take the category represented by the channel with the maximum probability score at each pixel position as the final category label to obtain the final classification map of surface features:

[0080]

[0081] In specific implementation, select the domestic high-resolution Gaofen-2 remote sensing image obtained in January 2015 in a certain area. Figures 3 to 5 The true color composite map of MS data, the PAN image, and the ground truth reference map are given. This dataset contains 5 surface feature categories: buildings, roads, water bodies, trees, and grasslands. Table 1 provides detailed information on the reference samples of each surface feature.

[0082] Table 1 Introduction to the total samples and the number of experimental training and test samples

[0083]

[0084]

[0085] To compare the performance of different methods, select the overall accuracy (OA for short), average accuracy (AA for short), and Kappa coefficient (Kappa for short) as evaluation indicators.

[0086] Such as Figures 6 to 10As shown, the results with the highest evaluation accuracy for 10 times of different methods are given, and the same training samples are used for each method. To compare the robustness of different methods, the standard deviation of the 10 results is given. The value after "±" in Table 2 represents the standard deviation. It can be seen that the standard deviation of the method proposed in the present invention is closest to that of the lowest CRHFF(F EMAP*PAN ) method, and it has better robustness.

[0087] Table 2 Precision evaluation results of different methods

[0088]

[0089] As can be seen from Table 2, when comparing the two methods without slicing, the average OA, AA, and Kappa values of the method proposed in the present invention are 98.97%, 99.27%, and 98.45% respectively, which are 4.64%, 2.68%, and 6.9% higher than those of GAFnet respectively. Especially in the building and road categories, compared with the GAFnet method, the method of the present invention has achieved significant improvement, and the class accuracies have increased by 6.61% and 6.31% respectively. Although among the slicing-based classification methods, the CRHFF(F EMAP*PAN ) method is superior to other methods, the performance of the method proposed in the present invention is higher than that of CRHFF(F EMAP*PAN ), especially in the two easily confused categories of buildings and roads, and the classification accuracies have increased by 0.82% and 2.33% respectively. From Figures 6 to 10 the visualization diagrams of the classification results of various methods, it can also be seen that the present invention has produced the best classification results, especially in the building and road categories, obtaining more homogeneous interiors and more complete building boundaries.

[0090] In summary, the results of a series of qualitative and quantitative experimental analyses show that the method for classifying surface elements of multi-spectral and panchromatic satellite remote sensing images based on the global collaborative fusion network proposed in the present invention can achieve higher-precision classification of surface elements compared with other methods, and has obvious advantages in accurately depicting the geometric boundaries of surface elements and the homogeneity of internal spectra, and has high robustness.

[0091] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations according to the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field based on the concept of the present invention through logical analysis, reasoning, or limited experiments on the basis of the prior art should be within the protection scope determined by the claims.

Claims

1. A method for classifying the surface of multi-spectral and panchromatic satellite images based on global collaborative fusion, characterized in that, It includes the following steps: S1: Obtain the multispectral satellite remote sensing image and the panchromatic satellite remote sensing image of the study area, and perform surface feature sample annotation to obtain the training sample map of the study area; S2: Construct a globally collaborative deep convolutional neural network for surface feature classification of multispectral and panchromatic satellite remote sensing images, which network includes two single-source branches and one multi-source branch; S3: Input the multispectral satellite remote sensing image and the panchromatic satellite remote sensing image of the study area obtained in step S1 and the corresponding training sample map into the globally collaborative deep convolutional neural network for surface feature classification of multispectral and panchromatic satellite remote sensing images constructed in step S2 to perform network training and obtain the trained network model; S4: Obtain the multispectral satellite remote sensing image and the panchromatic satellite remote sensing image of the study area to be classified, and input them into the network model trained in step S3 for prediction to obtain the probability classification maps of each network branch; S5: Perform decision-level fusion on the probability classification maps of the three network branches in step S4 by means of probability weighting to obtain the final surface feature classification map of the study area; The two single-source branches include a single-source multispectral deep and shallow feature fusion branch and a single-source panchromatic deep and shallow feature fusion branch. The multi-source branch is a multi-scale multispectral and panchromatic cross-feature fusion branch. The deep convolutional neural network for surface feature classification of multispectral and panchromatic satellite remote sensing images also includes the adaptive weighted cross-entropy loss of the three network branches; The calculation expression of the overall loss of the globally collaborative deep convolutional neural network for surface feature classification of multispectral and panchromatic satellite remote sensing images is: L total = λ1L MS + λ2L PAN + λ3L Fusion where L total is the loss of the overall deep convolutional neural network for classifying surface elements of global collaborative multi-spectral and panchromatic satellite remote sensing images, L MS , L PAN and L Fusion are the losses of the single-source multi-spectral deep and shallow feature fusion branch, the single-source panchromatic deep and shallow feature fusion branch, and the multi-scale multi-spectral and panchromatic cross-feature fusion branch, respectively, and λ1, λ2, and λ3 are the weight values corresponding to the losses; The calculation expression of the adaptive weighted cross-entropy loss of the three network branches is: where C is the number of surface feature classes marked in the study area, H and W are the height and width of the multispectral satellite remote sensing image, y is the training sample map, is the probability that the j-th channel of the last convolutional output layer of each branch belongs to class C at the image position (u, v), and 2. A method for classifying the surface of multi-spectral and panchromatic satellite images based on global collaborative fusion according to claim 1, characterized in that In step S1, the size of the multispectral satellite remote sensing image is H×W×B, the size of the panchromatic satellite remote sensing image is nH×nW×1, and the size of the training sample map is nH×nW×1, where B is the number of bands of the multispectral image and n is the multiple of the size of the panchromatic image relative to the size of the multispectral image.

3. A method for classifying the surface of multi - spectral and panchromatic satellite images based on global collaborative fusion according to claim 1, characterized in that, The single-source multispectral deep and shallow feature fusion branch includes a multispectral encoding module and a multispectral decoding module. The multispectral encoding module includes a sequentially connected multispectral input layer, encoding convolutional block MS1, spectral attention module 1, encoding convolutional block MS2, spectral attention module 2, encoding downsampling layer MS1, and encoding convolutional block MS3. The multispectral decoding module includes a sequentially connected decoding convolutional layer MS1, decoding upsampling layer MS1, decoding convolutional layer MS2, decoding upsampling layer MS2, decoding convolutional layer MS3, decoding upsampling layer MS3, decoding convolutional layer MS4, and multispectral output layer; The single-source panchromatic light and dark feature fusion branch includes a panchromatic encoding module and a panchromatic decoding module. The panchromatic encoding module includes a panchromatic input layer, an encoding convolutional block PAN1, an encoding downsampling layer PAN1, an encoding convolutional block PAN2, an encoding downsampling layer PAN2, an encoding convolutional block PAN3, an encoding downsampling layer PAN3, and an encoding convolutional block PAN4 connected in sequence. The panchromatic decoding module includes a decoding convolutional layer PAN1, a decoding upsampling layer PAN1, a decoding convolutional layer PAN2, a decoding upsampling layer PAN2, a decoding convolutional layer PAN3, a decoding upsampling layer PAN3, a decoding convolutional layer PAN4, and a panchromatic output layer connected in sequence.

4. A method for classifying the surface of multi - spectral and panchromatic satellite images based on global collaborative fusion according to claim 3, characterized in that, The encoding convolutional block MS1, the decoding convolutional layer MS1, the encoding convolutional block MS2, the decoding convolutional layer MS2, the encoding convolutional block MS3, the decoding convolutional layer MS3, the encoding convolutional block PAN1, the decoding convolutional layer PAN1, the encoding convolutional block PAN2, the decoding convolutional layer PAN2, the encoding convolutional block PAN3, the decoding convolutional layer PAN3, the encoding convolutional block PAN4, and the decoding convolutional layer PAN4 all contain a convolutional layer, a group normalization layer, and an activation layer.

5. A method for classifying the surface of multi - spectral and panchromatic satellite images based on global collaborative fusion according to claim 3, characterized in that, The spectral attention module 1 and the spectral attention module 2 adopt a squeeze-and-excitation attention module.

6. A method for classifying the surface of multi - spectral and panchromatic satellite images based on global collaborative fusion according to claim 3, characterized in that, In the multi-scale multispectral and panchromatic cross-feature fusion branch, the shallow features and deep features of the multispectral branch are fused by adding the encoding convolutional block MS2 to the decoding upsampling layer MS1. The shallow features and deep features of the panchromatic branch are fused by adding the encoding convolutional block PAN3 to the decoding upsampling layer PAN1, adding the encoding convolutional block PAN2 to the decoding upsampling layer PAN2, and adding the encoding convolutional block PAN1 to the decoding upsampling layer PAN3. A side connection convolutional layer is added after the shallow features of the multispectral branch and the shallow features of the panchromatic branch to adjust the number of channels of the shallow features so that it is the same as the number of channels of the corresponding deep features.

7. A method for classifying the surface of multi-spectral and panchromatic satellite images based on global collaborative fusion according to claim 3, characterized in that, The multi-scale multispectral and panchromatic cross-feature fusion branch includes an addition operation layer 1, a convolutional layer Fusion1, an upsampling layer Fusion1, an addition operation layer 2, a convolutional layer Fusion2, an upsampling layer Fusion2, an addition operation layer 3, a convolutional layer Fusion3, an upsampling layer Fusion3, a convolutional layer Fusion4, and a fusion output layer connected in sequence. The addition operation layer 1 includes the decoding convolutional layer MS1 + the decoding convolutional layer PAN1. The addition operation layer 2 includes the decoding convolutional layer MS2 + the decoding convolutional layer PAN2 + the upsampling layer Fusion1. The addition operation layer 3 includes the decoding convolutional layer MS3 + the decoding convolutional layer PAN3 + the upsampling layer Fusion2.

8. A method for classifying the surface of multi - spectral and panchromatic satellite images based on global collaborative fusion according to claim 1, characterized in that, In step S5, the process of decision-level fusion of the probability classification maps of the three network branches in step S4 includes: Denote and as the probability classification maps of surface elements for the single-source multi-spectral deep and shallow feature fusion branch, the single-source panchromatic deep and shallow feature fusion branch, and the multi-scale multi-spectral and panchromatic cross-feature fusion branch respectively, and calculate the weighted probability values of each channel of the network model: where λ1, λ2, and λ3 are the final parameters obtained after training the network model in step S3, C is the number of types of surface elements labeled in the study area, and the final surface element probability classification map is obtained by stacking the probability maps of each channel. At each pixel position of the final surface element probability classification map, the category represented by the channel with the maximum probability score is taken as the final category label, and the final surface element classification map is obtained: In the formula, is the final surface feature classification map.

Images