Remote sensing image scene classification system and method based on information fusion migration

Abstract

The present invention relates to the field of image processing technology, and in particular to a remote sensing image scene classification system and method based on information fusion migration. A source domain image is input into a multi-scale feature extraction branch to obtain source domain features, and a target domain image is input into the multi-scale feature extraction branch to obtain target domain features; a convolution operation and a multi-scale expansion attention mechanism are used to combine local features with global features, and the feature representation capability in a conditional adversarial domain adaptation process is effectively improved; the source domain features and the target domain features are jointly input into a conditional adversarial loss branch, and training is performed through a conditional adversarial domain adaptation method to reduce the feature distribution difference between the source domain and the target domain, thereby realizing classification of remote sensing scenes on an unlabeled data set and ensuring classification performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a remote sensing image scene classification system and method based on information fusion migration. Background Art

[0002] As one of the core tasks of remote sensing information processing and analysis, remote sensing image scene classification aims to use images or data acquired by high-altitude platforms such as satellites and drones to extract features of remote sensing scenes with complex semantic information through methods such as convolutional neural networks or transformers. These features are used to classify scenes into different categories, thereby automatically identifying and dividing surface cover types such as forests, farmlands, and airports. This helps to extract information from massive remote sensing images and is applied to tasks such as target detection, semantic segmentation, and change detection, providing strong technical support for further decision-making and planning.

[0003] With the continuous development of aerospace technology, the amount of remote sensing image data has increased rapidly. Although deep learning methods have achieved good results in processing remote sensing image tasks, there is a lack of high-quality labeled data sets to train deep learning methods. In practical applications, due to the influence of factors such as sensors, geographical locations, and imaging conditions, the distribution of training and test data may be different, that is, domain distribution differences. Therefore, it is necessary to study how to use existing labeled data sets to classify newly emerging unlabeled data sets. Domain adaptation, as a representative method in transfer learning, mainly transfers knowledge by reducing the distribution difference between the source domain and the target domain, so that limited samples can be used to improve the generalization ability of the model.

[0004] In recent years, more and more research work has focused on domain adaptation methods for processing remote sensing images. Processing remote sensing images on unlabeled datasets can solve problems caused by differences in domain distribution. Currently, based on transfer learning methods, deep learning models can learn knowledge between different fields. Remote sensing images can be classified through convolutional neural networks, deep feature alignment neural networks, or deep adversarial domain adaptation methods, or transformers and their variants can be applied to semantic segmentation tasks of remote sensing images. This not only improves the performance of the model, but also alleviates the differences in data distribution between domains.

[0005] In conventional image processing tasks, image features are divided into low-level, middle-level and high-level features. Low-level features include shape, texture and color, which can directly reflect visual information; middle-level features are semantic features, which can connect low-level and high-level features to reduce the semantic gap; high-level features reflect the overall semantic concept of the scene, involving behavior, emotion and scene semantics. For remote sensing images, remote sensing scenes have the diversity of objects and the complexity of spatial distribution. Classification only through low-level local features of the scene cannot obtain the global scene visual information and high-level semantic content. Therefore, in order to extract more effective scene features, it is necessary to establish a connection between the global and local features of the scene. Summary of the invention

[0006] In view of this, the invention aims to provide a remote sensing image scene classification system and method based on information fusion migration, which combines the local features based on the convolutional neural network and the global features based on the transformer by fusing the convolutional neural network and the improved transformer, and utilizes the convolution operation and the multi-scale expansion attention mechanism to effectively improve the feature representation ability in the conditional adversarial domain adaptation process, and further trains through the conditional adversarial domain adaptation method to reduce the feature distribution difference between the source domain and the target domain, so as to realize the classification of remote sensing scenes on unlabeled data sets and ensure the classification performance.

[0007] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0008] A remote sensing image scene classification system based on information fusion migration includes a multi-scale feature extraction branch and a conditional adversarial loss branch; wherein a source domain image is input into the multi-scale feature extraction branch to obtain source domain features, and a target domain image is input into the multi-scale feature extraction branch to obtain target domain features; the source domain features and the target domain features are input into the conditional adversarial loss branch together, thereby completing the label matching of the source domain image and the target domain image; the multi-scale feature extraction branch includes a residual network sub-branch, and a multi-scale expansion sub-branch for performing sliding window expansion attention operations of different scales; the source domain image or the target domain image is respectively input into the residual network sub-branch and the multi-scale expansion sub-branch, and the features output by the residual network sub-branch and the multi-scale expansion sub-branch are combined to obtain the source domain features or the target domain features accordingly.

[0009] Furthermore, in the multi-scale expansion sub-branch, the input image is passed through a shallow feature extraction module to obtain shallow features; the multi-scale expansion module performs sliding window expansion attention operations of different scales on the shallow features to obtain primary multi-scale features; the primary multi-scale features are transformed to obtain a multi-scale feature sequence; in the residual network sub-branch, the input image is passed through multiple cascaded residual modules to obtain primary residual features, and the primary residual features are transformed to obtain a residual feature sequence; the residual feature sequence is multiplied by the corresponding elements of the primary multi-scale features to obtain residual features; the multi-scale feature sequence is multiplied by the corresponding elements of the primary residual features to obtain multi-scale features; the multi-scale features are added to the corresponding elements of the residual features to obtain the output features of the multi-scale feature extraction branch.

[0010] Furthermore, in the shallow feature extraction module, multiple stacked convolution blocks are used to segment the input image; the segmented image blocks are input into the normalization layer to obtain shallow features.

[0011] Furthermore, in the multi-scale dilation module, the channels are divided into multiple different heads, and sliding window dilation attention operations are performed with different dilation rates in different heads. The obtained features are concatenated and linearly transformed, and the results are processed by a multi-layer perceptron to obtain multi-scale attention features. The dilation rates in sliding window dilation attention operations of different scales are different.

[0012] Furthermore, in the feature transformation, the input features are pooled by the average pooling layer and then input into the softmax layer to obtain the corresponding feature sequence.

[0013] Furthermore, in the conditional adversarial loss branch, the source classifier is used for feature learning and the domain discriminator is used for domain identification.

[0014] A remote sensing image scene classification method based on information fusion migration includes the following steps:

[0015] S1: Obtain a dataset, and obtain source domain images and their labels, and target domain images from the dataset;

[0016] S2: Constructing a remote sensing image scene classification system based on information fusion migration as provided by the present invention;

[0017] S3: The source domain image and the target domain image obtained in step S1 are sent as input to the classification system constructed in step S2 for training, and the predicted labels of the source domain and the target domain, as well as the predicted labels generated by the adversarial network are obtained as output to obtain a network model;

[0018] S4: Input the remote sensing image to be tested into the network model obtained in step S3 to obtain the corresponding category.

[0019] Furthermore, in step S3, the loss function used in training is as follows:

[0020] ;

[0021] Among them, i represents the number of samples, is the total number of samples in the source domain, j represents the number of categories, represents the total number of categories in the source domain, g represents the correct category of the source domain, Indicates that in the source domain The samples correspond to The predicted value of each category, Indicates that in the source domain The samples correspond to the correct category The predicted value of

[0022] And use the adversarial loss function L for optimization, namely:

[0023]

[0024] ;

[0025] in, and Represent the samples and labels of the source domain respectively, and Represent the source domain features and target domain features respectively, G represents the output of the source classifier in the conditional adversarial loss branch, D represents the output of the domain discriminator in the conditional adversarial loss branch, represents the likelihood function, c represents the category, represents the weight, and N represents the total number of samples.

[0026] Compared with the prior art, the invention can achieve the following beneficial effects:

[0027] (1) In the remote sensing image scene classification system and method based on information fusion migration described in the present invention, a conditional adversarial domain adaptation migration learning method is adopted to fuse the global features and local features in the remote sensing image. Compared with the existing technology, the network architecture has a good feature representation capability, and for unlabeled remote sensing image data sets, the model has good generalization ability;

[0028] (2) In the remote sensing image scene classification system and method based on information fusion migration created by the present invention, the original transformer is improved in view of the diversity of scenes and the complexity of spatial distribution of remote sensing images. For feature images, a sliding window is used and different expansion rates are set to perform self-attention calculation. At the same time, in order to mine the rich contextual information in remote sensing images, a multi-scale expansion attention mechanism is adopted. Multi-scale sliding window expansion attention is performed with different expansion rates for different scales, which can effectively extract semantic features of different scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:

[0030] Figure 1 A schematic diagram of a remote sensing image scene classification system based on information fusion migration according to an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the overall architecture of a remote sensing image scene classification system based on information fusion migration according to an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of the operation of the multi-scale dilated attention mechanism described in the embodiment of the present invention;

[0033] Figure 4 A flow chart of a remote sensing image scene classification method based on information fusion migration described in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0035] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0036] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0037] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0038] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0039] like Figures 1 to 3 As shown, the remote sensing image scene classification system based on information fusion migration described in the embodiment of the invention includes a multi-scale feature extraction branch and a conditional adversarial loss branch. The source domain image is input into the multi-scale feature extraction branch to obtain the source domain feature, and the target domain image is input into the multi-scale feature extraction branch to obtain the target domain feature; the source domain feature and the target domain feature are input into the conditional adversarial loss branch together, thereby completing the label matching of the source domain image and the target domain image.

[0040] The multi-scale feature extraction branch includes a residual network sub-branch and a multi-scale dilation sub-branch for performing sliding window dilation attention operations of different scales. The source domain image or the target domain image is input into the residual network sub-branch and the multi-scale dilation sub-branch respectively, and the features output by the residual network sub-branch and the multi-scale dilation sub-branch are combined to obtain the source domain features or the target domain features.

[0041] In the multi-scale expansion sub-branch, the input image is extracted by the shallow feature extraction module to obtain shallow features; the multi-scale expansion module performs sliding window expansion attention operations of different scales on the shallow features to obtain primary multi-scale features.

[0042] In the shallow feature extraction module, multiple stacked convolution blocks are used to segment the input image; the segmented image blocks are input to the normalization layer to obtain shallow features. In an embodiment of the present invention, multiple overlapping 3×3 convolution blocks are used to segment the input image, and the step size of the convolution kernel is alternately 1 or 2, and the resolution of the output feature image block is adjusted. This process can reduce the computational complexity to improve the efficiency of feature extraction while retaining important feature information.

[0043] In the multi-scale dilation module, after the input features are subjected to multiple parallel sliding window dilation attention operations of different scales, the obtained features are concatenated and linearly transformed, and the obtained results are processed by a multi-layer perceptron (MLP) to obtain primary multi-scale features. The sliding window dilation attention operation in the embodiment of the present invention adopts the paper "DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition" published in "IEEE Transactions on Multimedia", and the specific content is:

[0044] Centered on the feature map of the selected query, the keys and values ​​are sparsely selected through the sliding window, and then the self-attention calculation is performed on these image blocks, which can be expressed as:

[0045] ;

[0046] Where X represents the features obtained by the sliding window dilation attention operation, Q, K and V represent the query matrix, key matrix and value matrix of the input feature map respectively, r represents the dilation rate of the sliding window dilation attention operation, Represents a sliding window dilation attention operation.

[0047] For the query vector at position (i, j) in the initial feature map, sparsely select keys and values ​​for self-attention calculation in a sliding window of size w×w centered at (i, j). The output vector at position (i, j) can be defined as:

[0048] ;

[0049] in, represents the elements in feature X, represents the attention operation in the sliding window dilation attention operation, represents the elements in the query matrix, represents the key corresponding to the expansion rate r in the key matrix K, represents the value corresponding to the expansion rate r in the value matrix V, W represents the width of the feature map, and H represents the length of the feature map. For the query vector at position (i, j), the vector at coordinates The keys and values ​​of will participate in the self-attention calculation, The expression is:

[0050] ;

[0051] Among them, p and q are pointers inside the window respectively. Self-attention calculation is performed on all query image blocks in a sliding window manner. For the edge of the feature image, the padding strategy in the convolution operation is used to ensure the consistency of the size of the feature image. By sparsely selecting keys and values ​​centered on the query image block to construct features with long-distance dependencies, the problem of locality and sparsity can be effectively solved.

[0052] In order to extract multi-scale semantic information, in the embodiment of the present invention, for an input feature map , obtain the corresponding query by sliding window expansion attention operation ,key Sum , and then the feature map Divide the channels into N different heads and use different expansion rates in different heads Perform multi-scale sliding window expansion attention operation calculation, then splice the calculation results of the multi-scale sliding window expansion attention operation, and then perform linear mapping on the splicing results to obtain the output features , the corresponding calculation formula is:

[0053] ;

[0054] ;

[0055] in, Representation feature map The features assigned to the nth head, represents a linear mapping, Indicates feature concatenation. Output features After being processed by a multi-layer perceptron (MLP), primary multi-scale features are obtained. In the embodiment of the present invention, N=3, that is, the feature map Divide into 3 different heads by channel. By setting different expansion rates for different heads, semantic information of different scales can be effectively integrated, further reducing the redundancy of the self-attention mechanism and reducing the complex calculation cost.

[0056] In the embodiment of the present invention, a normalization layer is further provided between the linear mapping and the multi-layer perceptron (MLP), that is, the output feature After being processed by the multi-layer perceptron (MLP), the primary multi-scale features are obtained. After the multi-layer perceptron (MLP), there is also a normalization layer, which normalizes the primary multi-scale features. The primary multi-scale features after the normalization operation are transformed to obtain a multi-scale feature sequence.

[0057] In the residual network sub-branch, the input image is passed through multiple cascaded residual modules to obtain primary residual features. The residual module is used to extract local spatial information. Different residual blocks can capture multi-layer convolutional features. Shallow convolutional features usually contain low-level information, such as color, texture, and shape. Semantic information can usually be obtained in high-level convolutional features. Considering the complexity of the content of remote sensing scenes, features at different levels contribute to the semantic representation of remote sensing scenes. In an embodiment of the present invention, the residual network sub-branch is provided with 6 cascaded residual modules, each of which includes a convolutional layer and block normalization (BatchNorm). The sizes of the convolutional kernels corresponding to the convolutional layers in the 6 residual modules are 1×1, 3×3, 1×1, 1×1, 3×3, and 1×1, respectively. The primary residual features are subjected to feature transformation to obtain a residual feature sequence.

[0058] In the feature transformation in the multi-scale expansion sub-branch, the input features are pooled by the average pooling layer and then input into the softmax layer to obtain the corresponding feature sequence.

[0059] The residual feature sequence is multiplied by the corresponding elements of the primary multi-scale feature to obtain the residual feature, so as to enrich the local information of the branch; the multi-scale feature sequence is multiplied by the corresponding elements of the primary residual feature to obtain the multi-scale feature, so as to enhance the global perception ability of the residual network sub-branch; the multi-scale feature is added to the corresponding elements of the residual feature to obtain the output feature of the multi-scale feature extraction branch. The above process can greatly enhance the local details of the global features and the global perception of the local features.

[0060] In the conditional adversarial loss branch, the source classifier is used for feature learning and the domain discriminator is used for domain identification.

[0061] A remote sensing image scene classification method based on information fusion migration is applicable to a remote sensing image scene classification system based on information fusion migration provided by an embodiment of the present invention, comprising the following steps:

[0062] S1: Obtain a dataset, and obtain source domain images and their labels, and target domain images from the dataset.

[0063] In an embodiment of the present invention, the data set comes from three public data sets, including the paper "Bag-of-visual-words and spatial extensions for land-use classification" published by the University of California, Merced, which contains 2,100 remote sensing images divided into 21 scene categories; the paper "AID: A benchmark data set for performance evaluation of aerial scene classification" published by Wuhan University, which includes 10,000 remote sensing images divided into 30 scene categories; the paper "Remote sensing image scene classification: Benchmark and state of the art" published by Northwestern Polytechnical University, which contains 31,500 remote sensing images divided into 45 scene categories.

[0064] S2: Build a remote sensing image scene classification system.

[0065] S3: The source domain image and the target domain image obtained in step S1 are sent as input to the classification system constructed in step S2 for training, and the predicted labels of the source domain and the target domain, as well as the predicted labels generated by the adversarial network are obtained as output to obtain a network model;

[0066] In step S3, the loss function used in training is as follows:

[0067] ;

[0068] Among them, i represents the number of samples, is the total number of samples in the source domain, j represents the number of categories, represents the total number of categories in the source domain, g represents the correct category of the source domain, Indicates that in the source domain The samples correspond to The predicted value of each category, Indicates that in the source domain The samples correspond to the correct category The predicted value of

[0069] And use the adversarial loss function L for optimization, namely:

[0070]

[0071] ;

[0072] in, and Represent the samples and labels of the source domain respectively, and Represent the source domain features and target domain features respectively, G represents the output of the source classifier in the conditional adversarial loss branch, D represents the output of the domain discriminator in the conditional adversarial loss branch, represents the likelihood function, c represents the category, represents the weight, which is 1 in the embodiment of the present invention, and N represents the total number of samples. During the training process, SGD is used as the optimizer, the learning rate is set to 0.1, the batch size is set to 32, the epoch is 100, and the input scene is adjusted to 224×224.

[0073] S4: Input the remote sensing image to be tested into the network model obtained in step S3 to obtain the corresponding category.

[0074] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0075] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing image scene classification system based on information fusion migration, characterized by: It includes a multi-scale feature extraction branch and a conditional adversarial loss branch; among them, The source domain image is input into the multi-scale feature extraction branch to obtain the source domain features, and the target domain image is input into the multi-scale feature extraction branch to obtain the target domain features; the source domain features and the target domain features are input into the conditional adversarial loss branch together to complete the label matching of the source domain image and the target domain image; The multi-scale feature extraction branch includes a residual network sub-branch and a multi-scale dilation sub-branch for performing sliding window dilation attention operations of different scales; the source domain image or the target domain image is respectively input into the residual network sub-branch and the multi-scale dilation sub-branch, and the features output by the residual network sub-branch and the multi-scale dilation sub-branch are combined to obtain the source domain features or the target domain features accordingly; In the multi-scale expansion sub-branch, the input image is passed through the shallow feature extraction module to obtain shallow features; the multi-scale expansion module performs sliding window expansion attention operations of different scales on the shallow features; the multi-scale attention features are input to the deep feature extraction module to obtain primary multi-scale features; the primary multi-scale features are transformed to obtain a multi-scale feature sequence; In the residual network sub-branch, the input image is passed through multiple cascaded residual modules to obtain primary residual features, and the primary residual features are transformed to obtain a residual feature sequence; After the residual feature sequence is multiplied by the corresponding elements of the primary multi-scale feature, the residual feature is obtained; after the multi-scale feature sequence is multiplied by the corresponding elements of the primary residual feature, the multi-scale feature is obtained; after the multi-scale feature is added to the corresponding elements of the residual feature, the output feature of the multi-scale feature extraction branch is obtained.

2. The remote sensing image scene classification system based on information fusion migration according to claim 1 is characterized in that: In the shallow feature extraction module, multiple stacked convolution blocks are used to segment the input image; the segmented image blocks are input into the normalization layer to obtain shallow features.

3. The remote sensing image scene classification system based on information fusion migration according to claim 1 is characterized in that: In the multi-scale dilation module, the input features are divided into multiple heads according to the number of channels, and the sliding window dilation attention operation is performed in different heads with different dilation rates. The obtained features are concatenated and linearly transformed. The results are processed by a multi-layer perceptron to obtain multi-scale attention features. The dilation rates in the dilation operation of sliding windows of different scales are different.

4. The remote sensing image scene classification system based on information fusion migration according to claim 1 is characterized in that: In feature transformation, the input features are pooled by the average pooling layer and then input into the softmax layer to obtain the corresponding feature sequence.

5. The remote sensing image scene classification system based on information fusion migration according to claim 1 is characterized in that: In the conditional adversarial loss branch, the source classifier is used for feature learning and the domain discriminator is used for domain identification.

6. A remote sensing image scene classification method based on information fusion migration, characterized in that: The following steps are involved: S1: Obtain a dataset, and obtain source domain images and their labels, and target domain images from the dataset; S2: Constructing a remote sensing image scene classification system based on information fusion migration as described in any one of claims 1 to 5; S3: The source domain image and the target domain image obtained in step S1 are sent as input to the classification system constructed in step S2 for training, and the predicted labels of the source domain and the target domain, as well as the predicted labels generated by the adversarial network are obtained as output to obtain a network model; S4: Input the remote sensing image to be tested into the network model obtained in step S3 to obtain the corresponding category.

7. The remote sensing image scene classification method based on information fusion migration according to claim 6 is characterized in that: In step S3, the loss function used in training is as follows: ； Where i represents the number of samples, is the total number of samples in the source domain, j represents the number of categories, represents the total number of categories in the source domain, g represents the correct category of the source domain, Indicates that in the source domain The samples correspond to The predicted value of each category, Indicates that in the source domain The samples correspond to the correct category The predicted value of And use the adversarial loss function L for optimization, namely: ； in, and Represent the samples and labels of the source domain respectively, and Represent the source domain features and target domain features respectively, G represents the output of the source classifier in the conditional adversarial loss branch, D represents the output of the domain discriminator in the conditional adversarial loss branch, represents the likelihood function, c represents the category, represents the weight, and N represents the total number of samples.

Images