Remote sensing image classification system and method based on multi-modal feature fusion

Through multimodal feature fusion and model fusion, combined with LightGBM and multi-layer LSTM algorithm, the problem of insufficient computing complexity and generalization capabilities in remote sensing image classification is solved, and efficient and robust image classification results are achieved, suitable for large-scale Remote sensing image data.

CN118968172APending Publication Date: 2024-11-15JIANGSU TIANHUI SPATIAL INFORMATION RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411064694.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Multimodal feature fusion has problems such as high computational complexity, risk of overfitting and insufficient generalization capabilities in remote sensing image classification, especially in applications in different regions.

Method used

Adopt a remote sensing image classification system based on multimodal feature fusion, including preprocessing module, early fusion classification module, feature engineering module and model fusion classification module. Through feature fusion, dimensionality reduction, combination and weighting calculation, combined with LightGBM and multi-layer LSTM Algorithm to improve classification accuracy and robustness.

Benefits of technology

Effectively utilize multimodal data information, improve classification accuracy and efficiency, enhance the robustness of the model, reduce the influence of noise characteristics, and improve the reliability of the final classification results. It is suitable for classification tasks of large-scale remote sensing images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968172A_ABST
    Figure CN118968172A_ABST
Patent Text Reader

Abstract

The invention discloses a remote sensing image classification system and method based on multi-modal feature fusion, and belongs to the technical field of data fusion. The system comprises a preprocessing module, an early fusion classification module, a feature engineering module and a model fusion classification module. The preprocessing module is used for defining modalities on which image classification is based and collecting features corresponding to the modalities; the early fusion classification module is used for performing early fusion processing on the features obtained by the preprocessing module and training a model to classify the images; the feature engineering module is used for carrying out dimensionality reduction and combination processing on features; and the model fusion classification module is used for carrying out model fusion processing on the feature combination obtained by the feature engineering module, training a model for classification and carrying out weighted calculation on the classification probability to obtain a final classification result. The invention further provides a method for implementing the system. The conditions of feature redundancy and weak generalization ability in the prior art can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and in particular to a remote sensing image classification system and method based on multimodal feature fusion. Background Art

[0002] Remote sensing image classification is a technique to identify and classify surface objects or features by processing and analyzing remote sensing image data. Remote sensing images usually come from satellites, drones or aircraft, using various sensors to capture images of the surface. Remote sensing image classification is widely used in agriculture, environmental monitoring, urban planning, disaster management and other fields.

[0003] Multimodal feature fusion is a method of comprehensively analyzing features from different modalities to improve the accuracy of classification or recognition tasks. In remote sensing image classification, data captured by different sensors have different advantages and characteristics. By fusing these multimodal data, their complementary information can be fully utilized to improve the accuracy and robustness of classification.

[0004] When multimodal feature fusion is used for remote sensing image classification, the data dimension may be very high after fusing multimodal data, which greatly increases the computational complexity and overfitting risk. The feature differences between different modal data are large, making it difficult to effectively fuse them. In addition, the generalization ability of multimodal models in different regions is weak, making it difficult to effectively use them in other areas. Summary of the invention

[0005] The purpose of the present invention is to provide a remote sensing image classification system and method based on multimodal feature fusion to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a remote sensing image classification system based on multimodal feature fusion, the system comprising: a preprocessing module, an early fusion classification module, a feature engineering module and a model fusion classification module;

[0007] The preprocessing module is used to define the modalities on which image classification is based and to collect features corresponding to each modality; the early fusion classification module is used to perform early fusion processing on the features obtained by the preprocessing module and train the model to classify the image; the feature engineering module is used to perform dimensionality reduction processing on the features according to the classification obtained by the early fusion classification module and combine the features according to different classifications; the model fusion classification module is used to perform model fusion processing on the feature combination obtained by the feature engineering module, train the model for classification, and perform weighted calculation on the classification probability to obtain the final classification result;

[0008] The output end of the preprocessing module is connected to the input end of the early fusion classification module; the output end of the early fusion classification module is connected to the input end of the feature engineering module; the output end of the feature engineering module is connected to the input end of the model fusion classification module.

[0009] The preprocessing module includes: a modality definition unit, a data preparation unit and a feature extraction unit;

[0010] The modality definition unit is used to define the modality based on which the image classification is based; the data preparation unit is used to collect data information corresponding to different modalities; the feature extraction unit is used to process the data information corresponding to different modalities and extract features;

[0011] The output end of the modality definition unit is connected to the input end of the data preparation unit; the output end of the data preparation unit is connected to the input end of the feature extraction unit; the output end of the feature extraction unit is connected to the input end of the early fusion classification module.

[0012] The early fusion classification module includes: a feature fusion unit, a model training unit and a classification prediction unit;

[0013] The feature fusion unit is used to fuse features using early fusion; the model training a unit is used to train a classification model based on the features fused by the feature fusion unit and their corresponding classifications; the classification prediction a unit is used to predict the sample features of unknown labels based on the model trained by the model training a unit to obtain the probability of each classification;

[0014] The output end of the feature fusion unit is connected to the input end of the model training a unit; the output end of the model training a unit is connected to the input end of the classification prediction a unit; the output end of the classification prediction a unit is connected to the input end of the feature engineering module.

[0015] The feature engineering module includes: a feature dimension reduction unit and a feature combination unit;

[0016] The feature dimension reduction unit is used to perform dimension reduction processing on the features according to the feature data and the probability of each classification; the feature combination unit is used to combine the features obtained by dimension reduction of different classifications;

[0017] The output end of the feature dimension reduction unit is connected to the input end of the feature combination unit; the output end of the feature combination unit is connected to the input end of the model fusion classification module.

[0018] The model fusion classification module includes: a model fusion unit, a model training b unit, a classification prediction b unit and a weighted classification unit;

[0019] The model fusion unit is used to fuse features using model fusion; the model training b unit is used to train a classification model based on the features fused by the model fusion unit and their corresponding classifications; the classification prediction b unit is used to predict different combination features of samples with unknown labels based on the model trained by the model training b unit to obtain the probabilities of each classification in different combinations; the weighted classification unit is used to weight the predicted classification probabilities obtained from different combinations of samples to obtain the classification results;

[0020] The output end of the model fusion unit is connected to the input end of the model training b unit; the output end of the model training b unit is connected to the input end of the classification prediction b unit; the output end of the classification prediction b unit is connected to the input end of the weighted classification unit.

[0021] A remote sensing image classification method based on multimodal feature fusion, the method comprises the following steps:

[0022] S1. Define the modalities used for remote sensing image classification, collect data corresponding to each modality, and process the data to extract features;

[0023] S2, early integration of features, training of classification algorithms to perform classification prediction on new samples;

[0024] S3, reducing the dimension of features based on early fusion features and their classification prediction probabilities;

[0025] S4. Combine features according to different classifications, use model fusion for combined features, and train classification algorithms to perform classification prediction on unclassified samples;

[0026] S5. Perform weighted calculation on the different probabilities obtained by classifying and predicting samples according to different classifications and different feature combinations to obtain the final classification result.

[0027] In step S1, the modalities used for remote sensing image classification are defined as: {modal1, modal2,…, modal n};

[0028] The corresponding data of each mode collected by the sensor is:

[0029] modal1:[data 11 ,data 12 ,…,data 1i ];

[0030] modal2:[data 21 ,data 22 ,…,data 2j ];

[0031] …

[0032] model n :[data n1 ,data n2 ,…,data nk ];

[0033] The extracted features are directly represented by the data or obtained after data processing, and can be expressed as:

[0034] modal1:[feature 11 ,feature 12 ,…,feature 1r ];

[0035] modal2:[feature 21 ,feature 22 ,…,feature 2s ];

[0036] …

[0037] model n :[feature n1 ,feature n2 ,…,feature nt ];

[0038] Among them, i, j, k, n, r, s, t are positive integers; modal n Indicates the nth mode; data 1i Represents the i-th data of the first mode; data 2j Represents the jth data of the second mode; data nk Represents the kth data of the nth mode; feature 1r Represents the rth feature of the first mode; feature 2s Represents the sth feature of the second mode; feature nt represents the t-th feature of the n-th mode.

[0039] In step S2, the specific method of early fusion of features is as follows:

[0040] Concatenate the extracted features directly into a single high-dimensional vector: 11 ,…,feature 1r ,feature 21 ,…,feature 2s ,feature n1 ,…,feature nt ];

[0041] Use the machine learning algorithm LightGBM to train the classification model to predict unknown samples and output the classification prediction probability:

[0042] The sample format of the training set is: feature vector and label corresponding to the sample;

[0043] Construct a decision tree: Each tree recursively splits the data set and selects the best split point to minimize the cross entropy loss function

[0044] Among them, L is the cross entropy loss function, U is the total number of samples, V is the total number of categories, u∈{1,2,…,U}, v∈{1,2,…,V}, y uv represents the true label of sample u in classification v, p uv Represents the probability that sample u is predicted to be classified as v;

[0045] Gradient boosting: Initialize the model prediction to the mean of each classification, that is, Iterate the training tree, fit the residual each time, and update the model F m+1 (x) = F m (x)+η·T m (x);

[0046] Among them, x is the unknown sample, m is the number of iterations; F m (x) is the prediction model of the mth round, η is the learning rate, T m (x) is the prediction of the mth tree, F m+1 (x) is the updated model;

[0047] Softmax transformation: Apply the Softmax function to the predicted value of each category

[0048] Where, w∈{1,2,…,V}, F v (x) is the predicted value of category v, F w (x) is the predicted value of category w, p v is the probability of class v.

[0049] In step S3, the Spearman rank correlation coefficient is used to determine the impact of the feature on each classification probability:

[0050] Sort each variable and get their rank; for each pair of data points, calculate the difference in the rankings of the two variables; calculate the Spearman rank correlation coefficient based on the ranking difference:

[0051]

[0052] where \(c\in\{1,2,\ldots,e\}\), \(e\) is a positive integer, and \(d\) c is the difference in the rankings of two variables of the \(c\)-th sample, and \(e\) is the number of samples;

[0053] Calculate the Spearman rank correlation coefficient between each feature and each classification respectively; the closer the absolute value of the correlation coefficient is to 1, the greater the influence of the feature on the classification; set a threshold \(z_1\), where \(0\lt z_1\lt1\); when there is a feature whose Spearman rank correlation coefficient with the classification probability satisfies \(|1 - |\rho||\gt z_1\), it is considered that the influence of the feature on the classification probability is small;

[0054] For each classification, select the features that meet the threshold conditions to obtain the feature combination corresponding to each classification;

[0055] For each unknown sample, divide it into \(V\) different feature combinations according to the features corresponding to the \(V\) classifications;

[0056] In step S4, the specific way of using model fusion for the combined features is as follows:

[0057] For the feature combinations corresponding to each classification of the samples and unknown samples; construct a multi-layer LSTM model, input the features of the first modality into the first layer of LSTM to obtain the hidden layer state; concatenate the features of the second modality with the hidden layer state of the first layer of LSTM and input them into the second layer of LSTM to obtain the second layer hidden layer state; \(\cdots\); concatenate the features of the \(n\)-th modality with the hidden layer state of the \((n - 1)\)-th layer of LSTM and input them into the \(n\)-th layer of LSTM to obtain the \(n\)-th layer hidden layer state;

[0058] The \(n\)-th layer hidden layer state is the finally fused feature; use the samples as the training set to construct a LightGBM classification model to classify the unknown samples.

[0059] In step S5, for each unknown sample, divide it into \(V\) different feature combinations according to the \(V\) classifications and the features corresponding to the classifications; after classification, each of the \(V\) feature combinations of each sample outputs the prediction probability of each classification; the probability \(P\) of predicting as classification \(v\) in the classification result corresponding to the features of classification \(v\) v The weight is set to \(Q\), and the probabilities \(P_1\sim P\) of predicting as classification \(v\) in the classification results corresponding to the features of the remaining classifications v-1 , \(P\) v+1 \(\sim P\) V The weight is Then the predicted classification probability of classification \(v\) is Calculate the predicted probabilities of the \(V\) classifications, and select the classification corresponding to the maximum value as the final classification result.

[0060] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention can make full use of information from different data sources and improve classification accuracy and efficiency through feature fusion and model fusion of multimodal data; the present invention uses LightGBM and multi-layer LSTM machine learning algorithms, which can improve computing efficiency while ensuring classification effect, and is suitable for classification tasks of large-scale remote sensing images; the present invention uses a weighted calculation and dimensionality reduction method, which can enhance the robustness of the model, reduce the influence of noise characteristics, and improve the reliability of the final classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0062] Figure 1 It is a flow chart of a remote sensing image classification system based on multimodal feature fusion of the present invention;

[0063] Figure 2 It is a schematic diagram of the steps of the remote sensing image classification method based on multimodal feature fusion of the present invention;

[0064] Figure 3 It is a schematic diagram of weight allocation of the remote sensing image classification method based on multimodal feature fusion of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] See also Figure 1-Figure 3 ,The present invention provides a technical solution: a remote sensing image classification system based on multimodal feature fusion, the system comprising: a preprocessing module, an early fusion classification module, a feature engineering module and a model fusion classification module;

[0067] The preprocessing module is used to define the modalities on which image classification is based and to collect features corresponding to each modality; the early fusion classification module is used to perform early fusion processing on the features obtained by the preprocessing module and train the model to classify the image; the feature engineering module is used to perform dimensionality reduction processing on the features according to the classification obtained by the early fusion classification module and combine the features according to different classifications; the model fusion classification module is used to perform model fusion processing on the feature combination obtained by the feature engineering module, train the model for classification, and perform weighted calculation on the classification probability to obtain the final classification result;

[0068] The output end of the preprocessing module is connected to the input end of the early fusion classification module; the output end of the early fusion classification module is connected to the input end of the feature engineering module; the output end of the feature engineering module is connected to the input end of the model fusion classification module.

[0069] The preprocessing module includes: a modality definition unit, a data preparation unit and a feature extraction unit;

[0070] The modality definition unit is used to define the modality based on which the image classification is based; the data preparation unit is used to collect data information corresponding to different modalities; the feature extraction unit is used to process the data information corresponding to different modalities and extract features;

[0071] The output end of the modality definition unit is connected to the input end of the data preparation unit; the output end of the data preparation unit is connected to the input end of the feature extraction unit; the output end of the feature extraction unit is connected to the input end of the early fusion classification module.

[0072] The early fusion classification module includes: a feature fusion unit, a model training unit and a classification prediction unit;

[0073] The feature fusion unit is used to fuse features using early fusion; the model training a unit is used to train a classification model based on the features fused by the feature fusion unit and their corresponding classifications; the classification prediction a unit is used to predict the sample features of unknown labels based on the model trained by the model training a unit to obtain the probability of each classification;

[0074] The output end of the feature fusion unit is connected to the input end of the model training a unit; the output end of the model training a unit is connected to the input end of the classification prediction a unit; the output end of the classification prediction a unit is connected to the input end of the feature engineering module.

[0075] The feature engineering module includes: a feature dimension reduction unit and a feature combination unit;

[0076] The feature dimension reduction unit is used to perform dimension reduction processing on the features according to the feature data and the probability of each classification; the feature combination unit is used to combine the features obtained by dimension reduction of different classifications;

[0077] The output end of the feature dimension reduction unit is connected to the input end of the feature combination unit; the output end of the feature combination unit is connected to the input end of the model fusion classification module.

[0078] The model fusion classification module includes: a model fusion unit, a model training b unit, a classification prediction b unit and a weighted classification unit;

[0079] The model fusion unit is used to fuse features using model fusion; the model training b unit is used to train a classification model based on the features fused by the model fusion unit and their corresponding classifications; the classification prediction b unit is used to predict different combination features of samples with unknown labels based on the model trained by the model training b unit to obtain the probabilities of each classification in different combinations; the weighted classification unit is used to weight the predicted classification probabilities obtained from different combinations of samples to obtain the classification results;

[0080] The output end of the model fusion unit is connected to the input end of the model training b unit; the output end of the model training b unit is connected to the input end of the classification prediction b unit; the output end of the classification prediction b unit is connected to the input end of the weighted classification unit.

[0081] A remote sensing image classification method based on multimodal feature fusion, the method comprises the following steps:

[0082] S1. Define the modalities used for remote sensing image classification, collect data corresponding to each modality, and process the data to extract features;

[0083] S2, early integration of features, training of classification algorithms to perform classification prediction on new samples;

[0084] S3, reducing the dimension of features based on early fusion features and their classification prediction probabilities;

[0085] S4. Combine features according to different classifications, use model fusion for combined features, and train classification algorithms to perform classification prediction on unclassified samples;

[0086] S5. Perform weighted calculation on the different probabilities obtained by classifying and predicting samples according to different classifications and different feature combinations to obtain the final classification result.

[0087] In step S1, the modalities used for remote sensing image classification are defined as: {modal1, modal2,…, modal n};

[0088] The corresponding data of each mode collected by the sensor is:

[0089] modal1:[data 11 ,data 12 ,…,data 1i ];

[0090] modal2:[data 21 ,data22 ,…,data 2j ];

[0091] …

[0092] modal n :[datan1,datan2,…,data nk ];

[0093] The extracted features are directly represented by the data or obtained after data processing, and can be expressed as:

[0094] modal1:[feature 11 ,feature 12 ,…,feature 1r ];

[0095] modal2:[feature 21 ,feature 22 ,…,feature 2s ];

[0096] …

[0097] modal n :[feature n1 ,feature n2 ,…,feature nt ];

[0098] Among them, i, j, k, n, r, s, t are positive integers; modal n Indicates the nth mode; data 1i Represents the i-th data of the first mode; data 2j Represents the jth data of the second mode; data nk Represents the kth data of the nth mode; feature 1r Represents the rth feature of the first mode; feature 2s Represents the sth feature of the second mode; feature nt represents the t-th feature of the n-th mode.

[0099] In step S2, the specific method of early fusion of features is as follows:

[0100] Concatenate the extracted features directly into a single high-dimensional vector: 11 ,…,feature 1r ,feature 21 ,…,feature 2s ,featuren1 ,…,feature nt ];

[0101] Use the machine learning algorithm LightGBM to train the classification model to predict unknown samples and output the classification prediction probability:

[0102] The sample format of the training set is: feature vector and label corresponding to the sample;

[0103] Construct a decision tree: Each tree recursively splits the data set and selects the best split point to minimize the cross entropy loss function

[0104] Among them, L is the cross entropy loss function, U is the total number of samples, V is the total number of categories, u∈{1,2,…,U}, v∈{1,2,…,V}, y uv represents the true label of sample u in classification v, p uv Represents the probability that sample u is predicted to be classified as v;

[0105] Gradient boosting: Initialize the model prediction to the mean of each classification, that is, Iterate the training tree, fit the residual each time, and update the model F m+1 (x) = F m (x)+η·T m (x);

[0106] Among them, x is the unknown sample, m is the number of iterations; F m (x) is the prediction model of the mth round, η is the learning rate, T m (x) is the prediction of the mth tree, F m+1 (x) is the updated model;

[0107] Softmax transformation: Apply the Softmax function to the predicted value of each category

[0108] Where, w∈{1,2,…,V}, F v (x) is the predicted value of category v, F w (x) is the predicted value of category w, p v is the probability of class v.

[0109] In step S3, the Spearman rank correlation coefficient is used to determine the impact of the feature on each classification probability:

[0110] Sort each variable and get their rank; for each pair of data points, calculate the difference in the rankings of the two variables; calculate the Spearman rank correlation coefficient based on the ranking difference:

[0111]

[0112] where \(c\in\{1,2,\ldots,e\}\), \(e\) is a positive integer, and \(d\) c is the difference in the rankings of two variables of the \(c\)-th sample, and \(e\) is the number of samples;

[0113] Calculate the Spearman rank correlation coefficient between each feature and each classification respectively; the closer the absolute value of the correlation coefficient is to 1, the greater the impact of the feature on the classification; set a threshold \(z_1\), \(0\lt z_1\lt1\); when there is a feature whose Spearman rank correlation coefficient with the classification probability satisfies \(|1 - |\rho||\gt z_1\), it is considered that the impact of the feature on the classification probability is small;

[0114] For each classification, select the features that meet the threshold conditions to obtain the feature combination corresponding to each classification;

[0115] For each unknown sample, divide it into \(V\) different feature combinations according to the features corresponding to \(V\) classifications;

[0116] In step S4, the specific way of using model fusion for the combined features is as follows:

[0117] For the feature combinations corresponding to each classification of the samples and unknown samples; construct a multi-layer LSTM model, input the features of the first modality into the first layer of LSTM to obtain the hidden layer state; concatenate the features of the second modality with the hidden layer state of the first layer of LSTM and input them into the second layer of LSTM to obtain the second hidden layer state; \(\cdots\); concatenate the features of the \(n\)-th modality with the hidden layer state of the \((n - 1)\)-th layer of LSTM and input them into the \(n\)-th layer of LSTM to obtain the \(n\)-th hidden layer state;

[0118] The \(n\)-th hidden layer state is the finally fused feature; use the samples as the training set to construct a LightGBM classification model to classify the unknown samples.

[0119] In step S5, for each unknown sample, divide it into \(V\) different feature combinations according to \(V\) classifications and the features corresponding to the classifications; after classification, each of the \(V\) feature combinations of each sample outputs the prediction probability of each classification; the probability \(P\) of predicting as classification \(v\) in the feature classification result corresponding to classification \(v\) v The weight is set to \(Q\), and the probabilities \(P_1\sim P\) of predicting as classification \(v\) in the feature classification results corresponding to the remaining classifications v-1 , \(P\) v+1 \(\sim P\) V The weight is Then the predicted classification probability of classification \(v\) is Calculate the predicted probabilities of \(V\) classifications, and select the classification corresponding to the maximum value as the final classification result.

[0120] In this embodiment:

[0121] Classification: Sand, dunes and basins;

[0122] Define the modalities: optical imagery and radar imagery;

[0123] Data Collection:

[0124] Optical imagery: Image data is obtained using optical sensors on satellites or aerial platforms; usually multispectral or hyperspectral images, containing data from multiple bands;

[0125] Radar imagery: Image data is obtained using synthetic aperture radar sensors on satellites or aerial platforms; usually single-polarization or multi-polarization SAR images, containing data in different polarization modes;

[0126] Feature extraction, directly represented by data or obtained after data processing:

[0127] Optical imaging: spectral characteristics; texture characteristics; color characteristics;

[0128] Radar image: polarization features; texture features; interference features;

[0129] The extracted features are directly concatenated into a single high-dimensional vector: [spectral features, texture features, color features, polarization features, texture features, interference features];

[0130] Use the machine learning algorithm LightGBM to train the classification model to predict unknown samples and output the classification prediction probability;

[0131] The Spearman rank correlation coefficient between each feature and the three classifications was calculated respectively; for the three classifications, the features that met the threshold conditions were selected respectively to obtain the feature combination corresponding to each classification;

[0132] For each unknown sample, it is divided into different feature combinations according to the features corresponding to the three classifications;

[0133] For the feature combination corresponding to each classification of samples and unknown samples; construct a multi-layer LSTM model, input the features of the first modality into the first layer LSTM to obtain the hidden layer state; concatenate the features of the second modality with the hidden layer state of the first layer LSTM and input them into the second layer LSTM to obtain the second layer hidden layer state;

[0134] The state of the second hidden layer is the final fused feature; the sample is used as a training set to build a LightGBM classification model to classify unknown samples;

[0135] For each unknown sample, it is divided into three different feature combinations according to the three classifications and the features corresponding to the classifications; after the three feature combinations of each sample are classified, the predicted probability of each classification is output; the probability weight of the classified sand in the feature classification results corresponding to the classified sand is set to Q, and the probability weights of the other classifications corresponding to the feature classification results are Calculate the predicted probabilities of the three classifications and select the classification corresponding to the maximum value as the final classification result.

[0136] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0137] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image classification system based on multimodal feature fusion, characterized by: The system includes: a preprocessing module, an early fusion classification module, a feature engineering module and a model fusion classification module; The preprocessing module is used to define the modalities on which image classification is based and to collect features corresponding to each modality; the early fusion classification module is used to perform early fusion processing on the features obtained by the preprocessing module and train the model to classify the image; the feature engineering module is used to perform dimensionality reduction processing on the features according to the classification obtained by the early fusion classification module and combine the features according to different classifications; the model fusion classification module is used to perform model fusion processing on the feature combination obtained by the feature engineering module, train the model for classification, and perform weighted calculation on the classification probability to obtain the final classification result; The output end of the preprocessing module is connected to the input end of the early fusion classification module; the output end of the early fusion classification module is connected to the input end of the feature engineering module; the output end of the feature engineering module is connected to the input end of the model fusion classification module.

2. The remote sensing image classification system based on multimodal feature fusion according to claim 1, characterized in that: The preprocessing module includes: a modality definition unit, a data preparation unit and a feature extraction unit; The modality definition unit is used to define the modality based on which the image classification is based; the data preparation unit is used to collect data information corresponding to different modalities; the feature extraction unit is used to process the data information corresponding to different modalities and extract features; The output end of the modality definition unit is connected to the input end of the data preparation unit; the output end of the data preparation unit is connected to the input end of the feature extraction unit; the output end of the feature extraction unit is connected to the input end of the early fusion classification module.

3. The remote sensing image classification system based on multimodal feature fusion according to claim 1, characterized in that: The early fusion classification module includes: a feature fusion unit, a model training unit and a classification prediction unit; The feature fusion unit is used to fuse features using early fusion; the model training a unit is used to train a classification model based on the features fused by the feature fusion unit and their corresponding classifications; the classification prediction a unit is used to predict the sample features of unknown labels based on the model trained by the model training a unit to obtain the probability of each classification; The output end of the feature fusion unit is connected to the input end of the model training a unit; the output end of the model training a unit is connected to the input end of the classification prediction a unit; the output end of the classification prediction a unit is connected to the input end of the feature engineering module.

4. The remote sensing image classification system based on multimodal feature fusion according to claim 1, characterized in that: The feature engineering module includes: a feature dimension reduction unit and a feature combination unit; The feature dimension reduction unit is used to perform dimension reduction processing on the features according to the feature data and the probability of each classification; the feature combination unit is used to combine the features obtained by dimension reduction of different classifications; The output end of the feature dimension reduction unit is connected to the input end of the feature combination unit; the output end of the feature combination unit is connected to the input end of the model fusion classification module.

5. The remote sensing image classification system based on multimodal feature fusion according to claim 1, characterized in that: The model fusion classification module includes: a model fusion unit, a model training b unit, a classification prediction b unit and a weighted classification unit; The model fusion unit is used to fuse features using model fusion; the model training b unit is used to train a classification model based on the features fused by the model fusion unit and their corresponding classifications; the classification prediction b unit is used to predict the probabilities of different classifications of a sample with an unknown label for different combined features of the model trained by the model training b unit; the weighted classification unit is used to calculate the classification result by weighted calculation of the prediction classification probabilities obtained from different combinations of the sample. The output end of the model fusion unit is connected to the input end of the model training b unit; the output end of the model training b unit is connected to the input end of the classification prediction b unit; the output end of the classification prediction b unit is connected to the input end of the weighted classification unit.

6. A remote sensing image classification method based on multimodal feature fusion, characterized in that: This method includes the following steps: S1. Define the modalities used for remote sensing image classification, collect the corresponding data for each modality, and process the data to extract features. S2. Perform early fusion on the features, and train a classification algorithm to predict the classification of new samples. S3. Reduce the dimensions of the features according to the early fusion features and their classification prediction probabilities. S4. Combine the features according to different classifications, use model fusion for the combined features, and train a classification algorithm to predict the classification of unclassified samples. S5. Perform weighted calculation on the different probabilities obtained from the classification prediction of the sample according to different classifications and different feature combinations to obtain the final classification result.

7. The remote sensing image classification method based on multimodal feature fusion according to claim 6, characterized in that: In step S1, the modalities used for remote sensing image classification are defined as: {modal1, modal2,…, modal n }; The data corresponding to each modality collected by the sensor is: modal1:[data 11 ,data 12 ,…,data 1i ]; modal2:[data 21 ,data 22 ,…,data 2j ]; …… modal n :[data n1 ,data n2 ,…,data nk ]; The extracted features are directly represented by the data or obtained after data processing, and can be expressed as: modal1:[feature 11 ,feature 12 ,…,feature 1r ]; modaln2:[feature 21 ,feature 22 ,…,feature 2s ]; …… moodal n :[feature n1 ,feature n2 ,…,feature nt ]; Among them, i, j, k, n, r, s, t are positive integers; modal n Indicates the nth mode; data 1i Represents the i-th data of the first mode; data 2j Represents the jth data of the second mode; data nk Represents the kth data of the nth mode; feature 1r Represents the rth feature of the first mode; feature 2s Represents the sth feature of the second mode; feature nt represents the t-th feature of the n-th mode.

8. The remote sensing image classification method based on multimodal feature fusion according to claim 7, characterized in that: In step S2, the specific method for performing early fusion on the features is: Concatenate the extracted features directly into a single high-dimensional vector: 11 ,…,feature 1r ,feature 21 ,…,feature 2s ,feature n1 ,…,feature nt ]; Use the machine learning algorithm LightGBM to train a classification model to predict an unknown sample and output the classification prediction probability: The format of the training set samples is: feature vectors and the labels corresponding to the samples. Construct a decision tree: Each tree recursively splits the data set and selects the best split point to minimize the cross entropy loss function Among them, L is the cross entropy loss function, U is the total number of samples, V is the total number of categories, u∈{1,2,…,U}, v∈{1,2,…,V}, y uv represents the true label of sample u in classification v, p uv Represents the probability that sample u is predicted to be classified as v; Gradient boosting: Initialize the model prediction to the mean of each classification, that is, Iterate the training tree, fit the residual each time, and update the model F m+1 (x) = F m (x)+η·T m (x); Among them, x is the unknown sample, m is the number of iterations; F m (x) is the prediction model of the mth round, η is the learning rate, T m (x) is the prediction of the mth tree, F m+1 (x) is the updated model; Softmax transformation: Apply the Softmax function to the predicted value of each category Where, w∈{1,2,…,V}, F v (x) is the predicted value of category v, F w (x) is the predicted value of category w, p v is the probability of class v.

9. The remote sensing image classification method based on multimodal feature fusion according to claim 8, characterized in that: In step S3, use the Spearman rank correlation coefficient to judge the influence of the features on the classification probabilities of each class: Sort each variable to obtain their rankings; for each pair of data points, calculate the ranking differences of the two variables; calculate the Spearman rank correlation coefficient according to the ranking differences: Where c∈{1,2,…,e}, e is a positive integer, d c is the difference between the rankings of the two variables in the cth sample, and e is the number of samples; Calculate the Spearman rank correlation coefficient between each feature and each classification respectively; the closer the absolute value of the correlation coefficient is to 1, the greater the influence of the feature on the classification; set a threshold z1, 0 < z1 < 1; when there is a feature whose Spearman rank correlation coefficient with the classification probability satisfies |1 - |ρ|| > z1, it is considered that the influence of the feature on the classification probability is small. For each classification, select the features that meet the threshold conditions to obtain the feature combination corresponding to each classification. For each unknown sample, divide it into V different feature combinations according to the features corresponding to V classifications. In step S4, the specific method for using model fusion for the combined features is: For the feature combination corresponding to each classification of the sample and the unknown sample; construct a multi-layer LSTM model, input the features of the first modality into the first layer LSTM to obtain the hidden layer state; concatenate the features of the second modality with the hidden layer state of the first layer LSTM and input them into the second layer LSTM to obtain the second layer hidden layer state; ...; concatenate the features of the nth modality with the hidden layer state of the (n-1)th layer LSTM and input them into the nth layer LSTM to obtain the nth layer hidden layer state; The state of the nth hidden layer is the final fused feature; the sample is used as a training set to build a LightGBM classification model to classify unknown samples.

10. The remote sensing image classification method based on multimodal feature fusion according to claim 9, characterized in that: In step S5, for each unknown sample, it is divided into V different feature combinations according to V categories and the features corresponding to the categories; after each sample V feature combination is classified, the predicted probability of each category is output; the probability P of the feature classification result corresponding to category v is predicted as category v v The weight is set to Q, and the probability of the other classifications corresponding to the feature classification results predicted as classification v is P1~P v-1 ,P v+1 ~P V The weight is Then the predicted classification probability of class v is Calculate the predicted probabilities of V categories and select the category corresponding to the maximum value as the final classification result.