Remote sensing image cloud classification method, system and device based on multi-modal data fusion and medium

Abstract

The invention discloses a multi-modal data fused remote sensing image cloud classification method, system and device and a medium, and the method comprises the steps: obtaining multi-modal data and a known cloud type label, carrying out the preprocessing of the multi-modal data, forming input data, and dividing the input data and the cloud type label into a training set, a verification set and a test set according to a proportion; the wavelet consistency alignment module executes multi-scale wavelet transformation on the input data to obtain a wavelet feature tensor of each modal; constructing a cloud classification backbone network model; training and evaluating the cloud classification backbone network model by adopting the training set, the verification set and the test set to obtain an optimized cloud classification backbone network model, and inputting the optical remote sensing cloud image to be classified, the cloud profile radar reflectivity data and the atmospheric physical attribute data into the cloud classification backbone network model. And predicting the optical remote sensing cloud image, and outputting a cloud classification result label map corresponding to each pixel point in the target area. According to the invention, the cloud identification precision and stability in a complex scene are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image space-time fusion, and particularly relates to a multi-modal data fusion remote sensing image cloud classification method, system, device and medium. BACKGROUND

[0002] Cloud classification refers to a process of automatically identifying and classifying cloud bodies in the atmosphere according to their types, structures, distributions or physical properties by using remote sensing observation data. An accurate cloud classification result plays an important supporting role in typhoon path prediction, severe convective weather warning, radiation energy budget estimation and climate change modeling.

[0003] At present, remote sensing cloud classification methods can be divided into three categories: traditional image processing methods, machine learning methods and deep learning methods. Regarding the traditional image processing method, typical methods mostly rely on low-order image features such as gray level, edge and texture. For example, a cloud detection strategy based on a threshold value often uses a fixed brightness temperature or albedo value to segment cloud / non-cloud, but the traditional image processing method is difficult to accurately distinguish the complex morphology of clouds (such as thin clouds and mixed clouds), and is highly dependent on expert experience; although the texture classification method based on the co-occurrence matrix proposed by Haralick can describe the texture differences of cloud morphology, it is sensitive to scale changes and has poor generalization ability. Regarding the machine learning method, researchers attempt to convert remote sensing images into structured feature vectors and use classifiers to distinguish cloud types; for example, a support vector machine (SVM) can distinguish cloud types in a low-dimensional feature space, but it relies on manual feature design; for example, Zhang et al. constructed an SVM classifier based on MODIS cloud map brightness temperature features and structure parameters to distinguish cirrus clouds and cumulus clouds, but the SVM classifier is difficult to extend to high-dimensional data; in addition, random forest (RF) and KNN methods are also used for cloud recognition, but they have problems such as feature redundancy sensitivity and insufficient use of time series information. SUMMARY

[0004] The first object of the present application is to provide a multi-modal data fusion remote sensing image cloud classification method to improve the cloud recognition accuracy and stability in complex scenes.

[0005] The second object of the present application is to provide a multi-modal data fusion remote sensing image cloud classification system.

[0006] The third object of the present application is to provide an electronic device.

[0007] The fourth object of the present application is to provide a computer storage medium.

[0008] Technical solution: To achieve the above objects, the multi-modal data fusion remote sensing image cloud classification method according to the present application comprises the following steps: (1) Obtain multi-modal data from multiple remote sensing observation platforms and known cloud type labels, the multi-modal data including optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data, pre-process the multi-modal data to form input data, and divide the input data and the cloud type labels into a training set, a validation set and a test set according to a proportion; (2) The wavelet consistency alignment module performs multi-scale wavelet transform on the input data, extracts high-frequency and low-frequency sub-band coefficients, constructs a spectral distribution model between modalities based on wavelet energy spectrum, and aligns the features of different modal data through energy matching and spectral similarity constraint strategies to obtain a fusion representation with enhanced semantic consistency, that is, the wavelet feature tensor of each modality is obtained; (3) Construct a cloud classification backbone network model, the cloud classification backbone network model includes an encoder, a lightweight multi-modal wavelet fusion module and a decoder, wherein the encoder adopts a hierarchical structure composed of multiple levels, the encoder includes a PatchEmbedding module, a Swin-Transformer module and a down-sampling module, the wavelet feature tensor of each modality is input into the encoder for feature extraction, the Patch Embedding module is used for block embedding conversion, and then the Swin-Transformer module and the down-sampling module are used for feature extraction, and finally the minimum scale intermediate feature is obtained through the last level Swin-Transformer module, thereby obtaining a multi-scale intermediate feature map, and the multi-scale intermediate feature map is fused through the lightweight multi-modal wavelet fusion module at multiple scales to obtain a multi-scale fusion feature map; the decoder adopts a hierarchical structure symmetrical to the encoder, the decoder includes a Patch Embedding module, a Swin-Transformer module and an up-sampling module, the fused multi-scale fusion feature map is input into the decoder, the last level fusion feature map is input into the Swin-Transformer module and the up-sampling module for feature reconstruction after being input into the Swin-Transformer module and the up-sampling module through the Swin-Transformer module and the up-sampling module at each level, and then the sum of the fusion feature map of the last level and the fusion feature map of the previous level is input into the Swin-Transformer module and the up-sampling module for feature reconstruction, and finally the cloud classification result label map with the same resolution as the input is restored through the Patch Expanding module; (4) Train and evaluate the cloud classification backbone network model using the training set, the validation set and the test set to obtain an optimized cloud classification backbone network model, input the optical remote sensing cloud image, the cloud profile radar reflectivity data and the atmospheric physical property data to be classified into the optimized cloud classification backbone network model, and predict the optical remote sensing cloud image to be classified, and output the cloud classification result label map corresponding to each pixel point in the target region.

[0009] Optionally, the pre-processing in step (1) specifically includes the following steps: (1.1) Acquire optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, and unify the sampling time range and spatial coverage area. Then, register the cloud classification label of the optical remote sensing cloud image data with known cloud type labels through spatiotemporal registration. (1.2) Based on the imaging timestamps, orbital information and geographic location information of optical remote sensing cloud image data, the registration method is used to align different data sources to a unified geographic coordinate system; (1.3) Encapsulate optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data into input data with a consistent dimensional structure, and divide the input data and corresponding cloud classification labels into training set, validation set and test set according to the proportion.

[0010] Optionally, the wavelet consistency alignment module in step (2) performs multi-scale wavelet transform on the input data, specifically including the following steps: (2.1) Perform two-dimensional discrete wavelet transform on the input data respectively, and represent the input data as a set of multi-scale wavelet coefficients containing low-frequency approximate subbands and high-frequency detail subbands in multiple directions: , Among them Decompose hierarchy, Represents the first modal data type in the input data. Multi-scale wavelet coefficient set, subscript These refer to three different modal data types: opt, radar, and atm. The subscript opt ​​refers to the optical remote sensing cloud image data type, radar refers to the cloud profile radar reflectivity data type, and atm refers to the atmospheric physical property data type. Indicates the first The low-frequency approximate subband of the layer, Indicates the first Horizontal layer Directional high-frequency detail sub-band, Indicates the first Vertical layers Directional high-frequency detail sub-band, Indicates the first diagonal of the layer Directional high-frequency detail sub-band; (2.2) Calculate the energy spectrum for the wavelet subband coefficients at each mode and at each scale: , in For any modal data type in the input data, the first... Layer energy spectrum, Represents the spatial coordinates of the image; (2.3) Constructing inter-modal spectral distribution model, normalizing the energy spectrum value of different modalities at the same level into probability distribution form, which is used to describe the difference of wavelet energy response between modalities: , The normalized probability distribution of the energy spectrum value of any modal data type in the input data at the first level, while constructing the combination of inter-modal energy spectrum vectors , calculating the spectral distribution similarity between modalities, using KL divergence as the alignment index, and minimizing the probability distribution by introducing the spectral consistency loss function, so that the modalities tend to be unified in the multi-scale spatial frequency response: , Where KL is the KL divergence calculation method, subscript , is the spectral distribution similarity between two modalities; (2.4) According to the spectral distribution similarity, adjust the wavelet sub-band coefficients of each modality to match the energy space, and normalize each row, so that the "weight influence" of each modality on other modalities is 1. The final output of each modality is represented as the weighted sum of all modality features, and the weight is the spectral distribution similarity between the modalities: The final output of each modality is represented as the weighted sum of all modality features, and the weight is the spectral distribution similarity between the modalities: The wavelet feature tensor of each modality is obtained 、 、 ; , Where, is the normalized similarity weight between two modalities in the first level wavelet decomposition; , , , Where, is the normalized similarity weight between the opt modality and the opt modality in the first level wavelet decomposition, is the normalized similarity weight between the opt modality and the radar modality in the first level wavelet decomposition, is the normalized similarity weight between the opt modality and the atm modality in the first level wavelet decomposition, is the normalized similarity weight between the radar modality and the opt modality in the first level wavelet decomposition, It is in the Normalized similarity weights between radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between atm and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between ATM and Radar modes in layer wavelet decomposition. It is in the The normalized similarity weights between ATM modes in layer wavelet decomposition are used as weighting coefficients for wavelet feature co-reconstruction.

[0011] Optionally, step (3) of the Swin-Transformer module specifically includes the following steps: feature The input is processed by layer normalization in the LN module, and then input into the first layer standard window multi-head attention module. Within a fixed window partition, window-level multi-head self-attention computation is performed, denoted as: , For the input features, Indicates the size of the execution window; Adding residual connections yields features : , After residual connection After passing through the LN module again, the input is fed into the feedforward fully connected network MLP, where residual connections are added to obtain features. : , feature The input is processed by layer normalization in the LN module, and then input into the second layer shifted window multi-head attention module SW-MSA, which executes the shifted window attention mechanism. That is, it captures cross-window connection information through shifted windowing and adds residual connections to obtain features. : , in, This indicates that local attention calculation is performed after the adjacent layer windows are offset, which helps in modeling cross-window feature associations. Finally, the features After passing through the LN module again, the input is input into a feedforward fully connected network MLP, and a residual connection is added to obtain the feature : , wherein, is the output result of a Swin-Transformer module.

[0012] Optionally, the step S3 includes the following steps: First, the multi-scale intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of three different modalities are subjected to an encoder linear mapping to construct a cross-modal joint feature representation: , wherein, represents the intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of the optical remote sensing cloud image data; represents the intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of the cloud profile radar reflectivity data; represents the intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of the atmospheric physical property data; A channel selection mechanism is introduced to perform importance weighting on different modal dimension, and the multi-modal wavelet channel before fusion is denoted as C, and the fusion weight is obtained through the SE structure : , , wherein, GAP represents global average pooling; and are the learnable parameters of the fully connected layer; ⊙ is the channel-level point multiplication, and ReLU is the activation function, denotes a Sigmoid activation function, which is used to map the calculation result after the fully connected layer and the ReLU activation function to the interval (0, 1), thereby generating the channel selection weight γ; is the fusion result of the channel selection mechanism; The fused features are sent to a lightweight linear mapping layer: , wherein represents a dimension-compressed lightweight linear layer; The output features are fused with the cross-modal joint features: .

[0013] Based on the same inventive concept, the multi-modal data fusion remote sensing image cloud classification system comprises: The preparation data module is used for acquiring multi-modal data and known cloud type labels from multiple remote sensing observation platforms, the multi-modal data including optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data, the multi-modal data being preprocessed to form input data, and the input data and the cloud type labels being divided into a training set, a verification set and a test set according to a proportion; The wavelet transform module is used for performing multi-scale wavelet transform on the input data by the wavelet consistency alignment module, extracting high-frequency and low-frequency sub-band coefficients, constructing a spectral distribution model between modes based on wavelet energy spectrum, and performing feature alignment on different modal data through energy matching and spectral similarity constraint strategies to obtain a semantic consistency enhanced fusion representation, i.e., a wavelet feature tensor of each mode; The network construction module is used for constructing a cloud classification backbone network model, the cloud classification backbone network model including an encoder, a lightweight multi-modal wavelet fusion module and a decoder, wherein the encoder adopts a hierarchical structure composed of multiple levels, the encoder including a Patch Embedding module, a Swin-Transformer module and a down-sampling module, the wavelet feature tensor of each mode being input into the encoder for feature extraction, being converted by the Patch Embedding module for block embedding, and being extracted by the Swin-Transformer module and the down-sampling module at each level, and the minimum scale intermediate feature being obtained by the last level Swin-Transformer module to obtain a multi-scale intermediate feature map, the multi-scale intermediate feature map being fused by the lightweight multi-modal wavelet fusion module at multiple scales to obtain a multi-scale fusion feature map; the decoder adopts a hierarchical structure symmetrical to the encoder, the decoder including a Patch Embedding module, a Swin-Transformer module and an up-sampling module, the fused multi-scale fusion feature map being input into the decoder, the last level fusion feature map being reconstructed by the Swin-Transformer module and the up-sampling module at each level, being summed with the previous level fusion feature map, and being input into the Swin-Transformer module and the up-sampling module for feature reconstruction, and finally being restored to a cloud classification result label map with the same resolution as the input by the Patch Expanding module; The training and prediction module is used to train and evaluate the cloud classification backbone network model using training, validation, and test sets to obtain an optimized cloud classification backbone network model. The optical remote sensing cloud image to be classified, as well as cloud profile radar reflectivity data and atmospheric physical property data, are input into the optimized cloud classification backbone network model to predict the cloud image to be classified and output the cloud classification result label map corresponding to each pixel in the target area.

[0014] Optionally, the preprocessing in the data preparation module specifically includes: Acquire optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, and unify the sampling time range and spatial coverage area. Then, register the cloud classification labels of the optical remote sensing cloud image data with known cloud type labels through spatiotemporal registration. Based on the imaging timestamps, orbital information, and geographic location information of optical remote sensing cloud image data, a registration method is used to align different data sources to a unified geographic coordinate system. Optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data are encapsulated into input data with a consistent dimensional structure. The input data and corresponding cloud classification labels are divided into training set, validation set, and test set according to the proportion.

[0015] Optionally, the wavelet consistency alignment module based on the wavelet domain energy spectrum in the wavelet transform module performs multi-scale wavelet transform on the input data as follows: Perform two-dimensional discrete wavelet transforms on the input data to represent it as a set of multi-scale wavelet coefficients containing low-frequency approximate subbands and high-frequency detail subbands in multiple directions: , Among them Decompose hierarchy, Represents the first modal data type in the input data. Multi-scale wavelet coefficient set, subscript These refer to three different modal data types: opt, radar, and atm. The subscript opt ​​refers to the optical remote sensing cloud image data type, radar refers to the cloud profile radar reflectivity data type, and atm refers to the atmospheric physical property data type. Indicates the first The low-frequency approximate subband of the layer, Indicates the first Horizontal layer Directional high-frequency detail sub-band, Indicates the first Vertical layers Directional high-frequency detail sub-band, Indicates the first diagonal of the layer Directional high-frequency detail sub-band; Calculate the energy spectrum for the wavelet subband coefficients at each mode and at each scale: , in For any modal data type in the input data, the first... Layer energy spectrum, Represents the spatial coordinates of the image; A modal spectral distribution model is constructed to normalize the energy spectral values ​​of different modes at the same level into a probability distribution form, which is used to characterize the differences in wavelet energy responses between modes: , For any modal data type in the input data, the first... Normalized probability distribution of energy spectral values ​​at each level, and construction of energy spectral vector combinations between modes. The spectral distribution similarity between modes is calculated, and KL divergence is used as the alignment index. By introducing a spectral consistency loss function to minimize the probability distribution, the modes tend to be uniform in the multi-scale spatial frequency response. , Where KL represents the KL divergence calculation method, and the subscript... , The spectral distribution similarity between pairs of modes; Based on spectral distribution similarity, the wavelet subband coefficients of each mode are adjusted to match the energy space. Each row is normalized so that the "weighted influence" of the similarity of each mode on other modes sums to 1. The wavelet feature representation of the final output of each mode is a weighted sum of the features of all modes, where the weight is the spectral distribution similarity between a mode and other modes. The wavelet feature representation of the final output of each mode is a weighted sum of the features of all modes, resulting in the wavelet feature tensor of each mode. , , ; , in, It is in the Normalized similarity weights between pairwise modes in layered wavelet decomposition; , , , in, It is in the Normalized similarity weights between OPT modes in layer wavelet decomposition is a normalized similarity weight between the opt modality and the radar modality in the first layer wavelet decomposition, is a normalized similarity weight between the opt modality and the atm modality in the first layer wavelet decomposition, is a normalized similarity weight between the opt modality and the radar modality in the first layer wavelet decomposition, is a normalized similarity weight between the opt modality and the atm modality in the first layer wavelet decomposition, is a normalized similarity weight between the radar modality and the opt modality in the first layer wavelet decomposition, is a normalized similarity weight between the radar modality and the radar modality in the first layer wavelet decomposition, is a normalized similarity weight between the radar modality and the atm modality in the first layer wavelet decomposition, is a normalized similarity weight between the radar modality and the atm modality in the first layer wavelet decomposition, is a normalized similarity weight between the atm modality and the opt modality in the first layer wavelet decomposition, is a normalized similarity weight between the atm modality and the radar modality in the first layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the first layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the first layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the first layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the first layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the first layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the first layer wavelet decomposition,

[0016] Based on the same inventive concept, the electronic device provided by the application comprises a processor and a storage medium; The storage medium is used for storing instructions; The processor is used for operating according to the instructions to perform the steps of the method described above.

[0017] Based on the same inventive concept, the computer readable storage medium provided by the application has a computer program stored thereon, which is executed by a processor to realize the steps of the method described above.

[0018] Advantages: Compared with the prior art, the application has the following remarkable advantages: (1) The application fuses the wavelet decomposition features of multi-source remote sensing data and the deep Transformer structure, and realizes the collaborative modeling of different modalities in the unified spectral space and semantic space, thereby improving the cloud recognition accuracy and stability in complex scenes, aiming at the problems of weak multi-modal data fusion ability, limited spatial structure modeling, serious modality inconsistency and high model deployment complexity of the current remote sensing cloud classification method; (2) In the wavelet consistency alignment module, high and low frequency information of each mode is extracted through multi-scale wavelet transform, a spectrum distribution model is constructed based on an energy spectrum, and spectrum distribution differences between different modes are minimized by using KL divergence, so that the consistency of different modes in the frequency domain is realized, the inconsistent response problem between multi-modal data caused by different imaging mechanisms is effectively eliminated, the fusion quality is improved, the semantic consistency is enhanced while the diversity of mode characteristics is maintained, the energy weighted method is used for feature collaborative reconstruction, the expression ability of key modes is highlighted, the stability and discriminability of the final fusion representation are improved, and the subsequent classification accuracy is improved; (3) The cloud classification backbone network model adopts an encoder-decoder structure and introduces a multi-scale feature fusion module, the network model has the advantages of clear structure and module decoupling, and is convenient for unified processing and cross-scale fusion of different modal features, the encoder adopts a multi-layer Swin-Transformer structure, local and global spatial semantic information can be fully mined, the decoder uses a skip connection and an up-sampling strategy, high-resolution information recovery is ensured while the detail retention capability is enhanced, deep fusion is realized on a multi-scale semantic level, high discriminative features are generated, and the accuracy of cloud type identification is improved, and the overall architecture has strong adaptability and is easy to migrate and deploy; (4) The Swin-Transformer module combines window attention and shift window attention mechanisms, effectively reduces the complexity of global self-attention calculation, realizes more efficient feature modeling, the standard window attention (W-MSA) focuses on local context extraction, and the shift window attention (SW-MSA) enhances the cross-window information interaction capability, the hierarchical embedding and multi-scale modeling characteristics conform to the spatial hierarchical feature structure in remote sensing images, are helpful for detailed modeling of cloud texture, boundary, morphology and other features, improve the recognition accuracy and stability, and support flexible window size design, adapt to remote sensing image inputs of different resolutions; (5) The lightweight multi-modal wavelet fusion module introduces a channel selection mechanism and a lightweight linear mapping to fuse multi-modal features, can realize significant weighting between different modes, and highlight channels with higher information quantity or higher sensitivity to classification, dynamic channel weight learning is realized by using a Squeeze-and-Excitation structure, the fusion accuracy is improved, the network parameter quantity is low, the lightweight deployment requirement is met, and the module is suitable for embedded or resource-limited scenes, multi-scale fusion operation enhances the understanding of cloud structure on different spatial levels, and on the premise of not sacrificing performance, the model efficiency and inference speed are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of the present application; Figure 2This is a flowchart illustrating the wavelet consistency alignment module in this invention. Figure 3 This is a flowchart illustrating the cloud classification backbone network model in this invention. Figure 4 This is a schematic diagram of the framework of the Swin-Transformer module in this invention; Figure 5 This is a flowchart illustrating the lightweight multimodal wavelet fusion module of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0021] Example 1: As Figure 1 As shown in this embodiment, a remote sensing image cloud classification method based on multimodal data fusion includes the following steps: (1) Obtain multimodal data and known cloud type labels from multiple remote sensing observation platforms. The multimodal data includes optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data. The multimodal data are preprocessed to form input data. The input data and cloud type labels are divided into training set, validation set and test set according to the ratio. The preprocessing in step (1) specifically includes the following steps: (1.1) Acquire optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, and unify the sampling time range and spatial coverage area. Then, register the cloud classification labels of the optical remote sensing cloud image data with the cloud type labels provided by the CloudSat satellite through spatiotemporal registration. The optical remote sensing cloud image data can be data from the MODIS satellite or the FY-4 satellite. The cloud profile radar reflectivity data can be data from the radar CPR carried by the CloudSat satellite. The atmospheric physical property data includes numerical fields such as temperature, humidity, air pressure, and wind speed. (1.2) Based on the imaging timestamp, orbit information and geographic location information of optical remote sensing cloud image data, the registration method is used to align different data sources to a unified geographic coordinate system to ensure the consistency of multimodal data in time and space dimensions; the registration method can be affine transformation, affine projection or RPC model, etc. (1.3) Encapsulate the optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data into input data with a consistent dimensional structure. Upsample the low-resolution data in the input data using the interpolation resampling method. Convert the data format of each modality into a unified tensor structure input format to adapt to the input requirements of the wavelet consistency alignment module in step (2). Divide the input data and the corresponding cloud classification labels into training set, validation set and test set according to 7:2:1. (2) The wavelet consistency alignment module based on the wavelet domain energy spectrum performs multi-scale wavelet transform on the input data, extracts high-frequency and low-frequency subband coefficients, constructs a spectral distribution model between modes based on the wavelet domain energy spectrum, and performs feature alignment on different mode data through energy matching and spectral similarity constraint strategies to obtain a fusion representation with enhanced semantic consistency, that is, to obtain the wavelet feature tensor of each mode. like Figure 2 As shown, step (2) of the wavelet consistency alignment module based on the wavelet domain energy spectrum performs multi-scale wavelet transform on the input data, specifically including the following steps: (2.1) Perform two-dimensional discrete wavelet transform (DWT) on the preprocessed input data to represent the input data as a set of multi-scale wavelet coefficients containing low-frequency approximation (A) subbands and high-frequency detail subbands in multiple directions: , in To decompose the hierarchy, Represents the first modal data type in the input data. Multi-scale wavelet coefficient set, subscript These refer to three different modal data types: opt, radar, and atm. The subscript opt ​​refers to the optical remote sensing cloud image data type, radar refers to the cloud profile radar reflectivity data type, and atm refers to the atmospheric physical property data type. Indicates the first The low-frequency approximate subband of the layer, Indicates the first Horizontal layer Directional high-frequency detail sub-band, Indicates the first Vertical layers Directional high-frequency detail sub-band, Indicates the first diagonal of the layer Directional high-frequency detail sub-band; (2.2) Calculate the energy spectrum for the wavelet subband coefficients at each mode and at each scale: , in For any modal data type in the input data, the first... Layer energy spectrum, Represents the spatial coordinates of the image; (2.3) Construct an intermodal spectral distribution model to normalize the energy spectral values ​​of different modes at the same level into a probability distribution form, which is used to characterize the differences in wavelet energy responses between modes: , the first energy spectrum value normalization probability distribution under the hierarchy, while constructing the energy spectrum vector combination between modalities , calculate the spectral distribution similarity between modalities, use KL divergence (Kullback-Leibler divergence) as the alignment index, and minimize the probability distribution by introducing the spectral consistency loss function, so that each modality tends to be unified in the multi-scale spatial frequency response: , where KL is the KL divergence calculation method, and the subscript , is the spectral distribution similarity between two modalities; Specifically: , , , , , , , , , where is the spectral distribution similarity between the opt modality and the opt modality, is the spectral distribution similarity between the opt modality and the radar modality, is the spectral distribution similarity between the opt modality and the atm modality, is the spectral distribution similarity between the radar modality and the opt modality, is the spectral distribution similarity between the radar modality and the radar modality, is the spectral distribution similarity between the radar modality and the atm modality, is the spectral distribution similarity between the atm modality and the opt modality, is the spectral distribution similarity between the atm modality and the radar modality, is the spectral distribution similarity between the atm modality and the atm modality; (2.4) According to the spectral distribution similarity, adjust the wavelet sub-band coefficients of each modality to match the energy space, and normalize each row, so that the "weight influence" of each modality on other modalities is 1. The final output of each modality is represented as a weighted sum of all modality features, where the weight is the spectral distribution similarity between a modality and other modalities: The final output of each modality is represented as a weighted sum of all modality features, and the wavelet feature tensor of each modality is obtained , and , and then sent to the cloud classification backbone network model WaMCCformer for feature extraction; , wherein, is the normalized similarity weight between the two modalities in the wavelet decomposition of the first layer; Specifically, , , , , , , , , , ,

[0022] wherein, is the normalized similarity weight between the opt modality and the opt modality in the wavelet decomposition of the first layer, is the normalized similarity weight between the opt modality and the radar modality in the wavelet decomposition of the first layer, is the normalized similarity weight between the opt modality and the atm modality in the wavelet decomposition of the first layer, is the normalized similarity weight between the radar modality and the opt modality in the wavelet decomposition of the first layer, is the normalized similarity weight between the radar modality and the radar modality in the wavelet decomposition of the first layer, is the normalized similarity weight between the radar modality and the atm modality in the wavelet decomposition of the first layer, is the normalized similarity weight between the atm modality and the opt modality in the wavelet decomposition of the first layer, is the normalized similarity weight between the atm modality and the radar modality in the wavelet decomposition of the first layer, is the normalized similarity weight between the atm modality and the atm modality in the wavelet decomposition of the first layer, which is a weighting coefficient used in wavelet feature collaborative reconstruction.

[0023] (3) Construct a cloud classification backbone network model WaMCCformer to jointly model and information fuse the wavelet feature tensors of different modalities, and generate discriminative multi-modal fusion features; As Figure 3 shown, the cloud classification backbone network model WaMCCformer includes an encoder, a lightweight multi-modal wavelet fusion module, and a decoder, wherein the encoder includes a hierarchical structure composed of multiple levels, the encoder includes a PatchEmbedding module, a Swin-Transformer module, and a down-sampling module, the PatchEmbedding module is used to convert the wavelet feature tensor of each modality into a token representation through block embedding, and the Swin-Transformer module is used to extract spatial-semantic context features within a local window range; the down-sampling module is used to reduce the feature map resolution and increase the receptive field layer by layer; the encoder includes four down-sampling stages, wherein the first three stages each contain two Swin-Transformer modules, and the last layer contains one Swin-Transformer module; the wavelet feature tensor of each modality is input into the encoder for feature extraction, and then converted into a token representation through block embedding by the PatchEmbedding module, and then the features are extracted by the Swin-Transformer module and the down-sampling module step by step, and finally the minimum scale intermediate features are obtained through the last Swin-Transformer module, thereby obtaining a multi-scale intermediate feature map, and the multi-scale intermediate feature map is fused at multiple scales through the lightweight multi-modal wavelet fusion module to obtain a multi-scale fusion feature map; The decoder adopts a hierarchical structure symmetrical to the encoder, and the decoder includes a PatchEmbedding module, a Swin-Transformer module, and an up-sampling module, the fused multi-scale fusion feature map is input into the decoder, the last level fusion feature map is reconstructed through the Swin-Transformer module and the up-sampling module step by step, and then summed with the fusion feature map of the previous level, and then input into the Swin-Transformer module and the up-sampling module for feature reconstruction, and finally restored to a cloud class result label map with the same resolution as the input through the Patch Expanding module; The decoder adopts an up-sampling strategy to gradually restore the feature map resolution, each up-sampling stage is equipped with two or one Swin-Transformer module, which is used to further aggregate high-resolution features, and the corresponding encoder stage shares intermediate features through a skip connection, and finally restores a cloud class result label map with the same resolution as the input through the Patch Expanding module. The lightweight multi-modal wavelet fusion module constitutes a feature fusion and extraction stage, and the fusion position of each encoder level and the lightweight multi-modal wavelet fusion module is kept synchronous, ensuring that multi-modal features are guided and aggregated at different semantic levels, and the output multi-modal features have high discriminability and spatial consistency, which are used for cloud class result output.

[0024] As shown in Figure 4 , the step (3) includes the following steps: Input features are input into the LN (Layer Normalization) module for layer normalization processing, and then input into the first layer of the standard window multi-head attention module The window-level multi-head self-attention calculation is performed in the fixed window division, denoted as: , For the input features , represents the size of the window executed; The residual connection is added to obtain the feature : , The feature after the residual connection is input into the LN module again, and then input into the feedforward full connection network MLP, and the residual connection is added to obtain the feature : , The feature is input into the LN module for layer normalization processing, and then input into the second layer of the shift window multi-head attention module SW-MSA, and the shift window attention mechanism is executed, that is, the cross-window connection information is captured through the shift window processing, and the residual connection is added to obtain the feature : , Wherein, represents the local attention calculation after the window offset of the adjacent layer, which is helpful for modeling the cross-window feature association; Finally, the feature is input into the LN module again, and then input into the feedforward full connection network MLP, and the residual connection is added to obtain the feature : , Wherein, is the output result of a Swin-Transformer module.

[0025] The Swin-Transformer module can effectively capture the local-global cloud image spatial semantic information through the hierarchical combination of the W-MSA and SW-MSA attention mechanisms, and assist with the double-layer MLP and residual structure, which can improve the cloud class feature extraction capability, and is the core component module of the cloud classification backbone network model WaMCCformer.

[0026] As shown inFigure 5 As shown, the lightweight multi-modal wavelet fusion module in step S3 specifically includes the following steps: First, the multi-scale intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of three different modalities are subjected to encoder linear mapping to construct cross-modal joint feature representation: , Wherein, represents the intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of the optical remote sensing cloud image data; represents the intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of the cloud profile radar reflectivity data; represents the intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of the atmospheric physical property data; This step forms a multi-modal fusion base feature by retaining the original information of each modal feature, preparing for subsequent attention selection.

[0027] A channel selection mechanism is introduced to perform importance weighting on different modal dimensions. Let the multi-modal wavelet channel before fusion be C, and the fusion weight is obtained through the SE (Squeeze-and-Excitation) structure : , , Wherein, GAP represents global average pooling; and are the learnable parameters of the fully connected layer; ⊙ is the channel-level dot product, and ReLU is the activation function, denotes the Sigmoid activation function, which is used to map the calculation results after the fully connected layer and the ReLU activation function to the interval (0, 1), thereby generating the channel selection weight γ; is the fusion result of the channel selection mechanism; To ensure network lightweight, the fused features are sent to a lightweight linear mapping layer: , Wherein represents a dimension-compressed lightweight linear layer for mapping the multi-modal input back to the backbone network dimension; The output features are fused with the cross-modal joint features to inject information into the backbone network, enhance the cross-modal representation learning ability, and maintain information consistency: .

[0028] (4) training the cloud classification backbone network model using the training set and the validation set, performing inference evaluation on the test set using the trained cloud classification backbone network model, obtaining the classification performance indicators of the cloud classification backbone network model on each cloud type, inputting the optical remote sensing cloud image to be classified and the cloud profile radar reflectivity data and the atmospheric physical property data into the optimized cloud classification backbone network model, predicting the optical remote sensing cloud image to be classified, and outputting the cloud type label map corresponding to each pixel point in the target region, thereby realizing the automatic cloud classification task of the target region.

[0029] Embodiment 2: Based on the same inventive concept, a multi-modal data fusion remote sensing image cloud classification system is provided, which comprises: A data preparation module is configured to obtain multi-modal data from multiple remote sensing observation platforms and known cloud type labels, the multi-modal data including optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, and the multi-modal data is preprocessed to form input data, and the input data and the cloud type labels are divided into a training set, a validation set, and a test set according to a ratio.

[0030] In the data preparation module, the optical remote sensing cloud image data, the cloud profile radar reflectivity data, and the atmospheric physical property data are obtained, and the sampling time range and the spatial coverage area are unified, and then the time and space are registered with the cloud type labels provided by the CloudSat satellite to obtain the cloud classification labels of the optical remote sensing cloud image data; the optical remote sensing cloud image data can be selected from the data of the MODIS satellite or the FY-4 satellite, the cloud profile radar reflectivity data can be selected from the data of the radar CPR carried by the CloudSat satellite, and the atmospheric physical property data includes temperature, humidity, air pressure, wind speed, and other numerical fields.

[0031] Based on the imaging timestamp, orbit information, and geographic positioning information of the optical remote sensing cloud image data, a registration method is used to align different data sources to a unified geographic coordinate system, ensuring the correspondence consistency of the multi-modal data in the time and spatial dimensions; the registration method can be selected from affine transformation, affine projection, or RPC model.

[0032] The optical remote sensing cloud image data, the cloud profile radar reflectivity data, and the atmospheric physical property data are packaged into input data with consistent dimensional structures, the low-resolution data in the input data is up-sampled using an interpolation resampling method, the data forms of each modality are converted into a unified tensor structure input format to adapt to the input requirements of the wavelet consistency alignment module in the wavelet transform module; the input data and the corresponding cloud classification labels are divided into a training set, a validation set, and a test set according to a ratio of 7:2:1.

[0033] The wavelet transform module performs multi-scale wavelet transform on the input data based on the wavelet domain energy spectrum, extracts high-frequency and low-frequency subband coefficients, constructs a spectral distribution model between modes based on the wavelet domain energy spectrum, and performs feature alignment on different modal data through energy matching and spectral similarity constraint strategies to obtain a fusion representation with enhanced semantic consistency, that is, to obtain the wavelet feature tensor of each mode.

[0034] like Figure 2 As shown, the wavelet consistency alignment module based on wavelet domain energy spectrum in the wavelet transform module performs multi-scale wavelet transform on the input data as follows: The preprocessed input data is subjected to two-dimensional discrete wavelet transform (DWT) to represent the input data as a set of multi-scale wavelet coefficients containing low-frequency approximation (A) subbands and high-frequency detail subbands in multiple directions: , in To decompose the hierarchy, Represents the first modal data type in the input data. Multi-scale wavelet coefficient set, subscript These refer to three different modal data types: opt, radar, and atm. The subscript opt ​​refers to the optical remote sensing cloud image data type, radar refers to the cloud profile radar reflectivity data type, and atm refers to the atmospheric physical property data type. Indicates the first The low-frequency approximate subband of the layer, Indicates the first Horizontal layer Directional high-frequency detail sub-band, Indicates the first Vertical layers Directional high-frequency detail sub-band, Indicates the first diagonal of the layer Directional high-frequency detail sub-band.

[0035] Calculate the energy spectrum for the wavelet subband coefficients at each mode and at each scale: , in For any modal data type in the input data, the first... Layer energy spectrum, Represents the spatial coordinates of the image.

[0036] A modal spectral distribution model is constructed to normalize the energy spectral values ​​of different modes at the same level into a probability distribution form, which is used to characterize the differences in wavelet energy responses between modes: , The first energy spectrum value normalization probability distribution of any modal data type in the input data is constructed The energy spectrum vector combination between the modes is constructed at the same time The spectrum distribution similarity between the modes is calculated, the KL divergence (Kullback-Leibler divergence) is used as the alignment index, and the probability distribution is minimized by introducing the spectrum consistency loss function, so that the modes tend to be unified in the multi-scale space frequency response: , Wherein KL is the KL divergence calculation method, and the subscript , is the spectrum distribution similarity between the two modes; Specifically: , , , , , , , , , Wherein is the spectrum distribution similarity between the opt mode and the opt mode, is the spectrum distribution similarity between the opt mode and the radar mode, is the spectrum distribution similarity between the opt mode and the atm mode, is the spectrum distribution similarity between the radar mode and the opt mode, is the spectrum distribution similarity between the radar mode and the radar mode, is the spectrum distribution similarity between the radar mode and the atm mode, is the spectrum distribution similarity between the atm mode and the opt mode, is the spectrum distribution similarity between the atm mode and the radar mode, is the spectrum distribution similarity between the atm mode and the atm mode.

[0037] According to the spectrum distribution similarity, the wavelet sub-band coefficients of each mode are adjusted to match the energy space, and each row is normalized, so that the “weight influence” of each mode on other modes is 1. The final output of each mode is represented as the weighted sum of all modal features, and the weight is the spectrum distribution similarity between the mode and other modes: The final output of each mode is represented as the weighted sum of all modal features, and the wavelet feature tensor of each mode is obtained , and and then sent into a cloud classification backbone network model WaMCCformer for feature extraction; , wherein, is the normalized similarity weight between the two modes in the first layer wavelet decomposition; Specifically, , , , , , , , , , ,

[0038] wherein, is the normalized similarity weight between the opt mode and the opt mode in the first layer wavelet decomposition, is the normalized similarity weight between the opt mode and the radar mode in the first layer wavelet decomposition, is the normalized similarity weight between the opt mode and the atm mode in the first layer wavelet decomposition, is the normalized similarity weight between the radar mode and the opt mode in the first layer wavelet decomposition, is the normalized similarity weight between the radar mode and the radar mode in the first layer wavelet decomposition, is the normalized similarity weight between the radar mode and the atm mode in the first layer wavelet decomposition, is the normalized similarity weight between the atm mode and the opt mode in the first layer wavelet decomposition, is the normalized similarity weight between the atm mode and the radar mode in the first layer wavelet decomposition, is the normalized similarity weight between the atm mode and the atm mode in the first layer wavelet decomposition, which is a weighting coefficient used in wavelet feature collaborative reconstruction.

[0039] The network construction module is configured to construct a cloud classification backbone network model WaMCCformer, jointly model wavelet feature tensors of different modalities, and perform information fusion to generate discriminative multi-modal fusion features. As shown in Figure 3 The cloud classification backbone network model WaMCCformer includes an encoder, a lightweight multi-modal wavelet fusion module and a decoder. The encoder includes a hierarchical structure composed of multiple levels. The encoder includes a PatchEmbedding module, a Swin-Transformer module and a down-sampling module. The PatchEmbedding module is configured to convert the wavelet feature tensors of each modality into token representations through block embedding. The Swin-Transformer module is configured to extract spatial-semantic context features within a local window. The down-sampling module is configured to reduce the resolution of the feature map and increase the receptive field layer by layer. The encoder includes four down-sampling stages. The first three stages each include two Swin-Transformer modules, and the last stage includes one Swin-Transformer module. The wavelet feature tensors of each modality are input into the encoder for feature extraction. The wavelet feature tensors are converted into token representations through the PatchEmbedding module, and then are extracted through the Swin-Transformer modules and the down-sampling module layer by layer. The smallest scale intermediate features are obtained through the last Swin-Transformer module, thereby obtaining multi-scale intermediate feature maps. The multi-scale intermediate feature maps are fused at multiple scales through the lightweight multi-modal wavelet fusion module to obtain multi-scale fusion feature maps. The decoder adopts a hierarchical structure symmetrical to the encoder. The decoder includes a PatchEmbedding module, a Swin-Transformer module and an up-sampling module. The fused multi-scale fusion feature maps are input into the decoder. The last level fusion feature map is reconstructed through the Swin-Transformer module and the up-sampling module, and then is summed with the fusion feature map of the previous level. The summed fusion feature map is input into the Swin-Transformer module and the up-sampling module for feature reconstruction. Finally, the cloud classification result label map with the same resolution as the input is restored through the PatchExpanding module. The decoder gradually recovers the resolution of the feature map by using an upsampling strategy, each upsampling stage is equipped with two or one Swin-Transformer module, which is used for further aggregation of high-resolution features, and the corresponding encoder stage shares the intermediate features through a jump connection, and finally restores the cloud class result label map with the same resolution as the input through the Patch Expanding module. The lightweight multi-modal wavelet fusion module constitutes the feature fusion and extraction stage, and each encoder level is synchronized with the fusion position of the lightweight multi-modal wavelet fusion module, ensuring that multi-modal features are guided and aggregated at different semantic levels. The output multi-modal features have high discriminability and spatial consistency, which are used for cloud class result output.

[0040] As shown in Figure 4 , the Swin-Transformer module specifically includes: Input features are input into the first layer standard window multi-head attention module SW-MSA after layer normalization processing in the LN (Layer Normalization) module. The window-level multi-head self-attention calculation is performed in the fixed window division, denoted as: , is the input feature, denotes the size of the window executed; The residual connection is added to obtain the feature : , The feature after the residual connection is input into the feedforward full connection network MLP again after LN module, and the residual connection is added to obtain the feature : , The feature is input into the second layer shift window multi-head attention module SW-MSA after layer normalization processing in the LN module, and the shift window attention mechanism is executed, that is, the cross-window connection information is captured through the shift window processing, and the residual connection is added to obtain the feature : , wherein, denotes that the local attention calculation is performed after the window offset of the adjacent layer, which is helpful for modeling the cross-window feature association; Finally, the feature is input into the feedforward full connection network MLP again after the LN module, and the residual connection is added to obtain the feature : , wherein, is the output result of a Swin-Transformer module.

[0041] The Swin-Transformer module can effectively capture local-global cloud map spatial semantic information and improve cloud class feature extraction capability through the hierarchical combination of W-MSA and SW-MSA attention mechanisms, supplemented by a double-layer MLP and a residual structure, and is the core component module of the cloud classification backbone network model WaMCCformer.

[0042] As shown in Figure 5 , the lightweight multi-modal wavelet fusion module is specifically: First, the multi-scale intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of three different modalities are encoded and linearly mapped to construct cross-modal joint feature representation: , wherein, represents the intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of the optical remote sensing cloud image data; represents the intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of the cloud profile radar reflectivity data; represents the intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of the atmospheric physical property data; This step forms a multi-modal fusion basic feature by retaining the original information of each modal feature, preparing for subsequent attention selection.

[0043] A channel selection mechanism is introduced to perform importance weighting on different modal dimensions. Let the multi-modal wavelet channel before fusion be C, and the fusion weight is obtained through the SE (Squeeze-and-Excitation) structure : , , wherein, GAP represents global average pooling; and are the learnable parameters of the fully connected layer; ⊙ is the channel-level point multiplication, and ReLU is the activation function, denotes the Sigmoid activation function, which is used to map the calculation results after the fully connected layer and the ReLU activation function to the interval (0, 1), thereby generating the channel selection weight γ; is the fusion result of the channel selection mechanism; To ensure the lightness of the network, the fused features are sent to a light linear mapping layer: , wherein represents a dimension-compressed light linear layer for mapping the multi-modal input back to the backbone network dimension; The output features are fused with the cross-modal joint features to perform information injection on the backbone network, enhance the cross-modal representation learning capability, and maintain information consistency: .

[0044] The training prediction module is used to train the cloud classification backbone network model using the training set and the validation set, to infer and evaluate the test set using the trained cloud classification backbone network model, to obtain the classification performance indicators of the cloud classification backbone network model on each cloud type, to input the optical remote sensing cloud image to be classified and the cloud profile radar reflectivity data and the atmospheric physical property data into the optimized cloud classification backbone network model, to predict the optical remote sensing cloud image to be classified, to output the cloud type label map corresponding to each pixel point in the target region, and to realize the automatic cloud classification task of the target region.

[0045] Embodiment 3: An electronic device in this embodiment includes a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to perform the steps of the method as described above.

[0046] Embodiment 4: A computer readable storage medium in this embodiment has a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described above.

Claims

1. A remote sensing image cloud classification method based on multimodal data fusion, characterized in that, The method comprises the following steps: (1) obtaining multi-modal data from multiple remote sensing observation platforms and known cloud type labels, the multi-modal data including optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data, pre-processing the multi-modal data to form input data, and dividing the input data and the cloud type labels into a training set, a validation set and a test set according to a proportion; (2) a wavelet consistency alignment module performs multi-scale wavelet transform on the input data, extracts high-frequency and low-frequency sub-band coefficients, constructs a spectral distribution model between modalities based on wavelet energy spectrum, and aligns the features of different modal data through energy matching and spectral similarity constraint strategies to obtain a semantic consistency enhanced fusion representation, that is, the wavelet feature tensor of each modality is obtained; (3) a cloud classification backbone network model is constructed, the cloud classification backbone network model comprising an encoder, a lightweight multi-modal wavelet fusion module and a decoder, wherein the encoder adopts a hierarchical structure composed of multiple levels, and the encoder comprises a PatchEmbedding module, a Swin-Transformer module and a down-sampling module; the wavelet feature tensor of each modality is input into the encoder for feature extraction, is converted through block embedding by the Patch Embedding module, and is extracted by the Swin-Transformer module and the down-sampling module at each level; the minimum scale intermediate feature is obtained through the last level Swin-Transformer module, so that the multi-scale intermediate feature map is obtained; the multi-scale intermediate feature map is fused at multiple scales by the lightweight multi-modal wavelet fusion module to obtain a multi-scale fusion feature map; the decoder adopts a hierarchical structure symmetrical to the encoder, and the decoder comprises a Patch Embedding module, a Swin-Transformer module and an up-sampling module; the fused multi-scale fusion feature map is input into the decoder; the last level fusion feature map is reconstructed by the Swin-Transformer module and the up-sampling module at each level, is summed with the fusion feature map of the previous level, and is then input into the Swin-Transformer module and the up-sampling module for feature reconstruction; finally, the cloud classification result label map with the same resolution as the input is restored by the Patch Expanding module; (4) the training set, the validation set and the test set are used to train and evaluate the cloud classification backbone network model to obtain an optimized cloud classification backbone network model; the optical remote sensing cloud image to be classified, the cloud profile radar reflectivity data and the atmospheric physical property data are input into the optimized cloud classification backbone network model; the optical remote sensing cloud image to be classified is predicted, and a cloud classification result label map corresponding to each pixel point in a target region is output.

2. The method according to claim 1, wherein: The pre-processing in step (1) specifically comprises the following steps: (1.1) Obtain optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, unify the sampling time range and spatial coverage area, and then perform registration with known cloud type labels to obtain cloud classification labels for the optical remote sensing cloud image data; (1.2) Align different data sources to a unified geographic coordinate system based on the imaging timestamp, orbit information, and geographic positioning information of the optical remote sensing cloud image data; (1.3) Encapsulate the optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data into input data with consistent dimensional structures, and divide the input data and corresponding cloud classification labels into training sets, validation sets, and test sets according to a proportion. 3.The multi-modal data fusion-based remote sensing image cloud classification method of claim 1, wherein: The wavelet consistency alignment module in step (2) performs multi-scale wavelet transform on the input data, specifically including the following steps: (2.1) Perform two-dimensional discrete wavelet transform on the input data to represent the input data as a set of multi-scale wavelet coefficients including low-frequency approximation subbands and multiple directional high-frequency detail subbands: ， wherein decomposition level, denote the first layer multiscale wavelet coefficient set, subscript denote three different modal data types, respectively, opt, radar and atm, wherein subscript opt denotes the optical remote sensing cloud image data type, radar denotes the cloud profile radar reflectivity data type, and atm denotes the atmospheric physical property data type; denote the first layer low-frequency approximation subband, denote the first layer horizontal direction high-frequency detail subband, denote the first layer vertical direction high-frequency detail subband, denote the first layer diagonal direction high-frequency detail subband; (2.2) Calculate the energy spectrum of the wavelet subband coefficients at each scale and each modality: ， wherein is a first layer energy spectrum, denotes image space coordinates; (2.3) Construct an inter-modality spectral distribution model to normalize the energy spectrum values of different modalities at the same level into a probability distribution form, which is used to characterize the differences in wavelet energy response between modalities: ， The first energy spectrum value normalization probability distribution of any modal data type in the input data is calculated The energy spectrum value normalization probability distribution of any modal data type in the input data is calculated The spectrum distribution similarity between the modal data is calculated, the KL divergence is used as the alignment index, and the probability distribution is minimized by introducing the spectrum consistency loss function, so that the modal data tends to be unified in the multi-scale space frequency response. ， where KL is the KL divergence calculation method, subscript , is the spectral distribution similarity between each pair of modalities; (2.4) According to the spectral distribution similarity, the wavelet sub-band coefficients of each modality are adjusted to match the energy space, each row is normalized, and the "weight influence" of each modality on other modalities is 1. The final output of the wavelet feature of each modality is represented as a weighted sum of all modality features, and the weight is the spectral distribution similarity between one modality and other modalities: the final output of the wavelet feature of each modality is represented as a weighted sum of all modality features, and the wavelet feature tensor of each modality is obtained , , ; ， wherein, is the first wavelet decomposition of the first layer; is the normalized similarity weight between two modalities in the wavelet decomposition of the first layer; ， ， ， wherein, is a normalized similarity weight between the opt modality and the opt modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the opt modality and the radar modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the opt modality and the atm modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the radar modality and the opt modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the radar modality and the radar modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the radar modality and the atm modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the atm modality and the opt modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the atm modality and the radar modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the opt modality and the opt modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the opt modality and the radar modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the opt modality and the atm modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the radar modality and the opt modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the radar modality and the radar modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the radar modality and the atm modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the atm modality and the opt modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the atm modality and the radar modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the 2nd layer wavelet decomposition, 4. The method of claim 1, wherein: The Swin-Transformer module in step (3) specifically includes the following steps: Features The layer normalization processing is performed in the input LN module, and then the first layer standard window multi-head attention module is input The window-level multi-head self-attention calculation is performed within fixed window division, denoted as: ， for the input feature, denotes the size of the execution window; Adding a residual connection gets features : ， After the residual connection, the input is input to the feedforward fully connected network MLP again, and the residual connection is added to obtain the feature After the LN module, the input is input to the feedforward fully connected network MLP again, and the residual connection is added to obtain the feature : ， Features The layer normalization processing is performed in the input LN module, and then the second layer shift window multi-head attention module SW-MSA is input, and the shift window attention mechanism is executed, that is, the cross-window connection information is captured through the shift window processing, and the residual connection is added to obtain the features : ， wherein, denotes the local attention computation after the window offset of the adjacent layer, which helps to model the cross-window feature association. Finally the features After passing through the LN module again, input into the feedforward fully connected network MLP, add residual connection to get the features : ， wherein, is an output result of a Swin-Transformer module.

5. The method of claim 1, wherein: The lightweight multi-modal wavelet fusion module in step S3 specifically includes the following steps: First, the multi-scale intermediate features output by the Swin-Transformer module processing the wavelet tensor feature maps of the three different modalities are encoded and linearly mapped to construct cross-modality joint feature representations: ， wherein, represents the intermediate feature output by the Swin-Transformer module after processing the wavelet tensor feature map of the optical remote sensing cloud image data; represents the intermediate feature output by the Swin-Transformer module after processing the wavelet tensor feature map of the cloud profile radar reflectivity data; represents the intermediate feature output by the Swin-Transformer module after processing the wavelet tensor feature map of the atmospheric physical property data; A channel selection mechanism is introduced to perform importance weighting on different modal dimensions, where the multi-modal wavelet channels before fusion are denoted as C, and the fusion weights are obtained through the SE structure : ， ， wherein GAP represents global average pooling; and are learnable parameters of the fully connected layer; is a channel-level dot product, and ReLU is an activation function, denotes a Sigmoid activation function, which is used to map the calculation result after the fully connected layer and the ReLU activation function to the interval (0, 1), thereby generating a channel selection weight γ; is the fusion result of the channel selection mechanism. The fused features are sent to a light linear mapping layer: , wherein represents a dimensionally compressed light linear layer; The output features are fused with the cross-modal joint features: .

6. A remote sensing image cloud classification system of multi-modal data fusion, characterized in that, It includes: A data preparation module is used to obtain multi-modal data and known cloud type labels from multiple remote sensing observation platforms. The multi-modal data includes optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data. The multi-modal data is preprocessed to form input data, and the input data and cloud type labels are divided into training sets, validation sets, and test sets according to a proportion; A wavelet transform module is used to perform multi-scale wavelet transform on the input data by the wavelet consistency alignment module, extract high-frequency and low-frequency subband coefficients, construct an inter-modality spectral distribution model based on wavelet energy spectrum, and perform feature alignment on different modal data through energy matching and spectral similarity constraint strategies to obtain a semantic consistency enhanced fusion representation, i.e., the wavelet feature tensor of each modality. The network construction module is configured to construct a cloud classification backbone network model, the cloud classification backbone network model comprising an encoder, a lightweight multi-modal wavelet fusion module and a decoder, wherein the encoder adopts a hierarchical structure composed of multiple levels, the encoder comprising a Patch Embedding module, a Swin-Transformer module and a down-sampling module, the wavelet feature tensor of each modality being input into the encoder for feature extraction, the Patch Embedding module being used to perform block embedding conversion, the Swin-Transformer module and the down-sampling module being used to perform feature extraction step by step, the minimum scale intermediate feature being obtained through the last stage Swin-Transformer module, thereby obtaining a multi-scale intermediate feature map, the multi-scale intermediate feature map being fused on multiple scales through the lightweight multi-modal wavelet fusion module to obtain a multi-scale fusion feature map; The decoder adopts a hierarchical structure symmetrical to the encoder, the decoder comprising a Patch Embedding module, a Swin-Transformer module and an up-sampling module, the fused multi-scale fusion feature map being input into the decoder, the last stage fusion feature map being input into the Swin-Transformer module and the up-sampling module for feature reconstruction after being input into the Swin-Transformer module and the up-sampling module for feature reconstruction after being summed with the fusion feature map of the previous stage, and finally being restored to a cloud classification result label map with the same resolution as the input through the Patch Expanding module; The training and prediction module is configured to train and evaluate the cloud classification backbone network model using a training set, a validation set and a test set to obtain an optimized cloud classification backbone network model, input the optical remote sensing cloud image to be classified and the cloud profile radar reflectivity data and the atmospheric physical property data into the optimized cloud classification backbone network model, predict the optical remote sensing cloud image to be classified, and output a cloud classification result label map corresponding to each pixel point in the target region.

7. The remote sensing image cloud classification system of claim 1, wherein: The preprocessing in the preparation data module is specifically: Obtain optical remote sensing cloud image data, cloud profile radar reflectivity data, atmospheric physical property data, and unify the sampling time range and spatial coverage area, and then align the optical remote sensing cloud image data with the known cloud type label through time and space registration to obtain the cloud classification label of the optical remote sensing cloud image data; Based on the imaging timestamp, orbit information and geographic positioning information of the optical remote sensing cloud image data, different data sources are aligned to a unified geographic coordinate system by using a registration method; The optical remote sensing cloud image data, the cloud profile radar reflectivity data and the atmospheric physical property data are packaged into input data with consistent dimension structure, and the input data and the corresponding cloud classification label are divided into a training set, a validation set and a test set according to a proportion.

8. The remote sensing image cloud classification system of claim 1, wherein: The wavelet consistency alignment module based on the wavelet energy spectrum in the wavelet transform module performs multi-scale wavelet transform on the input data, specifically: The input data is respectively subjected to two-dimensional discrete wavelet transform, and the input data is represented as a set of multi-scale wavelet coefficients including a low-frequency approximation subband and a plurality of directional high-frequency detail subbands: ， wherein decomposition level, denote the first layer multiscale wavelet coefficient set, subscript denote three different modal data types, respectively, opt, radar and atm, wherein subscript opt denotes optical remote sensing cloud image data type, radar denotes cloud profile radar reflectivity data type, and atm denotes atmospheric physical property data type; denote the first layer low-frequency approximation subband, denote the first layer horizontal direction high-frequency detail subband, denote the first layer vertical direction high-frequency detail subband, denote the first layer diagonal direction high-frequency detail subband; An energy spectrum is calculated for each wavelet subband coefficient at each scale and each modality: ， wherein is a first layer energy spectrum, denotes image space coordinates; An inter-modality spectrum distribution model is constructed, and energy spectrum values of different modalities at the same level are normalized into a probability distribution form to depict the difference in wavelet energy response between modalities: ， The first energy spectrum value normalization probability distribution of any modal data type in the input data is calculated The energy spectrum vector combination between the modes is constructed, and the spectrum distribution similarity between the modes is calculated The KL divergence is used as an alignment index, the probability distribution is minimized by introducing a spectrum consistency loss function, and the modes tend to be unified in the multi-scale space frequency response. ， where KL is the KL divergence calculation method, subscript , is the spectral distribution similarity between each pair of modalities; According to the spectral distribution similarity, the wavelet sub-band coefficient of each modality is adjusted to match the energy space, each row is normalized, the weight influence of each modality on other modalities is 1, and the final output wavelet feature of each modality is represented as a weighted sum of all modality features, wherein the weight is the spectral distribution similarity between one modality and other modalities: the final output wavelet feature of each modality is represented as a weighted sum of all modality features, and the wavelet feature tensor of each modality is obtained 、 、 ; ， wherein, is the normalized similarity weight between the two modalities in the first layer wavelet decomposition; ， ， ， wherein, is a normalized similarity weight between the opt modality and the opt modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the opt modality and the radar modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the opt modality and the atm modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the radar modality and the opt modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the radar modality and the radar modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the radar modality and the atm modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the atm modality and the opt modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the atm modality and the radar modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the 1st layer wavelet decomposition, is a normalized similarity weight between the radar modality and the opt modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the radar modality and the radar modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the radar modality and the atm modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the atm modality and the opt modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the atm modality and the radar modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the 2nd layer wavelet decomposition, is a normalized similarity weight between the radar modality and the opt modality in the 3rd layer wavelet decomposition, is a normalized similarity weight between the radar modality and the radar modality in the 3rd layer wavelet decomposition, is a normalized similarity weight between the radar modality and the atm modality in the 3rd layer wavelet decomposition, is a normalized similarity weight between the atm modality and the opt modality in the 3rd layer wavelet decomposition, is a normalized similarity weight between the atm modality and the radar modality in the 3rd layer wavelet decomposition, is a normalized similarity weight between the atm modality and the atm modality in the 3rd layer wavelet decomposition, 9. An electronic device, comprising: The system comprises a processor and a storage medium; The storage medium is configured to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method of any one of claims 1-5.

10. A computer readable storage medium having stored thereon a computer program, characterized in that The program, when executed by the processor, implements the steps of the method of any one of claims 1-5.

Images