A multi-modal data fusion remote sensing image cloud classification method, system, device and medium

Abstract

The application discloses a kind of multi-modal data fusion remote sensing image cloud classification method, system, equipment and medium, including obtaining multi-modal data and known cloud type label, input data is formed to multi-modal data respectively by pre-processing, input data and cloud type label are divided into training set, verification set and test set according to proportion;Wavelet consistency alignment module carries out multi-scale wavelet transform to input data, obtains the wavelet feature tensor of each mode;Cloud classification backbone network model is constructed;Training set, verification set and test set are used to train and evaluate cloud classification backbone network model to obtain the optimized cloud classification backbone network model, input optical remote sensing cloud image to be classified and cloud profile radar reflectivity data and atmospheric physical property data into cloud classification backbone network model, optical remote sensing cloud image is predicted, and the cloud classification result label graph corresponding to each pixel point in target area is output.This application improves the cloud recognition precision and stability under complex scene.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image spatiotemporal fusion technology, and particularly relates to a remote sensing image cloud classification method, system, device and medium for multimodal data fusion. Background Technology

[0002] Cloud classification refers to the automated identification and classification of clouds in the atmosphere based on their type, structure, distribution, or physical properties using remote sensing observation data. Accurate cloud classification results play a crucial supporting role in typhoon track prediction, severe convective weather warning, radiation energy budget estimation, and climate change modeling.

[0003] Currently, remote sensing cloud classification methods can be broadly categorized into three main types: traditional image processing methods, machine learning methods, and deep learning methods. Traditional image processing methods typically rely on low-level image features such as grayscale, edges, and texture. For example, threshold-based cloud detection strategies often use fixed brightness temperature or albedo values ​​to segment clouds / non-clouds. However, traditional image processing methods struggle to accurately distinguish complex cloud morphologies (such as thin clouds and mixed clouds) and are highly dependent on expert experience. While the co-occurrence matrix-based texture classification method proposed by Haralick et al. can characterize differences in cloud morphology and texture, it is sensitive to scale changes and has poor generalization ability. Regarding machine learning methods, researchers have attempted to convert remote sensing images into structured feature vectors and use classifiers to distinguish cloud types. For example, Support Vector Machines (SVMs) can distinguish cloud types in low- to medium-dimensional feature spaces, but they rely on manual feature design. For instance, Zhang et al. constructed an SVM classifier based on brightness temperature features and structural parameters of MODIS cloud images to distinguish between cirrus and cumulus clouds, but the SVM classifier is difficult to extend to high-dimensional data. In addition, methods such as Random Forest (RF) and KNN have also been used in cloud identification, but they suffer from problems such as sensitivity to feature redundancy and insufficient utilization of temporal information. Summary of the Invention

[0004] Purpose of the invention: The first purpose of this invention is to provide a remote sensing image cloud classification method based on multimodal data fusion, so as to improve the accuracy and stability of cloud recognition in complex scenarios.

[0005] The second objective of this invention is to provide a remote sensing image cloud classification system based on multimodal data fusion.

[0006] A third objective of this invention is to provide an electronic device.

[0007] A fourth objective of this invention is to provide a computer storage medium.

[0008] Technical Solution: To achieve the above objectives, the present invention provides a multimodal data fusion-based remote sensing image cloud classification method, comprising the following steps: (1) Obtain multimodal data and known cloud type labels from multiple remote sensing observation platforms. The multimodal data includes optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data. The multimodal data are preprocessed to form input data. The input data and cloud type labels are divided into training set, validation set and test set according to the ratio. (2) The wavelet consistency alignment module performs multi-scale wavelet transform on the input data, extracts high-frequency and low-frequency subband coefficients, constructs a spectral distribution model between modes based on the wavelet domain energy spectrum, and performs feature alignment on different modal data through energy matching and spectral similarity constraint strategies to obtain a fusion representation with enhanced semantic consistency, that is, to obtain the wavelet feature tensor of each mode; (3) Construct a cloud classification backbone network model. The cloud classification backbone network model includes an encoder, a lightweight multimodal wavelet fusion module, and a decoder. The encoder adopts a hierarchical structure consisting of multiple levels. The encoder includes a PatchEmbedding module, a Swin-Transformer module, and a downsampling module. The wavelet feature tensor of each modality is input into the encoder for feature extraction. After block embedding transformation by the Patch Embedding module, feature extraction is performed using the progressively layered Swin-Transformer module and the downsampling module. Finally, the smallest-scale intermediate feature is obtained through the last-level Swin-Transformer module, thus obtaining a multi-scale intermediate feature map. The multi-scale intermediate feature map is fused at multiple scales by the lightweight multimodal wavelet fusion module to obtain a multi-scale fused feature map. The decoder adopts a hierarchical structure symmetrical to the encoder. The decoder includes a Patch... The Embedding module, the Swin-Transformer module, and the upsampling module are used to input the multi-scale fused feature map into the decoder. The last-level fused feature map is reconstructed by the Swin-Transformer module and the upsampling module at each level. It is then summed with the previous level fused feature map and input into the Swin-Transformer module and the upsampling module for feature reconstruction. Finally, the Patch Expanding module restores the cloud classification result label map with the same resolution as the input. (4) The cloud classification backbone network model is trained and evaluated using training set, validation set and test set to obtain the optimized cloud classification backbone network model. The optical remote sensing cloud image to be classified, as well as cloud profile radar reflectivity data and atmospheric physical property data are input into the optimized cloud classification backbone network model to predict the optical remote sensing cloud image to be classified and output the cloud classification result label map corresponding to each pixel in the target area.

[0009] Optionally, the preprocessing in step (1) specifically includes the following steps: (1.1) Acquire optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, and unify the sampling time range and spatial coverage area. Then, register the cloud classification label of the optical remote sensing cloud image data with known cloud type labels through spatiotemporal registration. (1.2) Based on the imaging timestamps, orbital information and geographic location information of optical remote sensing cloud image data, the registration method is used to align different data sources to a unified geographic coordinate system; (1.3) Encapsulate optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data into input data with a consistent dimensional structure, and divide the input data and corresponding cloud classification labels into training set, validation set and test set according to the proportion.

[0010] Optionally, the wavelet consistency alignment module in step (2) performs multi-scale wavelet transform on the input data, specifically including the following steps: (2.1) Perform two-dimensional discrete wavelet transform on the input data respectively, and represent the input data as a set of multi-scale wavelet coefficients containing low-frequency approximate subbands and high-frequency detail subbands in multiple directions: , Among them Decompose hierarchy, Represents the first modal data type in the input data. Multi-scale wavelet coefficient set, subscript These refer to three different modal data types: opt, radar, and atm. The subscript opt ​​refers to the optical remote sensing cloud image data type, radar refers to the cloud profile radar reflectivity data type, and atm refers to the atmospheric physical property data type. Indicates the first The low-frequency approximate subband of the layer, Indicates the first Horizontal layer Directional high-frequency detail sub-band, Indicates the first Vertical layers Directional high-frequency detail sub-band, Indicates the first diagonal of the layer Directional high-frequency detail sub-band; (2.2) Calculate the energy spectrum for the wavelet subband coefficients at each mode and at each scale: , in For any modal data type in the input data, the first... Layer energy spectrum, Represents the spatial coordinates of the image; (2.3) Construct an intermodal spectral distribution model to normalize the energy spectral values ​​of different modes at the same level into a probability distribution form, which is used to characterize the differences in wavelet energy responses between modes: , For any modal data type in the input data, the first... Normalized probability distribution of energy spectral values ​​at each level, and construction of energy spectral vector combinations between modes. The spectral distribution similarity between modes is calculated, and KL divergence is used as the alignment index. By introducing a spectral consistency loss function to minimize the probability distribution, the modes tend to be uniform in the multi-scale spatial frequency response. , Where KL represents the KL divergence calculation method, and the subscript... , The spectral distribution similarity between pairs of modes; (2.4) Based on the spectral distribution similarity, the wavelet subband coefficients of each mode are adjusted to match the energy space. Each row is normalized so that the "weighted influence" of the similarity of each mode on other modes sums to 1. The wavelet feature representation of the final output of each mode is the weighted sum of the features of all modes, where the weight is the spectral distribution similarity between a mode and other modes. The wavelet feature representation of the final output of each mode is the weighted sum of the features of all modes, thus obtaining the wavelet feature tensor of each mode. , , ; , in, It is in the Normalized similarity weights between pairwise modes in layered wavelet decomposition; , , , in, It is in the Normalized similarity weights between OPT modes in layer wavelet decomposition It is in the Normalized similarity weights between opt and radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between opt and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between atm and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between ATM and Radar modes in layer wavelet decomposition. It is in the The normalized similarity weights between ATM modes in layer wavelet decomposition are used as weighting coefficients for wavelet feature co-reconstruction.

[0011] Optionally, step (3) of the Swin-Transformer module specifically includes the following steps: feature The input is processed by layer normalization in the LN module, and then input into the first layer standard window multi-head attention module. Within a fixed window partition, window-level multi-head self-attention computation is performed, denoted as: , For the input features, Indicates the size of the execution window; Adding residual connections yields features : , After the residual is joined After passing through the LN module again, the input is fed into the feedforward fully connected network MLP, where residual connections are added to obtain features. : , feature The input is processed by layer normalization in the LN module, and then input into the second layer shifted window multi-head attention module SW-MSA, which executes the shifted window attention mechanism. That is, it captures cross-window connection information through shifted windowing and adds residual connections to obtain features. : , in, This indicates that local attention calculation is performed after the adjacent layer windows are offset, which helps in modeling cross-window feature associations. Finally, the features After passing through the LN module again, the input is fed into the feedforward fully connected network MLP, where residual connections are added to obtain features. : , in, This is the output of a Swin-Transformer module.

[0012] Optionally, the lightweight multimodal wavelet fusion module in step S3 specifically includes the following steps: First, the multi-scale intermediate features output from the wavelet tensor feature maps of three different modalities processed by the Swin-Transformer module are linearly mapped by the encoder to construct a cross-modal joint feature representation: , in, The intermediate features represent the output of the wavelet tensor feature map of optical remote sensing cloud image data after processing by the Swin-Transformer module; The intermediate features of the wavelet tensor feature map representing cloud profile radar reflectivity data are processed by the Swin-Transformer module. The intermediate features of the wavelet tensor feature map of atmospheric physical property data are represented by the output of the Swin-Transformer module. A channel selection mechanism is introduced to perform importance weighting across different modal dimensions. Let the multimodal wavelet channels before fusion be denoted as 𝐶, and the fusion weights are obtained through the SE structure. : , , GAP represents global average pooling; and Here are the learnable parameters of the fully connected layer; ⊙ represents the channel-level dot product; and ReLU is the activation function. Refers to the Sigmoid activation function, which is used to map the calculation results after passing through the fully connected layer and the ReLU activation function to the (0,1) interval, thereby generating the channel selection weight γ; The fusion result of the channel selection mechanism; The fused features are then fed into a lightweight linear mapping layer: , in Represents a lightweight linear layer with one-dimensional compression; The output features are fused with cross-modal joint features: .

[0013] Based on the same inventive concept, the present invention provides a multimodal data fusion remote sensing image cloud classification system, comprising: The data preparation module is used to acquire multimodal data and known cloud type labels from multiple remote sensing observation platforms. The multimodal data includes optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data. The multimodal data are preprocessed to form input data. The input data and cloud type labels are divided into training set, validation set, and test set according to the ratio. The wavelet transform module is used by the wavelet consistency alignment module to perform multi-scale wavelet transform on the input data, extract high-frequency and low-frequency subband coefficients, construct a spectral distribution model between modes based on the wavelet domain energy spectrum, and perform feature alignment on different modal data through energy matching and spectral similarity constraint strategies to obtain a fusion representation with enhanced semantic consistency, that is, to obtain the wavelet feature tensor of each mode. The network construction module is used to build the cloud classification backbone network model. The cloud classification backbone network model includes an encoder, a lightweight multimodal wavelet fusion module, and a decoder. The encoder adopts a hierarchical structure consisting of multiple layers, including a Patch Embedding module, a Swin-Transformer module, and a downsampling module. Wavelet feature tensors of each modality are input to the encoder for feature extraction. These are then processed by the Patch Embedding module through block embedding transformation, followed by feature extraction using successive Swin-Transformer and downsampling modules. Finally, the last-level Swin-Transformer module obtains the minimum-scale intermediate features, resulting in a multi-scale intermediate feature map. This multi-scale intermediate feature map is then fused across multiple scales by the lightweight multimodal wavelet fusion module to obtain a multi-scale fused feature map. The decoder adopts a hierarchical structure symmetrical to the encoder. The decoder includes a Patch... The Embedding module, the Swin-Transformer module, and the upsampling module are used to input the multi-scale fused feature map into the decoder. The last-level fused feature map is reconstructed by the Swin-Transformer module and the upsampling module at each level. It is then summed with the previous level fused feature map and input into the Swin-Transformer module and the upsampling module for feature reconstruction. Finally, the Patch Expanding module restores the cloud classification result label map with the same resolution as the input. The training and prediction module is used to train and evaluate the cloud classification backbone network model using training, validation, and test sets to obtain an optimized cloud classification backbone network model. The optical remote sensing cloud image to be classified, as well as cloud profile radar reflectivity data and atmospheric physical property data, are input into the optimized cloud classification backbone network model to predict the cloud image to be classified and output the cloud classification result label map corresponding to each pixel in the target area.

[0014] Optionally, the preprocessing in the data preparation module specifically includes: Acquire optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, and unify the sampling time range and spatial coverage area. Then, register the cloud classification labels of the optical remote sensing cloud image data with known cloud type labels through spatiotemporal registration. Based on the imaging timestamps, orbital information, and geographic location information of optical remote sensing cloud image data, a registration method is used to align different data sources to a unified geographic coordinate system. Optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data are encapsulated into input data with a consistent dimensional structure. The input data and corresponding cloud classification labels are divided into training set, validation set, and test set according to the proportion.

[0015] Optionally, the wavelet consistency alignment module based on the wavelet domain energy spectrum in the wavelet transform module performs multi-scale wavelet transform on the input data as follows: Perform two-dimensional discrete wavelet transforms on the input data to represent it as a set of multi-scale wavelet coefficients containing low-frequency approximate subbands and high-frequency detail subbands in multiple directions: , Among them Decompose hierarchy, Represents the first modal data type in the input data. Multi-scale wavelet coefficient set, subscript These refer to three different modal data types: opt, radar, and atm. The subscript opt ​​refers to the optical remote sensing cloud image data type, radar refers to the cloud profile radar reflectivity data type, and atm refers to the atmospheric physical property data type. Indicates the first The low-frequency approximate subband of the layer, Indicates the first Horizontal layer Directional high-frequency detail sub-band, Indicates the first Vertical layers Directional high-frequency detail sub-band, Indicates the first diagonal of the layer Directional high-frequency detail sub-band; Calculate the energy spectrum for the wavelet subband coefficients at each mode and at each scale: , in For any modal data type in the input data, the first... Layer energy spectrum, Represents the spatial coordinates of the image; A modal spectral distribution model is constructed to normalize the energy spectral values ​​of different modes at the same level into a probability distribution form, which is used to characterize the differences in wavelet energy responses between modes: , For any modal data type in the input data, the first... Normalized probability distribution of energy spectral values ​​at each level, and construction of energy spectral vector combinations between modes. The spectral distribution similarity between modes is calculated, and KL divergence is used as the alignment index. By introducing a spectral consistency loss function to minimize the probability distribution, the modes tend to be uniform in the multi-scale spatial frequency response. , Where KL represents the KL divergence calculation method, and the subscript... , The spectral distribution similarity between pairs of modes; Based on spectral distribution similarity, the wavelet subband coefficients of each mode are adjusted to match the energy space. Each row is normalized so that the "weighted influence" of the similarity of each mode on other modes sums to 1. The wavelet feature representation of the final output of each mode is a weighted sum of the features of all modes, where the weight is the spectral distribution similarity between a mode and other modes. The wavelet feature representation of the final output of each mode is a weighted sum of the features of all modes, resulting in the wavelet feature tensor of each mode. , , ; , in, It is in the Normalized similarity weights between pairwise modes in layered wavelet decomposition; , , , in, It is in the Normalized similarity weights between OPT modes in layer wavelet decomposition It is in the Normalized similarity weights between opt and radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between opt and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between atm and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between ATM and Radar modes in layer wavelet decomposition. It is in the The normalized similarity weights between ATM modes in layer wavelet decomposition are used as weighting coefficients for wavelet feature co-reconstruction.

[0016] Based on the same inventive concept, the present invention provides an electronic device including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method described above.

[0017] Based on the same inventive concept, the computer-readable storage medium of the present invention stores a computer program thereon, which, when executed by a processor, implements the steps of the method described above.

[0018] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) This invention addresses the problems of weak multimodal data fusion capability, limited spatial structure modeling, serious modal inconsistency, and high model deployment complexity of current remote sensing cloud classification methods. It integrates wavelet decomposition features and deep Transformer structure of multi-source remote sensing data to achieve collaborative modeling of different modalities under unified spectral and semantic spaces, thereby improving the accuracy and stability of cloud identification in complex scenarios. (2) In this invention, the wavelet consistency alignment module extracts the high and low frequency information of each mode through multi-scale wavelet transform, and constructs a spectral distribution model based on the energy spectrum. It uses KL divergence to minimize the spectral distribution differences between different modes, thereby achieving synergistic consistency between modes in the frequency domain. This can effectively eliminate the problem of inconsistent response between multi-modal data due to different imaging mechanisms and improve the fusion quality. While maintaining the diversity of modal features, it achieves semantic consistency enhancement. The use of energy weighting for feature synergistic reconstruction is beneficial to highlighting the expressive power of key modes. It improves the stability and discriminativeness of the final fusion representation, which helps to improve the subsequent classification accuracy. (3) The cloud classification backbone network model in this invention adopts an encoder-decoder structure and introduces a multi-scale feature fusion module. This network model has the advantages of clear structure and module decoupling, which facilitates the unified processing and cross-scale fusion of features of different modalities. The encoder adopts a multi-layer Swin-Transformer structure, which can fully mine local and global spatial semantic information. The decoder uses skip connections and upsampling strategies to ensure high-resolution information recovery while enhancing the ability to preserve details. Deep fusion is achieved at the multi-scale semantic level to generate highly discriminative features and improve the accuracy of cloud type recognition. The overall architecture has strong adaptability and is easy to migrate and deploy. (4) In this invention, the Swin-Transformer module effectively reduces the computational complexity of global self-attention by combining window attention and shifted window attention mechanisms, thereby achieving more efficient feature modeling; standard window attention (W-MSA) focuses on local context extraction, while shifted window attention (SW-MSA) enhances the ability to interact with information across windows; the hierarchical embedding and multi-scale modeling characteristics conform to the spatial hierarchical feature structure in remote sensing images; it helps to model the texture, boundary, and morphology of clouds in detail, thereby improving recognition accuracy and stability; it supports flexible window size design to adapt to remote sensing image inputs of different resolutions; (5) The lightweight multimodal wavelet fusion module of this invention introduces a channel selection mechanism and a lightweight linear mapping to perform multimodal feature fusion, which can achieve significant weighting between different modalities and highlight channels with higher information content or more sensitive to classification; it uses a Squeeze-and-Excitation structure to realize dynamic channel weight learning and improve fusion accuracy; it keeps the number of network parameters low, meets the requirements of lightweight deployment, and is suitable for embedded or resource-constrained scenarios; the multi-scale fusion operation enhances the model's understanding of cloud structure at different spatial levels; and it significantly improves model efficiency and inference speed without sacrificing performance. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2This is a flowchart illustrating the wavelet consistency alignment module in this invention. Figure 3 This is a flowchart illustrating the cloud classification backbone network model in this invention. Figure 4 This is a schematic diagram of the framework of the Swin-Transformer module in this invention; Figure 5 This is a flowchart illustrating the lightweight multimodal wavelet fusion module of this invention. Detailed Implementation

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0021] Example 1: As Figure 1 As shown in this embodiment, a remote sensing image cloud classification method based on multimodal data fusion includes the following steps: (1) Obtain multimodal data and known cloud type labels from multiple remote sensing observation platforms. The multimodal data includes optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data. The multimodal data are preprocessed to form input data. The input data and cloud type labels are divided into training set, validation set and test set according to the ratio. The preprocessing in step (1) specifically includes the following steps: (1.1) Acquire optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, and unify the sampling time range and spatial coverage area. Then, register the cloud classification labels of the optical remote sensing cloud image data with the cloud type labels provided by the CloudSat satellite through spatiotemporal registration. The optical remote sensing cloud image data can be data from the MODIS satellite or the FY-4 satellite. The cloud profile radar reflectivity data can be data from the radar CPR carried by the CloudSat satellite. The atmospheric physical property data includes numerical fields such as temperature, humidity, air pressure, and wind speed. (1.2) Based on the imaging timestamp, orbit information and geographic location information of optical remote sensing cloud image data, the registration method is used to align different data sources to a unified geographic coordinate system to ensure the consistency of multimodal data in time and space dimensions; the registration method can be affine transformation, affine projection or RPC model, etc. (1.3) Encapsulate the optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data into input data with a consistent dimensional structure. Upsample the low-resolution data in the input data using the interpolation resampling method. Convert the data format of each modality into a unified tensor structure input format to adapt to the input requirements of the wavelet consistency alignment module in step (2). Divide the input data and the corresponding cloud classification labels into training set, validation set and test set according to 7:2:1. (2) The wavelet consistency alignment module based on the wavelet domain energy spectrum performs multi-scale wavelet transform on the input data, extracts high-frequency and low-frequency subband coefficients, constructs a spectral distribution model between modes based on the wavelet domain energy spectrum, and performs feature alignment on different mode data through energy matching and spectral similarity constraint strategies to obtain a fusion representation with enhanced semantic consistency, that is, to obtain the wavelet feature tensor of each mode. like Figure 2 As shown, step (2) of the wavelet consistency alignment module based on the wavelet domain energy spectrum performs multi-scale wavelet transform on the input data, specifically including the following steps: (2.1) Perform two-dimensional discrete wavelet transform (DWT) on the preprocessed input data to represent the input data as a set of multi-scale wavelet coefficients containing low-frequency approximation (A) subbands and high-frequency detail subbands in multiple directions: , in To decompose the hierarchy, Represents the first modal data type in the input data. Multi-scale wavelet coefficient set, subscript These refer to three different modal data types: opt, radar, and atm. The subscript opt ​​refers to the optical remote sensing cloud image data type, radar refers to the cloud profile radar reflectivity data type, and atm refers to the atmospheric physical property data type. Indicates the first The low-frequency approximate subband of the layer, Indicates the first Horizontal layer Directional high-frequency detail sub-band, Indicates the first Vertical layers Directional high-frequency detail sub-band, Indicates the first diagonal of the layer Directional high-frequency detail sub-band; (2.2) Calculate the energy spectrum for the wavelet subband coefficients at each mode and at each scale: , in For any modal data type in the input data, the first... Layer energy spectrum, Represents the spatial coordinates of the image; (2.3) Construct an intermodal spectral distribution model to normalize the energy spectral values ​​of different modes at the same level into a probability distribution form, which is used to characterize the differences in wavelet energy responses between modes: , For any modal data type in the input data, the first... Normalized probability distribution of energy spectral values ​​at each level, and construction of energy spectral vector combinations between modes. The spectral distribution similarity between modes is calculated, and the Kullback-Leibler divergence is used as the alignment index. By introducing a spectral consistency loss function to minimize the probability distribution, the modes tend to be uniform in the multi-scale spatial frequency response. , Where KL represents the KL divergence calculation method, and the subscript... , The spectral distribution similarity between pairs of modes; Specifically: , , , , , , , , , in The spectral distribution similarity between the opt modes is... The spectral distribution similarity between the opt mode and the radar mode. The spectral distribution similarity between the opt mode and the atm mode. The spectral distribution similarity between the radar mode and the opt mode. The spectral distribution similarity between radar modes. The spectral distribution similarity between the radar mode and the atm mode. The spectral distribution similarity between the atm mode and the opt mode. The spectral distribution similarity between the atm mode and the radar mode. The spectral distribution similarity between atm modes; (2.4) Based on the spectral distribution similarity, the wavelet subband coefficients of each mode are adjusted to match the energy space. Each row is normalized so that the "weighted influence" of the similarity of each mode on other modes sums to 1. The wavelet feature representation of the final output of each mode is the weighted sum of the features of all modes, where the weight is the spectral distribution similarity between a mode and other modes. The wavelet feature representation of the final output of each mode is the weighted sum of the features of all modes, thus obtaining the wavelet feature tensor of each mode. , and The data is then fed into the WaMCCformer cloud classification backbone network model for feature extraction. , in, It is in the Normalized similarity weights between pairwise modes in layered wavelet decomposition; Specifically: , , , , , , , , , ,

[0022] in, It is in the Normalized similarity weights between OPT modes in layer wavelet decomposition It is in the Normalized similarity weights between opt and radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between opt and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between atm and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between ATM and Radar modes in layer wavelet decomposition. It is in the The normalized similarity weights between ATM modes in layer wavelet decomposition are used as weighting coefficients for wavelet feature co-reconstruction.

[0023] (3) Construct the cloud classification backbone network model WaMCCformer, jointly model and fuse wavelet feature tensors of different modalities, and generate multimodal fusion features with strong discriminative power; like Figure 3 As shown, the cloud classification backbone network model WaMCCformer includes an encoder, a lightweight multimodal wavelet fusion module, and a decoder. The encoder employs a hierarchical structure with multiple layers, comprising a Patch Embedding module, a Swin-Transformer module, and a downsampling module. The Patch Embedding module divides the wavelet feature tensors of each modality into blocks and embeds them into token representations. The Swin-Transformer module extracts spatial-semantic context features within a local window. The downsampling module progressively reduces the feature map resolution and increases the receptive field. The encoder consists of four downsampling stages, with the first three stages each containing two Swin-Transformer modules, and the last stage containing one Swin-Transformer module. The wavelet feature tensors of each modality are input to the encoder for feature extraction, and then processed by the Patch module. The Embedding module performs block embedding transformation, then uses the stepwise Swin-Transformer module and downsampling module for feature extraction, and then the final Swin-Transformer module to obtain the minimum scale intermediate features, thus obtaining a multi-scale intermediate feature map. The multi-scale intermediate feature map is fused at multiple scales by a lightweight multimodal wavelet fusion module to obtain a multi-scale fused feature map. The decoder adopts a hierarchical structure symmetrical to the encoder. The decoder includes a Patch Embedding module, a Swin-Transformer module, and an upsampling module. The fused multi-scale fused feature map is input into the decoder. The last-level fused feature map is reconstructed by the Swin-Transformer module and the upsampling module at each level. It is then summed with the previous level fused feature map and input back to the Swin-Transformer module and the upsampling module for feature reconstruction. Finally, the Patch Expanding module restores the cloud category result label map with the same resolution as the input. The decoder employs an upsampling strategy to progressively restore the feature map resolution. Each upsampling stage is equipped with two or one Swin-Transformer modules to further aggregate high-resolution features. Simultaneously, in the corresponding encoder stage, intermediate features are shared through skip connections. Finally, the Patch Expanding module restores the cloud category result label map to the same resolution as the input. A lightweight multimodal wavelet fusion module constitutes the feature fusion extraction stage. The fusion position of each encoder level and the lightweight multimodal wavelet fusion module is synchronized to ensure that multimodal features are guided and aggregated at different semantic levels. The output multimodal features possess high discriminativeness and spatial consistency, and are used for cloud category result output.

[0024] like Figure 4 As shown, step (3) of the Swin-Transformer module specifically includes the following steps: Input features The input is processed by the LN (Layer Normalization) module for layer normalization, and then input into the first-layer standard window multi-head attention module. Within a fixed window partition, window-level multi-head self-attention computation is performed, denoted as: , For the input features, Indicates the size of the execution window; Adding residual connections yields features : , After the residual is joined After passing through the LN module again, the input is fed into the feedforward fully connected network MLP, where residual connections are added to obtain features. : , feature The input is processed by layer normalization in the LN module, and then input into the second layer shifted window multi-head attention module SW-MSA, which executes the shifted window attention mechanism. That is, it captures cross-window connection information through shifted windowing and adds residual connections to obtain features. : , in, This indicates that local attention calculation is performed after the adjacent layer windows are offset, which helps in modeling cross-window feature associations. Finally, the features After passing through the LN module again, the input is fed into the feedforward fully connected network MLP, where residual connections are added to obtain features. : , in, This is the output of a Swin-Transformer module.

[0025] The Swin-Transformer module combines W-MSA and SW-MSA attention mechanisms in a hierarchical manner, and is supplemented by a two-layer MLP and residual structure. It can effectively capture local-global cloud spatial semantic information and improve the ability to extract cloud category features. It is a core component of the cloud classification backbone network model WaMCCformer.

[0026] like Figure 5 As shown, the lightweight multimodal wavelet fusion module in step S3 specifically includes the following steps: First, the multi-scale intermediate features output from the wavelet tensor feature maps of three different modalities processed by the Swin-Transformer module are linearly mapped by the encoder to construct a cross-modal joint feature representation: , in, The intermediate features represent the output of the wavelet tensor feature map of optical remote sensing cloud image data after processing by the Swin-Transformer module; The intermediate features of the wavelet tensor feature map representing cloud profile radar reflectivity data are processed by the Swin-Transformer module. The wavelet tensor feature map representing atmospheric physical property data is the intermediate feature output by the Swin-Transformer module. This step preserves the original information of each modality feature by forming a multimodal fusion base feature, which prepares for subsequent attention selection.

[0027] A channel selection mechanism is introduced to perform importance weighting across different modal dimensions. Let the multimodal wavelet channels before fusion be denoted as 𝐶, and the fusion weights are obtained through an SE (Squeeze-and-Excitation) structure. : , , GAP represents global average pooling; and Here are the learnable parameters of the fully connected layer; ⊙ represents the channel-level dot product; and ReLU is the activation function. Refers to the Sigmoid activation function, which is used to map the calculation results after passing through the fully connected layer and the ReLU activation function to the (0,1) interval, thereby generating the channel selection weight γ; The fusion result of the channel selection mechanism; To ensure the network remains lightweight, the fused features are fed into a lightweight linear mapping layer: , in This represents a lightweight linear layer with dimensional compression, used to map multimodal inputs back to the backbone network dimension; By fusing output features with cross-modal joint features, information is injected into the backbone network to enhance cross-modal representation learning capabilities and maintain information consistency. .

[0028] (4) The cloud classification backbone network model is trained using training and validation sets. The trained cloud classification backbone network model is used to perform inference evaluation on the test set to obtain the classification performance index of the cloud classification backbone network model on each cloud type. The optical remote sensing cloud image to be classified, as well as the cloud profile radar reflectivity data and atmospheric physical property data, are input into the optimized cloud classification backbone network model to predict the optical remote sensing cloud image to be classified and output the cloud type label map corresponding to each pixel in the target area to realize the automated cloud classification task of the target area.

[0029] Example 2: Based on the same inventive concept, the multimodal data fusion remote sensing image cloud classification system of the present invention includes: The data preparation module is used to acquire multimodal data and known cloud type labels from multiple remote sensing observation platforms. The multimodal data includes optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data. The multimodal data are preprocessed to form input data. The input data and cloud type labels are divided into training set, validation set, and test set according to the ratio.

[0030] The data preparation module acquires optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, and unifies the sampling time range and spatial coverage area. Then, it registers the cloud type labels provided by the CloudSat satellite through spatiotemporal registration to obtain cloud classification labels for the optical remote sensing cloud image data. The optical remote sensing cloud image data can be from MODIS or FY-4 satellites, the cloud profile radar reflectivity data can be from the CPR radar on the CloudSat satellite, and the atmospheric physical property data includes numerical fields such as temperature, humidity, air pressure, and wind speed.

[0031] Based on the imaging timestamps, orbital information, and geographic location information of optical remote sensing cloud image data, a registration method is used to align different data sources to a unified geographic coordinate system, ensuring the consistency of multimodal data in time and space dimensions. The registration method can be affine transformation, affine projection, or RPC model, etc.

[0032] Optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data are encapsulated into input data with a consistent dimensional structure. Low-resolution data in the input data are upsampled using an interpolation resampling method. The data format of each modality is converted into a unified tensor structure input format to meet the input requirements of the wavelet consistency alignment module in the wavelet transform module. The input data and corresponding cloud classification labels are divided into training set, validation set, and test set according to a 7:2:1 ratio.

[0033] The wavelet transform module performs multi-scale wavelet transform on the input data based on the wavelet domain energy spectrum, extracts high-frequency and low-frequency subband coefficients, constructs a spectral distribution model between modes based on the wavelet domain energy spectrum, and performs feature alignment on different modal data through energy matching and spectral similarity constraint strategies to obtain a fusion representation with enhanced semantic consistency, that is, to obtain the wavelet feature tensor of each mode.

[0034] like Figure 2 As shown, the wavelet consistency alignment module based on wavelet domain energy spectrum in the wavelet transform module performs multi-scale wavelet transform on the input data as follows: The preprocessed input data is subjected to two-dimensional discrete wavelet transform (DWT) to represent the input data as a set of multi-scale wavelet coefficients containing low-frequency approximation (A) subbands and high-frequency detail subbands in multiple directions: , in To decompose the hierarchy, Represents the first modal data type in the input data. Multi-scale wavelet coefficient set, subscript These refer to three different modal data types: opt, radar, and atm. The subscript opt ​​refers to the optical remote sensing cloud image data type, radar refers to the cloud profile radar reflectivity data type, and atm refers to the atmospheric physical property data type. Indicates the first The low-frequency approximate subband of the layer, Indicates the first Horizontal layer Directional high-frequency detail sub-band, Indicates the first Vertical layers Directional high-frequency detail sub-band, Indicates the first diagonal of the layer Directional high-frequency detail sub-band.

[0035] Calculate the energy spectrum for the wavelet subband coefficients at each mode and at each scale: , in For any modal data type in the input data, the first... Layer energy spectrum, Represents the spatial coordinates of the image.

[0036] A modal spectral distribution model is constructed to normalize the energy spectral values ​​of different modes at the same level into a probability distribution form, which is used to characterize the differences in wavelet energy responses between modes: , For any modal data type in the input data, the first... Normalized probability distribution of energy spectral values ​​at each level, and construction of energy spectral vector combinations between modes. The spectral distribution similarity between modes is calculated, and the Kullback-Leibler divergence is used as the alignment index. By introducing a spectral consistency loss function to minimize the probability distribution, the modes tend to be uniform in the multi-scale spatial frequency response. , Where KL represents the KL divergence calculation method, and the subscript... , The spectral distribution similarity between pairs of modes; Specifically: , , , , , , , , , in The spectral distribution similarity between the opt modes is... The spectral distribution similarity between the opt mode and the radar mode. The spectral distribution similarity between the opt mode and the atm mode. The spectral distribution similarity between the radar mode and the opt mode. The spectral distribution similarity between radar modes. The spectral distribution similarity between the radar mode and the atm mode. The spectral distribution similarity between the atm mode and the opt mode. The spectral distribution similarity between the atm mode and the radar mode. This represents the spectral distribution similarity between atm modes.

[0037] Based on spectral distribution similarity, the wavelet subband coefficients of each mode are adjusted to match the energy space. Each row is normalized so that the "weighted influence" of the similarity of each mode on other modes sums to 1. The wavelet feature representation of the final output of each mode is a weighted sum of the features of all modes, where the weight is the spectral distribution similarity between a mode and other modes. The wavelet feature representation of the final output of each mode is a weighted sum of the features of all modes, resulting in the wavelet feature tensor of each mode. , and The data is then fed into the WaMCCformer cloud classification backbone network model for feature extraction. , in, It is in the Normalized similarity weights between pairwise modes in layered wavelet decomposition; Specifically: , , , , , , , , , ,

[0038] in, It is in the Normalized similarity weights between OPT modes in layer wavelet decomposition It is in the Normalized similarity weights between opt and radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between opt and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between atm and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between ATM and Radar modes in layer wavelet decomposition. It is in the The normalized similarity weights between ATM modes in layer wavelet decomposition are used as weighting coefficients for wavelet feature co-reconstruction.

[0039] The network construction module is used to build the cloud classification backbone network model WaMCCformer, which jointly models and fuses information of wavelet feature tensors of different modalities to generate multimodal fusion features with strong discriminative power. like Figure 3 As shown, the cloud classification backbone network model WaMCCformer includes an encoder, a lightweight multimodal wavelet fusion module, and a decoder. The encoder employs a hierarchical structure with multiple layers, comprising a Patch Embedding module, a Swin-Transformer module, and a downsampling module. The Patch Embedding module divides the wavelet feature tensors of each modality into blocks and embeds them into token representations. The Swin-Transformer module extracts spatial-semantic context features within a local window. The downsampling module progressively reduces the feature map resolution and increases the receptive field. The encoder consists of four downsampling stages, with the first three stages each containing two Swin-Transformer modules, and the last stage containing one Swin-Transformer module. The wavelet feature tensors of each modality are input to the encoder for feature extraction, and then processed by the Patch module. The Embedding module performs block embedding transformation, then uses the stepwise Swin-Transformer module and downsampling module for feature extraction, and then the final Swin-Transformer module to obtain the minimum scale intermediate features, thus obtaining a multi-scale intermediate feature map. The multi-scale intermediate feature map is fused at multiple scales by a lightweight multimodal wavelet fusion module to obtain a multi-scale fused feature map. The decoder adopts a hierarchical structure symmetrical to the encoder. The decoder includes a Patch Embedding module, a Swin-Transformer module, and an upsampling module. The fused multi-scale fused feature map is input into the decoder. The last-level fused feature map is reconstructed by the Swin-Transformer module and the upsampling module at each level. It is then summed with the previous level fused feature map and input back to the Swin-Transformer module and the upsampling module for feature reconstruction. Finally, the Patch Expanding module restores the cloud category result label map with the same resolution as the input. The decoder employs an upsampling strategy to progressively restore the feature map resolution. Each upsampling stage is equipped with two or one Swin-Transformer modules to further aggregate high-resolution features. Simultaneously, in the corresponding encoder stage, intermediate features are shared through skip connections. Finally, the Patch Expanding module restores the cloud category result label map to the same resolution as the input. A lightweight multimodal wavelet fusion module constitutes the feature fusion extraction stage. The fusion position of each encoder level and the lightweight multimodal wavelet fusion module is synchronized to ensure that multimodal features are guided and aggregated at different semantic levels. The output multimodal features possess high discriminativeness and spatial consistency, and are used for cloud category result output.

[0040] like Figure 4 As shown, the Swin-Transformer module specifically consists of: Input features The input is processed by the LN (Layer Normalization) module for layer normalization, and then input into the first-layer standard window multi-head attention module. Within a fixed window partition, window-level multi-head self-attention computation is performed, denoted as: , For the input features, Indicates the size of the execution window; Adding residual connections yields features : , After the residual is joined After passing through the LN module again, the input is fed into the feedforward fully connected network MLP, where residual connections are added to obtain features. : , feature The input is processed by layer normalization in the LN module, and then input into the second layer shifted window multi-head attention module SW-MSA, which executes the shifted window attention mechanism. That is, it captures cross-window connection information through shifted windowing and adds residual connections to obtain features. : , in, This indicates that local attention calculation is performed after the adjacent layer windows are offset, which helps in modeling cross-window feature associations. Finally, the features After passing through the LN module again, the input is fed into the feedforward fully connected network MLP, where residual connections are added to obtain features. : , in, This is the output of a Swin-Transformer module.

[0041] The Swin-Transformer module combines W-MSA and SW-MSA attention mechanisms in a hierarchical manner, and is supplemented by a two-layer MLP and residual structure. It can effectively capture local-global cloud spatial semantic information and improve the ability to extract cloud category features. It is a core component of the cloud classification backbone network model WaMCCformer.

[0042] like Figure 5 As shown, the lightweight multimodal wavelet fusion module is specifically as follows: First, the multi-scale intermediate features output from the wavelet tensor feature maps of three different modalities processed by the Swin-Transformer module are linearly mapped by the encoder to construct a cross-modal joint feature representation: , in, The intermediate features represent the output of the wavelet tensor feature map of optical remote sensing cloud image data after processing by the Swin-Transformer module; The intermediate features of the wavelet tensor feature map representing cloud profile radar reflectivity data are processed by the Swin-Transformer module. The wavelet tensor feature map representing atmospheric physical property data is the intermediate feature output by the Swin-Transformer module. This step preserves the original information of each modality feature by forming a multimodal fusion base feature, which prepares for subsequent attention selection.

[0043] A channel selection mechanism is introduced to perform importance weighting across different modal dimensions. Let the multimodal wavelet channels before fusion be denoted as 𝐶, and the fusion weights are obtained through an SE (Squeeze-and-Excitation) structure. : , , GAP represents global average pooling; and Here are the learnable parameters of the fully connected layer; ⊙ represents the channel-level dot product; and ReLU is the activation function. Refers to the Sigmoid activation function, which is used to map the calculation results after passing through the fully connected layer and the ReLU activation function to the (0,1) interval, thereby generating the channel selection weight γ; The fusion result of the channel selection mechanism; To ensure the network remains lightweight, the fused features are fed into a lightweight linear mapping layer: , in This represents a lightweight linear layer with dimensional compression, used to map multimodal inputs back to the backbone network dimension; By fusing output features with cross-modal joint features, information is injected into the backbone network to enhance cross-modal representation learning capabilities and maintain information consistency. .

[0044] The training and prediction module is used to train the cloud classification backbone network model using training and validation sets, and to use the trained cloud classification backbone network model to perform inference evaluation on the test set, obtaining the classification performance index of the cloud classification backbone network model on various cloud types. The optical remote sensing cloud image to be classified, as well as cloud profile radar reflectivity data and atmospheric physical property data, are input into the optimized cloud classification backbone network model to predict the cloud image to be classified, and output the cloud type label map corresponding to each pixel in the target area, thereby realizing the automated cloud classification task for the target area.

[0045] Example 3: In this example, an electronic device includes a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method described above.

[0046] Example 4: In this example, a computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described above.

Claims

1. A remote sensing image cloud classification method based on multimodal data fusion, characterized in that, Includes the following steps: (1) Obtain multimodal data and known cloud type labels from multiple remote sensing observation platforms. The multimodal data includes optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data. The multimodal data are preprocessed to form input data. The input data and cloud type labels are divided into training set, validation set and test set according to the ratio. (2) The wavelet consistency alignment module performs multi-scale wavelet transform on the input data, extracts high-frequency and low-frequency subband coefficients, constructs a spectral distribution model between modes based on the wavelet domain energy spectrum, and performs feature alignment on different modal data through energy matching and spectral similarity constraint strategies to obtain a fusion representation with enhanced semantic consistency, that is, to obtain the wavelet feature tensor of each mode; (3) Construct a cloud classification backbone network model. The cloud classification backbone network model includes an encoder, a lightweight multimodal wavelet fusion module and a decoder. The encoder adopts a hierarchical structure composed of multiple levels. The encoder includes a PatchEmbedding module, a Swin-Transformer module and a downsampling module. The wavelet feature tensor of each mode is input into the encoder for feature extraction. After the Patch Embedding module performs block embedding transformation, the Swin-Transformer module and the downsampling module are used for feature extraction. After the last level of the Swin-Transformer module, the minimum scale intermediate features are obtained, thus obtaining a multi-scale intermediate feature map. The multi-scale intermediate feature map is fused at multiple scales by the lightweight multimodal wavelet fusion module to obtain a multi-scale fused feature map. The decoder adopts a hierarchical structure symmetrical to the encoder. The decoder includes a Patch Embedding module, a Swin-Transformer module, and an upsampling module. The fused multi-scale fused feature map is input into the decoder. The last-level fused feature map is reconstructed by the Swin-Transformer module and the upsampling module at each level. It is then summed with the previous level fused feature map and input back to the Swin-Transformer module and the upsampling module for feature reconstruction. Finally, the Patch Expanding module restores the cloud classification result label map with the same resolution as the input. (4) The cloud classification backbone network model is trained and evaluated using training set, validation set and test set to obtain the optimized cloud classification backbone network model. The optical remote sensing cloud image to be classified, as well as cloud profile radar reflectivity data and atmospheric physical property data are input into the optimized cloud classification backbone network model to predict the optical remote sensing cloud image to be classified and output the cloud classification result label map corresponding to each pixel in the target area.

2. The remote sensing image cloud classification method based on multimodal data fusion according to claim 1, characterized in that: The preprocessing in step (1) specifically includes the following steps: (1.1) Acquire optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, and unify the sampling time range and spatial coverage area. Then, register the cloud classification label of the optical remote sensing cloud image data with known cloud type labels through spatiotemporal registration. (1.2) Based on the imaging timestamps, orbital information and geographic location information of optical remote sensing cloud image data, the registration method is used to align different data sources to a unified geographic coordinate system; (1.3) Encapsulate optical remote sensing cloud image data, cloud profile radar reflectivity data and atmospheric physical property data into input data with a consistent dimensional structure, and divide the input data and corresponding cloud classification labels into training set, validation set and test set according to the proportion.

3. The remote sensing image cloud classification method based on multimodal data fusion according to claim 1, characterized in that: The wavelet consistency alignment module in step (2) performs multi-scale wavelet transform on the input data, specifically including the following steps: (2.1) Perform two-dimensional discrete wavelet transform on the input data respectively, and represent the input data as a set of multi-scale wavelet coefficients containing low-frequency approximate subbands and high-frequency detail subbands in multiple directions: ， Among them Decompose hierarchy, Represents the first modal data type in the input data. Multi-scale wavelet coefficient set, subscript These refer to three different modal data types: opt, radar, and atm. The subscript opt ​​refers to the optical remote sensing cloud image data type, radar refers to the cloud profile radar reflectivity data type, and atm refers to the atmospheric physical property data type. Indicates the first The low-frequency approximate subband of the layer, Indicates the first Horizontal layer Directional high-frequency detail sub-band, Indicates the first Vertical layers Directional high-frequency detail sub-band, Indicates the first diagonal of the layer Directional high-frequency detail sub-band; (2.2) Calculate the energy spectrum for the wavelet subband coefficients at each mode and at each scale: ， in For any modal data type in the input data, the first... Layer energy spectrum, Represents the spatial coordinates of the image; (2.3) Construct an intermodal spectral distribution model to normalize the energy spectral values ​​of different modes at the same level into a probability distribution form, which is used to characterize the differences in wavelet energy responses between modes: ， For any modal data type in the input data, the first... Normalized probability distribution of energy spectral values ​​at each level, and construction of energy spectral vector combinations between modes. The spectral distribution similarity between modes is calculated, and KL divergence is used as the alignment index. By introducing a spectral consistency loss function to minimize the probability distribution, the modes tend to be uniform in the multi-scale spatial frequency response. ， Where KL represents the KL divergence calculation method, and the subscript... , The spectral distribution similarity between pairs of modes; (2.4) Based on the spectral distribution similarity, the wavelet subband coefficients of each mode are adjusted to match the energy space. Each row is normalized so that the "weighted influence" of the similarity of each mode on other modes sums to 1. The wavelet feature representation of the final output of each mode is the weighted sum of the features of all modes, where the weight is the spectral distribution similarity between a mode and other modes. The wavelet feature representation of the final output of each mode is the weighted sum of the features of all modes, thus obtaining the wavelet feature tensor of each mode. , , ; ， in, It is in the Normalized similarity weights between pairwise modes in layered wavelet decomposition; ， ， ， in, It is in the Normalized similarity weights between OPT modes in layer wavelet decomposition It is in the Normalized similarity weights between opt and radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between opt and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between atm and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between ATM and Radar modes in layer wavelet decomposition. It is in the The normalized similarity weights between ATM modes in layer wavelet decomposition are used as weighting coefficients for wavelet feature co-reconstruction.

4. The remote sensing image cloud classification method based on multimodal data fusion according to claim 1, characterized in that: The Swin-Transformer module in step (3) specifically includes the following steps: feature The input is processed by layer normalization in the LN module, and then input into the first layer standard window multi-head attention module. Within a fixed window partition, window-level multi-head self-attention computation is performed, denoted as: ， For the input features, Indicates the size of the execution window; Adding residual connections yields features : ， After the residual is joined After passing through the LN module again, the input is fed into the feedforward fully connected network MLP, where residual connections are added to obtain features. : ， feature The input is processed by layer normalization in the LN module, and then input into the second layer shifted window multi-head attention module SW-MSA, which executes the shifted window attention mechanism. That is, it captures cross-window connection information through shifted windowing and adds residual connections to obtain features. : ， in, This indicates that local attention calculation is performed after the adjacent layer windows are offset, which helps in modeling cross-window feature associations. Finally, the features After passing through the LN module again, the input is fed into the feedforward fully connected network MLP, where residual connections are added to obtain features. : ， in, This is the output of a Swin-Transformer module.

5. The remote sensing image cloud classification method based on multimodal data fusion according to claim 1, characterized in that: The lightweight multimodal wavelet fusion module in step S3 specifically includes the following steps: First, the multi-scale intermediate features output from the wavelet tensor feature maps of three different modalities processed by the Swin-Transformer module are linearly mapped by the encoder to construct a cross-modal joint feature representation: ， in, The intermediate features represent the output of the wavelet tensor feature map of optical remote sensing cloud image data after processing by the Swin-Transformer module; The intermediate features of the wavelet tensor feature map representing cloud profile radar reflectivity data are processed by the Swin-Transformer module. The intermediate features of the wavelet tensor feature map of atmospheric physical property data are represented by the output of the Swin-Transformer module. A channel selection mechanism is introduced to perform importance weighting across different modal dimensions. Let the multimodal wavelet channels before fusion be denoted as 𝐶, and the fusion weights are obtained through the SE structure. : ， ， GAP represents global average pooling; and Here are the learnable parameters of the fully connected layer; ⊙ represents the channel-level dot product; and ReLU is the activation function. Refers to the Sigmoid activation function, which is used to map the calculation results after passing through the fully connected layer and the ReLU activation function to the (0,1) interval, thereby generating the channel selection weight γ; The fusion result of the channel selection mechanism; The fused features are then fed into a lightweight linear mapping layer: , in Represents a lightweight linear layer with one-dimensional compression; The output features are fused with cross-modal joint features: .

6. A remote sensing image cloud classification system based on multimodal data fusion, characterized in that, include: The data preparation module is used to acquire multimodal data and known cloud type labels from multiple remote sensing observation platforms. The multimodal data includes optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data. The multimodal data are preprocessed to form input data. The input data and cloud type labels are divided into training set, validation set, and test set according to the ratio. The wavelet transform module is used by the wavelet consistency alignment module to perform multi-scale wavelet transform on the input data, extract high-frequency and low-frequency subband coefficients, construct a spectral distribution model between modes based on the wavelet domain energy spectrum, and perform feature alignment on different modal data through energy matching and spectral similarity constraint strategies to obtain a fusion representation with enhanced semantic consistency, that is, to obtain the wavelet feature tensor of each mode. The network construction module is used to build the cloud classification backbone network model. The cloud classification backbone network model includes an encoder, a lightweight multimodal wavelet fusion module, and a decoder. The encoder adopts a hierarchical structure consisting of multiple layers. The encoder includes a Patch Embedding module, a Swin-Transformer module, and a downsampling module. The wavelet feature tensor of each modality is input into the encoder for feature extraction. After the Patch Embedding module performs block embedding transformation, the feature is extracted again using the successive Swin-Transformer module and the downsampling module. Finally, the smallest-scale intermediate feature is obtained through the last-level Swin-Transformer module, thus obtaining a multi-scale intermediate feature map. The multi-scale intermediate feature map is fused at multiple scales by the lightweight multimodal wavelet fusion module to obtain a multi-scale fused feature map. The decoder adopts a hierarchical structure symmetrical to the encoder. The decoder includes a Patch Embedding module, a Swin-Transformer module, and an upsampling module. The fused multi-scale fused feature map is input into the decoder. The last-level fused feature map is reconstructed by the Swin-Transformer module and the upsampling module at each level. It is then summed with the previous level fused feature map and input back to the Swin-Transformer module and the upsampling module for feature reconstruction. Finally, the Patch Expanding module restores the cloud classification result label map with the same resolution as the input. The training and prediction module is used to train and evaluate the cloud classification backbone network model using training, validation, and test sets to obtain an optimized cloud classification backbone network model. The optical remote sensing cloud image to be classified, as well as cloud profile radar reflectivity data and atmospheric physical property data, are input into the optimized cloud classification backbone network model to predict the cloud image to be classified and output the cloud classification result label map corresponding to each pixel in the target area.

7. The remote sensing image cloud classification system based on multimodal data fusion according to claim 1, characterized in that: The preprocessing in the data preparation module specifically includes: Acquire optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data, and unify the sampling time range and spatial coverage area. Then, register the cloud classification labels of the optical remote sensing cloud image data with known cloud type labels through spatiotemporal registration. Based on the imaging timestamps, orbital information, and geographic location information of optical remote sensing cloud image data, a registration method is used to align different data sources to a unified geographic coordinate system. Optical remote sensing cloud image data, cloud profile radar reflectivity data, and atmospheric physical property data are encapsulated into input data with a consistent dimensional structure. The input data and corresponding cloud classification labels are divided into training set, validation set, and test set according to the proportion.

8. The remote sensing image cloud classification system based on multimodal data fusion according to claim 1, characterized in that: The wavelet consistency alignment module based on wavelet domain energy spectrum in the wavelet transform module performs multi-scale wavelet transform on the input data as follows: Perform two-dimensional discrete wavelet transforms on the input data to represent it as a set of multi-scale wavelet coefficients containing low-frequency approximate subbands and high-frequency detail subbands in multiple directions: ， Among them Decompose hierarchy, Represents the first modal data type in the input data. Multi-scale wavelet coefficient set, subscript These refer to three different modal data types: opt, radar, and atm. The subscript opt ​​refers to the optical remote sensing cloud image data type, radar refers to the cloud profile radar reflectivity data type, and atm refers to the atmospheric physical property data type. Indicates the first The low-frequency approximate subband of the layer, Indicates the first Horizontal layer Directional high-frequency detail sub-band, Indicates the first Vertical layers Directional high-frequency detail sub-band, Indicates the first diagonal of the layer Directional high-frequency detail sub-band; Calculate the energy spectrum for the wavelet subband coefficients at each mode and at each scale: ， in For any modal data type in the input data, the first... Layer energy spectrum, Represents the spatial coordinates of the image; A modal spectral distribution model is constructed to normalize the energy spectral values ​​of different modes at the same level into a probability distribution form, which is used to characterize the differences in wavelet energy responses between modes: ， For any modal data type in the input data, the first... Normalized probability distribution of energy spectral values ​​at each level, and construction of energy spectral vector combinations between modes. The spectral distribution similarity between modes is calculated, and KL divergence is used as the alignment index. By introducing a spectral consistency loss function to minimize the probability distribution, the modes tend to be uniform in the multi-scale spatial frequency response. ， Where KL represents the KL divergence calculation method, and the subscript... , The spectral distribution similarity between pairs of modes; Based on spectral distribution similarity, the wavelet subband coefficients of each mode are adjusted to match the energy space. Each row is normalized so that the "weighted influence" of the similarity of each mode on other modes sums to 1. The wavelet feature representation of the final output of each mode is a weighted sum of the features of all modes, where the weight is the spectral distribution similarity between a mode and other modes. The wavelet feature representation of the final output of each mode is a weighted sum of the features of all modes, resulting in the wavelet feature tensor of each mode. , , ; ， in, It is in the Normalized similarity weights between pairwise modes in layered wavelet decomposition; ， ， ， in, It is in the Normalized similarity weights between OPT modes in layer wavelet decomposition It is in the Normalized similarity weights between opt and radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between opt and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between radar modes in layer wavelet decomposition. It is in the Normalized similarity weights between radar and atm modes in layer wavelet decomposition. It is in the Normalized similarity weights between atm and opt modes in layer wavelet decomposition It is in the Normalized similarity weights between ATM and Radar modes in layer wavelet decomposition. It is in the The normalized similarity weights between ATM modes in layer wavelet decomposition are used as weighting coefficients for wavelet feature co-reconstruction.

9. An electronic device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 5.

Images