Cloud type inversion model and system for remote sensing images based on symmetric cross attention

Abstract

The present invention discloses a remote sensing image cloud type inversion model and system based on symmetric cross-attention. The steps are as follows: obtain satellite remote sensing image data, extract cloud spectrum, texture and shape features after processing; perform feature encoding on longitude and latitude coordinates, terrain, underlying surface, and time elements; construct a symmetric cross-attention module to bidirectionally fuse images and geographic features, using time features as channel input; use CloudSat cloud products as true values, adopt the Adam optimizer and cross-entropy loss function to train the model, and verify it through indicators such as Kappa coefficient and producer accuracy. This method overcomes the shortcomings of traditional methods that rely only on two-dimensional images and do not take into account external driving factors. It can accurately classify cloud types and improve inversion accuracy and efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image cloud type inversion, and in particular relates to a remote sensing image cloud type inversion model and system based on symmetric cross attention. Background Art

[0002] Clouds play a crucial role in Earth's climate and atmospheric environment, regulating the transfer of radiative energy and influencing the radiation balance between the Earth's surface and the atmosphere. As a significant source of uncertainty in climate prediction and simulation, clouds are the core link between atmospheric process observations and understanding of the climate system. Advances in cloud monitoring technology directly drive the upgrading of numerical weather forecast models, enhanced extreme disaster warning capabilities, and the analysis of climate feedback mechanisms. Refined cloud type parameters not only reveal regional differences but also further enhance our understanding of the interaction between clouds and climate, providing applications in meteorological forecasting, climate modeling, and disaster monitoring and early warning.

[0003] From traditional cloud type inversion techniques to the currently popular deep learning-based cloud type inversion techniques, the core goal has always been to improve the accuracy and practicality of class inversion. Although deep learning methods surpass traditional techniques in accuracy, they primarily rely on image data for cloud type inversion and often fail to fully consider the influence of key factors such as geography and time on cloud formation and dissipation. Furthermore, these methods fail to effectively utilize prior knowledge about the temporal and spatial distribution patterns of different cloud types, which may affect the stability and reliability of cloud type inversion results. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a remote sensing image cloud type inversion model and system based on symmetric cross attention, which can improve the cloud type inversion capability.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solution, which includes the following steps:

[0006] (1) Data acquisition and preprocessing: collect FY-4A / 4B satellite image data from the Fengyun series, obtain geographic elements and time elements, unify the data format and remove outliers, and construct a temporally and spatially consistent dataset through spatiotemporal matching;

[0007] (2) Image feature extraction: constructing a set of cloud type-related spectral indices through band operations and extracting texture features of remote sensing images to reflect the shape and structure of clouds;

[0008] (3) Multi-factor feature coding: feature coding of latitude and longitude coordinates, terrain elements, underlying surface type features, and time elements using different coding methods;

[0009] (4) Symmetric cross attention fusion: construct a symmetric cross attention module to fuse image features such as spectrum, texture, shape, and geographic elements, and input the encoded time elements into the network as channels;

[0010] (5) Model training and verification: training a multi-dimensional information fusion processing network model, performing type inversion through multi-layer perceptron and Transformer blocks, learning and training the model with CloudSat cloud products as the true value, and using Kappa coefficient, producer accuracy, user accuracy and overall accuracy as quantitative evaluation indicators.

[0011] Furthermore, the step (1) includes the following steps:

[0012] (1.1) FY-4A / B satellites can easily obtain data that matches the temporal and spatial characteristics of CloudSat satellite cloud products. Download the required FY-4A / B satellite remote sensing image data from the official website and perform data normalization and data enhancement, including random rotation, flipping, cropping, and scaling.

[0013] (1.2) Obtain large-scale geographic information through ground surveys and remote sensing satellite imagery, including geographic elements such as latitude and longitude coordinates, underlying surface type, and topography, remove erroneous data and outliers, and convert data from different sources into a unified format to facilitate subsequent processing and analysis;

[0014] (1.3) Based on a large amount of historical data, remote sensing images, cloud type products, and geographic features are screened through spatiotemporal matching to construct a spatiotemporally consistent cloud type inversion dataset.

[0015] Furthermore, the step (2) includes the following steps:

[0016] (2.1) Use band operations to construct a set of spectral indices related to cloud types. By analyzing the reflectivity, absorptivity, and other information of images in different bands and combining them with the physical properties of clouds, a specific spectral index formula is designed to distinguish different types of clouds.

[0017] The Cloud Phase Index (CPI) is used to distinguish the phase of clouds (water cloud or ice cloud). A higher CPI value indicates that the cloud contains more ice crystals and may be an ice cloud; a lower CPI value indicates that the cloud is a water cloud. The calculation formula is as follows:

[0018]

[0019] in Indicates the reflectivity of the short-wave infrared band (such as 1.6μm), Indicates the reflectivity of the visible light band (such as 0.65μm);

[0020] The Cloud Thickness Index (CTI) estimates cloud thickness by analyzing the differences in cloud reflectivity across multiple wavelengths. Thicker clouds have higher reflectivity in the near-infrared band. The calculation formula is as follows:

[0021]

[0022] in Indicates the reflectivity of the near-infrared band (such as 0.86μm), Indicates the reflectivity of the visible light band (such as 0.65μm);

[0023] The cloud base height index (CBH) estimates cloud base height by analyzing the differences in cloud reflectivity or absorptivity in multiple band images. It is related to cloud temperature and water vapor content. The calculation formula is as follows:

[0024]

[0025] in Indicates the reflectivity in the near-infrared band (such as 11μm), Represents the reflectivity in the short-wave infrared band (e.g., 1.6 μm). Reflectivity and absorptivity are different aspects of the same physical process. They are directly related through the law of conservation of energy, and their sum is 1.

[0026] (2.2) Extracting texture features from remote sensing images to reflect the shape and structure of clouds. Texture features such as contrast, correlation, and energy can be calculated using the gray-level co-occurrence matrix to capture the texture details of clouds.

[0027] The gray-level co-occurrence matrix G(i, j) represents the probability of the occurrence of a pixel pair with gray-level values ​​i and j at a given direction and distance. By calculating the statistics of the gray-level co-occurrence matrix (such as energy, contrast, homogeneity, etc.), the values ​​reflecting the texture characteristics of the image can be extracted. The texture feature calculation formula is as follows:

[0028] energy:

[0029] Contrast ratio:

[0030] Homogeneity:

[0031] Dependencies:

[0032] in, and μc are the means along the rows and columns respectively, and are the standard deviations along the rows and columns, respectively.

[0033] Furthermore, the step (3) includes the following steps:

[0034] (3.1) Convert the geographic coordinates of longitude and latitude into sine and cosine function values, and map the geographic location information to a numerical range that the model can handle. This effectively captures the periodicity and continuity characteristics of the geographic location and solves the problems of discontinuity and scale inconsistency that may arise from directly using the latitude and longitude values. The specific formulas for sine and cosine encoding are as follows:

[0035] For each position pos and each dimension 2i, the positional encoding PE is defined as:

[0036]

[0037] For each position pos and each dimension 2i+1, the positional encoding PE is defined as:

[0038]

[0039] Among them, pos represents the position of the input sequence, d model Represents the dimension of the model embedding vector, the sine function is applied to all even dimensions (index 2i), and the cosine function is applied to all odd dimensions (index 2i+1);

[0040] (3.2) Terrain features are similar to image data and can be encoded through multi-layer convolution. A three-layer convolutional network is constructed with a convolution kernel size of 3×3 and a stride of 2 to achieve downsampling, reduce the size of the feature map, and extract higher-level features.

[0041] (3.3) The underlying surface feature data is normalized and mapped to a high-dimensional space through the embedding layer of the convolutional neural network. The input of the embedding layer is the discrete underlying surface feature value, and the output is a vector of fixed dimension, which is effectively encoded as an embedded vector. The normalization formula is as follows:

[0042]

[0043] Where x is the original data, is the normalized data, and are the minimum and maximum values ​​of the data respectively;

[0044] (3.4) The day, hour, minute, and second in the time element are all sparse data, and the embedding method is also used for feature encoding. The day, hour, minute, and second of each year are converted into index values ​​respectively. The corresponding embedding vector is obtained through the embedding layer, and the embedding vectors of day, hour, minute, and second are spliced ​​on the channel to form a comprehensive time feature vector.

[0045] Furthermore, the step (4) includes the following steps:

[0046] (4.1) Construct a symmetric cross-attention module to project geographic features (including longitude and latitude coordinates, terrain features, and underlying surface features) into query ,key Sum ; Project image features (including spectrum, texture, shape) as query ,key Sum ;

[0047] (4.2) Utilization and Multiply and calculate the similarity matrix, then weight Perform attention calculation to obtain the filtered image feature F image ;use and Multiply and calculate the similarity matrix, then weight Perform attention calculation to obtain the filtered geographic features F geo ;

[0048] For attention calculation from geographic features to image features:

[0049] F image

[0050] For attention calculation from image features to geographic features:

[0051] F geo

[0052] in is the dimension of the key vector, used to scale the attention scores;

[0053] (4.3) Concatenate the filtered geographic features and image features into vectors to form a comprehensive feature vector, and transform the features through global average pooling and multi-layer perceptron;

[0054] (4.4) Use the Softmax function to normalize the transformed feature vector to ensure the scale consistency of the features, calculate the weights of the two filtered features, and add the filtered image features and the original image features without attention filtering according to the weights to obtain the final fusion result;

[0055] (4.5) Add the temporal features directly as input channels to the network, process them in parallel with the geographic and image features, perform dimensionality reduction through the downsampling layer, and perform more efficient processing in subsequent network layers;

[0056] (4.6) The geographic features, image features, and time features are processed separately through the Transformer block to capture the long-range dependencies between features. The processed features are then upsampled through the upsampling layer to restore them to their original feature dimensions.

[0057] (4.7) The processed features are further processed through global average pooling and multi-layer perceptron to generate the final cloud type inversion results.

[0058] Furthermore, the step (5) includes the following steps:

[0059] (5.1) Model Training: The multi-dimensional information fusion processing model was trained using the preprocessed training data. The initial learning rate was set to 0.002, the batch size was 32, and the number of training rounds was set to 100. The Adam optimizer was used, which generally performs well due to its adaptive learning rate and momentum properties. The cross entropy loss function was used, and the network parameters were optimized using the backpropagation algorithm.

[0060] (5.2) Model validation: The cloud type inversion results of the model are relatively verified, with CloudSat cloud products as the true value, and the Kappa coefficient, producer accuracy, user accuracy and overall accuracy are used as quantitative evaluation indicators.

[0061] The present invention also discloses a remote sensing image cloud type inversion system based on symmetric cross attention, comprising:

[0062] Data preprocessing module: performs geometric correction, radiation correction, normalization, and data enhancement on FY-4A / 4B remote sensing images, unifies the formats of geographic and temporal elements, removes outliers, and performs time-space matching, and divides the training set into the validation set;

[0063] Image feature extraction module: includes a spectral index calculation unit and a texture feature extraction unit. The spectral index calculation unit calculates CPI, CTI, and CBH based on the band reflectance; the texture feature extraction unit calculates energy, contrast, homogeneity, and correlation using the gray-level co-occurrence matrix;

[0064] Multi-factor feature coding module: includes a latitude and longitude coordinate coding unit, a terrain coding unit, an underlying surface coding unit, and a time coding unit. The latitude and longitude coordinate coding unit uses sine and cosine functions to map longitude and latitude to numerical intervals; the terrain coding unit uses a three-layer convolutional network (3×3 convolution kernel, step size 2) to extract terrain features; the underlying surface coding unit uses normalization and then maps it into a high-dimensional vector through an embedding layer; the time coding unit converts the day, hour, minute, and second into an embedded vector and splices them together;

[0065] Symmetrical cross attention module: It includes feature projection unit, bidirectional attention calculation unit, and feature fusion unit. The feature projection unit projects geographic features and image features into Q, K, and V respectively; the bidirectional attention calculation unit performs F image and F geo The feature fusion unit concatenates the filtered features, performs global average pooling, MLP, Softmax normalization, and weight calculation to generate fused features. Temporal features are processed in parallel as independent channels, and the dimensions are adjusted through Transformer blocks and up / down sampling layers.

[0066] Model training and validation module: This module includes a training unit and a validation unit. The training unit trains the network based on the CloudSat true value using the Adam optimizer and the cross-entropy loss function. The validation unit calculates the Kappa coefficient, producer accuracy, user accuracy, and overall accuracy to evaluate the inversion results.

[0067] Furthermore, in the symmetric cross attention module, the bidirectional attention calculation of geographic features and image features is achieved through matrix multiplication, Softmax normalization and weighted summation, and the weight is determined by normalizing the transformed feature vector through the Softmax function.

[0068] The beneficial effects achieved by the present invention are:

[0069] 1. Break through the limitations of traditional methods and improve inversion accuracy

[0070] Multidimensional Information Fusion: Traditional methods rely solely on two-dimensional imagery, failing to consider the impact of geographic factors (latitude and longitude coordinates, topography, underlying surface) and time on cloud formation. This new method overcomes the limitations of single imagery by fusing multidimensional features such as spectral, textural, geographic, and temporal information, significantly improving the accuracy of cloud type inversion.

[0071] Quantitative indicator optimization: Using indicators such as Kappa coefficient, producer precision, user precision, and overall accuracy to verify, the model can achieve a reliable extension of "cloud binary classification → cloud multi-class classification", and the inversion effect is better than traditional methods.

[0072] 2. Symmetrical Cross-Attention Mechanism Enhances Feature Interaction

[0073] Bidirectional feature screening: Through bidirectional attention calculation of geographic features and image features (geography → image, image → geography), refined screening and fusion of cross-modal features are achieved, effectively capturing the correlation between the two types of features and improving feature expression capabilities.

[0074] Long-distance dependency modeling: Combined with Transformer block processing features, it captures the long-distance dependencies between different elements, enhances the model's understanding of the spatiotemporal distribution of cloud types, and improves the stability and reliability of inversion results.

[0075] 3. Multi-factor encoding improves feature expression capabilities

[0076] Efficient modeling of geographic features:

[0077] The longitude and latitude coordinates use sine and cosine position encoding to solve the discontinuity and scale problems of longitude and latitude values ​​and retain the periodicity and continuity of the geographic location.

[0078] The terrain elements are extracted with high-level features through convolutional networks, and the underlying surface types are mapped to high-dimensional space through normalization and embedding layers to achieve efficient encoding of geographic elements.

[0079] Dynamic fusion of temporal features: Time elements (day, hour, minute, second) are encoded into comprehensive feature vectors and input into the network as independent channels to capture the evolution of cloud types over time and enhance the model's adaptability to dynamic scenarios.

[0080] 4. Data adaptability and application scalability

[0081] Domestic satellite data support: Based on domestic satellite data such as FY-4A / 4B, combined with CloudSat true values, a temporally and spatially consistent dataset is constructed to reduce dependence on foreign data and adapt to domestic remote sensing monitoring needs.

[0082] Multi-scenario application potential: It can accurately invert cloud types (such as water clouds, ice clouds, thick clouds, thin clouds, etc.), provide richer cloud parameters for meteorological forecasting, climate modeling, disaster warning and other fields, and promote the upgrading of numerical weather forecast models and the improvement of extreme weather monitoring capabilities.

[0083] 5. System modular design improves practicality

[0084] Standardized processing flow: Through modular design including data preprocessing, feature extraction, encoding, attention fusion, and model training, the entire process from data input to cloud type inversion is automated, improving processing efficiency and reproducibility.

[0085] Enhanced generalization capability: Combining data augmentation (random rotation, flipping, etc.) and spatiotemporal matching technology to build a high-quality dataset and improve the model's generalization capability for remote sensing images from different regions and time periods. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a network structure diagram of a remote sensing image cloud type inversion model based on symmetric cross attention provided by the first embodiment of the present invention;

[0087] Figure 2 This is a structural diagram of a symmetric cross-attention module in a remote sensing image cloud type inversion model based on symmetric cross-attention provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0088] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0089] Example 1:

[0090] like Figure 1 As shown, this embodiment discloses a remote sensing image cloud type inversion model based on symmetric cross attention, and the specific training process includes the following steps:

[0091] (1) Acquire and integrate satellite data, mainly domestic satellite image data such as the Fengyun series FY-4A / 4B, and collect geographic and temporal elements.

[0092] Step (1) includes the following steps:

[0093] (1.1) The FY-4A / B satellite is located in geostationary orbit. Satellite image data, including spectral information of multiple bands, is downloaded from the official platform and pre-processed by geometric correction and radiation correction.

[0094] (1.2) By combining ground observation stations with satellite technology, we have collected and acquired comprehensive geographic information, including latitude and longitude coordinates, surface types, and topography.

[0095] (1.3) Based on abundant historical data, spatiotemporal matching technology is used to screen out data that is consistent with remote sensing images, cloud type products, and related geographic features. A spatiotemporally consistent cloud type inversion dataset is then constructed and divided into a training set and a validation set, providing a data foundation for cloud type inversion research.

[0096] (2) A set of spectral indices related to cloud types is constructed through band operations, and the texture features of remote sensing images are extracted to reflect the shape and structure of clouds.

[0097] Step (2) includes the following steps:

[0098] (2.1) Use band operations to construct a set of cloud-type-related spectral indices. By analyzing the reflectivity, absorptivity, and other information of images in different bands and combining them with the physical properties of clouds, we design specific spectral index formulas to distinguish different cloud types.

[0099] The Cloud Phase Index (CPI) is used to distinguish the phase of clouds (water cloud or ice cloud). A higher CPI value indicates that the cloud contains more ice crystals and may be an ice cloud; a lower CPI value indicates that the cloud is a water cloud. The calculation formula is as follows:

[0100]

[0101] in Indicates the reflectivity of the short-wave infrared band (such as 1.6μm), Indicates the reflectivity in the visible light band (such as 0.65μm).

[0102] The Cloud Thickness Index (CTI) estimates cloud thickness by analyzing the differences in cloud reflectivity in multiple band images. Thicker clouds have higher reflectivity in near-infrared band images. The calculation formula is as follows:

[0103]

[0104] in Indicates the reflectivity of the near-infrared band (such as 0.86μm), Indicates the reflectivity in the visible light band (such as 0.65μm).

[0105] The cloud base height index (CBH) estimates cloud base height by analyzing the differences in cloud reflectivity or absorptivity in multiple band images. It is related to cloud temperature and water vapor content. The calculation formula is as follows:

[0106]

[0107] in Indicates the reflectivity in the near-infrared band (such as 11μm), Represents the reflectivity in the shortwave infrared band (such as 1.6μm). Reflectivity and absorptivity are different aspects of the same physical process. They are directly related through the law of conservation of energy, and their sum is 1.

[0108] (2.2) By processing remote sensing images, the texture features of remote sensing images are extracted to reflect the shape and structure of clouds. The texture features of remote sensing images, such as contrast, correlation, energy, etc., can be calculated using the gray-level co-occurrence matrix to capture the texture details of clouds.

[0109] The gray-level co-occurrence matrix G(i, j) represents the probability of a pair of pixels with gray-level values ​​i and j occurring at a given direction and distance. By calculating the statistics of the gray-level co-occurrence matrix (such as energy, contrast, homogeneity, etc.), we can extract values ​​reflecting the texture characteristics of the image. The formula for calculating texture characteristics is as follows:

[0110] energy:

[0111] Contrast ratio:

[0112] Homogeneity:

[0113] Dependencies:

[0114] in, and μc are the means along the rows and columns respectively, and are the standard deviations along the rows and columns, respectively.

[0115] (3) Characteristic encoding of latitude and longitude coordinates, terrain elements, underlying surface type characteristics and time elements through different encoding methods;

[0116] Step (3) includes the following steps:

[0117] (3.1) Convert the geographic coordinates of longitude and latitude into sine and cosine function values, and map the geographic location information to a numerical range that the model can handle. This effectively captures the periodicity and continuity characteristics of the geographic location and solves the problems of discontinuity and scale inconsistency that may arise from directly using the latitude and longitude values. The specific formulas for sine and cosine encoding are as follows:

[0118] For each position pos and each dimension 2i, the positional encoding PE is defined as:

[0119]

[0120] For each position pos and each dimension 2i+1, the positional encoding PE is defined as:

[0121]

[0122] Among them, pos represents the position of the input sequence, d model Represents the dimension of the model embedding vector, the sine function is applied to all even dimensions (indexed 2i), and the cosine function is applied to all odd dimensions (indexed 2i+1).

[0123] (3.2) Terrain features are similar to image data and can be encoded through multi-layer convolution. A three-layer convolutional network is constructed with a convolution kernel size of 3×3 and a stride of 2 to achieve downsampling, reduce the size of the feature map, and extract higher-level features.

[0124] (3.3) The underlying surface feature data is normalized and mapped to a high-dimensional space through the embedding layer of the convolutional neural network. The input of the embedding layer is the discrete underlying surface feature value, and the output is a vector of fixed dimension, which is effectively encoded as an embedded vector. The normalization formula is as follows:

[0125]

[0126] Where x is the original data, is the normalized data, and are the minimum and maximum values ​​of the data respectively.

[0127] (3.4) The day, hour, minute, and second in the time element are all sparse data, and the embedding method is also used for feature encoding. The day, hour, minute, and second of each year are converted into index values ​​respectively. The corresponding embedding vector is obtained through the embedding layer, and the embedding vectors of day, hour, minute, and second are spliced ​​on the channel to form a comprehensive time feature vector.

[0128] (4) Construct a symmetrical cross-attention module to align and fuse image features such as spectral texture and shape, and geographical elements in feature space, and input the encoded time elements into the network as channels;

[0129] Step (4) includes the following steps:

[0130] (4.1) Construct a symmetric cross-attention module to project geographic features (including longitude and latitude coordinates, terrain features, and underlying surface features) into query ,key Sum ; Project image features (including spectrum, texture, shape) as query ,key Sum .

[0131] (4.2) Utilization and Multiply and calculate the similarity matrix, then weight Perform attention calculation to obtain the filtered image feature F image ;use and Multiply and calculate the similarity matrix, then weight Perform attention calculation to obtain the filtered geographic features F geo .

[0132] For attention calculation from geographic features to image features:

[0133]

[0134] For attention calculation from image features to geographic features:

[0135]

[0136] in is the dimension of the key vector, used to scale the attention scores.

[0137] (4.3) The filtered geographic features and image features are vector-concatenated to form a comprehensive feature vector, and the features are transformed through global average pooling and multi-layer perceptron.

[0138] (4.4) Use the Softmax function to normalize the transformed feature vector to ensure the scale consistency of the features, calculate the weights of the two filtered features, and add the filtered image features and the original image features without attention filtering according to the weights to obtain the final fusion result.

[0139] (4.5) The temporal features are directly added as input channels to the network, processed in parallel with the geographic features and image features, and dimensionality reduction is performed through the downsampling layer, so that they can be processed more efficiently in subsequent network layers.

[0140] (4.6) The geographic features, image features, and time features are processed separately through the Transformer block to capture the long-distance dependencies between the features; the processed features are then up-scaled through the upsampling layer to restore them to the original feature dimensions.

[0141] (4.7) The processed features are further processed through a multi-layer perceptron to generate the final cloud classification results.

[0142] (5) The multi-dimensional information fusion processing network model was trained and verified using CloudSat cloud products as the true value. Kappa coefficient, producer accuracy, user accuracy and overall accuracy were used as quantitative evaluation indicators.

[0143] Step (5) includes the following steps:

[0144] (5.1) Model Training: The model was trained using the preprocessed training data. The initial learning rate was set to 0.002, the batch size was 32, and the number of training epochs was set to 100. The Adam optimizer was used, which generally performs well due to its adaptive learning rate and momentum properties. Cross entropy was used as the loss function, and the network parameters were optimized using the backpropagation algorithm.

[0145] (5.2) Model validation: The model cloud type inversion results are relatively verified, with CloudSat cloud products as the true value, and the Kappa coefficient, producer accuracy, user accuracy and overall accuracy are used as quantitative evaluation indicators.

[0146] The Kappa coefficient is used to measure the consistency between the classification results and the random classification results. The calculation formula is:

[0147]

[0148] in is the overall accuracy, that is, the proportion of classification results consistent with the true value, is the chance agreement, which indicates the proportion of random classification results that are consistent with the true value.

[0149] Producer precision refers to the proportion of samples that are correctly classified among samples that actually belong to a certain category. For the i-th category, the calculation formula is:

[0150]

[0151] in It is the element in the jth row and ith column of the confusion matrix, indicating the number of samples that are actually in the ith class but are classified as the jth class.

[0152] User precision refers to the proportion of samples classified as a certain category that actually belong to that category. For category i, the calculation formula is:

[0153]

[0154] in It is the element in the jth row and ith column of the confusion matrix, indicating the number of samples that are actually in the ith class but are classified as the jth class.

[0155] The overall accuracy indicates the proportion of classification results that are consistent with the true value, and the calculation formula is:

[0156]

[0157] in is the element on the diagonal of the confusion matrix, indicating the number of samples correctly classified into the i-th class. N is the total number of samples.

[0158] Following these steps, an effective multidimensional information fusion processing network is constructed based on the collaborative feature coding and fusion of multiple factors, achieving excellent cloud type inversion results. This overcomes the shortcomings of traditional cloud type inversion methods that rely solely on two-dimensional image information and fail to consider external driving factors, and achieves a reliable extension of the "cloud binary classification" to multi-class cloud classification.

[0159] Example 2:

[0160] like Figure 2 As shown, this embodiment discloses a remote sensing image cloud type inversion system based on symmetric cross attention. The specific training process includes the following steps:

[0161] Data Preprocessing Module: This module preprocesses FY-4A / 4B remote sensing image data to ensure image quality. It inputs the collected raw remote sensing image data, geographic data, and time data, and outputs preprocessed remote sensing image data, standardized geographic data, and time data, which are divided into training and test sets.

[0162] Image Feature Extraction Module: This module is responsible for extracting spectral, texture, and shape features from multispectral images. Taking multispectral image data as input, the module analyzes the reflectance, absorptivity, and other information from different wavebands. In combination with cloud physical properties, it designs a specific spectral index formula to distinguish different cloud types. It also uses a gray-level co-occurrence matrix to calculate image texture features, such as contrast, correlation, and energy, to capture cloud texture details and reflect cloud shape and texture characteristics.

[0163] Multi-factor feature coding module: It integrates the latitude and longitude coordinate position coding, terrain feature coding, underlying surface feature coding and time feature coding modules to achieve the coding and fusion of multi-source features. The latitude and longitude coordinate position coding module inputs the longitude and latitude values ​​of the geographic coordinates, converts the longitude and latitude of the geographic coordinates into the values ​​of the sine and cosine functions, and maps the geographic location information to the numerical range that the model can handle, effectively capturing the periodicity and continuity characteristics of the geographic location; the terrain feature coding module constructs a 3-layer convolutional network, with the convolution kernel size of each layer being 3×3 and the step size being 2. The terrain feature image data is input into the convolutional network, and downsampling is achieved through layer-by-layer convolution operations to reduce the size of the feature map while extracting higher-level features; the underlying surface feature coding module inputs the distance The scattered underlying surface feature values ​​are normalized and the normalized features are input into the embedding layer of the convolutional neural network. The embedding layer maps the discrete underlying surface feature values ​​into a high-dimensional space and outputs a vector of fixed dimension, i.e., the embedding vector, thereby effectively encoding the underlying surface features. The time element feature encoding module inputs the day, hour, minute, and second in the time element, converts the day, hour, minute, and second of each year into index values ​​respectively, obtains the embedding vectors corresponding to the day, hour, minute, and second through the embedding layer, and splices them on the channel to form a comprehensive time feature vector.

[0164] Symmetrical cross attention module: First, the geographical features (including longitude and latitude coordinates, terrain features and underlying surface features) are projected into the query ,key Sum , projecting image features (including spectrum, texture, shape) into query ,key Sum Next, use and Multiply and calculate the similarity matrix, then weight Perform attention calculation to obtain the filtered image feature F image ;use and Multiply and calculate the similarity matrix, then weight Perform attention calculation to obtain the filtered geographic features F geo During the feature fusion and transformation phase, the filtered geographic features and image features are first concatenated to obtain fused features. Subsequently, a global average pooling operation is applied to the fused features. This operation compresses the feature map into a fixed-length vector by averaging the feature map over its spatial dimensions (height and width), thereby reducing the spatial dimension of the feature while preserving information in the channel dimension. The globally average pooled feature vector is then input into a multilayer perceptron (MLP), which consists of two fully connected layers activated by a nonlinear activation function. Specifically, the first fully connected layer maps the input features to a hidden layer, where a ReLU activation function introduces nonlinearity. The second fully connected layer further maps the hidden layer features to the target dimension, resulting in the transformed features. Finally, the transformed features are normalized using the Softmax function, and weights are calculated for the two filtered features. These weights reflect the importance of each feature in the fusion process. The filtered image features are summed with the original image features without attention filtering based on the weights to obtain the final fusion result. This fusion result integrates information from both geographic and image features, providing a richer feature representation for subsequent tasks. Temporal features are directly added as input channels to the network and processed in parallel with geographic and image features. Downsampling layers are used to reduce dimensionality, and upsampling layers are used to restore the processed features to their original dimensions. These features are then further processed through a multilayer perceptron to generate the final cloud classification results.

[0165] Model training and validation module: Use the data from the training set to train the multi-dimensional information fusion processing network, take CloudSat cloud products as the true value, and use Kappa coefficient, producer accuracy, user accuracy, and overall accuracy as quantitative evaluation indicators.

[0166] In summary, the beneficial effects of the present invention are as follows:

[0167] Multi-dimensional information fusion improves inversion accuracy: breaking through the limitation of traditional methods that rely only on two-dimensional images, fusing spectrum (constructing CPI, CTI, CBH index), texture (gray-level co-occurrence matrix to extract energy and other parameters), geography (latitude and longitude coordinates, terrain, underlying surface) and time elements, and compensating for the defects of single information through the collaboration of multi-source data, significantly improving the accuracy and stability of cloud type classification.

[0168] Symmetrical cross-attention enhances feature interaction: Through bidirectional attention calculation of geographic and image features (geography → image, image → geography), it achieves refined screening and fusion of cross-modal features. Combined with Transformer blocks, it captures long-range dependencies, strengthens the correlation between features, and improves the model's understanding of cloud spatiotemporal distribution patterns.

[0169] Multi-factor coding optimizes feature expression: Longitude and latitude coordinates use sine and cosine coding to solve the problem of scale discontinuity, terrain is extracted through a convolutional network, the underlying surface is normalized and embedded in a high-dimensional space, and time elements are encoded as comprehensive vectors to ensure that all types of data are efficiently integrated into the model and enhance feature representation capabilities.

[0170] Domestic data adaptation and application expansion: Based on FY-4A / 4B satellite data, a temporally and spatially consistent dataset is constructed to reduce dependence on foreign data. It supports multi-type classification such as water clouds / ice clouds, and can be applied to meteorological forecasting, climate modeling and other fields, promoting the upgrade of numerical forecast models and the improvement of extreme disaster warning capabilities.

[0171] Modular design improves practicality: The modularization of the entire process (preprocessing, feature extraction, encoding, fusion, and training) enables automated processing, combined with data enhancement and spatiotemporal matching technologies to improve model generalization capabilities and processing efficiency.

[0172] The foregoing description shows and describes several preferred embodiments of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the inventive concept described herein by the teachings above or by techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be within the scope of the appended claims.

Claims

1. A cloud type inversion model for remote sensing images based on symmetric cross attention, characterized by: The following steps are involved: (1) Data acquisition and preprocessing: collect FY-4A / 4B satellite image data from the Fengyun series, obtain geographic elements and time elements, unify the data format and remove outliers, and construct a temporally and spatially consistent dataset through spatiotemporal matching; (2) Image feature extraction: constructing a set of cloud type-related spectral indices through band operations and extracting texture features of remote sensing images to reflect the shape and structure of clouds; (3) Multi-factor feature coding, which uses different coding methods to encode the latitude and longitude coordinates, terrain elements, underlying surface type features, and time elements; (4) Symmetric cross attention fusion: construct a symmetric cross attention module to fuse the image features of spectral texture shape and geographical elements, and input the encoded time elements into the network as channels; (5) Model training and verification: training a multi-dimensional information fusion processing network model, performing type inversion through multi-layer perceptron and Transformer blocks, using CloudSat cloud products as the true value to train the model, and using Kappa coefficient, producer accuracy, user accuracy, and overall accuracy as quantitative evaluation indicators; Wherein said step (4) comprises the following steps: (4.1) Construct a symmetric cross-attention module to project the geographical features including longitude and latitude coordinates, terrain features and underlying surface features into the query Q geo , key K geo Sum V geo ; Project the image features including spectrum, texture, and shape into query Q image , key K image Sum V image ; (4.2) Using Q geo and K image Multiply and calculate the similarity matrix, then weight V image Perform attention calculation to obtain the filtered image feature F image ; Using Q image and K geo Multiply and calculate the similarity matrix, then weight V geo Perform attention calculation to obtain the filtered geographic features F geo ; For attention calculation from geographic features to image features: For attention calculation from image features to geographic features: where d k is the dimension of the key vector, used to scale the attention scores; (4.3) Concatenate the filtered geographic features and image features into vectors to form a comprehensive feature vector, and transform the features through global average pooling and multi-layer perceptron; (4.4) Use the Softmax function to normalize the transformed feature vector to ensure the scale consistency of the features, calculate the weights of the two filtered features, and add the filtered image features and the original image features without attention filtering according to the weights to obtain the final fusion result; (4.5) Add the temporal features directly as input channels to the network, process them in parallel with the geographic and image features, perform dimensionality reduction through the downsampling layer, and perform more efficient processing in subsequent network layers; (4.6) The geographic features, image features, and time features are processed separately through the Transformer block to capture the long-range dependencies between the features. The processed features are then up-sampled through the upsampling layer to restore them to their original feature dimensions. (4.7) The processed features are further processed through global average pooling and multi-layer perceptron to generate the final cloud type inversion results.

2. The remote sensing image cloud type inversion model based on symmetric cross attention according to claim 1 is characterized by: The step (1) comprises the following steps: (1.1) FY-4A / B can easily obtain data that matches the temporal and spatial characteristics of CloudSat satellite cloud products. Download the required FY-4A / B satellite remote sensing image data from the official website and perform data normalization and data enhancement, including random rotation, flipping, cropping, and scaling. (1.2) Obtain large-scale geographic information through ground surveys and remote sensing satellite imagery, including latitude and longitude coordinates, underlying surface type, and topographical features, remove erroneous data and outliers, and convert data from different sources into a unified format for subsequent processing and analysis; (1.3) Based on a large amount of historical data, remote sensing images, cloud type products and geographic feature data are screened through spatiotemporal matching to construct a spatiotemporally consistent cloud type inversion dataset.

3. The remote sensing image cloud type inversion model based on symmetric cross attention according to claim 1, characterized in that: The step (2) comprises the following steps: (2.1) Using band operations to construct a set of cloud-type-related spectral indices; by analyzing the reflectivity and absorptivity information of images in different bands and combining them with the physical properties of clouds, a specific spectral index formula is designed to distinguish different types of clouds; The Cloud Phase Index (CPI) is used to distinguish the phases of clouds, which can be water clouds or ice clouds. A higher CPI value indicates that the cloud contains more ice crystals and may be an ice cloud; a lower CPI value indicates that the cloud is a water cloud. The calculation formula is as follows: Among them B 短波红外 Indicates the reflectivity of the short-wave infrared band, B 可见光 Indicates the reflectivity of visible light band; The Cloud Thickness Index (CTI) estimates cloud thickness by analyzing the differences in cloud reflectivity across multiple wavelengths. Thicker clouds have higher reflectivity in the near-infrared band. The calculation formula is as follows: Among them B 近红外 represents the reflectivity in the near-infrared band, B 可见光 Indicates the reflectivity of visible light band; The cloud base height index (CBH) estimates cloud base height by analyzing the differences in cloud reflectivity and absorptivity in multiple waveband images. It is related to cloud temperature and water vapor content. The calculation formula is as follows: Among them B 热红外 represents the reflectivity in the near-infrared band, B 短波红外 Indicates the reflectivity in the shortwave infrared band; (2.2) Extracting texture features from remote sensing images to reflect the shape and structure of clouds. Texture features can be calculated using the gray-level co-occurrence matrix, which is a matrix of contrast, correlation, and energy to capture the texture details of clouds. The gray-level co-occurrence matrix G(i, j) represents the probability of the occurrence of a pixel pair with gray-level values ​​i and j at a given direction and distance. By calculating the statistics of the gray-level co-occurrence matrix, the value reflecting the texture characteristics of the image can be extracted. The texture feature calculation formula is as follows: energy: Contrast ratio: Homogeneity: Dependencies: Among them, μ r and μc are the means along the rows and columns respectively, σ r and σ c are the standard deviations along the rows and columns, respectively.

4. The remote sensing image cloud type inversion model based on symmetric cross attention according to claim 1 is characterized by: The step (3) comprises the following steps: (3.1) The geographic coordinates longitude and latitude are converted into the values ​​of sine and cosine functions, and the geographic location information is mapped to a numerical range that the model can handle. This effectively captures the periodicity and continuity characteristics of the geographic location and solves the problems of discontinuity and scale inconsistency that may arise from directly using the latitude and longitude values. The specific formulas for sine and cosine encoding are as follows: For each position pos and each dimension 2i, the positional encoding PE is defined as: For each position pos and each dimension 2i+1, the positional encoding PE is defined as: Among them, pos represents the position of the input sequence, d model Represents the dimension of the model embedding vector, the sine function is applied to all even dimensions, the even dimension index is 2i, and the cosine function is applied to all odd dimensions, the odd dimension index is 2i+1; (3.2) Terrain features are similar to image data and can be encoded through multi-layer convolution. A three-layer convolutional network is constructed with a convolution kernel size of 3×3 and a stride of 2 to achieve downsampling, reduce the size of the feature map, and extract higher-level features. (3.3) The underlying surface feature data is normalized and mapped to a high-dimensional space through the embedding layer of the convolutional neural network. The input of the embedding layer is the discrete underlying surface feature value, and the output is a vector of fixed dimension, which is effectively encoded as an embedded vector. The normalization formula is as follows: Where x is the original data, x norm is the normalized data, x min and x max are the minimum and maximum values ​​of the data respectively; (3.4) The day, hour, minute, and second in the time element are all sparse data, and the embedding method is also used for feature encoding. The day, hour, minute, and second of each year are converted into index values ​​respectively. The corresponding embedding vector is obtained through the embedding layer, and the embedding vectors of day, hour, minute, and second are spliced ​​on the channel to form a comprehensive time feature vector.

5. The remote sensing image cloud type inversion model based on symmetric cross attention according to claim 1, characterized in that: The step (5) comprises the following steps: (5.1) Model Training: The multi-dimensional information fusion processing model was trained using the preprocessed training data. The initial learning rate was set to 0.002, the batch size was 32, and the number of training rounds was set to 100. The Adam optimizer was used, which generally performs well due to its adaptive learning rate and momentum properties. The cross entropy loss function was used, and the network parameters were optimized using the backpropagation algorithm. (5.2) Model validation: The cloud type inversion results of the model are relatively verified, with CloudSat cloud products as the true value, and the Kappa coefficient, producer accuracy, user accuracy, and overall accuracy are used as quantitative evaluation indicators.

6. A remote sensing image cloud type inversion system based on symmetric cross attention, used to implement the remote sensing image cloud type inversion model based on symmetric cross attention according to any one of claims 1 to 5, characterized in that: include: Data preprocessing module: performs geometric correction, radiation correction, normalization and data enhancement on FY-4A / 4B remote sensing images, unifies the formats of geographic elements and time elements, removes outliers, and performs time-space matching, and divides the training set into the validation set; Image feature extraction module: includes a spectral index calculation unit and a texture feature extraction unit. The spectral index calculation unit calculates CPI, CTI, and CBH based on the band reflectance; the texture feature extraction unit calculates energy, contrast, homogeneity, and correlation using the gray-level co-occurrence matrix; Multi-factor feature coding module: includes a latitude and longitude coordinate coding unit, a terrain coding unit, an underlying surface coding unit, and a time coding unit. The latitude and longitude coordinate coding unit uses sine and cosine functions to map longitude and latitude to numerical intervals; the terrain coding unit uses a three-layer convolutional network with a convolution kernel of 3×3 and a step size of 2 to extract terrain features; the underlying surface coding unit is normalized and then mapped into a high-dimensional vector through an embedding layer; the time coding unit converts the day, hour, minute, and second into an embedded vector and splices them; Symmetrical Cross Attention Module: It includes a feature projection unit, a two-way attention calculation unit, and a feature fusion unit, wherein the feature projection unit projects the geographic features and image features into Q, K, and V respectively; the two-way attention calculation unit performs F image and F geo The bidirectional attention calculation is performed by the feature fusion unit. The features after splicing and filtering are subjected to global average pooling, MLP, Softmax normalization and weight calculation to generate fused features. The temporal features are processed in parallel as independent channels, and the dimensions are adjusted through the Transformer block and up / down sampling layer. Model training and validation module: This module includes a training unit and a validation unit. The training unit trains the network based on the CloudSat true value using the Adam optimizer and the cross-entropy loss function. The validation unit calculates the Kappa coefficient, producer accuracy, user accuracy, and overall accuracy to evaluate the inversion results.

7. The remote sensing image cloud type inversion system based on symmetric cross attention according to claim 6, characterized in that: In the symmetric cross-attention module, the bidirectional attention calculation of geographic features and image features is achieved through matrix multiplication, Softmax normalization and weighted summation, and the weight is determined by normalizing the transformed feature vector through the Softmax function.

Images