Remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion

Abstract

The application discloses a remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion, and relates to the technical field of image detection. The method first respectively performs spatial context feature coding (three times of context extraction and down-sampling to obtain features at different levels) and frequency domain structure feature strengthening coding (operations such as convolution, structure extraction, scaling and fusion); then the frequency domain strengthened features and the third spatial context features are deeply fused through twin self-attention fusion and multilayer perception machine processing, and finally the final result is obtained through context extraction, up-sampling, matrix operation and softmax activation. The method extracts spatial features at multiple levels, strengthens frequency domain features, processes information through multiple branches and multiple paths, gradually fuses and refines, and improves the cloud detection accuracy, robustness and reliability.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image detection, and particularly relates to a remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion. BACKGROUND

[0002] According to physical characteristics and distribution patterns, cloud layers can be roughly divided into thick clouds, uniform clouds, and thin cumulus clouds, and patchy cumulus clouds. In remote sensing images, thick clouds and uniform clouds are relatively easy to identify due to their large volume and regular shape. In contrast, thin cumulus clouds and patchy cumulus clouds are a major challenge in cloud detection due to their small volume, variable characteristics, and irregular shape. In particular, thin cumulus clouds with high transparency make the background features in remote sensing images complex, and the spectral signal often contains both cloud layer and underlying ground (such as vegetation area, soil area, water area, etc.) information, making it difficult to separate the cloud layer and cloud shadow parts, thereby increasing the difficulty of detection. Patchy cumulus clouds, which are irregular in shape and loose in structure, have a smaller particle size and present a more complex appearance in remote sensing images. Their spatial discontinuity and dynamic change characteristics further complicate accurate detection and identification.

[0003] In the field of cloud detection in optical satellite remote sensing images, current algorithms have shown a trend of diversification. Based on their basic principles, these algorithms can be roughly divided into three categories: spectral threshold method, statistical learning method, and deep learning method.

[0004] In the segmentation task of cloud layer and cloud shadow, the core challenge of the spectral threshold method is the similarity of spectral characteristics and the lack of spatial correlation between the two. Cloud shadow usually appears as a low-reflectance area, but its spectral characteristics (such as dark features in the visible light band) are highly overlapping with water bodies, dark vegetation, or mountain shadows, making it difficult to distinguish between them using only spectral threshold values. For example, cloud shadow in the near-infrared band may show weak reflection differences due to ground vegetation coverage, but such subtle changes are easily ignored by the rigid segmentation of threshold methods. At the same time, cloud shadow has a geometric projection relationship with the cloud layer, but the spectral threshold method lacks the ability to model the spatial context (such as the relative position of shadow and cloud, shape consistency), leading to the possibility of misclassifying shadow areas as independent features. In addition, the brightness and range of cloud shadow are influenced by cloud thickness, solar incidence angle, and ground albedo, and dynamic threshold adjustment cannot adapt to the coupled changes of multiple factors simultaneously, resulting in segmentation results with broken or fuzzy boundaries.

[0005] The disadvantages of statistical learning methods mainly lie in the weakness of joint spectral-spatial modeling and the neglect of dynamic projection relationships. The spectral characteristics of cloud shadows are highly overlapped with water bodies, dark vegetation, or terrain shadows. Although statistical models can enhance the discrimination through multi-band combination or texture feature enhancement, they are difficult to effectively model the complex projection relationships between clouds and shadows, which are jointly constrained by the solar elevation angle, the three-dimensional structure of the cloud layer, and the surface albedo. For example, the geometric shape and position of the shadow dynamically change with the height of the cloud layer, and traditional statistical methods lack explicit expression of the spatial topological constraints of cloud-shadows, resulting in spatial discontinuity or misplacement of the shadow segmentation results. In addition, the intensity of cloud shadows is modulated by the cloud thickness, showing a continuous gradual change. The decision mechanism of statistical learning methods based on discrete classification cannot accurately depict such continuous changes, especially in the shallow shadow area of thin cloud projection, where the model is easily disturbed by local noise, resulting in over-segmentation or under-segmentation problems.

[0006] Compared with spectral thresholding and statistical learning methods, deep learning methods exhibit higher detection accuracy and more extensive generalization performance. In addition, by incorporating attention mechanisms, multi-scale feature integration, and frequency domain feature extraction strategies, the model performance can be further optimized, enabling it to perform better in the face of complex and variable backgrounds and diverse cloud types. However, deep learning methods also face challenges such as high data labeling costs, long model training times, and large computational resource requirements. SUMMARY

[0007] The purpose of the present application is to provide a remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion. By respectively encoding the features of the original image from the spatial domain and the frequency domain, obtaining feature information in different domains, and then fusing these features, the final cloud detection output result is obtained to cope with the complex situation of cloud detection in remote sensing images and improve the accuracy and reliability of detection.

[0008] To achieve the above purpose, the embodiments of the present application provide a remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion, comprising:

[0009] Obtaining an original image, and respectively performing spatial domain context feature encoding and frequency domain structure feature enhancement encoding on the original image;

[0010] In the spatial domain context feature encoding process, the original image is subjected to continuous three times of context feature extraction and downsampling operations, and the result of the last context feature extraction and downsampling operation is taken as the input of the next context feature extraction and downsampling operation, to obtain the first spatial context feature, the second spatial context feature, and the third spatial context feature in turn;

[0011] In the frequency domain structure feature enhancement encoding process, the original image is sequentially subjected to 1*1 convolution operation, first structure feature extraction and first multi-path down-sampling operation to obtain the first structure feature; the output of the first structure feature extraction stage is subjected to scaling operation based on equal interval sampling to obtain the first scaling feature; the first structure feature and the first scaling feature are fused and sequentially subjected to second structure feature extraction and second multi-path down-sampling operation to obtain the second structure feature; the output of the second structure feature extraction stage is subjected to scaling operation based on equal interval sampling to obtain the second scaling feature; the second structure feature and the second scaling feature are fused and subjected to 1*1 convolution operation to obtain the frequency domain structure enhancement feature;

[0012] After the frequency domain structure enhancement feature and the third spatial context feature are subjected to normalization operation respectively, fusion operation of twin self-attention is performed to obtain the first output, and the first output and the third spatial context feature are subjected to matrix addition to obtain the twin self-attention output; after the twin self-attention output is subjected to normalization operation, multi-layer perception processing is performed to obtain the second output; the second output and the twin self-attention output are subjected to matrix addition to obtain the collaborative fusion feature;

[0013] The collaborative fusion feature is subjected to context feature extraction to obtain the first intermediate output, and the first intermediate output is sequentially subjected to 1*1 convolution operation and up-sampling processing to obtain the second intermediate output; the first intermediate output is further subjected to three consecutive up-sampling operations and context feature extraction, wherein the input of the first up-sampling operation and the context feature extraction is the first intermediate output and the third spatial context feature, the result of the first up-sampling operation and the context feature extraction and the second spatial context feature are taken as the input of the second up-sampling operation and the context feature extraction, the result of the second up-sampling operation and the context feature extraction and the first spatial context feature are taken as the input of the third up-sampling operation and the context feature extraction, and the result of the third up-sampling operation and the context feature extraction is taken as the third intermediate output; the third intermediate output and the second intermediate output are subjected to matrix element-by-element addition and average value operation, and subjected to softmax activation function processing to obtain the final cloud detection output result.

[0014] The above method according to the embodiments of the present application can further have the following additional technical features:

[0015] Further, in the spatial domain context feature encoding process, the context feature extraction includes:

[0016] The original input is layer normalized to obtain a third output, and the third output is processed in two parallel branches, wherein the first branch includes sequentially performing linear layer processing and activation function processing on the third output to obtain a fourth output; the second branch includes sequentially performing linear layer processing, convolution layer processing, activation function processing and state space model processing on the third output to obtain a fifth output; the fifth output and the fourth output are merged in an element-by-element multiplication manner, and then linear layer processing is performed to obtain a mixed feature; the mixed feature and the original input are added by residual connection to obtain a spatial domain feature, and the spatial domain feature is down-sampled to obtain a spatial context feature;

[0017] The original input of the first context feature extraction is an original image, and the obtained spatial context feature is a first spatial context feature; the original input of the second context feature extraction is the first spatial context feature, and the obtained spatial context feature is a second spatial context feature; the original input of the third context feature extraction is the second spatial context feature, and the obtained spatial context feature is a third spatial context feature.

[0018] Further, the state space model processing includes a sequence expansion preprocessing stage, an S6 deep feature extraction stage and a sequence merging post-processing stage.

[0019] In the sequence expansion preprocessing stage, based on a strategy of tiling along a fixed direction, the input of the state space model is sequentially expanded along four different directions to obtain sequence data.

[0020] In the S6 deep feature extraction stage, the sequence data is discretized to obtain sequence features from four different directions.

[0021] In the sequence merging post-processing stage, the sequence features from the four different directions are summed and merged to obtain an output tensor, i.e. the fifth output.

[0022] Further, the down-sampling operation includes sequentially performing block extraction, channel concatenation, flattening, normalization and channel dimension reduction operations on the spatial domain feature to obtain the spatial context feature.

[0023] Further, in the frequency domain structure feature enhancement encoding process, the structure feature extraction includes first structure feature extraction and second structure feature extraction, and the processes of the first structure feature extraction and the second structure feature extraction are consistent, including:

[0024] The structural feature input is subjected to 3*3 Gaussian filtering to obtain a smoothing processing output; the smoothing processing output is subjected to discrete Fourier transform to convert the structural feature input from a spatial domain to a frequency domain to obtain a first frequency domain output; the first frequency domain output is subjected to Butterworth high-pass filtering processing to obtain a second frequency domain output; the second frequency domain output is subjected to global average pooling processing and softmax activation function processing to obtain a third frequency domain output; the second frequency domain output and the third frequency domain output are subjected to matrix multiplication and then Laplace filtering processing to obtain a fourth frequency domain output; the fourth frequency domain output is subjected to inverse discrete Fourier transform processing and then matrix element-by-matrix element addition with the smoothing processing output to obtain a structural feature output;

[0025] The input of the first structural feature extraction is a result of 1*1 convolution operation on the original image, and the output is a result before the first multi-path down-sampling operation on the first structural feature; the input of the second structural feature extraction is a result of fusing the first structural feature and the first scaling feature, and the output is a result before the second multi-path down-sampling operation on the second structural feature.

[0026] Further, in the frequency domain structural feature strengthening encoding process, the multi-path down-sampling operation includes the first multi-path down-sampling operation and the second multi-path down-sampling operation, the processes of the first multi-path down-sampling operation and the second multi-path down-sampling operation are consistent, and include: performing average pooling operation on the multi-path down-sampling input to obtain a preliminary down-sampling result, performing channel separation on the preliminary down-sampling result to obtain a first down-sampling branch input, a second down-sampling branch input and a third down-sampling branch input; performing convolution processing on the first down-sampling branch input to obtain a first down-sampling branch output; performing average pooling operation on the second down-sampling branch input to obtain a second down-sampling branch output; performing maximum pooling operation and convolution processing on the third down-sampling branch input to obtain a third down-sampling branch output; performing concatenation operation on the first down-sampling branch output, the second down-sampling branch output and the third down-sampling branch output to obtain a multi-path down-sampling output;

[0027] The input of the first multi-path down-sampling operation is the output of the first structural feature extraction, and the output of the first multi-path down-sampling operation is the first structural feature; the input of the second multi-path down-sampling operation is the output of the second structural feature extraction, and the output of the second multi-path down-sampling operation is the second structural feature.

[0028] Compared with the prior art, the remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion provided by the embodiments of the present application has the following beneficial technical effects:

[0029] The embodiments of the present application can capture the spatial domain information of different scales and ranges in the image by gradually extracting different levels of spatial domain context features (first, second and third spatial context features) through three times of context feature extraction and down-sampling operations, which helps to understand the spatial distribution and context relationship of the cloud in the image; the frequency domain structure reinforced features are obtained by a series of frequency domain processing operations on the original image, including convolution, structure feature extraction, scaling, fusion, etc., which can mine the performance of the structure information of the cloud in the frequency domain, and complement the spatial domain features, providing more rich feature information for cloud detection; the depth fusion of spatial and frequency domain features is realized by operations such as twin self-attention fusion and multi-layer perception processing of the frequency domain structure reinforced features and the third spatial context features, which fully utilizes the feature advantages of different domains and improves the accuracy of cloud detection.

[0030] In the context feature extraction stage of the spatial domain context feature extraction, two parallel processing branches are adopted, one branch performs linear layer and activation function processing, and the other branch performs linear layer, convolution layer, activation function and state space model processing, and finally the outputs of the two branches are merged, which can extract spatial domain features from different angles and enhance the expression ability of the features; the state space model processing includes sequence expansion preprocessing, S6 deep feature extraction and sequence merging post-processing stages, which can mine the spatial sequence features of the cloud in the image through sequence expansion, discretization operation and sequence merging of the input, and further enrich the representation of the spatial domain features; in the frequency domain structure feature extraction process, a series of operations such as Gaussian filtering, discrete Fourier transform, Butterworth high-pass filtering, global average pooling, softmax activation function, matrix multiplication, Laplace filtering and inverse discrete Fourier transform are performed on the structure feature input, which can accurately extract the structure features of the cloud in the frequency domain and perform reinforcement processing to make the features more discriminative.

[0031] In the multi-path down-sampling operation in the frequency domain structure feature reinforcement encoding process, average pooling, channel separation, convolution and pooling operations of different branches are performed on the multi-path down-sampling input, and finally the outputs of each branch are concatenated, which can preserve the feature information of different scales and types in the down-sampling process, avoid information loss, and help to improve the robustness of cloud detection; in the feature fusion and output processing process, the features of different stages are gradually fused and refined through multiple matrix addition, normalization, softmax activation function and other operations, for example, the twin self-attention output and the multi-layer perception output are added to obtain the collaborative fusion features, and then the collaborative fusion features are subjected to context feature extraction, up-sampling and matrix element-by-element addition to obtain the cloud detection output result, which can make the output result more accurate and reliable. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A flowchart of a cloud shadow cross-domain feature fusion based remote sensing image cloud detection method of an embodiment of the present application is shown.

[0033] Figure 2 A structural diagram of a context feature extraction module of the cloud shadow cross-domain feature fusion based remote sensing image cloud detection method of an embodiment of the present application is shown.

[0034] Figure 3 A diagram of sequence expansion preprocessing and sequence merging postprocessing of the cloud shadow cross-domain feature fusion based remote sensing image cloud detection method of an embodiment of the present application is shown.

[0035] Figure 4 A diagram of Patch Merging of the cloud shadow cross-domain feature fusion based remote sensing image cloud detection method of an embodiment of the present application is shown.

[0036] Figure 5 A structural diagram of a structural feature extraction module of the cloud shadow cross-domain feature fusion based remote sensing image cloud detection method of an embodiment of the present application is shown.

[0037] Figure 6 A diagram of multi-path down-sampling of the cloud shadow cross-domain feature fusion based remote sensing image cloud detection method of an embodiment of the present application is shown.

[0038] Figure 7 A structural diagram of a cooperative fusion module of the cloud shadow cross-domain feature fusion based remote sensing image cloud detection method of an embodiment of the present application is shown.

[0039] Figure 8 A structural diagram of a decoding up-sampling unit of the cloud shadow cross-domain feature fusion based remote sensing image cloud detection method of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] In order to make the above objectives, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the purpose of description, only parts related to the present application are shown in the drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0041] The terms "comprises", "comprising", "includes", "including", "has", "having" and their conjugates, as used herein, are intended to cover the situation where individual elements have been described apart from those included in the process, method, system, product or apparatus that has been described in the specification. The intention is to allow for "additional" but "optional" elements, steps, or components that might be added to or performing in conjunction with the described process, method, system, product or apparatus.

[0042] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, or to a single alternative embodiment. It is explicitly contemplated that embodiments described herein can be combined with each other in their individual aspects.

[0043] As shown in Figure 1 The embodiment of the present application provides a remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion, which comprises the following steps:

[0044] In step 101, an original image is obtained, and spatial context feature encoding and frequency domain structure feature enhancement encoding are performed on the original image respectively.

[0045] The embodiment of the present application encodes the original image by a spatial context feature encoder (SCF-Encoder), and the basic unit of the spatial context feature encoder is composed of a context feature extraction module (CFEM). The module processes the original image through a series of linear transformations and discretization processes to enhance the performance of image spatial encoding. The context feature extraction module first regards its input as a state space, in which each pixel point or feature region represents a state. These states are updated with the iteration process to capture the dynamic changes in the original image, and then a learnable convolution kernel is used as a state transition matrix of a state space model (SSM) to transfer information between different feature channels, simulate the interaction between features, and finally the embodiment of the present application defines an observation model which maps the current state to its output. After the context feature extraction module, a patch merging operation is performed for down-sampling. The formula is as follows:

[0046]

[0047] Wherein, represents the original image, and the spatial context feature encoding is performed by representing the original image; representing the context feature extraction; representing the patch merging operation; representing the output of the three consecutive CFEMs and patch merging top-down operations in the SCF-Encoder, representing the first spatial context feature, the second spatial context feature and the third spatial context feature respectively.

[0048] As Figure 2 shown, in order to enable better feature extraction in the spatial domain, the embodiment of the application adopts a visual-based SSM structure for CFEM, which can capture long sequence context information in the SCF-Encoder, and use sequence expansion preprocessing and sequence merging postprocessing to traverse the spatial domain and convert the acyclic visual image into an ordered patch sequence, enhancing the applicability of CFEM in feature extraction in the spatial domain. The combination of the above two processes enhances the long-distance spatial feature modeling capability in the remote sensing image deep learning algorithm.

[0049] Specifically, within the CFEM, the input of the module is first subjected to layer normalization processing to obtain a third output, and then the third output is distributed to two parallel processing branches. The first branch is relatively simple, only containing a linear layer and subsequent activation function processing to obtain a fourth output. In contrast, the processing flow of the second branch is more complex, and after sequentially passing through a linear layer, a convolution layer and an activation function, a fifth output is obtained, which is finally input into the SSM unit for further processing.

[0050] After completing their respective processing flows, the outputs of the two branches are merged by element-wise multiplication, and an additional linear layer is used to mix the merged features to obtain mixed features. In order to further improve the expression ability of the model, the mixed features are then added to the original output through residual connection, thereby obtaining the spatial features. This design is used to fuse the feature information of different branches, and also enhances the gradient propagation ability of the model through residual connection, which helps to improve the training effect and generalization performance of the model.

[0051] The SSM unit of the embodiment of the application integrates three key components: a sequence expansion preprocessing stage, an S6 deep feature extraction stage and a sequence merging postprocessing stage, which together constitute an image processing process.

[0052] As Figure 3As shown, the sequence expansion preprocessing stage expands the input image along four different directions by a tiling strategy along a fixed direction, specifically, from the top-left corner to the bottom-right corner, from the bottom-right corner to the top-left corner, from the top-right corner to the bottom-left corner, and from the bottom-left corner to the top-right corner. This sequence expansion not only greatly expands the field of view of the image, enabling the model to capture more extensive spatial context information, but also ensures that each corner and detail in the image is fully focused and utilized through multi-directional sequential scanning. This preprocessing step lays a solid foundation for subsequent deep feature extraction, enabling the model to more accurately understand complex structure and texture information in the image. Similarly, the sequence merging post-processing stage is the process of restoring the image to its original size along a fixed direction.

[0053] Subsequently, the embodiment of the present application constructs a S6 deep feature extraction module to perform a S6 deep feature extraction stage, which constitutes the core element of the SSM and undertakes the key task of extracting deep spatial features from the sequence data after sequence expansion preprocessing. Since traditional convolutional neural networks are limited by local receptive fields, they are difficult to effectively capture long-range dependency information, which in turn limits the segmentation accuracy. Therefore, the module introduces a visual spatial state technique to capture context information in the feature map, and implements step-by-step abstraction processing and transformation on the input sequence to build long-range dependency relationships between pixels in the feature map. The design of this module deeply understands the characteristics of multi-directional input data, ensuring that information from different directions can be thoroughly scanned and extracted. This design strategy enriches the model's ability to capture a wide range of image features, improving the accuracy and efficiency of feature extraction, laying the foundation for the subsequent image segmentation fusion stage and providing feature representation support. When processing the state space in a computer system, the input sequence needs to be discretized, as shown in the following formula:

[0054]

[0055] wherein, represents the input feature map, with size parameters [B, L, D], representing batch size, token length after blocking, and feature dimension, respectively; represents a fully connected linear projection layer; represents applying an exponential operation to all elements in the matrix; represents a learnable parameter tensor.

[0056] The state of the S6 deep feature extraction module is represented by the following formula:

[0057]

[0058] wherein, denotes the hidden state at time t; denotes the hidden state at time t-1; denotes the input at time t.

[0059] The output of each time step is calculated using the hidden state at time t and the input at time t, and finally, the output tensor is collected using The operation collects the output vectors of all time steps from 1 to L into the final output tensor, as shown in the following formula:

[0060]

[0061] wherein, denotes the learnable parameter tensor; denotes the output of each time step ; denotes the final output tensor.

[0062] Finally, the sequence merging post-processing stage is a summation operation that combines sequence features from different directions. The purpose of this step is to integrate and fuse feature information from each direction, forming a more comprehensive and unified feature representation. Through the sequence merging mechanism, SSM can effectively utilize the feature information obtained from multi-direction scanning, further improving the model's ability to grasp global image features. At the same time, since the summation operation is simple and efficient, it can effectively integrate and utilize feature information without increasing excessive computational burden.

[0063] In summary, the operation processing of the entire SSM unit can be represented as the following formula:

[0064]

[0065] wherein, denotes the sequence merging post-processing operation; denotes the S6 deep feature extraction operation; denotes the sequence expansion preprocessing operation.

[0066] Patch Merging is the downsampling operation unit of SCF-Encoder, as shown in Figure 4As shown, the operation processes the output after CFEM processing. First, the patches are numbered and grouped in a specific order, with each group containing 4 adjacent small patches, numbered 1 to 4 in turn. Then, patches with the same number are extracted and integrated into the corresponding feature map respectively. Next, the patches within each group are spliced along a certain dimension to form a long vector. This step effectively converts the spatial adjacent information into a vector form, facilitating subsequent processing. In order to map the spliced long vector to a new feature space and adjust its channel number at the same time, channel dimension reduction is needed. The PatchMerging operation uses a fully connected layer. This fully connected layer realizes the conversion of dimensions and the recombination of features by performing linear transformation on the long vector. In this way, not only the spatial relationship between the original patches is preserved, but also more feature representations are introduced through the nonlinear transformation ability of the fully connected layer. Patch Merging operation reduces the spatial dimension of the feature map, reduces the computational complexity of the model, and improves the computational efficiency. At the same time, compared with the traditional pooling operation, PatchMerging avoids direct information loss, but preserves more detailed information by splicing adjacent patches.

[0067] Further, the embodiment of the present application encodes the original image by a frequency domain structural feature encoder (FSF-Encoder). The FSF-Encoder is mainly responsible for feature extraction of image information in the frequency domain. Through the cooperation of the internal structural feature extraction module (SFEM) for discrete Fourier transform and frequency domain filtering, efficient and accurate structural feature extraction and fusion are realized. In the discrete Fourier transform, the information in the frequency domain feature map is decomposed into low-frequency components and high-frequency components. The low-frequency components usually correspond to the main components in the image, i.e. large similar areas; while the high-frequency components more reflect the regions with large amplitude changes in the image, such as details and edge information. In the frequency domain filtering, the frequency domain feature map is processed by a filter to retain high-frequency features and fuse with the feature map of the SCF-Encoder in the subsequent steps, further highlighting the important details and edge information in the image, and strengthening the structural features of the cloud layer and cloud shadow.

[0068] To meet the input requirements of the extraction module, the embodiments of the present application first use a 1x1 convolution to process the channel dimension of the original image before performing SFEM processing. This step helps to reduce the computational complexity of subsequent processing, improve the efficiency of feature extraction, and meet the input requirements of subsequent SFEM.

[0069] The frequency domain feature extraction operation is followed by a downsampling operation. The embodiments of the present application use a multi-path downsampling strategy in the FSF-Encoder part. After this downsampling operation, the image data is gradually compressed while retaining its key information. Finally, after the frequency domain feature extraction and downsampling operations at each layer, the above two feature maps from different scales are fused, and a Resize operation based on equidistant sampling is used to reduce the size of the feature maps from the SFEM to avoid differences in the size of the feature maps. Finally, a 1x1 convolution is used again to expand the channels to meet the input requirements of the subsequent operations.

[0070] The embodiments of the present application represent the operations of the FSF-Encoder by formulas, which are as follows:

[0071]

[0072] wherein, represents the original image; represents a 1x1 convolution; represents the output after the 1x1 convolution on the original image; represents a structural feature extraction operation; represents a multi-path downsampling operation; represents an equidistant sampling scaling operation; represents a first scaled feature; represents a first structural feature; represents a feature after the fusion of the first scaled feature and the first structural feature; represents a second scaled feature; represents a second structural feature; represents a frequency domain structural reinforcement feature.

[0073] Feature extraction, as an important technique in the field of deep learning, its core lies in dynamically generating masks according to the data distribution of the network internal feature maps. This process aims to enhance the expression of key information by reweighting the feature maps, thereby improving the overall performance of the model. In the initial stage of training, the features extracted by the network are often filled with low-relevant semantics caused by the irregularity of the original input data and the immaturity of the feature mapping, which has an impact on the detection accuracy of the model. If these low-relevant semantic information is used to extract key features, it may not only mislead the model to focus on non-key information, but also further exacerbate the instability of the model performance. Moreover, conventional feature extraction techniques may not be able to effectively capture high-frequency details such as edge information in images, which are crucial for image segmentation.

[0074] In view of the above limitations, the embodiments of the present application propose a structural feature extraction module SFEM. This module uses a parameter-free method based on discrete Fourier transform, Gaussian filter, Butterworth high-pass filter and Laplacian operator to complete and optimize the features extracted by the shallow layer of the model. In contrast, the frequency domain feature extraction technique can reduce feature redundancy and improve the efficiency and accuracy of feature extraction by selectively extracting high-frequency and low-frequency components, achieving improved feature extraction capability of the model without additional training burden. The core of SFEM is to implement a fine processing strategy combining parameter-free transformation and kernel filter to achieve in-depth optimization of feature representation. This strategy effectively avoids the performance bottleneck caused by the inclusion of low-relevant semantic information in traditional methods, and provides a more reliable and efficient structural feature extraction path for cloud detection models in cloud detection tasks by extracting high-frequency information and Laplacian filtering.

[0075] As shown in Figure 5 , SFEM first uses a parameter-free discrete Fourier transform technique to convert its input from spatial domain to frequency domain. This step not only adds new dimensions to the subsequent processing of features, but also clearly distinguishes between low-frequency global structure and high-frequency detail information in the features. Based on the frequency domain information, Butterworth kernel filter and Laplacian filter are introduced to achieve precise filtering by regulating the frequency response. This processing procedure not only ensures the complete preservation of core structural information in the original image, but also reduces the noise introduced by imperfect input data or feature mapping, thereby improving the clarity and detail richness of the original image. After processing by SFEM, the influence of invalid semantic information in the original image on the model performance is greatly reduced, and at the same time, the detail information is highlighted. In particular, by filtering out low-frequency information, SFEM significantly enhances the edge details of the cloud layer and cloud shadow in the original image, enabling accurate differentiation between cloud layer and cloud shadow in complex scenes where cloud layer and cloud shadow are mixed.

[0076] Specifically, in the operation in the frequency domain, first, a 3x3 Gaussian filter is performed on the original image to obtain a smoothing output. The SFEM converts the smoothing output from the spatial domain to the frequency domain through a discrete Fourier transform. In this process, the information is decomposed into low-frequency components and high-frequency components. The low-frequency components usually correspond to the global structure and main features in the original image, while the high-frequency components more reflect the details and edge information of the original image. By filtering the smoothing output using different types of filters, the SFEM can selectively enhance or suppress information in a specific frequency range, thereby achieving further optimization.

[0077] wherein the formula of the discrete Fourier transform is as follows:

[0078]

[0079] wherein, SF and SW represent the height and width of the smoothing output, respectively; V represents a vertical frequency variable in the frequency domain, and the value range is 0~H-1; H represents a horizontal frequency variable in the frequency domain, and the value range is 0~W-1; j represents an imaginary unit; is a complex exponential function, representing a rotation factor; F1 represents the first frequency domain output. It should be noted that only the real number item after the discrete Fourier transform is used in the actual application process.

[0080] Then, through a Butterworth high-pass filter (BHF), low-frequency domain components are removed, and high-frequency information is passed to preliminarily extract structural features containing edge information, and the formula is as follows:

[0081]

[0082] wherein, Fcut represents the cutoff frequency of the BHF; N represents the order of the BHF; F represents the point in the frequency domain to the center; F represents the frequency component of the BHF at the point , that is, the second frequency domain output.

[0083] After passing through the Butterworth high-pass filter, the second frequency domain output is sequentially subjected to global average pooling and Softmax function processing to obtain a third frequency domain output. The second frequency domain output and the third frequency domain output are fused by multiplication operation according to corresponding channels, and the fused output is processed by a Laplace filter (LF) inside the SFEM to extract and enhance edge information, to obtain a fourth frequency domain output.

[0084] The Laplace filter is good at capturing and enhancing regions with sharp changes in gray value in an image by virtue of its ability to calculate the second derivative of the image. However, this feature may also have side effects, such as the edges appearing gray on a dark background or discontinuities in some regions of the image. In order to reconcile this sharpening process and potential defects, the usual strategy is to synthesize the image after Laplace processing with the original image.

[0085] Therefore, an embodiment of the present application also designs a feature fusion step in the internal processing flow of the SFEM. This step deeply fuses the feature map (smooth processing output) without inputting the LF with the feature map (fourth frequency domain output) after the LF fine processing. This fusion strategy not only retains the basic feature information of the original image (smooth processing output), but also introduces sharpening details in the Laplace image (fourth frequency domain output). These detail information is particularly important for the depiction of the edge region of the cloud layer and the cloud shadow. Specifically, the high-frequency information contained in the sharpening details effectively highlights the structural features of the cloud layer and the cloud shadow, i.e., the edge information at the boundary of the cloud layer and the cloud shadow, providing high-relevance structural feature information for subsequent steps. The formula of the frequency domain Laplace operator is as follows:

[0086]

[0087] represents the vertical frequency variable in the frequency domain, and the value range is 0~H-1; represents the horizontal frequency variable in the frequency domain, and the value range is 0~W-1; represents the frequency component of the Laplace filter at point .

[0088] Finally, the fourth frequency domain output is subjected to inverse discrete Fourier transform processing and added to the smooth processing output element by element to obtain a structural feature output. The formula is as follows:

[0089]

[0090] wherein, represents the original image; represents Gaussian filter processing; denotes a smoothing processing output; denotes a discrete Fourier transform processing; denotes a Butterworth high-pass filtering processing; denotes a second frequency domain output; denotes a global average pooling operation; denotes an activation function processing; denotes a third frequency domain output; denotes a matrix multiplication; denotes a Laplacian filtering processing; denotes an output after the Laplacian filtering processing; denotes an inverse discrete Fourier transform processing; denotes a matrix element-wise addition; denotes a structural feature extraction output.

[0091] As a parameter-free component, SFEM can be embedded into the FSF-Encoder as a basic extraction unit of frequency domain features, so that the entire model can improve the accuracy and robustness of structural feature extraction at a minimal cost. This design not only simplifies the training process of the model, but also significantly improves the generalization ability of the model when dealing with cloud segmentation and cloud shadow segmentation in cloud detection.

[0092] As shown in Figure 6 , the embodiment of the present application provides a multi-way down sampling strategy (Multi-way down, MW-Down), which aims to reduce the spatial resolution of feature maps in a diversified down sampling path manner, so that the model can capture the features of the image in a high-level receptive field and a diversified channel path.

[0093] Firstly, the strategy performs preliminary down sampling on the input feature map through average pooling operation, aiming to reduce the size of the feature map and thus reduce the computational load of subsequent processing. Then, MW-Down separates the channels of the feature map and sends them into three parallel down sampling operations. This parallel processing method extracts diversified feature information through different paths, enhancing the adaptability of the model to complex scenes, especially the scenes where clouds and cloud shadows are mixed. Finally, the feature maps after parallel down sampling are concatenated, fusing the information from different paths. This step helps to improve the recognition accuracy and collect key features of clouds and cloud shadows from the full channel field. The model can more accurately distinguish clouds and cloud shadows, thereby improving the overall performance of cloud detection.

[0094] Specifically, the embodiment of the present application first performs an average pooling operation on the input feature map to obtain a preliminary downsampling result, in which the number of channels remains unchanged. The channels are then separated to obtain the first downsampling branch input, the second downsampling branch input, and the third downsampling branch input, which are respectively fed into three parallel downsampling operations. For a feature map x input to a multi-channel downsampling operation, the size of the feature map after the channel separation operation is (C / 3, H, W). The first parallel downsampling operation is a convolutional layer with an input channel number of C / 3, an output channel number of C / 3, a convolution kernel of 3, a stride of n, and a padding of 1. The output size after convolution is (C / 3, H / n, W / n); the second branch only uses the average pooling operation with a stride of n; the third parallel downsampling operation goes through the maximum pooling and convolution layers in sequence, where the pooling layer has a stride of n, the convolution layer parameters are set to input channels of C / 3, output channels of C / 3, a convolution kernel of 1, a stride of 1, and a padding of 1. The output after pooling and convolution is (C / 3, H / n, W / n). Finally, the outputs from the three different downsampling operations are cascaded to obtain the multi-channel downsampled output. The formula is as follows:

[0095]

[0096] in, represents the average pooling operation; Indicates channel separation operation; represent the first down-sampling branch input, the second down-sampling branch input, and the third down-sampling branch input respectively; Respectively represent The output of the corresponding three parallel branches, and denote the convolution operations of the first down-sampling branch and the third down-sampling branch, respectively. represents the maximum pooling operation, represents the cascade operation of the channel dimension, Indicates multi-channel down-sampled output.

[0097] Step 102: After normalizing the frequency domain structure enhancement features and the third space context features respectively, perform a twin self-attention fusion operation to obtain a first output, perform matrix addition on the first output and the third space context features to obtain a twin self-attention output; after normalizing the twin self-attention output, perform multi-layer perceptron processing to obtain a second output; perform matrix addition on the second output and the twin self-attention output to obtain a collaborative fusion feature.

[0098] The embodiment of the application performs a decoding operation by constructing a cross-domain feature fusion decoder (CDF-Decoder), which generates a corresponding mask Mask according to the outputs of the SCF-Encoder and the FSF-Encoder. The mask is a two-dimensional matrix with the same size as the input image, and each element in the matrix represents whether the pixel at the corresponding position belongs to the target region, thereby realizing accurate segmentation of the target region in the image.

[0099] The CDF-Decoder internally contains a cooperative fusion module (CFM) for comprehensively and efficiently fusing feature information from the SCF-Encoder and the FSF-Encoder.

[0100] The design of this module takes into account the complementarity of frequency domain features and spatial domain features in cloud detection tasks, and introduces a self-attention mechanism based on a twin network. The self-attention mechanism can adaptively adjust the attention focus according to the context information of the input features, improving the model's adaptability to input changes. The twin network architecture maps features from different domains to the same feature space, preserving the intrinsic relationship between features while eliminating the specificity between different domains, achieving semantic fusion across branches.

[0101] As the fused features are passed on, the CDF-Decoder further guides these information to the subsequent processing structure. At this stage, the CFM plays a crucial role. It merges feature maps from different domains through normalization and twin self-attention, which not only preserves the multi-level information in the image, but also promotes the interaction and fusion of cross-domain features, providing more comprehensive feature support for the subsequent decoding part.

[0102] Specifically, as shown in Figure 7 The CFM module receives the output of the third layer of the SCF-Encoder (third spatial context feature) and the output of the entire FSF-Encoder (frequency domain structure reinforced feature) as input. These two sets of features represent different representations of the image in the frequency domain and the spatial domain, respectively, and each contains rich information for the cloud detection task.

[0103] In the first stage, in order to ensure that these features can be fully utilized in the subsequent processing process, the embodiment of the application first performs layer normalization on them respectively to eliminate the scale difference between different features and improve the stability of the model.

[0104] Subsequently, the processed features are sent into the Siam-SA mechanism. Siam-SA is a specially designed self-attention mechanism, the core of which is that it is composed of two self-attention units that are completely identical in structure and share weights. This design enables the features of the two branches to be processed under the same attention mechanism, thereby ensuring the consistency and effectiveness of feature fusion. Under the action of Siam-SA, the frequency domain features and the spatial domain features respectively undergo an attention calculation process, establishing the dependency relationship between the respective internal elements.

[0105] In the following second stage, in order to effectively fuse the features of the two branches, the output (first output) of Siam-SA is subjected to matrix addition operation by the embodiments of the present application. This step realizes the direct addition of features (matrix addition of the first output and the third spatial context feature), so that the frequency domain features and the spatial domain features can learn from each other and enhance each other in the fusion process, thereby generating more comprehensive and expressive feature representations.

[0106] On the fused features (twin self-attention output), the embodiments of the present application again perform layer normalization processing to further eliminate noise and unstable factors that may be introduced in the fusion process. Subsequently, these features are sent into a multi-layer perceptron to extract higher-level abstract features, obtaining a second output.

[0107] Finally, the second output and the twin self-attention output are subjected to matrix addition to obtain collaborative fusion features. The formula is as follows:

[0108]

[0109] wherein, represents the third spatial context feature, represents the frequency domain structure reinforced feature, represents the normalization operation, represents the fusion operation of the twin self-attention, represents the first output, represents the matrix addition operation, represents the multi-layer perceptron of the second stage, represents the twin self-attention output, represents the collaborative fusion feature.

[0110] In step 103, context feature extraction is performed on the cooperatively fused features to obtain a first intermediate output, and the first intermediate output is sequentially subjected to a 1x1 convolution operation and an upsampling process to obtain a second intermediate output; the first intermediate output is subjected to three consecutive upsampling operations and context feature extractions, wherein the input of the first upsampling operation and context feature extraction is the first intermediate output and the third spatial context feature, the result of the first upsampling operation and context feature extraction and the second spatial context feature serve as the input of the second upsampling operation and context feature extraction, the result of the second upsampling operation and context feature extraction and the first spatial context feature serve as the input of the third upsampling operation and context feature extraction, and the result of the third upsampling operation and context feature extraction serves as a third intermediate output; a matrix element-by-element addition and average value calculation operation is performed on the third intermediate output and the second intermediate output, and a softmax activation function process is performed to obtain a final cloud detection output result.

[0111] After the CFM module fuses features from different sources, it will enter the decoding upsampling module DUM. As shown in Figure 8 The cooperatively fused features first enter the independent CFEM to obtain a first intermediate output, and then the first intermediate output is subjected to channel adjustment and size restoration through a 1x1 convolution operation and upsampling to obtain a second intermediate output. These convolution operations enhance the local perception ability of the features, and through nonlinear transformations such as activation functions, nonlinear factors are introduced, so that the model can capture complex patterns and subtle changes in the image.

[0112] At the same time, the first intermediate output is sequentially fed into three CFEMs symmetrical to the SCF-Encoder through the CFEM. These modules make full use of the fused feature information and gradually decode the high-level feature maps back to a two-dimensional matrix mask with the same size as the input image through detailed decoding processing. The CFEM structure inside the CDF-Decoder is the same as that of the SCF-Encoder, which ensures that the model can accurately identify and distinguish cloud layers, shadows, and ground areas in the image.

[0113] The input of the first upsampling operation and context feature extraction is the first intermediate output and the third spatial context feature, the result of the first upsampling operation and context feature extraction and the second spatial context feature serve as the input of the second upsampling operation and context feature extraction, the result of the second upsampling operation and context feature extraction and the first spatial context feature serve as the input of the third upsampling operation and context feature extraction, and the result of the third upsampling operation and context feature extraction serves as a third intermediate output.

[0114] Finally, after the decoding process of the CDF-Decoder pre-operation, the matrix element-wise addition operation with the up-sampled image is performed again, and the mean value derived from the above matrix addition is finally generated in the mask matrix.

[0115] The third intermediate output and the second intermediate output are subjected to matrix element-wise addition and average operation, and subjected to softmax activation function processing to obtain the final cloud detection output result. The generated cloud detection mask identifies whether each pixel point in the image belongs to the target region, thereby realizing the segmentation of the cloud layer and cloud shadow of the remote sensing image. The formula is as follows:

[0116]

[0117] wherein, represents a collaborative fusion operation; represents matrix element-wise addition, represents a vector representation after sequentially passing through the cascade, convolution and the bottommost context feature extraction module operation, represents a feature map expansion operation, after which the width and height of the feature map will be increased to 2 times the original size, and the number of channels will be changed to 1 / 2 of the original number of channels, represents the feature map output after three CFEM bottom-up operations, represents an up-sampling operation, represents the mean operation after matrix element-wise addition of any number of matrices, represents the final cloud detection output result.

[0118] In addition, in the loss function design of the embodiments of the present application, a two-stage loss function design is adopted, and through two-stage weighted fusion of Focal Loss and Dice Loss, parameter fine-tuning at the end of model training is realized, and a higher-precision cloud detection effect is achieved. The formula is as follows:

[0119]

[0120] wherein, respectively represent the real label and the model inference value of the sample, is a weight hyperparameter of the two loss functions, is the iteration number of the current model, is a stage hyperparameter.

[0121] In the early stage of training, the model will use Dice Loss as the loss function in the process of back propagation and gradient update. This loss function is proposed in V-Net and aims to deal with the problem of uneven sample distribution in semantic segmentation tasks. This goal is also applicable to the cloud detection task of uneven distribution of cloud layers, cloud shadows and background parts. There are a considerable number of samples in the WFV dataset that do not contain cloud layers and cloud shadows inside, and all are background parts. Therefore, Dice Loss can be used as the loss function, which is derived from Dice Coefficient and is a measure function for evaluating the similarity of data samples, with a value range of [0, 1]. The formula is as follows:

[0122]

[0123] wherein, respectively represent the real label image and the model inference image of the sample, and respectively represent the number of positive sample pixels in the real label and the model inference value of the sample, is the intersection operation of the sample.

[0124] Therefore, Dice Coefficient is intuitively calculated as the similarity of X and Y, but in essence it is F1-score, which is the harmonic mean of Precision and Recall. Using Dice Loss as the loss function in the early stage of model training can directly optimize the F1-score indicator of the model, thereby optimizing the performance of cloud detection.

[0125] By deeply analyzing the calculation process of Dice Loss, it can be found that the loss function value of a pixel point is not only dependent on the comparison between the Ground Truth and the model inference value of the pixel point, but also closely related to its surrounding pixel points. This feature enables Dice Loss to consider the local context information of the current pixel point when evaluating the classification accuracy of the pixel point, thereby more comprehensively measuring the performance of the model in the segmentation task.

[0126] However, in the model training process, Dice Loss may cause a large fluctuation in the loss value due to the small number of positive samples and the inference error of positive samples, resulting in a dramatic change in the gradient. Focal Loss can smooth the loss of difficult and easy samples and to some extent alleviate the problem of unstable training. Therefore, the embodiments of the present application will use a loss function that combines Focal Loss and Dice Loss in the later stage of training to avoid the independent decision of the back propagation and gradient update process of the model by a single loss function. The formula is as follows:

[0127]

[0128] in, Represent the true label graph and model reasoning graph of the samples respectively, is the label value of the current sample, is the model’s predicted probability for the target class; is a balancing factor used to adjust the impact between positive and negative samples. is the focus factor, which adjusts the model's attention to easy-to-classify samples and difficult-to-classify samples. It is a One-Hot encoding operation.

[0129] By analyzing the calculation mechanism of Focal Loss in depth, it is not difficult to find that when facing a data set with uneven category distribution, Focal Loss introduces a balance factor , assigning a lower weight to the categories with a high proportion. This strategy effectively weakens the dominance of these categories in the loss function, allowing the model to pay more attention to the categories with a low proportion and improve their supervision contribution. On the other hand, the focus factor in Focal Loss When the predicted probability of a sample approaches 1, the model is very confident in the classification of the sample. This design mechanism reduces the impact of easy-to-classify samples on the overall loss, allowing the model to pay more attention to those difficult-to-classify samples. In the experiment, the behavior of the model can be further refined by flexibly adjusting the value of the focus factor. A larger focus factor This means that the model will focus more on samples that are more difficult to classify, thereby improving the model's generalization ability in complex scenarios.

[0130] It should be noted that, in the present application, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that processes, methods, articles, or apparatuses that comprise a list of elements are not limited to those elements, but can also include other elements not expressly listed, or can also include elements inherent in such processes, methods, articles, or apparatuses. Without further limitation, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element. In addition, it should be pointed out that the scope of the methods and apparatuses in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but can also include performing functions in a substantially simultaneous manner or in reverse order, for example, the described methods can be performed in an order different from that described, and various steps can also be added, omitted, or combined. In addition, features described with reference to certain examples can be combined in other examples.

[0131] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the specific embodiments described above, which are merely illustrative and not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. A remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion, characterized by: The method comprises: Acquire an original image, and perform spatial domain context feature encoding and frequency domain structural feature enhancement encoding on the original image; In the spatial context feature encoding process, the original image is subjected to three consecutive context feature extraction and downsampling operations, and the result of the previous context feature extraction and downsampling operation is used as the input of the next context feature extraction and downsampling operation to obtain a first spatial context feature, a second spatial context feature, and a third spatial context feature in sequence; In the frequency domain structural feature enhancement coding process, the original image is sequentially subjected to a 1×1 convolution operation, a first structural feature extraction, and a first multi-channel downsampling operation to obtain a first structural feature; a scaling operation based on equal-interval sampling is performed on the output of the first structural feature extraction stage to obtain a first scaling feature; the first structural feature and the first scaling feature are fused, and then a second structural feature extraction and a second multi-channel downsampling operation are sequentially performed to obtain a second structural feature; the output of the second structural feature extraction stage is subjected to a scaling operation based on equal-interval sampling to obtain a second scaling feature; the second structural feature and the second scaling feature are fused, and then a 1×1 convolution operation is performed to obtain a frequency domain structural enhancement feature; the first multi-channel downsampling operation is performed The operation is consistent with the process of the second multi-channel downsampling operation, including: performing an average pooling operation on the multi-channel downsampling input to obtain a preliminary downsampling result, performing channel separation on the preliminary downsampling result to obtain a first downsampling branch input, a second downsampling branch input, and a third downsampling branch input; performing convolution processing on the first downsampling branch input to obtain a first downsampling branch output; performing an average pooling operation on the second downsampling branch input to obtain a second downsampling branch output; performing a maximum pooling operation and convolution processing on the third downsampling branch input to obtain a third downsampling branch output; performing a cascade operation on the first downsampling branch output, the second downsampling branch output, and the third downsampling branch output to obtain a multi-channel downsampling output; After normalizing the frequency domain structure enhancement feature and the third spatial context feature respectively, a twin self-attention fusion operation is performed to obtain a first output, and the first output and the third spatial context feature are matrix-added to obtain a twin self-attention output; after normalizing the twin self-attention output, a multi-layer perceptron processing is performed to obtain a second output; the second output and the twin self-attention output are matrix-added to obtain a collaborative fusion feature; the twin self-attention fusion operation is composed of two self-attention units with exactly the same structure and shared weights, so that the frequency domain features and the spatial domain features undergo the attention calculation process respectively, and a dependency relationship is established between their internal elements; Context feature extraction is performed on the collaborative fusion feature to obtain a first intermediate output, and a 1×1 convolution operation and upsampling processing are performed on the first intermediate output in sequence to obtain a second intermediate output; then the first intermediate output is subjected to three consecutive upsampling operations and context feature extraction, wherein the input of the first upsampling operation and context feature extraction is the first intermediate output and the third spatial context feature, the result of the first upsampling operation and context feature extraction and the second spatial context feature are used as the input of the second upsampling operation and context feature extraction, the result of the second upsampling operation and context feature extraction and the first spatial context feature are used as the input of the third upsampling operation and context feature extraction, and the result of the third upsampling operation and context feature extraction is used as the third intermediate output; the third intermediate output and the second intermediate output are subjected to matrix element-by-element addition and then averaged, and softmax activation function processing is performed to obtain the final cloud detection output result.

2. The remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion according to claim 1, characterized in that: In the spatial domain context feature encoding process, the context feature extraction includes: The original input is subjected to layer normalization processing to obtain a third output, and the third output is subjected to two parallel processing branches, wherein the first branch includes sequentially performing linear layer processing and activation function processing on the third output to obtain a fourth output; the second branch includes sequentially performing linear layer processing, convolutional layer processing, activation function processing and state space model processing on the third output to obtain a fifth output, and the fifth output and the fourth output are merged in an element-by-element product manner and then subjected to linear layer processing to obtain a mixed feature; the mixed feature is subjected to residual connection addition with the original input Then, a spatial domain feature is obtained, and a downsampling operation is performed on the spatial domain feature to obtain a spatial context feature; the state space model processing includes a sequence expansion preprocessing stage, an S6 deep feature extraction stage, and a sequence merging post-processing stage; wherein, in the sequence expansion preprocessing stage, based on a strategy of tiling along a fixed direction, the input of the state space model is serialized and expanded along four different directions to obtain sequence data; the four different directions include from the upper left corner of the image to the lower right corner, from the lower right corner to the upper left corner, from the upper right corner to the lower left corner, and from the lower left corner to the upper right corner; In the S6 deep feature extraction stage, the sequence data is discretized to obtain sequence features from four different directions; in the sequence merging post-processing stage, the sequence features from the four different directions are summed and merged to obtain an output tensor, i.e., the fifth output; Among them, the original input of the first context feature extraction is the original image, and the obtained spatial context feature is the first spatial context feature; the original input of the second context feature extraction is the first spatial context feature, and the obtained spatial context feature is the second spatial context feature; the original input of the third context feature extraction is the second spatial context feature, and the obtained spatial context feature is the third spatial context feature.

3. The remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion according to claim 2, characterized in that: The downsampling operation includes sequentially performing block extraction, channel concatenation, flattening, normalization, and channel dimension reduction operations on the spatial domain features to obtain the spatial context features.

4. The remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion according to claim 1, characterized in that: In the frequency domain structural feature enhancement coding process, structural feature extraction includes first structural feature extraction and second structural feature extraction. The processes of the first structural feature extraction and the second structural feature extraction are consistent, including: Performing a 3×3 Gaussian filter on the structural feature input to obtain a smoothed output; performing a discrete Fourier transform on the smoothed output to convert the structural feature input from the spatial domain to the frequency domain to obtain a first frequency domain output; performing a Butterworth high-pass filter on the first frequency domain output to obtain a second frequency domain output; performing a global average pooling process and a softmax activation function process on the second frequency domain output to obtain a third frequency domain output; performing a Laplace filter process on the second frequency domain output and the third frequency domain output after matrix multiplication to obtain a fourth frequency domain output; performing an inverse discrete Fourier transform on the fourth frequency domain output and then adding the fourth frequency domain output to the smoothed output on a matrix element-by-matrix basis to obtain a structural feature output; Among them, the input of the first structural feature extraction is the result of a 1×1 convolution operation on the original image, and the output is the result before the first multi-way downsampling operation is performed on the first structural feature; the input of the second structural feature extraction is the result of fusing the first structural feature and the first scaling feature, and the output is the result before the second multi-way downsampling operation is performed on the second structural feature.

5. The remote sensing image cloud detection method based on cloud shadow cross-domain feature fusion according to claim 1 or 4, characterized in that: In the frequency domain structural feature enhancement encoding process, the multi-channel downsampling operation includes a first multi-channel downsampling operation and a second multi-channel downsampling operation, wherein the input of the first multi-channel downsampling operation is the output of the first structural feature extraction, and the output of the first multi-channel downsampling operation is the first structural feature; the input of the second multi-channel downsampling operation is the output of the second structural feature extraction, and the output of the second multi-channel downsampling operation is the second structural feature.

Images