Sea fog nowcasting method and system based on double-flow space-time Transform

Abstract

The invention discloses a sea fog nowcasting method based on double-flow space-time Transform, and belongs to the technical field of marine weather forecasting and artificial intelligence. The method comprises the following steps: preprocessing original observation data of a FY-4 satellite to obtain a standardized input tensor; constructing a physical characteristic flow based on a bright temperature difference and a multispectral characteristic flow based on a multispectral channel in parallel; performing space-time embedding and division on the double-flow features respectively; a Transform encoder is used for capturing the respective space-time dependency relationship; through a cross attention mechanism, taking physical features as query, and fusing multispectral feature information; and finally, performing up-sampling reconstruction through a layered decoder, and outputting a sea fog distribution prediction map in a future time period. According to the invention, the accuracy and reliability of sea fog nowcasting are effectively improved through deep fusion of a double-flow architecture and physical priori.

Description

Technical Field

[0001] This invention relates to the field of marine meteorological forecasting technology, specifically to a method and system for nowcasting sea fog based on a dual-stream spatiotemporal Transformer. Background Technology

[0002] Sea fog is a common low-visibility weather phenomenon in marine and coastal areas, and a key meteorological factor affecting maritime navigation safety. It causes over 2,000 maritime accidents globally each year, resulting in direct economic losses exceeding $5 billion. Sea fog forecasting not only relates to ship navigation safety but also directly impacts port operation scheduling, the selection of offshore construction windows, and operational decisions for emerging industries such as offshore wind power. Traditional sea fog forecasting primarily relies on numerical weather prediction and ensemble forecasting systems, but these are computationally time-consuming and cannot meet the real-time requirements of nowcasting. Furthermore, spatial resolution limitations prevent accurate characterization of sea fog boundary evolution. Existing deep learning methods (such as CNN and ConvLSTM) are computationally efficient but do not adequately utilize prior physical knowledge such as brightness temperature difference (BTD). They directly stack multispectral channel inputs, ignoring the uniqueness of physical characteristics like brightness temperature difference, causing crucial physical information to be drowned out by background noise.

[0003] Therefore, there is an urgent need for a technical solution that can integrate prior physical knowledge with multispectral observation data to effectively improve the accuracy of sea fog forecasts. Summary of the Invention

[0004] This invention addresses the aforementioned problems by providing a sea fog nowcasting method and system based on a dual-stream spatiotemporal Transformer. It aims to overcome the issues of poor timeliness and low accuracy in existing sea fog forecasts by proposing a dual-stream spatiotemporal Transformer-based method. This method constructs a "physical feature stream + multispectral feature stream" and utilizes a cross-attention mechanism to achieve intelligent fusion of heterogeneous features, thereby improving sea fog forecast accuracy.

[0005] This invention provides a sea fog nowcasting method based on a dual-stream spatiotemporal Transformer, characterized by the following steps: Step 1, Data preprocessing: Converting raw multispectral satellite observation data into a standardized input tensor to obtain raw satellite data; Step 2, Construction of multispectral feature stream: Extracting multispectral channel information from the raw satellite data and performing radiometric calibration to obtain a multispectral feature stream; Step 3, Construction of physical feature stream: Based on the raw satellite data, calculating multiple brightness temperature difference features using the brightness temperature difference method to construct a physical feature stream; Step 4, Spatiotemporal feature embedding: Based on the multispectral feature stream and physical feature embedding... The feature stream is divided into discrete spatiotemporal segments by 3D convolution and mapped to a high-dimensional feature space to add positional encoding, resulting in embedded features; Step 5, spatiotemporal feature encoding: The embedded features are input into a multi-layer Transformer encoder, and the spatiotemporal dependencies are captured through a self-attention mechanism to obtain physical features and multispectral features; Step 6, a dual-branch feature fusion module is designed: The complementary information of physical features and multispectral features is integrated through a cross-attention mechanism to obtain fused features; Step 7, a hierarchical decoder network is constructed: The fused features are input into the hierarchical decoder network to reconstruct a full-resolution sea fog prediction map for future time periods.

[0006] The sea fog nowcasting method based on dual-stream spatiotemporal Transformer provided by this invention may also have the following features: wherein, in step 3, the brightness temperature difference features include: a first brightness temperature difference feature BTD1, which is the difference between the brightness temperature of the 10.8μm band and the brightness temperature of the 3.75μm band; a second brightness temperature difference feature BTD2, which is the difference between the brightness temperature of the 10.8μm band and the brightness temperature of the 12.0μm band; and a third brightness temperature difference feature BTD3, which is the difference between the brightness temperature of the 3.75μm band and the brightness temperature of the 12.0μm band.

[0007] The sea fog near-term forecasting method based on dual-stream spatiotemporal Transformer provided by this invention may also have the following features: In step 4, the kernel size of the 3D convolution is (2, 16, 16) and the stride is (2, 16, 16), which converts the input tensor into 192 spatiotemporal segments; and each segment is mapped to a 256-dimensional feature space through a linear embedding layer.

[0008] The sea fog nowcasting method based on dual-stream spatiotemporal Transformer provided by this invention may also have the following features: in step 5, the multi-layer Transformer encoder contains 4 layers, each layer is composed of a multi-head self-attention network and a feedforward network; wherein, the number of heads in the multi-head self-attention network is 8, and the dimension of each head is 32.

[0009] The sea fog nowcasting method based on dual-stream spatiotemporal Transformer provided by this invention may also have the following features: In step 6, the cross-attention mechanism is specifically as follows: After physical features and multispectral features are encoded, the encoded physical features are used as queries, and the encoded multispectral features are used as keys and values ​​to calculate multi-head cross-attention and obtain the final fused features.

[0010] The sea fog nowcasting method based on dual-stream spatiotemporal Transformer provided by this invention may also have the following features: In step 7, the hierarchical decoder network performs spatial upsampling of the feature map through transposed convolution and introduces skip connections to retain multi-scale spatial detail information; the final output is activated by Sigmoid to generate a prediction map.

[0011] The sea fog nowcasting method based on dual-stream spatiotemporal Transformer provided in this invention may also include the following features: a model training step employing a combined loss function of weighted binary cross-entropy loss and Dice loss, wherein the positive sample weight of the weighted binary cross-entropy loss is 20.0, and the combined loss function takes the form of:

[0012]

[0013] In the formula, For the total loss function, For binary cross-entropy loss, For Dice's loss, and Let be the weighting coefficient, satisfying .

[0014] This invention provides a sea fog nowcasting system based on a dual-stream spatiotemporal Transformer, characterized by the following features: a data preprocessing module, which converts raw multispectral satellite observation data into standardized input tensors to obtain raw satellite data; a multispectral feature stream construction module, which extracts multispectral channel information from the raw satellite data and performs radiometric calibration to obtain a multispectral feature stream; a physical feature stream construction module, which calculates multiple brightness-temperature difference features based on the raw satellite data using the brightness-temperature difference method to construct a physical feature stream; a spatiotemporal feature embedding module, which divides the multispectral and physical feature streams into discrete spatiotemporal segments through 3D convolution and maps them to a high-dimensional feature space to add positional encoding to obtain embedded features; a spatiotemporal feature encoding module, which inputs the embedded features into a multi-layer Transformer encoder and captures spatiotemporal dependencies through a self-attention mechanism to obtain physical features and multispectral features; a dual-branch feature fusion module, which integrates the complementary information of physical features and multispectral features through a cross-attention mechanism to obtain fused features; and a prediction map generation module, which inputs the fused features into a hierarchical decoder network to reconstruct a full-resolution sea fog prediction map for the future time period.

[0015] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the sea fog nowcasting method based on dual-stream spatiotemporal Transformer of the present invention.

[0016] The present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the sea fog nowcasting method based on dual-stream spatiotemporal Transformer of the present invention. Attached Figure Description

[0017] Figure 1 This is an overall framework diagram of the sea fog nowcasting method based on dual-stream spatiotemporal Transformer in this embodiment of the invention;

[0018] Figure 2 This is a detailed architecture diagram of the DSST-Fog model in an embodiment of the present invention, including a dual-branch encoder, a cross-attention fusion module, and a prediction decoder. Detailed Implementation

[0019] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate a method and system for sea fog nowcasting based on a dual-stream spatiotemporal Transformer.

[0020] This implementation provides a sea fog nowcasting method based on a dual-stream spatiotemporal Transformer.

[0021] Sea fog nowcasting can be modeled as a spatiotemporal series prediction task. Given past... Multispectral satellite observation sequence at each time point ,in Let C represent the C-channel image at time i, where C is the number of channels, and H and W represent the height and width of the image, respectively. The goal is to predict the future. Sea fog distribution at a given moment ,in This is a binary sea fog mask. Therefore, to achieve the above objective, this embodiment proposes the following technical solution:

[0022] Figure 1 This is a diagram illustrating the overall framework of the sea fog nowcasting method based on dual-stream spatiotemporal Transformer in this invention.

[0023] Figure 2 This is a detailed architecture diagram of the DSST-Fog model in an embodiment of the present invention, including a dual-branch encoder, a cross-attention fusion module, and a prediction decoder.

[0024] like Figures 1-2 As shown, the sea fog nowcasting method based on dual-stream spatiotemporal Transformer involved in this embodiment includes the following steps:

[0025] Step S1, Data Preprocessing: Convert the raw multispectral satellite observation data into a standardized input tensor to obtain the raw satellite data.

[0026] In this embodiment, the raw observation data from the Fengyun-4 satellite is converted into a standardized input tensor. A standard latitude and longitude grid for the study area is defined, and a geolocation mapping relationship is established. A regular latitude and longitude grid with a fixed spatial resolution (0.1°) is defined for the study area. Subsequently, forward geolocation is performed using satellite orbit parameters and projection equations, accurately mapping each geographic coordinate point on the target grid to the pixel row and column number on the raw full-disk image of the Fengyun-4B satellite.

[0027] Step S2, Construction of multispectral feature stream: Extract multispectral channel information from the original satellite data and complete radiometric calibration to obtain the multispectral feature stream.

[0028] In this embodiment, multispectral channel information is extracted from the raw satellite data and radiometric calibration is performed. The FY-4B L1-level HDF format file at each time step is traversed. Using pre-calculated row and column indices, efficient sampling is performed from the raw count matrix of all 15 spectral channels. Based on the calibration coefficient table embedded in the file, the raw count values ​​are converted into brightness temperature or reflectance with clear physical meaning. These 15 channels constitute a multispectral feature stream.

[0029] Step S3, Construction of physical feature stream: Based on the original satellite data, multiple brightness temperature difference features are calculated using the brightness temperature difference method to construct the physical feature stream.

[0030] In this embodiment, to enhance the model's sensitivity to the physical properties of sea fog, brightness temperature difference (BTD) is introduced as the main input to the physical feature flow. The core of the BTD method lies in utilizing the differences in the radiative response of fog droplets across different bands to enhance the sea fog signal. Single-channel brightness temperature is easily affected by changes in surface temperature and atmospheric background radiation, while BTD can effectively suppress these combined effects through inter-band difference calculations, highlighting the unique spectral characteristics of the fog layer. Simultaneously, different band combinations exhibit varying sensitivities to microphysical parameters such as droplet size distribution, optical thickness, and liquid water content; the combined use of multiple BTD features can characterize sea fog properties from different perspectives. The specific implementation process is as follows: based on sea fog radiative transfer theory, three key brightness temperature difference features are calculated:

[0031]

[0032] Where BT represents brightness temperature, and the subscripts indicate different wavelengths. BTD1 represents the brightness temperature difference between 10.8μm and 3.75μm, which is one of the classic indicators for sea fog detection.

[0033]

[0034] This represents the brightness temperature difference between 10.8 μm and 12.0 μm. This "split window" feature is sensitive to the microphysical properties of water vapor and clouds in the lower atmosphere and is effective both day and night.

[0035]

[0036] This represents the brightness temperature difference between 3.75 μm and 12.0 μm. This feature combines information from the first two brightness temperature differences, providing a linear combined feature for the deep learning model.

[0037] Step S4, Spatiotemporal Feature Embedding: Based on the multispectral feature stream and physical feature stream, the features are divided into discrete spatiotemporal segments through 3D convolution and mapped to a high-dimensional feature space to add positional encoding, thus obtaining the embedded features. The specific process is as follows:

[0038] Step S4-1 involves converting the input data into a feature representation that can be processed by the Transformer. After calculating the BTD features as physical feature input, this embodiment uses 3D convolution to divide the input physical features and multi-channel feature data into discrete spatiotemporal segments, with each segment corresponding to a spatiotemporal window of 30 minutes × 1.6° × 1.6°. Subsequently, a 3D convolutional layer with a kernel size of (2, 16, 16) and a stride of (2, 16, 16) is used to convert the input tensor into N = 3 × 8 × 8 = 192 spatiotemporal segments.

[0039] Step S4-2 involves mapping the spatiotemporal segments to a high-dimensional feature space and adding positional encoding. Each segment is mapped to a D=256-dimensional feature space using a linear embedding layer.

[0040]

[0041] in, For convolution weights, It is a learnable positional encoding matrix. Let C represent the initial embedded features. Here, C represents the number of input channels, and T represents the number of input time steps (T = ...). h and w represent the spatial height and width of a single segment, respectively, N represents the total number of spatiotemporal segments obtained by segmentation (N=192), and D represents the dimension of the embedded features (D=256).

[0042] Step S5, Spatiotemporal Feature Encoding: The embedded features are input into a multi-layer Transformer encoder, and the spatiotemporal dependencies are captured through a self-attention mechanism to obtain physical features and multispectral features. The specific process is as follows:

[0043] In this embodiment, a multi-layer Transformer encoder is constructed to capture spatiotemporal dependencies through a self-attention mechanism. The Transformer encoder contains L=4 layers, each consisting of a multi-head self-attention network (MHSA) and a feedforward network (FFN). The specific implementation process is as follows:

[0044] Step S5-1: Implement the multi-head self-attention mechanism and model the global spatiotemporal interaction pattern. The calculation process of the l-th layer is as follows:

[0045]

[0046]

[0047]

[0048] Where h=8 is the number of attention heads. For each head dimension, This is the projection matrix. These are the query, key, and value matrices, respectively. This is the output of the i-th attention head; To output the projection matrix; This provides multi-headed attention output.

[0049] Step S5-2: Perform nonlinear feature transformation using a feedforward neural network. The FFN employs a two-layer fully connected network with a hidden dimension of 4D=1024.

[0050]

[0051]

[0052] For the output of multi-head attention, FFN is the feedforward network function, and LayerNorm is the layer normalization operation. This is the final output of the l-th layer. For the input of FFN, This is the weight matrix. is the bias term, and GELU is the activation function.

[0053] Step S6: Design a dual-branch feature fusion module: Integrate complementary information from physical features and multispectral features through a cross-attention mechanism to obtain fused features. The specific implementation process is as follows:

[0054] Multispectral feature enhancement guided by physical features is achieved through a cross-attention mechanism. After L=4 layers of encoding, the physical feature branch outputs... Multispectral branch output The cross-attention module achieves feature fusion through a query mechanism:

[0055]

[0056]

[0057]

[0058] In the formula, MHCA represents multi-head cross attention. , , These are the query, key, and value projection matrices, respectively. These are the physical characteristics after fusion. The final fused characteristics are:

[0059]

[0060] Step S7, Construct a hierarchical decoder network: Input the fused features into the hierarchical decoder network to reconstruct a full-resolution sea fog prediction map for future time periods. The specific process is as follows:

[0061] Step 7-1: Spatial upsampling of the feature map is achieved through transposed convolution. The decoder then fuses the features... Restoring to the Future Full-resolution sea fog prediction map at each time point. First, It is reshaped into a low-resolution 3D spatiotemporal feature map, and then upsampled through a series of 3D transposed convolutional layers.

[0062] To address the class imbalance where sea fog pixels account for only 1.09%, this invention employs a combination of weighted BCE and Dice loss:

[0063]

[0064] in For the total loss function, For binary cross-entropy loss, For Dice's loss, and Let be the weighting coefficient, satisfying .

[0065] The binary cross-entropy loss is:

[0066]

[0067] Dice's loss is:

[0068]

[0069] in This represents the total number of pixels. For real labels, To predict probabilities, For positive sample weights, This is a smoothing term to prevent division by zero errors. In the experiment, α=0.3, β=0.7, and w=20.0 were set.

[0070] Step 7-2 introduces skip connections to preserve multi-scale spatial details. To mitigate the loss of spatial details, this invention introduces skip connections similar to U-Net. Feature maps of different scales in the encoder are directly passed to the corresponding layers in the decoder. The final output is then activated by a Sigmoid function to generate predictions.

[0071]

[0072] in It is a probability prediction graph. It is the Sigmoid function, Decoder represents the entire decoder network, and S is the set of skip connection features.

[0073] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

[0074] The present invention has the following technical effects:

[0075] 1. Brightness temperature difference was introduced as the main input quantity of physical characteristic flux to highlight the unique spectral characteristics of sea fog;

[0076] 2. The cross-attention mechanism enables deep integration of prior physical knowledge and data-driven methods, providing high-accuracy forecasts;

[0077] 3. The dual-stream architecture independently encodes physical and multispectral features, avoiding information loss during early fusion;

[0078] 4. The combined loss function effectively handles the extreme class imbalance problem where the proportion of pixels in sea fog is low.

Claims

1. A sea fog nowcasting method based on dual-stream spatiotemporal Transformer, characterized in that, Includes the following steps: Step 1, Data Preprocessing: Convert the raw multispectral satellite observation data into a standardized input tensor to obtain the raw satellite data; Step 2, Construction of multispectral feature stream: Extract multispectral channel information from the original satellite data and complete radiometric calibration to obtain the multispectral feature stream; Step 3, Construction of physical feature stream: Based on the original satellite data, multiple brightness temperature difference features are calculated using the brightness temperature difference method to construct the physical feature stream; Step 4, Spatiotemporal feature embedding: Based on the multispectral feature stream and the physical feature stream, the features are divided into discrete spatiotemporal segments by 3D convolution and mapped to a high-dimensional feature space to add position encoding, thereby obtaining the embedded features; Step 5, Spatiotemporal feature encoding: The embedded features are input into a multi-layer Transformer encoder to capture spatiotemporal dependencies through a self-attention mechanism, thereby obtaining physical features and multispectral features; Step 6, design a dual-branch feature fusion module: integrate the complementary information of the physical features and the multispectral features through a cross-attention mechanism to obtain fused features; Step 7, Construct a hierarchical decoder network: Input the fused features into the hierarchical decoder network to reconstruct a full-resolution sea fog prediction map for future time periods.

2. The sea fog nowcasting method based on dual-stream spatiotemporal Transformer according to claim 1, characterized in that: in, In step 3, the brightness temperature difference characteristic includes: The first brightness temperature difference characteristic, BTD1, is the difference between the brightness temperature in the 10.8 μm band and the brightness temperature in the 3.75 μm band. The second brightness temperature difference feature, BTD2, is the difference between the brightness temperature in the 10.8 μm band and the brightness temperature in the 12.0 μm band. The third brightness temperature difference characteristic, BTD3, is the difference between the brightness temperature at the 3.75 μm band and the brightness temperature at the 12.0 μm band.

3. The sea fog nowcasting method based on dual-stream spatiotemporal Transformer according to claim 1, characterized in that: in, In step 4, the 3D convolution kernel size is (2, 16, 16) and the stride is (2, 16, 16), which converts the input tensor into 192 spatiotemporal segments; and each segment is mapped to a 256-dimensional feature space through a linear embedding layer.

4. The sea fog nowcasting method based on dual-stream spatiotemporal Transformer according to claim 1, characterized in that: in, In step 5, the multi-layer Transformer encoder contains 4 layers, each consisting of a multi-head self-attention network and a feedforward network; wherein, the number of heads in the multi-head self-attention network is 8, and the dimension of each head is 32.

5. The sea fog nowcasting method based on dual-stream spatiotemporal Transformer according to claim 1, characterized in that: in, In step 6, the cross-attention mechanism specifically refers to: After the physical features and the multispectral features are encoded, the encoded physical features are used as queries, and the encoded multispectral features are used as keys and values ​​to calculate multi-head cross-attention and obtain the final fused features.

6. The sea fog nowcasting method based on dual-stream spatiotemporal Transformer according to claim 1, characterized in that: in, In step 7, the hierarchical decoder network performs spatial upsampling of the feature map through transposed convolution and introduces skip connections to preserve multi-scale spatial details; the final output is activated by Sigmoid to generate a prediction map.

7. The sea fog nowcasting method based on dual-stream spatiotemporal Transformer according to claim 1, characterized in that, Also includes: The model training steps employ a combined loss function of weighted binary cross-entropy loss and Dice loss. The positive sample weights in the weighted binary cross-entropy loss are 20.

0. The combined loss function takes the following form: In the formula, For the total loss function, For binary cross-entropy loss, For Dice's loss, and Let be the weighting coefficient, satisfying .

8. A sea fog nowcasting system based on a dual-stream spatiotemporal Transformer, characterized in that, include: The data preprocessing module converts the raw multispectral satellite observation data into standardized input tensors to obtain the raw satellite data. The multispectral feature stream construction module extracts multispectral channel information from the satellite raw data and completes radiometric calibration to obtain the multispectral feature stream. The physical feature flow construction module calculates multiple brightness temperature difference features based on the satellite's original data using the brightness temperature difference method, and constructs the physical feature flow. The spatiotemporal feature embedding module, based on the multispectral feature stream and the physical feature stream, divides them into discrete spatiotemporal segments through 3D convolution and maps them to a high-dimensional feature space to add position encoding, thereby obtaining embedded features; The spatiotemporal feature encoding module inputs the embedded features into a multi-layer Transformer encoder, captures the spatiotemporal dependencies through a self-attention mechanism, and obtains physical features and multispectral features. The dual-branch feature fusion module integrates the complementary information of the physical features and the multispectral features through a cross-attention mechanism to obtain fused features; The prediction map generation module inputs the fused features into the hierarchical decoder network to reconstruct a full-resolution sea fog prediction map for future time periods.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images