A runway visual range prediction method based on multi-modal fusion

Abstract

The application relates to a runway visual range prediction method based on multi-modal fusion, which comprises the following steps: S1, constructing a time-space matched data set, including Himawari-9 satellite thermal infrared channel brightness temperature image data, airport site meteorological element data and RVR / MOR data observed at the airport, performing data cleaning work, and dividing the data into a training set, a verification set and a test set; S2, constructing an RVR / MOR prediction model based on a Cross ViViT model, training model parameters using the training set and the verification set, adjusting and optimizing the parameters, and evaluating the accuracy of the model using the test set; S3, utilizing the specific RVR prediction value mapped from the high-dimensional semantic representation presented by the constructed RVR / MOR prediction model; the method solves the problem that the current prediction effect is influenced by the phenomenon of response lag or prediction distortion in sudden weather events, and the complex generation mechanism of RVR is difficult to accurately depict by only relying on traditional statistical models or deep models based on a single mode.

Description

Technical Field

[0001] This invention belongs to the fields of meteorological technology and artificial intelligence technology, and in particular relates to a runway visual range prediction method based on multimodal fusion. Background Technology

[0002] Runway Visual Range (RVR) is rarely measured directly by observers; instead, it is calculated using automated measuring instruments. The intensity of lights along the airport runway and the ambient light level both affect RVR calculations. RVR reflects the current visibility conditions of the runway, and its predictions provide pilots and airlines with reaction time to future low-visibility conditions. Accurate RVR predictions are crucial for ensuring flight safety and improving airport operational efficiency.

[0003] In the early stages, many scholars used numerical computation or machine learning methods to predict RVR. In 2007, Zhu Lei et al. used support vector machines to predict the RVR of Urumqi Airport; in 2012, Faisal S. Boudala et al. proposed a parameterized method to establish a functional relationship between meteorological elements such as temperature, relative humidity, and wind speed and RVR, and applied it to Toronto Pearson International Airport; in 2016, Zeng Xiangneng et al. used iterative calculation to accurately measure RVR and verified the feasibility of the algorithm through experiments; in 2019, Dietz SJ et al. used a tree model to forecast low visibility events and achieved ideal prediction results at Vienna International Airport.

[0004] With the significant improvement in computing power, deep learning methods, with their stronger nonlinear modeling capabilities, have gradually become an important tool in runway visibility prediction research. In 2019, Deng Tuo et al. used an LSTM model to model visibility at the Beijing meteorological station, and the results showed that its prediction accuracy was better than traditional machine learning models such as random forests and multilayer perceptrons. In 2023, Huang Jialin et al. proposed an improved LSTM model based on meteorological observation data from the Zhengzhou Airport Station of the China Meteorological Administration, and achieved relatively ideal prediction results by selecting key input features based on principal component analysis. In 2022, Peng Lu et al. predicted RVR based on a Long Short-Term Memory (LSTM) network model, and the results showed that its prediction effect in the next two hours was better. In the same year, Zhu Guodong et al. used the automatic deep learning tool Auto-Keras to predict RVR and achieved good results that conformed to the RVR change trend. In 2024, Shankar et al. used a hybrid CNN and LSTM model to predict RVR, and used meteorological elements and historical RVR data to predict RVR in the next 1 to 10 hours over a longer period of time, and the results showed that the hybrid model was effective.

[0005] However, these models are prone to memory decay or forgetting key information when dealing with long-term dependencies, limiting their performance in some time-series prediction tasks. To address this issue, Vaswani et al. proposed a novel deep learning framework—Transformer—in 2017. This model introduces a self-attention mechanism, enabling it to flexibly focus on features at any position in the sequence when processing sequential data, effectively capturing long-distance dependencies and thus improving its ability to model potential correlation structures in time series. Compared to LSTM, it has a significant advantage in extracting global time-series features. In 2021, Haoyi Zhou et al. proposed the Informer model, specifically designed for long-term series prediction tasks. This model significantly reduces computational complexity through a sparse self-attention mechanism and demonstrates excellent performance in multiple weather prediction tasks. In 2023, Han Bo et al. applied the Informer model to airport visibility prediction with good results. In 2024, Ma Yuzhao et al. further proposed the TimeGAN-Informer model, achieving daily-scale visibility prediction.

[0006] Relative visibility dynamics (RVR) are influenced by multiple meteorological factors, such as temperature, humidity, and wind speed. These factors exhibit sequential dynamics over time and high heterogeneity and hierarchical structure in space. Therefore, relying solely on traditional statistical models or single-modal deep models often fails to accurately characterize their complex generation mechanisms, particularly in the case of sudden weather events, where response lags or prediction distortions are common. Furthermore, the cloud and fog structure information contained in remote sensing images is of significant reference value for visibility changes. Therefore, an effective prediction model must be able to integrate data from both image and time-series modalities and capture their complex nonlinear coupling relationships. Traditional models often focus on processing information in only one dimension, making it difficult to comprehensively reflect the complex evolution of RVR. The CrossViViT model, however, provides a more systematic solution. This model integrates a visual Transformer and a cross-modal attention mechanism, enabling it to process both remote sensing images and time-series data. In particular, its internal Cross-Transformer module allows the Query, Key, and Value to originate from inputs of different modalities, effectively addressing the differences in modality, spatiotemporal scale, and structure between ground observation and satellite remote sensing. This mechanism can simulate ground stations extracting the most relevant regional information from satellite images when making predictions, effectively achieving "point-area information coupling" and multi-scale feature alignment, thus enhancing the model's ability to understand spatiotemporal data. Summary of the Invention

[0007] In view of this, in order to address the problem that current methods relying solely on traditional statistical models or single-modal deep models often fail to accurately characterize the complex generation mechanism of RVR, especially in the case of sudden weather events where response lag or prediction distortion easily occurs, affecting the prediction effect, this invention provides a runway visual range (RVR) prediction method based on multimodal fusion. It uses the Cross ViViT model built based on Vision Transformer and Temporal Transformer, combined with the thermal infrared channel brightness temperature observed by the Himawari-9 satellite, meteorological elements observed by airport sites, and RVR, to achieve RVR prediction in a point-to-surface combined manner. It comprehensively utilizes satellite remote sensing imagery and ground observation data to improve the prediction accuracy of RVR in low visibility events.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A runway visual range prediction method based on multimodal fusion includes the following steps:

[0010] S1. Construct a spatiotemporal matching dataset, including Himawari-9 satellite thermal infrared channel brightness temperature image data, airport site meteorological element data, and airport observation RVR / MOR data. Clean the data and divide it into training set, validation set, and test set.

[0011] S2. Construct an RVR / MOR forecast model based on the Cross ViViT model. Use the training and validation sets from step S1 to train and optimize the model parameters. Use the test set to evaluate the model's accuracy. The Cross ViViT model uses the Vision Transformer (ViT) module as the image spatial extraction module to process thermal infrared brightness temperature image data from the geostationary meteorological satellite Himawari-9. The Cross ViViT model also uses the TemporalTransformer module as the time series extraction module to model the dynamic changes of multiple observed variables from a time dimension.

[0012] S3. Map the high-dimensional semantic representation presented by the RVR / MOR forecasting model constructed in step S2 to specific RVR prediction values, and put it into use in actual operations to perform operational forecasting of RVR.

[0013] Furthermore, in step S1, the runway visibility prediction dataset after data cleaning is randomly sampled in a 6:2:2 ratio, and the samples are divided into a training set, a validation set, and a test set. The training set is used to determine the prediction model, the validation set is used to determine the optimal model parameters for prediction, and the test set is used to verify the prediction effect.

[0014] Furthermore, the specific process of constructing the RVR / MOR forecast model in step S2 is as follows:

[0015] S21. For the input image of each time frame in the thermal infrared brightness temperature image data Divide it into non-overlapping, fixed-size patch blocks, with a size of ,in and These are the height and width of the image, respectively. This refers to the number of thermal infrared channels in the Himawari-9. Given the set patch size, it can be divided into a total of Each patch is flattened into a one-dimensional vector and then transformed using a learnable linear transformation matrix. Mapped to In the dimensional embedding space, a sequence of patch tokens is formed:

[0016]

[0017] At the same time, to preserve spatial location information, a location code is added to each patch. :

[0018]

[0019] For continuous The frame images are processed using the patch embedding method described above and stacked in chronological order to form the final input of the model:

[0020]

[0021] If the batch size input each time is Therefore, the final input dimension of the model is:

[0022]

[0023] S22. Input the model input sequence established in step S21 to the system... The Transformer encoder consists of stacked layers, with each layer applying a self-attention mechanism along the patch dimension to model the spatial structural relationships between patches within each frame. The attention calculation formula is as follows:

[0024]

[0025] in It is obtained by linearly transforming the patch token in each frame. For each dimension of attention head, This refers to the number of attention heads; since the model processes each frame of image in parallel, the encoder models each frame of image independently in the spatial dimension, and the output dimension remains consistent with the input.

[0026] .

[0027] S23. The input to the Temporal Transformer is a multivariate observation sequence from a ground weather station. Let its tensor dimension be:

[0028]

[0029] in, It is the batch size. It is the input time step. This refers to the number of input surface meteorological variables, which include temperature, air pressure, and relative humidity.

[0030] Specifically, this involves projecting the input variables uniformly to a fixed hidden dimension using a linear embedding layer or convolutional layer. : ;

[0031] S24. Project the input variables from step S23 onto a fixed-dimensional input variable and input them into a Temporal Transformer encoder, then model the time series using a standard self-attention mechanism. Each Transformer layer includes a multi-head self-attention mechanism, residual connections, and layer normalization. Each layer is in the time dimension. Self-attention is applied to model the long-term and short-term dependencies between different time steps. Finally, the encoder output dimension is consistent with the input dimension, containing feature representations that integrate historical context and variable interactions at each time step, which can effectively capture the trends and abrupt changes in local weather evolution. ;

[0032] S25. Establish a Cross-Attention module to connect the Vision Transformer module established in step S22 and the Temporal Transformer module established in step S24. In the Cross ViViT model structure, the Cross-Attention module is the key bridge connecting the spatial representation of the image and the temporal series representation.

[0033] Cross-Attention employs a standard multi-head cross-attention mechanism, using time-series features as queries and image spatial features as keys and values ​​to achieve intermodal information exchange. This module performs joint space-time modeling at each time step, where:

[0034] The query comes from the output of the Temporal Transformer module in step S24, and has the following shape:

[0035]

[0036] in For step S24 ;

[0037] The key / value comes from the output of the Vision Transformer module in step S22, and the original shape is:

[0038]

[0039] To facilitate subsequent operations, the output of the Vision Transformer module is flattened along the patch dimension and reconstructed as follows:

[0040]

[0041] This step enables the joint encoding of the spatial structure and temporal evolution of an image into a unified token sequence, allowing each temporal query to be aware of the context of the entire image's spatiotemporal domain.

[0042] Furthermore, in step S22, to enhance the model's learning ability under small sample conditions and its adaptability to missing data, a self-supervised pre-training strategy in the style of MAE is introduced; during the training phase, the patch token of each frame image is randomly occluded, and the model is trained to reconstruct the occluded token representation based on the unoccluded part.

[0043] Furthermore, in step S23, in order to capture time information, a learnable position code is added to each time step. : .

[0044] Furthermore, the core function of the Cross-Attention module in step S25 is to effectively inject the image spatial context features extracted by the Vision Transformer into the time series features extracted by the Temporal Transformer, thereby realizing the dynamic perception of remote sensing image information during the site RVR prediction process.

[0045] Furthermore, in order to maintain the consistency of cross-modal alignment, step S25 embeds spatial and temporal coordinates into the token sequence. Specifically, temporal step embedding and spatial location embedding are added to the Query and Key / Value respectively, and temporal and spatial information is input. Through this dual coordinate encoding mechanism, it is ensured that the time series features and the image spatial patch representation correspond in a unified semantic space, enabling the model to learn the interaction rules between time and space.

[0046] The output sequence after cross-attention fusion is further fed into the Temporal Transformer decoder to model the long-term dependencies and contextual trends between prediction time steps. The decoder module consists of several Transformer layers, with a structure similar to the aforementioned Temporal Transformer encoder, and the final output dimension is: .

[0047] Furthermore, in step S3, to map the high-dimensional semantic representation to specific RVR prediction values, a multilayer perceptron is connected after the decoder as the output head; the MLP includes several fully connected layers, and the final output dimension is: , This represents the RVR value for each prediction time step.

[0048] The runway visibility prediction method based on multimodal fusion disclosed in this invention has the following advantages:

[0049] 1) High-precision prediction capability: By constructing a cross-modal learning structure that integrates remote sensing images and ground station sequences, this invention can achieve high-precision prediction of near-surface meteorological elements such as runway visibility (RVR). This method can fully exploit the complementary features of spatiotemporal multi-source observation data, significantly improving the generalization ability, robustness, and practicality of the prediction model.

[0050] 2) Improve prediction accuracy: The Cross ViViT model, built based on Vision Transformer and Temporal Transformer, is used. Combined with the thermal infrared channel brightness temperature observed by Himawari-9 satellite, meteorological elements observed by airport stations, and RVR, RVR is predicted in a point-to-area combined approach. The prediction accuracy of RVR in low visibility events is improved by comprehensively utilizing satellite remote sensing imagery and ground observation data.

[0051] 3) Enhance model adaptability: Introduce a self-supervised pre-training strategy in the style of MAE to enhance the model's learning ability under small sample conditions and its adaptability to missing data, improve the model's ability to infer the overall spatial structure from partial information, and enhance its robustness and generalization ability under complex cloud field conditions.

[0052] 4) Effectively capture time dependencies: The Temporal Transformer module effectively mines time dependencies, coupling interactions between variables, and potential periodic trends in historical sequences, providing stable temporal priors for RVR prediction.

[0053] 5) Dynamic perception capability: The Cross-Attention module effectively injects the image spatial context features extracted by the Vision Transformer into the time series features extracted by the Temporal Transformer, realizing dynamic perception of remote sensing image information during the site RVR prediction process.

[0054] 6) Early identification capability: During the visibility improvement phase, the model demonstrates good trend prediction capability, can predict the timing of the turning point in advance, showing a certain "lead time", and has the ability to identify the trend of visibility improvement in advance.

[0055] This runway visual range (RVR) prediction method achieves high-precision prediction of near-surface meteorological elements such as runway visual range by constructing a cross-modal learning structure that fuses remote sensing images and ground station sequences. This method fully leverages the complementary features of spatiotemporal multi-source observation data, significantly improving the generalization ability, robustness, and practicality of the prediction model. It has broad application prospects in scenarios such as aviation meteorological support, providing pilots and airlines with reaction time to future low-visibility conditions, which is particularly important for ensuring flight safety and improving airport operational efficiency. Furthermore, through innovative multimodal fusion methods, it significantly improves the accuracy and reliability of RVR prediction, which has significant practical implications for aviation meteorological services and flight safety.

[0056] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0058] Figure 1This is a flowchart of the runway visual range prediction method based on multimodal fusion according to the present invention;

[0059] Figure 2 This is a model construction diagram of the runway visibility prediction method based on multimodal fusion according to the present invention;

[0060] Figure 3 This is a comparison chart of the actual and predicted RVR values ​​for the descending inflection point in this embodiment of the invention. Figure 3 (a) is a comparison chart of the actual and predicted RVR values ​​from 00:30 on March 11, 2025 to 02:30 on March 11, 2025. Figure 3 (b) is a comparison chart of the actual and predicted RVR values ​​from 01:15 on March 11, 2025 to 03:15 on March 11, 2025;

[0061] Figure 4 This is a comparison chart of the actual and predicted RVR values ​​for the stable low-value portion in this embodiment of the invention, specifically a comparison chart of the actual and predicted RVR values ​​from 02:10 on March 11, 2025 to 04:10 on March 11, 2025.

[0062] Figure 5 This is a comparison chart of the actual and predicted RVR values ​​in the rising transition section of this invention, specifically a comparison chart of the actual and predicted RVR values ​​from 03:23 on March 11, 2025 to 05:23 on March 11, 2025. Detailed Implementation

[0063] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0064] like Figure 1 The runway visual range prediction method based on multimodal fusion shown includes the following steps:

[0065] S1. Construct a spatiotemporal matching dataset, including Himawari-9 satellite thermal infrared channel brightness temperature image data, airport site meteorological element data, and airport observation RVR / MOR data. Clean the data and divide it into training set, validation set, and test set.

[0066] S2, such as Figure 2As shown, an RVR / MOR forecast model is constructed based on the Cross ViViT model. The model parameters are trained and adjusted using the training and validation sets in step S1, and the accuracy of the model is evaluated using the test set. The Cross ViViT model uses the Vision Transformer (ViT) module as the image spatial extraction module to process thermal infrared brightness temperature image data from the geostationary meteorological satellite Himawari-9. The Temporal Transformer module in the Cross ViViT model is used as the time series extraction module to model the dynamic changes of multiple observed variables from a time dimension.

[0067] Specifically, thermal infrared brightness temperature (RVR) image data, as observational data, is essentially a time-varying image sequence containing spatial structure information across multiple bands. It reflects atmospheric structure characteristics closely related to cloud and fog evolution and has significant indicative value for RVR changes. To effectively extract its spatiotemporal evolution features, a ViT-based image encoder module was constructed to perform deep modeling of spatial patterns in the image sequence, used to extract spatial information features from the thermal infrared channel images.

[0068] S21. For the input image of each time frame Divide it into non-overlapping, fixed-size patch blocks, with a size of ,in and These are the height and width of the image, respectively. This refers to the number of image channels, specifically the number of thermal infrared channels in the Himawari-9 model used in this study. Given the set patch size, it can be divided into a total of Each patch is flattened into a one-dimensional vector and then transformed using a learnable linear transformation matrix. Mapped to In the dimensional embedding space, a sequence of patch tokens is formed:

[0069]

[0070] At the same time, to preserve spatial location information, a location code is added to each patch. :

[0071]

[0072] For continuous The frame images are processed using the patch embedding method described above and stacked in chronological order to form the final input of the model:

[0073]

[0074] If the batch size input each time is Therefore, the final input dimension of the model is:

[0075]

[0076] S22. Input the model input sequence established in step S21 to the system... The Transformer encoder consists of stacked layers, with each layer applying a self-attention mechanism along the patch dimension to model the spatial structural relationships between patches within each frame. The attention calculation formula is as follows:

[0077]

[0078] in It is obtained by linearly transforming the patch token in each frame. For each dimension of attention head, This refers to the number of attention heads. Since the model processes each frame in parallel, the encoder independently models each frame spatially, ensuring the output dimension remains consistent with the input.

[0079]

[0080] To enhance the model's learning ability under limited sample conditions and its adaptability to missing data, a MAE-style self-supervised pre-training strategy was introduced. During the training phase, the patch tokens of each frame were randomly occluded, and the model was trained to reconstruct the occluded token representation based on the unoccluded parts. This training strategy effectively improved the model's ability to infer the overall spatial structure from partial information, enhancing its robustness and generalization ability under complex cloud conditions.

[0081] S23. To fully utilize the multivariate time-series information provided by ground meteorological stations, the model introduces a Temporal Transformer module to model the dynamic changes of multiple observed variables from a temporal perspective. This module can effectively uncover time dependencies, coupling interactions between variables, and potential periodic trends in historical sequences, thereby providing stable temporal priors for RVR prediction.

[0082] The input to the Temporal Transformer is a multivariate observation sequence from a ground weather station, and its tensor dimension is assumed to be:

[0083]

[0084] in, It is the batch size. It is the input time step. It is the number of input variables (such as temperature, air pressure, relative humidity, etc.).

[0085] First, a linear embedding layer or convolutional layer is used to uniformly project the input variables to a fixed hidden dimension. :

[0086]

[0087] In addition, to capture time information, a learnable position code is added to each time step. :

[0088]

[0089] S24. Project the input variables from step S23 onto a fixed-dimensional input variable and input them into a Temporal Transformer encoder, then model the time series using a standard self-attention mechanism. Each Transformer layer includes a multi-head self-attention mechanism, residual connections, and layer normalization. Each layer has a time dimension... Self-attention is applied to model the long-term and short-term dependencies between different time steps. Finally, the encoder output dimension is consistent with the input dimension, containing feature representations that integrate historical context and variable interactions at each time step, effectively capturing the trends and abrupt changes in local weather evolution.

[0090]

[0091] S25. Establish the Cross-Attention module. The Cross-Attention module connects the Vision Transformer module established in step S22 and the Temporal Transformer module established in step S24. In the Cross ViViT model structure, the Cross-Attention module is the key bridge connecting the spatial representation of the image and the temporal series representation. Its core function is to effectively inject the spatial context features extracted by the Vision Transformer into the temporal series features extracted by the Temporal Transformer, thereby realizing the dynamic perception of remote sensing image information during the site RVR prediction process.

[0092] Cross-Attention employs a standard multi-head cross-attention mechanism, using time-series features as queries and image spatial features as keys and values ​​to achieve intermodal information exchange. This module performs joint space-time modeling at each time step, where:

[0093] The query comes from the output of the Temporal Transformer module in step S24, and has the following shape:

[0094]

[0095] in For step S24 ;

[0096] The key / value comes from the output of the Vision Transformer module in step S22, and the original shape is:

[0097]

[0098] To facilitate subsequent operations, the output of the Vision Transformer module is flattened along the patch dimension and reconstructed as follows:

[0099]

[0100] This step enables the joint encoding of the spatial structure and temporal evolution of an image into a unified token sequence, allowing each temporal query to be aware of the context of the entire image's spatiotemporal domain.

[0101] Meanwhile, to maintain consistency in cross-modal alignment, spatial and temporal coordinate embeddings are applied to the token sequence. Specifically, temporal step embeddings and spatial location embeddings are added to the Query and Key / Value pairs, respectively, inputting both temporal and spatial information. This dual coordinate encoding mechanism ensures that time-series features correspond to image spatial patch representations within a unified semantic space, enabling the model to learn the interaction patterns between time and space.

[0102] The output sequence after cross-attention fusion is further fed into the Temporal Transformer decoder to model the long-term dependencies and contextual trends between prediction time steps. The decoder module consists of several Transformer layers, with a structure similar to the aforementioned Temporal Transformer encoder, and the final output dimension is:

[0103]

[0104] S3. Map the high-dimensional semantic representation presented by the RVR / MOR forecasting model constructed in step S2 to specific RVR prediction values, and put it into use in actual operations to perform operational forecasting of RVR.

[0105] To map the high-dimensional semantic representation to specific RVR predictions, a multilayer perceptron (MLP) is connected after the decoder as the output head. The MLP consists of several fully connected layers, and the final output dimension is:

[0106]

[0107] This represents the RVR value for each prediction time step.

[0108] Example

[0109] In meteorology, especially in contexts related to airport weather conditions, the descent inflection point, the stable low value phase, and the ascent inflection point are used to describe the curve characteristics of runway visual range (RVR) over time. The descent inflection point refers to the turning point where RVR begins to decline significantly from a relatively high value, usually indicating that weather conditions are beginning to deteriorate. The stable low value phase refers to the relatively stable low value stage that RVR enters after its decline, indicating relatively severe but stable weather conditions. The ascent inflection point refers to the turning point where RVR begins to rise significantly from a stable low value, indicating that weather conditions are beginning to improve.

[0110] Case analysis of runway 34L at Shanghai Pudong International Airport from 21:00 on February 27, 2025 to 7:00 on February 28, 2025.

[0111] The evaluation of this model can be divided into three parts: the declining inflection point, the stable low value point, and the rising inflection point. The following comparison is made by contrasting the actual and predicted RVR values ​​at this point in time. The predicted RVR values ​​are based on observation data from the airport station and satellite data from the previous hour.

[0112] 1) The downward turning point:

[0113] Figure 3 This illustrates the process of RVR declining from a high value to a low value during this period. Figure 3 (a) is a comparison chart of the actual and predicted RVR values ​​from 00:30 on March 11, 2025 to 02:30 on March 11, 2025. Figure 3(b) is a comparison chart of the actual and predicted RVR values ​​from 01:15 on March 11, 2025 to 03:15 on March 11, 2025. In the chart, the blue dotted line represents the actual observed RVR value, the orange dotted line represents the model's predicted value, and the red dashed line marks the forecast start time, i.e., the model no longer acquires subsequent actual information, and the orange curve thereafter is the pure prediction output. To further analyze trend changes, two fitting curves are also added to the chart: the blue solid line represents the linear fitting trend of the actual RVR data in the next hour, while the orange solid line represents the fitting trend of the model's predicted value within the same time period.

[0114] The results show that the model successfully identified the trend of RVR decreasing from high to low values ​​after the red dashed line, with the predicted curve starting to decline and gradually approaching the actual value. This indicates that the model can not only extract key features from historical information but also effectively predict the future evolution direction of RVR, that is, the predicted RVR value begins to shift from high to low. Although the predicted value differs significantly from the actual value, the model is able to capture the trend of change.

[0115] 2) Stable low value portion

[0116] Figure 4 This chart illustrates the process of RVR consistently reporting low values ​​during this period, specifically comparing the actual and predicted RVR values ​​from 02:10 on March 11, 2025 to 04:10 on March 11, 2025. In the chart, the blue dotted line represents the observed RVR value, the orange dotted line represents the model's predicted value, and the red dashed line marks the start time of the forecast, indicating that the model no longer acquires subsequent observed data; the orange curve thereafter represents the pure predicted output. To further analyze trend changes, two fitting curves are added to the chart: the blue solid line represents the linear fitting trend of the observed RVR data within the next hour, while the orange solid line represents the fitting trend of the model's predicted values ​​within the same time period.

[0117] As shown in the attached figures, the model effectively perceives persistent haze or low visibility backgrounds and maintains stable output. The RVR trend aligns with reality, but the model still exhibits some overestimation (optimism bias). In several low-value segments, the model's predicted values ​​are significantly higher than the actual values, failing to accurately reflect the actual low visibility risk. The reasons for this optimism bias may include: unbalanced training sample distribution: low visibility samples constitute a small proportion of the entire training set, causing the model to favor fitting normal weather conditions, resulting in insufficient generalization ability at extremely low values; saturated or blurred brightness and temperature images: at night or in high humidity environments, the brightness and temperature gradients in foggy areas of remote sensing images are small, making it difficult for the model to extract sufficiently significant image features; insufficient response of the objective function to extreme values: if the training loss function does not weight low-value regions, the model lacks sufficient optimization drive at extremely low values, leading to overestimation of predicted values. This provides directions for future model optimization.

[0118] 3) The upward turning point

[0119] Figure 5 This chart illustrates the process of RVR consistently reporting low values ​​during this period, comparing the actual and predicted RVR values ​​from 03:23 on March 11, 2025 to 05:23 on March 11, 2025. In the chart, the blue dotted line represents the observed RVR value, the orange dotted line represents the model's predicted value, and the red dashed line marks the start time of the forecast, indicating that the model no longer acquires subsequent observed data; the orange curve thereafter represents the pure predicted output. To further analyze trend changes, two fitting curves are added to the chart: the blue solid line represents the linear fitting trend of the observed RVR data within the next hour, while the orange solid line represents the fitting trend of the model's predicted values ​​within the same time period.

[0120] During the visibility improvement phase, the model demonstrated strong trend prediction capabilities. As shown in the attached chart, the RVR value gradually increased from a low level, and the model successfully captured this upward trend, even anticipating the timing of inflection points. The model predicted the RVR rebound signal at an early time point, with the predicted curve starting to rise before the actual curve showed a significant increase, indicating a certain "lead time." Although the predicted rise time was slightly earlier than the actual time in some samples, overall the model's ability to capture trends was strong, and the magnitude of the rise was basically consistent with the actual data, without significant "lag" or "misjudgment of rebound." This indicates that the model is highly sensitive to image and time-series features during the fog dissipation phase, possessing the ability to identify the visibility improvement trend in advance. This demonstrates the model's good performance in predicting the upward phase.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A runway visual range prediction method based on multi-modal fusion, characterized in that, Comprising the following steps: S1, constructing a spatio-temporal matched dataset, including Himawari-9 satellite thermal infrared channel brightness temperature image data, airport site meteorological element data and RVR / MOR data observed at the airport, performing data cleaning work, and dividing into a training set, a validation set and a test set; S2, constructing an RVR / MOR prediction model based on a Cross ViViT model, training and adjusting and optimizing the model parameters using the training set and the validation set in step S1, and using the test set to evaluate the accuracy of the model, wherein the Vision Transformer (ViT) module is used as an image space extraction module in the Cross ViViT model to process the thermal infrared brightness temperature image data from the geostationary meteorological satellite Himawari-9; the Temporal Transformer module is used as a time series extraction module in the Cross ViViT model to model the dynamic change process of multiple observation variables in the time dimension; The specific process of constructing the RVR / MOR prediction model in step S2 is as follows: S21, input image for each time frame of thermal infrared brightness temperature image data , which is divided into non-overlapping fixed-size patch blocks of size , where and are the height and width of the image, respectively, is the number of thermal infrared channels of Himawari-9, is the set patch size, which can be divided into patches in total; each patch is flattened into a one-dimensional vector and mapped into a -dimensional embedding space using a learnable linear transformation matrix to form a sequence of patch tokens: At the same time, to preserve spatial positional information, add position encoding for each patch : For consecutive frame images, the above patch embedding process is performed on each of them and stacked in time order to form the final input to the model: If the Batchsize is input each time , then the final input dimension of the model is: S22, inputting the model sequence established in step S21 to the model established in step S21 by Transformer encoders stacked layer by layer, each layer applying a self-attention mechanism on the patch dimension to model the spatial structural relationship between patches within each frame of image; The attention calculation formula is as follows: wherein is the patch token in each frame after linear transformation, is the dimension of each attention head, is the number of attention heads; since the model processes each frame image in parallel, the encoder models each frame image independently in the spatial dimension, and the output dimension remains consistent with the input: The input of the Temporal Transformer is a multivariate observation sequence from the ground meteorological station, and its tensor dimension is as follows: wherein, is the Batch size, is the input time step, is the number of input ground meteorological variables, ground meteorology including temperature, pressure, relative humidity; Specifically: uniform projection of input variables to a fixed hidden dimension through a linear embedding layer or a convolutional layer : ; S24, inputting the projection of step S23 to the input variable of fixed dimension into the Temporal Transformer encoder and modeling the time series by using the standard self-attention mechanism; each layer of the Transformer includes multi-head self-attention mechanism, residual connection, layer normalization, and each layer is in the time dimension applying self-attention to model the long and short term dependence between different time steps; the final encoder output dimension is consistent with the input dimension, and the feature representation of each time step includes the comprehensive history context and variable interaction, which can effectively capture the trend and mutation of local weather evolution: ; S25, establishing a Cross-Attention module for connecting the Vision Transformer module established in step S22 and the Temporal Transformer module established in step S24, wherein the Cross-Attention module is a key bridge connecting the image space representation and the time series representation in the Cross ViViT model structure; The Cross-Attention adopts a standard multi-head cross-attention mechanism, taking the time series features as the query (Query), the image space features as the key (Key) and the value (Value), and realizing the information interaction between the two modalities; this module performs spatial-temporal joint modeling at each time step, wherein: The Query comes from the output of the Temporal Transformer module in step S24, and the shape is as follows: wherein for step S24 ; The Key / Value comes from the output of the Vision Transformer module in step S22, and the original shape is as follows: In order to facilitate subsequent operations, the output of the Vision Transformer module is flattened along the patch dimension and reconstructed as follows: This step realizes the joint coding of the spatial structure of the image and the temporal evolution into a unified token sequence, so that each time series query can perceive the context of the entire image spatio-temporal domain; S3, mapping the high-dimensional semantic representation presented by the RVR / MOR prediction model constructed in step S2 to specific RVR prediction values, and putting it into use in actual business to predict RVR in a business-oriented manner.

2. The runway visual range prediction method based on multi-modal fusion according to claim 1, characterized in that, The runway visual range prediction data set processed by data cleaning in step S1 is randomly sampled according to a ratio of 6:2:2, and the samples are divided into a training set, a validation set and a test set, the training set is used to determine the prediction model, the validation set is used to determine the optimal model parameter of the prediction, and the test set is used to verify the prediction effect.

3. The runway visual range prediction method based on multi-modal fusion according to claim 1, wherein, In step S22, in order to enhance the learning ability of the model under small sample conditions and the adaptability to data loss, a self-supervised pre-training strategy in the MAE style is introduced; in the training stage, the patch token of each frame of image is randomly blocked, and the model is trained to reconstruct the blocked token representation according to the unblocked part.

4. The runway visual range prediction method based on multi-modal fusion according to claim 3, characterized in that, In step S23, to capture temporal information, a learnable positional encoding is added for each time step : .

5. The runway visual range prediction method based on multi-modal fusion according to claim 1, wherein, In step S25, the core function of the Cross-Attention module is to effectively inject the image space context features extracted by the Vision Transformer into the time sequence features extracted by the Temporal Transformer, so as to realize the dynamic perception of remote sensing image information in the station RVR prediction process.

6. The runway visual range prediction method based on multi-modal fusion according to claim 4, characterized in that, In step S25, in order to maintain the consistency of cross-modal alignment, spatial and time coordinate embedding is performed on the token sequence, that is, time step embedding and spatial position embedding are added to the Query and Key / Value respectively, and time and space information is input; through this double coordinate coding mechanism, it is ensured that the time sequence features and the image space patch expression correspond in the unified semantic space, so that the model can learn the interaction law between time and space; the output sequence after Cross-Attention fusion is further input into the Temporal Transformer decoder, which is used to model the long-term dependence relationship and context trend between each prediction time step; The decoder module consists of several layers of Transformers, with a similar structure as the aforementioned Temporal Transformer encoder, with the final output dimension being: .

7. The runway visual range prediction method based on multi-modal fusion according to claim 6, wherein, Step S3 To map the high-dimensional semantic representation into concrete RVR prediction values, a multi-layer perceptron is connected as an output head after the decoder; the MLP includes several fully connected layers, and the final output dimension is: , represents the RVR numerical value corresponding to each prediction time step.

Images