Solar irradiance prediction system and method based on dual attention multi-mode fusion

Abstract

The invention discloses a solar irradiance prediction system and method based on dual attention multi-modal fusion. The method comprises the following steps: performing spatial feature extraction on a sky imaging observation map after preprocessing and time synchronization by using a MobileNetV2 network to obtain a sky imaging time sequence feature sequence; inputting the sky imaging time sequence feature sequence into a DAT model to obtain a global sky imaging time sequence feature sequence, a local sky imaging time sequence feature sequence and a local convolution position coding residual term; obtaining an aggregated sky imaging time sequence feature sequence according to the global sky imaging time sequence feature sequence, the local sky imaging time sequence feature sequence and the local convolution position coding residual item; performing time-dependent modeling on the meteorological observation data by adopting a long short-term memory network to obtain a meteorological observation data feature sequence; and performing feature fusion on the aggregated sky imaging time sequence feature sequence and the meteorological observation data feature sequence through a trans-attention mechanism to obtain a fused joint feature sequence for solar irradiance prediction.

Description

Technical Field

[0001] This invention relates to the field of solar radiation information processing and meteorological data analysis, specifically to a solar irradiance prediction system and method based on dual attention multimodal fusion. Background Technology

[0002] Global solar irradiance (GSI) is a crucial physical quantity characterizing Earth's surface solar energy resources and a fundamental input parameter in photovoltaic (PV) power output estimation and power system operation analysis. Influenced by factors such as cloud formation, movement, dissipation, and local meteorological disturbances, irradiance often exhibits significant and rapid changes and uncertainties on a minute-scale timescale. Under high-proportion PV grid-connected conditions, these short-term fluctuations can impact inverter power output, grid voltage regulation, frequency maintenance, and reserve capacity configuration. Therefore, various operation and control systems place higher demands on short-scale irradiance prediction. Existing solar irradiance prediction methods mainly include models based on physical mechanisms, models based on statistical laws, and data-driven modeling methods. Physical models rely on calculations of cloud cover, atmospheric parameters, and radiative transfer, making it difficult to maintain stable accuracy under rapidly changing weather conditions. Statistical models are often based on time-series feature extraction and are insufficiently responsive to sudden cloud cover or local weather changes. Single-modal methods based on imaging data or meteorological measurements can capture some trends but cannot fully express the simultaneous spatial structure changes and environmental parameter changes in irradiance. With the widespread deployment of sky imaging equipment and meteorological monitoring devices in photovoltaic power plants, improving short-term forecast accuracy using multi-source observational data has become a research hotspot. However, existing research often focuses on feature extraction from specific data types or uses fixed methods to combine different data, failing to fully reflect the correlation and complementarity between various types of observational information. In complex weather conditions or rapidly changing scenarios, forecast accuracy and stability remain insufficient. Therefore, it is necessary to propose a technical solution that can comprehensively utilize the information features of image-based data and meteorological measurement data to improve the accuracy and adaptability of minute-level solar irradiance forecasts. Summary of the Invention

[0003] The purpose of this invention is to provide a solar irradiance prediction system based on dual attention multimodal fusion and a solar irradiance prediction method based on dual attention multimodal fusion. The system and method can improve the shortcomings of existing methods in terms of data utilization efficiency, spatiotemporal feature representation and adaptability to complex weather.

[0004] To achieve this objective, the present invention provides a solar irradiance prediction system based on dual attention multimodal fusion, comprising: The data acquisition module is used to collect sky imaging observation images and meteorological observation data of the area to be predicted, and to perform time alignment and data preprocessing operations on the sky imaging observation images and meteorological observation data to obtain preprocessed and time-synchronized sky imaging observation images and meteorological observation data. The image feature extraction and temporal modeling module is used to extract spatial features from the preprocessed and time-synchronized sky imaging observation image using the MobileNetV2 network, and obtain the sky imaging temporal feature sequence. The image processing module is used to input the sky imaging temporal feature sequence in parallel to the global attention branch and local window attention branch of the DAT structure. The global attention branch of the DAT structure is used to perform adaptive aggregation processing on the sky imaging temporal feature sequence to obtain the global sky imaging temporal feature sequence. The local window attention branch of the DAT structure is used to perform local window attention calculation and window stitching processing on the sky imaging temporal feature sequence to obtain the local sky imaging temporal feature sequence. According to the local convolution position enhancement mechanism of the DAT structure and the sky imaging temporal feature sequence, the local convolution position encoding residual term is calculated. The global sky imaging temporal feature sequence and the local sky imaging temporal feature sequence are sequentially processed by channel stitching, linear fusion layer integration, and residual term introduction to obtain the aggregated sky imaging temporal feature sequence. The meteorological sequence modeling module is used to perform time-dependent modeling of meteorological observation data using a long short-term memory network to obtain the feature sequence of meteorological observation data. The multimodal attention fusion module is used to fuse the aggregated sky imaging time-series feature sequences and meteorological observation data feature sequences through a cross-attention mechanism to obtain a fused joint feature sequence for solar irradiance prediction.

[0005] The beneficial effects of this invention are as follows: Addressing the difficulty in predicting rapidly changing solar irradiance on a minute-by-minute basis, existing technologies for short-term irradiance estimation suffer from the following shortcomings: Insufficient utilization of observational data from different sources: Existing methods often rely on single-type observational information, making it difficult to simultaneously reflect changes in cloud structure and environmental parameters, and limiting their responsiveness to sudden shading or rapid weather changes; Insufficient representation of the spatiotemporal characteristics of cloud changes: Some image- or sequence-based modeling methods cannot simultaneously consider the overall trend and local details of cloud movement, making it difficult to obtain stable spatiotemporal characteristic representations in minute-by-minute prediction scenarios; Insufficient representation of the correlation between multiple types of observational information: Existing methods often use simple combinations or fixed methods when processing multi-source data, making it difficult to reflect the differences in importance of different types of information under different weather conditions, and the prediction results are prone to fluctuations in complex scenarios; Difficulty in meeting the real-time and stability requirements of engineering scenarios: In application scenarios such as power plant operation control, grid regulation, or energy storage management, the response speed and stability requirements for short-term irradiance prediction are high, while existing methods still have insufficient accuracy and reliability under rapidly changing weather conditions.

[0006] This invention constructs a solar irradiance prediction model based on dual-attention multimodal fusion, simultaneously acquiring and preprocessing sky images and meteorological sequence data to ensure spatiotemporal alignment of multi-source information; it utilizes the MobileNetV2 network to efficiently extract spatial features from sky images, and then employs Dual-Attention... The Transformer (DAT) module processes the global attention branch and the local window attention branch in parallel and introduces the Local Convolutional Position Enhancement (LePE) mechanism to explicitly encode the relative position information between adjacent time steps in the spatial feature sequence of sky images. This reduces computational complexity while enhancing the spatiotemporal perception of cloud movement and distribution, overcoming the limitation of traditional Transformer models in neglecting local dynamics in long sequence modeling. An LSTM network is used to perform temporal modeling of environmental parameters such as temperature and humidity, statistically analyzing the changing characteristics of weather and environmental parameters on a short time scale. Through a cross-modal multi-head attention fusion module, image features are used as queries and meteorological features are used as keys to achieve dynamic alignment and adaptive weight allocation of multi-source information at the feature level. This enables the solar irradiance prediction model to accurately predict solar irradiance fluctuations in complex weather scenarios such as rapid cloud formation or localized severe convection. Attached Figure Description

[0007] Figure 1 This is a diagram of the multimodal deep learning framework for ultra-short-term global solar radiation prediction according to the present invention. Figure 2 This is a structural diagram of the Dual-Attention Transformer of the present invention; Figure 3Model comparison - MAE diagram for this invention; Figure 4 Model comparison - RMSE plot for this invention; Figure 5 The model comparison-nRMSE plot of this invention; Figure 6 This is a comparison chart of sunny and dry weather conditions according to the present invention; Figure 7 This is a cloud cover comparison chart for the present invention; Figure 8 This is a schematic diagram of the structure of the present invention; Detailed Implementation

[0008] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0009] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 like Figure 8 As shown, a solar irradiance prediction system based on dual-attention multimodal fusion includes: The data acquisition module is used to collect sky imaging observation images and meteorological observation data of the area to be predicted, and performs time alignment and data preprocessing operations on the sky imaging observation images and meteorological observation data to obtain preprocessed and time-synchronized sky imaging observation images and meteorological observation data. In some embodiments, the present invention can acquire sky image sequences through an infrared all-sky imager, acquire meteorological variables such as temperature, humidity, wind speed, wind direction, and atmospheric pressure through a meteorological monitoring device, and acquire solar irradiance measurements at corresponding times through a radiation sensor. All types of data are interpolated or resampled at a uniform time step, and denoising, missing value imputation, and numerical normalization are performed as needed to facilitate subsequent network training and inference.

[0010] The image feature extraction and temporal modeling module is used to extract spatial features from the preprocessed and time-synchronized sky imaging observation image using the MobileNetV2 network, and obtain the sky imaging temporal feature sequence. The image processing module is used to input the sky imaging temporal feature sequence in parallel to the global attention branch and local window attention branch of the DAT structure. The global attention branch of the DAT structure is used to perform adaptive aggregation processing on the sky imaging temporal feature sequence to obtain the global sky imaging temporal feature sequence. The local window attention branch of the DAT structure is used to perform local window attention calculation and window stitching processing on the sky imaging temporal feature sequence to obtain the local sky imaging temporal feature sequence. According to the local convolution position enhancement mechanism of the DAT structure and the sky imaging temporal feature sequence, the local convolution position encoding residual term is calculated. The global sky imaging temporal feature sequence and the local sky imaging temporal feature sequence are sequentially processed by channel stitching, linear fusion layer integration, and residual term introduction to obtain the aggregated sky imaging temporal feature sequence. The meteorological sequence modeling module is used to perform time-dependent modeling of meteorological observation data using a long short-term memory network to obtain the feature sequence of meteorological observation data. The multimodal attention fusion module is used to fuse the aggregated sky imaging time-series feature sequences and meteorological observation data feature sequences through a cross-attention mechanism to obtain a fused joint feature sequence for solar irradiance prediction.

[0011] like Figure 1 As shown, this invention designs a multimodal deep learning framework combining image and meteorological data, aiming to accurately predict future solar irradiance (GSI) values. The framework consists of an encoder that takes a sky image observation as input and a decoder that takes meteorological observation time-series data as input. At the encoder end: the input image first extracts basic features through a MobileNetV2 (a lightweight convolutional neural network) backbone network, and then feeds it into a Transformer (a deep learning transformer network) encoding stack integrating a Dual-Attention mechanism; each layer of the Transformer encoding stack, through Pre-LN (pre-layer normalization) residual connections, sequentially performs Masked Self-Attention (including Linear layers generating Q, K, V and a normalization function Softmax), Residual+LN (residual connections and layer normalization), Feed Forward (including Linear and activation functions), and other operations, finally outputting high-level image features F_image (i.e., At the decoder end: meteorological time series data (including solar radiance time series, i.e., raw meteorological sequence data) are input and flow into two processing branches in parallel. The solar radiance time series first calculates self-attention through a Masked Self-Attention module. The output of the Masked Self-Attention module is then processed by Residual+LN with residual connections and layer normalization before being input into the Cross Attention module. This module uses the image features F_image output by the encoder as keys (K) and values ​​(V) to achieve interaction between time series features and image features. Its output is again processed by Residual+LN and then fed into a Feed Forward network (composed of linear layers and nonlinear activations) for transformation. Finally, the Residual+LN module outputs the meteorological observation data feature sequence. Simultaneously, the raw meteorological sequence data... Input an independent LSTM block module, and the hidden state of the LSTM block module at the previous time step. With cell state For internal memory, the original meteorological sequence data is processed through an input gate activated by the σ (Sigmoid) function, a forget gate, an output gate, and a tanh activation function: the forget gate determines the... The information discarded in the input gate and the tanh function are used to generate new candidate information. The current cell state is obtained by combining the two. The output gate is based on The result after tanh scaling generates the current hidden state. This completes a cyclic update for one time step, constructing a complete time-dependent modeling path. Hidden states updated by LSTM block. The image is projected onto a meteorological feature Hmet through a Fully Connected layer. This meteorological feature Hmet is then input as a key-value pair to the Cross Attention module in the decoder stack. The image feature F_image and the meteorological feature Hmet undergo linear transformation through a Linear transformation layer, followed by a series of processes including Cross Attention, Concat, Inear (linear transformation), Feed Forward, Residual+LN, and Concat. Finally, the top-level MLP module performs the final irradiance value regression prediction.

[0012] The model consists of two main channels: an image channel and a meteorological channel, and processes image and meteorological data collaboratively through a cross-modal information interaction module. The overall processing flow is as follows: Sky imaging data and meteorological observation data corresponding to the area to be predicted are collected, and time alignment, outlier removal and normalization are performed on data from different sources to construct a time-synchronized multi-source observation sequence.

[0013] In the image channel, the input satellite cloud image is first processed through the MobileNetV2 network to extract the spatial features of the image; To further enhance the model's spatiotemporal perception of cloud distribution and motion, a DAT (Discretionary Attention) structure is designed. This structure integrates local and global attention mechanisms. Local attention captures detailed information in the image, while global attention enhances the model's understanding of the spatiotemporal evolution characteristics and distribution patterns of the cloud field. In the meteorological channel, the model employs an LSTM architecture to process meteorological time series data. Through the gating mechanism of LSTM, the model can accurately model the time-varying changes in historical meteorological data and extract time-series features, providing support for the prediction of GSI values.

[0014] To achieve cross-modal interaction between images and meteorological features, the model incorporates a Cross-Attention fusion module based on multi-head attention. This module learns the complex relationship between image spatial features and meteorological temporal features in multiple subspaces, thereby enhancing information fusion capabilities and improving GSI prediction accuracy.

[0015] Some technical solutions utilize the MobileNetV2 network to extract spatial features from preprocessed and time-synchronized sky imaging observation images to obtain a temporal feature sequence of sky imaging. The sequence of sky imaging observation images is as follows , , The number of channels is 1, and the height is , width is The set of real numbers in sky imaging observations. The time step for acquiring sky imaging observation images. This is a sky imaging observation image collected at the first moment. This is a sky imaging observation image acquired at the second moment. For the first Sky imaging observations are collected at all times, and spatial features are extracted from these sky imaging observations using MobileNetV2: ; in, for The spatial feature map of the sky corresponding to the sky imaging observation map at a given time. for Sky imaging observation image at that moment, This is the feature extraction function for the MobileNetV2 backbone network. The output feature dimension of the spatial features of sky imaging. for The height of the spatial feature map of the sky at any given time. for The width of the spatial feature map of the sky at any given time. for Spatial resolution of the spatial feature map of the sky at any given time. The dimension of the output feature is represented as Spatial resolution is The set of real numbers for the spatial feature maps of sky imaging. It belongs to the symbol; The sky imaging observation map is an infrared image. The MobileNetV2 network is a depthwise separable convolutional structure. Using MobileNetV2 to extract image features from the sky imaging observation map can extract spatial features rich in semantic information in an efficient and lightweight way. The sky imaging observation map is compressed into a low-dimensional (two-dimensional feature map with height and width) and high-information-density sky imaging spatial feature map (i.e., it can maximize the preservation of the image features of the sky imaging spatial feature map and reduce the amount of parameter calculation required for subsequent image processing, thereby reducing the computational cost.

[0016] The spatial dimension of the sky imaging spatial feature map is compressed using a global average pooling method: ; in, After global average pooling Sky spatial feature map at time of day. This is the global average pooling function. for Sky imaging spatial feature map at time of day in spatial location The feature vector at that location, The dimension of the output feature is represented as The set of real numbers for the spatial feature map of the sky imaging is used. In order to further reduce the computational cost, the spatial dimension of the spatial feature map of the sky imaging is compressed by the global average pooling method, which transforms the two-dimensional sky imaging spatial feature map into a compact one-dimensional sky imaging spatial feature map. The global average pooling method effectively preserves the spatial feature information of each feature channel while significantly reducing the computational complexity and the number of parameters.

[0017] After global average pooling is transformed by linear transformation Sky imaging spatial feature map at time of moment Projected onto a fixed dimension In the space, construct a temporal feature sequence of sky imaging: ; in, After linear transformation Sky spatial feature map at time of day. This is the weight matrix of the fully connected layer. This is the bias vector for the fully connected layer. To characterize the output feature dimension is The set of real numbers for the spatial feature maps of sky imaging. This is a temporal feature sequence of sky imaging. This is a temporal feature map of the sky image at the first moment. This is the temporal feature sequence of sky imaging at the second time step. For the first Temporal feature map of sky imaging at time 10:00. Characterizing time period The internal output feature dimension is The set of real numbers for the spatial feature map of the sky imaging.

[0018] The purpose of projecting the globally average pooled sky imaging spatial feature map into a fixed dimension by using linear transformation is to unify the dimensional representation of the linearly transformed sky imaging spatial feature map at different times, thus unifying the sky imaging spatial feature map after linear transformation at different times.

[0019] The structure of the Dual-Attention Transformer (DAT) module is as follows: Figure 2 As shown, the DAT module aims to extract long-range global context and high-resolution local details in parallel from the input feature sequence. The core architecture consists of a global attention branch and a local window attention branch, and integrates spatial location information through a dynamic location enhancement mechanism to generate richer feature representations. Under the multi-head attention mechanism, let the number of attention heads be... The dimensions of each head are DAT divides the attention head into two parts: the number of global branch heads. and the number of local branch heads The entire module employs a Pre-LN residual structure to stabilize training and accelerate convergence. The DAT module uses a parallel dual-branch structure to process the input sky imaging temporal feature sequence. The global attention branch first performs a deep sampling operation on the sky imaging temporal feature sequence. The deep-sampled sky imaging temporal feature sequence is then independently projected by multiple sets of Linear (linear transformation) layers and input into the Attention module for computation. The resulting output sky imaging temporal feature sequences from multiple global attention heads are concatenated along the feature dimension using a Concat operation. The concatenated sky imaging temporal feature sequence is then integrated by projection from a Linear layer to output the global sky imaging temporal feature sequence. The local window attention branch first performs a windowing operation on the sky imaging temporal feature sequence, dividing it into multiple local windows. The feature sequence within each window is projected by multiple sets of Linear layers and input into an independent Attention module for in-window self-attention computation. The output feature sequences of all windows are concatenated temporally by a Concat operation and then projected and fused through a Linear layer to obtain the local sky imaging temporal feature sequence. The local augmentation location encoding branch transforms the temporal feature sequence of sky imaging through a linear layer, inputs it into DWConv (depth-separable convolution) to generate location-aware features, and then feeds the residuals into the global attention branch and the local window attention branch. The outputs of the global attention branch and the local window attention branch are concatenated and linearly fused along the channel dimension in the Linear Fusion layer, and then added to the residuals of the local augmentation location encoding branch to obtain an augmented feature representation that aggregates global context, local details, and explicit location information.

[0020] Some technical solutions utilize the global attention branch of the DAT structure to adaptively aggregate the temporal feature sequence of sky imaging to obtain the global sky imaging temporal feature sequence. The global attention branch aims to capture input features. This addresses long-range temporal dependencies in the data while reducing computational complexity through downsampling and adaptive aggregation. To further reduce computational overhead, a global attention branch is applied to the temporal feature sequences of sky imaging. Downsampling is performed on the time axis: ; in, This is a sequence of temporal features of downsampled sky imaging. For downsampling function, To reduce the time step after sampling, To reduce sampling hyperparameters, This represents the number of samples processed in parallel batches. A set of temporal feature sequences of sky imaging with feature channel C after downsampling; Global attention branch has The global attention head, for the first Global attention heads ( The corresponding sky imaging temporal feature sequence is generated through linear projection. Global query vector, global key vector, and global value vector: ; in, For the first A global query vector for a global attention head. For the first Global key vectors of global attention heads For the first The global value vector of each global attention head. For the first Global query projection weight matrix of each global attention head For the first Global key projection weight matrix of each global attention head For the first The global value projection weight matrix of the i-th global attention head; The global query vector of each global attention head is composed of the temporal feature sequence of sky imaging. The calculation yielded that the first The global key vector of the first global attention head and the first global attention head The global value vector of each global attention head is calculated from the downsampled sky imaging temporal feature sequence, thereby achieving downsampling of the global key vector and global value vector for all attention heads in the global attention branch, reducing computational overhead. Maintaining the original temporal resolution of the global query vector ensures the detailed integrity of the sky imaging temporal feature sequence, while downsampling the global key vector and global value vector compresses the length of the sky imaging temporal feature sequence, enabling the solar radiation prediction model to efficiently capture long-distance temporal dependencies in the sky imaging temporal feature sequence (i.e., the correlation between current weather state data and past weather state data).

[0021] Each global attention head has its own independent set of learnable projection matrices. During training, different global attention heads will spontaneously learn to focus on different aspects of the input sky imaging temporal feature sequence. For example, some global attention heads are responsible for capturing the overall movement trend and direction of clouds in the sky; some global attention heads are responsible for analyzing the slow, periodic changes in cloud thickness or brightness; and some global attention heads are responsible for detecting sudden abnormal changes or short-term fluctuations in the sky imaging temporal feature sequence.

[0022] The first value is obtained by calculating the scaled dot product attention mechanism. Global attention weights for each global attention head: ; in, Let be the dimension of each global attention head in the global attention branch. For normalization function, for The global attention weights of each global attention head. for The global attention head weighted output of the size The transpose operator is used; each global attention head, through the scaling dot product attention mechanism, calculates the correlation between the global query vector and the global key vector, and then performs numerical scaling and normalization to obtain the corresponding global attention weight matrix; the global attention weight matrix performs weighted aggregation on the historical sky infrared image features at different times, and filters out the long-distance spatiotemporal dependency information with the most predictive value for the current weather state (i.e., the weighted output of the global attention head weights).

[0023] Will The weighted outputs of the global attention heads are concatenated and then projected through the global output matrix. The global sky imaging temporal feature sequence was obtained: ; in, This is a temporal feature sequence of global sky imaging. To be The weighted outputs of the global attention heads are concatenated along the feature channel dimension.

[0024] By stitching together the weighted outputs of multiple attention heads computed in parallel and then fusing them through a projection matrix, a global sky imaging temporal feature sequence is formed that can simultaneously capture multiple weather evolution patterns such as cloud movement, thickness changes, and brightness fluctuations. This enhances the ability to trace and predict the causes of minute-level fluctuations in solar irradiance in complex meteorological scenarios.

[0025] Some technical solutions utilize the local window attention branch of the DAT structure to perform local window attention calculation and window stitching processing on the temporal feature sequence of sky imaging to obtain the local sky imaging temporal feature sequence. The local window attention branch has multiple local window network layers, each of which stores the temporal feature sequence of sky imaging along the time axis. Divided into There are [number] timing windows, each with a length of [length]. : ; in, For the first A set of timing step indexes for each timing window. This represents the total number of timing windows; Local window attention branch has The local attention head, for the first... The first timing window Each local attention head generates a corresponding temporal feature sequence of sky imaging through linear projection. Local query vector, local key vector, and local value vector: Local window attention is a computationally efficient attention mechanism that segments the temporal feature sequence of sky imaging along the time axis and computes attention within local windows to efficiently capture short-range dependencies and local details in the sequence. The local window attention branch has multiple stacked network layers, each containing a local window attention sublayer, and each sublayer has multiple local attention heads. Each layer in the stacked network structure of the local window attention branch performs deeper feature processing based on the output of the previous layer. Each local attention head has a unique learnable local projection weight matrix (including local query projection weight matrix, local key projection weight matrix, and local value projection weight matrix). This learnable local projection weight matrix spontaneously differentiates during training, focusing on different aspects of the information within a single temporal window. For example, when analyzing a window containing several consecutive frames of sky images: the first local attention head is responsible for capturing the movement vectors of cloud edges in the infrared sky image within the temporal window; the second local attention head is used to analyze the continuous change trend of image brightness within the temporal window; and the third local attention head is used to detect specific texture patterns (such as cirrus cloud texture) within the temporal window.

[0026] ; in, For the local window attention branch, the local window network layer index is... For the first Layer, First The first timing window, the first Local query vectors of local attention heads For the first Sky imaging temporal feature sequence within a time window For the first Layer, First The first timing window, the first Local key vectors of a local attention head For the first Layer, First The first timing window, the first Local value vectors of local attention heads, For the first Layer, First Local query projection weight matrix of each local attention head For the first Layer, First Local bond projection weight matrix of a local attention head For the first Layer, First The local value projection weight matrix of each local attention head; By performing independent linear projection operations, the local query vector, local key vector, and local value vector of each local attention head in each network layer are obtained in each temporal window. This enables different attention heads to adaptively focus on different feature information of the sky imaging temporal feature sequence input in each temporal window. When calculating local attention, multi-view and fine-grained feature interaction is achieved, which improves the ability of local attention branches to represent diverse short temporal dependency patterns (i.e., the strong correlation between the sky imaging temporal feature sequence at a time point and the sky imaging temporal feature sequences at a limited number of neighboring previous time points in the time series).

[0027] No. Local attention weights within a time window and output for: ; in, For the first Layer, First The first timing window, the first Local attention head weighted output For the first Layer, First The first timing window, the first The local attention weight matrix of each local attention head. For the dimension of the local attention head, This is the matrix transpose operator. This is the normalization function; Calculating local attention weights and generating weighted outputs within a time window enables the multi-head mechanism of local attention branches to extract diverse local features from different subspaces and establish short-distance dependencies between adjacent time steps. Limiting computation to a single time window can reduce computational complexity.

[0028] The temporal feature maps of sky imaging output from all temporal windows and local attention heads are stitched together, and then projected through the local output matrix. Obtain the temporal feature sequence of local sky imaging : ; in, To stitch together the output of all windows along the spatial dimension, To stitch together the outputs of all attention heads along the channel dimension, For the first The local output projection weight matrix of the layer, For the first The total number of attention heads in the layer.

[0029] By concatenating the weighted outputs of multiple local attention heads computed in parallel within each time window, we can preserve the diverse short-term local features extracted by each attention head (such as subtle changes in cloud texture, edge motion vectors, brightness gradients, etc.) and achieve feature dimensionality reduction and cross-attention head information interaction through the local output projection matrix, thus obtaining a local sky imaging time-series feature sequence.

[0030] Some technical solutions involve calculating the local convolution position encoding residual term based on the local convolution position enhancement mechanism of the DAT structure and the temporal feature sequence of sky imaging. The local convolutional location enhancement mechanism processes the input temporal feature sequence of sky imaging through pointwise linear transformation and depthwise separable convolution. ,get: In the formula, Let be the pointwise linear transformation matrix. For point-by-point linear transformation bias term, For depthwise separable convolution, Encode residual terms for local convolution locations.

[0031] Local Convolutional Position Enhancement (LePE) is a positional encoding technique. Unlike global and local attention mechanisms, which disregard the input order of sky imaging temporal feature sequences and can still yield the same results even with shuffled input order, LePE addresses this by introducing a parameterized residual branch based on depthwise separable convolutions. This branch injects relative positional information from the local context into the sky imaging temporal feature sequence, enhancing the ability of both global and local attention mechanisms to capture the temporal structure and dynamic evolution of sky imaging temporal feature sequences.

[0032] In some technical solutions, the methods for obtaining the aggregated sky imaging time-series feature sequence by sequentially performing channel stitching, linear fusion layer integration, and residual term introduction on the global sky imaging time-series feature sequence and the local sky imaging time-series feature sequence include: global sky imaging temporal feature sequence Temporal feature sequences of local sky imaging The components are stitched together along the channel dimension and then processed through a linear fusion layer. Integration, while incorporating local convolutional position encoding residual terms. Obtain the temporal feature sequence of sky imaging output from the DAT structure. : ; The outputs of the global attention branch and the local branch are concatenated along the channel dimension, and a linear fusion layer is used to integrate multi-scale features, fusing long-distance and short-distance temporal dependence features to obtain a sky imaging temporal feature sequence representation with complementary information. At the same time, the residual term of Local Convolutional Position Enhancement (LePE) is introduced to enhance the encoding of temporal position information of the sky imaging temporal feature sequence, resulting in a sky imaging temporal feature sequence representation with complementary information and stronger temporal position awareness, thus improving the modeling accuracy and robustness of complex spatiotemporal patterns.

[0033] Will The layered DAT module is embedded into the standard Transformer structure through Pre-LN residual connections, resulting in a multi-layered stacked DAT and Transformer encoder network. ; The DAT and Transformer encoder networks refine the temporal feature sequences of sky imaging layer by layer. Each layer first processes the current input state. The (sky imaging temporal feature sequence) undergoes layer normalization to stabilize its data distribution. The normalized sky imaging temporal feature sequence is then input into a dual-attention transformation module, which integrates the global context and local details of the sky imaging temporal feature sequence in parallel, achieving deep interaction and enhancement of the sky imaging temporal features. The enhanced sky imaging temporal feature sequence is then connected to the initial input state via residual connections. Adding them together generates an intermediate state that contains both the original information and new knowledge. Intermediate state After layer normalization and flattening, the data is input into a feedforward neural network. The feedforward neural network consists of two learnable linear transformations and a GELU activation function. It performs independent and complex nonlinear transformations on the features at each location in the temporal features of sky imaging. The results of the linear transformations are also presented as residuals and intermediate states. Adding them together yields a richer and more abstract new state. This serves as the input for the next layer. By stacking multiple such processing layers, the original features... It is progressively abstracted and enhanced, gradually transformed into a high-level representation containing complex spatiotemporal context information. .

[0034] The sky imaging temporal feature sequence obtained from the DAT and Transformer encoder network By averaging and converging over time, we obtain: ; The time-axis averaging convergence method averages the sky imaging temporal feature sequences obtained from the DAT and Transformer encoder networks along the time dimension T, transforming the sky imaging temporal feature sequences into fixed-dimensional image modality feature vectors that represent global information within the entire time window. .

[0035] in, This is the temporal feature sequence of sky imaging after processing by the first DAT sublayer. For the first Sky imaging temporal feature sequence of layers, It is a dual attention switching module. For layer normalization function, for Sky imaging temporal feature sequence of layers, It is a feedforward neural network. For activation function, These are the feature vectors of the feedforward neural network. This is the first learnable weight matrix of the feedforward neural network. This is the second learnable weight matrix of the feedforward neural network; To aggregate the temporal feature sequences of sky imaging, The time step for acquiring sky imaging data. For the DAT and Transformer encoder network at time t The temporal feature sequence of sky imaging output from the layer.

[0036] The Transformer architecture is a deep learning model architecture based entirely on the self-attention mechanism, capable of processing the entire temporal feature sequence of sky imaging in parallel using its powerful long-range dependency modeling capabilities. Pre-LN (Pre-Layer Normalization) is a layer normalization preprocessing used to define the computation order within each sub-layer (such as the self-attention layer and the feedforward network layer) in the Transformer.

[0037] Some technical solutions employ Long Short-Term Memory (LSTM) networks to model the time dependence of meteorological observation data, thereby obtaining the characteristic sequences of the meteorological observation data. These methods include: In short-to-medium-term meteorological time series modeling, the evolution of meteorological variables exhibits clear continuity and causality. Unlike self-attention mechanisms that primarily rely on location encoding to capture sequential relationships, this scheme employs a Long Short-Term Memory (LSTM) network in the meteorological branch. LSTM, through its gating structure and recursive states, endogenously models time dependencies, better reflecting the statistical characteristics of meteorological data on short time scales. The original meteorological time series, composed of meteorological observation data, is as follows: , Given B independent samples, each sample being a sequence of length T, with each time step in the sequence represented by a set of real numbers in a d-dimensional real vector, perform MinMax scaling on the original meteorological time series: ; in, For normalized meteorological observation data, These are the minimum and maximum normalization functions; MinMax scaling is a data normalization method used to linearly scale raw meteorological time-series data to a fixed interval, typically [0, 1] or [-1, 1]. The numerical ranges and units of different features (such as temperature ℃, humidity %, and air pressure hPa) in the raw meteorological time-series data vary greatly. If directly input into a Long Short-Term Memory (LSTM) network, features with large numerical ranges (such as air pressure of 1000) will dominate gradient updates, while the effects of features with small numerical ranges (such as humidity of 0.6) will be suppressed. MinMax scaling normalizes the raw meteorological time-series data, facilitating efficient processing of the data.

[0038] An LSTM network has L layers of stacked LSTM units (i.e., cells of the LSTM network), with each layer having the following hidden states: (This refers to the collective term for hidden states, including...) and ), LSTM network The forgetting gate at time t is used to determine the cell state at the previous time step. The formula for calculating which meteorological observation data needs to be retained or discarded is as follows: ; in, This is the output of the forget gate, with values ​​ranging from (0,1); It is the Sigmoid activation function. The meteorological observation data is for the current time t. This is the weight matrix of the meteorological observation data at the current time t, representing the forgetting gate. Here is the weight matrix of the meteorological observation data at time t-1. For the bias term of the forget gate, This represents the hidden state at time t-1; The input gate of the LSTM network determines the meteorological observation data that should be introduced at the current moment and writes it into the cell state: ; ; ; in, This is the input gate output, with a value range of (0,1). Let be the weight matrix of the meteorological observation data of the input gate at the current time t. The weight matrix is ​​the meteorological observation data at time t-1 of the input gate. This is the bias term for the input gate; Candidate cell state, The hyperbolic tangent activation function is used. This is the weight matrix of meteorological observation data under candidate cell states. This is the weight matrix of meteorological observation data under candidate cell states. This is the input gate bias term for the candidate cell state; This represents the updated cell state. The cell state at time t-1. This is the Hadamard product algorithm; The output gate is used to determine which information from the current cell state will be passed to the hidden layer. Its calculation formula is as follows: ; ; in, This is the activation result of the output gate. To output the meteorological observation data at the current moment, This is the weight matrix corresponding to the hidden state in the previous time step. This is the bias term corresponding to the hidden state in the previous time step; It is the hidden state at the current time t; After stacking layers, the final hidden state of the top layer is taken as the temporal representation: ; in, For meteorological observation data characteristic sequences, This represents the hidden state of the final LSTM layer at the last moment.

[0039] LSTM networks, consisting of L layers of stacked LSTM units (i.e., the cells of the LSTM network), are a special type of recurrent neural network used to address the weakness of traditional RNNs in learning long-term dependencies when processing long sequences of data. LSTM networks introduce cell states; each LSTM cell in a layer uses a forget gate to view the meteorological observation data currently being input. The hidden state of the cell at the previous moment A forgetting coefficient is generated for each piece of information in the cell state, determining the information that needs to be forgotten from the cell state; candidate cell states. Depend on Activation function generation, representing potential new information, input gate Controlling candidate cell state How much of it is added to the new cell state? Controlling candidate cell state How many were added to the new cell state? In the middle; new cellular state First pass Function scaled to [-1, 1], output gate Controlling scaling How many are currently hidden? Output.

[0040] Based on the forgetting gate mechanism activated by the Sigmoid function, the LSTM network can dynamically evaluate the importance of various meteorological observation data in historical cell states and generate continuous weights between 0 and 1 for selective forgetting, thereby achieving adaptive adjustment of long-term dependencies in long sequences and enhancing the LSTM network's ability to remember and transmit key dynamic patterns in meteorological time series data.

[0041] Some technical solutions involve fusing aggregated temporal feature sequences of sky imaging and feature sequences of meteorological observation data through a cross-attention mechanism to obtain a fused joint feature sequence. A cross-attention mechanism is introduced to linearly project the aggregated sky imaging time-series feature sequence and meteorological observation data feature sequence to obtain the query vector Q, key vector K, and value vector V. The feature fusion between the aggregated sky imaging time-series feature sequence and meteorological observation data feature sequence is transformed into attention calculation with the sky imaging time-series feature sequence as the query and the meteorological observation data feature sequence as the key-value pair. ; in To aggregate the temporal feature sequences of sky imaging, To query the vector projection weight matrix, For meteorological observation data characteristic sequences, The weight matrix is ​​the projection of the key vectors. The weight matrix is ​​the projection of the value vector; Using image feature sequences as queries (Q) and meteorological feature sequences as keys (K) and values ​​(V), this method employs a cross-attention mechanism to treat image modalities as the subject of information retrieval. It selectively extracts the temporal environmental features (such as temperature and humidity trends) most relevant to the current sky image content from meteorological modalities, thereby capturing the complex nonlinear relationship between images and meteorological data. This avoids information redundancy or loss caused by simple information splicing or weighted averaging, and enhances the ability to collaboratively utilize multi-source heterogeneous information in minute-level solar irradiance prediction.

[0042] A multi-head attention mechanism is used to fuse the temporal feature sequence of sky imaging and the feature sequence of meteorological observation data, and each attention head in the multi-head attention mechanism is mapped separately into a low-dimensional subspace: ; in, For the first A low-dimensional query vector of attention heads For the first A low-dimensional key vector of an attention head. For the first A low-dimensional value vector of attention heads, For the first Each attention head can learn the query projection matrix. For the first A learnable key projection matrix for each attention head. For the first The attention head can learn the value projection matrix. Here, a multi-head attention mechanism is used to map each attention head to a low-dimensional subspace, which enables the parallel learning of different types of interaction patterns in multiple independent feature subspaces. This enhances the ability to capture features from multiple perspectives of sky imaging time-series feature sequences and meteorological observation data feature sequences. The low-dimensional mapping significantly reduces the computational complexity of each attention head and improves the overall computational efficiency.

[0043] Calculate the first Scaling dot product attention for each attention head: ; ; in, For the first The scaled dot product attention weight matrix of each attention head. It is a flexible maximum value function. For the first The dimension of attention heads For the first The query vector of each attention head transpose, For the first The output of each attention head computes scaled dot product attention, which assigns a normalized, focused weight distribution to each position in the query sequence, thereby dynamically identifying and enhancing the most relevant information from the key-value sequence, ultimately achieving accurate and adaptive feature fusion and context enhancement.

[0044] Calculating scaled dot product attention is the core step of the cross-modal multi-head attention fusion mechanism. It uses image feature sequences as queries and meteorological feature sequences as keys and values. It calculates the original relevance score between each query and key pair through dot product operation, and then normalizes it by scaling factor to maintain training stability. Finally, it normalizes the score into a probabilistic attention weight distribution through the Softmax function, so that each position in the query sequence can dynamically and adaptively focus on the most relevant part of the key and value sequences, providing a highly correlated joint feature representation for subsequent solar irradiance prediction.

[0045] The outputs of multiple attention heads are concatenated and projected to obtain the cross-attention outputs of the multiple attention heads. : ; in, To be The outputs of each attention head are concatenated. It is a cross-attention projection matrix; it effectively aggregates the diverse feature patterns extracted by multiple attention heads, and achieves cross-subspace information fusion and dimensionality reduction through a learnable projection matrix, resulting in a unified feature representation with stronger representation capabilities and higher information density.

[0046] Cross-attention output Further corrections are made through normalization, residual connections, and feedforward networks: in, This is a fusion feature between attention output and aggregated temporal feature sequences of sky imaging. It is a feedforward neural network. This refers to the fusion feature of the cross-attention output corrected by the feedforward network and the aggregated temporal feature sequence of sky imaging. This is the projection weight matrix used to match the input dimension skip connections; through the initial cross-attention output... By performing layer normalization, introducing dimension-matched residual connections, and enhancing nonlinear feedforward networks, fusion features can be formed that combine training stability, preservation of original information, and strong deep expressive capabilities.

[0047] The fusion features of the cross-attention output after feedforward network correction and the aggregated temporal feature sequence of sky imaging Pooling is performed along the query dimension to obtain the fused feature vector of sky imaging and meteorological data: ; in, Features for the fusion of sky imaging and meteorological data The pooling function based on the Lq norm performs pooling compression on the query dimension of the cross-modal fused feature sequence, aggregating the variable-length sequence containing rich spatiotemporal interaction information into a fixed-dimensional fused feature vector.

[0048] The feature vector fused from sky imaging and meteorological data is then combined with the feature sequence of meteorological observation data. After splicing, input the prediction head : ; in, This is the final fused joint feature sequence. For concatenation functions, This represents the predicted solar irradiance value. The last layer of the prediction head uses Sigmoid activation to ensure the output aligns with the normalized target range. After fusion, the joint features can be pooled or dimensionality reduced in the time dimension to obtain a fixed-dimensional feature vector for subsequent prediction. The fused feature vector from sky imaging and meteorological data is then concatenated with the feature sequence of meteorological observation data to obtain a comprehensive feature containing global information from multiple modalities (sky infrared images and meteorological data). This comprehensive feature is then used by the prediction head to generate accurate solar irradiance predictions using visual temporal features and meteorological environmental data.

[0049] In some embodiments, the input data required by this invention includes infrared observation image sequences, meteorological observation time series data, and global solar irradiance measurements at corresponding times. The aforementioned data can come from equipment such as infrared all-sky imagers, meteorological monitoring stations, and photometers conventionally deployed at photovoltaic power plant sites, or from other imaging or meteorological observation devices with equivalent functions. This invention does not limit image resolution, sampling interval, sensor model, or specific installation method; it only requires ensuring the synchronization and continuity of various observation data along the time axis to meet the requirements of multimodal modeling. Infrared images are used to characterize the spatial structure and dynamic evolution of clouds, meteorological sequences reflect the changing trends of environmental variables such as temperature, humidity, wind speed, wind direction, and atmospheric pressure, and irradiance measurements are used to provide supervisory signals for training and validation. Before using the data, conventional data cleaning, missing value completion, time alignment, outlier removal, and scale normalization can be performed on the imaging data and meteorological sequences to improve data quality and model stability. Weather conditions can be classified based on infrared image brightness distribution, cloud cover index, sky coverage, or other empirical rules to distinguish between clear sky and cloud-dominated scenarios, thus evaluating the model's predictive performance under different weather conditions. This invention does not rely on any specific dataset, sensor, or weather classification algorithm; it only requires input of multi-source synchronous observation data to complete the prediction task. Therefore, it has good substitutability and engineering feasibility, and can be applied to various photovoltaic power plants, meteorological monitoring scenarios, and new energy prediction systems.

[0050] The experimental setup was as follows: To verify the effectiveness and fairness of the proposed method, all experiments were conducted on a single GPU. To prevent overfitting during training, an early stopping strategy was employed, and the patience period was set to 7 training epochs. The Adam optimizer was selected, and the mean squared error loss was used as the optimization objective. The main hyperparameters of the model's network structure are shown in Table 1.

[0051] Table 1 Parameter settings for the network model The evaluation metrics are set as follows: To measure the predictive performance of the model, this invention uses three evaluation metrics, including root mean square error (RMSE), mean absolute error (MAE), and root mean square error (nRMSE). These three evaluation metrics are defined as follows: In the formula, This represents the predicted value of solar irradiance. This represents the true value of solar irradiance. This represents the average value of the true solar irradiance, where n is the total number of data samples.

[0052] like Figure 3 As shown, the evaluation indicator is MAE (unit: ( ) The figure compares the performance of different solar irradiance prediction models under two typical weather conditions: sunny (orange bars) and cloudy (blue bars). It primarily showcases the solar irradiance prediction performance of six models: FEDformer (Fourier Enhanced Decomposition Transformer model), Informer (a time series prediction model), Autoformer (Decomposition Transformers with Auto-Correlation), Reformer, Cross-ViViT (Cross-attention Multi-scale Vision Transformer), and the MDAT-LSTM model of this invention (a solar irradiance prediction model based on dual attention multimodal fusion). The figure shows that the MAE value of all models under Cloudy conditions is significantly higher than that under Sunny conditions, indicating that solar irradiance prediction is more difficult under variable cloud cover. Among the six models listed, the MDAT-LSTM model has the lowest MAE value under both weather conditions, indicating that the MDAT-LSTM model has the smallest prediction error and the highest accuracy.

[0053] Similarly, such as Figure 4 As shown, the evaluation indicator is RMSE (unit: ( ) The figure compares the performance of different solar irradiance prediction models under two typical weather conditions: sunny (orange bars) and cloudy (blue bars). It primarily shows the solar irradiance prediction performance of six models: FEDformer (Fourier Enhanced Decomposition Transformer model), Informer (a time series prediction model), Autoformer (Decomposition Transformers with Auto-Correlation), Reformer, Cross-ViViT (Cross-attention Multi-scale Vision Transformer), and the MDAT-LSTM model of this invention (a solar irradiance prediction model based on dual attention multimodal fusion). The figure shows that the MAE value of all models under the Cloudy condition is significantly higher than their RMSE value under the Sunny condition, indicating that solar irradiance prediction is more difficult under variable cloud conditions. Among the six models listed, the MDAT-LSTM model has the lowest RMSE value under both weather conditions, indicating that the MDAT-LSTM model has the smallest prediction error and the highest accuracy. Similarly, such as Figure 5 As shown, the performance comparison of different solar irradiance prediction models under two typical weather conditions—sunny (orange bars) and cloudy (blue bars)—is presented with nRMSE (%) as the evaluation metric. The figure mainly displays the solar irradiance prediction performance of six models: FEDformer (Fourier Enhanced Decomposition Transformer model), Informer (a time series prediction model), Autoformer (Decomposition Transformers with Auto-Correlation), Reformer, CrossViViT (Cross-attention Multi-scale Vision Transformer), and the MDAT-LSTM model of this invention (a solar irradiance prediction model based on dual attention multimodal fusion). The figure shows that the nRMSE value of all models under Cloudy conditions is significantly higher than that under Sunny conditions, indicating that solar irradiance prediction is more difficult under variable cloud cover. Among the six models listed, the MDAT-LSTM model has the lowest nRMSE value under both weather conditions, indicating that the MDAT-LSTM model has the smallest prediction error and the highest accuracy.

[0054] Figure 6The solar irradiance prediction curve for moderate clear skies includes the vertical axis GSI (unit: W / m²). 2 (Solar irradiance units), horizontal axis Time Step, and True GSI (Shaded) representing the true solar irradiance value, blue dashed line representing the single-modality (Informer model) solar irradiance prediction curve, orange dashed line representing the single-modality (FFDformer model) solar irradiance prediction curve, green dashed line representing the single-modality (Autofommer model) solar irradiance prediction curve, purple solid line representing the Cross-ViViT dual-modality solar irradiance prediction curve, and red solid line representing the LSTM-Simple dual-modality solar irradiance prediction curve. (LSTM-Simple)”; As can be seen from the main graph curve and the enlarged inset in the lower right corner, the multimodal method prediction curve, represented by the red and purple solid lines, is closer to the gray true value range of solar irradiance in terms of overall trend and detailed fluctuations compared to the other single-modal prediction curves. Especially in the enlarged area from time step 310 to 340, the multimodal curve can more stably follow the changes of the true value, while the single-modal curve shows more obvious fluctuations and deviations. This indicates that the multimodal method, which integrates information from multiple sources, can achieve more accurate and stable solar irradiance prediction technology results compared to methods that use only a single data source.

[0055] Figure 7 The solar irradiance prediction curve for moderately cloudy weather includes the vertical axis GSI (unit: W / m²). 2(Solar irradiance units), horizontal axis Time Step, and True GSI (Shaded) representing the true value of solar irradiance, blue dashed line representing the single-modality (Informer model) solar irradiance prediction curve, orange dashed line representing the single-modality (FFDformer model) solar irradiance prediction curve, green dashed line representing the single-modality (Autofommer model) solar irradiance prediction curve, purple solid line representing the Cross-ViViT dual-modality solar irradiance prediction curve, and red solid line representing the LSTM-Simple dual-modality solar irradiance prediction curve (the LSTM baseline model used for comparison). (LSTM-Simple)”; As can be seen from the main graph curve and the enlarged inset in the lower right corner, the multimodal method prediction curve, represented by the red and purple solid lines, is closer to the gray true value range of solar irradiance in terms of overall trend and detailed fluctuations compared to the other single-modal prediction curves. Especially in the enlarged area from time step 160 to 190, the multimodal curve can more stably follow the changes of the true value, while the single-modal curve shows more obvious fluctuations and deviations. This indicates that the multimodal method, which integrates information from multiple sources, can achieve more accurate and stable solar irradiance prediction technology results compared to methods that use only a single data source.

[0056] The experimental comparisons are as follows: To verify the effectiveness of the proposed multimodal deep learning ultra-short-term global solar irradiance prediction model based on sky images and meteorological sequences, we selected eight typical benchmark methods for comparative experiments. Specifically, the sky image prediction methods included the CNN, ConvLSTM, and MobileNetV2-Dual-AttentionTransformer (MDAT) models proposed in this paper; the meteorological sequence prediction methods included RNN, LSTM, GRU, and Transformer models; and the multimodal deep learning methods employed the MDAT-GRU and MDAT-LSTM models proposed in this paper. The comparative performance results of all models on the dataset are shown in Table 2.

[0057] Table 2 Comparison of Predictions Using Typical Benchmark Methods Based on the experimental results shown in Table 2, the following conclusions can be drawn: a. In image sequence modeling, MDAT significantly outperforms traditional baseline models. In sky image-based prediction tasks, the MDAT model achieves state-of-the-art results under both sunny and cloudy conditions. Under sunny and cloudy conditions, the RMSE of the MDAT model are 12.48 and 22.55, respectively, which are approximately 33.8% and 36.5% lower than CNNs, and approximately 83.8% and 74.7% lower than ConvLSTMs. This indicates that MDAT, by combining a lightweight convolutional structure with a dual attention mechanism, enhances its global spatiotemporal dependency modeling capabilities while maintaining efficient feature extraction. This allows the model to capture cloud distribution and motion information more comprehensively, thereby significantly improving the accuracy and stability of irradiance prediction.

[0058] b. Among all meteorological sequence prediction models, the LSTM model exhibits the best overall performance. Under sunny conditions, its RMSE is 3.74, 18.9% lower than RNN and 7.7% higher than GRU. Under cloudy conditions, its RMSE increases to 23.55, but is still 33.0% lower than RNN and 19.9% ​​lower than GRU. These results indicate that LSTM effectively captures the temporal dependence of meteorological factors through a gating mechanism, improving its adaptability to non-steady-state weather dynamics. In contrast, the Transformer model exhibits poor training stability under small sample sizes and high noise conditions, resulting in lower prediction accuracy compared to the recurrent model.

[0059] c. In the multimodal prediction task integrating sky imagery and meteorological sequence features, the proposed MDAT-LSTM model significantly outperforms the single-modal model, demonstrating clear superiority. Under clear conditions, its RMSE is 3.06, 18.2% lower than MDAT-GRU. Under cloudy conditions, its RMSE is 16.66, 33.9% lower than MDAT-GRU. The results indicate that deep fusion of imagery and meteorological modes can supplement spatial information and temporal dependencies, significantly improving the model's robustness and generalization ability under complex meteorological conditions, and accurately capturing rapid fluctuations in irradiance under cloudy conditions.

[0060] d. Based on the combined results of the nine models, the MDAT-LSTM model performed best in terms of RMSE and MAE. Compared to the single-modal LSTM, it reduced RMSE by 18.2% and 29.2% under sunny and cloudy conditions, respectively; compared to the single-modal MDAT, it reduced RMSE by 75.5% and 26.1%, respectively. This model combines the lightweight feature extraction of MobileNetV2, the global perception of the dual attention mechanism, and the temporal modeling advantages of LSTM, achieving deep feature fusion and maintaining high prediction stability and accuracy under cloudy conditions.

[0061] To further validate the effectiveness and advancement of the proposed model, we selected five state-of-the-art deep learning methods for comparison in prediction tasks: Cross Video Vision Transformer (CrossViViT), FEDformer, Informer, Autoformer, and Reformer. CrossViViT is a multimodal fusion model with the same input format as MDAT-LSTM; Informer, Autoformer, and Reformer are only applicable to time series prediction and use historical meteorological data as input. The hyperparameters of all benchmark models were set according to relevant literature to ensure fairness in the comparison. The comparative experimental results are shown in Table 3 and Figures 3-5, with comparisons of sunny and cloudy forecasts shown in the figures. Figure 6 As shown in Figure 7, the proposed MDAT-LSTM model demonstrates significant advantages under both weather conditions. Under sunny conditions, the RMSE, MAE, and nRMSE values ​​of the MDAT-LSTM model are 3.06, 2.47, and 0.42, respectively; while under cloudy conditions, these values ​​are 16.66, 11.69, and 3.11, respectively. Both sets of results significantly outperform all baseline models. Compared to the state-of-the-art time series model Autoformer, MDAT-LSTM reduces RMSE by 36.1% and 17.7% under sunny and cloudy conditions, respectively. Furthermore, compared to the multimodal model CrossViViT, the error is further reduced by approximately 70%. These results indicate that MDAT-LSTM exhibits higher prediction accuracy and robustness under various meteorological scenarios.

[0062] Table 3 Comparison of Prediction Model Performance The improved performance of the solar irradiance prediction model based on dual-attention multimodal fusion is mainly attributed to the synergistic effect of the multimodal structures. The MobileNetV2 module extracts features from satellite cloud images in a lightweight manner; the Dual-AttentionTransformer module further integrates local and global features, enhancing the model's ability to capture the spatiotemporal dynamics and distribution patterns of cloud fields; while LSTM captures the dynamic dependence of meteorological factors over time, ensuring the continuity and smoothness of the prediction results in the temporal dimension. Through the fusion of these structures, MDAT-LSTM can not only accurately fit stable irradiance trends under clear weather conditions but also effectively identify complex patterns of cloud evolution and radiation variation characteristics under cloudy weather conditions.

[0063] Example 2 A solar irradiance prediction method based on dual-attention multimodal fusion of the system, comprising: Collect sky imaging observation images and meteorological observation data of the area to be predicted, and perform time alignment and data preprocessing operations on the sky imaging observation images and meteorological observation data to obtain preprocessed and time-synchronized sky imaging observation images and meteorological observation data. Spatial features were extracted from the preprocessed and time-synchronized sky imaging observation images using the MobileNetV2 network to obtain the sky imaging temporal feature sequence. The temporal feature sequence of sky imaging is input in parallel to the global attention branch and the local window attention branch of the DAT structure. The global attention branch of the DAT structure is used to perform adaptive aggregation processing on the temporal feature sequence of sky imaging to obtain the global sky imaging temporal feature sequence. The local window attention branch of the DAT structure is used to perform local window attention calculation and window stitching processing on the temporal feature sequence of sky imaging to obtain the local sky imaging temporal feature sequence. According to the local convolution position enhancement mechanism of the DAT structure and the temporal feature sequence of sky imaging, the local convolution position encoding residual term is calculated. The global sky imaging temporal feature sequence and the local sky imaging temporal feature sequence are sequentially processed by channel stitching, linear fusion layer integration, and residual term introduction to obtain the aggregated sky imaging temporal feature sequence. Long Short-Term Memory (LSTM) networks are used to model the time dependence of meteorological observation data to obtain the characteristic sequence of meteorological observation data; By using a cross-attention mechanism, the time-series feature sequences of aggregated sky imaging and the feature sequences of meteorological observation data are fused to obtain a fused joint feature sequence, which is used for solar irradiance prediction.

[0064] Example 3 The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the method described in Embodiment 2.

[0065] This invention can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0066] It will be readily understood by those skilled in the art that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, combinations, substitutions, improvements, etc., made under the spirit and principles of the present invention are included within the protection scope of the present invention.

[0067] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A solar irradiance prediction system based on dual-attention multimodal fusion, characterized in that, It includes: The data acquisition module is used to collect sky imaging observation images and meteorological observation data of the area to be predicted, and to perform time alignment and data preprocessing operations on the sky imaging observation images and meteorological observation data to obtain preprocessed and time-synchronized sky imaging observation images and meteorological observation data. The image feature extraction and temporal modeling module is used to extract spatial features from the preprocessed and time-synchronized sky imaging observation image using the MobileNetV2 network, and obtain the sky imaging temporal feature sequence. The image processing module is used to input the sky imaging temporal feature sequence in parallel to the global attention branch and local window attention branch of the DAT structure. The global attention branch of the DAT structure is used to perform adaptive aggregation processing on the sky imaging temporal feature sequence to obtain the global sky imaging temporal feature sequence. The local window attention branch of the DAT structure is used to perform local window attention calculation and window stitching processing on the sky imaging temporal feature sequence to obtain the local sky imaging temporal feature sequence. According to the local convolution position enhancement mechanism of the DAT structure and the sky imaging temporal feature sequence, the local convolution position encoding residual term is calculated. The global sky imaging temporal feature sequence and the local sky imaging temporal feature sequence are sequentially processed by channel stitching, linear fusion layer integration, and residual term introduction to obtain the aggregated sky imaging temporal feature sequence. The meteorological sequence modeling module is used to perform time-dependent modeling of meteorological observation data using a long short-term memory network to obtain the feature sequence of meteorological observation data. The multimodal attention fusion module is used to fuse the aggregated sky imaging time-series feature sequences and meteorological observation data feature sequences through a cross-attention mechanism to obtain a fused joint feature sequence for solar irradiance prediction.

2. The solar irradiance prediction system based on dual attention multimodal fusion according to claim 1, characterized in that: Methods for extracting spatial features from preprocessed and time-synchronized sky imaging observations using the MobileNetV2 network to obtain temporal feature sequences of sky imaging include: The sequence of sky imaging observation images is as follows , , The number of channels is 1, and the height is , width is The set of real numbers in sky imaging observations. The time step for acquiring sky imaging observation images. This is a sky imaging observation image collected at the first moment. This is a sky imaging observation image acquired at the second moment. For the first Sky imaging observations are collected at all times, and spatial features are extracted from these sky imaging observations using MobileNetV2: ； in, for The spatial feature map of the sky corresponding to the sky imaging observation map at a given time. for Sky imaging observation image at that moment, This is the feature extraction function for the MobileNetV2 backbone network. The output feature dimension of the spatial features of sky imaging. for The height of the spatial feature map of the sky at any given time. for The width of the spatial feature map of the sky at any given time. for Spatial resolution of the spatial feature map of the sky at any given time. The dimension of the output feature is represented as Spatial resolution is The set of real numbers for the spatial feature maps of sky imaging. It belongs to the symbol; The spatial dimension of the sky imaging spatial feature map is compressed using a global average pooling method: ； in, After global average pooling Sky spatial feature map at time of day. This is the global average pooling function. for Sky imaging spatial feature map at time of day in spatial location The feature vector at that location, The dimension of the output feature is represented as The set of real numbers in the spatial feature map of sky imaging; After global average pooling is transformed by linear transformation Sky imaging spatial feature map at time of moment Projected onto a fixed dimension In the space, construct a temporal feature sequence of sky imaging: ； in, After linear transformation Sky spatial feature map at time of day. This is the weight matrix of the fully connected layer. This is the bias vector for the fully connected layer. To characterize the output feature dimension is The set of real numbers for the spatial feature maps of sky imaging. This is a temporal feature sequence of sky imaging. This is a temporal feature map of the sky image at the first moment. This is the temporal feature sequence of sky imaging at the second time step. For the first Temporal feature map of sky imaging at time 10:

00. Characterizing time period The internal output feature dimension is The set of real numbers for the spatial feature map of the sky imaging.

3. The solar irradiance prediction system based on dual attention multimodal fusion according to claim 2, characterized in that: Methods for adaptively aggregating temporal feature sequences of sky imaging using the global attention branch of the DAT structure to obtain global sky imaging temporal feature sequences include: Global attention branch for temporal feature sequences of sky imaging Downsampling is performed on the time axis: ； in, This is a sequence of temporal features of downsampled sky imaging. For downsampling function, To reduce the time step after sampling, To reduce sampling hyperparameters, This represents the number of samples processed in parallel batches. A set of temporal feature sequences of sky imaging with feature channel C after downsampling; Global attention branch has The global attention head, for the first Global attention heads ( The corresponding sky imaging temporal feature sequence is generated through linear projection. Global query vector, global key vector, and global value vector: ； in, For the first A global query vector for a global attention head. For the first Global key vectors of global attention heads For the first The global value vector of each global attention head. For the first Global query projection weight matrix of each global attention head For the first Global key projection weight matrix of each global attention head For the first The global value projection weight matrix of each global attention head; The first value is obtained by calculating the scaled dot product attention mechanism. Global attention weights for each global attention head: ； in, Let be the dimension of each global attention head in the global attention branch. For normalization function, for The global attention weights of each global attention head. for The global attention head weighted output of the size It is the transpose operator; Will The weighted outputs of the global attention heads are concatenated and then projected through the global output matrix. The global sky imaging temporal feature sequence was obtained: ； in, This is a temporal feature sequence of global sky imaging. To be The weighted outputs of the global attention heads are concatenated along the feature channel dimension.

4. The solar irradiance prediction system based on dual attention multimodal fusion according to claim 2, characterized in that: Methods for obtaining local sky imaging temporal feature sequences by utilizing the local window attention branch of the DAT structure to perform local window attention calculation and window stitching processing on the sky imaging temporal feature sequence include: The local window attention branch has multiple local window network layers, each of which stores the temporal feature sequence of sky imaging along the time axis. Divided into There are [number] timing windows, each with a length of [length]. : ； in, For the first A set of timing step indexes for each timing window. This represents the total number of timing windows; Local window attention branch has The local attention head, for the first... The first timing window Each local attention head generates a corresponding temporal feature sequence of sky imaging through linear projection. Local query vector, local key vector, and local value vector: in, For the local window attention branch, the local window network layer index is... For the first Layer, First The first timing window, the first Local query vectors of local attention heads For the first Sky imaging temporal feature sequence within a time window For the first Layer, First The first timing window, the first Local key vectors of a local attention head For the first Layer, First The first timing window, the first Local value vectors of local attention heads, For the first Layer, First Local query projection weight matrix of each local attention head For the first Layer, First Local bond projection weight matrix of a local attention head For the first Layer, First The local value projection weight matrix of each local attention head; No. Local attention weights within a time window and output for: ； in, For the first Layer, First The first timing window, the first Local attention head weighted output For the first Layer, First The first timing window, the first The local attention weight matrix of each local attention head. For the dimension of the local attention head, This is the matrix transpose operator. This is the normalization function; The temporal feature maps of sky imaging output from all temporal windows and local attention heads are stitched together, and then projected through the local output matrix. Obtain the temporal feature sequence of local sky imaging : ； in, To stitch together the output of all windows along the spatial dimension, To stitch together the outputs of all attention heads along the channel dimension, For the first The local output projection weight matrix of the layer, For the first The total number of attention heads in the layer.

5. A solar irradiance prediction system based on dual attention multimodal fusion according to claim 2, characterized in that: Methods for calculating the local convolution position encoding residual term based on the local convolution position enhancement mechanism of the DAT structure and the temporal feature sequence of sky imaging include: The local convolutional location enhancement mechanism processes the input temporal feature sequence of sky imaging through pointwise linear transformation and depthwise separable convolution. ,get: in, Let be the pointwise linear transformation matrix. For point-by-point linear transformation bias term, For depthwise separable convolution, Encode residual terms for local convolution locations.

6. A solar irradiance prediction system based on dual attention multimodal fusion according to claim 3, 4, or 5, characterized in that: The method for obtaining the aggregated sky imaging time-series feature sequence by sequentially performing channel stitching, linear fusion layer integration, and residual term introduction on the global sky imaging time-series feature sequence and the local sky imaging time-series feature sequence includes: global sky imaging temporal feature sequence Temporal feature sequences of local sky imaging The components are stitched together along the channel dimension and then processed through a linear fusion layer. Integration, while incorporating local convolutional position encoding residual terms. Obtain the temporal feature sequence of sky imaging output from the DAT structure. : ； Will The layered DAT module is embedded into the standard Transformer structure through Pre-LN residual connections, resulting in a multi-layered stacked DAT and Transformer encoder network. ； The sky imaging temporal feature sequence obtained from the DAT and Transformer encoder network By averaging and converging over time, we obtain: ； in, This is the temporal feature sequence of sky imaging after processing by the first DAT sublayer. For the first Sky imaging temporal feature sequence of layers, It is a dual attention switching module. For layer normalization function, for Sky imaging temporal feature sequence of layers, It is a feedforward neural network. For activation function, These are the feature vectors of the feedforward neural network. This is the first learnable weight matrix of the feedforward neural network. This is the second learnable weight matrix of the feedforward neural network; To aggregate the temporal feature sequences of sky imaging, The time step for acquiring sky imaging data. For the DAT and Transformer encoder network at time t The temporal feature sequence of sky imaging output from the layer.

7. The solar irradiance prediction system based on dual attention multimodal fusion according to claim 1, characterized in that: Methods for using Long Short-Term Memory (LSTM) networks to model the time dependence of meteorological observation data and obtain the characteristic sequences of meteorological observation data include: The original meteorological time series, composed of meteorological observation data, is , Given B independent samples, each sample being a sequence of length T, with each time step in the sequence represented by a set of real numbers in a d-dimensional real vector, perform MinMax scaling on the original meteorological time series: ； in, For normalized meteorological observation data, These are the minimum and maximum normalization functions; An LSTM network has L layers of stacked LSTM cells. The forgetting gate at time t is used to determine the cell state at the previous time step. The formula for calculating which meteorological observation data needs to be retained or discarded is as follows: ； in, This is the output of the forget gate, with values ​​ranging from (0,1); It is the Sigmoid activation function. The meteorological observation data is for the current time t. This is the weight matrix of the meteorological observation data at the current time t, representing the forgetting gate. Here is the weight matrix of the meteorological observation data at time t-1. For the bias term of the forget gate, This represents the hidden state at time t-1; The input gate of the LSTM network determines the meteorological observation data that should be introduced at the current moment and writes it into the cell state: ； ； ； in, This is the input gate output, with a value range of (0,1). Let be the weight matrix of the meteorological observation data of the input gate at the current time t. The weight matrix is ​​the meteorological observation data at time t-1 of the input gate. This is the bias term for the input gate; Candidate cell state, The hyperbolic tangent activation function is used. This is the weight matrix of meteorological observation data under candidate cell states. This is the weight matrix of meteorological observation data under candidate cell states. This is the input gate bias term for the candidate cell state; This represents the updated cell state. The cell state at time t-1. This is the Hadamard product algorithm; The output gate is used to determine which information from the current cell state will be passed to the hidden layer. Its calculation formula is as follows: ； ； in, This is the activation result of the output gate. To output the meteorological observation data at the current moment, This is the weight matrix corresponding to the hidden state in the previous time step. This is the bias term corresponding to the hidden state in the previous time step; It is the hidden state at the current time t; After stacking layers, the final hidden state of the top layer is taken as the temporal representation: ； in, For meteorological observation data characteristic sequences, This represents the hidden state of the final LSTM layer at the last moment.

8. The solar irradiance prediction system based on dual attention multimodal fusion according to claim 7, characterized in that: Methods for fusing aggregated temporal feature sequences of sky imaging and feature sequences of meteorological observation data through cross-attention mechanisms to obtain fused joint feature sequences include: A cross-attention mechanism is introduced to linearly project the aggregated sky imaging time-series feature sequence and meteorological observation data feature sequence to obtain the query vector Q, key vector K, and value vector V. The feature fusion between the aggregated sky imaging time-series feature sequence and meteorological observation data feature sequence is transformed into attention calculation with the sky imaging time-series feature sequence as the query and the meteorological observation data feature sequence as the key-value pair. ； in To aggregate the temporal feature sequences of sky imaging, To query the vector projection weight matrix, For meteorological observation data characteristic sequences, The weight matrix is ​​the projection of the key vectors. The weight matrix is ​​the projection of the value vector; A multi-head attention mechanism is used to fuse the temporal feature sequence of sky imaging and the feature sequence of meteorological observation data, and each attention head in the multi-head attention mechanism is mapped separately into a low-dimensional subspace: ； in, For the first A low-dimensional query vector of attention heads For the first A low-dimensional key vector of an attention head. For the first A low-dimensional value vector of attention heads, For the first Each attention head can learn the query projection matrix. For the first A learnable key projection matrix for each attention head. For the first The attention head can learn the value projection matrix. Calculate the first Scaling dot product attention for each attention head: ； ； in, For the first The scaled dot product attention weight matrix of each attention head. It is a flexible maximum value function. For the first The dimension of attention heads For the first The query vector of each attention head transpose, For the first The output of each attention head, The outputs of multiple attention heads are concatenated and projected to obtain the cross-attention outputs of the multiple attention heads. : ； in, To be The outputs of each attention head are concatenated. For attention projection matrix; Cross-attention output Further corrections are made through normalization, residual connections, and feedforward networks: in, This is a fusion feature between attention output and aggregated temporal feature sequences of sky imaging. It is a feedforward neural network. This refers to the fusion feature of the cross-attention output corrected by the feedforward network and the aggregated temporal feature sequence of sky imaging. This is the projection weight matrix used to match jump connections in the input dimension; The fusion features of the cross-attention output after feedforward network correction and the aggregated temporal feature sequence of sky imaging Pooling is performed along the query dimension to obtain the fused feature vector of sky imaging and meteorological data: ； in, Features for the fusion of sky imaging and meteorological data For pooling functions based on the Lq norm, The feature vector fused from sky imaging and meteorological data is then combined with the feature sequence of meteorological observation data. After splicing, input the prediction head : ； in, This is the final fused joint feature sequence. For concatenation functions, This represents the predicted solar irradiance value.

9. A solar irradiance prediction method based on dual-attention multimodal fusion of the system described in claim 1, characterized in that, include: Collect sky imaging observation images and meteorological observation data of the area to be predicted, and perform time alignment and data preprocessing operations on the sky imaging observation images and meteorological observation data to obtain preprocessed and time-synchronized sky imaging observation images and meteorological observation data. Spatial features were extracted from the preprocessed and time-synchronized sky imaging observation images using the MobileNetV2 network to obtain the sky imaging temporal feature sequence. The temporal feature sequence of sky imaging is input in parallel to the global attention branch and the local window attention branch of the DAT structure. The global attention branch of the DAT structure is used to perform adaptive aggregation processing on the temporal feature sequence of sky imaging to obtain the global sky imaging temporal feature sequence. The local window attention branch of the DAT structure is used to perform local window attention calculation and window stitching processing on the temporal feature sequence of sky imaging to obtain the local sky imaging temporal feature sequence. According to the local convolution position enhancement mechanism of the DAT structure and the temporal feature sequence of sky imaging, the local convolution position encoding residual term is calculated. The global sky imaging temporal feature sequence and the local sky imaging temporal feature sequence are sequentially processed by channel stitching, linear fusion layer integration, and residual term introduction to obtain the aggregated sky imaging temporal feature sequence. Long Short-Term Memory (LSTM) networks are used to model the time dependence of meteorological observation data to obtain the characteristic sequence of meteorological observation data; By using a cross-attention mechanism, the time-series feature sequences of aggregated sky imaging and the feature sequences of meteorological observation data are fused to obtain a fused joint feature sequence, which is used for solar irradiance prediction.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 9.

Images