Multi-modal ultra-short-term solar irradiation prediction method and system based on sky image

Abstract

The invention relates to the field of solar photovoltaic power generation, in particular to a multi-mode ultra-short-term solar irradiation prediction method and system based on a sky image, and the method comprises the steps: collecting a sky image and meteorological data shot by an all-sky imager; preprocessing the collected image, extracting cloud layer information through graying and a normalized red-blue ratio, and extracting dynamic characteristics by using a dense optical flow method; a 3D-ResNet model is adopted to extract spatial features of the image, and an Informer model is utilized to perform time sequence coding on historical irradiation data; and through a multi-modal interaction fusion matrix based on an attention mechanism, carrying out alignment fusion on the spatial features of the sky image and the time sequence features of the irradiation, and generating a predicted value of the solar irradiation. According to the invention, the precision of ultra-short-term solar irradiation prediction can be improved, and especially in a time range of 5-10 minutes, the method has relatively strong real-time performance and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of solar photovoltaic power generation, and in particular to a multi-modal ultra-short-term solar radiation prediction method and system based on sky images. Background Art

[0002] With the increasing global demand for clean energy, solar photovoltaic power generation has become an important way to solve energy shortages and environmental pollution. The randomness, intermittency and volatility of solar energy resources pose challenges to the stable operation of photovoltaic power generation systems, especially in a short period of time, changes in solar irradiation caused by weather conditions can cause drastic fluctuations in photovoltaic output power. These fluctuations seriously affect the grid-connected stability of photovoltaic power and the dispatching efficiency of the power grid. Therefore, accurate ultra-short-term solar irradiation prediction technology is crucial to improving the reliability of photovoltaic power generation systems.

[0003] At present, solar irradiation prediction technology mainly relies on the combination of historical meteorological data and physical models, but these methods have significant limitations in prediction accuracy and real-time performance. Specifically, the formation, movement and dissipation of clouds are the main factors that cause large changes in solar irradiation in a short period of time, and existing models based on meteorological data are difficult to effectively capture these rapid changes. Although physical models can explain the impact of cloud movement on irradiation to a certain extent, due to the complexity of atmospheric conditions and local climate differences, their prediction accuracy still cannot meet the scheduling needs of high-frequency photovoltaic power generation systems. Therefore, accurate ultra-short-term solar irradiation prediction technology is crucial to improving the reliability of photovoltaic power generation systems. Summary of the invention

[0004] Purpose of the invention: In view of the limitation that the single data source prediction model in the prior art cannot fully capture the impact of dynamic changes in the sky on irradiance, the present invention provides a multimodal ultra-short-term solar irradiance prediction method based on sky images. The method improves the prediction accuracy of short-term fluctuations in solar irradiance through the innovative design of multimodal data fusion and deep learning model, and is particularly suitable for ultra-short-term scheduling and management of photovoltaic power generation systems. By exploring the complementary characteristics of sky images and historical irradiation data, the ability to capture short-term weather changes is enhanced, thereby improving the accuracy of irradiance prediction. The present invention also discloses a multimodal ultra-short-term solar irradiance prediction system based on sky images.

[0005] Technical solution: The multimodal ultra-short-term solar radiation prediction method based on sky images disclosed in the present invention comprises:

[0006] Acquire sky images through a total sky imager, collect irradiance data and meteorological data at the same time, and calculate relevant irradiance data under theoretical clear sky, wherein the relevant irradiance data include global horizontal irradiance GHI, direct normal irradiance DNI and scattered horizontal irradiance DHI;

[0007] Grayscale processing is performed on the sky image, and cloud thickness information is extracted using a normalized red-to-blue ratio, and dynamic features are extracted using a dense optical flow method, and historical irradiation data and meteorological data are normalized to obtain preprocessed data and a sky image;

[0008] The features of multiple frames of continuous sky images are used as the input of the 3D convolutional neural network to extract the spatiotemporal features of the sky images, capture the dynamic changes of the clouds, and output the high-dimensional feature vector of the sky images. At the same time, the Informer model is used to encode the time series of irradiation data and meteorological data to obtain time series features.

[0009] Using the multimodal interaction matrix, the high-dimensional feature vector of the sky image and the time series features are concatenated through the column dimension to obtain the fused feature vector, and the fused features are input into the regression layer to generate the final irradiance prediction result.

[0010] Further, including:

[0011] The calculation of relevant irradiance data under theoretical clear sky includes:

[0012] First, calculate the global horizontal irradiance GHI as:

[0013]

[0014] Among them, H is the hour angle, φ is the latitude, δ is the solar declination angle, θ z is the solar zenith angle, θ A is the azimuth;

[0015] Secondly, the Perez model was used to calculate the GHI, DNI and DHI values ​​at each time point under theoretical clear sky.

[0016] GHI=exp(-cg2·airmass ab ·(fh1+fh2·(tl-1)))(2)

[0017] Among them, fh1, fh2, cg2 are altitude correction factors, airmass ab is the absolute air quality, tl is the Lincoln turbidity; bnci1 is the initial calculated normal direct radiation value, based on the influence of atmospheric transmittance and Lincoln turbidity, and bnci2 is another restriction condition based on GHI and zenith angle, which is used to constrain the value range of DNI:

[0018] bnci1 = b·exp(-0.09·airmass ab ·(tl-1)); (3)

[0019]

[0020] The final DNI takes the minimum value of bnci1 and bnci2:

[0021] DNI=min(bnci1,bnci2); (5)

[0022] DHI = GHI - DNI · cos θz (6).

[0023] Further, including:

[0024] The use of normalized red-to-blue ratio I NRBR Extract cloud thickness information, expressed as:

[0025]

[0026] Among them, B is the blue channel value, R is the red channel value, and I NRBR Ability to highlight the distribution of clouds in the image.

[0027] Further, including:

[0028] The dense optical flow method is used to extract dynamic features, and the motion velocity vector field of the cloud layer is calculated between consecutive image frames to capture the movement information of the cloud layer in all directions.

[0029] The dense optical flow method calculation specifically includes:

[0030] I(x,y,t)=I(x+u,y+v,t+Δ t ); (8)

[0031] Where (x, y) represents the pixel position in the image at time t, (x+u, y+v) represents the position of the pixel after the pixel changes at time t+Δt, and I(x, y, t) represents the pixel value of the image at position (x, y) at time t.

[0032] Perform a first-order Taylor expansion on the right side of the equation to obtain:

[0033]

[0034] is the gradient of the image in the x and y directions, u and v are the pixel movement speeds that need to be solved, that is, the optical flow components;

[0035] Assuming that the optical flow of the image is a constant value in a window of size m×m, we can get the following set of equations:

[0036]

[0037] The system of equations is represented by the abbreviation A·V=b, and the linear equation is solved:

[0038] V=(A T A) -1 A T b; (11)

[0039] Where V = [u, v] T is the optical flow vector of each pixel on the x,y axis;

[0040] Therefore, the optical flow vector at each pixel on the cloud map is calculated to obtain the same pixel size as the original cloud map. Two dynamic feature optical flow images.

[0041] Further, including:

[0042] The method uses the features of multiple frames of continuous sky images as the input of the 3D convolutional neural network, extracts the spatiotemporal features of the sky images, captures the dynamic changes of the clouds, and outputs the high-dimensional feature vector of the sky images, specifically including:

[0043] Input data processing: Each input contains multiple frames of continuous sky image features The image size is uniformly adjusted to 128×128 pixels; after downsampling and normalization, it is represented by vector x and input into the 3D-ResNet module, where I Gray is the feature of the image after grayscale conversion;

[0044] 3D convolution operation: In 3D-ResNet, convolution operations are performed in both spatial and temporal dimensions. The size of each 3D convolution kernel is k×k×d, where k is the convolution kernel size in the spatial dimension and d is the convolution depth in the temporal dimension. The specific operations of 3D convolution are as follows:

[0045]

[0046] Among them, Y i,j,t represents the output feature value at position (i, j, t), W p,q,r is the 3D convolution kernel weight, b is the bias term, (p, q, r) represents the index inside the convolution kernel, W pqr ·X i+p,j+q,t+r Is a dot product operation, representing an operation logic X i+p,j+q,t+r Represents a value in this local sub-block, which is located at the (t+r)th time step, (i+p)th width position and (j+q)th height position of the input feature map. The subscripts t, i, and j mean the time, width, and height position of the output feature map. The final output is expressed as

[0047] Residual connection: In order to avoid the vanishing gradient problem in deep networks, the input features are directly passed to the convolution output. The calculation method is as follows:

[0048] F(x)=x+Conv3D(x) (13)

[0049] Among them, Conv3D is the abbreviation of the 3D convolution operation calculation process mentioned above.

[0050] Further, including:

[0051] The Informer model is used to encode the time series of irradiation data and meteorological data to obtain high-dimensional time series features. include:

[0052] Input data processing: Perform 1D convolution operation on the preprocessed multidimensional time series data, and add the position information of the time series data to the input features for position encoding. Includes: global horizontal irradiance GHI, direct normal irradiance DNI, diffuse horizontal irradiance DHI and meteorological data;

[0053] Sparse self-attention mechanism: Generates a sparse self-attention score, calculated as follows:

[0054]

[0055] Where Q, K, and V represent the query, key, and value matrices respectively, and d k For the scaling factor, in sparse self-attention, only some key-value pairs with the largest contribution value are selected for calculation, reducing redundant calculations and thus improving the computational efficiency of the model;

[0056] Multi-head self-attention mechanism: By running multiple attention heads in parallel, the model captures the characteristics of irradiance and meteorological data at different time scales. The formula is as follows:

[0057] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (15)

[0058] Each attention head performs calculations on different feature dimensions, and the results are concatenated and transformed linearly to obtain the final output;

[0059]

[0060] Among them, AB[·] represents the attention calculation link including sparse self-attention mechanism and multi-head self-attention mechanism, including multi-head self-attention, Conv1d(·) represents the convolution filter with a convolution kernel size of 3 in the time dimension, and the ELU(·) activation function is used to improve the nonlinear mapping ability of this link. Finally, a maximum pooling layer with a stride of 2 is added, and Informer(·) is used to represent the entire time feature extraction link, that is:

[0061]

[0062] Further, including:

[0063] The step of concatenating the high-dimensional feature vector of the sky image and the time series feature through the column dimension to obtain the fused feature vector specifically includes:

[0064] First, in order to achieve deep fusion between different modal features, an interaction matrix based on the self-attention mechanism is constructed to capture the complementary relationship between image and time series features. The generation formula of the interaction matrix Θ is as follows:

[0065]

[0066] Among them, Q and K represent query and key respectively. According to the calculation above, here, the output of 3D convolution is Indicates that the output of the multi-head self-attention mechanism is It means that q α A modal input is required for calculation, k β is the input of another modality, d k Represents the high dimension mapped to, q I represents the sky image feature as the query, k I Indicates that time series features are used as keys;

[0067] Secondly, using the interaction matrix, the spatial and temporal series features are concatenated through the column dimension to obtain the fused feature vector. The calculation formula of the fusion process is as follows:

[0068]

[0069] in, and They are sky image and time series features respectively, and Concat represents the concatenation operation.

[0070] On the other hand, the present invention also provides a multi-modal ultra-short-term solar radiation prediction system based on sky images, characterized in that the system comprises:

[0071] The acquisition module is used to acquire sky images through a full sky imager, collect irradiance data and meteorological data at the same time, and calculate relevant radiation data under theoretical clear sky, wherein the relevant radiation data includes global horizontal irradiance GHI, direct normal irradiance DNI and scattered horizontal irradiance DHI;

[0072] A preprocessing module is used to grayscale the sky image, extract cloud thickness information using a normalized red-to-blue ratio, extract dynamic features using a dense optical flow method, and normalize historical radiation data and meteorological data to obtain preprocessed data and a sky image;

[0073] The feature extraction module is used to take the features of multiple frames of continuous sky images as the input of the 3D convolutional neural network, extract the spatiotemporal features of the sky images, capture the dynamic changes of the clouds, output the high-dimensional feature vector of the sky images, and use the Informer model to encode the time series of irradiation data and meteorological data to obtain time series features;

[0074] The fusion module uses the multimodal interaction matrix to concatenate the high-dimensional feature vector of the sky image and the time series features through the column dimension to obtain the fused feature vector, and inputs the fused features into the regression layer to generate the final irradiance prediction result.

[0075] Further, including:

[0076] In the feature extraction module,

[0077] The features of multiple frames of continuous sky images are used as the input of the 3D convolutional neural network to extract the spatiotemporal features of the sky images, capture the dynamic changes of the clouds, and output the high-dimensional feature vector of the sky images, including:

[0078] Input data processing: Each input contains multiple frames of continuous sky image features The image size is uniformly adjusted to 128×128 pixels; after downsampling and normalization, it is represented by vector x and input into the 3D-ResNet module, where I Gray is the image feature after the image is grayed;

[0079] 3D convolution operation: In 3D-ResNet, convolution operations are performed in both spatial and temporal dimensions. The size of each 3D convolution kernel is k×k×d, where k is the convolution kernel size in the spatial dimension and d is the convolution depth in the temporal dimension. The specific operations of 3D convolution are as follows:

[0080]

[0081] Among them, Y i,j,t represents the output feature value at position (i, j, t), Wp,q,r is the 3D convolution kernel weight, b is the bias term, (p, q, r) represents the index inside the convolution kernel, W pqr ·X i+p,j+q,t+r Is a dot product operation, representing an operation logic X i+p,j+q,t+r Represents a value in this local sub-block, which is located at the (t+r)th time step, (i+p)th width position and (j+q)th height position of the input feature map. The subscripts t, i, and j mean the time, width, and height position of the output feature map. The final output is expressed as

[0082] Residual connection: In order to avoid the vanishing gradient problem in deep networks, the input features are directly passed to the convolution output. The calculation method is as follows:

[0083] F(x)=x+Conv3D(x);

[0084] Among them, Conv3D is the abbreviation of the 3D convolution operation calculation process mentioned above.

[0085] Further, including:

[0086] The Informer model is used to encode the time series of irradiation data and meteorological data. Get high-dimensional time series features include:

[0087] Input data processing: Perform 1D convolution operation on the preprocessed multidimensional time series data, and add the position information of the time series data to the input features for position encoding. Includes: global horizontal irradiance GHI, direct normal irradiance DNI, diffuse horizontal irradiance DHI and meteorological data;

[0088] Sparse self-attention mechanism: Generates a sparse self-attention score, calculated as follows:

[0089]

[0090] Where Q, K, and V represent the query, key, and value matrices respectively, and d k For the scaling factor, in sparse self-attention, only some key-value pairs with the largest contribution value are selected for calculation, reducing redundant calculations and thus improving the computational efficiency of the model;

[0091] Multi-head self-attention mechanism: By running multiple attention heads in parallel, the model captures the characteristics of irradiance and meteorological data at different time scales. The formula is as follows:

[0092] MultiHead(Q,K,V)=Concat(head1,...,head h )W O ;

[0093] Each attention head performs calculations on different feature dimensions, and the results are concatenated and transformed linearly to obtain the final output;

[0094]

[0095] Among them, AB[·] represents the attention calculation link including sparse self-attention mechanism and multi-head self-attention mechanism, including multi-head self-attention, Conv1d(·) represents the convolution filter with a convolution kernel size of 3 in the time dimension, and the ELU(·) activation function is used to improve the nonlinear mapping ability of this link. Finally, a maximum pooling layer with a stride of 2 is added, and Informer(·) is used to represent the entire time feature extraction link, that is:

[0096]

[0097] Finally, the present invention also provides a computer-readable storage medium, on which a multimodal ultra-short-term solar irradiance prediction program based on sky images is stored. When the multimodal ultra-short-term solar irradiance prediction program based on sky images is executed by a processor, the steps of the multimodal ultra-short-term solar irradiance prediction method based on sky images as described above are implemented.

[0098] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0099] The present invention combines the spatial characteristics of sky images and time series irradiance data, realizes the deep fusion of different modal data through the cross-attention mechanism, makes full use of the complementary information of images and time series, and significantly improves the irradiance prediction accuracy, especially in the time range of 5 to 10 minutes, with strong real-time and accuracy; the present invention combines the dense optical flow method with 3D-ResNet, which can accurately capture the dynamic changes of clouds in continuous time frames, and provides strong support for cloud movement and occlusion information for short-term irradiance prediction; the present invention supports adaptive adjustment of feature weights, that is, the feature fusion matrix of the present invention contains a large number of hyperparameters. During the model training process, the parameters are continuously updated during the reverse adjustment of the loss function, so that the model can flexibly respond to real-time weather changes, thereby significantly enhancing the accuracy and robustness of the prediction under changeable weather conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 The figure shows a schematic flow chart of a multi-modal ultra-short-term solar irradiation prediction method based on sky images in an example of the present invention;

[0101] Figure 2 The figure shows a flow chart of a data preprocessing module of a multi-modal ultra-short-term solar irradiation prediction method based on sky images in an example of the present invention;

[0102] Figure 3 The figure shows a schematic diagram of a 3D-ResNet module flow of a multi-modal ultra-short-term solar irradiation prediction method based on sky images in an example of the present invention;

[0103] Figure 4 The figure shows a schematic diagram of the flow of an Informer module of a multi-modal ultra-short-term solar irradiation prediction method based on sky images in an example of the present invention;

[0104] Figure 5 Shown is a schematic diagram of the multimodal fusion module flow of a multimodal ultra-short-term solar irradiation prediction method based on sky images in an example of the present invention. DETAILED DESCRIPTION

[0105] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0106] The present invention discloses a multi-modal ultra-short-term solar radiation prediction method based on sky images, comprising the following steps:

[0107] Step 1: Obtain sky images through the all-sky imager, collect irradiance data and meteorological data at the same time, and calculate relevant irradiance data under theoretical clear sky;

[0108] In this embodiment, a full sky imager is used to capture sky images in real time to capture the dynamic changes of the cloud layer. The sampling interval of the image data is 1 minute, which can record rapidly changing weather conditions and provide rich cloud space information.

[0109] In this embodiment, the irradiance data includes data such as global horizontal irradiance GHI, direct normal irradiance DNI, and scattered horizontal irradiance DHI. These parameters can intuitively reflect the changes in irradiance under different weather conditions.

[0110] Meteorological data include meteorological factors related to irradiance, such as temperature, humidity, air pressure, wind speed and wind direction, which are used to supplement the external environment information of irradiance.

[0111] First, calculate the global horizontal irradiance GHI as:

[0112]

[0113] Among them, H is the hour angle, φ is the latitude, δ is the solar declination angle, θ z is the solar zenith angle, θ A is the azimuth;

[0114] Secondly, the Perez model was used to calculate the GHI, DNI and DHI values ​​at each time point under theoretical clear sky.

[0115] GHI=exp(-cg2·airmass ab ·(fh1+fh2·(tl-1)))(2)

[0116] Among them, fh1, fh2, cg2 are altitude correction factors, airmass ab is the absolute air quality, tl is the Lincoln turbidity; bnci1 is the initial calculated normal direct radiation value, based on the influence of atmospheric transmittance and Lincoln turbidity, and bnci2 is another restriction condition based on GHI and zenith angle, which is used to constrain the value range of DNI:

[0117] bnci1 = b·exp(-0.09·airmass ab ·(tl-1)); (3)

[0118]

[0119] The final DNI takes the minimum value of bnci1 and bnci2:

[0120] DNI=min(bnci1,bnci2); (5)

[0121] DHI = GHI - DNI · cos θz (6).

[0122] Step 2: Grayscale the sky image and use the normalized red-to-blue ratio (NRBR) to extract cloud thickness information. Then use the dense optical flow method to extract dynamic features. At the same time, normalize the historical irradiance data and meteorological data.

[0123] In this embodiment, the step specifically includes:

[0124] S2.1, image grayscale processing. Color cloud images contain red, green, and blue channel information, and the amount of data is large, which can easily cause image processing difficulties and long computing time. The information contained in the grayscale image is sufficient to reflect the size and thickness of the cloud, so the image is grayscaled;

[0125] S2.2, sky image processing. Based on the grayscale image, the normalized red-to-blue ratio (NRBR) index is used to extract cloud thickness information to distinguish different cloud density characteristics. NRBR Extract cloud thickness information, expressed as:

[0126]

[0127] Among them, B is the blue channel value, R is the red channel value, and I NRBR Ability to highlight the distribution of clouds in the image.

[0128] In this embodiment, NRBR can highlight the distribution of clouds in the image, especially in distinguishing thinner clouds from clear skies.

[0129] S2.3, dynamic feature extraction using dense optical flow method. The dense optical flow method is used to calculate the motion velocity vector field of the cloud between consecutive image frames to capture the movement information of the cloud in all directions. By analyzing these velocity vectors, the movement trend and speed of the cloud can be more accurately reflected, providing the model with dynamic features of the cloud occlusion trend.

[0130] The dense optical flow method is used to extract dynamic features, and the motion velocity vector field of the cloud layer is calculated between consecutive image frames to capture the movement information of the cloud layer in all directions.

[0131] The dense optical flow method calculation specifically includes:

[0132] I(x,y,t)=I(x+u,y+v,t+Δ t ); (8)

[0133] Where (x, y) represents the pixel position in the image at time t, (x+u, y+v) represents the position of the pixel after the pixel changes at time t+Δt, and I(x, y, t) represents the pixel value of the image at position (x, y) at time t.

[0134] Perform a first-order Taylor expansion on the right side of the equation to obtain:

[0135]

[0136] is the gradient of the image in the x and y directions, u and v are the pixel movement speeds that need to be solved, that is, the optical flow components;

[0137] Assuming that the optical flow of the image is a constant value in a window of size m×m, we can get the following set of equations:

[0138]

[0139] The system of equations is represented by the abbreviation A·V=b, and the linear equation is solved:

[0140] V=(A T A) -1 A T b; (11)

[0141] Where V = [u, v] T is the optical flow vector of each pixel on the x,y axis;

[0142] Therefore, the optical flow vector at each pixel on the cloud map is calculated to obtain the same pixel size as the original cloud map. Two dynamic feature optical flow images.

[0143] S2.4, time series data processing. Align the irradiance and meteorological data by timestamp to ensure synchronization with the sky image. Standardize all time series data and normalize the data to the [-1,1] interval to eliminate the dimensional differences between different data sources and improve data consistency.

[0144] Step 3: Use the 3D convolutional neural network 3D-ResNet to extract the spatiotemporal features of the sky image and capture the dynamic changes of the clouds.

[0145] In this embodiment, the step includes:

[0146] Input data processing: Each input contains multiple frames of continuous sky image features The image size is uniformly adjusted to 128×128 pixels; after downsampling and normalization, it is represented by vector x and input into the 3D-ResNet module, where I Gray is the feature of the image after grayscale conversion;

[0147] 3D convolution operation: In 3D-ResNet, convolution operations are performed in both spatial and temporal dimensions. The size of each 3D convolution kernel is k×k×d, where k is the convolution kernel size in the spatial dimension and d is the convolution depth in the temporal dimension. The specific operations of 3D convolution are as follows:

[0148]

[0149] Among them, Y i,j,t represents the output feature value at position (i, j, t), W p,q,r is the 3D convolution kernel weight, b is the bias term, (p, q, r) represents the index inside the convolution kernel, W pqr ·X i+p,j+q,t+r Is a dot product operation, representing an operation logic X i+p,j+q,t+rRepresents a value in this local sub-block, which is located at the (t+r)th time step, (i+p)th width position and (j+q)th height position of the input feature map. The subscripts t, i, and j mean the time, width, and height position of the output feature map. The final output is expressed as

[0150] Residual connection: In order to avoid the vanishing gradient problem in deep networks, the input features are directly passed to the convolution output. The calculation method is as follows:

[0151] F(x)=x+Conv3D(x)(13)

[0152] Among them, Conv3D is the abbreviation of the 3D convolution operation calculation process mentioned above.

[0153] S4: Use the Informer model to encode the historical irradiation data in time series to capture long-term and short-term dependencies;

[0154] In this embodiment, the Informer model is used to encode the time series of irradiation data and meteorological data. Get high-dimensional time series features Informer has significant advantages in processing long time series and is suitable for capturing short-term trend changes in irradiance, including:

[0155] Input data processing: Perform 1D convolution operation on the preprocessed multidimensional time series data, and add the position information of the time series data to the input features for position encoding. Includes: global horizontal irradiance GHI, direct normal irradiance DNI, scattered horizontal irradiance DHI and meteorological data; meteorological data includes: such as temperature, humidity, air pressure, wind speed and wind direction, etc.

[0156] Sparse self-attention mechanism: Generates a sparse self-attention score, calculated as follows:

[0157]

[0158] Where Q, K, and V represent the query, key, and value matrices respectively, and d k For the scaling factor, in sparse self-attention, only some key-value pairs with the largest contribution value are selected for calculation, reducing redundant calculations and thus improving the computational efficiency of the model;

[0159] Multi-head self-attention mechanism: By running multiple attention heads in parallel, the model captures the characteristics of irradiance and meteorological data at different time scales. The formula is as follows:

[0160] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (15)

[0161] Each attention head performs calculations on different feature dimensions, and the results are concatenated and transformed linearly to obtain the final output;

[0162]

[0163] Among them, AB[·] represents the attention calculation link including sparse self-attention mechanism and multi-head self-attention mechanism, including multi-head self-attention, Conv1d(·) represents the convolution filter with a convolution kernel size of 3 in the time dimension, and the ELU(·) activation function is used to improve the nonlinear mapping ability of this link. Finally, a maximum pooling layer with a stride of 2 is added, and Informer(·) is used to represent the entire time feature extraction link, that is:

[0164]

[0165] Step 5: Generate the predicted value of solar irradiance by combining the spatial characteristics of sky images with the temporal characteristics of historical irradiance data through the multimodal interaction module.

[0166] In this embodiment, the high-dimensional feature vector of the sky image and the time series feature are spliced ​​through the column dimension to obtain the fused feature vector, specifically including:

[0167] First, in order to achieve deep fusion between different modal features, an interaction matrix based on the self-attention mechanism is constructed to capture the complementary relationship between image and time series features. The generation formula of the interaction matrix Θ is as follows:

[0168]

[0169] Among them, Q and K represent query and key respectively. According to the calculation above, here, the output of 3D convolution is Indicates that the output of the multi-head self-attention mechanism is It means that q α A modal input is required for calculation, k β is the input of another modality, d k Represents the high dimension mapped to, q I represents the sky image feature as the query, k I Indicates that time series features are used as keys;

[0170] Secondly, using the interaction matrix, the spatial and temporal series features are concatenated through the column dimension to obtain the fused feature vector. The calculation formula of the fusion process is as follows:

[0171]

[0172] in, and They are sky image and time series features respectively, and Concat represents the concatenation operation.

[0173] This embodiment further illustrates the present invention in conjunction with the accompanying drawings and specific implementation methods, and the examples are as follows:

[0174] The experimental data used in the example analysis comes from a public dataset in Folsom, California, covering the period from 2014 to 2015 with a temporal resolution of 1 minute. It includes local measurement data synchronized with the time of the sky image, such as global horizontal irradiance GHI, direct normal irradiance DNI, diffuse horizontal irradiance DHI, air temperature, humidity, relative air pressure, and wind speed. After further processing of the raw data, external data such as clear sky irradiance, such as solar zenith angle, solar azimuth, and solar position, can be obtained.

[0175] 1. Data preprocessing

[0176] Data cleaning and screening: Clean the experimental data and remove dates with consecutive missing timestamps to ensure the timestamp alignment between the sky image and the irradiance data; synchronously add external data, such as timestamp-based clear sky irradiance data, to enrich the characteristic information of irradiance prediction.

[0177] Sky image preprocessing: The original sky image is downsampled from 1536×1536 pixels to 128×128 pixels, and the bilinear interpolation method is used to retain the key information of cloud characteristics while reducing the amount of calculation; the RGB sky image is converted to a grayscale image to reduce image redundant information so that the model pays more attention to the outline and distribution of the cloud layer. The normalized red-to-blue ratio NRBR is calculated to highlight the thick and thin features of the cloud layer; the cloud movement velocity vector field between adjacent frames is calculated by the dense optical flow method to extract the dynamic information of the cloud layer, including the movement direction and speed of the cloud layer in space.

[0178] Time series data processing: All time series data are aligned with image data by timestamp to ensure data synchronization. In addition, the data is normalized to the interval [-1,1] to eliminate the dimensional differences between different data sources and ensure the consistency of model input.

[0179] Dataset division: The data is divided into training set, validation set and test set, with the data from 2014 as the training set, the odd-numbered days in 2015 for validation, and the even-numbered days for testing. This division scheme ensures the diversity of the data set and covers different weather conditions to improve the generalization ability of the model.

[0180] (II) Model training

[0181] The experiment was implemented in Python 3.8 using the PyTorch framework and accelerated by CUDA 11.1. In terms of hardware configuration, NVIDIA GeForce RTX 3090 GPU and Dell PowerEdge T640 server were used to ensure efficient model training. The Adam optimizer was used for model training, and the initial learning rate was set to 1e-4. The model input was a 10-minute historical data sequence, and the prediction time window was set to 5 minutes.

[0182] 3D-ResNet module settings. The 3D-ResNet module contains a 3D convolution layer, using a 7×7×7 convolution kernel and initialized to 64 filters. The module consists of four residual blocks, and the number of filters in each block increases from 64 to 512 layer by layer, gradually extracting spatiotemporal features. Figure 3 shown.

[0183] Informer module settings. The Informer time series encoder contains a 1D convolution layer with a kernel size of 3×3 to align the length of the time series with the spatial features output by the 3D-ResNet module. Figure 4 shown.

[0184] Multimodal fusion module. The fusion module receives feature inputs from different sources, including sky image features extracted by the 3D-ResNet module and time series features encoded by the Informer module; it uses a cross-attention mechanism to calculate the similarity between queries, keys, and values, and generates attention weights; it generates a feature fusion matrix to combine sky image features with time series features to form a comprehensive feature representation; the fused feature vector is passed to the regression layer to generate the final ultra-short-term irradiance prediction result. Figure 5 shown.

[0185] The present invention proposes an ultra-short-term solar irradiance prediction model based on multimodal data fusion, which is particularly suitable for real-time prediction needs in photovoltaic power generation. By introducing multimodal fusion of sky images, irradiance data and meteorological data, the model shows excellent results in capturing short-term cloud dynamics and irradiance fluctuations.

[0186] The present invention also provides another aspect, that is, a multi-modal ultra-short-term solar radiation prediction system based on sky images, the system comprising:

[0187] The acquisition module is used to acquire sky images through a full sky imager, collect irradiance data and meteorological data at the same time, and calculate relevant irradiance data under theoretical clear sky, wherein the relevant irradiance data includes global horizontal irradiance GHI, direct normal irradiance DNI and scattered horizontal irradiance DHI;

[0188] A preprocessing module is used to grayscale the sky image, extract cloud thickness information using a normalized red-to-blue ratio, extract dynamic features using a dense optical flow method, and normalize historical irradiation data and meteorological data to obtain preprocessed data and a sky image;

[0189] The feature extraction module is used to take multiple frames of continuous sky images as the input of the 3D convolutional neural network, extract the spatiotemporal features of the sky images, capture the dynamic changes of the clouds, output the high-dimensional feature vector of the sky images, and use the Informer model to encode the time series of irradiation data and meteorological data to obtain time series features;

[0190] The fusion module uses the multimodal interaction matrix to concatenate the high-dimensional feature vector of the sky image and the time series features through the column dimension to obtain the fused feature vector, and inputs the fused features into the regression layer to generate the final irradiance prediction result.

[0191] Further, including:

[0192] In the feature extraction module, multiple frames of continuous sky images are used as 3D convolutional neural network inputs to extract the spatiotemporal features of the sky images, capture the dynamic changes of the clouds, and output high-dimensional feature vectors of the sky images, specifically including:

[0193] Input data processing: Each input contains multiple frames of continuous sky images, and the image size is uniformly adjusted to 128×128 pixels; the image is grayscaled, downsampled and normalized, and then represented by x and input into the 3D-ResNet module;

[0194] 3D convolution operation: In 3D-ResNet, convolution operations are performed in both spatial and temporal dimensions. The size of each 3D convolution kernel is k×k×d, where k is the convolution kernel size in the spatial dimension and d is the convolution depth in the temporal dimension. The specific operations of 3D convolution are as follows:

[0195]

[0196] Among them, Y i,j,t Represents the output feature value at position (i, j, t), W is the 3D convolution kernel weight, and b is the bias term. Residual connection: In order to avoid the gradient vanishing problem in deep networks, the input features are directly passed to the convolution output. The calculation method is as follows:

[0197] F(x)=x+Conv3D(x)(9)

[0198] Other technical features of the multimodal ultra-short-term solar irradiation prediction system based on sky images of the present invention are similar to those of the corresponding multimodal ultra-short-term solar irradiation prediction method based on sky images, and will not be repeated here.

[0199] Finally, the present invention also provides a computer-readable storage medium, on which a multimodal ultra-short-term solar irradiance prediction program based on sky images is stored. When the multimodal ultra-short-term solar irradiance prediction program based on sky images is executed by a processor, the steps of the multimodal ultra-short-term solar irradiance prediction method based on sky images as described above are implemented.

[0200] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0202] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multimodal ultra-short-term solar irradiance prediction method based on sky images, characterized in that: The method includes: Acquire sky images through a total sky imager, collect irradiance data and meteorological data at the same time, and calculate relevant irradiance data under theoretical clear sky, wherein the relevant irradiance data include global horizontal irradiance GHI, direct normal irradiance DNI and scattered horizontal irradiance DHI; Grayscale processing is performed on the sky image, and cloud thickness information is extracted using a normalized red-to-blue ratio, and dynamic features are extracted using a dense optical flow method, and historical irradiation data and meteorological data are normalized to obtain preprocessed data and a sky image; The features of multiple frames of continuous sky images are used as the input of the 3D convolutional neural network to extract the spatiotemporal features of the sky images, capture the dynamic changes of the clouds, and output the high-dimensional feature vector of the sky images. At the same time, the Informer model is used to encode the time series of irradiation data and meteorological data to obtain time series features. Using the multimodal interaction matrix, the high-dimensional feature vector of the sky image and the time series features are concatenated through the column dimension to obtain the fused feature vector, and the fused features are input into the regression layer to generate the final irradiance prediction result.

2. The multimodal ultra-short-term solar radiation prediction method based on sky images according to claim 1 is characterized in that: The calculation of relevant irradiance data under theoretical clear sky includes: First, calculate the global horizontal irradiance GHI as: Among them, H is the hour angle, φ is the latitude, δ is the solar declination angle, θ z is the solar zenith angle, θ A is the azimuth; Secondly, the Perez model was used to calculate the GHI, DNI and DHI values ​​at each time point under theoretical clear sky. GHI=exp(-cg2·airmass ab ·(fh1+fh2·(tl-1)))(2) Among them, fh1, fh2, cg2 are altitude correction factors, airmass ab is the absolute air quality, tl is the Lincoln turbidity; bnci1 is the initial calculated normal direct radiation value, based on the influence of atmospheric transmittance and Lincoln turbidity, and bnci2 is another restriction condition based on GHI and zenith angle, which is used to constrain the value range of DNI: bnci1=b·exp(-0.09·airmass ab ·(tl-1));(3) bnci2=(1-(0.1-0.2·exp(-tl) / (0.1+0.882 / fh1)) / cosθ z ; (4) The final DNI takes the minimum value of bnci1 and bnci2: DNI=min(bnci1,bnci2); (5) DHI = GHI - DNI · cos θz (6).

3. The multimodal ultra-short-term solar radiation prediction method based on sky images according to claim 2 is characterized in that: The use of normalized red-to-blue ratio I NRBR Extract cloud thickness information, expressed as: Among them, B is the blue channel value, R is the red channel value, and I NRBR Ability to highlight the distribution of clouds in the image.

4. The multimodal ultra-short-term solar radiation prediction method based on sky images according to claim 3 is characterized in that: The dense optical flow method is used to extract dynamic features, and the motion velocity vector field of the cloud layer is calculated between consecutive image frames to capture the movement information of the cloud layer in all directions. The dense optical flow method calculation specifically includes: I(x,y,t)=I(x+u,y+v,t+Δ t ); (8) Where (x, y) represents the pixel position in the image at time t, (x+u, y+v) represents the position of the pixel after the pixel changes at time t+Δt, and I(x, y, t) represents the pixel value of the image at position (x, y) at time t. Perform a first-order Taylor expansion on the right side of the equation to obtain: is the gradient of the image in the x and y directions, u and v are the pixel movement speeds that need to be solved, that is, the optical flow components; Assuming that the optical flow of the image is a constant value in a window of size m×m, we can get the following set of equations: The system of equations is represented by the abbreviation A·V=b, and the linear equation is solved: V=(A T A) -1 A T b; (11) Where V = [u, v] T is the optical flow vector of each pixel on the x,y axis; Therefore, the optical flow vector at each pixel on the cloud map is calculated to obtain the same pixel size as the original cloud map. Two dynamic feature optical flow images.

5. The multimodal ultra-short-term solar radiation prediction method based on sky images according to claim 3 is characterized in that: The method uses the features of multiple frames of continuous sky images as the input of the 3D convolutional neural network, extracts the spatiotemporal features of the sky images, captures the dynamic changes of the clouds, and outputs the high-dimensional feature vector of the sky images, specifically including: Input data processing: Each input contains multiple frames of continuous sky image features The image size is uniformly adjusted to 128×128 pixels; after downsampling and normalization, it is represented by vector x and input into the 3D-ResNet module, where I Gray is the feature of the image after grayscale conversion; 3D convolution operation: In 3D-ResNet, convolution operations are performed in both spatial and temporal dimensions. The size of each 3D convolution kernel is k×k×d, where k is the convolution kernel size in the spatial dimension and d is the convolution depth in the temporal dimension. The specific operations of 3D convolution are as follows: Among them, Y i,j,t represents the output feature value at position (i, j, t), W p,q,r is the 3D convolution kernel weight, b is the bias term, (p, q, r) represents the index inside the convolution kernel, W pqr ·X i+p,j+q,t+r Is a dot product operation, representing an operation logic X i+p,j+q,t+r Represents a value in this local sub-block, which is located at the (t+r)th time step, (i+p)th width position and (j+q)th height position of the input feature map. The subscripts t, i, and j mean the time, width, and height position of the output feature map. The final output is expressed as Residual connection: In order to avoid the vanishing gradient problem in deep networks, the input features are directly passed to the convolution output. The calculation method is as follows: F(x)=x+Conv3D(x) (13) Among them, Conv3D is the abbreviation of the 3D convolution operation calculation process mentioned above.

6. The multimodal ultra-short-term solar radiation prediction method based on sky images according to claim 5 is characterized in that: The Informer model is used to encode the time series of irradiation data and meteorological data to obtain high-dimensional time series features. include: Input data processing: Perform 1D convolution operation on the preprocessed multidimensional time series data, and add the position information of the time series data to the input features for position encoding. Includes: global horizontal irradiance GHI, direct normal irradiance DNI, diffuse horizontal irradiance DHI and meteorological data; Sparse self-attention mechanism: Generates a sparse self-attention score, calculated as follows: Where Q, K, and V represent the query, key, and value matrices respectively, and d k For the scaling factor, in sparse self-attention, only some key-value pairs with the largest contribution value are selected for calculation, reducing redundant calculations and thus improving the computational efficiency of the model; Multi-head self-attention mechanism: By running multiple attention heads in parallel, the model captures the characteristics of irradiance and meteorological data at different time scales. The formula is as follows: MultiHead(Q,K,V)=Concat(head1,...,head h )W O (15) Each attention head performs calculations on different feature dimensions, and the results are concatenated and transformed linearly to obtain the final output; Among them, AB[·] represents the attention calculation link including sparse self-attention mechanism and multi-head self-attention mechanism, including multi-head self-attention, Conv1d(·) represents the convolution filter with a convolution kernel size of 3 in the time dimension, and the ELU(·) activation function is used to improve the nonlinear mapping ability of this link. Finally, a maximum pooling layer with a stride of 2 is added, and Informer(·) is used to represent the entire time feature extraction link, that is:

7. The multimodal ultra-short-term solar radiation prediction method based on sky images according to claim 6 is characterized in that: The step of obtaining a fused feature vector by splicing the high-dimensional feature vector of the sky image and the time series feature through the column dimension specifically includes: First, in order to achieve deep fusion between different modal features, an interaction matrix based on the self-attention mechanism is constructed to capture the complementary relationship between image and time series features. The generation formula of the interaction matrix Θ is as follows: Where Q and K represent query and key respectively. Here, in formula (18), the output of 3D convolution is I is used to represent the output of the multi-head self-attention mechanism. Using G to represent, q α A modal input is required for calculation, k β is the input of another modality, d k Represents the high dimension mapped to, q I represents the sky image feature as the query, k G Indicates that time series features are used as keys; Secondly, using the interaction matrix, the spatial and temporal series features are concatenated through the column dimension to obtain the fused feature vector. The calculation formula of the fusion process is as follows: in, and They are sky image and time series features respectively, and Concat represents the concatenation operation.

8. A multi-modal ultra-short-term solar irradiance prediction system based on sky images, characterized in that: The system includes: The acquisition module is used to acquire sky images through a full sky imager, collect irradiance data and meteorological data at the same time, and calculate relevant radiation data under theoretical clear sky, wherein the relevant radiation data includes global horizontal irradiance GHI, direct normal irradiance DNI and scattered horizontal irradiance DHI; A preprocessing module is used to grayscale the sky image, extract cloud thickness information using a normalized red-to-blue ratio, extract dynamic features using a dense optical flow method, and normalize historical radiation data and meteorological data to obtain preprocessed data and a sky image; The feature extraction module is used to take the features of multiple frames of continuous sky images as the input of the 3D convolutional neural network, extract the spatiotemporal features of the sky images, capture the dynamic changes of the clouds, output the high-dimensional feature vector of the sky images, and use the Informer model to encode the time series of irradiation data and meteorological data to obtain time series features; The fusion module uses the multimodal interaction matrix to concatenate the high-dimensional feature vector of the sky image and the time series features through the column dimension to obtain the fused feature vector, and inputs the fused features into the regression layer to generate the final irradiance prediction result.

9. The multimodal ultra-short-term solar radiation prediction system based on sky images according to claim 8, characterized in that: In the feature extraction module, The features of multiple frames of continuous sky images are used as the input of the 3D convolutional neural network to extract the spatiotemporal features of the sky images, capture the dynamic changes of the clouds, and output the high-dimensional feature vector of the sky images, including: Input data processing: Each input contains multiple frames of continuous sky image features The image size is uniformly adjusted to 128×128 pixels; after downsampling and normalization, it is represented by vector x and input into the 3D-ResNet module, where I Gray is the feature of the image after grayscale conversion; 3D convolution operation: In 3D-ResNet, convolution operations are performed in both spatial and temporal dimensions. The size of each 3D convolution kernel is k×k×d, where k is the convolution kernel size in the spatial dimension and d is the convolution depth in the temporal dimension. The specific operations of 3D convolution are as follows: Among them, Y i,j,t represents the output feature value at position (i, j, t), W p,q,r is the 3D convolution kernel weight, b is the bias term, (p, q, r) represents the index inside the convolution kernel, W pqr ·X i+p,j+q,t+r Is a dot product operation, representing an operation logic X i+p,j+q,t+r Represents a value in this local sub-block, which is located at the (t+r)th time step, (i+p)th width position and (j+q)th height position of the input feature map. The subscripts t, i, and j mean the time, width, and height position of the output feature map. The final output is expressed as Residual connection: In order to avoid the vanishing gradient problem in deep networks, the input features are directly passed to the convolution output. The calculation method is as follows: F(x)=x+Conv3D(x); Among them, Conv3D is the abbreviation of the 3D convolution operation calculation process mentioned above.

10. The multi-modal ultra-short-term solar radiation prediction system based on sky images according to claim 9, characterized in that: The Informer model is used to encode the time series of irradiation data and meteorological data to obtain high-dimensional time series features. include: Input data processing: Perform 1D convolution operation on the preprocessed multidimensional time series data, and add the position information of the time series data to the input features for position encoding. Includes: global horizontal irradiance GHI, direct normal irradiance DNI, diffuse horizontal irradiance DHI and meteorological data; Sparse self-attention mechanism: Generates a sparse self-attention score, calculated as follows: Where Q, K, and V represent the query, key, and value matrices respectively, and d k For the scaling factor, in sparse self-attention, only some key-value pairs with the largest contribution value are selected for calculation, reducing redundant calculations and thus improving the computational efficiency of the model; Multi-head self-attention mechanism: By running multiple attention heads in parallel, the model captures the characteristics of irradiance and meteorological data at different time scales. The formula is as follows: MultiHead(Q,K,V)=Concat(head1,...,head h )W O ; Each attention head performs calculations on different feature dimensions, and the results are concatenated and transformed linearly to obtain the final output; Among them, AB[·] represents the attention calculation link including sparse self-attention mechanism and multi-head self-attention mechanism, including multi-head self-attention, Conv1d(·) represents the convolution filter with a convolution kernel size of 3 in the time dimension, and the ELU(·) activation function is used to improve the nonlinear mapping ability of this link. Finally, a maximum pooling layer with a stride of 2 is added, and Informer(·) is used to represent the entire time feature extraction link, that is:

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multimodal ultra-short-term solar irradiance prediction program based on sky images, and when the multimodal ultra-short-term solar irradiance prediction program based on sky images is executed by a processor, the steps of the multimodal ultra-short-term solar irradiance prediction method based on sky images as described in any one of claims 1 to 7 are implemented.

Images