Photovoltaic ultra-short-term power prediction method and system based on multi-modal information fusion

Abstract

The invention discloses a photovoltaic ultra-short-term power prediction method and system based on multi-modal information fusion, and belongs to the technical field of photovoltaic power generation prediction. The method comprises the steps: obtaining an all-sky image sequence, segmenting a cloud layer and a sky region, and extracting cloud layer boundary features; thickness features are analyzed by calculating cloud pixel point brightness indexes, overall motion features of a cloud layer are determined based on adjacent frame displacement vectors, a time sequence sub-image covering a sun area in the future is reversely captured in combination with sun position coordinates, and then time sequence features and image features are extracted by adopting a dual-channel fusion mechanism. And finally, introducing a weather-dependent decoder: identifying weather types through a lightweight classifier, dynamically weighting the fused features based on the types, and outputting a photovoltaic power prediction result. According to the invention, through a multi-scale feature cooperation and dynamic weighting mechanism, the prediction robustness under a complex meteorological condition is significantly improved.

Description

TECHNICAL FIELD

[0001] The application relates to a multi-modal information fusion photovoltaic ultra-short-term power prediction method and system, and belongs to the technical field of photovoltaic power generation prediction BACKGROUND

[0002] With large-scale grid connection of photovoltaic power stations, the output is easily affected by weather mutations (such as dynamic cloud cover), resulting in short-term power fluctuations, which significantly increases the difficulty of power distribution network scheduling. Traditional photovoltaic power prediction methods mainly rely on a single data source (such as satellite images or time series historical data), which is difficult to effectively associate large-scale cloud dynamic characteristics with photovoltaic power fluctuation rules, and has low prediction accuracy under complex weather conditions. At the same time, the existing technology is insufficiently robust to time series noise and sensor missing data, and has defects such as calculation redundancy and noise sensitivity. These problems restrict the stability and generalization ability of the prediction model, and cannot meet the demand for high-precision ultra-short-term power prediction. Therefore, a new method is needed that can fuse multi-modal information, adapt to complex weather changes and improve prediction robustness. SUMMARY

[0003] The purpose of the present application is to provide a multi-modal information fusion photovoltaic ultra-short-term power prediction method and system, which can solve the problems of low prediction accuracy under complex weather conditions and insufficient robustness to time series noise and sensor missing data in the prior art.

[0004] To solve the above technical problems, the present application is realized by the following technical scheme.

[0005] In a first aspect, the present application provides a multi-modal information fusion photovoltaic ultra-short-term power prediction method, comprising: obtaining a full-sky image sequence and preliminarily dividing the full-sky image sequence into cloud and sky regions; performing boundary recognition on sky pixel points in the sky region, extracting cloud boundary features according to the boundary recognition result, and redividing the cloud and sky regions; calculating the brightness index of each cloud pixel point in the redivided cloud, and extracting the thickness feature of the cloud according to the brightness index; based on the full-sky image sequence of adjacent two frames, calculating the displacement vector of each cloud pixel point, and determining the overall motion feature of the cloud by averaging all displacement vectors; determining the sun center coordinates according to the current solar zenith angle and azimuth angle, and combining the displacement vector to reversely capture the time sequence sub-image covering the future solar region; inputting the historical photovoltaic power time sequence, the clear sky index time sequence converted from the all-sky image sequence, and the solar zenith angle time sequence into the GRU model to obtain time sequence features, and inputting the cloud layer boundary features, thickness features, motion features, and time sequence sub-image features into the Swin-Transformer model to obtain image features, and performing weighted fusion on the image features and the time sequence features to generate multi-modal fusion features; outputting a weather type from a lightweight weather classifier using a historical all-sky image sequence, and performing weighted summation on the multi-modal fusion features according to the weather type through a weather-dependent decoder to output a photovoltaic power prediction result.

[0006] With reference to the first aspect, further, the all-sky image sequence is divided into a cloud layer and a sky region, including: randomly selecting two pixel points as initial clustering centers of the sky region and the cloud layer; calculating the Euclidean distance of each pixel point in the all-sky image sequence to the initial clustering center, if the Euclidean distance of the current pixel point to the sky region is less than the Euclidean distance of the current pixel point to the cloud layer, the current pixel point is assigned to the sky region, if the Euclidean distance of the current pixel point to the sky region is greater than the Euclidean distance of the current pixel point to the cloud layer, the current pixel point is assigned to the cloud layer; recomputing the new center of each cluster according to the current pixel assignment result; repeating the operations of assigning pixel points and updating clustering centers until the clustering centers no longer change.

[0007] With reference to the first aspect, further, boundary identification is performed on the sky pixel points in the sky region, including: if the ratio of the sky pixel points to the total pixel points is greater than a preset pixel threshold, the initial clustering result is adopted; if the ratio of the sky pixel points to the total pixel points is less than the preset pixel threshold, a boundary correction operation is performed, specifically including: based on each sky pixel point, calculating the gray difference between it and the adjacent pixel points, if the gray difference is greater than a preset gradient threshold, it is marked as a boundary point, if the gray difference is not greater than the preset gradient threshold, it is marked as a non-boundary point; traversing all the sky pixel points, if the current sky pixel point is marked as a boundary point, its sky region attribution is retained, if the current sky pixel point is marked as a non-boundary point, and there is no any boundary point in its neighborhood, the current sky pixel point is classified into the cloud layer; based on the above re-clustering result, the sky region and the cloud layer are re-divided.

[0008] With reference to the first aspect, further, the thickness feature of the cloud layer is extracted according to the luminance index, comprising: Based on the each cloud pixel point, the corresponding luminance index L is calculated according to its RGB value; The luminance index L is compared with a preset luminance threshold If the luminance index L is greater than the preset luminance threshold , the current cloud pixel point is marked as a first type of pixel; if the luminance index L is less than the preset luminance threshold , the current cloud pixel point is marked as a second type of pixel; The proportion of the first type of pixel in the total pixel N of the all-sky image, the proportion of the second type of pixel in the total pixel N of the all-sky image, and the average value T of the luminance index L of all cloud pixel points are calculated respectively, and are taken as the thickness feature of the cloud layer.

[0009] With reference to the first aspect, further, the overall motion feature of the cloud layer is determined by averaging all displacement vectors, comprising: Based on the sequence of two consecutive all-sky images, a flow constraint equation is established for each cloud pixel point in the cloud layer; For each cloud pixel point, the flow constraint equations of the associated pixel points in its neighborhood are combined to construct an overdetermined equation, and the displacement vector of each cloud pixel point is obtained by solving the overdetermined equation; The average value of all cloud pixel displacement vectors is calculated, and the overall motion feature of the cloud layer is determined according to the average value.

[0010] With reference to the first aspect, further, the displacement vector expression of each cloud pixel point is:

[0011] Wherein, u represents the moving direction of the cloud layer; v represents the speed of the cloud layer; n represents the number of measurement points; i represents the i-th measurement point; represents the change rate of the i-th measurement point in the x-axis direction; represents the change rate of the i-th measurement point in the y-axis direction; represents the gray scale change caused by time change.

[0012] With reference to the first aspect, further, the sun center coordinates are determined according to the current sun zenith angle and azimuth angle, and the time sequence sub-image covering the future sun area is captured in reverse combination with the displacement vector, comprising: According to the sun zenith angle And the azimuth angle And the radius r of the all-sky image sequence, the pixel coordinates of the sun center in the all-sky image are calculated; wherein the expression of the pixel coordinates of the sun center in the all-sky image is: ; ; wherein, denotes the horizontal coordinate of the sun center point; denotes the vertical coordinate of the sun center point; r denotes the radius of the all-sky image sequence; denotes the zenith angle; denotes the azimuth angle; capturing a time-series sub-image covering the sun region within a preset time along the reverse direction of the displacement vector from the pixel coordinate of the sun center as a starting point.

[0013] With reference to the first aspect, further, the generating the multi-modal fusion feature comprises: performing global average pooling operation on the image feature and the time-series feature respectively to obtain a first reduced dimension feature and a second reduced dimension feature ; concatenating the first reduced dimension feature and the second reduced dimension feature to obtain a preliminary fusion feature ; processing the preliminary fusion feature through a fully connected layer and a ReLU activation function to obtain a weight coefficient of the image feature and a weight coefficient of the time-series feature ; based on the weight coefficient of the image feature and the weight coefficient of the time-series feature , performing weighted fusion on the image feature and the time-series feature to generate a multi-modal fusion feature .

[0014] With reference to the first aspect, further, the weather-dependent decoder comprises a plurality of parallel sub-decoders; performing global average pooling operation on the multi-modal fusion feature to obtain a one-dimensional fusion feature; processing the one-dimensional fusion feature using the plurality of parallel sub-decoders to generate power prediction values corresponding to different weather types for each sub-decoder; analyzing historical all-sky image sequences through a lightweight weather classifier to output weather condition features; based on the weather condition features, calculating a fusion weight vector for each sub-decoder; performing weighted summation on the power prediction values according to the fusion weight vector to generate a photovoltaic power prediction result.

[0015] In a second aspect, a photovoltaic ultra-short-term power prediction system based on multi-modal information fusion comprises: An image preliminary division module is configured to obtain a sequence of all-sky images and preliminarily divide the sequence of all-sky images into a cloud layer and a sky region; A first feature extraction module is configured to perform boundary recognition on sky pixel points in the sky region, extract cloud layer boundary features according to the boundary recognition result, and redivide the cloud layer and the sky region; A second feature extraction module is configured to calculate a brightness index of each cloud pixel point in the redivided cloud layer and extract thickness features of the cloud layer according to the brightness index; A third feature extraction module is configured to calculate a displacement vector of each cloud pixel point based on two adjacent frames of the sequence of all-sky images, and determine overall motion features of the cloud layer by averaging all displacement vectors; A fourth feature extraction module is configured to determine a sun center coordinate according to a current solar zenith angle and azimuth angle, and reversely capture a time-series sub-image covering a future solar region in combination with the displacement vector; An adaptive fusion module comprises: A time-series encoding unit is configured to input a historical photovoltaic power time series, a clearness index time series converted from the sequence of all-sky images, and a solar zenith angle time series into a GRU model to obtain time-series features An image encoding unit is configured to input the cloud layer boundary features, the thickness features, the motion features, and the time-series sub-image features into a Swin-Transformer model to obtain image features; A data fusion unit is configured to perform weighted fusion of the image features and the time-series features to generate multi-modal fusion features; A decoding prediction module is configured to output a weather type from a lightweight weather classifier using a historical sequence of all-sky images, and perform weighted summation of the multi-modal fusion features according to the weather type through a weather-dependent decoder to output a photovoltaic power prediction result.

[0016] Compared with the prior art, the present application has the following beneficial effects: The improved clustering-boundary algorithm, cloud layer feature extraction, LK optical flow method (Lucas-Kanade optical flow method) for capturing dynamic changes of the cloud layer, sub-image capture of the solar region, and the design of the adaptive fusion module and the weather-dependent decoder significantly improve the accuracy and reliability of photovoltaic ultra-short-term power prediction. In particular, under complex weather conditions, it not only reduces the sensitivity to sensor noise and missing data, but also enhances the stability and generalization ability of the model, thereby effectively solving the shortcomings of traditional methods in prediction accuracy and robustness, and greatly improving the problem of power distribution network scheduling. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 Fig. 1 shows a structural diagram of an adaptive fusion module provided by an embodiment of the present application; Figure 2 Fig. 2 shows a photovoltaic ultra-short-term power prediction model based on multi-modal information fusion of historical data and sky images provided by an embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solutions of the present application, but not limitations of the technical solutions of the present application. In the case of no conflict, the technical features in the embodiments of the present application and the embodiments can be combined with each other.

[0019] The term "and / or", only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / ", generally represents that the associated objects before and after it are in an "or" relationship. Embodiment 1

[0020] This embodiment introduces a multi-modal information fusion photovoltaic ultra-short-term power prediction method, which includes the following steps: Step S1, acquiring a full-sky image sequence, and preliminarily dividing the full-sky image sequence into cloud layer and sky region; First, continuously collect a full-sky image sequence, and then apply a k-means clustering algorithm to preliminarily identify clouds and sky in the full-sky image sequence, so as to roughly divide the full-sky image sequence into cloud layer and sky region.

[0021] Step S2, boundary recognition is performed on the sky pixel points in the sky region, cloud layer boundary features are extracted according to the boundary recognition result, and the cloud layer and the sky region are re-divided; In the embodiment of the present application, in order to solve the problem that thick clouds are easily misidentified as sky, boundary recognition is performed on all sky pixel points, the above-mentioned roughly divided region, i.e. the initial clustering result, is corrected according to the boundary recognition result, and the cloud layer and the sky region are further re-divided.

[0022] Step S3, calculating the brightness index of each cloud pixel point in the re-divided cloud layer, and extracting the thickness feature of the cloud layer according to the brightness index; Based on the reclassified cloud layer and sky area, the brightness index of each cloud pixel point is calculated by using the color information of each cloud pixel point to distinguish thin clouds and thick clouds, wherein the brightness index L is positively correlated with the light transmittance of the cloud layer, that is, the larger the brightness index, the better the light transmittance, and the cloud layer is classified as a thin cloud; otherwise, the smaller the brightness index, the stronger the light blocking, and the cloud layer is classified as a thick cloud.

[0023] In order to evaluate the cloud thickness in the whole all-sky image sequence, the average value of all cloud pixel points in the red, green and blue color channels (RGB) is calculated to comprehensively reflect the thickness of the cloud layer. The thickness not only considers the proportion of the cloud layer coverage (cloud cover), but also combines the color depth information (thickness), so as to more comprehensively describe the cloud thickness.

[0024] Step S4, based on the adjacent two frames of all-sky image sequence, the displacement vector of each cloud pixel point is calculated, and the overall motion feature of the cloud layer is determined by averaging all displacement vectors; Step S5, the sun center coordinates are determined according to the current sun zenith angle and azimuth angle, and the time sequence sub-image covering the sun region in the future is captured in combination with the displacement vector; The calculated displacement vector is used to predict the moving direction and speed of the cloud layer in the future; based on the moving direction and speed, the time sequence sub-image covering the sun region in the preset time is captured from the pixel coordinates of the sun center as the starting point in the opposite direction of the displacement vector, and each time sequence sub-image has the same resolution. Finally, the captured time sequence sub-image is used for further feature extraction and model training to improve the prediction accuracy.

[0025] Step S6, the historical photovoltaic power time sequence, the clear sky index time sequence converted from the all-sky image sequence and the sun zenith angle time sequence are input into the GRU model (gated recurrent unit), to obtain the time sequence feature, and the cloud layer boundary feature, the thickness feature, the motion feature and the time sequence sub-image feature are input into the Swin-Transformer model (sliding window transformer), to obtain the image feature, and the image feature and the time sequence feature are weighted and fused to generate a multi-modal fusion feature; The two types of data features are captured by global average pooling, and are simplified and connected to form a preliminary multi-modal fusion feature.

[0026] Step S7, the weather type is output by the lightweight weather classifier using the historical all-sky image sequence, and the multi-modal fusion feature is weighted and summed according to the weather type by the weather dependent decoder to output the photovoltaic power prediction result.

[0027] First, the multi-modal fusion feature is converted into one-dimensional feature by global average pooling, and each sub-decoder generates different photovoltaic prediction results for different weather types.

[0028] In the embodiment of the present application, a lightweight weather classifier is introduced to extract the current weather condition by analyzing the historical all-sky image sequence, and the weather information is represented in the form of soft labels, which reflects the similarity between different weather states and the closeness of the current weather to the predefined weather types, for example, giving a probability distribution such as [sunny 20%, cloudy 70%, overcast 10%].

[0029] Based on the weather information, the weights of the outputs of each sub-decoder are calculated, and the weights of each sub-decoder are weighted and summed to obtain the final photovoltaic power prediction value adjusted according to the current weather information. Embodiment 2

[0030] Based on the same inventive concept as Embodiment 1, this embodiment introduces a multi-modal information fusion photovoltaic ultra-short-term power prediction method, which includes: (1) Cloud boundary feature extraction: a. Cloud-sky recognition based on k-means algorithm The k-means algorithm is applied to the preliminary cloud-sky recognition of the all-sky image sequence, which divides the image into two parts of cloud layer and sky area, including: Step S11, initialization of cluster centers: randomly select two pixel points as the initial cluster centers of the sky area and the cloud layer, and set the distribution of the two initial cluster centers as and ; Step S12, distance of cloud pixels to each cluster center: calculate the Euclidean distance of each pixel point in the all-sky image sequence to the initial cluster center; The expression of the distance of the cloud pixels to each cluster center is: (1) Wherein, represents the set of pixel points in the all-sky image sequence; represents the set of all pixel points of the jth class; represents the color value of the ith cloud pixel point in the red channel; represents the color value of the jth cloud pixel point in the red channel; represents the color value of the ith cloud pixel point in the green channel; represents the color value of the jth cloud pixel point in the green channel; represents the color value of the ith cloud pixel point in the blue channel; represents the color value of the jth cloud pixel point in the blue channel.

[0031] Step S13, assign cloud pixel points: assign each cloud pixel point to the nearest cluster center, specifically:​ If the Euclidean distance from the current pixel point to the sky region is less than the Euclidean distance from the current pixel point to the cloud layer ( ), the current pixel point is assigned to the sky region; if the Euclidean distance from the current pixel point to the sky region is greater than the Euclidean distance from the current pixel point to the cloud layer ( ), the current pixel point is assigned to the cloud layer. Step S14, recalculate the cluster center: according to the current pixel assignment result, recalculate the new center of each cluster; wherein the new center of the cluster is the average value of the color values of all pixel points in the region, and its expression is: (2) wherein, represents the number of all pixel points in the region.

[0032] Step S15, iterative optimization: repeat the operations of assigning cloud pixel points and updating cluster centers until the cluster center no longer changes or the preset maximum iteration number is reached.

[0033] b. Boundary correction After the above k-means clustering is completed, in order to further improve the accuracy, the boundary of the sky pixel points in the sky region is identified, if the ratio of the sky pixel points to the total pixel points is greater than a preset pixel threshold, the initial clustering result is adopted; if the ratio of the sky pixel points to the total pixel points is less than the preset pixel threshold, the boundary correction operation is performed, in the embodiment of the present application, the pixel threshold is set to 30%.

[0034] wherein, the boundary correction operation includes: Step S111, calculate the gradient difference: based on each sky pixel point, calculate the gray difference between it and the adjacent pixel point, if the gray difference is greater than a preset gradient threshold, it is marked as a boundary point, if the gray difference is not greater than the preset gradient threshold, it is marked as a non-boundary point; Specifically, the calculation expression of the gray difference is: (3) wherein, represents the gray difference; i represents the gray value of the current pixel point; represents the gray value of the adjacent pixel point of the current pixel point; represents the set gradient threshold.

[0035] Step S112, correct clustering errors: traverse all sky pixel points, if the current sky pixel point is marked as a boundary point, keep its sky region attribution; if the sky pixel point is marked as a non-boundary point, and there is no any boundary point in its neighborhood, consider that the sky pixel point is a clustering error point, and classify it from the sky region to the cloud layer; Step S113, re-divide the sky region and the cloud layer based on the above re-clustering result.

[0036] In summary, through the above k-means algorithm, the whole sky image sequence can be initially divided into the cloud layer and the sky region, and the subsequent boundary correction step further improves the accuracy of classification, ensuring that the final cloud layer and sky region are more accurate, providing reliable data support for subsequent cloud type classification and solar irradiance prediction.

[0037] (2) Cloud thickness feature Through the improved clustering-boundary correction algorithm in step (1), the cloud layer and the sky region are accurately identified, and the brightness index of each cloud pixel point is calculated based on its color information to distinguish between thin clouds and thick clouds: brighter clouds are considered as thin clouds, and darker ones are classified as thick clouds. Then, in order to evaluate the cloud thickness in the whole image, the average value of all cloud pixel points in the red, green and blue color channels is calculated to comprehensively reflect the thickness of the cloud layer. In this way, not only the proportion of cloud coverage is considered, but also the information of color depth is combined, so that the cloud thickness feature is more comprehensively described.

[0038] Specifically, according to the brightness index, the thickness feature of the cloud layer is extracted, including the following steps: Step S21, calculate the brightness index: based on each cloud pixel point, calculate the corresponding brightness index L according to its RGB value; The brightness index reflects the overall brightness of the cloud pixel point, and its expression is: (4) Where L represents the brightness index; R represents the color value of the red channel; G represents the color value of the green channel; B represents the color value of the blue channel.

[0039] Step S22, set threshold: compare the brightness index L with the preset brightness threshold , if the brightness index L is greater than the preset brightness threshold , the current cloud pixel point is marked as the first type of pixel, i.e. thin cloud; if the brightness index L is less than the preset brightness threshold , the current cloud pixel point is marked as the second type of pixel, i.e. thick cloud; Step S23, calculate cloud coverage: calculate the proportion of the first type of pixel in the total number of pixels N of the all-sky image, and the proportion of the second type of pixel in the total number of pixels N of the all-sky image, respectively. The expression of the proportion of the first cloud pixel point and the second cloud pixel point in the all-sky image is: (5) wherein, represents the proportion of all cloud pixel points in the all-sky image; represents the number of cloud pixel points; N represents the total number of all cloud pixel points and sky pixel points in the all-sky image.

[0040] Step S24, calculate cloud layer thickness: calculate the average value T of the brightness index L of all cloud pixel points, and the expression is: (6) wherein, i represents the index value of the cloud pixel point; T represents the average value of the brightness index.

[0041] Step S25, cloud layer type classification: take the proportion of the first cloud pixel point and the second cloud pixel point in the all-sky image and the average value of the brightness index L as the thickness characteristics of the cloud layer.

[0042] Further, based on the proportion of the sky pixel point and the first cloud pixel and the second cloud pixel, the cloud-sky type of the entire all-sky image sequence is determined, and the expression is: (7) wherein, type represents the cloud-sky type; Clearsky represents clear sky; Block cloud represents cumulus cloud; Thin cloud represents the first pixel point, i.e. thin cloud; Thick cloud represents the second pixel point, i.e. thick cloud; represents the total number of sky pixels; represents the total number of thin cloud pixels; represents the total number of thick cloud pixels; N represents the total number of pixels.

[0043] Through the above steps, the numerical characteristics of the cloud layer type are extracted from the obtained cloud layer characteristics, and are classified into clear sky, cumulus cloud, thin cloud and thick cloud, thereby providing key input for subsequent solar irradiance prediction.

[0044] (3) Cloud layer motion characteristics Firstly, a series of all-sky image sequences arranged in time sequence are needed, which are obtained by the all-sky imager at fixed time intervals (for example, one picture per minute), and contain the position and shape change information of the cloud layer at different time points.

[0045] Then, the following assumptions are determined: 1) the gray value of the corresponding pixel point remains unchanged between adjacent frames; which means that the pixel color at the same position will not change significantly within an hour interval, and the expression of the gray value is: (8) wherein, represents the gray value of the full-sky image at position (x, y) and time t; x represents the coordinate of the full-sky image on the x-axis; y represents the coordinate of the full-sky image on the y-axis; t represents time.

[0046] This sub-condition ensures the validity of the Taylor expansion and simplifies the subsequent calculation 2) the target relative displacement is small; which means that all pixel points in a small area move at the same speed, which helps to reduce the calculation complexity and improve stability.

[0047] 3) the pixel points in the neighborhood have the same motion trend.

[0048] Then, the Lucas-Kanade method is used to calculate the displacement vector of all cloud pixel points, and the specific steps are as follows: Step S31, based on all continuous two-frame full-sky image sequences, a flow constraint equation is established for each cloud pixel point in the cloud layer; Specifically, the Taylor expansion is performed on the right side of the above formula (8), and the expression of the first derivative is as follows: (9) wherein, represents a high-order term.

[0049] It should be noted that under the small displacement assumption, the high-order term can be ignored, and the simplified equation is as follows: (10) Let , then formula (10) can be rewritten in matrix form, that is, the flow constraint equation is as follows: (11) wherein, represents the spatial gradient x component; represents the spatial gradient y component; u represents the moving direction of the cloud layer; v represents the speed of the cloud layer; represents the gray value change caused by time change.

[0050] Step S32, for each cloud pixel point, the flow constraint equations of the associated pixel points in its neighborhood are combined to construct an overdetermined equation, and the displacement vector of each cloud pixel point is obtained by solving the overdetermined equation; It should be noted that for a pixel point and other pixel points in its neighborhood, an overdetermined equation system can be constructed. If considering n pixel points in the neighborhood of a pixel point, the following equation system can be obtained: (12) wherein, represents the first spatial gradient x component; represents the second spatial gradient x component; represents the n-th spatial gradient x component; represents the first spatial gradient y component; represents the second spatial gradient y component; represents the n-th spatial gradient y component; represents the first temporal gradient; represents the second temporal gradient; represents the n-th temporal gradient.

[0051] The formula (12) is solved using the least square method to find the optimal displacement vector [x, y] as follows: (13) wherein, represents the rate of change of the i-th measurement point in the x-axis direction; represents the rate of change of the i-th measurement point in the y-axis direction; n represents the number of measurement points; i represents the i-th measurement point.

[0052] Step S33, calculate the average value of all cloud pixel displacement vectors as the motion vector of the all-sky image sequence.

[0053] This step helps to capture the overall moving trend of the cloud layer in the entire image, rather than relying only on the motion information of a single pixel point. Specifically, the displacement vectors of each pixel point are added and divided by the number of pixel points to obtain an average displacement vector representing the motion trend of the entire image.

[0054] (4) Time sequence sub-image features Capture the time sequence sub-image covering the future solar region, including the following steps: Step S41, determine the pixel coordinates (center of the circle) of the sun center in the image according to the zenith angle and azimuth angle information of the sun at that time.

[0055] Specifically, according to the zenith angle and azimuth angle of the sun and the radius r of the all-sky image sequence, the pixel coordinates of the sun center in the all-sky image are calculated; wherein the expression of the pixel coordinates of the sun center in the all-sky image is: (14) (15) wherein, denotes the horizontal coordinate of the sun center point; denotes the vertical coordinate of the sun center point; r denotes the radius of the all-sky image sequence; denotes the zenith angle; denotes the azimuth angle; Step S42, starting from the pixel coordinates of the sun center, capture time-series sub-images covering the sun area within a preset time along the reverse direction of the displacement vector.

[0056] Specifically, starting from the sun center and moving in the direction opposite to the cloud movement direction, ten sub-images covering the sun area every minute in the next ten minutes are captured frame by frame from the current time point, and the resolution of each sub-image is 60x60 pixels. Finally, these captured sub-images will be used for further feature extraction and model training to improve the prediction accuracy.

[0057] The above process not only focuses on the area where the sun itself is located, but also takes into account the dynamic changes of the surrounding clouds, thereby providing key input data for subsequent feature extraction and solar irradiance prediction. In this way, local features closely related to the change of solar irradiance can be extracted from the original all-sky image, improving the accuracy of the prediction model.

[0058] Referring to Figure 2 , the photovoltaic ultra-short-term power prediction model based on the multi-modal information fusion of historical data and sky images is described as follows (5)~(6), and the specific process is as follows: (5) Multi-modal fusion features Step S51, input the historical photovoltaic power time series, the clear sky index time series converted from the all-sky image sequence, and the sun zenith angle time series into the GRU model to obtain time-series features; Step S52, input the cloud boundary feature, thickness feature, motion feature and time-series sub-image feature obtained in (1)~(4) into the Swin-Transformer model to obtain image features; Step S53, weight and fuse the image features and time-series features to generate multi-modal fusion features.

[0059] Specifically, the process of generating multi-modal fusion features is as follows: Step S531, global information capture: perform global average pooling operations on the image features and the time-series features to obtain first reduced dimension features and second reduced dimension features , whose expressions are as follows: (16) wherein, denotes the first reduced dimension feature; denotes the first reduced dimension feature; denotes the image feature; denotes the time sequence feature.

[0060] Step S532, preliminary fusion: the first reduced dimension feature and the second reduced dimension feature are spliced to obtain a preliminary fusion feature , and the expression is as follows: (17) wherein, denotes the preliminary fusion feature.

[0061] Step S533, learning hidden correlation: the preliminary fusion feature is processed through a full connection layer plus a ReLU activation function to obtain the weight coefficient of the image feature and the weight coefficient of the time sequence feature . Referring to Figure 1 , the preliminary fusion feature is processed through a full connection layer plus a ReLU activation function, so that the model learns the hidden correlation between the image feature and the time sequence feature; then, another full connection layer is used to learn the weight, and a Sigmoid function is applied to adjust the weight to 0-1, to calculate the weight coefficient of the image feature and the weight coefficient of the time sequence feature .

[0062] wherein, the expression of the weight coefficient of the image feature is as follows: (18) wherein, denotes the weight coefficient of the image feature; ReLU denotes an activation function; denotes a full connection layer.

[0063] Step S534, based on the weight coefficient of the image feature and the weight coefficient of the time sequence feature , the image feature and the time sequence feature are input to an adaptive weight module for weighted fusion to generate a multi-modal fusion feature , and the expression is as follows: (19) wherein, denotes the multi-modal fusion feature.

[0064] The above steps work together to enable the AFM (adaptive module) to adaptively adjust the weights according to the specific characteristics of each feature and its hidden correlations, thereby highlighting the task-related aspects of the features in the fusion process. This not only enhances the feature representation capability of the model, but also improves the overall performance. In this way, AFM can effectively integrate information from different modalities, such as sky images and photovoltaic power generation data, to improve prediction accuracy.

[0065] (6) Future photovoltaic power prediction A weather-dependent decoder is introduced, which includes multiple parallel sub-decoders, each generating different prediction outputs for different weather types. A lightweight weather classifier is also introduced to analyze historical all-sky image sequences to extract the current weather conditions and represent weather information through soft labels, reflecting the similarity between different weather states and the closeness of the current weather to predefined weather types.

[0066] Specifically, generating a photovoltaic power prediction result includes the following steps: Step S61, global average pooling: performing global average pooling on the multi-modal fusion feature to obtain a one-dimensional fusion feature, whose expression is: (20) wherein, denotes the one-dimensional fusion feature.

[0067] Step S62, multi-sub-decoder processing: using the plurality of parallel sub-decoders to process the one-dimensional fusion feature, using the plurality of parallel sub-decoders to process the one-dimensional fusion feature, to generate power prediction values corresponding to different weather types for each sub-decoder; wherein each sub-decoder is composed of two fully connected layers FC, and a ReLU activation function is added after each layer.

[0068] The expression of the power prediction value is: (21) wherein, denotes the output of the i-th sub-decoder, and for all sub-decoders, the output dimension is where B is the batch size and T is the time span of the prediction.

[0069] Step S63, obtaining weather information: analyzing the historical all-sky image sequence through the lightweight weather classifier to output weather condition features; Specifically, to determine the fusion weights of these different outputs, it is necessary to first extract information of the current weather condition from the historical sky image sequence. This is achieved through a lightweight weather classifier, which includes a convolutional layer and two residual blocks to extract hidden features, and then a simple classification head composed of two fully connected layers to obtain soft labels for classification.

[0070] The output weather type expression is: (22) where, represents the weather condition feature corresponding to the i-th sky image, where it is assumed that the weather is classified into three types: sunny, cloudy, and overcast; represents the i-th sky image.

[0071] Step S64, based on the weather condition feature, calculate the fusion weight vector of each sub-decoder; This step ensures that the weather condition closer to the current time point plays a more important role in making accurate predictions.

[0072] Step S65, final prediction result fusion: the power prediction value is weighted and summed according to the fusion weight vector to generate the future photovoltaic power prediction result.

[0073] The expression of the future photovoltaic power prediction result is: (23) where, represents the future photovoltaic power prediction result; i represents the index number of the sub-decoder; k represents the number of sub-decoders; represents the dynamic weight of the i-th sub-decoder; represents the output of the i-th sub-decoder.

[0074] In this way, the WDD (Weather Dependent Decoder) can dynamically adjust parameters according to real-time weather conditions, thereby improving the accuracy of predictions under different weather conditions. This mechanism not only considers the impact of weather changes on photovoltaic power generation, but also allows the model to learn the similarity between different weather conditions through soft labels, further improving the adaptability and prediction accuracy of the model. Embodiment 3

[0075] A multi-modal information fusion photovoltaic ultra-short-term power prediction system, comprising: An image preliminary division module is configured to obtain a sequence of full-sky images and preliminarily divide the sequence of full-sky images into cloud layers and sky regions. The first feature extraction module is configured to perform boundary recognition on sky pixels in the sky region, extract cloud layer boundary features according to a boundary recognition result, and redivide the cloud layer and the sky region; The second feature extraction module is configured to calculate a brightness index of each cloud pixel in the newly divided cloud layer, and extract a thickness feature of the cloud layer according to the brightness index; The third feature extraction module is configured to calculate a displacement vector of each cloud pixel based on the sequence of full-sky images of adjacent two frames, and determine an overall motion feature of the cloud layer by averaging all displacement vectors; The fourth feature extraction module is configured to determine a sun center coordinate according to a current sun zenith angle and azimuth, and reversely capture a time sequence sub-image covering a sun region in the future in combination with the displacement vector; The adaptive fusion module comprises: The time sequence encoding unit is configured to input a historical photovoltaic power time sequence, a clear sky index time sequence converted from the sequence of full-sky images, and a sun zenith angle time sequence into a GRU model to obtain time sequence features The image encoding unit is configured to input the cloud layer boundary features, the thickness features, the motion features, and the time sequence sub-image features into a Swin-Transformer model to obtain image features; The data fusion unit is configured to perform weighted fusion on the image features and the time sequence features to generate multi-modal fusion features; The decoding prediction module is configured to output a weather type from a lightweight weather classifier by using a historical sequence of full-sky images, and perform weighted summation on the multi-modal fusion features according to the weather type by using a weather-dependent decoder to output a photovoltaic power prediction result.

[0076] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative rather than limiting. Those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and these are all within the protection of the present application.

Claims

1. A photovoltaic ultra-short-term power prediction method based on multimodal information fusion, characterized in that, include: Acquire a full-sky image sequence and initially divide the full-sky image sequence into cloud and sky regions; Boundary recognition is performed on the sky pixels in the sky region, and cloud boundary features are extracted based on the boundary recognition results to re-divide the cloud and sky regions. Calculate the brightness index of each cloud pixel in the re-divided cloud layer, and extract the thickness feature of the cloud layer based on the brightness index; Based on a sequence of all-sky images from two adjacent frames, the displacement vector of each cloud pixel is calculated, and the overall motion characteristics of the cloud layer are determined by averaging all displacement vectors. The coordinates of the solar center are determined based on the current solar zenith angle and azimuth angle, and the displacement vector is used to capture the temporal sub-image of the future solar coverage area in reverse. The historical photovoltaic power time series, the clear sky index time series converted from the all-sky image series, and the solar zenith angle time series are input into the GRU model to obtain time series features. The cloud boundary features, thickness features, motion features, and time series sub-image features are input into the Swin-Transformer model to obtain image features. The image features and the time series features are then weighted and fused to generate multimodal fusion features. Using historical all-sky image sequences, a lightweight weather classifier outputs the weather type, and a weather-dependent decoder performs a weighted summation of the multimodal fusion features based on the weather type to output the photovoltaic power prediction result.

2. The photovoltaic ultra-short-term power prediction method based on multimodal information fusion according to claim 1, characterized in that, The entire sky image sequence is divided into cloud and sky regions, including: Two pixels are randomly selected as the initial cluster centers for the sky region and the cloud layer; Calculate the Euclidean distance from each pixel in the all-sky image sequence to the initial cluster center. If the Euclidean distance from the current pixel to the sky region is less than the Euclidean distance from the current pixel to the cloud layer, then the current pixel is assigned to the sky region; if the Euclidean distance from the current pixel to the sky region is greater than the Euclidean distance from the current pixel to the cloud layer, then the current pixel is assigned to the cloud layer. Based on the current pixel allocation results, recalculate the new center for each cluster; Repeat the operations of allocating pixels and updating cluster centers until the cluster centers no longer change.

3. The photovoltaic ultra-short-term power prediction method based on multimodal information fusion according to claim 1, characterized in that, Boundary recognition of sky pixels in the sky region includes: If the ratio of the sky pixels to the total pixels is greater than a preset pixel threshold, the initial clustering result is used; If the ratio of the sky pixels to the total pixels is less than a preset pixel threshold, a boundary correction operation is performed, specifically including: For each sky pixel, calculate the grayscale difference between it and its neighboring pixels. If the grayscale difference is greater than a preset gradient threshold, it is marked as a boundary point. If the grayscale difference is not greater than the preset gradient threshold, it is marked as a non-boundary point. Traverse all sky pixels. If the current sky pixel is marked as a boundary point, retain its sky region affiliation. If the current sky pixel is marked as a non-boundary point and there are no boundary points in its neighborhood, classify the current sky pixel as a cloud. Based on the above re-clustering results, the sky region and cloud layer are re-divided.

4. The photovoltaic ultra-short-term power prediction method based on multimodal information fusion according to claim 1, characterized in that, Extracting cloud thickness features based on the brightness index includes: Based on each cloud pixel, the corresponding brightness index L is calculated according to its RGB value; The brightness index L is compared with a preset brightness threshold. If the brightness index L is greater than the preset brightness threshold, the comparison is made. If the current cloud pixel is not more than a preset brightness threshold, then mark it as a first-class pixel; Then, the current cloud pixel will be marked as a second type of pixel; The proportions of the first type of pixels to the total number of pixels N in the sky image, the proportions of the second type of pixels to the total number of pixels N in the sky image, and the average value T of the brightness index L of all cloud pixels are calculated and used as the thickness feature of the cloud layer.

5. The photovoltaic ultra-short-term power prediction method based on multimodal information fusion according to claim 1, characterized in that, The method of determining the overall motion characteristics of the cloud layer by averaging all displacement vectors includes: Based on two consecutive frames of all-sky image sequence, an optical flow constraint equation is established for each cloud pixel within the cloud layer; For each cloud pixel, an overdetermined equation is constructed by combining the optical flow constraint equations of its neighboring pixels. The displacement vector of each cloud pixel is obtained by solving the overdetermined equation. Calculate the average value of all cloud pixel displacement vectors, and determine the overall motion characteristics of the cloud layer based on the average value.

6. The photovoltaic ultra-short-term power prediction method based on multimodal information fusion according to claim 5, characterized in that, The displacement vector expression for each cloud pixel is: ； Where u represents the direction of cloud movement; v represents the speed of cloud movement; n represents the number of measurement points; and i represents the i-th measurement point. This represents the rate of change of the i-th measurement point along the x-axis. This represents the rate of change of the i-th measurement point along the y-axis. This indicates the change in grayscale value caused by changes over time.

7. The photovoltaic ultra-short-term power prediction method based on multimodal information fusion according to claim 1, characterized in that, The coordinates of the solar center are determined based on the current solar zenith angle and azimuth angle. Then, the displacement vector is used to capture a temporal sub-image of the region that will cover the sun in the future, including: According to the solar zenith angle and azimuth Given the radius r of the entire sky image sequence, calculate the pixel coordinates of the sun's center in the entire sky image; wherein, the expression for the pixel coordinates of the sun's center in the entire sky image is: ； ； in, The x-coordinate representing the center point of the sun; The vertical coordinate of the sun's center point is represented by r; r represents the radius of the entire sky image sequence. Indicates the zenith angle; Indicates azimuth; Starting from the pixel coordinates of the center of the sun, a temporal sub-image covering the solar region is captured along the opposite direction of the displacement vector within a preset time period.

8. The photovoltaic ultra-short-term power prediction method based on multimodal information fusion according to claim 1, characterized in that, The generated multimodal fusion features include: For the image features With time series characteristics Perform global average pooling operations separately to obtain the first dimensionality reduction feature. Second dimensionality reduction features ; The first dimensionality reduction feature Second dimensionality reduction features By splicing the parts together, preliminary fusion characteristics are obtained. ; The preliminary fusion features are processed using a fully connected layer and a ReLU activation function. Processing is performed to obtain image features. Weighting coefficients and time series characteristics Weighting coefficients; Based on the above weighting coefficients and The weighting coefficients of the image features With time series characteristics The inputs are weighted and fused to generate multimodal fusion features. .

9. The photovoltaic ultra-short-term power prediction method based on multimodal information fusion according to claim 8, characterized in that, The weather-dependent decoder includes multiple parallel sub-decoders; For the multimodal fusion features Perform global average pooling to obtain one-dimensional fused features; The one-dimensional fused features are processed using the multiple parallel sub-decoders to generate power prediction values ​​for different weather types for each sub-decoder. A lightweight weather classifier is used to analyze historical all-sky image sequences and output weather condition features. Based on the weather condition characteristics, calculate the fusion weight vector for each sub-decoder; The predicted power values ​​are weighted and summed according to the fusion weight vector to generate the photovoltaic power prediction result.

10. A photovoltaic ultra-short-term power prediction system with multimodal information fusion, characterized in that, include: The image preliminary segmentation module is used to acquire the whole sky image sequence and initially segment the whole sky image sequence into cloud and sky regions; The first feature extraction module is used to perform boundary recognition on the sky pixels in the sky region, extract cloud boundary features based on the boundary recognition results, and re-divide the cloud and sky regions. The second feature extraction module is used to calculate the brightness index of each cloud pixel in the re-divided cloud layer, and extract the thickness feature of the cloud layer based on the brightness index. The third feature extraction module is used to calculate the displacement vector of each cloud pixel based on the full sky image sequence of two adjacent frames, and determine the overall motion characteristics of the cloud by averaging all displacement vectors. The fourth feature extraction module is used to determine the coordinates of the solar center based on the current solar zenith angle and azimuth angle, and to capture the temporal sub-images of the future solar coverage area in reverse by combining the displacement vector. The adaptive fusion module includes: The time-series encoding unit is used to input the historical photovoltaic power time series, the clear sky index time series converted from the all-sky image series, and the solar zenith angle time series into the GRU model to obtain time-series features. The image encoding unit is used to input the cloud boundary features, thickness features, motion features, and temporal sub-image features into the Swin-Transformer model to obtain image features; The data fusion unit is used to weightedly fuse the image features and the temporal features to generate multimodal fusion features; The decoding and prediction module is used to utilize historical all-sky image sequences, output weather types by a lightweight weather classifier, and then use a weather-dependent decoder to perform a weighted summation of the multimodal fusion features based on the weather types to output photovoltaic power prediction results.

Images