A multi-modal ultra-short-term photovoltaic power generation power prediction method and system

Abstract

The embodiment of the present application provides a kind of multimodal ultra-short-term photovoltaic power generation power prediction method and system, the method comprises: obtaining historical sky image data, and synchronously collecting real-time power data, to obtain feature image and feature video;Feature image sequence is input into ST-GEN pre-training model and is iteratively trained, to determine the ST-GEN pre-training model after training;Feature video is input into VST-GEN pre-training model and is iteratively trained, to determine the VST-GEN pre-training model after training;ST-GEN pre-training model, VST-GEN pre-training model and the encoding model corresponding to real-time power data are multimodal fusion, to build multimodal model, based on multimodal model, the prediction value of photovoltaic power generation power corresponding to sky image data is output.

Description

Technical Field

[0001] This invention relates to the field of photovoltaic power generation technology, and in particular to a multimodal ultra-short-term photovoltaic power generation prediction method and system. Background Technology

[0002] With advancements in technology and market integration, solar power generation is experiencing significant growth. Despite this rapid development, challenges remain. One major obstacle is its intermittent nature, which can lead to instability in grid integration. Solar power relies on sunlight, which is not constant and fluctuates due to factors such as cloud cover, day / night cycles, and seasonal variations. These fluctuations cause rapid and unpredictable changes in power generation, thus affecting grid stability. Ultra-short-term (USST) photovoltaic (PV) power forecasting utilizes existing historical power generation data and other data to predict PV power generation for the next few minutes to hours. This type of forecasting is crucial for optimizing the integration of solar energy with the grid. Accurate USST solar power forecasting can reduce energy losses and ensure a continuous and stable energy supply.

[0003] Currently, existing technologies for ultra-short-term photovoltaic (PV) power generation forecasting can be broadly categorized into two types to address the aforementioned issues. One type relies on numerical information, such as historical PV power generation data and weather forecast data, to estimate future ultra-short-term PV power output. However, cloud movement is difficult to reflect in numerical data, making accurate predictions challenging for numerically based methods. The other type combines sky image data with other data, such as historical PV power generation data, to estimate future PV power output. However, methods using sky images typically struggle to extract sufficient information from sky image sequences, resulting in lower accuracy. Summary of the Invention

[0004] To address the problems existing in the prior art, embodiments of the present invention provide a multimodal ultra-short-term photovoltaic power generation prediction method and system.

[0005] This invention provides a multimodal ultra-short-term photovoltaic power generation prediction method, the method comprising:

[0006] Historical sky image data is acquired, and real-time power data is collected simultaneously. The historical sky image data is preprocessed to obtain feature images and feature videos.

[0007] The feature image is input into the ST-GEN pre-trained model for training, including image occlusion, feature encoding and image restoration steps, and iterative training is performed using the loss function between the restored image and the original image to determine the trained ST-GEN pre-trained model.

[0008] The feature video is input into the VST-GEN pre-trained model for training, including feature encoding and feature dilation steps. The VST-GEN pre-trained model is then iteratively trained using the loss function between the feature-dilated predicted video sequence and the actual video sequence at the target time to determine the trained VST-GEN pre-trained model.

[0009] The ST-GEN pre-trained model, VST-GEN pre-trained model, and the encoding model corresponding to the real-time power data are fused in a multimodal manner to construct a multimodal model. Based on the multimodal model, the predicted value of photovoltaic power generation corresponding to the sky image data is output.

[0010] In one embodiment, the method further includes:

[0011] The feature image is segmented along its width and height dimensions to generate a corresponding image block feature sequence. Each image block is transformed into a feature vector through linear projection during model training.

[0012] The feature video is segmented in terms of its width, height, and time to generate a corresponding spatiotemporal video block feature sequence. Each video block is mapped to a feature vector during model training.

[0013] In one embodiment, the method further includes:

[0014] Set the features of some image patches in the image patch feature sequence to zero;

[0015] The image patch feature sequence is processed by a multi-stage feature encoder to calculate window self-attention, and the window boundaries are shifted to calculate the self-attention between windows. Then, adjacent image patches are merged until the ST-GEN intermediate features are output in the final stage.

[0016] The ST-GEN intermediate features and the image block features with zero features are input into the image block recovery layer to generate the restored image corresponding to the image block with zero features.

[0017] In one embodiment, the method further includes:

[0018] The feature encoder performs window self-attention calculation on the spatiotemporal video block feature sequence through a multi-stage feature encoder, shifts the window boundaries, calculates self-attention between windows, and then merges the features of adjacent video blocks until the VST-GEN intermediate features are output in the final stage.

[0019] The intermediate features of VST-GEN are input into the feature dilation layer for reverse block merging, and the predicted video at the corresponding time point is output.

[0020] In one embodiment, the method further includes:

[0021] The ST-GEN pre-trained model and the VST-GEN pre-trained model interact to find the cross-correlation relationship between image features and video features, and combine their own residuals to generate enhanced visual features through MLP fusion.

[0022] The enhanced visual features are interacted with real-time power data to find the cross-correlation relationship between the enhanced visual features and power features, and combined with their own residuals, multimodal fusion features are generated through MLP fusion.

[0023] In one embodiment, the method further includes:

[0024] Using the image features as a query, find the relevant motion context in the video features, including:

[0025] ,

[0026] in, To combine image features with motion context, The query vector represents the image features. The key vector for video features. Let K be the dimension of the key vector. This is a value vector of video features. Image features;

[0027] Using the video features as a query, find static reference points in the image features, including:

[0028] ,

[0029] in, To combine video features at static reference points, The query vector represents the video features. The key vector of image features, A vector of image feature values. For video features;

[0030] The process of finding the cross-correlation relationship between image features and video features, and combining it with its own residuals, uses MLP fusion to generate enhanced visual features, including:

[0031] ,

[0032] in, To enhance visual features;

[0033] Using the enhanced visual features as a query, the corresponding power generation mode is searched within the power features, including:

[0034] ,

[0035] in, For power generation mode, To enhance the query vector of visual features, The key vector represents the power characteristics. This is a value vector representing the power characteristics;

[0036] Using the power feature as a query, the enhanced visual features are used to find visual evidence that causes power fluctuations, including:

[0037] ,

[0038] in, As visual evidence, This is the query vector for power features. To enhance the key vector of visual features, To enhance the value vector of visual features, Power characteristics;

[0039] The process of finding the cross-correlation relationship between enhanced visual features and power features, and combining it with its own residuals, generates multimodal fusion features through MLP fusion, including:

[0040] ,

[0041] in, This is a multimodal fusion feature.

[0042] This invention provides a multimodal ultra-short-term photovoltaic power generation prediction system, the system comprising:

[0043] The acquisition module is used to acquire historical sky image data and simultaneously collect real-time power data, and preprocess the historical sky image data to obtain feature images and feature videos.

[0044] The first training module is used to input the feature image into the ST-GEN pre-training model for training, including image occlusion, feature encoding and image restoration steps, and iteratively trains the model using the loss function between the restored image and the original image to determine the trained ST-GEN pre-training model.

[0045] The second training module is used to input the feature video into the VST-GEN pre-trained model for training, including feature encoding and feature dilation steps, and to perform iterative training using the loss function between the feature-dilated predicted video sequence and the actual video sequence at the target time to determine the trained VST-GEN pre-trained model.

[0046] The multimodal training module is used to perform multimodal fusion of the ST-GEN pre-trained model, the VST-GEN pre-trained model, and the encoding model corresponding to the real-time power data to construct a multimodal model. Based on the multimodal model, the module outputs the predicted value of photovoltaic power generation corresponding to the sky image data.

[0047] In one embodiment, the system further includes:

[0048] The image segmentation module is used to cut and divide the feature image in its width and height dimensions to generate corresponding image block feature sequences. Each image block is transformed into a feature vector through linear projection during its model training.

[0049] The video segmentation module is used to cut and divide the feature video in terms of its width, height, and time, and generate corresponding spatiotemporal video block feature sequences. Each video block is mapped to a feature vector during its model training.

[0050] This invention provides an electronic device, including a processor and a memory;

[0051] The processor is connected to the memory;

[0052] The memory is used to store executable program code;

[0053] The processor runs a program corresponding to the executable program code stored in the memory to perform the methods described in one or more embodiments.

[0054] This invention provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described multimodal ultra-short-term photovoltaic power generation prediction method.

[0055] In view of the above, in one or more embodiments of this specification, historical sky image data is acquired, and real-time power data is collected simultaneously. The historical sky image data is preprocessed to obtain feature images and feature videos. The feature images are input into the ST-GEN pre-trained model for training, including image occlusion, feature encoding, and image restoration steps. Iterative training is performed using the loss function between the restored image and the original image to determine the trained ST-GEN pre-trained model. The feature videos are input into the VST-GEN pre-trained model for training, including feature encoding and feature dilation steps. Iterative training is performed using the loss function between the predicted video sequence after feature dilation and the actual video sequence at the target time to determine the trained VST-GEN pre-trained model. The ST-GEN pre-trained model, the VST-GEN pre-trained model, and the encoding model corresponding to the real-time power data are fused in a multimodal manner to construct a multimodal model. Based on the multimodal model, the predicted value of photovoltaic power generation corresponding to the sky image data is output. This allows the use of a multi-modal cross-attention structure to fuse features from different modalities, enabling better fusion of features from three modalities and resulting in more accurate power generation prediction results. Furthermore, in the multimodal models, the ST-Gen model can fuse the feature encoding layer and the pixel restoration layer. By training to restore the original sky image, the feature encoding layer of the ST-Gen model can better capture the state information of the sky image. The VST-Gen model can fuse the feature encoding layer and the feature dilation layer. By training to predict future sky images, the feature encoding layer of the VST-Gen model can better capture the motion information of clouds in the sky image, further improving the accuracy of the prediction results. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart of a multimodal ultra-short-term photovoltaic power generation prediction method provided in one embodiment of this specification.

[0058] Figure 2 This is a schematic diagram of the structure of a multimodal ultra-short-term photovoltaic power generation prediction system provided in one embodiment of this specification.

[0059] Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0060] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0061] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term shall remain consistent throughout the specification.

[0062] like Figure 1 As shown, this embodiment of the invention provides a multimodal ultra-short-term photovoltaic power generation prediction method, including:

[0063] Step S102: Acquire historical sky image data and simultaneously collect real-time power data. Preprocess the historical sky image data to obtain feature images and feature videos.

[0064] Specifically, historical sky image data is acquired, and real-time power data is simultaneously collected via the power plant's power meter. The timestamps of both are strictly aligned, and the sampling frequencies are matched. For sky image data acquisition, a vertically upward fisheye camera can capture full-hemisphere sky images, as the fisheye perspective maximizes sky coverage, avoids obstruction, and provides complete cloud movement observation. The acquired data is then preprocessed, including data cleaning and filtering to exclude all photovoltaic power generation data before sunrise and after sunset. For problematic photovoltaic power generation data, including null values, zero values, and negative values, linear interpolation is used to repair and complete the data based on preceding and following photovoltaic power generation data. Regarding data normalization, considering the large fluctuations and uneven distribution of photovoltaic power generation data, traditional Min-Max and Z-Score normalization methods may lead to decreased accuracy. Therefore, a dynamic normalization method can be adopted, which applies the sequence data g of each set of historical photovoltaic power generation data during the training process. i Let i = 1, 2, ..., k, where k is the total number of historical power generation sequence data and h is the future photovoltaic power generation sequence data used as a label. i Let i = 1, 2, ..., m, where m is the total number of future power generation sequence data. Its normalization method can be written as:

[0065]

[0066]

[0067] and This represents normalized historical and future photovoltaic power generation data.

[0068] For sky image data, the preprocessing step can involve labeling and distinguishing different parts. A threshold segmentation method can be used, dividing the sky image into three parts: sky, clouds, and sun, by setting different thresholds. Then, each part is re-labeled to obtain a feature image, where the sky is labeled 0, clouds are labeled 1, and the sun is labeled 2. The labeled feature image has its pixel channels reduced from 3 to 1, resulting in more regular features that are easier for the deep learning model to utilize in subsequent steps. Simultaneously, the feature images corresponding to 16 adjacent sky images at each time step are concatenated using a matrix method to obtain a feature video.

[0069] Step S104: Input the feature image into the ST-GEN pre-trained model for training, including image occlusion, feature encoding and image restoration steps, and perform iterative training using the loss function between the restored image and the original image to determine the trained ST-GEN pre-trained model.

[0070] Specifically, for the preprocessed feature images and videos, the pixel information of the images and videos is transformed into serialization tokens that the Transformer architecture can process. For the feature images, the block layer divides the image along its width and height spatial dimensions, cutting the entire image into a series of regular image blocks (e.g., 16x16 pixel squares). Each image block is transformed into a feature vector through linear projection. The feature vectors of all image blocks are arranged in spatial order, thus forming the image block feature sequence.

[0071] Furthermore, after image and video segmentation, the feature sequences of image patches are trained by constructing an ST-Gen pre-trained model. The training steps may include zeroing (masking) the features of a portion of the image patches (e.g., 50% of the image patches). After image masking, the model is forced to learn a strong semantic understanding of the overall layout and context of the sky, clouds, and sun from the remaining visible patches.

[0072] The image patch feature sequence is then processed through a Swing Transformer-based feature encoding layer in the pre-trained model. This process involves multiple stages: each stage consists of several Swing Transformer Blocks and a block merging layer. In each stage, the Swing Transformer Block first divides the feature map into non-overlapping local windows. Then, a Window Multi-head Self-Attention (W-MSA) layer is used to compute local self-attention features within each window. Next, a Shifted Window Multi-head Self-Attention (SW-MSA) module is used to compute inter-window self-attention features, which are then input into the next Swing Transformer Block. After multiple Swing Transformer Blocks, adjacent blocks are merged through a block merging layer to gradually reduce the feature map resolution and increase the number of channels, simulating the role of pooling layers in CNNs, and gradually constructing a feature pyramid from local details to global semantics. The merged feature maps are fed into the next stage, which continues to use the Swin Transformer Block and block merging layer to reduce the feature map resolution and increase the number of channels until the final stage outputs features, which are the state features of the intermediate sky image in ST-Gen. These features are then fed into the image patch reconstruction layer. That is, the high-level semantic features output by the encoder (ST-Gen intermediate features) are fed together with mask tokens (representing occluded blocks) into the lightweight image patch reconstruction layer. Its task is to predict the corresponding original image patch features for each mask token based on the context. In other words, it generates the occluded image patch features from the mask generation module.

[0073] Then, the ST-Gen pre-trained model is trained. The masked image patch features, the image patch features generated by the mask generation module, and the corresponding original input image patch features are input into the loss function. The training loss function is:

[0074]

[0075] In the formula N M x represents the total number of occluded image patch features in a batch. i The feature of the i-th original input image patch, y iThe model reconstructs the features of the i-th occluded image patch. Based on the loss function calculation results, the model parameters are updated using the backpropagation algorithm. This training process improves the feature encoding layer's ability to extract features from sky images in the ST-GEN model. By training the pre-trained ST-Gen model, the model learns the compositional patterns of sky scenes, thereby acquiring robust and semantically rich feature representations.

[0076] Step S106: Input the feature video into the VST-GEN pre-trained model for training, including feature encoding and feature dilation steps, and perform iterative training using the loss function between the predicted video sequence after feature dilation and the actual video sequence at the target time to determine the trained VST-GEN pre-trained model.

[0077] Specifically, for feature videos, the chunking layer operates in three dimensions: width, height, and time. A series of consecutive frames are treated as a three-dimensional cube, which is then divided into smaller spatiotemporal cube chunks along both time and space. Each such cube chunk is also mapped to a feature vector. Ultimately, the feature vectors of all spatiotemporal blocks are arranged in spatiotemporal order to form a spatiotemporal video block feature sequence.

[0078] Furthermore, a VST-Gen pre-trained model is constructed to train the spatiotemporal video patch feature sequences. Compared to the ST-Gen pre-trained model, which processes feature images, the VST-Gen pre-trained model takes spatiotemporal video patch feature sequences as input. Its goal is to compress and abstract specific feature sequences into high-level semantic features (VST-Gen intermediate features) containing core motion patterns through a deep network.

[0079] The spatiotemporal video block feature sequences are then processed through a 3D Swin Transformer-based feature encoding layer in the pre-trained model. This process includes multiple stages: each stage consists of several 3D Swin Transformer Blocks and a block merging layer. In each stage, the 3D Swin Transformer Block first divides the feature map into non-overlapping local windows. Then, a 3D Window Multi-head Self-Attention (3D W-MSA) layer is used to calculate local self-attention features within each window. Next, a 3D Shifted Window Multi-head Self-Attention (3D SW-MSA) module is used to calculate self-attention features between windows. Unlike the 2D version of ST-Gen, the 3D windows are defined in three dimensions: time, height, and width. These features are then fed into the next 3D Swin Transformer Block. After multiple 3D Swin Transformer Blocks, adjacent blocks are merged through a block merging layer to progressively reduce the feature map resolution and increase the number of channels. The merged features are fed into the next stage, which continues to use a Swin Transformer Block and a block merging layer to reduce the feature map resolution and increase the number of channels until the final stage. The output of the final stage is the motion features of the VST-Gen intermediate historical sky image, which is fed into the feature dilation layer. This layer consists of four stages. Each stage consists of several 3D Swin Transformer Blocks and video block dilation units. In each stage, the high-level sky image sequence features are first fed into the 3D Swin Transformer Block, and the output features are then fed into the video block dilation unit to perform the reverse operation of the block merging layer, that is, to decompose and expand the high-level feature blocks in the spatiotemporal dimension (e.g., split a feature block into four smaller, spatiotemporally adjacent feature blocks), gradually restoring the resolution of the feature map. After four stages, the output features are passed through a multilayer perceptron network to obtain the sky video image sequence corresponding to the predicted power generation time point. The VST-Gen pre-trained model computes attention within a local spatiotemporal cube (such as a small region within several consecutive frames), directly capturing the local spatiotemporal patterns of cloud movement and deformation within that small region and over a short period of time. It then ensures information interaction between different spatiotemporal windows, thereby modeling the motion propagation of clouds over a larger area (such as the overall movement direction, speed, and diffusion trend of cloud clusters).

[0080] Similarly, the VST-Gen pre-trained model is trained. Each set of sky image sequences from the training set is input into the model, and the output is the sky image sequence corresponding to the predicted power generation time point. This image sequence and the sky image sequence corresponding to the actual predicted power generation time point are input into the loss function, which is trained as follows:

[0081]

[0082] In the formula N P This represents the total number of predicted sky image pixels in a batch. For the j-th sky image pixel prediction value, The values ​​represent the corresponding real-world sky image pixels. Based on the loss function calculation results, the model parameters are updated using the backpropagation algorithm. This training process enhances the ability of its feature encoding layer to extract motion features from historical sky images. This enables subsequent multi-level attention modules to obtain strong signals about future sky conditions during multimodal fusion, thereby significantly improving power prediction, especially the ability to predict sharp rises and falls in power caused by cloud movement.

[0083] Step S108: Multimodal fusion is performed on the ST-GEN pre-trained model, the VST-GEN pre-trained model, and the encoding model corresponding to the real-time power data to construct a multimodal model. Based on the multimodal model, the predicted value of photovoltaic power generation corresponding to the sky image data is output.

[0084] Specifically, the ST-Gen and VST-Gen pre-trained models trained in the above steps are fused with the encoding module corresponding to historical power generation to construct a multi-modal cross-attention module. In the construction step of the multi-modal cross-attention module, a Transformer-based multi-modal cross-attention module is constructed to fuse information from different modalities. This module takes as input the state features of the intermediate sky image output by the ST-Gen test model, the motion features of the intermediate historical sky image output by the VST-Gen test model, and the historical power generation features output by the historical power generation encoding model, and outputs the predicted ultra-short-term power generation result. Unlike general cross-attention structures that can only achieve the fusion of two modalities, the multi-modal cross-attention structure can achieve the fusion of three modalities. The steps are as follows, which can be divided into two steps:

[0085] First, it enables full information exchange between the "current cloud state" (from ST-Gen) and the "historical cloud movement trend" (from VST-Gen).

[0086] Assumption Features of the sky image at the current moment, Features of historical sky image sequences Based on historical photovoltaic power generation characteristics, the calculation formula for the multiple cross-attention structure is as follows:

[0087]

[0088]

[0089] The main purpose of the above formula is to calculate the corresponding query, key, and value features based on the features of the input sky image and the features of the historical sky image sequence.

[0090] Using the image features as a query, find the relevant motion context in the video features, including:

[0091] ,

[0092] in, To combine image features with motion context, The query vector represents the image features. The key vector for video features. Let K be the dimension of the key vector. This is a value vector of video features. Image features;

[0093] Using the video features as a query, find static reference points in the image features, including:

[0094] ,

[0095] in, To combine video features at static reference points, The query vector represents the video features. The key vector of image features, A vector of image feature values. For video features;

[0096] The process of finding the cross-correlation relationship between image features and video features, and combining it with its own residuals, uses MLP fusion to generate enhanced visual features, including:

[0097] ,

[0098] in, To enhance visual features;

[0099] In the above formula, the sky image features at the current moment are used. For the query, look for features in historical sky image sequences (video features). Searching for related motion contexts and features of historical sky image sequences For the query, the sky image features at the current time. Find key static reference points in Then, the two results are concatenated with their own residuals and spliced ​​together. After MLP fusion, a unified enhanced visual feature XO that contains both instantaneous state and motion trend is generated.

[0100] Secondly, the fused high-level visual features XO are correlated with the historical power features XG that characterize the operation patterns of the power plant to establish a mapping model between "sky image" and "power generation".

[0101]

[0102]

[0103] The main purpose of the above formula is to calculate the corresponding query, key, and value features based on the fusion features of the input sky image features and historical sky image sequence features, as well as the historical power generation features.

[0104] Using the enhanced visual features as a query, the corresponding power generation mode is searched within the power features, including:

[0105] ,

[0106] in, For power generation mode, To enhance the query vector of visual features, The key vector represents the power characteristics. This is a value vector representing the power characteristics;

[0107] Using the power feature as a query, the enhanced visual features are used to find visual evidence that causes power fluctuations, including:

[0108] ,

[0109] in, As visual evidence, This is the query vector for power features. To enhance the key vector of visual features, To enhance the value vector of visual features, Power characteristics;

[0110] The process of finding the cross-correlation relationship between enhanced visual features and power features, and combining it with its own residuals, generates multimodal fusion features through MLP fusion, including:

[0111] ,

[0112] in, This is a multimodal fusion feature.

[0113] In the above formulas, visual features are enhanced. For Query, search for the corresponding power generation mode in the historical power characteristics XG. and based on historical power characteristics For queries, enhance visual features Searching for visual evidence that causes power fluctuations Then, the two results are concatenated with their own residuals and spliced ​​together. After MLP fusion, the final output is a three-modal deep fusion feature. .

[0114] In the formula For querying the matrix, The key matrix, For value matrices, The output features are a three-modal fused feature, and the MLP is a multilayer perceptron network. The fused features are then passed through the output layer of the multilayer perceptron to obtain the final photovoltaic power generation prediction value.

[0115] This invention provides a multimodal ultra-short-term photovoltaic (PV) power prediction method. It acquires historical sky image data and simultaneously collects real-time power data. The historical sky image data is preprocessed to obtain feature images and feature videos. The feature images are input into an ST-GEN pre-trained model for training, including image occlusion, feature encoding, and image restoration steps. Iterative training is performed using a loss function between the restored image and the original image to determine the trained ST-GEN pre-trained model. The feature videos are input into a VST-GEN pre-trained model for training, including feature encoding and feature dilation steps. Iterative training is performed using a loss function between the predicted video sequence after feature dilation and the actual video sequence at the target time to determine the trained VST-GEN pre-trained model. The ST-GEN pre-trained model, the VST-GEN pre-trained model, and the encoding model corresponding to the real-time power data are fused in a multimodal manner to construct a multimodal model. Based on the multimodal model, the predicted PV power corresponding to the sky image data is output. This method utilizes a multi-modal cross-attention structure to fuse features from different modalities, resulting in a more accurate PV power prediction. Furthermore, in the multimodal models, the ST-Gen model can fuse the feature encoding layer and the pixel restoration layer. By training to restore the original sky image, the feature encoding layer of the ST-Gen model can better capture the state information of the sky image. The VST-Gen model can fuse the feature encoding layer and the feature dilation layer. By training to predict future sky images, the feature encoding layer of the VST-Gen model can better capture the motion information of clouds in the sky image, further improving the accuracy of the prediction results.

[0116] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a multimodal ultra-short-term photovoltaic power generation prediction system provided in an embodiment of this application. Figure 2 As shown, the system includes:

[0117] The acquisition module S202 is used to acquire historical sky image data and simultaneously collect real-time power data, and preprocess the historical sky image data to obtain feature images and feature videos.

[0118] The first training module S204 is used to input the feature image into the ST-GEN pre-training model for training, including image occlusion, feature encoding and image restoration steps, and to perform iterative training using the loss function between the restored image and the original image to determine the trained ST-GEN pre-training model.

[0119] The second training module S206 is used to input the feature video into the VST-GEN pre-training model for training, including feature encoding and feature dilation steps, and to perform iterative training using the loss function between the feature-dilated predicted video sequence and the actual video sequence at the target time to determine the trained VST-GEN pre-training model.

[0120] The multimodal training module S208 is used to perform multimodal fusion of the ST-GEN pre-trained model, the VST-GEN pre-trained model, and the encoding model corresponding to the real-time power data to construct a multimodal model. Based on the multimodal model, it outputs the predicted value of photovoltaic power generation corresponding to the sky image data.

[0121] In another embodiment, a multimodal ultra-short-term photovoltaic power generation prediction system further includes:

[0122] The image segmentation module is used to cut and divide the feature image in its width and height dimensions to generate corresponding image block feature sequences. Each image block is transformed into a feature vector through linear projection during its model training.

[0123] The video segmentation module is used to cut and divide the feature video in terms of its width, height, and time, and generate corresponding spatiotemporal video block feature sequences. Each video block is mapped to a feature vector during its model training.

[0124] Those skilled in the art will clearly understand that the technical solutions of the embodiments of this application can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0125] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0126] See Figure 3 It shows a schematic diagram of the structure of an electronic device according to an embodiment of this application, which can be used to implement... Figure 1 The method in the illustrated embodiment. (As shown) Figure 3 As shown, the electronic device 300 may include: at least one processor 301, at least one network interface 304, user interface 303, memory 305, and at least one communication bus 302.

[0127] The communication bus 302 is used to enable communication between these components.

[0128] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0129] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0130] The processor 301 may include one or more processing cores. The processor 301 connects to various parts within the electronic device 300 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 305, and by calling data stored in the memory 305. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 301 may integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 301 and may be implemented as a separate chip.

[0131] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory 305 may include a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. Figure 3 As shown, the memory 305, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and program instructions.

[0132] exist Figure 3 In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and acquire user input data; while the processor 301 can be used to call the image-based interactive application stored in the memory 305 and specifically perform the following operations: acquire historical sky image data and synchronously collect real-time power data; preprocess the historical sky image data to obtain feature images and feature videos; input the feature images into the ST-GEN pre-trained model for training, including image occlusion, feature encoding, and image restoration steps, and iteratively train using the loss function between the restored image and the original image to determine the trained ST-GEN pre-trained model; input the feature videos into the VST-GEN pre-trained model for training, including feature encoding and feature dilation steps, and iteratively train using the loss function between the predicted video sequence after feature dilation and the actual video sequence at the target time to determine the trained VST-GEN pre-trained model; perform multimodal fusion of the ST-GEN pre-trained model, the VST-GEN pre-trained model, and the encoding model corresponding to the real-time power data to construct a multimodal model; and output the predicted value of photovoltaic power generation corresponding to the sky image data based on the multimodal model.

[0133] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0134] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0135] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0140] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0141] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

Claims

1. A multimodal ultra-short-term photovoltaic power generation prediction method, the method comprising: Historical sky image data is acquired, and real-time power data is collected simultaneously. The historical sky image data is preprocessed to obtain feature images and feature videos. The feature image is input into the ST-GEN pre-trained model for training, including image occlusion, feature encoding and image restoration steps, and iterative training is performed using the loss function between the restored image and the original image to determine the trained ST-GEN pre-trained model. The feature video is input into the VST-GEN pre-trained model for training, including feature encoding and feature dilation steps. The VST-GEN pre-trained model is then iteratively trained using the loss function between the feature-dilated predicted video sequence and the actual video sequence at the target time to determine the trained VST-GEN pre-trained model. The ST-GEN pre-trained model, VST-GEN pre-trained model, and the encoding model corresponding to the real-time power data are fused in a multimodal manner to construct a multimodal model. Based on the multimodal model, the predicted value of photovoltaic power generation corresponding to the sky image data is output.

2. The method according to claim 1, characterized in that, The method further includes: The feature image is segmented along its width and height dimensions to generate a corresponding image block feature sequence. Each image block is transformed into a feature vector through linear projection during model training. The feature video is segmented in terms of its width, height, and time to generate a corresponding spatiotemporal video block feature sequence. Each video block is mapped to a feature vector during model training.

3. The method according to claim 2, characterized in that, The step of inputting the feature image into the ST-GEN pre-trained model for training includes: Set the features of some image patches in the image patch feature sequence to zero; The image patch feature sequence is processed by a multi-stage feature encoder to calculate window self-attention, and the window boundaries are shifted to calculate the self-attention between windows. Then, adjacent image patches are merged until the ST-GEN intermediate features are output in the final stage. The ST-GEN intermediate features and the image blocks with zero features are input into the image block restoration layer to generate the restored image corresponding to the image block features with zero features.

4. The method according to claim 2, characterized in that, The step of inputting the feature video into the VST-GEN pre-trained model for training includes: The video block feature sequence is processed by a multi-stage feature encoder to calculate window self-attention, and the window boundaries are shifted to calculate the self-attention between windows. Then, adjacent video blocks are merged until the VST-GEN intermediate features are output in the final stage. The intermediate features of VST-GEN are input into the feature dilation layer for reverse block merging, and the predicted video at the corresponding time point is output.

5. The method according to claim 2, characterized in that, The step of fusing the ST-GEN pre-trained model, the VST-GEN pre-trained model, and the encoding model corresponding to the real-time power data into a multimodal model includes: The ST-GEN pre-trained model and the VST-GEN pre-trained model interact to find the cross-correlation relationship between image features and video features, and combine their own residuals to generate enhanced visual features through MLP fusion. The enhanced visual features are interacted with real-time power data to find the cross-correlation relationship between the enhanced visual features and power features, and combined with their own residuals, multimodal fusion features are generated through MLP fusion.

6. The method according to claim 5, characterized in that, The method further includes: Using the image features as a query, find the relevant motion context in the video features, including: ， in, To combine image features with motion context, The query vector represents the image features. The key vector for video features. Let K be the dimension of the key vector. This is a value vector of video features. Image features; Using the video features as a query, find static reference points in the image features, including: ， in, To combine video features at static reference points, The query vector represents the video features. The key vector of image features. A vector of image feature values. For video features; The process of finding the cross-correlation relationship between image features and video features, and combining it with its own residuals, uses MLP fusion to generate enhanced visual features, including: ， in, To enhance visual features; Using the enhanced visual features as a query, the corresponding power generation mode is searched within the power features, including: ， in, For power generation mode, To enhance the query vector of visual features, The key vector represents the power characteristics. This is a value vector representing the power characteristics; Using the power feature as a query, the enhanced visual features are used to find visual evidence that causes power fluctuations, including: ， in, As visual evidence, This is the query vector for power features. To enhance the key vector of visual features, To enhance the value vector of visual features, Power characteristics; The process of finding the cross-correlation relationship between enhanced visual features and power features, and combining it with its own residuals, generates multimodal fusion features through MLP fusion, including: ， in, This is a multimodal fusion feature.

7. A multimodal ultra-short-term photovoltaic power generation prediction system, characterized in that, The system includes; The acquisition module is used to acquire historical sky image data and simultaneously collect real-time power data, and preprocess the historical sky image data to obtain feature images and feature videos. The first training module is used to input the feature image into the ST-GEN pre-training model for training, including image occlusion, feature encoding and image restoration steps, and iteratively trains the model using the loss function between the restored image and the original image to determine the trained ST-GEN pre-training model. The second training module is used to input the feature video into the VST-GEN pre-trained model for training, including feature encoding and feature dilation steps, and to perform iterative training using the loss function between the feature-dilated predicted video sequence and the actual video sequence at the target time to determine the trained VST-GEN pre-trained model. The multimodal training module is used to perform multimodal fusion of the ST-GEN pre-trained model, the VST-GEN pre-trained model, and the encoding model corresponding to the real-time power data to construct a multimodal model. Based on the multimodal model, the module outputs the predicted value of photovoltaic power generation corresponding to the sky image data.

8. The system according to claim 7, characterized in that, The system also includes: The image segmentation module is used to cut and divide the feature image in its width and height dimensions to generate corresponding image block feature sequences. Each image block is transformed into a feature vector through linear projection during its model training. The video segmentation module is used to cut and divide the feature video in terms of its width, height, and time, and generate corresponding spatiotemporal video block feature sequences. Each video block is mapped to a feature vector during its model training.

9. An electronic device, comprising a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-6.

Images