Hard and soft prompt fused wind power prediction method based on large language model

Abstract

The invention relates to the technical field of new energy and intelligent power systems, in particular to a big language model-based hard-soft prompt fusion wind power prediction method, which comprises the following steps of: modeling a prediction task, determining a data dimension, dividing a non-overlapping training / test set, and preprocessing original data at the same time; numerical wind speed, power and missing information are converted into a structured natural language (hard prompt) and coded into a token sequence to be input into a model; then, a soft prompt matrix is generated through lower projection, a multi-layer residual self-attention block and upper projection, and after mapping, the soft prompt matrix is fused with keys / values of all layers of Transform; training only optimizes soft prompt parameters, and reasoning is combined with hard and soft prompt to output future power prediction. According to the method, the problems of data missing, poor generalization and resource consumption in the prior art are solved, the prediction robustness and generalization ability are improved, the cost is reduced, and intelligent operation and maintenance of the wind power plant are facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of new energy and smart power systems, in particular to a hard-soft prompt fusion wind power prediction method based on a large language model. BACKGROUND

[0002] With the transformation of global energy structure towards low carbonization and renewable, wind power, as an important part of clean energy, has played a crucial role in the power system. Wind power output is influenced by multiple factors such as meteorological environment, topography and unit operating state, and has significant randomness and volatility. Accurate wind power prediction is not only the core of ensuring the safe and stable operation of the power system and improving the new energy consumption capacity, but also the basis for realizing economic dispatching and intelligent operation and maintenance of wind farms. Existing wind power prediction methods mainly include physical modeling methods, statistical modeling methods and data-driven methods based on deep learning. Physical models describe the wind energy conversion process through numerical weather prediction and fluid mechanics equations, although they have a certain theoretical basis, but the model parameters are numerous, the calculation complexity is high, and the description of the microenvironment of the wind farm is limited. Statistical models (such as ARMA, ARIMA, etc.) rely on historical data to mine time series patterns, which are suitable for short-term linear characteristics, but it is difficult to capture nonlinear and high-order dependence relationships in wind power data. In recent years, deep learning methods (such as LSTM, GRU, Transformer, etc.) have been widely used in wind power prediction due to their strong nonlinear modeling ability and end-to-end learning advantages. However, existing deep learning methods generally rely on complete and continuous historical observation data, and in the face of large-scale data missing caused by sensor failures, communication abnormalities, and other factors in actual wind farms, as well as zero-sample prediction scenarios with significant differences between new wind farm distribution characteristics and historical samples, the model prediction accuracy and generalization ability are significantly reduced, making it difficult to meet the requirements of high robustness and strong adaptability in engineering practice. Therefore, there is an urgent need for a new wind power prediction method and system that can maintain high accuracy, high robustness and strong generalization ability in complex scenarios such as data missing, wind power fluctuation and dynamic changes in wind farm distribution. SUMMARY

[0003] In view of the above, the present application proposes a hard-soft prompt fusion wind power prediction method based on a large language model to solve at least one of the problems in the above background technology.

[0004] The present application provides a hard-soft prompt fusion wind power prediction method based on a large language model, comprising the following steps: Mathematically model the wind power prediction task, specify the number of wind farms, feature dimensions, time series data length, and divide the training set and test set to ensure no overlap between them; The numerical historical wind speed and wind power data sequences and missing information are transformed into structured natural language instructions through rule templates. The instructions are then segmented and encoded into token sequences and input into the large language model. The token sequence is denoted as z=concat(P,x,y), where P is a learnable prefix token, x is a text token sequence transformed from historical observations and missing information, and y is the prediction target token used for supervision during training. The embedding representation corresponding to the prefix token is compressed to a low-dimensional space by the down projection matrix, and then input into a multi-layer residual self-attention block for processing. The output of each layer is activated by ReLU and added to the input residual. The output is aggregated by average pooling and finally restored to the original embedding dimension by the up projection matrix and GeLU activation to obtain the soft cue matrix. The soft cue matrix is ​​mapped to the soft cue vectors of each layer and head of the large language model Transformer structure through a linear transformation. These vectors are then concatenated with the key and value of the main model's self-attention to obtain the enhanced key and value. Based on the enhanced key and value, the self-attention calculation process is rewritten, and the model generates the output token in an autoregressive manner to obtain the wind power prediction distribution. During the training phase, the cross-entropy loss function is used to supervise the prediction of tokens, and only the soft cue adapter and related parameters are optimized, while the main model parameters are frozen. During the inference phase, the input is hard cue text containing historical observations and missing data, and the frozen large language model and the trained soft cue adapter are used to output the future L-step wind power prediction results.

[0005] In some embodiments, the optimization objective formula for the mathematical modeling is: ; in For parameters to be optimized, To train the number of wind farms, For the first Length of time series data for each wind field For loss function, For the prediction model to be trained, This indicates input from the history window. For the prediction target; the test wind field prediction formula is: ,in , To test the wind collection field.

[0006] In some embodiments, the formula for the i-th head self-attention output of the multi-layer residual self-attention block is: ,in, These are the learnable query, key, and value weights, respectively. output weights shared by all layers, embedding representation in low-dimensional space; The multi-layer residual self-attention block aggregation output formula is: where N is the number of heads, AvgPool is the average pooling operation; the soft prompt matrix formula is: where is the upper projection matrix, ≪ , is the original embedding dimension.

[0007] In some embodiments, the enhanced key and value formulas are respectively: , , where , is the soft prompt vector of the lth layer and the hth head, , is the key and value of the lth layer and the hth head of the main model, and concat is the concatenation operation.

[0008] In some embodiments, the self-attention calculation process formula is: where is the self-attention output of the lth layer and the hth head, is the query of the lth layer and the hth head of the main model. The prediction distribution formula is: where represents the transformer calculation after n layers of fusion of hard and soft prompts, is the initial hidden layer state, is the output mapping matrix, is the predicted output token, is the input token sequence, is the main model parameter.

[0009] In some embodiments, the cross-entropy loss function formula is: where is the predicted output token of the ith sample, is the input token sequence of the ith sample.

[0010] In some embodiments, the mathematical modeling of the wind power prediction task, the number of wind farms, the feature dimension, the length of the time series data, the division of the training set and the test set, and the ensuring that there is no overlap between the two include: The preprocessing operations of normalization, missing value marking, and outlier removal are performed on the original data.

[0011] Compared with the prior art, the beneficial effects of the present application are that the numerical time series data such as wind speed, power and missing conditions are converted into structured natural language instructions by the hard prompt encoder, effectively solving the problem of mismatch between the original numerical sequence and the input structure of the large language model, and significantly improving the robustness of the model in predicting missing data and extreme fluctuations.

[0012] A soft prompt adapter module is designed, which adopts a bottleneck structure and a residual self-attention mechanism to realize efficient fine-tuning of the large language model. By freezing the main model parameters and fine-tuning only a small number of soft prompt parameters, the resource consumption in the model training and deployment process is significantly reduced, while the zero-shot generalization ability across wind farms is improved. In the inference stage, combined with the hybrid mechanism of hard prompt and soft prompt, the context understanding ability of the large language model and the task adaptation ability of the soft prompt are fully utilized to realize high-precision wind power prediction in complex scenarios such as multiple wind farms, multiple missing data and multiple fluctuations, effectively improving the intelligent operation and maintenance level of the wind farm and the engineering application value.

[0013] The foregoing general description and the following detailed description are merely exemplary and explanatory, rather than limiting the present disclosure.

[0014] Other features and aspects of the present disclosure will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0016] Figure 1 A flowchart of a hard-soft prompt fusion wind power prediction method based on a large language model is provided for the embodiments of the present application. Figure 2 A soft-hard mixed prompt mechanism diagram of a hard-soft prompt fusion wind power prediction method based on a large language model is provided for the embodiments of the present application. Figure 3 A soft prompt adapter mechanism diagram based on a multi-layer residual self-attention module of a hard-soft prompt fusion wind power prediction method based on a large language model is provided for the embodiments of the present application. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] See Figures 1-3 As shown, a wind power prediction method based on a large language model and fusion of hardware and software prompts according to an embodiment of this application includes the following steps: Mathematical modeling is performed for the wind power prediction task, which clarifies the number of wind farms, feature dimensions, and time series data length, and divides the training set and test set to ensure that the two sets do not overlap. The numerical historical wind speed and wind power data sequences and missing information are transformed into structured natural language instructions through rule templates. The instructions are then segmented and encoded into token sequences and input into the large language model. The token sequence is denoted as z=concat(P,x,y), where P is a learnable prefix token, x is a text token sequence transformed from historical observations and missing information, and y is the prediction target token used for supervision during training. The embedding representation corresponding to the prefix token is compressed to a low-dimensional space by the down projection matrix, and then input into a multi-layer residual self-attention block for processing. The output of each layer is activated by ReLU and added to the input residual. The output is aggregated by average pooling and finally restored to the original embedding dimension by the up projection matrix and GeLU activation to obtain the soft cue matrix. The soft cue matrix is ​​mapped to the soft cue vectors of each layer and head of the large language model Transformer structure through a linear transformation. These vectors are then concatenated with the key and value of the main model's self-attention to obtain the enhanced key and value. Based on the enhanced key and value, the self-attention calculation process is rewritten, and the model generates the output token in an autoregressive manner to obtain the wind power prediction distribution. During the training phase, the cross-entropy loss function is used to supervise the prediction of tokens, and only the soft cue adapter and related parameters are optimized, while the main model parameters are frozen. During the inference phase, the input is hard cue text containing historical observations and missing data, and the frozen large language model and the trained soft cue adapter are used to output the future L-step wind power prediction results.

[0019] In some specific embodiments, the optimization objective formula for the mathematical modeling is: ; in For parameters to be optimized, To train the number of wind farms, For the first Length of time series data for each wind field For loss function, For the prediction model to be trained, This indicates input from the history window. For the prediction target; the test wind field prediction formula is: ,in , To test the wind collection field.

[0020] In some specific embodiments, the formula for the i-th head self-attention output of the multi-layer residual self-attention block is: ,in, These are the learnable query, key, and value weights, respectively. The output weights are shared by all layers. Embedded representation in low-dimensional space; The formula for the aggregated output of multi-layer residual self-attention blocks is: Where N is the number of heads, and AvgPool is the average pooling operation; the formula for the soft suggestion matrix is: in Let be the upward projection matrix. ≪ , This represents the original embedding dimension.

[0021] In some specific embodiments, the enhanced key and value formulas are as follows: , , in , These are the soft cue vectors for the h-th head in the l-th layer, respectively. , These are the key and value of the l-th layer and h-th head of the main model, respectively, and concat is the concatenation operation.

[0022] In some specific embodiments, the formula for the self-attention calculation process is as follows: in This is the self-attention output of the h-th head in the l-th layer. The query at the h-th head of the l-th layer of the main model; The formula for predicting the distribution is: in This indicates that the transformation calculation has been performed after n layers of hardware and software feedback integration. This is the initial hidden layer state. To output the mapping matrix, To predict the output token, Given the input token sequence, These are the parameters of the main model.

[0023] In some specific embodiments, the cross-entropy loss function is formulated as follows: in Let the predicted output token be the i-th sample. Let be the input token sequence for the i-th sample.

[0024] In some specific embodiments, the step of mathematically modeling the wind power prediction task, specifying the number of wind farms, feature dimensions, and time series data length, and dividing the training set and test set while ensuring that they do not overlap, includes: Perform preprocessing operations on the raw data, including normalization, missing value marking, and outlier removal.

[0025] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0026] It should be understood that: Step 1: Data Preparation. In this invention, a rigorous mathematical modeling of the wind power prediction task is first required. Assume there are a total of... Each wind farm is in Data of length was collected under several features (such as wind speed at different altitudes and historical wind power). Time series data. For each time step... , No. The data for each wind field can be denoted as: The corresponding wind power is The training wind field set is as follows: The test wind field collection is And the two do not overlap, that is The goal of model training is to optimize parameters using only the wind field from the training set. This enables accurate prediction of future winds even in unseen test wind fields. The specific optimization objective for the wind power output of the step is: in For loss function, For the prediction model to be trained, This indicates input from the history window. For the predicted target. Ultimately, in the test wind field The model can be applied directly without further training: This step also includes normalizing the original data and marking missing values ​​(e.g., using "..."). <unk>" indicates deletion), outlier rejection, etc. Preprocessing operations ensure that subsequent inputs are uniform and have good numerical stability.

[0027] Step two: Hard prompt encoding (HardPromptEncoder) In the hard prompt encoding stage, the present application converts the numerical type of historical wind speed and wind power data sequence and the missing condition into structured natural language instructions through a rule template. Specifically, for each time step , first read the observation value of each feature in turn, if the feature is missing at this time, use the placeholder <unk>Replace it; otherwise, retain its value. Then, generate a natural language description using the following template: " For wind farm 1, the four features related to power generation in the historical operation data are: (11.2, <unk> ,< / unk> 0.1701,9.623),(19.2, <unk> ,15.3,-0.0971),…,(0.14, <unk>,0.1041,0.592). Then what should be the power output of the wind farm for the next 16 time steps? The corresponding powers are: 120, 130, 128, 135 kW. Please predict the power for the next 1 hour.< / unk> < / unk> Figure 3 θ The above text was segmented and encoded into a token sequence, denoted as ,in For learnable prefix tokens, A text token sequence transformed from historical observations and missing information. The target token is predicted (used for supervision during training). The entire token sequence. The data is input into a Large Language Model (LLM) as the conditional context for wind power prediction. This step formally transforms the original numerical prediction task into a natural language generation task, enabling the LLM to leverage its powerful semantic modeling and contextual understanding capabilities.

[0028] Step 3: SoftPromptAdapter.

[0029] This invention, in addition to the main structure of the large language model, designs a soft prompt adapter to enhance the model's adaptability to the spatiotemporal characteristics of wind power, specifically as follows: ϕ As shown. First, let the embedding representation corresponding to a prefix token of length P be... Then through the lower projection matrix Compress it into a low-dimensional space ,in .then, Input multiple layers of residual self-attention blocks sequentially, each layer's first... The self-attention output of the size is: in, These are the learnable query, key, and value weights, respectively. The output weights are shared across all layers. Each layer's output is processed... After activation, the result is added to the input residual, and finally, the outputs of all layers are aggregated using average pooling to obtain: Then, through the upper projection matrix GeLU activation restores the low-dimensional aggregated features to their original embedding dimensions, ultimately yielding the soft cue matrix: This soft cue matrix contains all trainable parameters and can efficiently adapt to prediction tasks under different wind fields and missing scenarios, provided that the large language model is frozen.

[0030] Step four: Soft prompt hierarchical mechanism In the hard-soft prompt hybrid prediction mechanism, the soft prompt matrix obtained in step three is first mapped to the soft prompt vector of each layer and each head through linear transformation For each layer l and each head h of the large language model Transformer structure, the soft prompt vector is respectively concatenated with the key and value of the main model self-attention to obtain the enhanced key and value: Step five: Hard-soft prompt hybrid prediction mechanism On the basis of step four, the self-attention calculation process of each head is rewritten as In this way, the soft prompt vector directly affects the context modeling of self-attention in each layer and each head, effectively introducing prior knowledge for wind power prediction tasks and missing scenarios. Finally, the model generates output tokens in an autoregressive manner, and the prediction distribution is: where represents the transformer calculation after n layers of fusion of hard and soft prompts, is the output mapping matrix.

[0031] Step six: Model training and reasoning In the training stage, the prediction tokens of all training samples are supervised by using the cross-entropy loss function, and the loss function is: where only the soft prompt adapter and related parameters Figure 1 are optimized, and the main model parameters Figure 1 are kept frozen. During the training process, the soft prompt and adapter parameters are continuously adjusted through backpropagation, so that the model can maintain high prediction accuracy under different wind farms and different missing conditions. In the reasoning stage, in the test wind farm or actual application scenario, only the hard prompt text (containing historical observations and missing marks) of the same format needs to be input, and the frozen large language model and the trained soft prompt adapter can directly output the future L-step wind power prediction results. Since the soft prompt adapter has the generalization ability across wind farms and across missing scenarios, this method can still achieve high robustness and high accuracy of wind power prediction in complex environments such as data missing and extreme fluctuations.

[0032] ​​​​The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0033] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0034] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. ​ one or more flowcharts and / or blocks ​ one or more flowcharts and / or blocks

[0035] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing the technical solutions of the present application, but not for limiting it. Although the present application is described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalent replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.< / unk> < / unk>

Claims

1. A wind power prediction method based on a large language model that integrates hardware and software prompts, characterized in that, Includes the following steps: Mathematical modeling is performed for the wind power prediction task, which clarifies the number of wind farms, feature dimensions, and time series data length, and divides the training set and test set to ensure that the two sets do not overlap. The numerical historical wind speed and wind power data sequences and missing information are transformed into structured natural language instructions through rule templates. The instructions are then segmented and encoded into token sequences and input into the large language model. The token sequence is denoted as z=concat(P,x,y), where P is a learnable prefix token, x is a text token sequence transformed from historical observations and missing information, and y is the prediction target token used for supervision during training. The embedding representation corresponding to the prefix token is compressed to a low-dimensional space by the down projection matrix, and then input into a multi-layer residual self-attention block for processing. The output of each layer is activated by ReLU and added to the input residual. The output is aggregated by average pooling and finally restored to the original embedding dimension by the up projection matrix and GeLU activation to obtain the soft cue matrix. The soft cue matrix is ​​mapped to the soft cue vectors of each layer and head of the large language model Transformer structure through a linear transformation. These vectors are then concatenated with the key and value of the main model's self-attention to obtain the enhanced key and value. Based on the enhanced key and value, the self-attention calculation process is rewritten, and the model generates the output token in an autoregressive manner to obtain the wind power prediction distribution. During the training phase, the cross-entropy loss function is used to supervise the prediction of tokens, and only the soft cue adapter and related parameters are optimized, while the main model parameters are frozen. During the inference phase, the input is hard cue text containing historical observations and missing data, and the frozen large language model and the trained soft cue adapter are used to output the future L-step wind power prediction results.

2. The wind power prediction method based on a large language model with fusion of hardware and software prompts according to claim 1, characterized in that, The optimization objective formula for the mathematical modeling is: ； in For parameters to be optimized, To train the number of wind farms, For the first Length of wind field time series data For loss function, For the prediction model to be trained, This indicates input from the history window. For the prediction target; the test wind field prediction formula is: ,in , To test the wind collection field.

3. The wind power prediction method based on a large language model with fusion of hardware and software prompts according to claim 2, characterized in that, The formula for the i-th head self-attention output of the multi-layer residual self-attention block is: ,in, These are the learnable query, key, and value weights, respectively. The output weights are shared by all layers. Embedded representation in low-dimensional space; The formula for the aggregated output of multi-layer residual self-attention blocks is: Where N is the number of heads, and AvgPool is the average pooling operation; the formula for the soft suggestion matrix is: in Let be the upward projection matrix. ≪ , This represents the original embedding dimension.

4. The wind power prediction method based on a large language model with fusion of hardware and software prompts according to claim 3, characterized in that, The enhanced key and value formulas are as follows: , ， in , These are the soft cue vectors for the h-th head in the l-th layer, respectively. , These are the key and value of the l-th layer and h-th head of the main model, respectively, and concat is the concatenation operation.

5. The wind power prediction method based on a large language model with fusion of hardware and software prompts according to claim 4, characterized in that, The formula for calculating self-attention is: in This is the self-attention output of the h-th head in the l-th layer. The query at the h-th head of the l-th layer of the main model; The formula for predicting the distribution is: in This indicates that the transformation calculation has been performed after n layers of hardware and software feedback integration. This is the initial hidden layer state. To output the mapping matrix, To predict the output token, Given the input token sequence, These are the parameters of the main model.

6. The wind power prediction method based on a large language model with fusion of hardware and software prompts according to claim 5, characterized in that, The formula for the cross-entropy loss function is: in The predicted output token for the i-th sample. Let be the input token sequence for the i-th sample.

7. The wind power prediction method based on a large language model with fusion of hardware and software prompts according to claim 6, characterized in that, The mathematical modeling of the wind power prediction task, which clarifies the number of wind farms, feature dimensions, and time series data length, and divides the training and test sets to ensure that they do not overlap, includes: Perform preprocessing operations on the raw data, including normalization, missing value marking, and outlier removal.

Images