Power system time sequence prediction model training method, system and device based on large language model weight fine tuning and storage medium

Abstract

The invention discloses an electric power system time sequence prediction model training method, system and device based on large language model weight fine tuning and a storage medium. The method comprises the steps of obtaining time sequence data of different fields and performing data cleaning to obtain a data set; performing weight freezing on a feedforward neural network and multi-head attention of the large language model, and performing first-stage training on the large language model by using the data set; reconstructing an input layer of the large language model to enable the reconstructed input layer to generate a corresponding natural language instruction according to the time sequence data; and freezing weight parameters of the large language model, and performing second-stage training on the large language model by using source load data of the power system to adjust parameters of the reconstructed input layer so as to obtain a time sequence prediction pre-training model. According to the method, the prediction effect can be improved while the weight of the large language model backbone network adapts to the specific task of time sequence prediction.

Description

Technical Field

[0001] The present invention relates to a power system time series prediction technology, and in particular to a power system time series prediction model training method, system, device and storage medium based on large language model weight fine-tuning. Background Art

[0002] The output of renewable energy in the power system is greatly affected by weather factors and has strong randomness and volatility. The source-load forecast has large temporal and spatial differences depending on the usage scenario. At the same time, the calculation model used for source-load forecasting faces problems such as massive input data, high-dimensional features, and strong coupling. Traditional forecasting methods require the establishment of dedicated models for specific scenarios, which is time-consuming and labor-intensive, with poor model generalization ability, high subsequent operation and maintenance costs, and complex technology. Therefore, it is urgent to establish a forecasting model with strong generalization ability, high forecasting accuracy, and can be used in a variety of forecasting scenarios.

[0003] The rapid development of large language base model technology has promoted the technical iteration and update of computer vision, natural language processing and other fields. Research results have shown that the large language base model weights with huge parameter scale have strong learning reasoning and pattern recognition capabilities in cross-domain tasks, and are better than traditional prediction solutions in extreme scenarios such as zero samples and small samples.

[0004] However, the large language model is trained and generated for natural language tasks. During the training and reasoning process, its input and output are in the form of text and characters. If the large language model needs to be applied to time series prediction tasks, the key technical difficulty lies in how to adapt the weights of the large language model backbone network to the specific tasks of time series prediction. In time series prediction, both input and output are continuous data points, not text and characters. The encoding table established by the original model (i.e., the mapping relationship from text characters to vector space) is no longer applicable, and the weights within the large language model need to be adjusted in a targeted manner to adapt to the source and load prediction tasks within the power system.

[0005] In order to adapt the weights of the backbone network of the large language model to the specific task of time series prediction, existing research has made some attempts. For example, the patent application with publication number CN117634740A discloses a power load prediction method based on a large language model. In terms of solution, the time series is converted into a structured text input model, and then the loss function is introduced to update the weight parameters of the entire model in the gradient descent to adapt to the time series prediction task. There are three shortcomings of this solution: 1. Time series are discrete data points, which are significantly different from text forms and cannot be converted losslessly, which will inevitably lead to information loss and reduce the prediction effect; 2. In terms of model structure, there is no adaptive design and improvement for time series prediction tasks, and the text input-text output structure is still used; 3. In the gradient update, the model parameters are updated through supervised fine-tuning. Instead of deeply exploring the specific scenarios of time series and text input to optimize the weights of the large language model, all model parameters are updated. The parameter update of the FFN and multi-head self-attention mechanism structure in the time series prediction fine-tuning may destroy the knowledge graph learned by the large language model in pre-training, causing the model performance to degrade. Summary of the invention

[0006] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems in the prior art, a method, system, device and storage medium for training a power system time series prediction model based on fine-tuning of the weights of a large language model are provided, so that the weights of the backbone network of the large language model can be adapted to the specific tasks of time series prediction while improving the prediction effect.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: In a first aspect, the present invention provides a method for training a power system time series prediction model based on fine-tuning of a large language model weight, comprising the following steps: Obtain time series data from different fields and clean the data to get the data set; Freeze the weights of the feedforward neural network and multi-head attention in the decoder and encoder of each layer of the large language model, and use the dataset to perform the first phase of training on the large language model to adjust the weight parameters of the position encoder, layer normalization network, and input embedding layer of the large language model; Reconstructing the input layer of the large language model so that the reconstructed input layer generates corresponding natural language instructions according to the time series data and inputs the natural language instructions into the large language model; Freeze the weight parameters of the large language model, and use the power system source and load data to perform the second stage training on the large language model to adjust the parameters of the reconstructed input layer to obtain a time series prediction pre-training model. ; Furthermore, after obtaining the time series prediction pre-training model, it also includes: inputting the power system time series data and task instructions into the trained time series prediction pre-training model to obtain corresponding prediction results.

[0008] Furthermore, each decoder and encoder of the large language model includes at least one basic network unit, and the calculation process of the basic network unit on the time series data includes: The input time series is reversibly normalized and divided into multiple segments. Each segment is input into the input embedding layer to obtain the corresponding embedding vector. The embedding vector is superimposed with the encoding information output by the position encoder to obtain the encoded vector. Input the encoded vector into the multi-head attention mechanism to calculate the corresponding attention output result; The attention output result is input into the normalization layer to obtain the normalized result, and then the normalized result is input into the feedforward neural network. Finally, the output result of the feedforward neural network is processed again by the normalization layer to obtain the final output result of the current basic network unit.

[0009] Furthermore, when the reconstructed input layer generates corresponding natural language instructions according to the time series data, it includes: After the time series data is input into the embedding layer after being fragmented and normalized, Extract time series feature information to form a vector representation of the time series; Pass the preset natural language input containing the command prompt words through the text input embedding layer Then a vector representation of natural language is formed; The vector representation of the time series and the vector representation of the natural language are concatenated to obtain the natural language instructions and use them as the input of the large language model.

[0010] Furthermore, the preset natural language input containing the command prompt word is passed through the text input embedding layer When the vector representation of natural language is formed, it specifically includes: The text that will be input in the preset natural language According to the character table Perform word segmentation to form corresponding results ,in Indicates word segmentation tokens, is the number of tokens, which are then fed into the embedding layer via text input The linear mapping network parameters map each word segmentation token to a high-dimensional space to form a text embedding vector And as a vector representation of natural language , It is the backbone dimension of the large language model.

[0011] Furthermore, when the vector representation of the input sequence and the vector representation of the natural language are concatenated, the vector representation of the time series and the vector representation of the natural language are concatenated along the backbone dimension of the large language model.

[0012] Furthermore, the weights of the feedforward neural network and the multi-head attention in the decoders and encoders of each layer of the large language model are frozen, and the large language model is trained in the first stage using the data set, and the weight parameters of the large language model are frozen, and the large language model is trained in the second stage using the power system source and load data, all of which include: Add unfrozen model parameters to the parameter set , add the frozen model parameters to the parameter set ; Taking the historical time series data in the data set or power system source and load data as input, the prediction results are output according to the large language model. and the corresponding true value Calculate the loss function for the current prediction ; Calculating the loss function About parameter collection Gradient , and use this to update the first-order moment estimate and the second moment estimate ; According to the first-order moment estimate and the second moment estimate The calculated results of the calculation are used to perform bias correction to obtain the corrected first-order moment estimate and the corrected second moment estimate ; Use the corrected first moment estimate and the corrected second moment estimate Update parameter set The model parameters in .

[0013] Furthermore, the weights of the feedforward neural network and multi-head attention in the decoder and encoder of each layer of the large language model are frozen, and when the large language model is trained in the first stage using the dataset, the unfrozen model parameters are added to the parameter set. Specifically, the position encoding parameters, embedding layer parameters and normalization layer parameters are added to the parameter set , add the frozen model parameters to the parameter set Specifically, the feedforward neural network model parameters and the multi-head attention mechanism model parameters are added to the parameter set .

[0014] Further, the weight parameters of the large language model are frozen, and when the large language model is trained in the second stage using the power system source and load data, the unfrozen model parameters are added to the parameter set. Specifically, the time series input in the reconstructed input layer is embedded in the layer The network parameters and text input embedding layer The network parameters are added to the parameter set , add the frozen model parameters to the parameter set Specifically, the position encoding parameters, embedding layer parameters, normalization layer parameters, feedforward neural network model parameters and multi-head attention mechanism model parameters are added to the parameter set .

[0015] In a second aspect, the present invention further proposes a power system time series prediction model training system based on large language model weight fine-tuning, comprising: The data set generation unit is used to obtain time series data in different fields and obtain data sets after data cleaning; The first-stage training unit is used to freeze the weights of the feedforward neural network and the multi-head attention in the decoders and encoders of each layer of the large language model, and use the data set to perform the first-stage training on the large language model to adjust the weight parameters of the position encoder, the layer normalization network, and the input embedding layer of the large language model; A model reconstruction unit, used to reconstruct the input layer of the large language model so that the reconstructed input layer generates corresponding natural language instructions according to the time series data, and inputs the natural language instructions into the large language model; The second stage training unit is used to freeze the weight parameters of the large language model and use the power system source and load data to perform the second stage training on the large language model to adjust the parameters of the reconstructed input layer to obtain a time series prediction pre-training model.

[0016] Furthermore, it also includes: The result prediction unit is used to input the power system time series data and task instructions into the trained time series prediction pre-training model to obtain the corresponding prediction results.

[0017] Furthermore, each decoder and encoder of the large language model includes at least one basic network unit, and the calculation process of the basic network unit on the time series data includes: The input time series is reversibly normalized and divided into multiple segments. Each segment is input into the input embedding layer to obtain the corresponding embedding vector. The embedding vector is superimposed with the encoding information output by the position encoder to obtain the encoded vector. Input the encoded vector into the multi-head attention mechanism to calculate the corresponding attention output result; The attention output result is input into the normalization layer to obtain the normalized result, and then the normalized result is input into the feedforward neural network. Finally, the output result of the feedforward neural network is processed again by the normalization layer to obtain the final output result of the current basic network unit.

[0018] Furthermore, when the reconstructed input layer generates corresponding natural language instructions according to the time series data, it includes: After the time series data is input into the embedding layer after being fragmented and normalized, Extract time series feature information to form a vector representation of the time series; Pass the preset natural language input containing the command prompt words through the text input embedding layer Then a vector representation of natural language is formed; The vector representation of the time series and the vector representation of the natural language are concatenated to obtain the natural language instructions and use them as the input of the large language model.

[0019] Furthermore, the preset natural language input containing the command prompt word is passed through the text input embedding layer When the vector representation of natural language is formed, it specifically includes: The text that will be input in the preset natural language According to the character table Perform word segmentation to form corresponding results ,in Indicates word segmentation tokens, is the number of tokens, which are then fed into the embedding layer via text input The linear mapping network parameters map each word segmentation token to a high-dimensional space to form a text embedding vector And as a vector representation of natural language , It is the backbone dimension of the large language model.

[0020] Furthermore, when the vector representation of the input sequence and the vector representation of the natural language are concatenated, the vector representation of the time series and the vector representation of the natural language are concatenated along the backbone dimension of the large language model.

[0021] Furthermore, the first-stage training unit freezes the weights of the feedforward neural network and the multi-head attention in the decoders and encoders of each layer of the large language model, and uses the data set to perform the first-stage training on the large language model, and the second-stage training unit freezes the weight parameters of the large language model, and uses the power system source and load data to perform the second-stage training on the large language model, both of which include: Add unfrozen model parameters to the parameter set , add the frozen model parameters to the parameter set ; Taking the historical time series data in the data set or power system source and load data as input, the prediction results are output according to the large language model. and the corresponding true value Calculate the loss function for the current prediction ; Calculating the loss function About parameter collection Gradient , and use this to update the first-order moment estimate and the second moment estimate ; According to the first-order moment estimate and the second moment estimate The calculated results of the calculation are used to perform bias correction to obtain the corrected first-order moment estimate and the corrected second moment estimate ; Use the corrected first moment estimate and the corrected second moment estimate Update parameter set The model parameters in .

[0022] Furthermore, the first-stage training unit freezes the weights of the feedforward neural network and the multi-head attention in the decoder and encoder of each layer of the large language model, and adds the unfrozen model parameters to the parameter set when the large language model is trained in the first stage using the data set. Specifically, the position encoding parameters, embedding layer parameters and normalization layer parameters are added to the parameter set , add the frozen model parameters to the parameter set Specifically, the feedforward neural network model parameters and the multi-head attention mechanism model parameters are added to the parameter set .

[0023] Furthermore, the second-stage training unit freezes the weight parameters of the large language model, and when the power system source and load data are used to perform the second-stage training on the large language model, the unfrozen model parameters are added to the parameter set. Specifically, the time series input in the reconstructed input layer is embedded in the layer The network parameters and text input embedding layer The network parameters are added to the parameter set , add the frozen model parameters to the parameter set Specifically, the position encoding parameters, embedding layer parameters, normalization layer parameters, feedforward neural network model parameters and multi-head attention mechanism model parameters are added to the parameter set .

[0024] In the third aspect, the present invention also proposes a power system time series prediction model training device based on large language model weight fine-tuning, comprising a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the power system time series prediction model training method based on large language model weight fine-tuning described in the first aspect.

[0025] In a fourth aspect, the present invention further proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the power system time series prediction model training method based on large language model weight fine-tuning as described in the first aspect.

[0026] Compared with the prior art, the advantages of the present invention are: When solving the problem of "how to adapt the weights of the backbone network of a large language model to the specific task of time series prediction", the present invention refines the adaptation process into two-stage tasks. The first-stage task focuses on the targeted weight adjustment within the large language model, retains the key knowledge graph inside the large language model, freezes the weight parameters of key structures such as the feedforward neural network and the multi-head attention mechanism, and only allows the position encoding and normalization layers to participate in parameter updates, so as to adapt to the parameter dimensions of the time series prediction input in large-scale open source time series data training; the second-stage task focuses on the reconstruction of the input and output layers of the large language model. The reconstructed input layer directly uses the time series as input, and designs a structured instruction generation scheme, which can automatically generate corresponding natural language instructions based on the time series and preset task instructions, and provide supplementary descriptions of the time series data features and task scenarios through natural language task instructions, so as to provide additional information to improve the prediction effect; then through the two-stage training, the alignment of the time series input and the text input in the vector space is realized, and the prediction effect is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0028] Figure 2 This is the parameter adjustment status of the large language model in the first stage training of the embodiment of the present invention.

[0029] Figure 3 It is the model input layer structure reconstructed by the embodiment of the present invention and the adjustment of model parameters in the two-stage training. DETAILED DESCRIPTION

[0030] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0031] Embodiment 1 This embodiment proposes a power system time series prediction model training method based on large language model weight fine-tuning, through the technical means of network parameter freezing and the technical method of two-stage pre-training model parameter adjustment, so that the large language model can adapt to the source and load prediction task in the power system scenario during the weight adjustment process. Figure 1 As shown, the method of this embodiment includes the following steps: S1) Open source time series data collection: Obtain open source time series data in different fields; S2) Data cleaning and dataset construction: Data sets are obtained after data cleaning of open source time series data in different fields; S3) First stage training: Freeze the weights of the feedforward neural network and multi-head attention in the decoder and encoder of each layer of the large language model, and use the dataset to perform the first stage training on the large language model to adjust the weight parameters of the position encoder, layer normalization network, and input embedding layer of the large language model; S4) Model input layer construction driven by natural language instructions: reconstructing the input layer of the large language model so that the reconstructed input layer generates corresponding natural language instructions according to the time series data and inputs them into the large language model; S5) Second-stage training: freezing the weight parameters of the large language model, and performing second-stage training on the large language model using power system source and load data to adjust the parameters of the reconstructed input layer to obtain a time series prediction pre-training model; S6) Model application: Input the power system time series data and task instructions into the trained time series prediction pre-training model to obtain the corresponding prediction results.

[0032] The following is a detailed description of each step.

[0033] The initial weights of the large language model are oriented to various language tasks, and the weights need to be adjusted through training to gradually adapt them to the prediction tasks of time series. Therefore, rich time series data is needed to support the weight adjustment of the large language model. In step S1 of this embodiment, obtaining open source time series data in different fields specifically involves collecting publicly available data sets from the Internet, including time series data from different fields such as transportation, finance, electricity, and weather, so that the model can give full play to its cross-domain learning and reasoning capabilities and improve the generalization performance of the model. In the process of collecting time series data, the basic information of the open source data set is counted, including: the number of time series data points, the number of variables, the time series interval, the time span, the field to which the data belongs, and other basic information.

[0034] Since there are a lot of low-quality data in the open source time series data, including missing values, distorted values ​​and erroneous values. This embodiment performs a deeper processing on the collected data through step S2. In step S2, data cleaning is specifically to randomly sample time series fragments from each data set, mark and remove missing values ​​and outliers, and improve the overall quality of the data set through data preprocessing. When constructing a data set, follow the principle of uniform sampling to ensure that each batch of data input to the model comes from time series in different fields (electricity, transportation, finance, weather, etc.) and the data ratio is close to balance, that is, evenly distributed in proportion.

[0035] In step S3 of this embodiment, the first stage of training is performed to update and adjust some weight parameters of the large language model during the training process.

[0036] In this embodiment, the large language model adopts the GPT2 model. GPT2 is an open source large language basic model based on the Transformer architecture, which is generated by training with rich natural language data. The structure of its open source model weight is as follows: Figure 2 As shown in , the model consists of multiple encoders and decoders connected in series. Each encoder and decoder includes at least one basic network unit. A basic network unit includes multi-head attention, layer normalization, and feedforward neural network layers. Figure 2 As shown in Figure 1, the calculation process of the basic network unit for time series data includes: Assume that the input time series data is , where represents the number of input variables of the time series, Represents the time step. First, the input time series is reversibly normalized so that the data mean is 0 and the standard deviation is 1. The processed data is .Will Divide into several lengths Each fragment is represented by Each fragment passes through the input embedding layer to form an embedding vector , is the backbone dimension of the GPT2 model, is the linear layer network parameter of the embedding layer. Embedding vector superimposed position encoder Output the encoded information and get the encoded vector , the expression is as follows:

[0037] Will contain location information The encoding vector After inputting the GPT2 model, the encoded vector The multi-head attention mechanism is used to calculate the corresponding attention output result. In this process, the query matrix is ​​first calculated , key-value matrix and the weight matrix , the expression is as follows: , ,

[0038] in, , and are the weights of the query matrix, key value matrix and weight matrix respectively; based on , , Three matrices are used for attention calculation, and the expressions are as follows:

[0039] in, Indicates In multi-head attention calculation, the final attention output result is Depend on The calculation results of the attention heads are concatenated and calculated, and the expression is as follows:

[0040] in, It is the linear change matrix after concatenating the outputs of multiple attention heads.

[0041] The attention output result is input into the normalization layer to obtain the normalized result. The output of the multi-head attention is normalized and residually connected to prevent the gradient from disappearing during training. The operation process is as follows:

[0042] in, Represents the normalization layer in the GPT2 model.

[0043] Then the normalized result Input the feedforward neural network, the operation process is:

[0044] As can be seen from the formula, the feedforward neural network consists of two layers of fully connected networks, where , is the weight matrix, , is the bias term.

[0045] Finally, the output result of the feedforward neural network is processed again by the normalization layer to obtain the final output result of the current basic network unit, which is expressed as follows:

[0046] The above is a data operation process in a basic network unit. The GPT2 model uses multiple identical basic network units to process input data in series to achieve deep feature extraction at different scales. After processing by the basic network units of the layer, the final output result can be expressed as , and finally the prediction result of the time series is obtained through the output projection layer , the expression is as follows:

[0047] In the formula, The output projection layer parameters of the model.

[0048] From the above content, it can be seen that the self-attention layer and feedforward neural network layer in the large language model contain most of the knowledge from the pre-trained language model, which is the basis for the model to present powerful learning reasoning and pattern recognition capabilities. In order to adapt it to the tasks in the time series prediction scenario, it is necessary to adjust the weights of the GPT2 basic model through the gradient descent algorithm during training. In this embodiment, the model parameter freezing technology is used to train and adjust some network layers of the large language model to enhance the model's adaptability to time series data. Specifically, the weights of the feedforward neural network and multi-head attention in each layer of the decoder and encoder are frozen to prevent the weight parameters from changing during the model training process and affecting the model's reasoning ability. And the time series data from different fields constructed in step S2 is used as input for model training. The specific update steps are as follows: Suppose the total parameter set of the GPT2 model is , where the set of parameters that need to be updated is , the set that is frozen and does not participate in parameter update is , and the parameter sets satisfy and When freezing the weights of the feedforward neural network and multi-head attention in the decoder and encoder of each layer of the large language model, the position encoding parameters, embedding layer parameters, and normalization layer parameters are added to the parameter set. , add the feedforward neural network model parameters and the multi-head attention mechanism model parameters to the parameter set .

[0049] During the training process, the historical time series data in the dataset is used as input, and the prediction results are output by the large language model. and the corresponding true value Calculate the loss function for the current prediction , the expression is as follows:

[0050] in, Represents all samples in a batch; According to the loss calculation results, the back propagation algorithm is used to calculate the loss function with respect to the set of parameters to be updated. Gradient , use the Adam optimization algorithm to update the parameter set according to the gradient The model parameters in include: Setting the learning rate , the decay rate of the first-order moment estimate , the decay rate of the second-order moment estimate , introduce a small constant Avoid situations where the denominator is zero.

[0051] In this embodiment, due to The parameters in the set are frozen, so only the loss function needs to be calculated About parameter collection Gradient , without calculating the gradient , calculate the gradient Then, the first-order moment estimate is updated and the second moment estimate , the expression is as follows:

[0052]

[0053] in, represents the first-order moment estimate before updating, represents the second-order moment estimate before updating; because and The initial value of is 0, and it approaches 0 in the early iterations. It needs to be estimated based on the first-order moment and the second moment estimate The calculated results of the calculation are used to perform bias correction to obtain the corrected first-order moment estimate and the corrected second moment estimate , the expression is as follows:

[0054]

[0055] in, is the decay rate of the first-order moment estimate The tth power, is the decay rate of the second-order moment estimate t to the power of Use the corrected first moment estimate and the corrected second moment estimate Update parameter set The model parameters in are expressed as follows:

[0056] in, represents the model parameters before updating, Represents the updated model parameters.

[0057] Through the above steps, step S3 adaptively adjusts the weights of the GPT2 large language model through parameter freezing and fine-tuning methods to adapt it to the scale characteristics of time series data. After the first stage of training, the weights of the large language model are reasonably fine-tuned, and after the prediction loss function converges, it can be considered that the current large language model has adapted to various downstream tasks in the time series prediction scenario.

[0058] In order to give full play to the potential of the large language model in time series prediction tasks (for example, increasing logical reasoning capabilities, introducing additional information about prediction scenarios and task descriptions), step S4 of this embodiment constructs an outer layer reprogramming framework to efficiently combine time series data with natural language input to further improve the model's reasoning capabilities. Specifically, the input layer reconstructed in step S4 of this embodiment discards the original input layer of the GPT2 model and redesigns the model input layer driven by natural language instructions, such as Figure 3 As shown in the figure, the reconstructed input layer additionally introduces task instructions in natural language to provide additional explanations for prediction tasks and application scenarios. The vector input to the large language model through the reconstructed input layer is composed of two parts. One part is the time series data input, which is embedded in the time series input layer after fragmentation and normalization. (This is the reconstructed temporal input embedding layer in the input layer, rather than the temporal input embedding layer in GPT2) Extract temporal feature information to form a vector representation of the input sequence The other part is the natural language instruction input, which is formed into a vector representation of natural language after passing through the input embedding layer of the large language model. .

[0059] Therefore, in step S4 of this embodiment, when the reconstructed input layer generates corresponding natural language instructions according to the time series data, it includes: After the time series data is input into the embedding layer after being fragmented and normalized, Extract time series feature information to form a vector representation of the time series ; Pass the preset natural language input containing the command prompt words through the text input embedding layer Then form a vector representation of natural language , specifically, the text input by the preset natural language According to the character table Perform word segmentation to form corresponding results ,in Indicates word segmentation tokens, is the number of tokens, which are then fed into the embedding layer via text input The linear mapping network parameters map each word segmentation token to a high-dimensional space to form a text embedding vector And as a vector representation of natural language , It is the backbone dimension of the large language model; Represent the time series as a vector and vector representation of natural language Concatenation, specifically the vector representation of the time series and the vector representation of natural language concatenated along the backbone dimension of the large language model , thereby obtaining natural language instructions and using them as input for the large language model , further enhancing the model's reasoning ability for time series concepts, the expression is as follows:

[0060] in, Represents a concatenation operation.

[0061] In this embodiment, the natural language instructions input into the large language model should correspond to the input time series data, and the generation format is fixed: "basic description of the data set + prediction task description + input time series data mean information + input time series data standard deviation information + input data trend information".

[0062] The following uses the open source dataset ETTh1 as an example to further explain how natural language instructions are generated. For a batch of time series input data from this dataset, the natural language instructions generated according to the above generation format are as follows: -Basic description of the dataset: "The ETTh1 dataset is an open source dataset in the power field. It contains 2 years of data points with a time interval of 1 hour. It contains 7 groups of variables. The target variable is transformer oil temperature, and the remaining 6 groups are covariates related to the target variable." -The prediction task description is: "Input the time series data of 192 points in the past and predict the time series data of 32 points in the future." -Input time series data mean information: "The mean of the input data is 12.316" -Enter the standard deviation information of the time series data: "The standard deviation is 3.263" - Trend information of input data: "Increasing trend" The result of splicing the above information in sequence is the natural language instructions corresponding to the data set and the current input time series data. Natural language task instructions provide a supplementary description of the time series data characteristics and task scenarios, which can provide additional information to improve the prediction effect.

[0063] In order to further improve the performance of the model in the power system source and load prediction scenario, step S5 of this implementation performs the second stage training based on the above steps, with the goal of optimizing and adjusting the parameters of the model input layer driven by the constructed natural language instructions. In step S5, the power system source and load data includes wind turbines, photovoltaic power stations and load data from busbars, feeders, and substations, such as Figure 3 As shown in the figure, the weight parameters of the large language model need to be frozen during the training process, and the text input embedding layer in the input layer reconstructed using the power system source and load data Parameters and temporal input embedding layer The parameters are optimized and adjusted until the model parameters converge and reach the optimal prediction state, so that it can normalize the time series input and effectively extract the semantic feature information of the time series.

[0064] In step S5 of this embodiment, the specific steps of freezing the weight parameters of the large language model and using the power system source and load data to perform the second stage training on the large language model are at the same level as the steps of freezing the weights of the feedforward neural network and multi-head attention in the decoders and encoders of each layer of the large language model in step S3, and using the data set to perform the first stage training on the large language model, except that the parameter set of the model parameters that are not frozen is Specifically includes the time series input embedding layer in the reconstructed input layer The network parameters and text input embedding layer The network parameters of the model are frozen. Specifically, it includes position encoding parameters, embedding layer parameters, normalization layer parameters, feedforward neural network model parameters, and multi-head attention mechanism model parameters. The subsequent training loss function and gradient descent process are consistent with the first-stage training.

[0065] After completing the above steps in order, a time series prediction pre-training model based on the weight of the large language model can be constructed. In step S6 of this embodiment, when applying the model for time series prediction, the following factors can be combined to determine whether the time series prediction pre-training model needs to be further trained: If the task of the pre-trained model is very similar to the time series prediction task of the actual application, the pre-trained model can be used directly for prediction. The prediction result can be obtained by simply inputting the power system time series data into the model.

[0066] If there is a significant difference between the distribution of the power system time series data to be predicted and the data used in the second-stage training, it is necessary to use the power system time series data to be predicted to fine-tune the time series prediction pre-training model so that the model can adapt to the specific input distribution and characteristics.

[0067] Embodiment 2 Based on the method of the first embodiment, this embodiment proposes a power system time series prediction model training system based on large language model weight fine-tuning, including: The data set generation unit is used to obtain time series data in different fields and obtain data sets after data cleaning; The first-stage training unit is used to freeze the weights of the feedforward neural network and the multi-head attention in the decoders and encoders of each layer of the large language model, and use the data set to perform the first-stage training on the large language model to adjust the weight parameters of the position encoder, the layer normalization network, and the input embedding layer of the large language model; A model reconstruction unit, used to reconstruct the input layer of the large language model so that the reconstructed input layer generates corresponding natural language instructions according to the time series data, and inputs the natural language instructions into the large language model; A second-stage training unit, used for freezing the weight parameters of the large language model, and performing second-stage training on the large language model using power system source and load data to adjust the parameters of the reconstructed input layer to obtain a time series prediction pre-training model; The result prediction unit is used to input the power system time series data and task instructions into the trained time series prediction pre-training model to obtain the corresponding prediction results.

[0068] In this embodiment, corresponding to the method of embodiment 1, the data set generation unit obtains open source time series data in different fields by collecting publicly available data sets from the Internet, including time series data from different fields such as transportation, finance, electricity, and weather, so that the model can give full play to the cross-domain learning and reasoning capabilities and improve the generalization performance of the model. In the process of collecting time series data, the basic information of the open source data set is counted, including: the number of time series data points, the number of variables, the time series interval, the time span, the field to which the data belongs, and other basic information. The data set generation unit performs data cleaning by randomly sampling time series fragments from each data set, marking and removing missing values ​​and outliers, and improving the overall quality of the data set through data preprocessing. When constructing a data set, according to the principle of uniform sampling, ensure that each batch of data input to the model comes from time series in different fields (electricity, transportation, finance, weather, etc.) and the data ratio is close to balance, that is, evenly distributed in proportion.

[0069] In this embodiment, corresponding to the method of embodiment 1, the large language model adopts the GPT2 model, and each layer of decoder and encoder includes at least one basic network unit, and a basic network unit includes multi-head attention, layer normalization, and feedforward neural network layer. The calculation process of the basic network unit on time series data includes: For the input time series After reversible normalization, it is divided into multiple segments , each fragment Input to the input embedding layer to get the corresponding embedding vector , The backbone dimension of the GPT2 model is to embed the vector and superimpose the position encoder Output the encoded information and get the encoded vector , the expression is as follows: ; The encoded vector Input the multi-head attention mechanism to calculate the corresponding attention output result. In this process, first calculate the query matrix , key-value matrix and the weight matrix , the expression is as follows: , ,

[0070] in, , and are the weights of the query matrix, key value matrix and weight matrix respectively; based on , , Three matrices are used for attention calculation, and the expressions are as follows:

[0071] in, Indicates In multi-head attention calculation, the final attention output result is Depend on The calculation results of the attention heads are concatenated and calculated, and the expression is as follows:

[0072] in, It is the linear change matrix after concatenating the outputs of multiple attention heads; The attention output result is input into the normalization layer to obtain the normalized result. The output of the multi-head attention is normalized and residually connected to prevent the gradient from disappearing during training. The operation process is as follows:

[0073] in, Represents the normalization layer in the GPT2 model; Then the normalized result is input into the feedforward neural network, and the operation process is:

[0074] As can be seen from the formula, the feedforward neural network consists of two layers of fully connected networks, where , is the weight matrix, , is the bias term; Finally, the output result of the feedforward neural network is processed again by the normalization layer to obtain the final output result of the current basic network unit, which is expressed as follows:

[0075] The above is a data operation process in a basic network unit. The GPT2 model uses multiple identical basic network units to process input data in series to achieve deep feature extraction at different scales. After processing by the basic network units of the layer, the final output result can be expressed as , and finally the prediction result of the time series is obtained through the output projection layer , the expression is as follows:

[0076] In the formula, The output projection layer parameters of the model.

[0077] In this embodiment, corresponding to the method in the first embodiment, when the input layer reconstructed by the model reconstruction unit generates corresponding natural language instructions according to the time series data, it includes: After the time series data is input into the embedding layer after being fragmented and normalized, Extract time series feature information to form a vector representation of the time series ; Pass the preset natural language input containing the command prompt words through the text input embedding layer Then form a vector representation of natural language , specifically, the text input by the preset natural language According to the character table Perform word segmentation to form corresponding results ,in Indicates word segmentation tokens, is the number of tokens, which are then fed into the embedding layer via text input The linear mapping network parameters map each word segmentation token to a high-dimensional space to form a text embedding vector And as a vector representation of natural language , It is the backbone dimension of the large language model; Represent the time series as a vector and vector representation of natural language Concatenation, specifically the vector representation of the time series and the vector representation of natural language concatenated along the backbone dimension of the large language model , thereby obtaining natural language instructions and using them as input for the large language model , further enhancing the model's reasoning ability for time series concepts, the expression is as follows:

[0078] in, Represents a concatenation operation.

[0079] In this embodiment, corresponding to the method of embodiment 1, the first-stage training unit freezes the weights of the feedforward neural network and the multi-head attention in the decoders and encoders of each layer of the large language model, and uses the data set to perform the first-stage training on the large language model, and the second-stage training unit freezes the weight parameters of the large language model, and uses the power system source and load data to perform the second-stage training on the large language model, both of which include: Add unfrozen model parameters to the parameter set , add the frozen model parameters to the parameter set ; Taking the historical time series data in the data set or power system source and load data as input, the prediction results are output according to the large language model. and the corresponding true value Calculate the loss function for the current prediction , the expression is as follows:

[0080] in, Represents all samples in a batch; Calculating the loss function About parameter collection Gradient , and use this to update the first-order moment estimate and the second moment estimate , the expression is as follows:

[0081]

[0082] in, represents the first-order moment estimate before updating, represents the second-order moment estimate before updating; is the decay rate of the first-order moment estimate, is the decay rate of the second-order moment estimate; According to the first-order moment estimate and the second moment estimate The calculated results of the calculation are used to perform bias correction to obtain the corrected first-order moment estimate and the corrected second moment estimate , the expression is as follows:

[0083]

[0084] in, is the decay rate of the first-order moment estimate The tth power, is the decay rate of the second-order moment estimate t to the power of Use the corrected first moment estimate and the corrected second moment estimate Update parameter set The model parameters in are expressed as follows:

[0085] in, represents the model parameters before updating, represents the updated model parameters, is the learning rate, is a constant.

[0086] In this embodiment, corresponding to the method of the first embodiment, the first-stage training unit freezes the weights of the feedforward neural network and the multi-head attention in the decoder and encoder of each layer of the large language model, and adds the unfrozen model parameters to the parameter set when the large language model is trained in the first stage using the data set. Specifically, the position encoding parameters, embedding layer parameters and normalization layer parameters are added to the parameter set , add the frozen model parameters to the parameter set Specifically, the feedforward neural network model parameters and the multi-head attention mechanism model parameters are added to the parameter set .

[0087] In this embodiment, corresponding to the method of the first embodiment, the second-stage training unit freezes the weight parameters of the large language model, and when the power system source and load data are used to perform the second-stage training on the large language model, the unfrozen model parameters are added to the parameter set. Specifically, the time series input in the reconstructed input layer is embedded in the layer The network parameters and text input embedding layer The network parameters are added to the parameter set , add the frozen model parameters to the parameter set Specifically, the position encoding parameters, embedding layer parameters, normalization layer parameters, feedforward neural network model parameters and multi-head attention mechanism model parameters are added to the parameter set .

[0088] Embodiment 3 The present embodiment proposes a power system time series prediction model training device based on large language model weight fine-tuning, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the power system time series prediction model training method based on large language model weight fine-tuning described in Example 1.

[0089] Embodiment 4 This embodiment proposes a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the power system time series prediction model training method based on large language model weight fine-tuning described in Embodiment 1 are implemented.

[0090] In summary, the present invention proposes a method, system, device and storage medium for training a power system time series prediction model based on fine-tuning of the weight of a large language model. Based on the pre-trained model weights of the open source large language model GPT2, the technical means of freezing network parameters and the technical method of adjusting the parameters of the two-stage pre-trained model are used to enable the large language model to adapt to the source-load prediction task in the power system scenario during the weight adjustment process. The time series prediction pre-training model constructed by the present invention has significant advantages in generalization performance, and the prediction accuracy in special prediction scenarios such as small samples and zero samples is significantly improved compared to traditional prediction methods.

[0091] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, devices or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it.

[0092] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0093] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0094] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0095] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A power system time series prediction model training method based on large language model weight fine-tuning, characterized in that: The following steps are involved: Obtain time series data from different fields and clean the data to get the data set; Freeze the weights of the feedforward neural network and multi-head attention in the decoder and encoder of each layer of the large language model, and use the dataset to perform the first phase of training on the large language model to adjust the weight parameters of the position encoder, layer normalization network, and input embedding layer of the large language model; Reconstructing the input layer of the large language model so that the reconstructed input layer generates corresponding natural language instructions according to the time series data and inputs the natural language instructions into the large language model; The weight parameters of the large language model are frozen, and the large language model is trained in the second stage using the power system source and load data to adjust the parameters of the reconstructed input layer to obtain a time series prediction pre-training model.

2. The power system time series prediction model training method based on large language model weight fine-tuning according to claim 1 is characterized in that: After obtaining the time series prediction pre-training model, the method further includes: inputting the power system time series data into the trained time series prediction pre-training model to obtain corresponding prediction results.

3. The power system time series prediction model training method based on large language model weight fine-tuning according to claim 1 is characterized in that: Each decoder and encoder of the large language model includes at least one basic network unit, and the calculation process of the basic network unit on the time series data includes: The input time series is reversibly normalized and divided into multiple segments. Each segment is input into the input embedding layer to obtain the corresponding embedding vector. The embedding vector is superimposed with the encoding information output by the position encoder to obtain the encoded vector. Input the encoded vector into the multi-head attention mechanism to calculate the corresponding attention output result; The attention output result is input into the normalization layer to obtain the normalized result, and then the normalized result is input into the feedforward neural network. Finally, the output result of the feedforward neural network is processed again by the normalization layer to obtain the final output result of the current basic network unit.

4. The power system time series prediction model training method based on large language model weight fine-tuning according to claim 1 is characterized in that: The reconstructed input layer generates corresponding natural language instructions based on time series data, including: After the time series data is input into the embedding layer after being fragmented and normalized, Extract time series feature information to form a vector representation of the time series; Pass the preset natural language input containing the command prompt words through the text input embedding layer Then a vector representation of natural language is formed; The vector representation of the time series and the vector representation of the natural language are concatenated to obtain the natural language instructions and use them as the input of the large language model.

5. The power system time series prediction model training method based on large language model weight fine-tuning according to claim 4 is characterized in that: Pass the preset natural language input containing the command prompt words through the text input embedding layer When the vector representation of natural language is formed, it specifically includes: The text that will be input in the preset natural language According to the character table Perform word segmentation to form corresponding results ,in Indicates word segmentation tokens, is the number of tokens, which are then fed into the embedding layer via text input The linear mapping network parameters map each word segmentation token to a high-dimensional space to form a text embedding vector And as a vector representation of natural language , It is the backbone dimension of the large language model.

6. The power system time series prediction model training method based on large language model weight fine-tuning according to claim 4 is characterized in that: When concatenating the vector representation of the input sequence and the vector representation of the natural language, specifically, the vector representation of the time series and the vector representation of the natural language are concatenated along the backbone dimension of the large language model.

7. The power system time series prediction model training method based on large language model weight fine-tuning according to claim 1 is characterized in that: Freezing the weights of the feedforward neural network and the multi-head attention in the decoders and encoders of each layer of the large language model, and using the data set to perform the first-stage training of the large language model, as well as freezing the weight parameters of the large language model and using the power system source and load data to perform the second-stage training of the large language model, all include: Add unfrozen model parameters to the parameter set , add the frozen model parameters to the parameter set ; Taking the historical time series data in the data set or power system source and load data as input, the prediction results are output according to the large language model. and the corresponding true value Calculate the loss function for the current prediction ; Calculating the loss function About parameter collection Gradient , and use this to update the first-order moment estimate and the second moment estimate ; According to the first-order moment estimate and the second moment estimate The calculated results of the calculation are used to perform bias correction to obtain the corrected first-order moment estimate and the corrected second moment estimate ; Use the corrected first moment estimate and the corrected second moment estimate Update parameter set The model parameters in .

8. The power system time series prediction model training method based on large language model weight fine-tuning according to claim 7 is characterized in that: Freeze the weights of the feedforward neural network and multi-head attention in the decoder and encoder of each layer of the large language model, and add the unfrozen model parameters to the parameter set when using the dataset to perform the first stage training of the large language model Specifically, the position encoding parameters, embedding layer parameters and normalization layer parameters are added to the parameter set , add the frozen model parameters to the parameter set Specifically, the feedforward neural network model parameters and the multi-head attention mechanism model parameters are added to the parameter set .

9. The power system time series prediction model training method based on large language model weight fine-tuning according to claim 7 is characterized in that: Freeze the weight parameters of the large language model, and when the large language model is trained in the second phase using the power system source and load data, add the unfrozen model parameters to the parameter set Specifically, the time series input in the reconstructed input layer is embedded in the layer The network parameters and text input embedding layer The network parameters are added to the parameter set , add the frozen model parameters to the parameter set Specifically, the position encoding parameters, embedding layer parameters, normalization layer parameters, feedforward neural network model parameters and multi-head attention mechanism model parameters are added to the parameter set .

10. A power system time series prediction model training system based on large language model weight fine-tuning, characterized in that: include: The data set generation unit is used to obtain time series data in different fields and obtain data sets after data cleaning; The first-stage training unit is used to freeze the weights of the feedforward neural network and the multi-head attention in the decoders and encoders of each layer of the large language model, and use the data set to perform the first-stage training on the large language model to adjust the weight parameters of the position encoder, the layer normalization network, and the input embedding layer of the large language model; A model reconstruction unit, used to reconstruct the input layer of the large language model, so that the reconstructed input layer generates corresponding natural language instructions according to the time series data and inputs the natural language instructions into the large language model; The second stage training unit is used to freeze the weight parameters of the large language model and use the power system source and load data to perform the second stage training on the large language model to adjust the parameters of the reconstructed input layer to obtain a time series prediction pre-training model.

11. The power system time series prediction model training system based on large language model weight fine-tuning according to claim 10, characterized in that: Also includes: The result prediction unit is used to input the power system time series data and task instructions into the trained time series prediction pre-training model to obtain the corresponding prediction results.

12. The power system time series prediction model training system based on large language model weight fine-tuning according to claim 10, characterized in that: Each decoder and encoder of the large language model includes at least one basic network unit, and the calculation process of the basic network unit on the time series data includes: The input time series is reversibly normalized and divided into multiple segments. Each segment is input into the input embedding layer to obtain the corresponding embedding vector. The embedding vector is superimposed with the encoding information output by the position encoder to obtain the encoded vector. Input the encoded vector into the multi-head attention mechanism to calculate the corresponding attention output result; The attention output result is input into the normalization layer to obtain the normalized result, and then the normalized result is input into the feedforward neural network. Finally, the output result of the feedforward neural network is processed again by the normalization layer to obtain the final output result of the current basic network unit.

13. The power system time series prediction model training system based on large language model weight fine-tuning according to claim 10, characterized in that: The reconstructed input layer generates corresponding natural language instructions based on time series data, including: After the time series data is input into the embedding layer after being fragmented and normalized, Extract time series feature information to form a vector representation of the time series; Pass the preset natural language input containing the command prompt words through the text input embedding layer Then a vector representation of natural language is formed; The vector representation of the time series and the vector representation of the natural language are concatenated to obtain the natural language instructions and use them as the input of the large language model.

14. The power system time series prediction model training system based on large language model weight fine-tuning according to claim 13, characterized in that: Pass the preset natural language input containing the command prompt words through the text input embedding layer When the vector representation of natural language is formed, it specifically includes: The text that will be input in the preset natural language According to the character table Perform word segmentation to form corresponding results ,in Indicates word segmentation tokens, is the number of tokens, which are then fed into the embedding layer via text input The linear mapping network parameters map each word segmentation token to a high-dimensional space to form a text embedding vector And as a vector representation of natural language , It is the backbone dimension of the large language model.

15. The power system time series prediction model training system based on large language model weight fine-tuning according to claim 13, characterized in that: When concatenating the vector representation of the input sequence and the vector representation of the natural language, specifically, the vector representation of the time series and the vector representation of the natural language are concatenated along the backbone dimension of the large language model.

16. The power system time series prediction model training system based on large language model weight fine-tuning according to claim 10, characterized in that: The first-stage training unit freezes the weights of the feedforward neural network and the multi-head attention in the decoders and encoders of each layer of the large language model, and uses the data set to perform the first-stage training on the large language model, and the second-stage training unit freezes the weight parameters of the large language model, and uses the power system source and load data to perform the second-stage training on the large language model, both of which include: Add unfrozen model parameters to the parameter set , add the frozen model parameters to the parameter set ; Taking the historical time series data in the data set or power system source and load data as input, the prediction results are output according to the large language model. and the corresponding true value Calculate the loss function for the current prediction ; Calculating the loss function About parameter collection Gradient , and use this to update the first-order moment estimate and the second moment estimate ; According to the first-order moment estimate and the second moment estimate The calculated results of the calculation are used to perform bias correction to obtain the corrected first-order moment estimate and the corrected second moment estimate ; Use the corrected first moment estimate and the corrected second moment estimate Update parameter set The model parameters in .

17. The power system time series prediction model training system based on large language model weight fine-tuning according to claim 16, characterized in that: The first-stage training unit freezes the weights of the feedforward neural network and multi-head attention in the decoder and encoder of each layer of the large language model, and adds the unfrozen model parameters to the parameter set when using the dataset to perform the first-stage training of the large language model. Specifically, the position encoding parameters, embedding layer parameters and normalization layer parameters are added to the parameter set , add the frozen model parameters to the parameter set Specifically, the feedforward neural network model parameters and the multi-head attention mechanism model parameters are added to the parameter set .

18. The power system time series prediction model training system based on large language model weight fine-tuning according to claim 16, characterized in that: The second-stage training unit freezes the weight parameters of the large language model, and when the power system source and load data are used to perform the second-stage training on the large language model, the unfrozen model parameters are added to the parameter set. Specifically, the time series input in the reconstructed input layer is embedded in the layer The network parameters and text input embedding layer The network parameters are added to the parameter set , add the frozen model parameters to the parameter set Specifically, the position encoding parameters, embedding layer parameters, normalization layer parameters, feedforward neural network model parameters and multi-head attention mechanism model parameters are added to the parameter set .

19. A power system time series prediction model training device based on large language model weight fine-tuning, comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the power system time series prediction model training method based on large language model weight fine-tuning as described in any one of claims 1 to 9.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the power system time series prediction model training method based on large language model weight fine-tuning as described in any one of claims 1 to 9 are implemented.

Images