Method for multi-task risk assessment of sea wind grid-connected system based on human feedback reinforcement learning

Abstract

A sea wind grid-connected system multi-task risk assessment method based on human feedback reinforcement learning is disclosed, and belongs to the technical field of power grids. The method comprises the following steps: obtaining a training set based on a plurality of time series data of a sea wind grid-connected system, wherein the training set comprises a first sub-training set, a second sub-training set and a third sub-training set; the first sub-training set is used for training an abnormal value detection task; the second sub-training set is used for training a missing value filling task; the third sub-training set is used for training a prediction task; the output of the prediction task is used for risk assessment; a multi-task risk assessment model is established; the multi-task risk assessment model is trained using the training set; in the process of training the multi-task risk assessment model, a human feedback reinforcement learning algorithm is used to perform reinforcement learning on the multi-task risk assessment model; and risk assessment is performed based on the trained multi-task risk assessment model. The method can improve the accuracy of risk assessment of the model in the multi-task sea wind grid-connected system.

Description

Technical Field

[0001] This disclosure relates to the field of power grid technology, and in particular to a multi-task risk assessment method for offshore wind grid-connected systems based on human feedback reinforcement learning. Background Technology

[0002] Offshore wind grid-connected systems are management systems for offshore wind power grid connection. Offshore wind power grid connection refers to large-scale grid-connected wind turbines directly connected to the power grid at sea. With the continuous increase in the scale of offshore wind power grid connection and the increasing complexity of its operating environment, the amount of data that offshore wind grid-connected systems need to process is also constantly increasing. The risk assessment process for offshore wind grid-connected systems typically includes: establishing multiple tasks to process the acquired data, and conducting a risk assessment based on the results of the data processing.

[0003] In related technologies, risk assessment methods include: establishing a risk assessment model and training the risk assessment model using a task to achieve risk assessment.

[0004] However, modern wind power grid-connected systems suffer from large data processing volumes and numerous tasks. When a risk assessment model trained on a single task is used to assess risk in a multi-task offshore wind grid-connected system, the accuracy of the risk assessment is low. This is especially true under conditions such as large-scale unstable renewable energy access and significant fluctuations in power load, which can easily lead to model failure or inaccurate risk assessment. Summary of the Invention

[0005] This disclosure provides a multi-task risk assessment method for offshore wind grid-connected systems based on human feedback reinforcement learning, which can improve the accuracy of risk assessment in multi-task offshore wind grid-connected systems. The technical solution includes at least the following components:

[0006] Firstly, a multi-task risk assessment method for a wind-powered grid-connected system based on human feedback reinforcement learning is provided, comprising: acquiring a training set based on multiple time-series data of the wind-powered grid-connected system, wherein the training set includes a first sub-training set, a second sub-training set, and a third sub-training set, wherein the first sub-training set is used to train an outlier detection task, the second sub-training set is used to train a missing value imputation task, and the third sub-training set is used to train a prediction task; establishing a multi-task risk assessment model, wherein the multi-task risk assessment model is a natural language model, and the multi-task risk assessment model is used to perform the outlier detection task, the missing value imputation task, and the prediction task, wherein the output of the prediction task is used for risk assessment; training the multi-task risk assessment model using the training set, wherein during the training process, a human feedback reinforcement learning algorithm is used to perform reinforcement learning on the multi-task risk assessment model; and performing a risk assessment on the wind-powered grid-connected system based on the trained multi-task risk assessment model.

[0007] Optionally, the reinforcement learning algorithm for the multi-task risk assessment model using human feedback includes: training a reward model based on a human feedback dataset; and using a proximal policy optimization algorithm to perform reinforcement learning on the multi-task risk assessment model based on the reward model.

[0008] Optionally, the reinforcement learning of the multi-task risk assessment model based on the reward model and employing a proximal policy optimization algorithm includes: generating multiple first sequences based on a first policy of the multi-task risk assessment model, wherein the first policy is the policy of the multi-task risk assessment model in the (k-1)th iteration; evaluating the multiple first sequences based on the reward model to obtain the cumulative reward of each first sequence; determining the advantage function at each time step based on the cumulative reward and value function of each first sequence, wherein the value function is determined based on the proximal policy optimization algorithm; and updating the first policy based on the loss function of the proximal policy optimization algorithm and the advantage function at each time step to complete the kth iteration; wherein k is a positive integer.

[0009] Optionally, the loss function of the near-end policy optimization algorithm is expressed by the following formula:

[0010]

[0011] in, Let be the loss function of the near-end policy optimization algorithm. For the expectation, This is the ratio of the probability of the policy in the (k-1)th iteration to the probability of the policy in the kth iteration. For the aforementioned advantage function, This is a truncation function. The shearing threshold, As a weighting factor, For the first strategy Compared with the initial strategy The KL divergence between them, the initial strategy is the strategy of the multi-task risk assessment model before the first iteration.

[0012] Optionally, the loss function of the reward model is determined based on the Bradley-Terry model; or, the loss function of the reward model is log-likelihood loss.

[0013] Optionally, the step of acquiring a training set based on multiple time-series data of the sea-wind grid-connected system includes: acquiring a time-series dataset, which includes multiple time-series data of the sea-wind grid-connected system; performing downsampling processing on the multiple time-series data; and acquiring the training set based on the downsampled time-series dataset.

[0014] Secondly, a multi-task risk assessment device for a wind-powered grid-connected system based on human feedback reinforcement learning is also provided, comprising: an acquisition module for acquiring a training set based on multiple time-series data of the wind-powered grid-connected system, the training set including a first sub-training set, a second sub-training set, and a third sub-training set, wherein the first sub-training set is used to train an outlier detection task, the second sub-training set is used to train a missing value imputation task, and the third sub-training set is used to train a prediction task; a modeling module for establishing a multi-task risk assessment model, the multi-task risk assessment model being a natural language model, the multi-task risk assessment model being used to perform the outlier detection task, the missing value imputation task, and the prediction task, the output of the prediction task being used for risk assessment; a training module for training the multi-task risk assessment model using the training set, wherein during the training of the multi-task risk assessment model, a human feedback reinforcement learning algorithm is used to perform reinforcement learning on the multi-task risk assessment model; and a risk assessment module for performing a risk assessment on the wind-powered grid-connected system based on the trained multi-task risk assessment model.

[0015] Optionally, the training module is further used to train a reward model based on a human feedback dataset; and based on the reward model, a proximal policy optimization algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

[0016] Optionally, the training module is further configured to generate multiple first sequences based on the first strategy of the multi-task risk assessment model, wherein the first strategy is the strategy of the multi-task risk assessment model in the (k-1)th iteration; evaluate the multiple first sequences based on the reward model to obtain the cumulative reward of each first sequence; determine the advantage function at each time step based on the cumulative reward and value function of each first sequence, wherein the value function is determined based on the proximal policy optimization algorithm; and update the first strategy based on the loss function of the proximal policy optimization algorithm and the advantage function at each time step to complete the kth iteration; wherein k is a positive integer.

[0017] Optionally, in the training module, the loss function of the near-end policy optimization algorithm is expressed by the following formula:

[0018]

[0019] in, Let be the loss function of the near-end policy optimization algorithm. For the expectation, This is the ratio of the probability of the policy in the (k-1)th iteration to the probability of the policy in the kth iteration. For the aforementioned advantage function, This is a truncation function. The shearing threshold, As a weighting factor, For the first strategy Compared with the initial strategy The KL divergence between them, the initial strategy is the strategy of the multi-task risk assessment model before the first iteration.

[0020] Optionally, in the training module, the loss function of the reward model is determined based on the Bradley-Terry model; or, the loss function of the reward model is log-likelihood loss.

[0021] Optionally, the acquisition module is further configured to acquire a time-series dataset, which includes multiple time-series data of the offshore wind grid-connected system; perform downsampling processing on the multiple time-series data; and acquire the training set based on the downsampled time-series dataset.

[0022] Thirdly, a computer device is also provided, comprising: a memory and a processor, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to perform the multi-task risk assessment method for a sea-wind grid-connected system based on human feedback reinforcement learning as described in the above embodiments.

[0023] Fourthly, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor, thereby performing the multi-task risk assessment method for a sea-wind grid-connected system based on human feedback reinforcement learning as described in the above embodiments.

[0024] Fifthly, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0025] The beneficial effects of the technical solutions provided in this disclosure include at least the following:

[0026] In this embodiment, by employing human feedback reinforcement learning to train the multi-task risk assessment model, the multi-task execution capability of the model can be improved, ultimately optimizing its performance in outlier detection, missing value imputation, and prediction tasks. This enables the model to accurately execute multiple tasks and quickly adapt to changes in different scenarios.

[0027] After the multi-task risk assessment model is trained, it can preprocess the time-series data obtained from the offshore wind grid-connected system by performing outlier detection and missing value imputation tasks. By performing prediction tasks, the multi-task risk assessment model can predict the future time-series data of the offshore wind grid-connected system based on the preprocessed time-series data, thereby enabling a stable and accurate risk assessment of the offshore wind grid-connected system based on the future time-series data. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart is shown below illustrating a multi-task risk assessment method for a sea breeze grid-connected system based on human feedback reinforcement learning, provided in an exemplary embodiment of this disclosure.

[0030] Figure 2 A flowchart is shown below illustrating a multi-task risk assessment method for a sea breeze grid-connected system based on human feedback reinforcement learning, provided in another exemplary embodiment of this disclosure.

[0031] Figure 3 This illustration shows a schematic diagram of the structure of a multi-task risk assessment device for a sea breeze grid-connected system based on human feedback reinforcement learning, provided in an exemplary embodiment of this disclosure.

[0032] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0033] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, but do not exclude other elements or objects.

[0034] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0035] Figure 1 A flowchart illustrating a multi-task risk assessment method for a wind-powered grid-connected system based on human feedback reinforcement learning, provided in an exemplary embodiment of this disclosure, is shown. This method can be executed by a computer device. See also Figure 1 The method includes:

[0036] In step 101, a training set is obtained based on multiple time-series data from the offshore wind grid-connected system.

[0037] The training set includes a first sub-training set, a second sub-training set, and a third sub-training set. The first sub-training set is used to train the outlier detection task, the second sub-training set is used to train the missing value imputation task, and the third sub-training set is used to train the prediction task.

[0038] Optionally, a first cue word is present before any time series data in the first sub-training set. The first cue word is used to instruct the multi-task risk assessment model to perform the outlier detection task.

[0039] A second cue word precedes any time series data in the second sub-training set. The second cue word is used to instruct the multi-task risk assessment model to perform the missing value imputation task.

[0040] A third cue word precedes any time series data in the third sub-training set. The third cue word is used to instruct the multi-task risk assessment model to perform the prediction task.

[0041] In this embodiment, three types of tasks are distinguished by a first prompt word, a second prompt word, and a third prompt word. When the multi-task risk assessment model performs multiple tasks, it can understand which task needs to be performed on the subsequent time-series data based on the prompt words preceding the time-series data.

[0042] For example, the first prompt word is The time-series data in the first training subset are as follows:

[0043] In this sequence, Indicates the start marker of time series data. This indicates the end marker of time series data. This is a separator used to separate the question from the answer; Outlier labels are generated manually during training. This is equivalent to inputting a problem into a multi-task risk assessment model. This is equivalent to inputting the answer into a multi-task risk assessment model.

[0044] During training, both the question and answer are input into the multi-task risk assessment model simultaneously. After training, only the question without outlier labels needs to be input. For the outlier labeling task, the question input into the multi-task risk assessment model after training is, for example: After receiving a question, the multi-task risk assessment model uses outlier markers to identify outliers in the question. The model's output, for example, is... .

[0045] The essence of training a multi-task risk assessment model using the first subset of training set is to train the model's ability to accurately use outlier labeling symbols. After training, the multi-task risk assessment model can accurately label outliers in time series data using outlier labeling symbols.

[0046] For example, the second prompt word is The time-series data in the second training set are as follows:

[0047] In this sequence, Mark missing values ​​with symbols. For the missing values ​​to be filled, missing value labels and the filled missing values ​​are implemented through manual annotation during training. This is equivalent to inputting a problem into a multi-task risk assessment model. This is equivalent to inputting the answer into a multi-task risk assessment model.

[0048] During training, both the question and answer are input into the multi-task risk assessment model simultaneously. After training, only the question without missing value markers needs to be input. For the missing value imputation task, the question input into the multi-task risk assessment model after training is, for example: After receiving a question, the multi-task risk assessment model identifies the missing values ​​and fills in the corresponding missing values. The model's output is, for example,... .

[0049] The essence of training a multi-task risk assessment model using a second sub-training set is to train the model's ability to accurately identify and fill in missing values. After training, the multi-task risk assessment model can accurately identify and fill in missing values ​​in time-series data.

[0050] For example, the third prompt word is The time-series data in the third training set are as follows:

[0051] In this sequence, the preceding time series data is the question for the following time series data, and the following time series data is the answer for the preceding time series data. For example, This is a prediction problem; the predicted answer is the sequence in... The value of the next moment after the current moment is also When training with time-series data from the third training set, the answer can be masked first, and then the mask removed after the model outputs the prediction. For example, when inputting this sequence into a multi-task risk assessment model, the answer can be masked first. Part of it is set as a mask, in the model based on Output prediction After the next value, then remove the sequence. The mask is then removed, and the model can then be used to remove the mask. predict The next mask is used, and so on, until the mask of all data in the sequence is removed.

[0052] For prediction tasks, after training, simply input the third cue word and the data to be predicted into the multi-task risk assessment model. The model can predict the next value after the last value of the input data; this next value is the predicted value. For example, the sequence input into the multi-task risk assessment model for a prediction task might be... The output of the multi-task risk assessment model is: ,in This refers to the predicted value generated by the model.

[0053] The essence of using a third sub-training set to train a multi-task risk assessment model is to train the model's ability to make accurate predictions.

[0054] The aforementioned prompts, missing value markers, and outlier markers can be defined as data belonging to a fixed category. This fixed-category data does not require regression during processing. Typically, time-series data obtained from the sea-wind grid-connected system is continuous and high-precision. Higher data precision results in more categories during subsequent encoding. Directly inputting time-series data from the sea-wind grid-connected system into the model would cause the probability of each category to approach zero due to the excessive number of categories. Therefore, it is necessary to downsample the acquired time-series data to reduce its precision, decrease the number of categories, and simultaneously classify the data.

[0055] Optionally, the method includes the following step ac.

[0056] Step a: Obtain the time series dataset.

[0057] The time-series dataset includes multiple time-series data points from the offshore wind-connected grid system. The time-series dataset can be represented as... ,in, In the first A certain time-series data at a given moment.

[0058] Optionally, multiple time-series data of the offshore wind grid-connected system include operating parameters such as power output, wind speed, and equipment status.

[0059] Step b involves downsampling multiple time-series data.

[0060] Optionally, downsampling can be implemented using formula (1).

[0061] (1)

[0062] In formula (1), In the first A certain time-series data at a given moment, To The data after downsampling The maximum value in the time series dataset. It is the minimum value in the time series dataset. Indicates rounding down. It is a discrete constant, and its value can be determined empirically, usually taking a very small value. The discrete series is typically equal to the number of categories in the downsampled time-series dataset. For example, if the discrete series is 3, then the time-series dataset will also have 3 categories. The value of the discrete series is an empirical value, and this disclosure does not limit its selection.

[0063] Downsampling enables the classification of data in time-series datasets. When the subsequent multi-task risk assessment model makes predictions, the predicted value is not a specific numerical value but rather a category.

[0064] Step c: Obtain the training set based on the downsampled time series dataset.

[0065] After downsampling multiple time series data in the time series dataset, the data in the time series dataset can be divided into three subsets. The data in these three subsets are then manually labeled according to the format of the first, second, and third training subsets, thus obtaining the first, second, and third training subsets, which together constitute the training set.

[0066] In step 102, a multi-task risk assessment model is established.

[0067] The multi-task risk assessment model is a natural language model. It is used to perform outlier detection, missing value imputation and prediction tasks. The output of the prediction task is used for risk assessment.

[0068] In step 103, a multi-task risk assessment model is trained using the training set.

[0069] In the process of training the multi-task risk assessment model, the reinforcement learning algorithm from human feedback (RLHF) is used to perform reinforcement learning on the multi-task risk assessment model.

[0070] In step 104, a risk assessment is performed on the offshore wind grid-connected system based on the trained multi-task risk assessment model.

[0071] In this embodiment, by employing human feedback reinforcement learning to train the multi-task risk assessment model, the multi-task execution capability of the model can be improved, ultimately optimizing its performance in outlier detection, missing value imputation, and prediction tasks. This enables the model to accurately execute multiple tasks and quickly adapt to changes in different scenarios.

[0072] After the multi-task risk assessment model is trained, it can preprocess the time-series data obtained from the offshore wind grid-connected system by performing outlier detection and missing value imputation tasks. By performing prediction tasks, the multi-task risk assessment model can predict the future time-series data of the offshore wind grid-connected system based on the preprocessed time-series data, thereby enabling a stable and accurate risk assessment of the offshore wind grid-connected system based on the future time-series data.

[0073] Figure 2 A flowchart illustrating a multi-task risk assessment method for a wind-powered grid-connected system based on human feedback reinforcement learning, provided in another exemplary embodiment of this disclosure, is shown. This method can be executed by a computer device. See also Figure 2 The method includes:

[0074] In step 201, a training set is obtained based on multiple time-series data of the offshore wind grid-connected system.

[0075] The training set includes a first sub-training set, a second sub-training set, and a third sub-training set. The first sub-training set is used to train the outlier detection task, the second sub-training set is used to train the missing value imputation task, and the third sub-training set is used to train the prediction task.

[0076] The details of step 201 are the same as those in step 101 above, and will not be described in detail here.

[0077] In step 202, a multi-task risk assessment model is established.

[0078] The multi-task risk assessment model is a natural language model. It is used to perform outlier detection, missing value imputation, and prediction tasks, with the output of the prediction task used for risk assessment.

[0079] For example, the multi-task risk assessment model established in step 202 is a pre-trained natural language model, such as a GPT (Generative Pre-Training) model, a BERT (Bidirectional Encoder Representations from Transformers) model, or an LLaMA (Large Language Model Meta AI) model. This multi-task risk assessment model can be represented as follows: This indicates that the goal of the multi-task risk assessment model is to assess the input sequence. Predict the next token in the input sequence. .

[0080] Optionally, before performing step 203, the method further includes: performing preliminary training on the multi-task risk assessment model in step 202 using a multi-task dataset. Here, the data in the multi-task dataset also needs to be processed in a similar manner to the data in the training set before it can be used for preliminary training of the multi-task risk assessment model. The data in the multi-task dataset can be any type of multi-task data, and this disclosure does not limit this.

[0081] For example, when using data from a multi-task dataset to initially train a multi-task risk assessment model, the loss function can be the cross-entropy loss, expressed by formula (2).

[0082] (2)

[0083] In formula (2), This indicates a token in the sequence input to the multi-task risk assessment model during the initial training process.

[0084] In step 203, a multi-task risk assessment model is trained using the training set.

[0085] During the training of the multi-task risk assessment model, a human feedback reinforcement learning algorithm is used to reinforce the model.

[0086] Optionally, step 203 includes the following steps:

[0087] Step d: Train a reward model based on the human feedback dataset.

[0088] Optionally, step d includes: obtaining a human feedback dataset; building a reward model; and training the reward model using the human feedback dataset.

[0089] The data in the human feedback dataset reflects human preferences. For example, with the participation of human evaluators, pairwise comparisons can be made of different outputs generated by a model under the same input to collect human preference data. For instance, for the same input, a model generates two candidate outputs, s1 and s2, and humans choose the better one, thus forming a data point in the human feedback dataset.

[0090] Reward models, such as neural networks, can be trained using human feedback datasets. This allows the reward model to learn human preferences and provide reward signals for subsequent reinforcement learning optimization.

[0091] In one possible implementation, the loss function of the reward model is determined based on the Bradley-Terry model, in which case the loss function of the reward model is expressed by formula (3).

[0092] (3)

[0093] In formula (3), The loss function for the reward model, , This represents two candidate outputs generated by a model given the same input. This represents the candidate outputs in terms of human preferences. Better than candidate output . This indicates that the reward model is for candidate outputs. The score, This indicates that the reward model is for candidate outputs. The scoring here refers to selecting the better candidate output. For example, we can define that the better candidate output is scored as 1, and the other as scored as 0. The reward model selects the better candidate output based on the score value.

[0094] In another possible implementation, the loss function of the reward model is the log-likelihood loss. There are many related technologies that describe how to implement the log-likelihood loss, so details will not be elaborated here.

[0095] Once the reward model is trained, the better candidate output selected by the reward model will be close to the better candidate output selected by humans.

[0096] Step e: Based on the reward model, a proximal policy optimization algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

[0097] The process by which a multi-task risk assessment model generates a sequence based on a sequence of time-series data input into the training set can be regarded as an MDP (Markov Decision Process).

[0098] The MDP process includes: state ,action ,award And strategies.

[0099] Given time-series data as input to a sequence in a multi-task risk assessment model, the model generates a sequence. During the process of the multi-task risk assessment model generating a sequence, the state represents the sequence currently generated by the model. The action indicates that the multi-task risk assessment model selects the next generated token. ,award This refers to the cumulative reward obtained by evaluating the generated sequence using a reward model after the model has generated the sequence.

[0100] For example, if the multi-task risk assessment model ultimately generates a sequence of 10 tokens, the model generates them one by one, starting from the first token, until all 10 tokens are generated. During the generation of these 10 tokens, each token generation essentially changes the state once. Assuming the 8th token has been generated, the current state is... Then, the multi-task risk assessment model selects one token from multiple tokens as... After the action is executed, a state transition occurs, forming a new state. After all 10 tokens have been generated, a reward model is used to evaluate all 10 tokens as a whole to obtain the cumulative reward for the sequence.

[0101] A strategy comprises multiple strategy elements, each representing the probability of taking a particular action in a given state. These multiple strategy elements can represent all actions and probabilities of the multi-task risk assessment model under each state. Essentially, the strategy is the set of model parameters in the multi-task risk assessment model.

[0102] Before performing reinforcement learning, the initial policy is first defined as the set of model parameters in the multi-task risk assessment model after preliminary training in step 202. Then, the proximal policy optimization (PPO) algorithm can be used to iterate the multi-task risk assessment model multiple times.

[0103] In this embodiment of the disclosure, the optimization objective of the strategy of the multi-task risk assessment model is expressed by formula (4).

[0104] (4)

[0105] In formula (4), The updated multi-task risk assessment model's strategy, i.e., the current strategy, represents multiple parameters of the multi-task risk assessment model. The set, This is the initial strategy. This represents the operation of finding the maximum value, where the variables are multiple parameters in the multi-task risk assessment model. . Represents a reward model. This indicates that the multi-task risk assessment model is in strategy The expected reward is as follows As a weighting factor, For the current strategy Compared with the initial strategy The KL (Kullback-Leibler) divergence between the two sides can control the magnitude of policy updates and prevent the model from forgetting previous knowledge.

[0106] The optimization objective is to maximize the expected reward of the model's current policy while preventing the model from completely forgetting the content it was previously trained on.

[0107] Optionally, for the k-th iteration, step e includes the following four steps.

[0108] The first step is to generate multiple first sequences based on the first strategy of the multi-task risk assessment model.

[0109] The first strategy is the strategy of the multi-task risk assessment model in the (k-1)th iteration, which is the model parameter of the multi-task risk assessment model in the (k-1)th iteration.

[0110] During the k-th iteration of training, time-series data from multiple sequences in the training set are input into the multi-task risk assessment model. For each input sequence of time-series data, the multi-task risk assessment model generates a first sequence, thus generating multiple first sequences.

[0111] The second step is to evaluate multiple first sequences based on the reward model and obtain the cumulative reward for each first sequence.

[0112] The third step is to determine the dominance function for each time step based on the cumulative reward and value function of each first sequence.

[0113] The value function is determined based on the PPO algorithm. The PPO algorithm includes two neural networks that share the same underlying language model parameters. One neural network generates the policy, i.e., the multi-task risk assessment model, while the other neural network is the value function. The value function inherits the weights from the initial training of the multi-task risk assessment model during initialization. The value function is used to predict the value of a state.

[0114] A time step is the time when a token is generated, or the time when a state transition occurs.

[0115] Optionally, the advantage function can be expressed by formula (5).

[0116] (5)

[0117] In formula (5), Let the dominance function be defined at a given time step. The cumulative reward for the first sequence, Represents the value function in relation to the state Value rating, Represents the value function in relation to the state Value rating, status It is a state The next state, The discount factor is a pre-given empirical value.

[0118] In formula (5), before the first sequence is generated, the reward model cannot generate the cumulative reward of the first sequence. Therefore, in the time steps before the first sequence is generated... Obtained by fitting a multi-task risk assessment model; in the time step where the first sequence is generated, Generated by a reward model.

[0119] The fourth step is to update the first policy based on the loss function of the near-end policy optimization algorithm and the advantage function at each time step to complete the k-th iteration.

[0120] Optionally, the loss function of the PPO algorithm is expressed by formula (6).

[0121] (6)

[0122] In formula (6), Let PPO be the loss function. For the expectation, The probability of the policy in the (k-1)th iteration The probability of the policy in the k-th iteration The ratio of is expressed as . For the dominant function, For truncation function, it means The value of needs to meet the range . This is the shearing threshold, used to limit the magnitude of policy updates. As a weighting factor, First strategy Compared with the initial strategy The KL divergence between them, the initial strategy is the strategy of the multi-task risk assessment model before the first iteration.

[0123] By substituting the advantage function value at each time step into formula (6), the loss of the PPO algorithm in the kth generation can be calculated. Based on this loss, the first strategy of the multi-task risk assessment model can be optimized, thus completing the kth iteration. Here, k is a positive integer.

[0124] Repeating steps one through four allows for multiple iterations of training. Through these iterations, the multi-task risk assessment model continuously improves its multi-task performance during reinforcement learning. Furthermore, the introduction of KL divergence constraints ensures the model retains its existing knowledge, preventing the forgetting of previously trained content during reinforcement learning.

[0125] Optionally, after training, the performance of the multi-task risk assessment model also needs to be evaluated. This includes testing the model's task performance in different task scenarios, evaluating its performance on outlier detection, missing value imputation, and prediction tasks, and deploying the model in a real-world system environment to verify its practicality and stability in risk identification and early warning tasks.

[0126] In step 204, a risk assessment is performed on the offshore wind grid-connected system based on the trained multi-task risk assessment model.

[0127] In this embodiment, after the multi-task risk assessment model is trained, it preprocesses the time-series data obtained from the offshore wind grid-connected system by performing outlier detection and missing value imputation tasks. Then, by performing prediction tasks, the model predicts future time-series data of the offshore wind grid-connected system based on the preprocessed data, thereby enabling accurate risk assessment of the system based on this future time-series data. For example, a threshold can be set for each operating parameter; if the predicted time-series data for a certain operating parameter exceeds the threshold, it indicates that the operating parameter poses a risk.

[0128] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the above method embodiments.

[0129] Figure 3 This illustration shows a schematic diagram of a multi-task risk assessment device for a sea-wind grid-connected system based on human feedback reinforcement learning, provided in an exemplary embodiment of this disclosure. (See also...) Figure 3 The multi-task risk assessment device 300 for the sea wind grid-connected system based on human feedback reinforcement learning includes: an acquisition module 301, a modeling module 302, a training module 303, and a risk assessment module 304.

[0130] The acquisition module 301 is used to acquire training sets of multiple time series data based on the offshore wind grid-connected system. The training sets include a first sub-training set, a second sub-training set, and a third sub-training set. The first sub-training set is used to train the outlier detection task, the second sub-training set is used to train the missing value imputation task, and the third sub-training set is used to train the prediction task.

[0131] Modeling module 302 is used to establish a multi-task risk assessment model. The multi-task risk assessment model is a natural language model. The multi-task risk assessment model is used to perform outlier detection tasks, missing value imputation tasks, and prediction tasks. The output of the prediction task is used for risk assessment.

[0132] The training module 303 is used to train a multi-task risk assessment model using a training set. During the training of the multi-task risk assessment model, a human feedback reinforcement learning algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

[0133] The risk assessment module 304 is used to conduct risk assessments on the offshore wind grid-connected system based on the trained multi-task risk assessment model.

[0134] Optionally, the training module 303 is also used to train a reward model based on a human feedback dataset; and based on the reward model, a proximal policy optimization algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

[0135] Optionally, the training module 303 is further configured to generate multiple first sequences based on the first strategy of the multi-task risk assessment model, wherein the first strategy is the strategy of the multi-task risk assessment model in the (k-1)th iteration; evaluate the multiple first sequences based on the reward model to obtain the cumulative reward of each first sequence; determine the advantage function at each time step based on the cumulative reward and value function of each first sequence, wherein the value function is determined based on the proximal policy optimization algorithm; and update the first strategy based on the loss function of the proximal policy optimization algorithm and the advantage function at each time step to complete the kth iteration; wherein k is a positive integer.

[0136] Optionally, in training module 303, the loss function of the near-end policy optimization algorithm is expressed by the following formula:

[0137]

[0138] in, Let be the loss function of the near-end policy optimization algorithm. For the expectation, This is the ratio of the probability of the policy in the (k-1)th iteration to the probability of the policy in the kth iteration. For the dominant function, This is a truncation function. The shearing threshold, As a weighting factor, First strategy Compared with the initial strategy The KL divergence between them, the initial strategy is the strategy of the multi-task risk assessment model before the first iteration.

[0139] Optionally, in training module 303, the loss function of the reward model is determined based on the Bradley-Terry model; or, the loss function of the reward model is log-likelihood loss.

[0140] Optionally, the acquisition module 301 is also used to acquire a time-series dataset, which includes multiple time-series data of the offshore wind grid-connected system; perform downsampling processing on the multiple time-series data; and acquire a training set based on the downsampled time-series dataset.

[0141] It should be noted that the risk assessment of the multi-task risk assessment device for a wind-powered grid-connected system based on human feedback reinforcement learning provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the multi-task risk assessment device for a wind-powered grid-connected system based on human feedback reinforcement learning provided in the above embodiments and the multi-task risk assessment method for a wind-powered grid-connected system based on human feedback reinforcement learning belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0142] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in the various embodiments of this disclosure can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0143] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer, mobile phone, or communication device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. For example... Figure 4 As shown, the computer device 400 includes a processor 401 and a memory 402.

[0145] Processor 401 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0146] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 402 is used to store at least one instruction, which is executed by the processor 401 to implement the multi-task risk assessment method for a wind-driven grid-connected system based on human feedback reinforcement learning provided in this disclosure embodiment.

[0147] Those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on computer device 400, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0148] This disclosure also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of a computer device, the computer device is able to execute the multi-task risk assessment method for a wind-driven grid-connected system based on human feedback reinforcement learning provided in this disclosure.

[0149] This disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the multi-task risk assessment method for a wind-driven grid-connected system based on human feedback reinforcement learning provided in this disclosure.

[0150] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A multi-task risk assessment method for a wind-powered grid-connected system based on human feedback reinforcement learning, characterized in that, The method includes: Training sets are obtained based on multiple time-series data from the offshore wind grid-connected system. The training sets include a first sub-training set, a second sub-training set, and a third sub-training set. The first sub-training set is used to train the outlier detection task, the second sub-training set is used to train the missing value imputation task, and the third sub-training set is used to train the prediction task. A multi-task risk assessment model is established, and the multi-task risk assessment model is initially trained using a multi-task dataset. The multi-task risk assessment model is a natural language model. The multi-task risk assessment model is used to perform the outlier detection task, the missing value imputation task, and the prediction task. The output of the prediction task is used for risk assessment. The multi-task risk assessment model is trained using a training set. During the training process, a human feedback reinforcement learning algorithm is used to perform reinforcement learning on the multi-task risk assessment model. Based on the trained multi-task risk assessment model, a risk assessment is performed on the offshore wind grid connection system. The training set for acquiring multiple time-series data based on the offshore wind grid connection system includes: Obtain a time-series dataset, which includes multiple time-series data of the offshore wind grid-connected system; Downsampling processing is performed on multiple of the aforementioned time-series data; The training set is obtained based on the time-series dataset after downsampling.

2. The method according to claim 1, characterized in that, The reinforcement learning of the multi-task risk assessment model using a human feedback reinforcement learning algorithm includes: A reward model was trained based on a human feedback dataset. Based on the reward model, a near-end policy optimization algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

3. The method according to claim 2, characterized in that, The step of using a reward model-based, near-end policy optimization algorithm to perform reinforcement learning on the multi-task risk assessment model includes: Based on the first strategy of the multi-task risk assessment model, multiple first sequences are generated, where the first strategy is the strategy of the multi-task risk assessment model in the (k-1)th iteration. The first sequences are evaluated based on the reward model to obtain the cumulative reward for each first sequence; Based on the cumulative reward and value function of each of the first sequences, the advantage function for each time step is determined, wherein the value function is determined based on the proximal policy optimization algorithm; Based on the loss function of the near-end policy optimization algorithm and the advantage function at each time step, the first policy is updated to complete the k-th iteration; Where k is a positive integer.

4. The method according to claim 3, characterized in that, The loss function of the near-end policy optimization algorithm is expressed by the following formula: in, Let be the loss function of the near-end policy optimization algorithm. For the expectation, This is the ratio of the probability of the policy in the (k-1)th iteration to the probability of the policy in the kth iteration. For the aforementioned advantage function, This is a truncation function. The shearing threshold, As a weighting factor, For the first strategy Compared with the initial strategy The KL divergence between them, the initial strategy is the strategy of the multi-task risk assessment model before the first iteration.

5. The method according to any one of claims 2 to 4, characterized in that, The loss function of the reward model is determined based on the Bradley-Terry model; or, The loss function of the reward model is log-likelihood loss.

6. A multi-task risk assessment device for a sea-wind grid-connected system based on human feedback reinforcement learning, characterized in that, The device includes: The acquisition module is used to acquire a training set based on multiple time series data of the offshore wind grid-connected system. The training set includes a first sub-training set, a second sub-training set, and a third sub-training set. The first sub-training set is used to train the outlier detection task, the second sub-training set is used to train the missing value imputation task, and the third sub-training set is used to train the prediction task. The modeling module is used to establish a multi-task risk assessment model. The multi-task risk assessment model is initially trained using a multi-task dataset. The multi-task risk assessment model is a natural language model. The multi-task risk assessment model is used to perform the outlier detection task, the missing value imputation task, and the prediction task. The output of the prediction task is used for risk assessment. The training module is used to train the multi-task risk assessment model using a training set. During the training of the multi-task risk assessment model, a human feedback reinforcement learning algorithm is used to perform reinforcement learning on the multi-task risk assessment model. The risk assessment module is used to conduct a risk assessment of the offshore wind grid connection system based on the trained multi-task risk assessment model. The acquisition module is also used to acquire a time-series dataset, which includes multiple time-series data of the offshore wind grid-connected system; Downsampling processing is performed on multiple of the aforementioned time-series data; The training set is obtained based on the time-series dataset after downsampling.

7. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method of any one of claims 1 to 5.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images