Multi-task risk assessment method for sea wind grid-connected system based on human feedback reinforcement learning

Abstract

The invention discloses a multi-task risk assessment method for a sea wind grid-connected system based on human feedback reinforcement learning, and belongs to the technical field of power grids, and the method comprises the steps: obtaining a training set based on a plurality of time sequence data of the sea wind grid-connected system, the training set comprising a first sub-training set, a second sub-training set and a third sub-training set, the first sub-training set is used for training an abnormal value detection task, the second sub-training set is used for training a missing value filling task, the third sub-training set is used for training a prediction task, and the output of the prediction task is used for risk assessment; establishing a multi-task risk assessment model; training a multi-task risk assessment model by adopting the training set, and performing reinforcement learning on the multi-task risk assessment model by adopting a human feedback reinforcement learning algorithm in the process of training the multi-task risk assessment model; and performing risk assessment based on the trained multi-task risk assessment model. According to the method, the accuracy degree of risk assessment of the model under the multi-task sea wind grid-connected system can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of electric power grids, and in particular to a multi-task risk assessment method for an offshore wind grid-connected system based on human feedback reinforcement learning. Background Art

[0002] The offshore wind grid-connected system is also the management system for offshore wind power grid-connected. Offshore wind power grid-connected refers to large-scale wind turbines that are directly connected to the grid at sea. With the continuous increase in the scale of offshore wind power grid-connected access and the increasingly complex operating environment, the amount of data that the offshore wind grid-connected system needs to process is also increasing. The risk assessment process of the offshore wind grid-connected system usually includes: establishing multiple tasks to process the acquired data, and conducting risk assessment based on the results of data processing.

[0003] In the related art, the risk assessment method includes: establishing a risk assessment model, and using a task to train the risk assessment model to achieve risk assessment.

[0004] However, modern wind power grid-connected systems have problems with large data processing volumes and many tasks to be processed. When a risk assessment model trained with a single task is used to perform risk assessment in a multi-task offshore wind grid-connected system, the accuracy of risk assessment is low, especially under conditions such as large-scale unstable new energy access and large fluctuations in power load. Model failure or inaccurate risk assessment is more likely to occur. Summary of the invention

[0005] The present disclosure provides a multi-task risk assessment method for an offshore wind grid-connected system based on human feedback reinforcement learning, which can improve the accuracy of risk assessment in an offshore wind grid-connected system with a multi-task model. The technical solution includes at least the following solutions: In a first aspect, a multi-task risk assessment method for an offshore wind grid-connected system based on human feedback reinforcement learning is provided, comprising: obtaining a training set based on multiple time series data of the offshore wind grid-connected system, the training set comprising a first sub-training set, a second sub-training set and a third sub-training set, the first sub-training set being used to train an outlier detection task, the second sub-training set being used to train a missing value filling task, and the third sub-training set being used to train a prediction task; establishing a multi-task risk assessment model, the multi-task risk assessment model being a natural language model, the multi-task risk assessment model being used to perform the outlier detection task, the missing value filling task and the prediction task, and the output of the prediction task being used to perform risk assessment; training the multi-task risk assessment model with the training set, and in the process of training the multi-task risk assessment model, performing reinforcement learning on the multi-task risk assessment model with a human feedback reinforcement learning algorithm; and performing risk assessment on the offshore wind grid-connected system based on the trained multi-task risk assessment model.

[0006] Optionally, the use of a human feedback reinforcement learning algorithm to perform reinforcement learning on the multi-task risk assessment model includes: training a reward model based on a human feedback data set; and based on the reward model, using a proximal strategy optimization algorithm to perform reinforcement learning on the multi-task risk assessment model.

[0007] Optionally, the reward model-based, proximal strategy optimization algorithm is used to perform reinforcement learning on the multi-task risk assessment model, including: based on the first strategy of the multi-task risk assessment model, generating multiple first sequences, the first strategy is the strategy of the multi-task risk assessment model in the k-1th iteration; based on the reward model, evaluating the multiple first sequences to obtain the cumulative reward of each first sequence; based on the cumulative reward and value function of each first sequence, determining the advantage function of each time step, the value function is determined based on the proximal strategy optimization algorithm; based on the loss function of the proximal strategy optimization algorithm and the advantage function of each time step, updating the first strategy to complete the kth iteration; wherein k is a positive integer.

[0008] Optionally, the loss function of the proximal strategy optimization algorithm is expressed by the following formula:

[0009] in, is the loss function of the proximal strategy optimization algorithm, For expectations, is the ratio of the probability of the strategy in the k-1th iteration to the probability of the strategy in the kth iteration, is the advantage function, is the truncation function, is the clipping threshold, is the trade-off coefficient, The first strategy With the initial strategy The KL divergence between , the initial strategy is the strategy of the multi-task risk assessment model before the first iteration.

[0010] Optionally, the loss function of the reward model is determined based on a Bradley-Terry model; or, the loss function of the reward model is a log-likelihood loss.

[0011] Optionally, the method of acquiring a training set based on multiple time series data of an offshore wind grid-connected system comprises: acquiring a time series data set, wherein the time series data set includes multiple time series data of the offshore wind grid-connected system; downsampling the multiple time series data; and acquiring the training set based on the time series data set after the downsampling process.

[0012] In the second aspect, a multi-task risk assessment device for an offshore wind grid-connected system based on human feedback reinforcement learning is also provided, including: an acquisition module, used to acquire a training set based on multiple time series data of the offshore wind grid-connected system, the training set including a first sub-training set, a second sub-training set and a third sub-training set, the first sub-training set is used to train an outlier detection task, the second sub-training set is used to train a missing value filling task, and the third sub-training set is used to train a prediction task; a modeling module, used to establish a multi-task risk assessment model, the multi-task risk assessment model is a natural language model, the multi-task risk assessment model is used to perform the outlier detection task, the missing value filling task and the prediction task, and the output of the prediction task is used to perform risk assessment; a training module, used to train the multi-task risk assessment model using a training set, and in the process of training the multi-task risk assessment model, a human feedback reinforcement learning algorithm is used to perform reinforcement learning on the multi-task risk assessment model; a risk assessment module, used to perform risk assessment on the offshore wind grid-connected system based on the trained multi-task risk assessment model.

[0013] Optionally, the training module is also used to train a reward model based on a human feedback data set; based on the reward model, a proximal policy optimization algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

[0014] Optionally, the training module is also used to generate multiple first sequences based on the first strategy of the multi-task risk assessment model, where the first strategy is the strategy of the multi-task risk assessment model in the k-1th iteration; evaluate the multiple first sequences based on the reward model to obtain the cumulative reward of each first sequence; determine the advantage function of each time step based on the cumulative reward and value function of each first sequence, and the value function is determined based on the proximal strategy optimization algorithm; update the first strategy based on the loss function of the proximal strategy optimization algorithm and the advantage function of each time step to complete the kth iteration; wherein k is a positive integer.

[0015] Optionally, in the training module, the loss function of the proximal strategy optimization algorithm is expressed using the following formula:

[0016] in, is the loss function of the proximal strategy optimization algorithm, For expectations, is the ratio of the probability of the strategy in the k-1th iteration to the probability of the strategy in the kth iteration, is the advantage function, is the truncation function, is the clipping threshold, is the trade-off coefficient, The first strategy With the initial strategy The KL divergence between , the initial strategy is the strategy of the multi-task risk assessment model before the first iteration.

[0017] Optionally, in the training module, the loss function of the reward model is determined based on a Bradley-Terry model; or, the loss function of the reward model is a log-likelihood loss.

[0018] Optionally, the acquisition module is also used to acquire a time series data set, which includes a plurality of the time series data of the sea wind grid-connected system; downsample the plurality of the time series data; and acquire the training set based on the time series data set after the downsampling process.

[0019] In a third aspect, a computer device is also provided, including: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor, so as to execute the multi-task risk assessment method of the offshore wind grid-connected system based on human feedback reinforcement learning described in the above embodiment.

[0020] In a fourth aspect, a computer-readable storage medium is also provided, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor, so as to execute the multi-task risk assessment method of offshore wind grid-connected system based on human feedback reinforcement learning described in the above embodiment.

[0021] In a fifth aspect, a computer program product is provided, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method described in the first aspect is implemented.

[0022] The beneficial effects brought by the technical solution provided by the embodiments of the present disclosure include at least: In the disclosed embodiment, by adopting human feedback reinforcement learning to train the multi-task risk assessment model, the multi-task execution capability of the multi-task risk assessment model can be improved, and the performance of the multi-task risk assessment model in outlier detection tasks, missing value filling tasks and prediction tasks can be finally optimized, so that it can accurately perform multiple tasks and quickly adapt to changes in different scenarios.

[0023] After the training of the multi-task risk assessment model is completed, the multi-task risk assessment model can preprocess the time series data obtained from the offshore wind grid-connected system by performing outlier detection tasks and missing value filling tasks; the multi-task risk assessment model can predict the future time series data of the offshore wind grid-connected system based on the preprocessed time series data by performing prediction tasks, so as to perform a stable and accurate risk assessment of the offshore wind grid-connected system based on the future time series data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A flowchart of a multi-task risk assessment method for an offshore wind grid-connected system based on human feedback reinforcement learning provided by an exemplary embodiment of the present disclosure is shown; Figure 2 A flowchart of a multi-task risk assessment method for an offshore wind grid-connected system based on human feedback reinforcement learning provided by another exemplary embodiment of the present disclosure is shown; Figure 3 A schematic structural diagram of a multi-task risk assessment device for an offshore wind grid-connected system based on human feedback reinforcement learning provided by an exemplary embodiment of the present disclosure is shown; Figure 4 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Unless otherwise defined, the technical or scientific terms used herein shall have the usual meanings understood by persons of ordinary skill in the field to which the present disclosure belongs. The words "first", "second", "third" and similar words used in the patent application specification and claims of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not indicate a quantitative limitation, but rather indicate the presence of at least one. Words such as "include" or "comprise" and similar words mean that the elements or objects appearing before "include" or "comprises" cover the elements or objects listed after "include" or "comprises" and their equivalents, and do not exclude other elements or objects.

[0027] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0028] Figure 1 A flowchart of a multi-task risk assessment method for an offshore wind grid-connected system based on human feedback reinforcement learning provided by an exemplary embodiment of the present disclosure is shown, and the method can be executed by a computer device. Figure 1 , the method comprising: In step 101, a training set is obtained based on a plurality of time series data of an offshore wind grid-connected system.

[0029] The training set includes a first sub-training set, a second sub-training set and a third sub-training set. The first sub-training set is used to train the outlier detection task, the second sub-training set is used to train the missing value filling task, and the third sub-training set is used to train the prediction task.

[0030] Optionally, there is a first prompt word before any time series data in the first sub-training set, and the first prompt word is used to instruct the multi-task risk assessment model to perform an outlier detection task.

[0031] There is a second prompt word before any time series data in the second sub-training set, and the second prompt word is used to instruct the multi-task risk assessment model to perform a missing value filling task.

[0032] There is a third prompt word before any time series data in the third sub-training set, and the third prompt word is used to instruct the multi-task risk assessment model to perform a prediction task.

[0033] In the disclosed embodiment, the first prompt word, the second prompt word and the third prompt word are used to distinguish the three tasks. When the subsequent multi-task risk assessment model performs multiple tasks, the model can understand which task needs to be performed on the subsequent time series data through the prompt words before the time series data.

[0034] For example, the first prompt word is , the time series data in the first sub-training set is as follows: In this sequence, Indicates the start marker of time series data. Indicates the end marker of time series data. is a separator, used to separate questions and answers; is an outlier marker symbol, which is implemented through manual annotation. During the training process, This is equivalent to the problem of inputting into a multi-task risk assessment model. This is equivalent to the answer input to the multi-task risk assessment model.

[0035] During the training process, questions and answers are simultaneously input into the multi-task risk assessment model. After the training is completed, only questions without outlier markers need to be input into the multi-task risk assessment model. For the outlier marker task, the questions input into the multi-task risk assessment model after training are, for example, ; After receiving the question, the multi-task risk assessment model will use the outlier marker to mark the outliers in the question. The output of the model is, for example, .

[0036] The essence of using the first sub-training set to train the multi-task risk assessment model is to train the multi-task risk assessment model to accurately use the outlier marker symbol. After the training is completed, the multi-task risk assessment model can accurately use the outlier marker symbol to mark the outliers in the time series data.

[0037] For example, the second prompt word is , the time series data in the second sub-training set is as follows: In this sequence, Mark missing values ​​with symbols. To fill in the missing values, the missing value markers and the filled missing values ​​are manually labeled. During the training process, This is equivalent to the problem of inputting into a multi-task risk assessment model. This is equivalent to the answer input to the multi-task risk assessment model.

[0038] During the training process, questions and answers are simultaneously input into the multi-task risk assessment model. After the training is completed, only questions without missing value markers need to be input into the multi-task risk assessment model. For the missing value filling task, the questions input into the multi-task risk assessment model after training are, for example, After receiving the question, the multi-task risk assessment model will identify the missing values ​​in the question and fill in the corresponding missing values. The output of the model is, for example, .

[0039] The essence of using the second sub-training set to train the multi-task risk assessment model is to train the multi-task risk assessment model to accurately identify missing values ​​and accurately fill in missing values. After the training is completed, the multi-task risk assessment model can accurately identify missing values ​​in time series data and accurately fill in the missing values.

[0040] For example, the third prompt word is , the time series data in the third sub-training set is as follows: In this sequence, the previous time series data is the question of the next time series data, and the next time series data is the answer to the previous time series data. For example, It is a prediction problem, and the predicted answer is the sequence in The value of the next moment at the current moment is When using the time series data in the third sub-training set for training, the answer can be set as a mask first, and then the mask can be removed after the model outputs the prediction result. For example, when the sequence is input into the multi-task risk assessment model, The part of is set as a mask, and the model is based on Output predicted After the next value of , the model can be based on unmasking predict The next mask is added, and so on until all the data in the sequence are unmasked.

[0041] For the prediction task, after training is completed, only the third prompt word and the data to be predicted need to be input into the multi-task risk assessment model. The model can predict the next value of the last value of the input data, and the next value of the last value is the predicted value. For example, the sequence of a prediction task input into the multi-task risk assessment model is , then the output of the multi-task risk assessment model is ,in This is the predicted value predicted by the model.

[0042] The essence of using the third sub-training set to train the multi-task risk assessment model is to train the multi-task risk assessment model's ability to accurately predict.

[0043] The above-mentioned prompt words, missing value markers and outlier markers can be defined as fixed category data, and these fixed category data do not need to be regressed during the processing process. Usually, the time series data obtained from the offshore wind grid-connected system is continuous and high-precision data. The higher the accuracy of the data, the more data categories there are in the subsequent encoding process. If the time series data obtained from the offshore wind grid-connected system is directly input into the model, the probability of each category will be infinitely close to 0 due to too many categories. Therefore, it is necessary to downsample the acquired time series data first to reduce the accuracy of the time series data, reduce the categories of the time series data, and classify the time series data at the same time.

[0044] Optionally, the method comprises the following steps ac.

[0045] Step a: Obtain a time series data set.

[0046] The time series data set includes multiple time series data of the offshore wind grid-connected system. The time series data set can be expressed as ,in, For the Some kind of time series data at a certain moment.

[0047] Optionally, the multiple time series data of the offshore wind grid-connected system include operating parameters such as power output, wind speed, and equipment status.

[0048] Step b: down-sample the multiple time series data.

[0049] Optionally, formula (1) is used to implement downsampling processing.

[0050] (1) In formula (1), For the Some kind of time series data at a certain moment, For After downsampling, is the maximum value in the time series data set. is the minimum value in the time series data set, Indicates rounding down. It is a discrete constant, and its value can be determined based on experience, usually taking a very small value. is a discrete series, and usually the value of the discrete series is equal to the value of the category of the data in the time series data set after downsampling. For example, if the discrete series is 3, then the category of the data in the time series data set is also 3. The value of the discrete series is an empirical value, and the embodiments of the present disclosure do not limit this.

[0051] Through downsampling, the data in the time series data set can be classified. When the subsequent multi-task risk assessment model makes a prediction, the predicted value obtained is not a specific value but a category.

[0052] Step c: obtaining a training set based on the time series data set after downsampling.

[0053] After downsampling multiple time series data in the time series data set, the data in the time series data set can be divided into three subsets, and the data in the three subsets are manually labeled according to the formats of the first sub-training set, the second sub-training set and the third sub-training set, so as to obtain the first sub-training set, the second sub-training set and the third sub-training set. The first sub-training set, the second sub-training set and the third sub-training set constitute the training set.

[0054] In step 102, a multi-task risk assessment model is established.

[0055] The multi-task risk assessment model is a natural language model. The multi-task risk assessment model is used to perform outlier detection tasks, missing value filling tasks, and prediction tasks. The output of the prediction task is used for risk assessment. In step 103, the multi-task risk assessment model is trained using the training set.

[0056] In the process of training the multi-task risk assessment model, the Reinforcement Learning from Human Feedback (RLHF) algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

[0057] In step 104, a risk assessment of the offshore wind grid-connected system is performed based on the trained multi-task risk assessment model.

[0058] In the disclosed embodiment, by adopting human feedback reinforcement learning to train the multi-task risk assessment model, the multi-task execution capability of the multi-task risk assessment model can be improved, and the performance of the multi-task risk assessment model in outlier detection tasks, missing value filling tasks and prediction tasks can be finally optimized, so that it can accurately perform multiple tasks and quickly adapt to changes in different scenarios.

[0059] After the training of the multi-task risk assessment model is completed, the multi-task risk assessment model can preprocess the time series data obtained from the offshore wind grid-connected system by performing outlier detection tasks and missing value filling tasks; the multi-task risk assessment model can predict the future time series data of the offshore wind grid-connected system based on the preprocessed time series data by performing prediction tasks, so as to perform a stable and accurate risk assessment of the offshore wind grid-connected system based on the future time series data.

[0060] Figure 2 A flowchart of a multi-task risk assessment method for an offshore wind grid-connected system based on human feedback reinforcement learning provided by another exemplary embodiment of the present disclosure is shown, and the method can be executed by a computer device. Figure 2 , the method comprising: In step 201, a training set is obtained based on a plurality of time series data of an offshore wind grid-connected system.

[0061] The training set includes a first sub-training set, a second sub-training set and a third sub-training set. The first sub-training set is used to train the outlier detection task, the second sub-training set is used to train the missing value filling task, and the third sub-training set is used to train the prediction task.

[0062] The relevant contents of step 201 refer to the aforementioned step 101, and the detailed description is omitted here.

[0063] In step 202, a multi-task risk assessment model is established.

[0064] The multi-task risk assessment model is a natural language model. The multi-task risk assessment model is used to perform outlier detection tasks, missing value filling tasks and prediction tasks, where the output of the prediction task is used for risk assessment.

[0065] Exemplarily, the multi-task risk assessment model established in step 202 is a pre-trained natural language model, such as a GPT (Generative Pre-Training) model, a BERT (Bidirectional Encoder Representations from Transformers) model, a LLaMA (Large Language Model Meta AI) model, etc. The multi-task risk assessment model can be expressed as , indicating that the goal of the multi-task risk assessment model is to Predict the next token in the input sequence .

[0066] Optionally, before executing step 203, the method further includes: using a multi-task data set to perform preliminary training on the multi-task risk assessment model in step 202. Here, the data in the multi-task data set also needs to be processed in a similar manner to the data in the training set before it can be used to perform preliminary training on the multi-task risk assessment model. The data in the multi-task data set can be any type of multi-task data, which is not limited in the embodiments of the present disclosure.

[0067] For example, when the multi-task risk assessment model is initially trained using data in the multi-task dataset, the loss function may be a cross entropy loss, which is represented by formula (2).

[0068] (2) In formula (2), Represents a token in the sequence that is input to the multi-task risk assessment model during the initial training process.

[0069] In step 203, the multi-task risk assessment model is trained using the training set.

[0070] In the process of training the multi-task risk assessment model, a human feedback reinforcement learning algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

[0071] Optionally, step 203 includes the following steps:

[0072] Step d: Train the reward model based on the human feedback dataset Optionally, step d includes: obtaining a human feedback dataset; establishing a reward model; and training the reward model using the human feedback dataset.

[0073] The data in the human feedback dataset reflects human preferences. For example, with the participation of human evaluators, the different output results generated by a model under the same input can be compared pairwise to collect human preference data. For example, for the same input, a model generates two candidate outputs s1 and s2, and humans choose the better one, which constitutes a data in the human feedback dataset.

[0074] The reward model is, for example, a neural network. Using a human feedback dataset to train the reward model can enable the reward model to learn human preferences and provide reward signals for subsequent reinforcement learning optimization.

[0075] In a possible implementation, the loss function of the reward model is determined based on the Bradley-Terry model. In this case, the loss function of the reward model is expressed using formula (3).

[0076] (3) In formula (3), is the loss function of the reward model, , represents two candidate outputs generated by a model given the same input. Indicates that in human preference, the candidate output Better than candidate output . Represents the reward model for candidate output The rating, Represents the reward model for candidate output Scoring here refers to selecting which candidate output is better. For example, you can define two candidate outputs, the better one is scored as 1, and the other is scored as 0. The reward model determines which candidate output is selected as the better candidate output by the score value.

[0077] In another possible implementation, the loss function of the reward model is log-likelihood loss. There are many implementation methods of log-likelihood loss in the related art, and detailed description is omitted here.

[0078] After the reward model is trained, the better candidate outputs selected by the reward model are close to the better candidate outputs selected by humans.

[0079] In step e, based on the reward model, a proximal policy optimization algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

[0080] The process of generating a sequence of time series data from a sequence of input in the training set can be regarded as MDP (Markov Decision Process).

[0081] The MDP process includes: ,action ,award and strategies.

[0082] For a sequence of time series data input into the multi-task risk assessment model, the model generates a sequence. In the process of the multi-task risk assessment model generating a sequence, the state represents the sequence currently generated by the multi-task risk assessment model. , the action represents the next token generated by the multi-task risk assessment model ,award It is the cumulative reward obtained by using the reward model to evaluate the sequence generated by the model after the model generates the sequence.

[0083] For example, if the sequence length generated by the multi-task risk assessment model is 10 tokens, the multi-task risk assessment model generates tokens one by one from the first token until 10 tokens are generated. In the process of generating these 10 tokens, each token generated is equivalent to a change in state. Assuming that the 8th token has been generated, the state at this time is , then the action of the multi-task risk assessment model is to select a token from multiple tokens as After the action is executed, a state transition occurs, forming a new state After all 10 tokens are generated, the reward model is used to evaluate the 10 tokens as a whole to obtain the cumulative reward of the sequence.

[0084] The strategy includes multiple strategy elements, and any strategy element represents the probability of taking a certain action in a certain state. Multiple strategy elements in the strategy can represent all actions and probabilities of the multi-task risk assessment model in each state. The essence of the strategy is the set of model parameters in the multi-task risk assessment model.

[0085] Before reinforcement learning is performed, the initial strategy is first defined as a set of model parameters in the multi-task risk assessment model after preliminary training in step 202. Then, the multi-task risk assessment model may be iterated multiple times using a proximal policy optimization (PPO) algorithm.

[0086] In the disclosed embodiment, the optimization target of the strategy of the multi-task risk assessment model is expressed by formula (4).

[0087] (4) In formula (4), is the updated strategy of the multi-task risk assessment model, that is, the current strategy, which represents multiple parameters of the multi-task risk assessment model A collection of is the initial strategy. Represents the maximum value operation, and the variables are multiple parameters in the multi-task risk assessment model . represents the reward model, Represents the multi-task risk assessment model in strategy The expected reward under is the trade-off coefficient, For current strategy With the initial strategy The KL (Kullback-Leibler) divergence between them can control the update amplitude of the strategy and prevent the model from forgetting previous knowledge.

[0088] This optimization objective is used to maximize the expected reward of the model's current strategy while preventing the model from completely forgetting what it had previously trained.

[0089] Optionally, for the kth iteration, step e includes the following four steps.

[0090] In the first step, a plurality of first sequences are generated based on the first strategy of the multi-task risk assessment model.

[0091] The first strategy is the strategy of the multi-task risk assessment model in the k-1th iteration, that is, the model parameters of the multi-task risk assessment model in the k-1th iteration.

[0092] In the kth iteration training process, multiple sequences of time series data in the training set are input into the multi-task risk assessment model. Each time a sequence of time series data is input, the multi-task risk assessment model can generate a first sequence, thereby generating multiple first sequences.

[0093] In the second step, multiple first sequences are evaluated based on the reward model to obtain the cumulative reward of each first sequence.

[0094] In the third step, based on the cumulative reward and value function of each first sequence, the advantage function of each time step is determined.

[0095] The value function is determined based on the PPO algorithm. The PPO algorithm includes two neural networks that share the same basic language model parameters. One neural network is used to generate strategies, that is, the multi-task risk assessment model, and the other neural network is the value function. The value function inherits the weights of the initial training of the multi-task risk assessment model during initialization. The value function is used to predict the value of the state.

[0096] A time step is the time when a token is generated, or the time when a state transfer occurs.

[0097] Optionally, the advantage function is expressed using formula (5).

[0098] (5) In formula (5), is the advantage function at a certain time step, is the cumulative reward of the first sequence, Represents the value function of the state The value rating of Represents the value function of the state Value rating, status Yes Status The next state, is the discount factor, which is a pre-given empirical value.

[0099] In formula (5), before the first sequence is generated, the reward model cannot generate the cumulative reward of the first sequence. Then, in the time step before the first sequence is generated, It is obtained by fitting the multi-task risk assessment model; in the time step when the first sequence is generated, Generated by the reward model.

[0100] In the fourth step, based on the loss function of the proximal policy optimization algorithm and the advantage function of each time step, the first policy is updated to complete the kth iteration.

[0101] Optionally, the loss function of the PPO algorithm is expressed using formula (6).

[0102] (6) In formula (6), is the loss function of the PPO algorithm, For expectations, is the probability of the strategy in the k-1th iteration The probability of the strategy in the kth iteration The ratio of . is the advantage function, is the truncation function, indicating The value must satisfy the range . is the clipping threshold, which is used to limit the amplitude of the policy update. is the trade-off coefficient, First strategy With the initial strategy The KL divergence between , the initial strategy is the strategy of the multi-task risk assessment model before the first iteration.

[0103] Substituting the advantage function value of each time step into formula (6), the loss of the PPO algorithm in the kth generation can be calculated. Based on this loss, the first strategy of the multi-task risk assessment model can be optimized, thereby completing the kth iteration. Where k is a positive integer.

[0104] Repeating the first to fourth steps above can perform multiple iterative training. Through multiple iterative training, the multi-task risk assessment model continuously improves its multi-task execution capabilities during the reinforcement learning process. At the same time, due to the introduction of the KL divergence constraint, the multi-task risk assessment model can maintain the original knowledge and avoid forgetting the previously trained content during the reinforcement learning process.

[0105] Optionally, after training is completed, the performance of the multi-task risk assessment model needs to be evaluated. The method includes: testing the task execution effect of the model in different task scenarios, evaluating its performance on outlier detection tasks, missing value filling tasks, and prediction tasks. Deploy the model in the actual system environment to verify its practicality and stability in risk identification and early warning tasks.

[0106] In step 204, risk assessment is performed on the offshore wind grid-connected system based on the trained multi-task risk assessment model.

[0107] In the disclosed embodiment, after the multi-task risk assessment model is trained, the multi-task risk assessment model can pre-process the time series data obtained from the sea wind grid-connected system by performing outlier detection tasks and missing value filling tasks; the multi-task risk assessment model can predict the future time series data of the sea wind grid-connected system based on the pre-processed time series data by performing prediction tasks, so that the sea wind grid-connected system can be accurately risk assessed based on the future time series data. For example, a threshold value can be set for each operating parameter. If the time series data predicted by the prediction task for a certain operating parameter is greater than the threshold value, it means that the operating parameter is at risk.

[0108] The following are device embodiments of the present application. For details not described in detail in the device embodiments, reference may be made to the above method embodiments.

[0109] Figure 3A schematic diagram of a multi-task risk assessment device for an offshore wind grid-connected system based on human feedback reinforcement learning provided by an exemplary embodiment of the present disclosure is shown. Figure 3 The multi-task risk assessment device 300 for an offshore wind grid-connected system based on human feedback reinforcement learning includes: an acquisition module 301 , a modeling module 302 , a training module 303 and a risk assessment module 304 .

[0110] The acquisition module 301 is used to acquire a training set based on multiple time series data of the offshore wind grid-connected system. The training set includes a first sub-training set, a second sub-training set and a third sub-training set. The first sub-training set is used to train an outlier detection task, the second sub-training set is used to train a missing value filling task, and the third sub-training set is used to train a prediction task.

[0111] The modeling module 302 is used to establish a multi-task risk assessment model. The multi-task risk assessment model is a natural language model. The multi-task risk assessment model is used to perform outlier detection tasks, missing value filling tasks and prediction tasks. The output of the prediction task is used for risk assessment.

[0112] The training module 303 is used to train the multi-task risk assessment model using a training set. During the training of the multi-task risk assessment model, a human feedback reinforcement learning algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

[0113] The risk assessment module 304 is used to perform risk assessment on the offshore wind grid-connected system based on the trained multi-task risk assessment model.

[0114] Optionally, the training module 303 is further used to train a reward model based on a human feedback data set; based on the reward model, a proximal policy optimization algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

[0115] Optionally, the training module 303 is also used to generate multiple first sequences based on the first strategy of the multi-task risk assessment model, where the first strategy is the strategy of the multi-task risk assessment model in the k-1th iteration; evaluate the multiple first sequences based on the reward model to obtain the cumulative reward of each first sequence; determine the advantage function of each time step based on the cumulative reward and value function of each first sequence, and the value function is determined based on the proximal strategy optimization algorithm; update the first strategy based on the loss function of the proximal strategy optimization algorithm and the advantage function of each time step to complete the kth iteration; wherein k is a positive integer.

[0116] Optionally, in the training module 303, the loss function of the proximal strategy optimization algorithm is expressed by the following formula:

[0117] in, is the loss function of the proximal policy optimization algorithm, For expectations, is the ratio of the probability of the strategy in the k-1th iteration to the probability of the strategy in the kth iteration, is the advantage function, is the truncation function, is the clipping threshold, is the trade-off coefficient, First strategy With the initial strategy The KL divergence between , the initial strategy is the strategy of the multi-task risk assessment model before the first iteration.

[0118] Optionally, in the training module 303, the loss function of the reward model is determined based on the Bradley-Terry model; or, the loss function of the reward model is log-likelihood loss.

[0119] Optionally, the acquisition module 301 is further used to acquire a time series data set, which includes multiple time series data of the offshore wind grid-connected system; downsample the multiple time series data; and acquire a training set based on the downsampled time series data set.

[0120] It should be noted that: when the multi-task risk assessment device for sea wind grid-connected system based on human feedback reinforcement learning provided in the above embodiment performs risk assessment, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the multi-task risk assessment device for sea wind grid-connected system based on human feedback reinforcement learning provided in the above embodiment and the multi-task risk assessment method embodiment for sea wind grid-connected system based on human feedback reinforcement learning belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0121] The division of modules in the embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present disclosure may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0122] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a terminal device (which can be a personal computer, mobile phone, or communication device, etc.) or a processor (processor) to execute all or part of the steps of the method of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0123] Figure 4 Schematic diagram of the structure of the computer device provided by the embodiment of the present disclosure. Figure 4 As shown, the computer device 400 includes: a processor 401 and a memory 402 .

[0124] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0125] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the multi-task risk assessment method for the sea wind grid-connected system based on human feedback reinforcement learning provided in the embodiment of the present disclosure.

[0126] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the computer device 400, and the computer device 400 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0127] The embodiment of the present disclosure also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of a computer device, the computer device is enabled to execute the multi-task risk assessment method for offshore wind grid-connected systems based on human feedback reinforcement learning provided in the embodiment of the present disclosure.

[0128] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the multi-task risk assessment method for an offshore wind grid-connected system based on human feedback reinforcement learning provided in the embodiments of the present disclosure.

[0129] The above description is only an optional embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multi-task risk assessment method for offshore wind grid-connected systems based on human feedback reinforcement learning, characterized in that: The method comprises: A training set is obtained based on multiple time series data of the sea wind grid-connected system, wherein the training set includes a first sub-training set, a second sub-training set and a third sub-training set, wherein the first sub-training set is used for training an outlier detection task, the second sub-training set is used for training a missing value filling task, and the third sub-training set is used for training a prediction task; Establishing a multi-task risk assessment model, wherein the multi-task risk assessment model is a natural language model, and the multi-task risk assessment model is used to perform the outlier detection task, the missing value filling task, and the prediction task, and the output of the prediction task is used to perform risk assessment; The multi-task risk assessment model is trained using a training set, and during the training of the multi-task risk assessment model, a human feedback reinforcement learning algorithm is used to perform reinforcement learning on the multi-task risk assessment model; Based on the trained multi-task risk assessment model, a risk assessment is performed on the offshore wind grid-connected system.

2. The method according to claim 1, characterized in that The use of a human feedback reinforcement learning algorithm to perform reinforcement learning on the multi-task risk assessment model includes: Train a reward model based on a human feedback dataset; Based on the reward model, a proximal strategy optimization algorithm is used to perform reinforcement learning on the multi-task risk assessment model.

3. The method according to claim 2, characterized in that The reward model-based method uses a proximal strategy optimization algorithm to perform reinforcement learning on the multi-task risk assessment model, including: Generate a plurality of first sequences based on a first strategy of the multi-task risk assessment model, wherein the first strategy is a strategy of the multi-task risk assessment model in a k-1th iteration; Evaluate the plurality of first sequences based on the reward model to obtain a cumulative reward for each of the first sequences; Determining an advantage function for each time step based on the cumulative reward and a value function of each of the first sequences, wherein the value function is determined based on the proximal policy optimization algorithm; Based on the loss function of the proximal policy optimization algorithm and the advantage function at each time step, updating the first policy to complete a kth iteration; Wherein, k is a positive integer.

4. The method according to claim 3, characterized in that The loss function of the proximal strategy optimization algorithm is expressed by the following formula: in, is the loss function of the proximal strategy optimization algorithm, For expectations, is the ratio of the probability of the strategy in the k-1th iteration to the probability of the strategy in the kth iteration, is the advantage function, is the truncation function, is the clipping threshold, is the trade-off coefficient, The first strategy With the initial strategy The KL divergence between , the initial strategy is the strategy of the multi-task risk assessment model before the first iteration.

5. The method according to any one of claims 2 to 4, characterized in that The loss function of the reward model is determined based on the Bradley-Terry model; or, The loss function of the reward model is log-likelihood loss.

6. The method according to any one of claims 1 to 4, characterized in that: The method of acquiring a training set based on multiple time series data of the sea wind grid-connected system includes: Acquire a time series data set, wherein the time series data set includes a plurality of time series data of the sea wind grid-connected system; Performing downsampling processing on the plurality of time series data; The training set is obtained based on the time series data set after downsampling.

7. A multi-task risk assessment device for sea wind grid-connected system based on human feedback reinforcement learning, characterized in that: The device comprises: An acquisition module, used for acquiring a training set based on multiple time series data of the sea wind grid-connected system, wherein the training set includes a first sub-training set, a second sub-training set and a third sub-training set, wherein the first sub-training set is used for training an outlier detection task, the second sub-training set is used for training a missing value filling task, and the third sub-training set is used for training a prediction task; A modeling module, used to establish a multi-task risk assessment model, wherein the multi-task risk assessment model is a natural language model, and the multi-task risk assessment model is used to perform the outlier detection task, the missing value filling task and the prediction task, and the output of the prediction task is used to perform risk assessment; A training module, used for training the multi-task risk assessment model using a training set, and in the process of training the multi-task risk assessment model, using a human feedback reinforcement learning algorithm to perform reinforcement learning on the multi-task risk assessment model; A risk assessment module is used to perform risk assessment on the offshore wind grid-connected system based on the trained multi-task risk assessment model.

8. A computer device, characterized in that: The computer device comprises: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images