Wind power cabin equipment quality evaluation and maintenance decision-making method based on reinforcement learning

Abstract

The invention relates to a wind power cabin equipment quality evaluation and maintenance decision-making method based on reinforcement learning, and belongs to the field of wind turbine generator operation and maintenance. According to the method, the state degradation model of the wind power cabin is established by using the Markov decision process, and the state transition probability is estimated in combination with the dynamic integrated probabilistic neural network, so that the prediction accuracy is improved. Weibull distribution parameter estimation is carried out on the fault data of the subcomponents of the wind turbine cabin through the grey model, life reliability analysis of the components is completed, then a reliability function of the wind turbine cabin is obtained, and equipment quality evaluation is achieved. According to the method, a maintenance method combining post maintenance and opportunity maintenance is adopted, an objective function of a maintenance decision model is established according to a component reliability function and a maintenance cost model, and an optimal maintenance strategy is solved by using an AD3QN algorithm, so that the minimum maintenance cost is realized, and the operation efficiency and economic benefits of the wind power plant are improved.

Description

Technical Field

[0001] The present invention belongs to the field of wind turbine operation and maintenance, and relates to a wind turbine cabin equipment quality assessment and maintenance decision-making method based on reinforcement learning. Background Art

[0002] The wind turbine cabin contains key electromechanical equipment for wind turbine operation, such as gearboxes, generators, yaw systems, electrical cabinets, hydraulic systems and cooling elements, which often fail. The various operation and maintenance work in the wind farm is mainly carried out by referring to the maintenance manual provided by the wind turbine manufacturer. However, since the operation and maintenance manual provided by the manufacturer is relatively simple, there is a certain deviation from the actual operation of the wind turbine. The results show that this method cannot effectively deal with various problems that occur in the operation of wind turbines. Therefore, the present invention proposes a wind turbine cabin equipment quality assessment and maintenance decision-making based on reinforcement learning.

[0003] Reinforcement learning refers to the learning of an agent from the environment to behavior, with the goal of maximizing or minimizing the reward value of the behavior. The difference from other machine learning methods is that the agent in this learning method is not told what action to take, but needs to interact with the environment through continuous attempts, and then choose actions based on the information obtained from the interaction as output to the external environment. Executing this action will lead to a change in the state of the environment, and the agent will receive rewards or penalties from the environment. The goal of the agent is to maximize or minimize the rewards or costs fed back from the environment due to the selected actions. Therefore, if the behavior of the agent can achieve the above goals, the agent's tendency to take such behavior in the future will be strengthened. The entire calculation process of the current reinforcement learning method can be described as: based on the system's operation process, relying on the Markov decision process to establish a system operation model, and solving the model through different algorithms.

[0004] Although reinforcement learning algorithms are superior to other algorithms in some aspects, most research work is still in the theoretical analysis stage, and few studies directly apply reinforcement learning algorithms to the operation and maintenance practice of wind turbines. Therefore, conducting research on wind turbine cabin maintenance decision-making based on reinforcement learning has important practical value.

[0005] The design life of wind turbines is 20 years. As the units installed earlier continue to operate for a period of time, equipment failures and maintenance issues will become increasingly prominent. For example, the complex structure of wind turbines and the harsh working environment lead to frequent failures of gearbox components, low availability, "under-maintenance" and "over-maintenance" problems caused by traditional maintenance strategies, and the failure to effectively utilize the data recorded by the unit system and maintenance personnel at this stage. These problems urgently need to be solved through targeted research to reduce the operation and maintenance costs of wind power equipment, improve the economic efficiency of wind power, and ensure the healthy development of wind power. Summary of the invention

[0006] In view of this, the object of the present invention is to provide a wind turbine cabin equipment quality assessment and maintenance decision-making method based on reinforcement learning.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A wind turbine cabin equipment quality assessment and maintenance decision-making method based on reinforcement learning includes the following steps:

[0009] Constructing a wind turbine nacelle state degradation model:

[0010] Define the operating status of wind turbine nacelle components and establish the state space;

[0011] Define maintenance measures for wind turbine nacelle components and establish maintenance measure sets;

[0012] The dynamic integrated probabilistic neural network DEPNN is used to estimate the state transition probability and establish a state transition probability model.

[0013] According to the state transition probability model, the degradation model of wind turbine cabin components is constructed;

[0014] Constructing a wind turbine nacelle maintenance decision model:

[0015] The gray model GM(1,1) is used to estimate the parameters of the Weibull distribution model of the wind turbine nacelle subcomponent failure data, and the reliability and failure rate function of the subcomponent are obtained.

[0016] According to the reliability and failure rate functions of the subcomponents, combined with different weights, the reliability function of the wind turbine nacelle is calculated;

[0017] According to the reliability function, the component status is divided and the maintenance strategy is determined;

[0018] Based on the changes in component reliability before and after maintenance and other factors such as downtime, the single maintenance cost of each component is calculated, and the total cost of the component during the operating cycle is accumulated;

[0019] Establish the objective function of the maintenance decision model, which aims to comprehensively consider the maintenance cost, component reliability and overall system efficiency;

[0020] Optimal maintenance decision for wind turbine components based on reinforcement learning:

[0021] The maintenance decision model is solved using the AD3QN algorithm to obtain the minimum maintenance cost.

[0022] Furthermore, the operating states of the wind turbine cabin components are divided into the following categories:

[0023] Brand new condition means that the parts have just been installed and are in optimal operating condition with 100% reliability;

[0024] Operational status means that the component is in normal working condition, the reliability gradually decreases, but it can still continue to operate;

[0025] A fault state means that the reliability of a component has dropped to a certain level and it cannot continue to operate and needs to be repaired or replaced.

[0026] Furthermore, the maintenance measures for the wind turbine cabin components include:

[0027] Minimum maintenance: After the component is minimally maintained, the reliability of the component does not change, that is, the reliability is:

[0028]

[0029] Where R(t) is the reliability function of the component after the μth consecutive minimum maintenance, t represents time, γ represents the location parameter of the Weibull distribution, δ represents the scale parameter of the Weibull distribution, and m represents the shape parameter of the Weibull distribution;

[0030] Incomplete maintenance: Introduce the maintenance effect attenuation factor of the service age regression factor into the formula of the reliability function, expressed as:

[0031]

[0032] Where μ is the μth uninterrupted incomplete maintenance of the component, ξ is the total number of uninterrupted incomplete maintenance, and λ is [μ] is the service life regression factor of the component after the μth incomplete maintenance without interruption, is the time interval between the μth and μ+1th consecutive incomplete maintenance, R [μ] (t) is the reliability function of the component after the μth consecutive incomplete maintenance; j represents the number of maintenance times, and k represents the serial number of the maintenance component;

[0033] The expression of the service age regression factor is:

[0034]

[0035] Replacement repair: When a vulnerable part fails, or when a vulnerable part does not fail but reaches the threshold for opportunity repair, only replacement repair is considered. After replacement, the reliability of the part is considered to be restored to the initial state, that is, the reliability starts from 1 again;

[0036] No maintenance.

[0037] Furthermore, the objective function of the maintenance decision model is:

[0038]

[0039] Among them, C total represents the total cost of the wind turbine nacelle operation cycle, K represents the total number of wind turbine nacelle subcomponents, k represents the subcomponent serial number of the wind turbine nacelle, ω k represents the weight of the total cost of the kth subcomponent operation cycle to the total cost of the wind turbine cabin, C k represents the total cost of the kth subcomponent operation cycle, C vt represents the cost caused by the change of wind turbine nacelle reliability, R k Represents the reliability value of the kth subcomponent.

[0040] Further, the AD3QN algorithm comprises the following steps:

[0041] Initialize the parameters of the attention mechanism, fully connected layer, estimation network and target network, and initialize the experience replay buffer;

[0042] For each training cycle, the wind turbine nacelle status is obtained from the wind turbine monitoring system, the input features are processed using the attention mechanism, and maintenance actions are selected;

[0043] After performing the maintenance action, observe the reward and the next state, and store the experience in the experience pool;

[0044] Randomly sample a batch of experiences from the experience pool to train the target network structure and update the network parameters;

[0045] The trained AD3QN algorithm is used to solve the wind turbine nacelle maintenance decision model and select the optimal maintenance action to achieve the maximum and minimum total maintenance cost of the wind turbine nacelle during the operation cycle.

[0046] The beneficial effects of the present invention are:

[0047] (1) Based on the basic reliability theory and combined with the Markov decision process, the present invention introduces the dynamic integrated probabilistic neural network (DEPNN) to estimate the transition probability and establish a degradation model for the wind turbine cabin. The DEPNN algorithm integrates multiple probabilistic neural network models and can dynamically adjust the model parameters and structure according to environmental changes, thereby reducing the possible deviation and variance of a single model, improving the accuracy of the overall prediction, and providing an effective mathematical tool for formulating maintenance strategies.

[0048] (2) The present invention sets the fault data of each subcomponent of the wind turbine nacelle as Weibull distribution, uses the gray model GM (1,1) to generate gray, estimates the parameters of the Weibull distribution model of each subcomponent, and completes the life reliability analysis of the component based on the parameter estimation results. According to the reliability function of the nacelle subcomponents, calculations are performed according to different weight values ​​to obtain the reliability function of the wind turbine nacelle, and complete the quality evaluation of the nacelle equipment.

[0049] (3) The present invention divides the different states of components according to the reliability function, adopts a maintenance method that combines post-maintenance and opportunistic maintenance, and adopts different maintenance strategies according to different reliability thresholds. After taking different maintenance measures for the components, they can be restored to the corresponding set reliability function. Then, the single maintenance cost of each sub-component can be obtained based on the changes in different reliability before and after each maintenance, combined with consideration of factors such as downtime. The total cost of the component during the operating cycle can be obtained by accumulating the maintenance cost. The maintenance cost of different components is calculated according to different weight values, and the cost generated by the cabin due to the change in reliability during the operating cycle is introduced. A more accurate wind turbine cabin maintenance decision model can be established.

[0050] (4) The present invention introduces reinforcement learning theory and AD3QN algorithm to solve such problems. The AD3QN algorithm extracts key information from the input state by introducing an attention mechanism, separates the value function and the advantage function, and enables the network to more efficiently evaluate the overall value of the state and the advantage of each action. By continuously learning and adjusting the maintenance strategy, the minimum total maintenance cost for the long-term operation of the cabin is obtained.

[0051] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0053] Figure 1 It is a flow chart of the present invention;

[0054] Figure 2 It is the basic model of MDP;

[0055] Figure 3 Categorize maintenance strategies;

[0056] Figure 4 Provides a basic framework for reinforcement learning;

[0057] Figure 5 It is the AD3QN network structure. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0059] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0060] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0061] In view of the current problems of high failure rate of wind turbine cabin components, large maintenance workload and high maintenance costs in China, the present invention has carried out a study on maintenance decision-making technology based on the combination of Markov decision process and reinforcement learning. In order to effectively solve the above problems, firstly, based on the Markov decision process, the DEPNN framework learning transfer probability model is introduced, and the degradation model of the wind turbine cabin is constructed on this basis. This model can accurately reflect the state changes and degradation process of the wind turbine cabin during operation. The gray model is used to estimate the parameters of the Weibull distribution model of the failure data of the wind turbine cabin sub-components, and the reliability and failure rate functions of the sub-components can be obtained, and then the reliability function of the wind turbine cabin can be obtained according to different weights. The reliability function provides an important mathematical basis for the quality assessment of wind turbine cabin equipment.

[0062] Next, the maintenance plan adopts a combination of post-maintenance and opportunistic maintenance. According to the different operating conditions of the wind turbine nacelle sub-components and the changes in component reliability before and after maintenance, the maintenance cost of the component under various maintenance decisions is calculated. This process not only takes into account the cost of the maintenance operation itself, but also comprehensively evaluates the downtime losses and other related economic losses that may be caused by component failure, thereby more comprehensively reflecting the economic efficiency of maintenance. According to the changes in the reliability function during the operation cycle of the wind turbine nacelle, the comprehensive cost of the wind turbine nacelle is calculated.

[0063] On this basis, the objective function of the maintenance decision model is further constructed. This objective function aims to comprehensively consider the maintenance cost, component reliability and overall system efficiency to ensure the scientificity and rationality of the decision. In order to find the best maintenance strategy, the AD3QN (Attention-Based Dueling Double Deep Q-Network) algorithm is adopted. This is an effective reinforcement learning method that combines the advantages of the D3QN algorithm with the attention mechanism, and can optimize the decision-making process through continuous iteration and learning. Ultimately, the goal of the present invention is to achieve the optimal maintenance decision for the wind turbine cabin in each state, so that the total maintenance cost during the operation cycle is minimized, thereby improving the overall operating efficiency and economic benefits of the wind farm.

[0064] This method can not only effectively reduce the failure rate of cabin components, but also significantly reduce the maintenance workload and costs, thus providing technical support for the sustainable development of the domestic wind power industry. This research has important theoretical significance and practical application value, and provides new ideas and solutions for the subsequent maintenance and management of wind power equipment.

[0065] The main function flow chart of the present invention is as follows Figure 1 shown.

[0066] Step 1: Construct a wind turbine cabin state degradation model:

[0067] The design of the present invention relies on the Markov decision process to establish a state maintenance decision model. In the actual operation process, the state of certain systems or components in the wind turbine set undergoes a degradation process, that is, after the wind turbine set is manufactured and established, during its operation, the systems or components in the wind turbine set will be damaged due to various impacts until failure occurs, and this process is often continuous and long-term. Based on whether the component can be restored to normal operation through maintenance measures after a failure, the components can be divided into two categories: repairable components and non-repairable components. In the present invention, the analysis and discussion of the component maintenance decision model are all carried out on the wind turbine cabin, which are all repairable components, and their operating status can be intervened through maintenance measures.

[0068] 1.1. Markov decision theory:

[0069] The present invention regards the optimization problem based on the maintenance decision of wind turbine components as a random sequence decision problem, and uses the Markov decision process to construct the degradation model of the components, uses deep reinforcement learning to optimize the Markov decision process, and finally uses the AD3QN algorithm in reinforcement learning to solve the maintenance decision objective function. In the Markov decision process, the main factors that determine the final operation process of the system include: the state transition law in the system and the selected decision plan, and the decision maker obtains certain benefits through the interaction of the above factors. The Markov decision process adds the decision process of the intelligent agent on the basis of the Markov process. In each state, the system will select an action from the action set, and then the system will obtain the benefits related to the current state and action. The positive or negative of this benefit depends on the characteristics of each system. At the same time, the action taken will cause the state of the system to transfer, which will affect the state of the system at the next decision point.

[0070] The Markov decision process has the following characteristics:

[0071] (1) The system's state transition process is memoryless;

[0072] (2) The state transition process of the system is uncertain;

[0073] (3) During the operation of the system, each state can be obtained through detection.

[0074] like Figure 2 As shown in the figure, this figure describes that under the Markov decision process, after the agent performs an action, it will obtain new current state information and the immediate reward that the system can obtain for this action.

[0075] MDP usually consists of the following five groups:

[0076] {S,A,P(s′|s,a),R(s,a,s′),γ} (1-1)

[0077] Among them, s, s′∈S, a∈A;

[0078] Here is the meaning of each part of the quintuple group:

[0079] (1) S: represents the set of eigenvalues ​​of the system in its actual state, that is, the state space of the system.

[0080] (2) A: represents the set of all actions that the system can take based on its state, that is, the decision space of the system.

[0081] (3) P(s′|s,a): represents the state transition probability during the operation of the system. The system is in state s at a certain decision point. After taking action a, the probability that the system is in state s′ when it reaches the next decision point. Where a refers to the action taken by the system in state i.

[0082] (4) R(s,a,s′): represents the benefits or costs obtained by the system during the period of time when the system is in state s at a decision moment and moves to state s′ after taking action a.

[0083] (5)γ: discount coefficient.

[0084] This paper proposes a framework based on dynamic ensemble probabilistic neural network (DEPNN) for learning state transition probability models in Markov decision processes (MDPs). The framework combines ensemble learning, probabilistic neural network (PNN) and dynamic adjustment strategy to improve the model's prediction accuracy for state transition probabilities. By constructing multiple PNN models for ensemble learning and using dynamic adjustment strategy to update model weights or retrain models according to prediction errors, accurate estimation of MDP state transition probabilities is achieved.

[0085] First, to learn the state transition probability in the Markov decision process, we construct a separate PNN for each state s and action a, denoted as PNN s,a These PNN models use a radial basis function (RBF) network structure, which outputs an estimate of the transition probability of the next possible state by inputting the feature vector and action information of the current state.

[0086] Then, we use the ensemble learning strategy to average or weighted average the prediction results of multiple PNN models to obtain a more accurate estimate of the state transition probability. For each state s and action a, we train k PNN models, denoted as For each state s and action a, the prediction results of k PNNs are integrated by weighted average:

[0087]

[0088] where ω j is the weight of each PNN model, and

[0089] In order to further improve the adaptability of the model, we introduced a dynamic adjustment strategy. This strategy dynamically updates the weight ω of each PNN model according to the prediction error e of the current model. j , and retrain or replace part of the model when the prediction error exceeds a certain threshold.

[0090]

[0091] in, is the prediction error of the jth model at time t, and λ is the adjustment factor. If the prediction error exceeds a certain threshold, part of the PNN model is retrained or replaced.

[0092] Through the above steps, an integrated probabilistic neural network model capable of dynamically adjusting weights and retraining was constructed to learn the state transition probability of the Markov decision process.

[0093] 1.2 Construction of degradation model:

[0094] The construction of the multi-stage degradation model of wind turbine components mainly consists of three parts, including: the definition of component status, and the definition and classification of component failures. The multi-stage degradation model based on the Markov decision process is introduced:

[0095] The component state is defined, and the component operation state is divided into n different stages according to the change of reliability during the entire operation process of the component, which are recorded as 1, 2, ..., n. The state set S is as follows:

[0096] S={1,2,…,n-1,n} (1-4)

[0097] Combining formula (1-8), it can be seen that: the component is in a new state in state 1; when the component is in any state from state 2 to n-1, the component can still continue to operate, but its operating efficiency, random failure rate, etc. will be affected; the component is in a faulty state in state n, at which time the component can only be replaced, and the component cannot continue to operate before it is replaced. Among them, in the selected state i, j in the state set S of the component, if there is 1≤i≤j≤n, then it can be considered that the degradation level of the component in state i is less than that in state j.

[0098] The fault definition simply divides faults into two types, including random faults that occur randomly in components in various states and degradation faults that occur when the reliability of components reaches a threshold during operation.

[0099] The state transfer process is described as follows: Since it is necessary to find the optimal component maintenance strategy in the maintenance decision model of component status, this paper adopts the Markov decision process of discontinuous state in continuous time to describe the transfer of component status.

[0100] The continuous-time Markov decision process of component state degradation is expressed as follows:

[0101] {X(t),t≥0} (1-5)

[0102] In formula (1-9), t represents time, and X(t) represents the state of the component at time point t. All possible values ​​of X(t) constitute the complete state space of the component, that is, the state set S defined in formula (1-9). The operating state of the component is divided by the reliability curve of the component.

[0103] The behavior set consists of N different maintenance measures, namely A = {a1, a2, ..., a n}. Take different maintenance measuresa n The structure undergoes different changes. When the degraded structure is in state i, if action a is taken n , the probability of the structure entering state j is recorded as P(j|i,a n ), and the cost of the corresponding maintenance behavior is recorded as C. Different behaviors under different states will produce different maintenance effects.

[0104] During the entire operation of the component, at time t, the probability of the component in each state is expressed as follows:

[0105] p i (t)=Pr{X(t)=i},i=1,2,...n (1-6)

[0106] At any time point t, the probability of a part falling into each state can be expressed as follows:

[0107] P(t)={p1(t),p2(t),...,p n-1 (t),p n (t)} (1-7)

[0108] The sum of the probabilities of the components in various states at time t and the values ​​of each probability satisfy the following conditions:

[0109]

[0110]

[0111] In the present invention, it is assumed that the state transition process of the component under the decision model satisfies the Markov property, that is, the operation process of the component after a certain decision point is determined by the state of the component at the decision point and the decision actions of each decision point in the future, and is independent of the state and decision before the current decision point, as shown below:

[0112]

[0113] Among them, the state transition probability during the operation of the component is generally represented by a transition probability matrix, which can be achieved through the method introduced in 1.1.

[0114] The second part builds a wind turbine nacelle maintenance decision model

[0115] 2.1 Establishment of wind turbine nacelle structure and reliability model

[0116] The working principle of wind turbines is that the blades exposed outside the nacelle capture the natural wind in the air and rotate themselves to convert wind energy into mechanical energy, and then transfer energy to the gearbox through the axis and main shaft system. The gearbox then transfers the energy to the generator, and finally converts the mechanical kinetic energy generated by the rotation of the rotor into target electrical energy. The main shaft, gearbox and generator inside the wind turbine nacelle are the core equipment for the normal operation of the wind turbine and the realization of power generation goals, and are also the equipment with high incidence of failure inside the wind turbine nacelle.

[0117] Most of the faults in the engine room occur in the main drive chain system. Common faults are listed below:

[0118] (1) The generator, gearbox, etc. overheat and overtemperature occurs;

[0119] (2) Oil or water leakage occurs in hydraulic equipment, gear boxes, generators, etc.;

[0120] (3) The wiring cables become loose;

[0121] (4) Loose fan bearings or large gaps in the turntable shaft bearings can cause abnormal vibration and noise in the generator:

[0122] (5) If the tower of a wind turbine is tilted too much, it will cause huge safety hazards to the cabin.

[0123] Based on the specific calculation of the maintenance cost C required for different maintenance measures of the component status, this paper adopts a component maintenance cost calculation method based on the change in component reliability before and after maintenance. Therefore, it is necessary to use the gray model GM (1,1) to generate gray for the fault data of the wind turbine nacelle components, fully mine the data information, estimate the three-parameter Weibull distribution parameters for the fault data of the wind turbine nacelle, and complete the reliability analysis of the wind turbine nacelle life based on the parameter estimation results.

[0124] ① Three-parameter Weibull distribution model

[0125] In the study of Weibull distribution of wind turbine nacelle component failure data, the three-parameter Weibull distribution introduces a location parameter compared to the two-parameter Weibull distribution, which can more fully and flexibly describe the distribution law of component failure data and can be applied to the reliability analysis of component failure data. However, the introduction of the location parameter also increases the complexity of parameter estimation. The distribution function F(t), probability density function f(t) and reliability function R(t) of the three-parameter Weibull distribution are:

[0126]

[0127] Where γ is the location parameter, m is the shape parameter, and δ is the scale parameter.

[0128] ② Parameter estimation method of three-parameter Weibull distribution based on grey model GM(1,1)

[0129] For the fault data of a certain component in the wind turbine nacelle, the common parameter estimation methods of its Weibull distribution model include correlation coefficient optimization method, moment method, regression estimation method and grey estimation method. Among them, the grey model GM(1,1) has good parameter estimation performance when analyzing sample fault data. By solving the inverse function, F(t) in equation (2-1) can be transformed into:

[0130]

[0131] make: Formula (2-2) can be transformed into:

[0132]

[0133] At this time, the relationship between formula (2-3) and the time response function of the grey model GM (1,1) is:

[0134]

[0135] In the formula, a and b are the parameters to be estimated. Using the idea of ​​least squares estimation method, let:

[0136]

[0137]

[0138] R is the failure time t i The mean of is generated. At this time, the least squares estimation result is:

[0139]

[0140] Then the estimated results of the location parameter γ and shape parameter m in the Weibull distribution of the component are:

[0141]

[0142]

[0143] Similarly, for the scale parameter δ in the Weibull distribution, let:

[0144]

[0145] N=[t1 t2 …t n ] T (2-11)

[0146] The least squares estimation result is:

[0147]

[0148] At this time, the estimation result of the scale parameter δ in the three-parameter Weibull distribution is:

[0149]

[0150] 2.2 Wind turbine cabin equipment quality assessment

[0151] From the analysis and calculation in 2.1, it can be seen that by obtaining fault data and performing parameter estimation, the reliability function of a certain component of the wind turbine nacelle can be obtained. Since there are multiple subcomponents in the wind turbine nacelle, it is assumed that there are k (k = 1, ..., K) subcomponents inside the nacelle, and the reliability function of each subcomponent is R k (t), then the reliability function of the entire wind turbine cabin is designed as:

[0152] R vt (t)=α1R1(t)+α2R2(t)+…+α K-1 R K-1 (t)+α K R K (t) (2-14)

[0153] Among them, α k The reliability function of each subcomponent is the weight of the reliability function of the entire nacelle. vt (t) is determined by the reliability of the subcomponents and has more practical significance in engineering.

[0154] According to the reliability function R of the wind turbine nacelle vt (t) Evaluate the quality of cabin equipment. vt When (t)≤0.4, the cabin equipment is of low quality; when 0.4<R vt (t)≤0.8, the cabin equipment is of medium quality; when 0.8<R vt When (t)≤1, the cabin equipment is of high quality.

[0155] 2.3 Wind turbine nacelle maintenance measures

[0156] The maintenance cost of wind turbines is very high, and maintenance strategy is one of the main factors affecting the maintenance cost. According to the actual operation of the wind farm, adopting a reasonable maintenance strategy for each component of the wind turbine can significantly improve the economic benefits of the wind farm. At present, wind farms generally adopt a maintenance strategy that combines post-maintenance and preventive replacement maintenance. Post-maintenance is a maintenance activity that occurs after a unit component fails, and preventive replacement maintenance is a periodic replacement of unit components. This maintenance strategy ignores the time correlation of the maintenance of each component and increases the maintenance cost.

[0157] Therefore, the present invention adopts a method combining post-fault maintenance and opportunistic maintenance. Opportunistic maintenance is to determine whether the non-faulty parts of the wind turbine set need preventive replacement and maintenance when post-fault maintenance is performed on the faulty parts of the wind turbine set, thereby reducing the number of wind turbine set shutdowns due to maintenance. That is, when a faulty part occurs, the present invention performs post-fault maintenance on the faulty part, and at the same time, performs opportunistic maintenance, i.e., repairs or replacements, on other parts that have not failed but whose reliability has dropped to a lower level, such as Figure 3 shown.

[0158] There are many ways to repair different parts, and the reliability recovery of parts is closely related to their repair methods. In order to facilitate the calculation of the reliability function of parts after repair, the post-repair and opportunity maintenance methods are classified into four types, namely minimum maintenance (a1), incomplete maintenance (a2), replacement maintenance (a3) ​​and no maintenance (a4).

[0159] ① Minimum maintenance (a1): For some large components, power off and restart operations do not change the original reliability of the components, and the reliability curve of the components remains unchanged after maintenance. After the components are minimally repaired, the reliability of the components does not change, that is, the reliability remains:

[0160]

[0161] Where R(t) is the reliability function of the component after the μth consecutive minimum maintenance.

[0162] ② Incomplete maintenance (a2): After an incomplete maintenance of a component, the reliability of the component cannot be restored to its initial state, and with the accumulation of uninterrupted incomplete maintenance times, the reliability recovery ability of the component will gradually weaken. Therefore, the traditional Weibull distribution can no longer be used to describe its reliability change. Introducing the service age regression factor and the maintenance effect attenuation factor into the reliability function formula can be expressed as:

[0163]

[0164] Where μ is the μth uninterrupted incomplete maintenance of the component, ξ is the total number of uninterrupted incomplete maintenance, and λ is [μ]is the service life regression factor of the component after the μth incomplete maintenance without interruption, is the time interval between the μth and μ+1th consecutive incomplete maintenance, R [μ] (t) is the reliability function of the component after the μth consecutive incomplete maintenance. The expression of the service age regression factor is:

[0165]

[0166] ③ Replacement repair (a3): When a vulnerable part fails, or when a vulnerable part does not fail but reaches the threshold for opportunity repair, usually only replacement repair is considered. After replacement, it is considered that the reliability of the component has returned to its initial state, that is, the reliability starts from 1 again.

[0167] 2.4 Wind turbine nacelle failure maintenance cost model

[0168] Based on the correlation between various components of the wind turbine nacelle, the following assumptions are made before establishing the maintenance cost model:

[0169] 1) The failure rates of each subcomponent in the wind turbine nacelle follow a three-parameter Weibull distribution and are independent of each other;

[0170] 2) Failure of any component in the wind turbine nacelle will cause system failure and require shutdown for maintenance. The time of shutdown for maintenance cannot be ignored compared to the operating time of the wind turbine.

[0171] The maintenance cost C of a single wind turbine cabin designed in the present invention is total Depends on the total maintenance cost C of each subcomponent operation cycle k , and the cost C of each maintenance causing the change in the reliability of the wind turbine nacelle vt , C total It can be expressed as:

[0172]

[0173] Where K is the total number of cabin subcomponents, ω k It is the weight of the maintenance cost of different parts in the total maintenance cost.

[0174] in:

[0175]

[0176]

[0177] In formula (2-21), C kol is the cost of a single maintenance of subcomponent k, and ξ represents the total number of maintenance times for the component. vt is the reliability value of the wind turbine nacelle, ΔRvt represents the change in reliability of the wind turbine nacelle during the operation cycle, and c is the set fixed cost.

[0178] C kol =C e +C m +C f (2-21)

[0179] In formula (2-23), C e is the cost of power loss caused by wind turbine shutdown, C m The direct cost of maintenance, C f It is the fixed cost in maintenance. The details are as follows:

[0180] (1) Electricity loss cost C e , occurs during post-maintenance, which requires downtime, so the power loss is mainly determined by the total downtime of the wind turbine. Usually, the maintenance downtime of a wind turbine mainly includes the following four stages, namely, reaching the unit stage (T1), climbing onto the nacelle stage (T2), hoisting tools stage (T3) and component maintenance stage (T4).

[0181]

[0182] Where τ is the capacity factor of the wind turbine (the wind turbine will not run at full load when it is working normally), P is the rated power of the wind turbine, C ep When an incomplete maintenance method is adopted for a component, the component maintenance phase time (T4) is jointly affected by the processing time of the maintenance and replacement activities and the reliability recovery level of the component after maintenance compared with before maintenance, which can be expressed as:

[0183]

[0184]

[0185] Where t k is the processing time for the replacement activity of component k, ΔR k is the reliability recovery level of component k after repair compared with that before repair, R k is the reliability of component k.

[0186] (2) Direct maintenance cost C m , is simply affected by the maintenance method. When performing replacement maintenance, the direct cost is simply affected by the price of the parts. k When implementing incomplete repairs, the direct cost depends on the combined impact of the price of the parts and the level of reliability recovery after the repair compared to before the repair, that is:

[0187]

[0188]

[0189] Where c k is the maintenance cost of replacing component k, ΔR k is the change of component reliability before and after maintenance, R k is the reliability of component k.

[0190] (3) Fixed cost C f , mainly for the costs of manpower, equipment and transportation, especially the cost of deploying large maintenance equipment. Therefore, if some simple and minimal maintenance operations are performed, there is no need to start equipment and transportation costs, which can be ignored. Therefore, when performing post-maintenance, C f It only exists in incomplete repair and replacement repair. When opportunistic repair is performed on some parts, since fixed costs have been incurred when the faulty parts are repaired afterwards, the fixed cost C here is f Just take 0.

[0191] In summary, a maintenance decision model is established, with the minimum total cost of the wind turbine cabin operation cycle as the objective function of the maintenance decision:

[0192]

[0193] Step 3: Optimal maintenance decision of wind turbine components based on reinforcement learning:

[0194] This paper establishes a multi-state degradation system model based on the Markov decision process. According to the different states of the system, the optimal maintenance strategy is adopted to minimize the total cost of the wind turbine nacelle system in the entire operation cycle. Reinforcement learning is a decision-making tool that can effectively solve the maintenance decision optimization problem. This paper introduces the AD3QN algorithm to solve the optimal maintenance strategy and obtain the minimum maintenance cost C total .

[0195] like Figure 4 As shown in Figure 1, the reinforcement learning algorithm includes four key elements: system model, immediate benefits, state value function, and action strategy. The system will explore through trial and error to obtain immediate rewards and then choose the appropriate strategy.

[0196] The wind turbine nacelle takes decision a in state s n , the probability of the system state transferring to s′ is recorded as: P(s′|s,a), which is solved by the DEPNN model. The cost of the system state transferring to s′ is recorded as R(s′|s,a), which is the cost of taking decision a n The value of the cost to make the repairs.

[0197] The D3QN algorithm incorporates the idea of ​​the Double DQN algorithm on the basis of the Dueling DQN algorithm. It has improved the calculation method of the target value compared with the Dueling DQN algorithm, and can effectively solve the problem that the algorithm fails to select the optimal action due to overestimation of the expected value Q value. On this basis, the present invention designs the AD3QN algorithm, that is, introduces the attention mechanism on the basis of the D3QN algorithm to help the model focus on the most relevant part of the input features. Design an attention layer to calculate the attention weight of the input features.

[0198] The network structure of AD3QN algorithm is as follows Figure 5 As shown in the figure, it consists of an attention layer, a fully connected layer, an estimation network, a target network, and an experience pool. The attention layer extracts the target features of the input data of the wind turbine cabin components, uses a simple fully connected network to generate attention scores, and converts them into weights through a softmax function.

[0199] For the input feature, that is, the state s of the component, calculate the attention weight α:

[0200]

[0201] Among them, W a and b a is a learning parameter. Multiply the original state features of the wind turbine nacelle components by the attention weight to generate weighted features:

[0202]

[0203] The attention layer outputs weighted features to the fully connected layer, which can improve the model's ability to respond to important features. The fully connected layer is a competitive network structure, which is divided into an advantage function network and value function network The advantage function is related to the status of the wind turbine cabin components and the maintenance actions taken, while the value function is only related to the status of the components. Finally, the expected value Q of the cumulative discount reward is output, that is:

[0204]

[0205] In the formula, represents the state of the component after processing, a represents the maintenance action taken on the component, α is the network parameter of the value function branch, β is the network parameter of the advantage function branch, and θ is the network parameter of the common part.

[0206] The AD3QN algorithm uses two networks with the same structure and initial parameters, namely the estimation network and the target network. The state action value function is calculated by the estimation network, and the target value function is calculated by the target network. The experience replay mechanism effectively utilizes historical data, which is conducive to the convergence of the algorithm. t+1 The action a corresponding to the maximum Q value in the state max ,Right now:

[0207]

[0208] Then the target Q value in the target network is calculated based on the selected action, that is:

[0209]

[0210] The loss function of AD3QN is:

[0211] Q loss =E[(Q t -Q(s,a;θ) 2 ] (3-6)

[0212] In the formula, Q t and Q(s,a;θ) are the value functions of the target network and the estimated network respectively. By minimizing Q loss function to update the network parameters.

[0213] The AD3QN algorithm is used to solve the optimal solution for wind turbine cabin maintenance decision. The specific implementation steps are as follows:

[0214] (1) Initialize the parameters of the attention mechanism, fully connected layer, estimation network, and target network, and initialize the experience replay buffer.

[0215] (2) For each training cycle, the wind turbine nacelle status is obtained from the wind turbine monitoring system, the input features are processed using the attention mechanism, and maintenance actions are selected. After performing the maintenance action, the reward and the next state are observed. The experience (s, a, r, s′) is stored in the experience pool.

[0216] (3) Randomly sample a batch of experiences from the experience pool to train the target network structure. For each experience, use the estimation network to select the best maintenance action for the next state, and use the target network and the selected best action to calculate the target value Q t . The parameters of the estimation network and the target network are updated using the mean squared error loss function.

[0217] (4) At regular intervals, the parameters of the estimated network are copied to the target network.

[0218] (5) Use the trained AD3QN algorithm to solve the wind turbine cabin maintenance decision model. Select the optimal maintenance action a in each state to achieve Q t Maximum, total maintenance cost of wind turbine cabin during operation period C total Minimum.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A wind turbine cabin equipment quality assessment and maintenance decision-making method based on reinforcement learning, characterized by: The following steps are involved: Constructing a wind turbine nacelle state degradation model: Define the operating status of wind turbine nacelle components and establish the state space; Define maintenance measures for wind turbine nacelle components and establish maintenance measure sets; The dynamic integrated probabilistic neural network DEPNN is used to estimate the state transition probability and establish a state transition probability model. According to the state transition probability model, the degradation model of wind turbine cabin components is constructed; Constructing a wind turbine nacelle maintenance decision model: The gray model GM(1,1) is used to estimate the parameters of the Weibull distribution model of the wind turbine nacelle subcomponent failure data, and the reliability and failure rate function of the subcomponent are obtained. According to the reliability and failure rate functions of the subcomponents, combined with different weights, the reliability function of the wind turbine nacelle is calculated; According to the reliability function, the component status is divided and the maintenance strategy is determined; Based on the changes in component reliability before and after maintenance and other factors such as downtime, the single maintenance cost of each component is calculated, and the total cost of the component during the operating cycle is accumulated; Establish the objective function of the maintenance decision model, which aims to comprehensively consider the maintenance cost, component reliability and overall system efficiency; Optimal maintenance decision for wind turbine components based on reinforcement learning: The maintenance decision model is solved using the AD3QN algorithm to obtain the minimum maintenance cost.

2. The wind turbine cabin equipment quality assessment and maintenance decision-making method based on reinforcement learning according to claim 1 is characterized by: The operating states of the wind turbine nacelle components are divided into the following categories: Brand new condition means that the parts have just been installed and are in optimal operating condition with 100% reliability; Operational status means that the component is in normal working condition, the reliability gradually decreases, but it can still continue to operate; A fault state means that the reliability of a component has dropped to a certain level and it cannot continue to operate and needs to be repaired or replaced.

3. The wind turbine cabin equipment quality assessment and maintenance decision-making method based on reinforcement learning according to claim 1 is characterized by: The maintenance measures for the wind turbine cabin components include: Minimum maintenance: After the component is minimally maintained, the reliability of the component does not change, that is, the reliability is: Where R(t) is the reliability function of the component after the μth consecutive minimum maintenance, t represents time, γ represents the location parameter of the Weibull distribution, δ represents the scale parameter of the Weibull distribution, and m represents the shape parameter of the Weibull distribution; Incomplete maintenance: Introduce the maintenance effect attenuation factor of the service age regression factor into the formula of the reliability function, expressed as: Where μ is the μth uninterrupted incomplete maintenance of the component, ξ is the total number of uninterrupted incomplete maintenance, and λ is [μ] is the service life regression factor of the component after the μth incomplete maintenance without interruption, is the time interval between the μth and μ+1th consecutive incomplete maintenance, R [μ] (t) is the reliability function of the component after the μth consecutive incomplete maintenance; j represents the number of maintenance times, and k represents the serial number of the maintenance component; The expression of the service age regression factor is: Replacement repair: When a vulnerable part fails, or when a vulnerable part does not fail but reaches the threshold for opportunity repair, only replacement repair is considered. After replacement, the reliability of the part is considered to be restored to the initial state, that is, the reliability starts from 1 again; No maintenance.

4. The wind turbine cabin equipment quality assessment and maintenance decision-making method based on reinforcement learning according to claim 1 is characterized by: The objective function of the maintenance decision model is: Among them, C total represents the total cost of the wind turbine nacelle operation cycle, K represents the total number of wind turbine nacelle subcomponents, k represents the subcomponent serial number of the wind turbine nacelle, ω k represents the weight of the total cost of the kth subcomponent operation cycle to the total cost of the wind turbine cabin, C k represents the total cost of the kth subcomponent operation cycle, C vt represents the cost caused by the change of wind turbine nacelle reliability, R k Represents the reliability value of the kth subcomponent.

5. The wind turbine cabin equipment quality assessment and maintenance decision-making method based on reinforcement learning according to claim 1 is characterized by: The AD3QN algorithm includes the following steps: Initialize the parameters of the attention mechanism, fully connected layer, estimation network and target network, and initialize the experience replay buffer; For each training cycle, the wind turbine nacelle status is obtained from the wind turbine monitoring system, the input features are processed using the attention mechanism, and maintenance actions are selected; After performing the maintenance action, observe the reward and the next state, and store the experience in the experience pool; Randomly sample a batch of experiences from the experience pool to train the target network structure and update the network parameters; The trained AD3QN algorithm is used to solve the wind turbine nacelle maintenance decision model and select the optimal maintenance action to achieve the maximum and minimum total maintenance cost of the wind turbine nacelle during the operation cycle.

Images