Equipment optimal maintenance strategy searching method and system based on reinforcement learning

Through a method based on reinforcement learning, a dynamic attention integration Markov model and multi-agent framework are constructed, combined with a generative adversarial network, the complexity and insufficient capture capabilities of equipment maintenance strategies in the existing technology are solved, and the overall maximum is achieved Excellent, interpretable and timely maintenance decisions improve the operational efficiency and economicality of equipment maintenance.

CN118941275APending Publication Date: 2024-11-12SHENHUA HOLLYSYS INFORMATION TECH CO LTD
View PDF 0 Cites 11 Cited by

Patent Information

Application Number
CN202411242401.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing equipment maintenance strategies have problems such as strong subjectivity and limited capture capabilities when dealing with complex industrial systems, and it is difficult to make global optimal maintenance decisions, resulting in poor policy timeliness and effectiveness.

Method used

Using a reinforcement learning-based method, by constructing a dynamic attention integration Markov model and a multi-agent reinforcement learning framework, combined with a generative adversarial network, a comprehensive optimal maintenance strategy that can maximize the service life of the equipment and reduce maintenance costs is trained.

Benefits of technology

It realizes global optimality, interpretability and timeliness maintenance decisions, comprehensively captures the impact of complex environments, distributes policy decisions, introduces expert experience constraints, and quickly obtains global optimal maintenance solutions, improving operational efficiency and economy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941275A_ABST
    Figure CN118941275A_ABST
Patent Text Reader

Abstract

The invention provides an equipment optimal maintenance strategy searching method and system based on reinforcement learning. According to the method, a dynamic attention integrated Markov model is constructed to capture complex environmental influence, a multi-agent reinforcement learning framework distributed exploration maintenance strategy combination is set, and generative adversarial network training is adopted to optimize the strategy. The generator network outputs a comprehensive maintenance strategy capable of globally optimizing long-term income, and the discriminator network takes expert experience and long-term income as constraint conditions to guide the generator. According to the method, the defects of an existing manual rule and data model prediction method are overcome, the generated maintenance strategy has global optimality, interpretability and timeliness, the service life of equipment can be prolonged to the maximum extent, the maintenance cost can be reduced, and the method has great significance in improving the operation efficiency of the industrial equipment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of industrial equipment maintenance optimization, and in particular to a method and system for searching for an optimal equipment maintenance strategy based on reinforcement learning. Background Art

[0002] Equipment maintenance decision-making is a key link in achieving efficient operation in the manufacturing industry. How to formulate the optimal maintenance strategy to extend the service life of equipment while minimizing maintenance costs has always been a core technical issue in this field.

[0003] At present, the formulation of equipment maintenance strategies mainly relies on two technical routes: decision-making methods based on artificial experience rules and decision-making methods based on data model prediction. The artificial experience rule method summarizes the long-term operation and maintenance experience of experts to form a series of experience rule libraries for maintenance time points and operations to guide actual decision-making. The method based on data model prediction uses technologies such as machine learning to mine the evolution laws of equipment status from historical operation data, predict future failures, and formulate maintenance plans accordingly.

[0004] However, the above two traditional methods have obvious shortcomings when dealing with complex industrial systems. Manual rule-based decision-making is highly subjective and it is difficult to accurately describe the transfer mechanism of the system under a dynamic environment; and the existing data model prediction method simplifies the complexity of the system and has limited capture capabilities. It cannot fully consider the multiple environmental factors that affect the operation of the equipment, resulting in poor timeliness and effectiveness of the strategy. More importantly, these methods lack global optimization considerations for long-term operation and maintenance performance (such as service life, maintenance costs, etc.), and the benefits of the formulated strategies are often not maximized. Therefore, a new technical solution is needed to solve the shortcomings of the existing methods, so as to obtain globally optimal equipment maintenance decisions. Summary of the invention

[0005] In order to solve the problems of the prior art, the embodiment of the present invention provides a method and system for searching for an optimal maintenance strategy for equipment based on reinforcement learning. The technical solution is as follows:

[0006] On the one hand, a method for searching for an optimal maintenance strategy for equipment based on reinforcement learning is provided, which is used to obtain a comprehensive optimal maintenance strategy that can maximize the service life of the equipment and reduce the maintenance cost through reinforcement learning training according to the operating status data of the equipment, and includes the following steps:

[0007] A) Construct a dynamic attention integrated Markov model, use the historical data of equipment operation to extract the multiple environmental factors that affect the equipment state transition, use the multi-head attention mechanism to dynamically calculate the weight of each environmental factor on the equipment state, and integrate the weighted environmental factors with the current state of the equipment and the historical state sequence to establish a dynamic attention integrated Markov model that describes the state transition process of the equipment in a complex environment;

[0008] B) Set up a multi-agent reinforcement learning framework, treat different equipment maintenance operations as independent agents, and each agent has different state observations and reward and punishment function designs for specific subsystems of the equipment. Output the combined maintenance strategy through collaborative games, and use the multi-agent policy gradient algorithm to train each agent to achieve strategy coordination and consistency, and output the overall maintenance strategy;

[0009] C) Generative adversarial models are used for strategy optimization training. The generator network is set to output the maintenance strategy, and the discriminator network judges the quality of the strategy. Through adversarial training, the generator output strategy gradually approaches the ideal optimal distribution. The maintenance expert data and the long-term benefits of the strategy are used as the discriminator input to guide the generator to output a reasonable, explainable and globally optimal final maintenance strategy.

[0010] Furthermore, in the above step B), a value decomposition multi-agent reinforcement learning algorithm such as COMA, QProp or QMIX is used for training.

[0011] Furthermore, in the above step C), an improved GAN objective function and loss such as WGAN-GP or LSGAN are used for adversarial training.

[0012] Furthermore, during the adversarial training process, the existing maintenance expert experience data is input into the discriminator as positive samples to guide the generator to output a more explanatory strategy.

[0013] Furthermore, during the adversarial training process, the long-term benefit value of the strategy is input as a label into the discriminator to guide the generator to output an optimal strategy with higher benefits.

[0014] Furthermore, when constructing the dynamic attention ensemble Markov model in the above step A), techniques such as residual connection and layer normalization are used to improve the model's expressiveness.

[0015] Furthermore, an offline policy evaluation optimization step is included, and a reinforcement learning algorithm such as PPO or SAC is used to further optimize the maintenance policy output by the generator in a simulation environment.

[0016] Furthermore, the dynamic attention integrated Markov model constructed in the above step A) and the multi-agent reward and punishment function set in step B) are jointly designed based on multiple influencing factors such as energy loss, fault downtime, and maintenance cost during equipment operation.

[0017] Furthermore, the equipment is a generator set, a wind turbine or other industrial production equipment.

[0018] On the other hand, a system for searching for an optimal maintenance strategy for equipment is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed, the method for searching for an optimal maintenance strategy for equipment based on reinforcement learning is implemented.

[0019] The technical solution provided by the embodiment of the present invention has the following beneficial effects:

[0020] 1. Comprehensively capture the impact of complex environments

[0021] By constructing a dynamic attention integrated Markov model, it is possible to dynamically integrate the impact of multiple environmental factors such as temperature, humidity, and load on the equipment state transition, and comprehensively characterize the operating rules of the equipment under complex working conditions. Compared with the existing single state model, it has stronger expressive ability.

[0022] 2. Implement distributed policy decision-making

[0023] A multi-agent reinforcement learning framework is adopted, and different maintenance operations of equipment are regarded as independent agents. Each agent focuses on a specific subsystem and outputs a combined maintenance strategy through collaborative game, which realizes distributed parallelism of strategy decision-making and avoids the limitations of centralized decision-making.

[0024] 3. Introducing expert experience constraints

[0025] During the adversarial training process of the generative adversarial network, historical maintenance expert data is input into the discriminator network as positive examples to guide the generator network to output a maintenance strategy that is explainable and consistent with expert experience, effectively improving the rationality and interpretability of the strategy.

[0026] 4. Maximize long-term benefits

[0027] By introducing long-term operation and maintenance performance indicators (such as equipment service life, maintenance cost, etc.) into the labels of adversarial training, the final strategy output by the generator network can globally optimize long-term benefits, improving the economy and foresight of decision-making.

[0028] 5. Accelerate the strategy convergence process

[0029] The generative adversarial network training framework accelerates the convergence process of strategy exploration. At the same time, combined with reinforcement learning algorithms such as PPO for further optimization, it can quickly obtain the globally optimal maintenance plan and shorten the decision-making cycle.

[0030] In general, the reinforcement learning-based equipment optimal maintenance strategy search method proposed in the present invention has better solved the defects of the existing technology and can intelligently generate maintenance decision-making plans with global optimality, explainability and timeliness, which is of great significance for improving the operational efficiency and economy of industrial equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 1 is a flow chart of a method for searching for an optimal equipment maintenance strategy based on reinforcement learning according to Embodiment 1 of the present invention;

[0033] Figure 2 This is a module diagram of an equipment optimal maintenance strategy search system according to Example 2 of the present invention. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0035] The present invention proposes a method for searching for optimal equipment maintenance strategies based on reinforcement learning. By constructing a dynamic attention integrated Markov model to capture the impact of complex environments, setting up a multi-agent framework for distributed exploration of maintenance operation combinations, and adopting a generative adversarial network training optimization strategy, a comprehensive maintenance strategy that is globally optimal and interpretable by expert experience is finally output.

[0036] The specific technical solutions are as follows:

[0037] Constructing a dynamic attention integration state model

[0038] The historical data of equipment operation is used to extract multiple environmental factors (temperature, humidity, load, etc.) that affect the state transition of the equipment. The multi-head attention mechanism is used to dynamically calculate the weight of each factor on the equipment state. The weighted environmental factors are integrated with the current state of the equipment, and the dynamic attention integrated Markov model is constructed to describe the complex state transition through the gated recurrent unit and the historical state sequence model. This model can effectively capture the impact of environmental micro-changes on the equipment state.

[0039] Setting up the multi-agent reinforcement learning framework

[0040] Different maintenance operations of equipment are regarded as independent agents. Each agent controls a maintenance behavior and has different state observation and reward and punishment function designs, corresponding to the characteristics of different subsystems of the equipment. Multi-agents output combined maintenance strategies as behavioral interactions through collaborative games. Value decomposition multi-agent algorithms such as COMA, QProp or QMIX are used for training to achieve coordination and consistency among the strategies of each agent and output the overall maintenance strategy.

[0041] Strategy optimization training based on generative adversarial model

[0042] The generator network G is set to output the maintenance strategy, and the discriminator network D is set to judge the quality of the strategy. Adopt WGAN-GP or LSGAN to improve the GAN objective function and loss for adversarial training, so that the output strategy of the generator G gradually approaches the ideal optimal distribution. The existing maintenance expert experience data is input into the discriminator D as a positive sample to guide G to output a more interpretable strategy. At the same time, the long-term benefit value of the strategy (such as service life, maintenance cost) is input into D as a label to guide G to output the strategy with the best benefit. Adversarial training accelerates the convergence of the strategy, and G finally outputs the optimal maintenance strategy that is globally optimal and interpretable by expert experience.

[0043] Offline strategy evaluation and optimization

[0044] Using reinforcement learning algorithms such as PPO and SAC, the maintenance strategy output by the generator G is further optimized in a simulation environment. Environmental rewards and penalties are designed based on multiple influencing factors such as energy loss, downtime, and maintenance costs of equipment operation to maximize the benefits of the strategy.

[0045] The dynamic attention integration model in this invention captures the micro-influence of the environment, the multi-agent framework realizes the distributed collaborative decision-making of the maintenance strategy, and the generative adversarial network introduces expert knowledge and long-term benefit considerations, effectively overcoming the limitations of existing maintenance strategies. The output of the final strategy has global optimality and interpretability. This method can be applied to the optimal maintenance decision-making of industrial production equipment such as generator sets and wind turbines.

[0046] OK, I will give two specific examples to illustrate the implementation details of the invented method and system respectively:

[0047] Example 1

[0048] This embodiment takes a generator set as an example to illustrate the specific implementation process of a method for searching for an optimal maintenance strategy for equipment based on reinforcement learning of the present invention. Figure 1 :

[0049] 1. Constructing a dynamic attention integration state model

[0050] First, the operating data of the generator set for the past five years is collected, including environmental factors such as ambient temperature, humidity, load level, as well as status information such as unit fault codes and downtime duration.

[0051] An 8-head attention mechanism is adopted to dynamically integrate the influence weights of environmental factors and input them into the gated recurrent unit together with the current state code of the generator and the state sequence of the past 12 hours to construct an attention integrated hidden Markov model that describes the state transfer law of the unit.

[0052] 2. Setting up the multi-agent reinforcement learning framework

[0053] Three intelligent agents are set up to correspond to the cylinder maintenance, circuit inspection and cooling system maintenance operations of the unit respectively.

[0054] The reward and punishment function of the cylinder maintenance agent takes into account the output power attenuation; the reward and punishment function of the circuit maintenance agent takes into account the downtime of circuit failure; and the reward and punishment function of the cooling system maintenance agent takes into account the coolant replenishment cost.

[0055] The QMIX algorithm is used to train the above three agents so that they can learn to select appropriate single maintenance operations in their respective states and coordinate single operations to form an overall maintenance strategy combination.

[0056] 3. Generative Adversarial Network Strategy Optimization

[0057] Set the generator G network to output the maintenance strategy, and the discriminator D network to evaluate the strategy. Use the WGAN-GP algorithm to train G and D adversarially:

[0058] D constantly tries to distinguish the real expert repair data from G’s output strategy, while G strives to make the output strategy realistic enough to deceive D.

[0059] At the same time, the long-term service life of the unit after the strategy is executed is input as a label into D, thereby guiding G to output a strategy that maximizes the service life.

[0060] After enough training rounds, G converges and outputs a generator maintenance strategy that is globally optimal and consistent with expert experience.

[0061] 4. Offline strategy optimization

[0062] The strategy output by G is executed in a simulation environment, and the PPO algorithm is used to further fine-tune and optimize the strategy parameters based on the unit operating status and profit score fed back by the model.

[0063] Example 2

[0064] This embodiment provides a system framework for implementing the above-mentioned method for searching for the optimal maintenance strategy for equipment based on reinforcement learning. Figure 2 :

[0065] 1. Data processing module

[0066] Environmental factors, fault codes, state quantities and other features are extracted from equipment operation data as input data for state modeling and agent training.

[0067] 2. State Modeling Module

[0068] Including sub-modules such as attention layer and gated recurrent unit, it can build a dynamic attention integrated Markov model to model the state transition of the device under the influence of complex environment.

[0069] 3. Multi-agent reinforcement learning module

[0070] It includes three sub-agents and their reward and punishment calculation units. Each agent controls a specific maintenance operation and realizes strategy collaborative output through algorithms such as QMIX.

[0071] 4. Generate adversarial training module

[0072] Including generator network, discriminator network, WGAN-GP algorithm, etc., the equipment maintenance strategy output by the generator is optimized through adversarial training.

[0073] 5. Strategy evaluation and optimization module

[0074] Build a simulation running environment, including the PPO algorithm, and optimize the strategy offline based on the simulation execution feedback.

[0075] 6. Optimal strategy output module

[0076] The global optimal maintenance strategy solution obtained through training of the above modules is visually displayed and output for users to use.

[0077] The above functional modules are interconnected through a high-speed network, and parallel computing is performed on a GPU server cluster to achieve fast training and strategy optimization. This system architecture can efficiently implement the overall technical solution of the present invention.

[0078] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for searching for optimal maintenance strategies for equipment based on reinforcement learning, which is used to obtain the comprehensive optimal maintenance strategy that can maximize the service life of the equipment and reduce the maintenance cost through reinforcement learning training according to the operating status data of the equipment. It is characterized in that The following steps are involved: A) Construct a dynamic attention integrated Markov model, use the historical data of equipment operation to extract the multiple environmental factors that affect the equipment state transition, use the multi-head attention mechanism to dynamically calculate the weight of each environmental factor on the equipment state, and integrate the weighted environmental factors with the current state of the equipment and the historical state sequence to establish a dynamic attention integrated Markov model that describes the state transition process of the equipment in a complex environment; B) Set up a multi-agent reinforcement learning framework, treat different equipment maintenance operations as independent agents, and each agent has different state observations and reward and punishment function designs for specific subsystems of the equipment. Output the combined maintenance strategy through collaborative games, and use the multi-agent policy gradient algorithm to train each agent to achieve strategy coordination and consistency, and output the overall maintenance strategy; C) Generative adversarial models are used for strategy optimization training. The generator network is set to output the maintenance strategy, and the discriminator network judges the quality of the strategy. Through adversarial training, the generator output strategy gradually approaches the ideal optimal distribution. The maintenance expert data and the long-term benefits of the strategy are used as the discriminator input to guide the generator to output a reasonable, explainable and globally optimal final maintenance strategy.

2. According to claim 1, a method for searching for optimal equipment maintenance strategy based on reinforcement learning, It is characterized in that In the above step B), the value decomposition multi-agent reinforcement learning algorithm of COMA, QProp or QMIX is used for training.

3. The method for searching for optimal equipment maintenance strategy based on reinforcement learning according to claim 1 or 2, It is characterized in that In the above step C), an improved GAN objective function and loss such as WGAN-GP or LSGAN are used for adversarial training.

4. A method for searching for an optimal equipment maintenance strategy based on reinforcement learning according to any one of claims 1 to 3, It is characterized in that During the adversarial training process, the existing maintenance expert experience data is input into the discriminator as positive samples to guide the generator to output a more explanatory strategy.

5. A method for searching for an optimal equipment maintenance strategy based on reinforcement learning according to any one of claims 1 to 4, It is characterized in that During the adversarial training process, the long-term benefit value of the strategy is input into the discriminator as a label to guide the generator to output the optimal strategy with higher benefit.

6. A method for searching for an optimal equipment maintenance strategy based on reinforcement learning according to any one of claims 1 to 5, It is characterized in that When constructing the dynamic attention ensemble Markov model in the above step A), residual connection and layer normalization techniques are used to improve the model's expressiveness.

7. A method for searching for an optimal equipment maintenance strategy based on reinforcement learning according to any one of claims 1 to 6, It is characterized in that It further includes an offline policy evaluation optimization step, using a reinforcement learning algorithm such as PPO or SAC to further optimize the maintenance policy output by the generator in a simulation environment.

8. A method for searching for an optimal equipment maintenance strategy based on reinforcement learning according to any one of claims 1 to 7, It is characterized in that The dynamic attention integrated Markov model constructed in the above step A) and the multi-agent reward and punishment function set in step B) are designed in combination with multiple influencing factors such as energy loss, fault downtime, and maintenance cost during equipment operation.

9. A method for searching for an optimal equipment maintenance strategy based on reinforcement learning according to any one of claims 1 to 8, It is characterized in that The equipment is a generator set, a wind turbine or other industrial production equipment.

10. An equipment optimal maintenance strategy search system, It is characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 9 when being executed.

Citation Information

Cited By

  • Equipment fault maintenance management method and system based on augmented reality technology

    CN119648201A

  • Multi-agent cooperative vehicle chassis maintenance method, device, equipment and medium

    CN119941216A

  • Multi-agent deep reinforcement learning method based on human guidance

    CN119990246A

  • A Human-Guided Multi-Agent Deep Reinforcement Learning Method

    CN119990246B

  • Aircraft reinforcement learning predictive maintenance decision-making method under residual life uncertainty

    CN120013530A