A non-periodic dynamic detection and maintenance method for multi-state systems based on gaussian demand and customized PPO

Abstract

This invention discloses a non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO (Progressive Point of Action), comprising the following steps: A) System and demand modeling: including a multi-state system model, time-varying demand modeling, and definition of detection and maintenance actions; B) Continuous-time MDP modeling: constructing the dynamic detection and maintenance decision problem as a continuous-time Markov decision process, including state space, action space, state transition probabilities, reward function, and objective function; C) Customized PPO algorithm framework: designing a deep reinforcement learning framework based on PPO, adapting to the hybrid action space, and efficiently solving the MDP model. The demand modeling of this invention is accurate: by using a Gaussian process to model time-varying demand, it can simultaneously capture the expected trend, random fluctuations, and time correlation of demand. Compared with traditional constant, linear, or simplified Markov demand modeling, it is more in line with actual industrial scenarios and effectively reduces the risk of supply-demand mismatch.

Description

Technical Field

[0001] This invention relates to the field of industrial system maintenance decision-making technology, specifically to a non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO. Background Technology

[0002] Multi-state systems (MSSs) are widespread in industrial production. Their components gradually degrade due to wear, fatigue, environmental factors, etc., and the system performance changes dynamically over time. To ensure the normal operation of the system and meet user needs, detection and maintenance actions are crucial. Traditional maintenance decision-making methods for multi-state systems have many limitations and are difficult to adapt to the complex needs of actual industrial scenarios. 1. Simplified demand modeling: Existing studies often assume that user demand is constant or linearly changing, while in actual industrial scenarios, demand often exhibits complex random fluctuations. Simplified modeling can easily lead to supply and demand mismatch, causing economic losses or safety risks. Some methods that simplify time-varying demand modeling, such as discrete-state Markov processes, cannot accurately capture the uncertainty and time correlation of demand.

[0003] 2. Insufficient flexibility in time processing: Traditional methods often use time discretization processing, which relies on fixed decision intervals and is difficult to adapt flexibly to demand fluctuations. This leads to unreasonable timing of detection and maintenance, either missing the best maintenance window and causing system failure, or excessive detection and increasing operation and maintenance costs.

[0004] 3. Difficulty in solving complex systems: The state space and action space of multi-state systems grow exponentially with the number of components. Traditional dynamic programming methods face the "curse of dimensionality" and are difficult to solve the maintenance decision problem of large-scale systems. Existing reinforcement learning methods are mostly designed for purely discrete or purely continuous action spaces, which cannot adapt to the mixed action space requirements of "discrete maintenance actions + continuous decision intervals" in detection and maintenance decision-making.

[0005] 4. Imbalance between decision-making efficiency and optimization effect: For large-scale systems with multiple components and complex structures, existing methods either have extremely low solution efficiency, which cannot meet the real-time decision-making needs of actual industrial scenarios, or they cannot guarantee the optimization effect of the decision, and cannot achieve the comprehensive minimization of detection costs, maintenance costs and unmet needs costs.

[0006] Gaussian processes (GPs), as powerful time series modeling tools, can simultaneously characterize the expected value and temporal correlation of demand, making it possible to accurately model complex time-varying demands. The proximal policy optimization (PPO) algorithm in deep reinforcement learning (DRL) has advantages such as stable training, excellent performance, and strong scalability, and is suitable for complex sequence decision problems. However, no research has yet combined Gaussian process demand modeling with a customized PPO algorithm to construct a continuous-time dynamic detection and maintenance framework adapted to a hybrid action space to solve the maintenance decision problem of multi-state systems under complex time-varying demands. To this end, we propose a non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO. Summary of the Invention

[0007] The purpose of this invention is to provide a non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian requirements and customized PPO, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO, comprising the following steps: A. System and Requirements Modeling: This includes multi-state system models, time-varying requirements modeling, and definition of detection and maintenance actions; B. Continuous-time MDP modeling: The dynamic detection and maintenance decision problem is constructed as a continuous-time Markov decision process, including state space, action space, state transition probability, reward function and objective function; C. Customized PPO Algorithm Framework: Design a deep reinforcement learning framework based on PPO, adapt to the hybrid action space, and efficiently solve the MDP model. It includes the agent network architecture and network training mechanism. The agent network architecture adopts the actor-critic framework, where the actor module is a parallel architecture, including discrete actor network, continuous actor network, critic network and parallel actor network. The network training mechanism includes policy optimization, value network update, training stability guarantee and exploration rate scheduling. D. Dynamic Detection and Maintenance Decision Process: This includes initialization, demand prediction, agent training, decision execution, and dynamic adaptation. Agent training involves the interaction between the agent and the system environment. At each decision moment: the current system state s=(S, T, D) is observed, the parallel actor network outputs discrete maintenance action A and continuous decision interval τ, the detection action is executed, the actual state of the component is obtained, maintenance action A is executed, and detection cost and maintenance cost are generated. The system runs within the decision interval τ, the component degenerates according to the state transition probability, the cost of unmet demand within this interval is calculated, and empirical data (s, a, r, s') is recorded, where r is the negative value of the total cost. The parameters of the actor network and the critic network are updated based on the empirical data, and iterative training continues until the network converges.

[0009] Preferably, the multi-state system model is a multi-state system composed of N independent components, each component having M discrete states, different states corresponding to different performance capacities, and the state transitions follow a Poisson process, the residence time satisfies an exponential distribution, and the overall performance capacity of the system is obtained by aggregating the performance capacities of the components through the structure function φ(·), the series system adopts a minimization function, and the parallel system adopts a summation function.

[0010] Preferably, the time-varying demand modeling adopts Gaussian process (GP) to model user demand. The demand model is represented as GP(μ(t), k(t, t')), where μ(t) is the mean function (representing expected demand) and k(t, t') is the covariance function (representing the correlation of demand at different time points). By configuring different covariance kernel functions (such as periodic kernel, linear kernel, Marton kernel), non-stationary, multimodal, and abrupt complex demand curves can be generated to accurately capture the random fluctuation characteristics of actual demand.

[0011] Preferably, the detection action defined by the detection and maintenance action is an operation that allows for the complete observability of the component state, resulting in a fixed detection cost. The detection time is negligible relative to the task cycle. The maintenance action includes no maintenance, partial repair, and complete repair. The maintenance action a_i of component i can restore it from state s_i to state r_i (r_i ≥ s_i). The maintenance cost consists of fixed costs and variable costs related to state restoration.

[0012] Preferably, the state space consists of component state vector S, remaining task time T, and historical user demand information D, i.e., s=(S, T, D), all variables are discrete values, the state space is a discrete set, and the action space is a hybrid discrete-continuous space, including a discrete maintenance action set A and a continuous decision interval τ, i.e., a=(A, τ), where A is the combination of all component maintenance actions, and τ is the time interval from the current decision time to the next decision time.

[0013] Preferably, the state transition probability is based on the component state transition strength, calculated using the Kolmogorov differential equation to determine the probability of a component transitioning from state s_i to state s_i' within time τ. The system state transition probability is obtained by multiplying the independent transition probabilities of each component. The reward function is the negative of the total cost between adjacent decision times. The total cost includes detection cost, maintenance cost, and unmet demand cost. The unmet demand cost is calculated by dividing the decision interval into sub-intervals and integrating the difference between the system performance capacity and the demand. Simultaneously, the objective function minimizes the expected total cost within the task cycle, i.e., the optimal value function V. (s) represents the minimum expected cost over the remaining task time.

[0014] Preferably, the discrete actor network takes the system state s as input and outputs the probability distribution of discrete maintenance action A. The maintenance action of each component is determined by sampling. The continuous actor network takes the system state s as input and outputs the exponential distribution rate parameter of the decision interval τ, which follows this exponential distribution. The critic network takes the system state s and action a as input and evaluates the optimal value function V(s), providing a value benchmark for the policy update of the actor network. The parallel actor network shares the input layer and part of the hidden layer, and the output layer independently generates discrete maintenance actions and continuous decision intervals, which together constitute a complete decision.

[0015] Preferably, the strategy optimization adopts the pruning agent objective function of PPO to optimize the parameters of the discrete actor network and the continuous actor network respectively, avoiding training instability caused by excessive policy update amplitude. The loss functions of the discrete actor network and the continuous actor network both include policy gradient terms and pruning constraint terms. The relative value of actions is quantified by the advantage function. The value network updates the commentator network parameters by minimizing the mean square error between the predicted value and the target value of the value function. Training stability is guaranteed by using KL divergence hard constraints to control the difference between the old and new strategies. The Adam optimizer is used for gradient descent updates. In the early stage of training, a hard constraint of [5, 50] days is applied to the decision interval of the continuous actor network output to avoid training performance degradation due to output instability. The exploration rate of the greedy strategy of exploration rate scheduling decreases linearly from 0.9 to 0.1 with the number of training iterations, balancing exploration and utilization.

[0016] Preferably, the initialization involves constructing a multi-state system model and a Gaussian process demand model, setting component parameters, cost parameters, and task cycle, initializing the network parameters (discrete actor network, continuous actor network, critic network) and training hyperparameters (iteration count, batch size, learning rate) of the customized PPO algorithm, generating time-varying demand trajectories within the task cycle based on the Gaussian process model, and obtaining demand expectations and uncertainty information at each time point. Decision execution involves deploying the optimized strategy to the actual system after training. At each decision moment, the current system state is input, and the optimal maintenance action and the next decision interval are output in real time, achieving non-periodic dynamic detection and maintenance. Dynamic adaptation involves re-calling the Gaussian process model to update demand prediction when demand characteristics or system parameters change, and fine-tuning the trained agent based on the new demand trajectory to achieve dynamic adaptation of the decision strategy.

[0017] Preferably, the implementation environment preparation and system parameter configuration are also included. The implementation environment preparation includes both hardware and software environments. The hardware environment uses an Intel Core i7 or higher processor, at least 16GB of RAM, and an NVIDIA GeForce RTX 2080 or higher graphics card (supporting CUDA acceleration) to meet the computational requirements of large-scale system model training and inference. The software environment uses Ubuntu 18.04 LTS as the operating system, Python 3.7 or higher as the programming language, and PyTorch as the deep learning framework. Version 1.8.0 and above includes auxiliary libraries such as NumPy (numerical computation), SciPy (probability distribution and differential equation solving), Matplotlib (result visualization), and GPy (Gaussian process modeling). The system parameter configuration uses a 14-component multi-state coal transportation system as an example, including system component parameters, task and requirement parameters, cost parameters, and algorithm hyperparameters. The 14 components are divided into 5 parallel subsystems (with subsystems connected in series). The performance capacity, state transition intensity, and maintenance cost parameters (fixed cost and variable cost) of each component are shown in the attached diagram. Each component has multiple discrete states, and state transitions follow a Poisson process. The task cycle for the task and requirement parameters is 90 days. User requirements are modeled using a Gaussian process with a mean of 150 tons / day. Variance and length scale parameters are set as specified. The covariance function uses a squared exponential kernel. The cost parameter is calculated per unit time. The unit cost for unmet demand is a preset value (e.g., $100 / (ton × day)). The system-level fixed maintenance cost and detection cost are set as specified. The component maintenance cost is calculated as "fixed cost + variable cost of state recovery". The maintenance cost between adjacent states is equal. The maximum number of training iterations of the algorithm hyperparameters is a preset value (e.g., 300). The number of steps per iteration is a preset value (e.g., 2048). The parallel actor network (discrete + continuous) contains 3 hidden layers with 64, 16, and 16 neurons respectively. The first two layers are shared layers. The critic network contains 2 hidden layers with 16 neurons per layer. The learning rate is 3e-4. The pruning threshold is 0.2. The discount factor is 0.99. The advantage function estimation window is 5. The Monte Carlo simulation sample size is 1000. The exploration rate decreases linearly from 0.9 to 0.1. The continuous decision interval is constrained to [5, 50] days for the first 175 epochs of training.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Accurate demand modeling: By using Gaussian process modeling to model time-varying demand, it can simultaneously capture the expected trend, random fluctuations and time correlation of demand. Compared with traditional constant, linear or simplified Markov demand modeling, it is more in line with actual industrial scenarios and effectively reduces the risk of supply and demand mismatch.

[0019] 2. High decision-making flexibility: The continuous-time decision-making framework is constructed, and the decision interval is dynamically optimized by the algorithm. It does not rely on fixed-time discretization, and can flexibly adapt to demand fluctuations and system state changes, accurately grasp the best time for detection and maintenance, and balance failure risk and operation and maintenance costs.

[0020] 3. Hybrid Action Space Adaptation: By designing a parallel actor network, the joint optimization of discrete maintenance actions and continuous decision intervals is achieved, which solves the problem that traditional reinforcement learning algorithms are difficult to adapt to hybrid action spaces, resulting in better decision integrity and optimization effect.

[0021] 4. High efficiency in solving large-scale systems: The customized PPO algorithm can effectively cope with the "curse of dimensionality" of multi-state systems. Compared with traditional dynamic programming methods, the solution efficiency is greatly improved (the computation time is only 0.63% of that of dynamic programming in Markov requirement scenarios), and it can guarantee a near-global optimal solution (relative error of only 3.95%). It can still converge stably in large-scale complex systems such as 14-component coal transportation systems and has good scalability.

[0022] 5. Optimal Overall Cost: By jointly optimizing detection cost, maintenance cost, and unmet demand cost, the expected sum of the three is minimized. Under various demand scenarios such as constant, linear increase / decrease, Markov, and Gaussian, compared with the discrete-time PPO algorithm, it further reduces the computation time and improves the economic efficiency of operation and maintenance while ensuring the optimization effect. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the relationship between time-varying demand, system performance capacity, and unmet demand in this invention. Figure 2 This is a schematic diagram of the decision-making moments and possible actions of the present invention (including two scenarios: detection only and detection + maintenance). Figure 3 A sample diagram of the demand trajectory generated by the Gaussian process of this invention (showing random fluctuations and time correlation); Figure 4 This is a schematic diagram of the dynamics of the agent-environment interaction in this invention; Figure 5 This is a diagram of the parallel actor network framework based on a random strategy of the present invention; Figure 6 This is a schematic diagram of the three-component multi-state traffic transmission system of the present invention (components 1 and 2 are connected in parallel and then connected in series with component 3). Figure 7 The following is a diagram illustrating the convergence process of the continuous-time customized PPO under four simple demand scenarios of this invention (a-constant demand; b-increasing demand; c-decreasing demand; d-Markov demand). Figure 8This is a diagram illustrating the convergence process of the continuous-time customized PPO under the Gaussian requirement scenario of this invention. Figure 9 The diagram shows the discrete-time PPO convergence process under four simple demand scenarios of this invention. Figure 10 This is a schematic diagram of the 14-component multi-state coal transportation system of the present invention (including 5 parallel subsystems connected in series between the subsystems). Figure 11 This is a diagram illustrating the continuous-time customized PPO convergence process in the coal transportation system under Gaussian demand according to the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] In the description of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," and "connected," etc., should be interpreted broadly. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0026] Please see Figures 1-11 A non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO includes the following steps: A. System and Requirements Modeling: This includes multi-state system models, time-varying requirements modeling, and definition of detection and maintenance actions; B. Continuous-time MDP modeling: The dynamic detection and maintenance decision problem is constructed as a continuous-time Markov decision process, including state space, action space, state transition probability, reward function and objective function; C. Customized PPO Algorithm Framework: Design a deep reinforcement learning framework based on PPO, adapt to the hybrid action space, and efficiently solve the MDP model. It includes the agent network architecture and network training mechanism. The agent network architecture adopts the actor-critic framework, where the actor module is a parallel architecture, including discrete actor network, continuous actor network, critic network and parallel actor network. The network training mechanism includes policy optimization, value network update, training stability guarantee and exploration rate scheduling. D. Dynamic Detection and Maintenance Decision Process: This includes initialization, demand prediction, agent training, decision execution, and dynamic adaptation. Agent training involves the interaction between the agent and the system environment. At each decision moment: the current system state s=(S, T, D) is observed, the parallel actor network outputs discrete maintenance action A and continuous decision interval τ, the detection action is executed, the actual state of the component is obtained, maintenance action A is executed, and detection cost and maintenance cost are generated. The system runs within the decision interval τ, the component degenerates according to the state transition probability, the cost of unmet demand within this interval is calculated, and empirical data (s, a, r, s') is recorded, where r is the negative value of the total cost. The parameters of the actor network and the critic network are updated based on the empirical data, and iterative training continues until the network converges.

[0027] The multistate system model consists of N independent components, each component has M discrete states, different states correspond to different performance capacities, the state transitions follow a Poisson process, the residence time satisfies an exponential distribution, the overall performance capacity of the system is obtained by aggregating the performance capacities of the components through the structure function φ(·), the series system uses a minimization function, and the parallel system uses a summation function.

[0028] Time-varying demand modeling uses Gaussian processes (GP) to model user demand. The demand model is represented as GP(μ(t), k(t,t')), where μ(t) is the mean function (representing expected demand) and k(t, t') is the covariance function (representing the correlation of demand at different time points). By configuring different covariance kernel functions (such as periodic kernel, linear kernel, Marton kernel), non-stationary, multimodal, and abrupt complex demand curves can be generated to accurately capture the random fluctuation characteristics of actual demand.

[0029] The detection and maintenance actions are defined as operations that allow for the observation of the component's state. They generate a fixed detection cost, and the detection time is negligible relative to the task cycle. Maintenance actions include no maintenance, partial repair, and complete repair. The maintenance action a_i of component i can restore it from state s_i to state r_i (r_i ≥ s_i). The maintenance cost consists of fixed costs and variable costs related to state restoration.

[0030] The state space consists of component state vector S, remaining task time T, and historical user demand information D, i.e., s=(S,T,D). All variables are discrete values, the state space is a discrete set, and the action space is a hybrid discrete-continuous space, including a discrete set of maintenance actions A and a continuous decision interval τ, i.e., a=(A,τ), where A is the combination of all component maintenance actions, and τ is the time interval from the current decision time to the next decision time.

[0031] The state transition probability is based on the component's state transition strength. It is calculated using the Kolmogorov differential equation, determining the probability of a component transitioning from state s_i to state s_i' within time τ. The system state transition probability is obtained by multiplying the independent transition probabilities of each component. The reward function is the negative of the total cost between adjacent decision points. The total cost includes detection cost, maintenance cost, and unmet demand cost. The unmet demand cost is calculated by dividing the decision interval into sub-intervals and integrating the difference between the system's performance capacity and the demand. Simultaneously, the objective function minimizes the expected total cost within the task cycle, i.e., the optimal value function V. (s) represents the minimum expected cost over the remaining task time.

[0032] The discrete actor network takes the system state s as input and outputs the probability distribution of discrete maintenance action A. The maintenance action of each component is determined by sampling. The continuous actor network takes the system state s as input and outputs the exponential distribution rate parameter of the decision interval τ, which follows this exponential distribution. The critic network takes the system state s and action a as input and evaluates the optimal value function V(s), providing a value benchmark for the policy update of the actor network. The parallel actor network shares the input layer and part of the hidden layer, and the output layer independently generates discrete maintenance actions and continuous decision intervals, which together constitute the complete decision.

[0033] The strategy optimization adopts the pruning agent objective function of PPO to optimize the parameters of discrete and continuous actor networks respectively, avoiding training instability caused by excessive policy update amplitude. The loss functions of discrete and continuous actor networks both include policy gradient terms and pruning constraint terms. The relative value of actions is quantified by the advantage function. The value network updates the commentator network parameters by minimizing the mean square error between the predicted value and the target value of the value function. Training stability is guaranteed by using KL divergence hard constraints to control the difference between the old and new policies. The Adam optimizer is used for gradient descent updates. In the early stage of training, a hard constraint of [5, 50] days is applied to the decision interval of the continuous actor network output to avoid training performance degradation due to output instability. The exploration rate of the greedy policy of exploration rate scheduling decreases linearly from 0.9 to 0.1 with the number of training iterations, balancing exploration and utilization.

[0034] Initialization involves constructing a multi-state system model and a Gaussian process demand model, setting component parameters, cost parameters, and task cycle, and initializing the network parameters (discrete actor network, continuous actor network, critic network) and training hyperparameters (number of iterations, batch size, learning rate) of the customized PPO algorithm. Demand prediction generates time-varying demand trajectories within the task cycle based on the Gaussian process model, obtaining demand expectations and uncertainty information at each time point. Decision execution involves deploying the optimized strategy to the actual system after training. At each decision moment, the current system state is input, and the optimal maintenance action and the next decision interval are output in real time, achieving non-periodic dynamic detection and maintenance. Dynamic adaptation involves re-calling the Gaussian process model to update demand prediction when demand characteristics or system parameters change, and fine-tuning the trained agent based on the new demand trajectory to achieve dynamic adaptation of the decision strategy.

[0035] Initialize discrete policy network parameters ; Initialize continuous policy network parameters ; Initialize value network parameters ; Initialize discrete old policy network parameters ; Initialize the network parameters of the continuous old policy ; for

[0036] Initialize system state Remaining task time User needs ; for do According to the old discrete strategy and continuous old strategies choose and ; Execute discrete actions And calculate the total cost incurred during that period. ; Update environment status ; The trajectory data is stored in the experience pool; if then Initialize environment state ; end if end for Randomly sample small batches of experience data from the experience pool and calculate... ; For do Updated based on (1) to (8) , , ; end for end for Policy optimization aims to minimize the loss function and uses a probability ratio truncation mechanism and advantage function estimation to ensure the stability of the update process. For discrete stochastic policies... Its loss function is defined as: (1) in The ratio of the probabilities of the new and old policies on discrete actions is used to measure the new policy before and after the update. and old strategies The degree of difference. Its expression is as follows: (2) The function uses a truncation mechanism; the specific expression is as follows: (3) Continuous random strategy The loss function is: (4) Similarly The probability ratio of a continuous random policy is calculated using the following formula: (5) (6) In the above loss function, It is the advantage estimation function, used to calculate the relative benefit of performing a certain action relative to the average level in a given state.

[0037] (7) The commentator network updates by minimizing the value function loss, and its update form is as follows: (8) The task period for this case study is 90 days. Component parameters are shown in Table 1, and state transition strengths are shown in Table 2. The penalty cost per unit supply-demand gap is... Yuan (per day × tons). System fixed maintenance cost. Yuan, system observation cost Yuan.

[0038] Table 1 Parameters of each component (Performance capacity: tons / day, Cost: Yuan)

[0039] Table 2. State transition intensity of each component (year) )

[0040] This also includes implementation environment preparation and system parameter configuration. Implementation environment preparation includes both hardware and software environments. The hardware environment requires an Intel Core i7 or higher processor, at least 16GB of RAM, and an NVIDIA GeForce RTX 2080 or higher graphics card (supporting CUDA acceleration) to meet the computational needs of large-scale system model training and inference. The software environment uses Ubuntu 18.04 LTS as the operating system, Python 3.7 or higher as the programming language, and PyTorch as the deep learning framework. Version 1.8.0 and above includes auxiliary libraries such as NumPy (numerical computation), SciPy (probability distribution and differential equation solving), Matplotlib (result visualization), and GPy (Gaussian process modeling). The system parameter configuration uses a 14-component multi-state coal transportation system as an example, including system component parameters, task and requirement parameters, cost parameters, and algorithm hyperparameters. The 14 components are divided into 5 parallel subsystems (with subsystems connected in series). The performance capacity, state transition intensity, and maintenance cost parameters (fixed cost and variable cost) of each component are shown in the attached diagram. Each component has multiple discrete states, and state transitions follow a Poisson process. The task cycle for the task and requirement parameters is 90 days. User requirements are modeled using a Gaussian process with a mean of 150 tons / day. Variance and length scale parameters are set as specified. The covariance function uses a squared exponential kernel. The cost parameter is calculated per unit time. The unit cost for unmet demand is a preset value (e.g., $100 / (ton × day)). The system-level fixed maintenance cost and detection cost are set as specified. The component maintenance cost is calculated as "fixed cost + variable cost of state recovery". The maintenance cost between adjacent states is equal. The maximum number of training iterations of the algorithm hyperparameters is a preset value (e.g., 300). The number of steps per iteration is a preset value (e.g., 2048). The parallel actor network (discrete + continuous) contains 3 hidden layers with 64, 16, and 16 neurons respectively. The first two layers are shared layers. The critic network contains 2 hidden layers with 16 neurons per layer. The learning rate is 3e-4. The pruning threshold is 0.2. The discount factor is 0.99. The advantage function estimation window is 5. The Monte Carlo simulation sample size is 1000. The exploration rate decreases linearly from 0.9 to 0.1. The continuous decision interval is constrained to [5, 50] days for the first 175 epochs of training.

[0041] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO, characterized in that: Includes the following steps: A. System and Requirements Modeling: This includes multi-state system models, time-varying requirements modeling, and definition of detection and maintenance actions; B. Continuous-time MDP modeling: The dynamic detection and maintenance decision problem is constructed as a continuous-time Markov decision process, including state space, action space, state transition probability, reward function and objective function; C. Customized PPO Algorithm Framework: Design a deep reinforcement learning framework based on PPO, adapt to the hybrid action space, and efficiently solve the MDP model. It includes the agent network architecture and network training mechanism. The agent network architecture adopts the actor-critic framework, where the actor module is a parallel architecture, including discrete actor network, continuous actor network, critic network and parallel actor network. The network training mechanism includes policy optimization, value network update, training stability guarantee and exploration rate scheduling. D. Dynamic Detection and Maintenance Decision Process: This includes initialization, demand prediction, agent training, decision execution, and dynamic adaptation. Agent training involves the interaction between the agent and the system environment. At each decision moment: the current system state s=(S, T,D) is observed, the parallel actor network outputs discrete maintenance action A and continuous decision interval τ, the detection action is executed, the actual state of the component is obtained, maintenance action A is executed, and detection cost and maintenance cost are generated. The system runs within the decision interval τ, the component degenerates according to the state transition probability, the cost of unmet demand within this interval is calculated, and empirical data (s, a, r, s') is recorded, where r is the negative value of the total cost. The parameters of the actor network and the critic network are updated based on the empirical data, and iterative training continues until the network converges.

2. The non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO as described in claim 1, characterized in that: The multi-state system model consists of N independent components, each component having M discrete states. Different states correspond to different performance capacities, and the state transitions follow a Poisson process. The residence time follows an exponential distribution. The overall performance capacity of the system is obtained by aggregating the performance capacities of the components using the structure function φ(·). The series system uses a minimization function, and the parallel system uses a summation function.

3. The non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO as described in claim 1, characterized in that: The time-varying demand modeling adopts Gaussian process (GP) to model user demand. The demand model is represented as GP(μ(t), k(t, t')), where μ(t) is the mean function and k(t, t') is the covariance function. By configuring different covariance kernel functions, non-stationary, multimodal, and abrupt complex demand curves can be generated to accurately capture the random fluctuation characteristics of actual demand.

4. The non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO as described in claim 1, characterized in that: The detection and maintenance actions defined are operations that allow for the complete observation of component states, resulting in a fixed detection cost. The detection time is negligible relative to the task cycle. Maintenance actions include no maintenance, partial repair, and complete repair. The maintenance action a_i of component i can restore it from state s_i to state r_i (r_i ≥ s_i). The maintenance cost consists of fixed costs and variable costs related to state restoration.

5. The non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO as described in claim 1, characterized in that: The state space consists of component state vector S, remaining task time T, and historical user demand information D, i.e., s=(S, T, D). All variables are discrete values, the state space is a discrete set, and the action space is a hybrid discrete-continuous space, including a discrete maintenance action set A and a continuous decision interval τ, i.e., a=(A, τ), where A is the combination of all component maintenance actions, and τ is the time interval from the current decision time to the next decision time.

6. The non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO as described in claim 1, characterized in that: The state transition probability is based on the component state transition strength. It is calculated using the Kolmogorov differential equation to determine the probability of a component transitioning from state s_i to state s_i' within time τ. The system state transition probability is obtained by multiplying the independent transition probabilities of each component. The reward function is the negative of the total cost between adjacent decision times. The total cost includes detection cost, maintenance cost, and unmet demand cost. The unmet demand cost is calculated by dividing the decision interval into sub-intervals and integrating the difference between the system performance capacity and the demand. Simultaneously, the objective function minimizes the expected total cost within the task cycle, i.e., the optimal value function V. (s) represents the minimum expected cost over the remaining task time.

7. The non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO as described in claim 1, characterized in that: The discrete actor network takes the system state s as input and outputs the probability distribution of discrete maintenance action A. The maintenance action of each component is determined by sampling. The continuous actor network takes the system state s as input and outputs the exponential distribution rate parameter of the decision interval τ, which follows this exponential distribution. The critic network takes the system state s and action a as input and evaluates the optimal value function V(s), providing a value benchmark for the policy update of the actor network. The parallel actor network shares the input layer and part of the hidden layer, and the output layer independently generates discrete maintenance actions and continuous decision intervals, which together constitute a complete decision.

8. The non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO as described in claim 1, characterized in that: The strategy optimization adopts the pruning proxy objective function of PPO to optimize the parameters of discrete actor networks and continuous actor networks respectively, avoiding training instability caused by excessive policy update amplitude. The loss functions of discrete actor networks and continuous actor networks both include policy gradient terms and pruning constraint terms. The relative value of actions is quantified by the advantage function. The value network updates the commentator network parameters by minimizing the mean square error between the predicted value and the target value of the value function. Training stability is guaranteed by using KL divergence hard constraints to control the difference between the old and new policies. The Adam optimizer is used for gradient descent updates. In the early stage of training, a hard constraint of [5, 50] days is applied to the decision interval of the continuous actor network output to avoid training performance degradation due to output instability. The exploration rate of the greedy policy of exploration rate scheduling decreases linearly from 0.9 to 0.1 with the number of training iterations, balancing exploration and utilization.

9. The non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO as described in claim 1, characterized in that: The initialization involves constructing a multi-state system model and a Gaussian process demand model, setting component parameters, cost parameters, and task cycles, initializing the network parameters and training hyperparameters of the customized PPO algorithm, generating time-varying demand trajectories within the task cycle based on the Gaussian process model, and obtaining demand expectations and uncertainty information at each time point. Decision execution involves deploying the optimized strategy to the actual system after training. At each decision moment, the current system state is input, and the optimal maintenance action and the next decision interval can be output in real time, realizing non-periodic dynamic detection and maintenance. Dynamic adaptation involves re-calling the Gaussian process model to update demand prediction when demand characteristics or system parameters change, and fine-tuning the trained agent based on the new demand trajectory to achieve dynamic adaptation of the decision strategy.

10. The non-periodic dynamic detection and maintenance method for multi-state systems based on Gaussian demand and customized PPO as described in claim 1, characterized in that: This also includes preparation of the implementation environment and configuration of system parameters. The implementation environment preparation includes both hardware and software environments. The hardware environment requires an Intel Core i7 or higher processor, at least 16GB of RAM, and an NVIDIA GeForce RTX 2080 or higher graphics card to meet the computational needs of large-scale system model training and inference. The software environment uses Ubuntu 18.04 LTS as the operating system, Python 3.7 or higher as the programming language, and PyTorch as the deep learning framework. Version 1.8.0 and above, with auxiliary libraries including NumPy, SciPy, Matplotlib, and GPy, are used as examples of system parameter configurations. These parameters include system component parameters, task and requirement parameters, cost parameters, and algorithm hyperparameters. The 14 components are divided into 5 parallel subsystems. The performance capacity, state transition strength, and maintenance cost parameters of each component are shown in the attached diagram. Each component has multiple discrete states, and state transitions follow a Poisson process. The task cycle for the task and requirement parameters is 90 days. User requirements are modeled using a Gaussian process with a mean of 150 tons / day. Variance and length scale parameters are set as specified. The covariance function uses a squared exponential kernel. The cost parameter represents the cost per unit time unit of unmet demand. The system-level fixed maintenance cost and detection cost are set as preset values. The component maintenance cost is calculated as "fixed cost + variable cost of state recovery". The maintenance cost between adjacent states is equal. The maximum number of training iterations of the algorithm hyperparameters is a preset value, and the number of steps in each iteration is a preset value. The parallel actor network contains 3 hidden layers with 64, 16 and 16 neurons respectively. The first two layers are shared layers. The critic network contains 2 hidden layers with 16 neurons in each layer. The learning rate is 3e-4, the pruning threshold is 0.2, the discount factor is 0.99, the advantage function estimation window is 5, the Monte Carlo simulation sample size is 1000, and the exploration rate decreases linearly from 0.9 to 0.

1. The continuous decision interval is constrained to [5,50] days for the first 175 epochs of training.

Images