Reinforcement learning-based device maintenance strategy dynamic optimization method and system

Abstract

This invention relates to the field of equipment maintenance technology, and more particularly to a method and system for dynamically optimizing equipment maintenance strategies based on reinforcement learning. The method extracts equipment state features and deduces evolution paths, then clusters the data and calculates transition probabilities and difficulty coefficients to generate a candidate action set and construct a reward function. A policy network is trained using the state and actions as input, and after iterative optimization, a decision network is obtained to output maintenance decisions. This invention can dynamically optimize maintenance strategies, improving maintenance efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of equipment maintenance technology, and in particular to a method and system for dynamic optimization of equipment maintenance strategies based on reinforcement learning. Background Technology

[0002] In the field of industrial equipment maintenance, traditional maintenance strategies mainly rely on preventive maintenance based on fixed rules and predictive maintenance based on historical failure data. Preventive maintenance typically sets fixed maintenance cycles based on equipment manufacturer recommendations or experience, and performs repairs or replaces parts according to plan regardless of the actual operating condition of the equipment.

[0003] Existing methods lack the ability to deeply explore and proactively intervene in the inherent dynamic laws governing equipment state evolution. The evolution of equipment from a healthy state to a faulty state is not a single linear path, but rather involves multiple potential paths and evolution rates. Conventional predictive models typically focus only on predicting the final failure point or modeling a single evolution trend, ignoring the diversity and uncertainty of state evolution paths. This results in reactive maintenance strategies or those based on single-scenario assumptions, failing to dynamically assess the potential impact of different intervention actions (i.e., maintenance decisions) on the long-term evolution trajectory of the equipment during operation. Consequently, it is difficult to achieve a dynamic and globally optimal balance between maintenance costs, equipment availability, and risk control. Summary of the Invention

[0004] This invention provides a method and system for dynamic optimization of equipment maintenance strategies based on reinforcement learning, which can solve the problems in the prior art.

[0005] A first aspect of the present invention provides a method for dynamically optimizing equipment maintenance strategies based on reinforcement learning, comprising:

[0006] The system acquires the operating status data of the target device and performs multi-scale feature extraction to generate a state vector. Based on the state vector, it infers and generates multiple evolution paths, calculates the similarity of the evolution paths and clusters them into multiple path clusters, assigns cluster weights to each path cluster and extracts the evolution rate, and calculates the inter-cluster transition probability.

[0007] The transition difficulty coefficient from each path cluster to the target path cluster is calculated based on the evolution rate and inter-cluster transition probability. A candidate action set is generated based on the transition difficulty coefficient, and a mapping relationship between each candidate action in the candidate action set and the path cluster is established.

[0008] For each candidate action in the candidate action set, the associated path cluster corresponding to the candidate action is determined based on the mapping relationship. The change in cluster weight of the associated path cluster and the path cluster transfer trajectory after executing the candidate action are calculated based on the inter-cluster transfer probability of the associated path cluster. A reward function is constructed based on the change in cluster weight and the path cluster transfer trajectory.

[0009] The policy network is trained with the state vector as the state, the candidate action set as the action space, and the reward function as the optimization objective. The state vector is updated according to the path cluster transition trajectory, and the cluster weights and inter-cluster transition probabilities are updated according to the change in cluster weights. The network is iteratively trained until convergence to obtain the decision network. The operating status data of the target device is input into the decision network, and the maintenance decision is output and executed.

[0010] The process involves acquiring the target device's operational status data and performing multi-scale feature extraction to generate a state vector. Based on this state vector, multiple evolution paths are derived. The similarity of these evolution paths is calculated, and they are clustered into multiple path clusters. Cluster weights are assigned to each path cluster, and the evolution rate is extracted. The inter-cluster transition probability is calculated, including:

[0011] Wavelet decomposition is performed on the operational status data at multiple time scales to obtain components at each scale. Time-domain statistics and frequency-domain energy distributions are calculated for each scale component. The degradation contribution of each scale component is calculated based on the time-domain statistics and frequency-domain energy distributions. The scale components are weighted and fused according to the degradation contribution to generate a state vector.

[0012] The degradation contribution of each scale component is normalized to obtain the inference weight corresponding to each scale component. The inference interval is calculated based on the time domain statistics, and the inference amplitude is calculated based on the frequency domain energy distribution. Multiple evolution paths are generated by coordinating the inference weight, inference interval and inference amplitude corresponding to each scale component.

[0013] Morphological features and evolution speed features are extracted from multiple evolution paths. Morphological similarity between evolution paths is calculated based on morphological features, and velocity similarity between evolution paths is calculated based on evolution speed features. The morphological similarity and velocity similarity are weighted to obtain a comprehensive similarity.

[0014] A similarity matrix is ​​constructed based on the comprehensive similarity, and spectral clustering is performed to obtain multiple path clusters. Cluster weights are assigned to each path cluster, the evolution rate is extracted, and the inter-cluster transition probability is calculated.

[0015] The degradation contribution of each scale component is normalized to obtain the inference weight corresponding to each scale component. The inference interval is calculated based on time-domain statistics, and the inference amplitude is calculated based on frequency-domain energy distribution. Multiple evolution paths are generated by co-inferring the inference weight, inference interval, and inference amplitude corresponding to each scale component, including:

[0016] The degradation contribution of each scale component is normalized to obtain the inference weight corresponding to each scale component;

[0017] The fluctuation amplitude and frequency of the state vector are extracted based on time-domain statistics. The rate of change is calculated based on the fluctuation amplitude and frequency and mapped to the inference interval.

[0018] Based on the frequency domain energy distribution, the energy proportion of each frequency band is extracted, and the energy concentration is calculated and mapped to the extrapolation amplitude according to the energy proportion.

[0019] The time step corresponding to each scale component is obtained by weighting the inference interval based on the inference weight, and the numerical change corresponding to each scale component is obtained by weighting the inference amplitude based on the inference weight.

[0020] Generate a random perturbation sequence that follows a normal distribution. Superimpose the random perturbation sequence with the numerical change to obtain the perturbed numerical change. Advance the state vector according to the time step and update the state vector value according to the perturbed numerical change. Generate multiple evolution paths through iterative advancement.

[0021] The path variance among multiple evolution paths is calculated as the path dispersion. The dispersion deviation is calculated based on the path dispersion and a preset dispersion threshold. The variance parameter of the random perturbation sequence is adjusted according to the dispersion deviation. The random perturbation sequence is regenerated using the adjusted variance parameter, and the evolution path generation process is re-executed. The adjustment is iteratively repeated until the path dispersion converges within the dispersion threshold range.

[0022] The transition difficulty coefficient from each path cluster to the target path cluster is calculated based on the evolution rate and inter-cluster transition probability. A candidate action set is generated based on the transition difficulty coefficient, and the mapping relationship between each candidate action in the candidate action set and the path cluster is established, including:

[0023] The evolution rate of each path cluster and the inter-cluster transition probability between each path cluster and the target path cluster are mapped to a two-dimensional transition space. A transition potential energy field is constructed in the two-dimensional transition space with the target path cluster as the center. The potential energy value of each path cluster in the transition potential energy field is calculated as the transition difficulty coefficient from each path cluster to the target path cluster.

[0024] In the potential energy field, regions with abnormal potential energy gradients are identified as transfer obstacle zones. The distance from each path cluster to the transfer obstacle zone is calculated, and the transfer difficulty coefficient is corrected based on the distance to obtain the corrected transfer difficulty coefficient.

[0025] The path cluster with the highest difficulty coefficient of the correction transfer is selected as the path cluster to be optimized. The spatial distribution features of the transfer obstacle area are extracted, and candidate actions to bypass the transfer obstacle area are generated based on the spatial distribution features to form a candidate action set.

[0026] Calculate the position offset vector generated by each candidate action in the candidate action set in the transferred potential energy field. Calculate the similarity between the position offset vector and the vector pointing from the path cluster to the target path cluster. Based on the vector similarity, establish the mapping relationship between each candidate action in the candidate action set and the path cluster to be optimized.

[0027] Based on the inter-cluster transition probability of associated path clusters, the change in cluster weights and the path cluster transition trajectory after executing candidate actions are calculated. The reward function is then constructed based on the change in cluster weights and the path cluster transition trajectory, including:

[0028] Extract the inter-cluster transition probability of the associated path cluster, simulate the execution of candidate actions to adjust the inter-cluster transition probability of the associated path cluster to obtain the adjusted inter-cluster transition probability, recalculate the potential energy value of the associated path cluster in the transfer potential energy field based on the adjusted inter-cluster transition probability, and calculate the difference in potential energy value before and after the execution of candidate actions as the change in cluster weight.

[0029] Extract the starting coordinates of the associated path cluster before executing the candidate action and the target coordinates after executing the candidate action in the transfer potential energy field. Connect the starting coordinates and the target coordinates to form the path cluster transfer trajectory. Extract the boundary coordinates of the transfer obstacle area in the transfer potential energy field. Calculate the overlap length between the path cluster transfer trajectory and the boundary coordinates of the transfer obstacle area as the obstacle crossing amount.

[0030] The potential energy reduction benefit is calculated based on the cluster weight change, and the obstacle crossing penalty is calculated based on the obstacle crossing amount. The potential energy reduction benefit and the obstacle crossing penalty are weighted and summed to construct the reward function.

[0031] The policy network is trained using a state vector as the state, a candidate action set as the action space, and a reward function as the optimization objective. The state vector is updated based on the path cluster transition trajectories, and the cluster weights and inter-cluster transition probabilities are updated based on changes in cluster weights. Iterative training until convergence yields the decision network, which includes:

[0032] The target coordinates of the path cluster transfer trajectory are extracted and the current coordinates of the corresponding path cluster in the state vector are calculated to obtain the coordinate difference. The coordinate difference is then superimposed on the coordinate features of the corresponding path cluster to obtain the updated coordinate features.

[0033] Calculate the potential energy gradient of the corresponding path cluster in the transferred potential energy field, and then replace the original path cluster features in the state vector with the updated coordinate features and the potential energy gradient to obtain the updated state vector.

[0034] The cluster weights of the corresponding path clusters are adjusted according to the change in cluster weights to obtain the updated cluster weights. The inter-cluster transition strength is calculated based on the updated cluster weights and the spatial distance between path clusters and normalized to obtain the updated inter-cluster transition probability. An inter-cluster transition probability matrix containing all path clusters is constructed and the largest eigenvalue is extracted as the transition stability index.

[0035] The updated state vector and transition stability index are input into the policy network to calculate the reward function value of the current round and update the policy network parameters. The reward function values ​​of multiple rounds are accumulated to construct a reward sequence and then smoothed to obtain a smooth reward curve.

[0036] The rate of change of the smoothed reward curve is extracted as the convergence rate. When the convergence rate is less than a preset rate threshold, the policy network training is considered to have converged and the decision network is output.

[0037] Input the target device's operational status data into the decision network, output maintenance decisions, and execute them, including:

[0038] Collect real-time operating status data of the target equipment and extract equipment operating parameter features, equipment health status features and equipment historical maintenance features to construct an operating status feature vector;

[0039] The operating status feature vector is input into the decision network. The decision network extracts the correlation pattern between the equipment operating parameter features and the equipment health status features. Based on the correlation pattern, the state deviation of the equipment operating parameter features and the health decay of the equipment health status features are calculated. The state deviation and health decay are fused to generate equipment risk assessment indicators.

[0040] Based on the equipment risk assessment indicators and the equipment's historical maintenance characteristics, the expected benefit value and execution cost value corresponding to each maintenance action in the candidate action set are calculated. A comprehensive score is calculated based on the expected benefit value and execution cost value. The maintenance action with the highest comprehensive score is selected as the target maintenance action. The target maintenance action is converted into a maintenance task instruction and sent to the target equipment to perform the maintenance operation.

[0041] A second aspect of this invention provides a dynamic optimization system for equipment maintenance strategies based on reinforcement learning, comprising:

[0042] The state evolution unit is used to acquire the operating state data of the target device and perform multi-scale feature extraction to generate a state vector. Based on the state vector, multiple evolution paths are generated, the similarity of the evolution paths is calculated and they are clustered into multiple path clusters. Cluster weights are assigned to each path cluster and the evolution rate is extracted. The inter-cluster transition probability is calculated.

[0043] The transfer mapping unit is used to calculate the transfer difficulty coefficient from each path cluster to the target path cluster based on the evolution rate and inter-cluster transfer probability, generate a candidate action set based on the transfer difficulty coefficient, and establish the mapping relationship between each candidate action in the candidate action set and the path cluster.

[0044] The function construction unit is used to determine the associated path clusters corresponding to each candidate action in the candidate action set based on the mapping relationship, calculate the change in cluster weights and the path cluster transfer trajectory after executing the candidate action based on the inter-cluster transfer probability of the associated path clusters, and construct the reward function based on the change in cluster weights and the path cluster transfer trajectory.

[0045] The decision execution unit is used to train the policy network with the state vector as the state, the candidate action set as the action space, and the reward function as the optimization objective. It updates the state vector according to the path cluster transition trajectory, updates the cluster weight and inter-cluster transition probability according to the change in cluster weight, and iterates the training until convergence to obtain the decision network. It inputs the operating status data of the target device into the decision network, outputs maintenance decisions, and executes them.

[0046] A third aspect of the present invention provides an electronic device, comprising:

[0047] processor;

[0048] Memory used to store processor-executable instructions;

[0049] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0050] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0051] This method enables dynamic optimization of equipment maintenance strategies, significantly improving the adaptability and accuracy of maintenance decisions. Through multi-scale feature extraction and evolution path deduction, it comprehensively captures the complex dynamics and potential evolution trends of equipment operating states, providing a deep state representation for subsequent decisions. Path clustering and weight allocation mechanisms effectively extract core evolution patterns in the high-dimensional state space, reducing decision complexity. Simultaneously, by quantifying the correlation and transformation characteristics between different state clusters through evolution rate and transition probability, it ensures a close correlation between maintenance actions and the actual evolution patterns of the equipment state. By calculating the cluster weight changes and transition trajectories after executing actions, a reward function is constructed, transforming abstract maintenance effects into quantifiable and evaluable action effect features. This accurately reflects the actual intervention effect and long-term impact of different maintenance actions on equipment state evolution. A reinforcement learning training framework is constructed using state vectors, action space, and reward functions. Based on real-time operating state data, it can automatically output optimal maintenance decisions, ensuring equipment reliability and operating efficiency while optimizing maintenance resource allocation, reducing unplanned downtime, and ultimately achieving the goals of reducing total lifecycle maintenance costs and improving overall equipment performance. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the dynamic optimization method for equipment maintenance strategies based on reinforcement learning, as described in an embodiment of the present invention.

[0053] Figure 2 This is a flowchart of the state evolution path generation and iterative adjustment process in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0056] Figure 1 This is a flowchart illustrating the dynamic optimization method for equipment maintenance strategies based on reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the dynamic optimization method for equipment maintenance strategies based on reinforcement learning includes:

[0057] The system acquires the operating status data of the target device and performs multi-scale feature extraction to generate a state vector. Based on the state vector, it infers and generates multiple evolution paths, calculates the similarity of the evolution paths and clusters them into multiple path clusters, assigns cluster weights to each path cluster and extracts the evolution rate, and calculates the inter-cluster transition probability.

[0058] The transition difficulty coefficient from each path cluster to the target path cluster is calculated based on the evolution rate and inter-cluster transition probability. A candidate action set is generated based on the transition difficulty coefficient, and a mapping relationship between each candidate action in the candidate action set and the path cluster is established.

[0059] For each candidate action in the candidate action set, the associated path cluster corresponding to the candidate action is determined based on the mapping relationship. The change in cluster weight of the associated path cluster and the path cluster transfer trajectory after executing the candidate action are calculated based on the inter-cluster transfer probability of the associated path cluster. A reward function is constructed based on the change in cluster weight and the path cluster transfer trajectory.

[0060] The policy network is trained with the state vector as the state, the candidate action set as the action space, and the reward function as the optimization objective. The state vector is updated according to the path cluster transition trajectory, and the cluster weights and inter-cluster transition probabilities are updated according to the change in cluster weights. The network is iteratively trained until convergence to obtain the decision network. The operating status data of the target device is input into the decision network, and the maintenance decision is output and executed.

[0061] The process involves acquiring the target device's operational status data and performing multi-scale feature extraction to generate a state vector. Based on this state vector, multiple evolution paths are derived. The similarity of these evolution paths is calculated, and they are clustered into multiple path clusters. Cluster weights are assigned to each path cluster, and the evolution rate is extracted. The inter-cluster transition probability is calculated, including:

[0062] Wavelet decomposition is performed on the operational status data at multiple time scales to obtain components at each scale. Time-domain statistics and frequency-domain energy distributions are calculated for each scale component. The degradation contribution of each scale component is calculated based on the time-domain statistics and frequency-domain energy distributions. The scale components are weighted and fused according to the degradation contribution to generate a state vector.

[0063] The degradation contribution of each scale component is normalized to obtain the inference weight corresponding to each scale component. The inference interval is calculated based on the time domain statistics, and the inference amplitude is calculated based on the frequency domain energy distribution. Multiple evolution paths are generated by coordinating the inference weight, inference interval and inference amplitude corresponding to each scale component.

[0064] Morphological features and evolution speed features are extracted from multiple evolution paths. Morphological similarity between evolution paths is calculated based on morphological features, and velocity similarity between evolution paths is calculated based on evolution speed features. The morphological similarity and velocity similarity are weighted to obtain a comprehensive similarity.

[0065] A similarity matrix is ​​constructed based on the comprehensive similarity, and spectral clustering is performed to obtain multiple path clusters. Cluster weights are assigned to each path cluster, the evolution rate is extracted, and the inter-cluster transition probability is calculated.

[0066] The system collects operational status data from sensors, including multi-dimensional information such as vibration signals, temperature data, and current signals. The collected operational status data is then decomposed using wavelet decomposition across multiple time scales. A suitable wavelet basis function, such as the Daubechies wavelet or Morlet wavelet, is selected to perform multi-level decomposition of the original signal. After decomposition, approximate and detail components at different scales are obtained. The approximate components reflect the low-frequency trends of the signal, while the detail components reflect the high-frequency fluctuations. Setting the decomposition level to 5 levels yields 5 detail components at 5 scales and 1 final approximate component, for a total of 6 scale components.

[0067] Time-domain statistics are calculated for each scale component. These statistics include characteristic parameters such as mean, variance, kurtosis, kurtosis, and skewness. The mean reflects the central tendency of the signal, the variance reflects the dispersion, the kurtosis reflects the impulse characteristics, the kurtosis reflects the sharpness of the signal distribution, and the skewness reflects the symmetry of the signal distribution. For the i-th scale component, its mean, standard deviation, and kurtosis coefficient are calculated. Simultaneously, frequency domain analysis is performed on each scale component. The time-domain signal is converted to the frequency domain using a Fast Fourier Transform (FFT) to obtain the frequency domain energy distribution. The energy proportion of each frequency band is calculated, with a focus on energy changes within a characteristic frequency range, which is typically associated with the equipment's failure modes.

[0068] The degradation contribution of each scale component is calculated based on time-domain statistics and frequency-domain energy distribution. The degradation contribution reflects the degree of influence of that scale component on the overall degradation state of the equipment. For the i-th scale component, its degradation contribution comprehensively considers the rate of change of time-domain characteristics and the rate of increase of frequency-domain energy. The time-domain statistics are compared with the baseline values ​​under the equipment's healthy state to calculate the degree of deviation. The frequency-domain energy distribution is compared with the energy distribution under normal operating conditions to identify energy growth in abnormal frequency bands. The calculation of the degradation contribution considers both abrupt change characteristics and trend characteristics. Abrupt change characteristics are reflected through kurtosis and kurtosis, while trend characteristics are reflected through changes in mean and variance.

[0069] The degradation contribution is normalized so that the sum of the degradation contributions of all scale components is 1. Key feature parameters are extracted for each scale component, with 8 feature parameters extracted for each scale component, including 4 time-domain statistics and 4 frequency-domain features. The normalized degradation contribution is used as a weighting coefficient to weight the feature parameters of the corresponding scale. The weighted feature parameters are arranged in scale order to form a multi-dimensional state vector with 48 dimensions (6 scales × 8 features). The state vector not only contains the original feature information but also incorporates the degree of contribution of each scale to the degradation state.

[0070] The degradation contribution of each scale component is normalized to obtain the inference weight corresponding to each scale component. The inference weight represents the degree of influence of each scale component on the overall evolution path during the future state inference process. Scale components with high degradation contribution receive larger inference weights and have a more significant impact on the path morphology during the inference process. The inference interval is calculated based on time-domain statistics, and the inference interval determines the time step of state inference. The rate of change of time-domain statistics is analyzed. Fast-changing statistics are used with smaller inference intervals to capture rapid changes, while slow-changing statistics are used with larger inference intervals to improve computational efficiency. The basic inference interval is set to 10 sampling periods and is dynamically adjusted according to the rate of change of variance.

[0071] The extrapolation amplitude is calculated based on the frequency domain energy distribution. This amplitude determines the change in state variables at each extrapolation step. The fluctuation characteristics of the frequency domain energy distribution are analyzed; frequency bands with drastic energy fluctuations correspond to larger extrapolation amplitudes, while frequency bands with relatively stable energy correspond to smaller extrapolation amplitudes. A mapping relationship between frequency domain energy changes and state evolution amplitudes is established through historical data statistics. For a given characteristic parameter, its extrapolation amplitude is positively correlated with the energy standard deviation of the corresponding frequency band.

[0072] Multiple evolution paths are generated through collaborative extrapolation based on the extrapolation weights, intervals, and magnitudes corresponding to each scale component. The extrapolation process employs the Monte Carlo method, introducing random disturbance terms to simulate uncertainties in actual operation. For each extrapolation step, the state value at the next moment is calculated based on the current state, extrapolation weights, intervals, and magnitudes. The random disturbance term follows a normal distribution, with its standard deviation proportional to the extrapolation magnitude. By changing the random seed, 30 different evolution paths are generated. Each evolution path contains 200 extrapolation steps, covering the entire process of the equipment progressing from its current state to a potential failure state.

[0073] Morphological and evolution rate features are extracted from the generated multiple evolution paths. Morphological features describe the path's geometry, including curvature, inflection point locations, and fluctuation amplitude. A discrete curvature calculation method is used to calculate the local curvature value for each point on the path. Curvature abrupt change points are identified as key inflection points, reflecting the stage-specific characteristics of equipment state evolution. The overall fluctuation amplitude of the path is calculated to measure the degree to which the path deviates from the average trend. Evolution rate features reflect the rate of change of the path over time. The Euclidean distance between adjacent points on the path is calculated, representing the amount of state change per unit time. The average evolution rate and variance of the entire path are statistically analyzed; the average evolution rate reflects the overall rate of degradation, and the variance reflects the stationarity of the evolution process.

[0074] The morphological similarity between evolutionary paths is calculated based on morphological features. For paths Pm and Pn, their curvature sequences, inflection point sequences, and fluctuation amplitudes are extracted. A dynamic time warping algorithm is used to calculate the similarity between curvature sequences, which can handle different path lengths and time axis alignment issues. The overlap of inflection point positions is calculated; paths with close inflection point positions have similar stage-specific evolutionary characteristics. After normalizing the fluctuation amplitudes, the difference is calculated. The morphological similarity is then obtained by combining the curvature similarity, inflection point overlap, and fluctuation difference.

[0075] The velocity similarity between evolution paths is calculated based on evolution velocity characteristics. The average evolution velocity and the variance of the evolution velocity of two paths are compared. Similar average evolution velocities indicate that the overall degradation rates of the two paths are similar, while similar velocity variances indicate similar stationarity of the degradation process. Velocity similarity is calculated using a Gaussian kernel function, with the velocity difference as the input to the kernel function; the smaller the velocity difference, the higher the similarity.

[0076] The overall similarity is obtained by weighting morphological similarity and velocity similarity. The weight for morphological similarity is set to 0.6, and the weight for velocity similarity is set to 0.4. The larger weight for morphological similarity is because the geometry of the path provides a more intuitive and stable description of the evolution pattern. The overall similarity is calculated for all path pairs, constructing a 30×30 similarity matrix. The similarity matrix is ​​symmetric, with diagonal elements all equal to 1.

[0077] Spectral clustering is performed based on a similarity matrix constructed using comprehensive similarity. The similarity matrix is ​​converted into a degree matrix and a Laplacian matrix. Eigenvalues ​​and eigenvectors of the Laplacian matrix are calculated, and the eigenvectors corresponding to the top 8 largest eigenvalues ​​are selected to form a feature space. K-means clustering is performed on the 30 paths in the feature space, and the optimal number of clusters is determined to be 5 using the elbow rule. The clustering results divide the 30 evolution paths into 5 path clusters, with paths within the same cluster exhibiting similar evolution patterns and velocity characteristics.

[0078] Assign cluster weights to each path cluster. Cluster weights reflect the probability or reliability of the path cluster's occurrence. Count the number of paths contained in each path cluster; clusters with more paths receive larger initial weights. Adjust the weights based on the density of paths within the path cluster. Density is measured by the average similarity between paths within the cluster; higher density indicates a more stable and reliable evolution pattern for the cluster. Normalize the weights of the five path clusters so that the sum of the weights is 1.

[0079] Extract the evolution rate of each path cluster. Average the evolution rate characteristics of all paths within a cluster to obtain the representative evolution rate of that cluster. The evolution rate indicates how quickly the equipment condition deteriorates along the evolution mode represented by that cluster. Path clusters with fast evolution rates correspond to rapid degradation modes and require priority attention and intervention. Path clusters with slow evolution rates correspond to slow degradation modes and can be managed with preventative maintenance strategies.

[0080] Calculate the inter-cluster transition probability. Analyze the changes in the path cluster to which the evolution path belongs in different time periods. Divide each path into several time periods and determine which path cluster the path segment corresponding to each time period belongs to. Count the number of path segments that transition from cluster i to cluster j, divide by the total number of path segments in cluster i, and obtain the transition probability from cluster i to cluster j. Construct a 5×5 inter-cluster transition probability matrix, where the sum of the elements in each row of the matrix is ​​1. The transition probability reflects the likelihood of the equipment state switching between different evolution modes, providing state information for the formulation of subsequent maintenance strategies.

[0081] like Figure 2 The diagram shows the flowchart of state evolution path generation and iterative adjustment in this embodiment.

[0082] The degradation contribution of each scale component is normalized to obtain the inference weight corresponding to each scale component. The inference interval is calculated based on time-domain statistics, and the inference amplitude is calculated based on frequency-domain energy distribution. Multiple evolution paths are generated by co-inferring the inference weight, inference interval, and inference amplitude corresponding to each scale component, including:

[0083] The degradation contribution of each scale component is normalized to obtain the inference weight corresponding to each scale component;

[0084] The fluctuation amplitude and frequency of the state vector are extracted based on time-domain statistics. The rate of change is calculated based on the fluctuation amplitude and frequency and mapped to the inference interval.

[0085] Based on the frequency domain energy distribution, the energy proportion of each frequency band is extracted, and the energy concentration is calculated and mapped to the extrapolation amplitude according to the energy proportion.

[0086] The time step corresponding to each scale component is obtained by weighting the inference interval based on the inference weight, and the numerical change corresponding to each scale component is obtained by weighting the inference amplitude based on the inference weight.

[0087] Generate a random perturbation sequence that follows a normal distribution. Superimpose the random perturbation sequence with the numerical change to obtain the perturbed numerical change. Advance the state vector according to the time step and update the state vector value according to the perturbed numerical change. Generate multiple evolution paths through iterative advancement.

[0088] The path variance among multiple evolution paths is calculated as the path dispersion. The dispersion deviation is calculated based on the path dispersion and a preset dispersion threshold. The variance parameter of the random perturbation sequence is adjusted according to the dispersion deviation. The random perturbation sequence is regenerated using the adjusted variance parameter, and the evolution path generation process is re-executed. The adjustment is iteratively repeated until the path dispersion converges within the dispersion threshold range.

[0089] After obtaining the target equipment's operational status data and generating a state vector through multi-scale feature extraction, multiple evolution paths need to be generated through a collaborative inference mechanism. First, the degradation contribution of each scale component is normalized to establish an inference weight system. The degradation contribution reflects the degree of influence of different scale features on equipment performance degradation, obtained by calculating the performance degradation correlation coefficient of each scale component in historical data. The normalization process uses a summation normalization method, standardizing the sum of the degradation contributions of all scale components to 1, ensuring that the sum of the inference weights corresponding to each scale component equals 1, thus ensuring the rationality of subsequent weighted calculations. Low-frequency scale components typically reflect the overall degradation trend of the equipment, and their inference weights are relatively large, generally between 0.4 and 0.6; high-frequency scale components reflect local fluctuation characteristics, and their inference weights are relatively small, generally between 0.1 and 0.3.

[0090] After establishing the extrapolation weighting system, the extrapolation interval and extrapolation amplitude need to be determined from both the time and frequency domains. During the time-domain statistics extraction process, the fluctuation characteristics of the time-series data of the state vector are analyzed, and the fluctuation amplitude is obtained by calculating the difference sequence of state values ​​at adjacent time points. The fluctuation amplitude is represented by the standard deviation of the difference sequence; a larger standard deviation indicates more drastic state changes. The fluctuation frequency is obtained by counting the number of times the state value crosses the mean line per unit time, reflecting the activity level of state oscillations. The calculation of the rate of change comprehensively considers both the fluctuation amplitude and the fluctuation frequency, and is expressed as their product, reflecting the overall intensity of state changes. An inverse mapping relationship is established between the extrapolation interval and the rate of change: a larger rate of change corresponds to a smaller extrapolation interval, ensuring more intensive sampling during rapid state changes; a smaller rate of change corresponds to a larger extrapolation interval, allowing for a more appropriate extension of the extrapolation step size during stable state phases. The specific mapping is implemented using a piecewise linear function. The baseline extrapolation interval is set to 2 hours. When the rate of change exceeds the baseline threshold, the extrapolation interval is shortened proportionally, with a minimum of 0.5 hours. When the rate of change is lower than the baseline threshold, the extrapolation interval is extended proportionally, with a maximum of 8 hours.

[0091] The extraction of frequency domain energy distribution is achieved through spectral analysis of the state vector. After converting the time-domain signal to the frequency domain, the energy proportion of each frequency band is calculated. Frequency bands are divided using a logarithmic scale, into three intervals: low-frequency, mid-frequency, and high-frequency. The low-frequency band corresponds to the gradual change trend of the equipment, the mid-frequency band corresponds to the periodic maintenance cycle, and the high-frequency band corresponds to random noise. The energy proportion is obtained by calculating the ratio of the sum of the squares of the spectral amplitudes within each frequency band to the total energy. The energy concentration index is used to measure the distribution characteristics of frequency domain energy. A higher energy concentration indicates that energy is mainly concentrated in a certain frequency band, while a lower energy concentration indicates that energy is evenly distributed across all frequency bands. The energy concentration is calculated using the negative form of Shannon entropy, i.e., taking the logarithm of the energy proportion of each frequency band, weighting the sum, and then taking the negative sign. A direct proportional relationship is established between the extrapolation amplitude and the energy concentration. High energy concentration indicates that the state evolution has a clear dominant pattern, in which case the extrapolation amplitude should be set larger to quickly reflect the evolution trend; low energy concentration indicates that the state evolution has strong randomness, in which case the extrapolation amplitude should be set smaller to avoid over-extrapolation. The range of the extrapolation magnitude is set to 0.3 to 1.5 times the current standard deviation of the state vector.

[0092] After obtaining the extrapolation interval and magnitude, a weighted adjustment based on the extrapolation weights is required. The time step is calculated separately for each scale component. The extrapolation interval is multiplied by the extrapolation weight of the corresponding scale component to obtain the time step for that scale component. Since the evolution rates of different scale components vary, slowly changing scale components require longer time steps to reflect their evolutionary characteristics, while rapidly changing scale components require shorter time steps to capture rapid fluctuations. The calculation of numerical change is also performed separately for each scale component. The extrapolation magnitude is multiplied by the extrapolation weight of the corresponding scale component, and then multiplied by the current value of that scale component to obtain the numerical change for that scale component. This weighting method ensures that different scale components evolve collaboratively according to their importance.

[0093] To reflect the uncertainty of equipment state evolution, a random perturbation mechanism is introduced. A random perturbation sequence following a normal distribution is generated, with the mean set to 0 and the variance parameter initially set to 20% of the extrapolation amplitude. The random perturbation sequence is superimposed with the numerical change to obtain the perturbed numerical change. This approach introduces reasonable randomness while maintaining the evolution trend, making the generated evolution path more consistent with the random fluctuation characteristics of actual equipment state. The generation of the evolution path adopts an iterative approach. Each iteration advances the time axis by the time step corresponding to each scale component, while simultaneously updating the values ​​of each scale component of the state vector according to the perturbed numerical change. The advancement process continues until a preset extrapolation duration is reached, typically set to the next 30 to 90 days. By independently executing the above advancement process multiple times, each time using a different random perturbation sequence, multiple evolution paths with random differences are generated, typically ranging from 50 to 200 paths.

[0094] The generated multiple evolution paths need to meet reasonable dispersion requirements; they should not be too concentrated, leading to insufficient path diversity, nor too dispersed, causing a loss of representativeness. Path variance, as a measure of path dispersion, is obtained by calculating the variance of the state values ​​of all evolution paths at each time point and then averaging it over the time dimension. The dispersion threshold is determined based on the statistical characteristics of the state values ​​in historical data, generally set to 0.6 to 1.2 times the standard deviation of historical state values. Dispersion deviation is obtained by calculating the difference between the current path dispersion and the lower and upper bounds of the dispersion threshold. When the path dispersion is below the lower bound, the dispersion deviation is negative, requiring an increase in the variance parameter of the random disturbance; when the path dispersion is above the upper bound, the dispersion deviation is positive, requiring a decrease in the variance parameter of the random disturbance. The adjustment of the variance parameter adopts a proportional control method, with the adjustment magnitude proportional to the dispersion deviation, and the proportionality coefficient is set to 0.15. After adjustment, the random disturbance sequence is regenerated using the new variance parameter, the evolution path generation process is re-executed, a new set of paths is obtained, and the path dispersion is recalculated. The iterative adjustment process continues until the path dispersion converges to a dispersion threshold. The convergence criterion is that the change in path dispersion over three consecutive iterations is less than 5% of the dispersion threshold. The maximum number of iterations is limited to 15. If this limit is exceeded, the set of paths closest to the dispersion threshold is used as the final result.

[0095] The multiple evolution paths generated through the aforementioned collaborative extrapolation mechanism fully consider the differentiated evolution patterns of characteristics at different scales, while introducing reasonable randomness to reflect the uncertainty of actual equipment states. Simultaneously, dispersion control ensures the quality of the path set. This provides a reliable foundation for state evolution prediction in maintenance strategy optimization. The collaborative extrapolation of each scale component ensures the complete preservation of multi-dimensional characteristics of state evolution, the extrapolation weight mechanism ensures the dominant role of important scale features in path generation, and the dual constraints of the time and frequency domains ensure that the extrapolation process conforms to both the statistical laws of time series and spectral characteristics, achieving a balance between path diversity and representativeness.

[0096] The transition difficulty coefficient from each path cluster to the target path cluster is calculated based on the evolution rate and inter-cluster transition probability. A candidate action set is generated based on the transition difficulty coefficient, and the mapping relationship between each candidate action in the candidate action set and the path cluster is established, including:

[0097] The evolution rate of each path cluster and the inter-cluster transition probability between each path cluster and the target path cluster are mapped to a two-dimensional transition space. A transition potential energy field is constructed in the two-dimensional transition space with the target path cluster as the center. The potential energy value of each path cluster in the transition potential energy field is calculated as the transition difficulty coefficient from each path cluster to the target path cluster.

[0098] In the potential energy field, regions with abnormal potential energy gradients are identified as transfer obstacle zones. The distance from each path cluster to the transfer obstacle zone is calculated, and the transfer difficulty coefficient is corrected based on the distance to obtain the corrected transfer difficulty coefficient.

[0099] The path cluster with the highest difficulty coefficient of the correction transfer is selected as the path cluster to be optimized. The spatial distribution features of the transfer obstacle area are extracted, and candidate actions to bypass the transfer obstacle area are generated based on the spatial distribution features to form a candidate action set.

[0100] Calculate the position offset vector generated by each candidate action in the candidate action set in the transferred potential energy field. Calculate the similarity between the position offset vector and the vector pointing from the path cluster to the target path cluster. Based on the vector similarity, establish the mapping relationship between each candidate action in the candidate action set and the path cluster to be optimized.

[0101] After obtaining the evolution rate and inter-cluster transition probability of each path cluster, it is necessary to quantify the difficulty of each path cluster transitioning to the target path cluster. The evolution rate reflects how quickly the state of a path cluster changes, while the inter-cluster transition probability describes the possibility of mutual transformation between path clusters. Integrating information from these two dimensions can more comprehensively characterize the complexity of the transition process.

[0102] Specifically, the evolution rate of each path cluster Inter-cluster transition probability from the path cluster to the target path cluster Using these coordinates as the x and y axes respectively, all path clusters are mapped into a two-dimensional transfer space. In this space, each path cluster corresponds to a coordinate point. The location of the target path cluster in this space is denoted as... ,in It is usually 1, representing the probability that the target path cluster maintains its own state.

[0103] A transfer potential energy field is constructed centered on the location of the target path cluster. This field quantifies the "resistance" that each path cluster must overcome to transfer to the target path cluster. The construction of the potential energy field is based on the distance decay principle, meaning that the farther a path cluster is from the target path cluster in the two-dimensional transfer space, the higher its potential energy value and the greater the difficulty of the transfer. The potential energy value of path cluster i in the transfer potential energy field is calculated. First, calculate the Euclidean distance between path cluster i and the target path cluster in the two-dimensional transition space. , The potential energy value is proportional to the distance, and the reciprocal of the evolution rate is introduced as a weighting factor because path clusters with lower evolution rates experience slower state changes and are more difficult to transition. Therefore, the potential energy value can be expressed as... ,in This is an adjustment coefficient, typically ranging from 0.1 to 0.5. This potential energy value represents the difficulty coefficient of the transition from path cluster i to the target path cluster.

[0104] In the actual two-dimensional transfer space, there may be regions where the potential energy distribution exhibits anomalies, manifested as abrupt changes in the potential energy gradient or local extrema. These regions correspond to obstacles in the path cluster transfer process, which may be due to a sharp decrease in the transfer probability caused by certain specific state combinations, or a nonlinear change in the evolution rate in these regions. Identifying these transfer obstacle regions is crucial for developing effective maintenance strategies.

[0105] Calculate the potential gradient at each location in the two-dimensional transfer space. Where x represents the evolution rate dimension and y represents the transition probability dimension. The gradient is calculated using the finite difference method for a given location in space. Its gradient x-component is The y-component of the gradient is ,in and The sampling step size is used to calculate the gradient magnitude. When the gradient magnitude of a certain region exceeds twice the standard deviation of the overall gradient magnitude mean, that region is marked as a potential gradient anomaly region. Connectivity analysis is performed on all marked anomaly regions, and adjacent anomaly points are aggregated to form transfer barrier regions.

[0106] For each path cluster, calculate the shortest distance from its position in the 2D transfer space to the boundaries of each transfer obstacle zone. Using a point-to-polygon boundary distance calculation method, traverse all boundary segments of the obstacle zone, calculate the perpendicular distance from the path cluster's position point to each segment, and take the minimum value as the distance from that path cluster to that obstacle zone. , where k represents the k-th obstacle zone. When a path cluster is close to an obstacle zone, its transition process is more easily affected by that obstacle, requiring a correction to the original transition difficulty coefficient. The correction formula is: ,in This is the obstacle impact coefficient, which typically ranges from 0.3 to 0.8. To prevent small constants from being divided by zero, a value of 0.01 is used. This modified transfer difficulty coefficient more accurately reflects the comprehensive difficulties that path families need to overcome in the actual transfer process.

[0107] Traverse all path clusters and find the corrected transition difficulty coefficient. The highest-ranking path cluster is marked as the path cluster to be optimized. This path cluster represents the state that is currently the most difficult to transition to the target path cluster, and is the focus of maintenance strategy optimization.

[0108] For the path cluster to be optimized, candidate actions that can effectively reduce its transition difficulty need to be generated. The spatial distribution characteristics of the obstacle transition area determine the optimal path to bypass the obstacle. The geometric features of the obstacle area, including its center location, are extracted. The spatial relationship between the location of the path cluster to be optimized, the center location of the obstacle area, and the location of the target path cluster is calculated, considering its coverage area and shape characteristics.

[0109] When the obstacle zone is located on a direct path between the path cluster to be optimized and the target path cluster, a detour action needs to be generated. The direct path vector from the path cluster to the target path cluster is calculated. And the vector pointing from the path cluster to be optimized to the center of the obstacle region. Determine the angle between two vectors. When the included angle is less than 45 degrees and the distance between the obstacle zone and the direct path is less than a threshold, the obstacle zone is confirmed to be blocking the direct path. At this point, two candidate detour directions are generated along a direction perpendicular to the direct path vector, shifting left and right respectively. The shift angle is dynamically determined based on the coverage area of ​​the obstacle zone, typically between 30 and 60 degrees. For each detour direction, a series of candidate actions of varying intensities are generated. The intensity parameter controls the strength of the maintenance operation, ranging from minor adjustments to severe interventions, forming a set of candidate actions.

[0110] When an obstacle zone is not on a direct path, but the path cluster to be optimized is close to the obstacle zone, candidate actions far from the obstacle zone are generated to ensure that the transfer process does not accidentally get stuck in the obstacle zone. At the same time, accelerated transfer actions are generated along the direct direction. Taking advantage of the current low evolution rate of the path cluster to be optimized, its state change rate is increased through maintenance operations, thus speeding up the transfer process to the target path cluster.

[0111] After constructing the candidate action set, it is necessary to establish the mapping relationship between each candidate action and the path cluster to be optimized, and quantify the promoting effect of each candidate action on the path cluster transition. For each candidate action in the candidate action set... This simulates the positional changes of the path cluster to be optimized in the two-dimensional transition space after the action is performed. A set of parameters is adjusted, which alters the evolution rate and transition probability of the path cluster. Based on the action parameters, the change in evolution rate after executing the action is calculated. and the change in transition probability The new position of the path cluster to be optimized in the transferred potential field is: The position offset vector relative to the original position is Calculate the direction vector from the path cluster to the target path cluster. The cosine of the angle between the position offset vector and the direction vector is calculated as the similarity score. . Indicates candidate actions After execution, the position offset vector of the path cluster to be optimized in the two-dimensional transfer space.

[0112] Higher similarity indicates that the candidate action is more likely to promote the transition of the path cluster to the target path cluster. Weights are assigned to each candidate action based on similarity, establishing a mapping relationship between each candidate action and the path cluster to be optimized. Candidate actions with a similarity greater than 0.8 are marked as efficient actions and given priority for execution; actions with a similarity between 0.5 and 0.8 are marked as suboptimal actions; and actions with a similarity less than 0.5 are marked as inefficient actions and are only considered when no other options are available.

[0113] Based on the inter-cluster transition probability of associated path clusters, the change in cluster weights and the path cluster transition trajectory after executing candidate actions are calculated. The reward function is then constructed based on the change in cluster weights and the path cluster transition trajectory, including:

[0114] Extract the inter-cluster transition probability of the associated path cluster, simulate the execution of candidate actions to adjust the inter-cluster transition probability of the associated path cluster to obtain the adjusted inter-cluster transition probability, recalculate the potential energy value of the associated path cluster in the transfer potential energy field based on the adjusted inter-cluster transition probability, and calculate the difference in potential energy value before and after the execution of candidate actions as the change in cluster weight.

[0115] Extract the starting coordinates of the associated path cluster before executing the candidate action and the target coordinates after executing the candidate action in the transfer potential energy field. Connect the starting coordinates and the target coordinates to form the path cluster transfer trajectory. Extract the boundary coordinates of the transfer obstacle area in the transfer potential energy field. Calculate the overlap length between the path cluster transfer trajectory and the boundary coordinates of the transfer obstacle area as the obstacle crossing amount.

[0116] The potential energy reduction benefit is calculated based on the cluster weight change, and the obstacle crossing penalty is calculated based on the obstacle crossing amount. The potential energy reduction benefit and the obstacle crossing penalty are weighted and summed to construct the reward function.

[0117] After obtaining the mapping relationship between candidate actions and path clusters, for each candidate action in the candidate action set, it is necessary to analyze its impact on associated path clusters in depth. First, the associated path clusters corresponding to the candidate action are extracted. These associated path clusters refer to one or more path clusters directly associated with the candidate action in the mapping relationship. For each associated path cluster, its current inter-cluster transition probability matrix is ​​extracted. This matrix records the probability distribution of the path cluster transitioning to other path clusters. For example, if the associated path clusters are... Then extract its path to all other path clusters. The transition probability.

[0118] When simulating the execution of the current candidate action, the inter-cluster transition probability is adjusted based on the maintenance intensity and type of the action. Different maintenance actions affect the transition probability in different ways. Preventative maintenance actions typically reduce the transition probability to faulty path clusters while increasing the transition probability to healthy path clusters; restorative maintenance actions mainly change the transition direction of the current path cluster. During the adjustment process, the original transition probability is combined with the influence factor of the maintenance action to obtain the adjusted inter-cluster transition probability. The influence factor is determined based on the specific type and intensity of the maintenance action, and is usually derived from historical maintenance data statistics.

[0119] Within the conceptual framework of a transferred potential energy field, each path cluster corresponds to a specific potential energy value. The level of the potential energy value reflects the quality of the equipment state represented by that path cluster; healthy path clusters correspond to lower potential energy values, while path clusters with high failure risk correspond to higher potential energy values. Based on the adjusted inter-cluster transfer probabilities, the potential energy values ​​of associated path clusters in the transferred potential energy field are recalculated. The calculation of the potential energy value comprehensively considers the probability of the path cluster transferring to each target path cluster and the potential energy level of each target path cluster, and is obtained using a weighted summation method. Specifically, associated path clusters... Adjusted potential energy value It is determined by the weighted sum of the transition probability to all reachable path clusters and the corresponding potential energy value, while also adding the basic potential energy of the path cluster itself.

[0120] Calculate the difference in potential energy value before and after executing the candidate action, i.e., the original potential energy value. With the adjusted potential energy value difference This serves as the change in cluster weight. This change directly reflects the quantitative indicator of the degree to which executing the candidate action improves the state of the path cluster. When When the value is positive, it indicates that the potential energy has decreased and the equipment condition has improved; when... A negative value indicates an increase in potential energy, which may be due to improper maintenance leading to state deterioration. For a candidate action associated with multiple path clusters, the change in cluster weight for each associated path cluster is calculated separately, and then weighted and summed according to the current cluster weights of each path cluster to obtain the overall change in cluster weight for the candidate action.

[0121] In the spatial representation of the transferred potential energy field, each path cluster corresponds to a specific spatial coordinate position. These coordinates are not physical spatial coordinates, but rather two-dimensional or three-dimensional representation coordinates obtained by dimensionality reduction mapping based on the path cluster features in a high-dimensional feature space. The starting coordinates of the associated path clusters before executing the candidate action are extracted; these coordinates represent the current position of the path cluster in the potential energy field. Simultaneously, based on the adjusted inter-cluster transition probabilities and potential energy distribution, the target position to which the associated path clusters are most likely to transfer after executing the candidate action is determined; the coordinates corresponding to this position are the target coordinates. The determination of the target coordinates considers the position of the target path cluster with the highest transition probability and the descent direction of the potential energy gradient.

[0122] Connecting the starting and target coordinates to form a straight path or a curved path considering the potential energy gradient constitutes the path cluster transition trajectory. The path cluster transition trajectory characterizes the expected direction and distance of movement of the associated path cluster in the state space after executing the candidate action. In practical calculations, this trajectory is represented parametrically, for example, by generating a sequence of discrete sampling points on the trajectory using linear interpolation or Bézier curve fitting.

[0123] Several transfer barrier regions exist within the transferred potential energy field. These barrier regions represent high-risk areas or insurmountable state boundaries during the equipment's state evolution process. Transfer barrier regions may correspond to unstable operating states, critical fault states, or maintenance-restricted areas of the equipment. The boundary coordinates of each transfer barrier region in the transferred potential energy field are extracted; these boundaries are typically represented as polygons or closed curves. For each barrier region, its boundary coordinate sequence is obtained.

[0124] The spatial relationship between the path cluster transfer trajectory and the boundaries of each transfer obstacle zone is calculated to determine whether the trajectory crosses an obstacle zone. A line segment and polygon intersection detection algorithm from computational geometry is used to check each line segment on the trajectory for intersection with the obstacle zone boundary. When an intersection is detected, the length of the intersection portion is calculated. For a trajectory that may intersect with multiple obstacle zones, the length of each intersection segment is calculated separately, and the total overlap length is obtained. The overlap length, as the obstacle crossing amount, quantifies the degree to which a high-risk area needs to be traversed during the execution of the candidate action. The larger the obstacle crossing amount, the more dangerous the maintenance action is or the less it conforms to the natural laws of equipment condition evolution.

[0125] Based on cluster weight change Calculate the potential energy reduction benefit. The potential energy reduction benefit reflects the improvement in equipment condition resulting from performing the candidate action. A linear or nonlinear mapping function is used to convert the potential energy change into a benefit value. When the potential energy decreases (i.e. When the potential energy decreases, the reward is positive; when the potential energy increases, the reward is negative. Scaling factors and offset terms can be introduced to adapt the range of the reward value to the reward scale of the reinforcement learning framework. For multiple associated path clusters, the potential energy reduction reward is a weighted sum of the rewards of each path cluster, with the weights determined by the current cluster weight of each path cluster.

[0126] Based on obstacle crossing amount Calculate the obstacle crossing penalty. The obstacle crossing penalty negatively incentivizes actions involving traversing high-risk areas during execution. Penalty value. The penalty is positively correlated with the length of obstacle crossing; the longer the crossing, the heavier the penalty. Different forms of penalty can be used, such as linear penalty, quadratic penalty, or exponential penalty, selected based on the risk tolerance of the specific application scenario. Linear penalty is suitable for scenarios with uniform risk distribution, while exponential penalty is suitable for scenarios requiring strict avoidance of high-risk areas. The penalty value can also consider the danger level of the obstacle crossing area, assigning different penalty weighting coefficients to different obstacle areas.

[0127] Reduce potential energy yield Obstacle course penalty Perform a weighted summation to construct a reward function for the candidate action. .in and This is a weighting coefficient used to balance the relative importance of rewards and penalties in the reward function. The choice of weighting coefficient is determined based on the actual maintenance objective; if more emphasis is placed on state improvement, then it should be increased. If greater emphasis is placed on the safety and stability of the maintenance process, then improvements will be made. By adjusting the weight coefficients, flexible configuration of different maintenance strategy preferences can be achieved. Once the reward function is constructed, it provides a clear optimization objective for training the reinforcement learning policy network, enabling the network to learn to select the optimal maintenance action in different states that effectively improves equipment condition while avoiding high-risk operations.

[0128] The policy network is trained using a state vector as the state, a candidate action set as the action space, and a reward function as the optimization objective. The state vector is updated based on the path cluster transition trajectories, and the cluster weights and inter-cluster transition probabilities are updated based on changes in cluster weights. Iterative training until convergence yields the decision network, which includes:

[0129] The target coordinates of the path cluster transfer trajectory are extracted and the current coordinates of the corresponding path cluster in the state vector are calculated to obtain the coordinate difference. The coordinate difference is then superimposed on the coordinate features of the corresponding path cluster to obtain the updated coordinate features.

[0130] Calculate the potential energy gradient of the corresponding path cluster in the transferred potential energy field, and then replace the original path cluster features in the state vector with the updated coordinate features and the potential energy gradient to obtain the updated state vector.

[0131] The cluster weights of the corresponding path clusters are adjusted according to the change in cluster weights to obtain the updated cluster weights. The inter-cluster transition strength is calculated based on the updated cluster weights and the spatial distance between path clusters and normalized to obtain the updated inter-cluster transition probability. An inter-cluster transition probability matrix containing all path clusters is constructed and the largest eigenvalue is extracted as the transition stability index.

[0132] The updated state vector and transition stability index are input into the policy network to calculate the reward function value of the current round and update the policy network parameters. The reward function values ​​of multiple rounds are accumulated to construct a reward sequence and then smoothed to obtain a smooth reward curve.

[0133] The rate of change of the smoothed reward curve is extracted as the convergence rate. When the convergence rate is less than a preset rate threshold, the policy network training is considered to have converged and the decision network is output.

[0134] After completing the aforementioned steps, an iterative training mechanism is needed to gradually converge the policy network into a usable decision network. This process involves dynamic updates to the state vector, adaptive adjustments to cluster weights and inter-cluster transition probabilities, and setting training termination conditions based on convergence criteria.

[0135] After each candidate action is executed, the state vector needs to be updated based on the path cluster transition trajectory. The path cluster transition trajectory essentially records the movement process of a path cluster in the state space, and this trajectory can be described by target coordinates. Specifically, after executing a candidate action, the relevant path cluster will undergo a position change in the state space, and the endpoint of this change is the target coordinate. The coordinate features of the current path cluster are extracted from the state vector. These coordinate features may be in multi-dimensional vector form, reflecting the positional information of the path cluster in various dimensions of the state space. The difference between the target coordinates and the current coordinates is calculated; this difference represents the amount of movement of the path cluster. The calculated coordinate difference is directly superimposed onto the coordinate features of the corresponding path cluster to update the coordinate features. This superposition operation uses vector addition to ensure that the positional information in the state space is fed back in real time.

[0136] Building upon the updated coordinate features, the concept of a potential energy field is further introduced to enhance the expressive power of the state vector. A transfer potential energy field is constructed, forming an energy distribution in the state space, with different potential energy values ​​corresponding to the positions of different path clusters. The potential energy gradient of the corresponding path cluster at its current position is calculated, reflecting the direction and magnitude of the driving force for the path cluster's transfer to its surroundings. The calculation of the potential energy gradient can be comprehensively determined based on the spatial relationship between the path cluster and its surrounding path clusters, as well as the cluster weights of each path cluster. The updated coordinate features are concatenated with the calculated potential energy gradient to form a comprehensive feature vector containing both positional and transfer trend information. This comprehensive feature vector replaces the original path cluster feature portion in the state vector, resulting in the updated state vector. The updated state vector not only reflects the current position of the path cluster but also incorporates its dynamic transfer tendency within the potential energy field.

[0137] Simultaneously, cluster weights and inter-cluster transition probabilities also need to be updated synchronously based on the action execution effect. The change in cluster weight directly reflects the impact of the executed action on the importance of the path cluster. The change in cluster weight is added to the current cluster weight of the corresponding path cluster to obtain the updated cluster weight value. This adjustment process ensures that the path cluster weights can dynamically change with the execution of maintenance actions, accurately reflecting the importance of each path cluster in its current state. Based on the updated cluster weights, the inter-cluster transition probability is recalculated. The calculation of the inter-cluster transition strength comprehensively considers two factors: first, the spatial distance between path clusters—path clusters that are closer together are easier to transition between; second, the updated cluster weights—path clusters with higher weights have a stronger attraction to surrounding path clusters. The difference in cluster weights is combined with the reciprocal of the spatial distance to calculate the original value of the inter-cluster transition strength. All possible inter-cluster transition strengths are normalized so that the sum of the transition probabilities from any path cluster to all other path clusters equals 1, thus obtaining the updated inter-cluster transition probability.

[0138] To evaluate the overall stability of inter-cluster transfers, an inter-cluster transfer probability matrix is ​​constructed. The rows and columns of this matrix correspond to each path cluster, and the matrix elements represent the probability of transferring from one path cluster to another. Eigenvalue decomposition is performed on this matrix, and the largest eigenvalue is extracted as a transfer stability index. This index reflects the convergence characteristics of the inter-cluster transfer system. The closer the largest eigenvalue is to 1, the more stable the transfer system is, and the more balanced the transfer relationships between path clusters are. When the largest eigenvalue is much greater than 1, it indicates that the weights of some path clusters are continuously increasing, and the system has not yet reached stability; when the largest eigenvalue is close to 1, it indicates that the weights and transfer relationships of each path cluster have tended to a dynamic equilibrium.

[0139] The updated state vector and transition stability index are simultaneously input into the policy network. After receiving these two inputs, the policy network performs forward propagation calculations through its internal neural network layers. The network first extracts features from the state vector, then combines this with the transition stability index to evaluate the overall quality of the current state. Based on the definition of the reward function, the reward function value for the current round is calculated. This value comprehensively reflects the degree of optimization of the state vector after executing the action and the improvement in the stability of the inter-cluster transition system. Based on the calculated reward function value, the parameters of the policy network are updated using gradient descent or its variants. The parameter update process follows the principle of maximizing cumulative reward, enabling the policy network to gradually learn the optimal maintenance decision strategy.

[0140] During multiple training rounds, the reward function value for each round is recorded, and a reward sequence is constructed in chronological order. This reward sequence intuitively reflects the performance trend of the policy network as it progresses through training. However, because random and exploratory actions during training can cause significant fluctuations in reward values, directly using the original reward sequence is insufficient to accurately determine the convergence state. Therefore, the reward sequence is smoothed using methods such as moving averages or exponentially weighted averages to eliminate the impact of short-term fluctuations and extract the long-term trend, resulting in a smoothed reward curve. The smoothed curve more clearly displays the overall improvement in the policy network's performance.

[0141] The convergence state of training is evaluated based on a smoothed reward curve. The rate of change of the smoothed reward curve over the most recent several rounds is extracted. This rate of change is obtained by calculating the slope of the curve or the difference between adjacent points, and is used as the convergence rate. The convergence rate reflects how quickly the policy network's performance improves. In the early stages of training, the convergence rate is usually large, indicating rapid improvement in network performance; as training progresses, the network gradually approaches the optimal policy, and the convergence rate gradually decreases. A convergence rate threshold is preset. This threshold is determined based on practical application requirements and training experience, and is usually set to a small positive value, such as a reward value change of less than one-thousandth of the total reward value. When the calculated convergence rate is less than the preset rate threshold, the policy network is considered to have reached convergence, and the training process terminates. At this point, the policy network has learned a stable decision policy, and its output is used as the decision network.

[0142] In subsequent practical applications, real-time operational status data of the target equipment is input into the trained decision network. Based on the input status data, the decision network automatically selects the optimal maintenance action and outputs a specific maintenance decision, including maintenance type, maintenance time point, and maintenance parameters. Executing this maintenance decision enables dynamic optimization maintenance of the target equipment, extending its lifespan and reducing the failure rate. Throughout the training process, by continuously updating the state vector, cluster weights, and transition probabilities, the policy network accurately captures the evolution patterns of the equipment's state, ultimately forming a highly efficient maintenance decision-making capability.

[0143] Input the target device's operational status data into the decision network, output maintenance decisions, and execute them, including:

[0144] Collect real-time operating status data of the target equipment and extract equipment operating parameter features, equipment health status features and equipment historical maintenance features to construct an operating status feature vector;

[0145] The operating status feature vector is input into the decision network. The decision network extracts the correlation pattern between the equipment operating parameter features and the equipment health status features. Based on the correlation pattern, the state deviation of the equipment operating parameter features and the health decay of the equipment health status features are calculated. The state deviation and health decay are fused to generate equipment risk assessment indicators.

[0146] Based on the equipment risk assessment indicators and the equipment's historical maintenance characteristics, the expected benefit value and execution cost value corresponding to each maintenance action in the candidate action set are calculated. A comprehensive score is calculated based on the expected benefit value and execution cost value. The maintenance action with the highest comprehensive score is selected as the target maintenance action. The target maintenance action is converted into a maintenance task instruction and sent to the target equipment to perform the maintenance operation.

[0147] After the decision network is trained, the real-time operating status data of the target device needs to be input into the decision network to output the optimal maintenance decision and execute the corresponding maintenance operation.

[0148] When collecting real-time operating status data of the target equipment, data is acquired through a sensor network deployed at key parts of the equipment. Sensor types include vibration sensors, temperature sensors, pressure sensors, current sensors, and acoustic sensors. Vibration sensors collect vibration acceleration signals during equipment operation, with a sampling frequency set between 1000Hz and 10000Hz, adaptively adjusted according to the equipment's rotational speed. Temperature sensors collect temperature change data for major components of the equipment, including bearing temperature, motor temperature, and ambient temperature. Pressure sensors collect pressure fluctuation information from hydraulic or pneumatic systems. Current sensors monitor the equipment's power consumption, recording current waveforms and power factor. Acoustic sensors capture sound signals generated during equipment operation to identify abnormal noise patterns.

[0149] When extracting equipment operating parameter features from the collected real-time operating status data, time-domain and frequency-domain features are extracted from the vibration signal. Time-domain features include root mean square value, peak value, peak-to-peak value, waveform factor, impulse factor, and margin factor. Frequency-domain features are obtained by obtaining the power spectral density through fast Fourier transform, and the amplitude and phase information of characteristic frequency components are extracted. Sliding window statistical analysis is performed on the temperature data to calculate the temperature change rate, temperature fluctuation amplitude, and temperature stability index. Pressure pulsation frequency, pressure peak frequency, and pressure fluctuation standard deviation are extracted from the pressure data. Active power, reactive power, harmonic content, and current imbalance are calculated from the current data. The extracted multi-dimensional operating parameter features are combined to form the equipment operating parameter feature vector.

[0150] When extracting equipment health status features, health status assessment is based on the changing trends of equipment operating parameters. By establishing baseline values ​​for parameters during normal equipment operation, the deviation of current operating parameters from these baseline values ​​is calculated. For vibration signals, the ratio of current vibration intensity to historical normal vibration intensity is calculated. For temperature data, a temperature anomaly index is calculated; the anomaly index increases when the temperature exceeds a preset threshold. The health status evaluation results from multiple parameters are combined, and a weighted fusion method is used to generate an equipment health status feature vector. The weights are allocated according to the degree of influence of each parameter on the equipment health status.

[0151] When extracting historical maintenance features of equipment, past maintenance records are retrieved from the equipment maintenance history database, including maintenance type, maintenance time, maintenance cycle, maintenance duration, type of replaced parts, and equipment performance recovery status after maintenance. The execution frequency of various maintenance actions is statistically analyzed, and the average maintenance interval is calculated. The duration of maintenance effect, i.e., the duration after maintenance operations are performed and the equipment returns to normal operation, is analyzed. Recurring patterns of equipment failure modes are extracted to identify frequently occurring failure types. The above historical maintenance information is then encoded to generate a historical maintenance feature vector for the equipment.

[0152] When constructing the operational status feature vector, the equipment operating parameter feature vector, equipment health status feature vector, and equipment historical maintenance feature vector are concatenated and fused. Before concatenation, each feature vector is normalized to eliminate the influence of different feature dimensions. The normalization method uses min-max normalization or Z-score standardization to ensure that the numerical range of each feature component is consistent. The dimension of the concatenated operational status feature vector is typically between 50 and 200 dimensions, adjusted according to the complexity of the equipment.

[0153] After the operational status feature vector is input into the decision network, the feature extraction module within the network performs deep feature learning on the input features. The decision network employs a multi-layer fully connected neural network or residual neural network structure, introducing nonlinear transformation capabilities through activation functions. During feature extraction, the network automatically learns the correlation patterns between equipment operating parameter features and equipment health status features, identifying which changes in operating parameters significantly impact the health status.

[0154] When calculating the state deviation of equipment operating parameter characteristics based on correlation patterns, the deviation of each operating parameter characteristic from the normal operating baseline value is calculated, and the deviation is measured using Euclidean distance or Mahalanobis distance. A larger state deviation value indicates a greater difference between the current operating state and the normal state of the equipment. When calculating the health decay of equipment health status characteristics, the trend of health status characteristics over time is analyzed, and a health decay curve is fitted. The health decay reflects the rate of deterioration of the equipment's health status; a higher decay rate indicates a faster deterioration of the equipment's health status.

[0155] When generating equipment risk assessment indicators from the fusion of state deviation and health degradation, a weighted summation method or adaptive fusion through a neural network layer is used. The equipment risk assessment indicators comprehensively reflect the current operational risk level of the equipment; higher values ​​indicate a greater risk of equipment failure and the need for timely maintenance intervention.

[0156] When calculating the expected benefit value for each maintenance action in the candidate action set based on equipment risk assessment indicators and historical maintenance characteristics, the expected benefit value includes the degree of equipment performance improvement after maintenance, the extended equipment lifespan, and the potential losses avoided. For preventative maintenance actions, the expected benefit value is calculated by estimating the downtime losses caused by avoiding failures. For corrective maintenance actions, the expected benefit value is assessed based on the level of equipment performance recovery after maintenance. The expected benefit value also considers historical effect data of the same maintenance actions in the equipment's historical maintenance characteristics, obtaining the average benefit value through historical data statistics.

[0157] When calculating the execution cost of each maintenance action, the cost includes direct costs and indirect costs. Direct costs include labor costs, material costs for replacement parts, and costs for using maintenance tools. Indirect costs include production losses due to equipment downtime, energy consumption during maintenance, and the time cost of post-maintenance setup. The execution cost varies significantly depending on the type of maintenance action; simple lubrication maintenance has lower costs, while component replacement or system overhaul has higher costs.

[0158] When calculating the comprehensive score based on expected benefits and execution costs, the comprehensive score reflects the cost-effectiveness of maintenance actions. The comprehensive score is calculated as the ratio of expected benefits to execution costs, or using a linear weighted combination method, assigning positive weight to benefits and negative weight to costs. The comprehensive score also incorporates equipment risk assessment indicators as adjustment factors; when equipment risk is high, the weight of expected benefits is increased, and when equipment risk is low, the weight of execution costs is increased, thus achieving risk-oriented maintenance decisions.

[0159] After selecting the maintenance action with the highest overall score as the target maintenance action, a feasibility verification is performed to check whether the current conditions for executing the maintenance action are met, including the availability of maintenance personnel, spare parts inventory, and production planning. If the current conditions are not met, the second-highest rated and feasible maintenance action is selected as the alternative target maintenance action.

[0160] When converting target maintenance actions into maintenance task instructions, these instructions include the maintenance action type, execution time, operation steps, required spare parts list, personnel requirements, and safety precautions. The instructions are encoded using a structured data format for easy parsing and execution by automated systems. Maintenance execution time is optimized and scheduled according to the production plan, prioritizing periods of equipment idle time or planned downtime to minimize production disruption.

[0161] When a maintenance task instruction is sent to the target equipment for maintenance operations, the instruction is transmitted to the equipment control system or maintenance management system via the Industrial Internet or fieldbus. Upon receiving the instruction, the maintenance management system automatically generates a maintenance work order and assigns it to the appropriate maintenance personnel. The maintenance personnel prepare the necessary spare parts and tools according to the work order and arrive at the equipment site at the designated time to perform the maintenance operation. The execution of the maintenance operation is recorded in real time during the maintenance process. After maintenance, equipment performance is verified to ensure that the maintenance effect meets the expected goals. The maintenance result data is fed back to the decision network for subsequent model updates and optimizations, forming a closed-loop maintenance decision optimization system.

[0162] A second aspect of this invention provides a dynamic optimization system for equipment maintenance strategies based on reinforcement learning, comprising:

[0163] The state evolution unit is used to acquire the operating state data of the target device and perform multi-scale feature extraction to generate a state vector. Based on the state vector, multiple evolution paths are generated, the similarity of the evolution paths is calculated and they are clustered into multiple path clusters. Cluster weights are assigned to each path cluster and the evolution rate is extracted. The inter-cluster transition probability is calculated.

[0164] The transfer mapping unit is used to calculate the transfer difficulty coefficient from each path cluster to the target path cluster based on the evolution rate and inter-cluster transfer probability, generate a candidate action set based on the transfer difficulty coefficient, and establish the mapping relationship between each candidate action in the candidate action set and the path cluster.

[0165] The function construction unit is used to determine the associated path clusters corresponding to each candidate action in the candidate action set based on the mapping relationship, calculate the change in cluster weights and the path cluster transfer trajectory after executing the candidate action based on the inter-cluster transfer probability of the associated path clusters, and construct the reward function based on the change in cluster weights and the path cluster transfer trajectory.

[0166] The decision execution unit is used to train the policy network with the state vector as the state, the candidate action set as the action space, and the reward function as the optimization objective. It updates the state vector according to the path cluster transition trajectory, updates the cluster weight and inter-cluster transition probability according to the change in cluster weight, and iterates the training until convergence to obtain the decision network. It inputs the operating status data of the target device into the decision network, outputs maintenance decisions, and executes them.

[0167] A third aspect of the present invention provides an electronic device, comprising:

[0168] processor;

[0169] Memory used to store processor-executable instructions;

[0170] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0171] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0172] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamic optimization of equipment maintenance strategy based on reinforcement learning, characterized in that, include: The system acquires the operating status data of the target device and performs multi-scale feature extraction to generate a state vector. Based on the state vector, it infers and generates multiple evolution paths, calculates the similarity of the evolution paths and clusters them into multiple path clusters, assigns cluster weights to each path cluster and extracts the evolution rate, and calculates the inter-cluster transition probability. The transition difficulty coefficient from each path cluster to the target path cluster is calculated based on the evolution rate and inter-cluster transition probability. A candidate action set is generated based on the transition difficulty coefficient, and a mapping relationship between each candidate action in the candidate action set and the path cluster is established. For each candidate action in the candidate action set, the associated path cluster corresponding to the candidate action is determined based on the mapping relationship. The change in cluster weight of the associated path cluster and the path cluster transfer trajectory after executing the candidate action are calculated based on the inter-cluster transfer probability of the associated path cluster. A reward function is constructed based on the change in cluster weight and the path cluster transfer trajectory. The policy network is trained with the state vector as the state, the candidate action set as the action space, and the reward function as the optimization objective. The state vector is updated according to the path cluster transition trajectory, and the cluster weights and inter-cluster transition probabilities are updated according to the change in cluster weights. The network is iteratively trained until convergence to obtain the decision network. The operating status data of the target device is input into the decision network, and the maintenance decision is output and executed.

2. The method of claim 1, wherein, The process involves acquiring the target device's operational status data and performing multi-scale feature extraction to generate a state vector. Based on this state vector, multiple evolution paths are derived. The similarity of these evolution paths is calculated, and they are clustered into multiple path clusters. Cluster weights are assigned to each path cluster, and the evolution rate is extracted. The inter-cluster transition probability is calculated, including: Wavelet decomposition is performed on the operational status data at multiple time scales to obtain components at each scale. Time-domain statistics and frequency-domain energy distributions are calculated for each scale component. The degradation contribution of each scale component is calculated based on the time-domain statistics and frequency-domain energy distributions. The scale components are weighted and fused according to the degradation contribution to generate a state vector. The degradation contribution of each scale component is normalized to obtain the inference weight corresponding to each scale component. The inference interval is calculated based on the time domain statistics, and the inference amplitude is calculated based on the frequency domain energy distribution. Multiple evolution paths are generated by coordinating the inference weight, inference interval and inference amplitude corresponding to each scale component. Morphological features and evolution speed features are extracted from multiple evolution paths. Morphological similarity between evolution paths is calculated based on morphological features, and velocity similarity between evolution paths is calculated based on evolution speed features. The morphological similarity and velocity similarity are weighted to obtain a comprehensive similarity. A similarity matrix is ​​constructed based on the comprehensive similarity, and spectral clustering is performed to obtain multiple path clusters. Cluster weights are assigned to each path cluster, the evolution rate is extracted, and the inter-cluster transition probability is calculated.

3. The method of claim 2, wherein, The degradation contribution of each scale component is normalized to obtain the inference weight corresponding to each scale component. The inference interval is calculated based on time-domain statistics, and the inference amplitude is calculated based on frequency-domain energy distribution. Multiple evolution paths are generated by co-inferring the inference weight, inference interval, and inference amplitude corresponding to each scale component, including: The degradation contribution of each scale component is normalized to obtain the inference weight corresponding to each scale component; The fluctuation amplitude and frequency of the state vector are extracted based on time-domain statistics. The rate of change is calculated based on the fluctuation amplitude and frequency and mapped to the inference interval. Based on the frequency domain energy distribution, the energy proportion of each frequency band is extracted, and the energy concentration is calculated and mapped to the extrapolation amplitude according to the energy proportion. The time step corresponding to each scale component is obtained by weighting the inference interval based on the inference weight, and the numerical change corresponding to each scale component is obtained by weighting the inference amplitude based on the inference weight. Generate a random perturbation sequence that follows a normal distribution. Superimpose the random perturbation sequence with the numerical change to obtain the perturbed numerical change. Advance the state vector according to the time step and update the state vector value according to the perturbed numerical change. Generate multiple evolution paths through iterative advancement. The path variance among multiple evolution paths is calculated as the path dispersion. The dispersion deviation is calculated based on the path dispersion and a preset dispersion threshold. The variance parameter of the random perturbation sequence is adjusted according to the dispersion deviation. The random perturbation sequence is regenerated using the adjusted variance parameter, and the evolution path generation process is re-executed. The adjustment is iteratively repeated until the path dispersion converges within the dispersion threshold range.

4. The method according to claim 1, characterized in that, The transition difficulty coefficient from each path cluster to the target path cluster is calculated based on the evolution rate and inter-cluster transition probability. A candidate action set is generated based on the transition difficulty coefficient, and the mapping relationship between each candidate action in the candidate action set and the path cluster is established, including: The evolution rate of each path cluster and the inter-cluster transition probability between each path cluster and the target path cluster are mapped to a two-dimensional transition space. A transition potential energy field is constructed in the two-dimensional transition space with the target path cluster as the center. The potential energy value of each path cluster in the transition potential energy field is calculated as the transition difficulty coefficient from each path cluster to the target path cluster. In the potential energy field, regions with abnormal potential energy gradients are identified as transfer obstacle zones. The distance from each path cluster to the transfer obstacle zone is calculated, and the transfer difficulty coefficient is corrected based on the distance to obtain the corrected transfer difficulty coefficient. The path cluster with the highest difficulty coefficient of the correction transfer is selected as the path cluster to be optimized. The spatial distribution features of the transfer obstacle area are extracted, and candidate actions to bypass the transfer obstacle area are generated based on the spatial distribution features to form a candidate action set. Calculate the position offset vector generated by each candidate action in the candidate action set in the transferred potential energy field. Calculate the similarity between the position offset vector and the vector pointing from the path cluster to the target path cluster. Based on the vector similarity, establish the mapping relationship between each candidate action in the candidate action set and the path cluster to be optimized.

5. The method according to claim 1, characterized in that, Based on the inter-cluster transition probability of associated path clusters, the change in cluster weights and the path cluster transition trajectory after executing candidate actions are calculated. The reward function is then constructed based on the change in cluster weights and the path cluster transition trajectory, including: Extract the inter-cluster transition probability of the associated path cluster, simulate the execution of candidate actions to adjust the inter-cluster transition probability of the associated path cluster to obtain the adjusted inter-cluster transition probability, recalculate the potential energy value of the associated path cluster in the transfer potential energy field based on the adjusted inter-cluster transition probability, and calculate the difference in potential energy value before and after the execution of candidate actions as the change in cluster weight. Extract the starting coordinates of the associated path cluster before executing the candidate action and the target coordinates after executing the candidate action in the transfer potential energy field. Connect the starting coordinates and the target coordinates to form the path cluster transfer trajectory. Extract the boundary coordinates of the transfer obstacle area in the transfer potential energy field. Calculate the overlap length between the path cluster transfer trajectory and the boundary coordinates of the transfer obstacle area as the obstacle crossing amount. The potential energy reduction benefit is calculated based on the cluster weight change, and the obstacle crossing penalty is calculated based on the obstacle crossing amount. The potential energy reduction benefit and the obstacle crossing penalty are weighted and summed to construct the reward function.

6. The method according to claim 1, characterized in that, The policy network is trained using a state vector as the state, a candidate action set as the action space, and a reward function as the optimization objective. The state vector is updated based on the path cluster transition trajectories, and the cluster weights and inter-cluster transition probabilities are updated based on changes in cluster weights. Iterative training until convergence yields the decision network, which includes: The target coordinates of the path cluster transfer trajectory are extracted and the current coordinates of the corresponding path cluster in the state vector are calculated to obtain the coordinate difference. The coordinate difference is then superimposed on the coordinate features of the corresponding path cluster to obtain the updated coordinate features. Calculate the potential energy gradient of the corresponding path cluster in the transferred potential energy field, and then replace the original path cluster features in the state vector with the updated coordinate features and the potential energy gradient to obtain the updated state vector. The cluster weights of the corresponding path clusters are adjusted according to the change in cluster weights to obtain the updated cluster weights. The inter-cluster transition strength is calculated based on the updated cluster weights and the spatial distance between path clusters and normalized to obtain the updated inter-cluster transition probability. An inter-cluster transition probability matrix containing all path clusters is constructed and the largest eigenvalue is extracted as the transition stability index. The updated state vector and transition stability index are input into the policy network to calculate the reward function value of the current round and update the policy network parameters. The reward function values ​​of multiple rounds are accumulated to construct a reward sequence and then smoothed to obtain a smooth reward curve. The rate of change of the smoothed reward curve is extracted as the convergence rate. When the convergence rate is less than a preset rate threshold, the policy network training is considered to have converged and the decision network is output.

7. The method according to claim 1, characterized in that, Input the target device's operational status data into the decision network, output maintenance decisions, and execute them, including: Collect real-time operating status data of the target equipment and extract equipment operating parameter features, equipment health status features and equipment historical maintenance features to construct an operating status feature vector; The operating status feature vector is input into the decision network. The decision network extracts the correlation pattern between the equipment operating parameter features and the equipment health status features. Based on the correlation pattern, the state deviation of the equipment operating parameter features and the health decay of the equipment health status features are calculated. The state deviation and health decay are fused to generate equipment risk assessment indicators. Based on the equipment risk assessment indicators and the equipment's historical maintenance characteristics, the expected benefit value and execution cost value corresponding to each maintenance action in the candidate action set are calculated. A comprehensive score is calculated based on the expected benefit value and execution cost value. The maintenance action with the highest comprehensive score is selected as the target maintenance action. The target maintenance action is converted into a maintenance task instruction and sent to the target equipment to perform the maintenance operation.

8. A dynamic optimization system for equipment maintenance strategies based on reinforcement learning, used to implement the method as described in any one of claims 1-7, characterized in that, include: The state evolution unit is used to acquire the operating state data of the target device and perform multi-scale feature extraction to generate a state vector. Based on the state vector, multiple evolution paths are generated, the similarity of the evolution paths is calculated and they are clustered into multiple path clusters. Cluster weights are assigned to each path cluster and the evolution rate is extracted. The inter-cluster transition probability is calculated. The transfer mapping unit is used to calculate the transfer difficulty coefficient from each path cluster to the target path cluster based on the evolution rate and inter-cluster transfer probability, generate a candidate action set based on the transfer difficulty coefficient, and establish the mapping relationship between each candidate action in the candidate action set and the path cluster. The function construction unit is used to determine the associated path clusters corresponding to each candidate action in the candidate action set based on the mapping relationship, calculate the change in cluster weights and the path cluster transfer trajectory after executing the candidate action based on the inter-cluster transfer probability of the associated path clusters, and construct the reward function based on the change in cluster weights and the path cluster transfer trajectory. The decision execution unit is used to train the policy network with the state vector as the state, the candidate action set as the action space, and the reward function as the optimization objective. It updates the state vector according to the path cluster transition trajectory, updates the cluster weight and inter-cluster transition probability according to the change in cluster weight, and iterates the training until convergence to obtain the decision network. It inputs the operating status data of the target device into the decision network, outputs maintenance decisions, and executes them.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images