Cable accessory intelligent maintenance strategy optimization method based on reinforcement learning

Abstract

The invention discloses a cable accessory intelligent maintenance strategy optimization method based on reinforcement learning, and the method comprises the following steps: S1, obtaining data, and constructing a state feature sequence, an action sequence and a degradation observation sequence; s2, training a hidden semi-Markov model to obtain a hidden state transition relation and duration expression; s3, constructing a semi-Markov decision process according to the hidden state transition relation and the duration expression, and generating a simulation trajectory; s4, extracting time weighted features of the expert trajectory and the simulation trajectory, and constructing a maximum entropy inverse reinforcement learning optimization target; s5, iteratively solving reward function parameters; s6, executing increment updating of the reward function; and S7, inputting the reward function into the state evolution model, and outputting a maintenance strategy. According to the invention, cable accessory degradation accurate identification and maintenance strategy dynamic optimization are realized, and maintenance timeliness, accuracy and operation reliability are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for power equipment, and in particular to an optimization method for intelligent maintenance strategies of cable accessories based on reinforcement learning. Background Technology

[0002] As key components for connection, insulation, and terminal sealing in power systems, cable accessories directly affect the reliability and security of the distribution network. Currently, cable accessory maintenance primarily relies on inspection records, partial discharge detection, insulation resistance testing, and environmental monitoring, using manual experience to assess the degree of degradation and formulate maintenance plans.

[0003] Traditional methods have limitations in identifying degradation stages, analyzing the correlation between maintenance behaviors, and generating strategies. The degradation process is implicit and time-dependent; existing methods often employ fixed thresholds or simple trend analysis, failing to express the hidden state changes within each degradation stage or accurately describe the duration characteristics of each stage, leading to unstable degradation state inferences. Current maintenance strategy generation is primarily based on human experience, rule bases, or static models, failing to consider the differences in action effects across different degradation stages or the impact of action duration on degradation evolution, resulting in strategies lacking specificity and temporal rationality. When using expert maintenance trajectories for learning, existing techniques typically treat trajectories with equal weight, neglecting the knowledge decay problem caused by the timeliness of trajectories, making it difficult to effectively integrate new data into the model and achieve dynamic updates to maintenance strategies.

[0004] Therefore, how to provide a reinforcement learning-based intelligent maintenance strategy optimization method for cable accessories is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose an intelligent maintenance strategy optimization method for cable accessories based on reinforcement learning. This invention constructs a degradation stage sequence, expresses the duration characteristics of actions, generates a time-weighted reward expression, and implements incremental updates of the reward function to form a maintenance strategy that can be dynamically adjusted according to the operating status. It has the advantages of accurate degradation identification, reasonable action timing, and timely strategy updates.

[0006] An intelligent maintenance strategy optimization method for cable accessories based on reinforcement learning according to an embodiment of the present invention includes the following steps: S1. Acquire cable accessory operation data, environmental parameters, monitoring records and expert maintenance trajectories, construct state characteristic sequences and action sequences, and form a degradation stage observation sequence; S2. Train a hidden semi-Markov model based on the observation sequence of the degradation stage, set the set of degradation stages, the stage transition probability matrix and the stage duration distribution, and obtain the hidden state transition relationship and duration expression. S3. Construct a semi-Markov decision process based on the hidden state transition relationship and duration expression, set the state space, action space, state transition rules and action duration constraints, form a state evolution model, and generate a simulated trajectory based on the state evolution model; S4. Define a time-weighted feature set containing state, action, and action duration; extract time-weighted feature information from expert-maintained trajectory and simulated trajectory; and construct a maximum entropy inverse reinforcement learning optimization objective. S5. Initialize the reward function parameters, calculate the expert trajectory probability distribution based on maximum entropy inverse reinforcement learning, perform gradient iteration to solve for the reward function parameters, and generate the current reward function. S6. Receive newly added expert-maintained trajectories, introduce trajectory time labels and data weight decay factors, extract newly added time-weighted feature information, perform incremental updates of reward function parameters based on maximum entropy inverse reinforcement learning, and generate updated reward functions. S7. Input the reward function into the state evolution model, use the reinforcement learning algorithm to execute the strategy iterative solution, and output the cable accessory maintenance strategy.

[0007] Optionally, S1 specifically includes: S11. Obtain cable accessory operation data, environmental parameters and monitoring records, and align current amplitude, load level, conductor temperature, ambient temperature, ambient humidity, sheath current and partial discharge detection data in chronological order to form a state characteristic sequence; S12. Obtain the expert maintenance trajectory and arrange the inspection actions, replacement actions, maintenance actions and pre-test actions according to the maintenance start time and maintenance end time to construct the action sequence; S13. Based on partial discharge detection data, insulation resistance test data, conductor temperature and ambient humidity, perform degradation-related feature extraction, arrange the extracted features in chronological order to form a degradation stage observation sequence.

[0008] Optionally, S3 specifically includes: S21. Based on the observation sequence of the degradation stage, a set of degradation stages for the hidden semi-Markov model is set, and the degradation stages are divided into initial degradation stage, slow degradation stage, accelerated degradation stage and critical degradation stage according to a preset order to form a set of degradation stages. S22. Set a state duration distribution function for each degradation stage in the degradation stage set. Based on the statistical distribution of degradation characteristics in each time period in the degradation stage observation sequence, construct the duration distribution of duration characteristics of each degradation stage to form a duration distribution set. S23. Based on the changes in degradation characteristics between adjacent time periods in the degradation stage observation sequence, calculate the state transition frequency between each degradation stage, and generate a degradation stage transition probability matrix based on the transition frequency. S24. Input the degradation stage observation sequence, degradation stage set, duration distribution set, and degradation stage transition probability matrix into the hidden semi-Markov model, perform model training, and obtain the hidden semi-Markov model parameters. S25. Based on the trained hidden semi-Markov model, perform hidden state inference on the observation sequence of the degradation stage to generate the hidden degradation stage sequence. S26. The sequence of hidden degeneration stages and their corresponding durations are expressed as hidden state transition relations and durations.

[0009] Optionally, S3 specifically includes: S31. Combine the temporally corresponding degeneracy stages and state characteristics into state elements, arrange all state elements in chronological order to form the state space of a semi-Markov decision process; add inspection actions, maintenance actions, pre-test actions and replacement actions into the action space according to preset action numbers. S32. Based on the hidden state transition relationship, establish a state transition connection between the starting state element and the target state element corresponding to each stage, and write the transition probability of the corresponding stage in the hidden state transition relationship into the state transition connection to form a state transition rule. S33. Based on the duration expression, read each degenerate stage in the degenerate stage sequence, write the duration distribution corresponding to the degenerate stage in the duration expression into the state element associated with the degenerate stage, and form an action duration constraint for all actions in the action space by referencing the duration distribution. S34. Integrate the state space, action space, state transition rules and action duration constraints according to the preset structure to construct a state evolution model. Call the state evolution model to execute state reading, action selection, state transition and action duration allocation in sequence from any starting time index to generate state sequence, action sequence and action duration sequence, forming a simulation trajectory.

[0010] Optionally, S4 specifically includes: S41. Set a time-weighted feature template, determine the state feature dimension and action coding dimension, and concatenate the state feature and action coding corresponding to each time index to generate a basic feature vector; read the state sequence, action sequence and action duration sequence of the expert maintenance trajectory according to the time index in sequence, and construct the expert basic feature vector sequence. S42. Read the corresponding action duration for each basic feature vector in the expert basic feature vector sequence, multiply the action duration as a weight with each component of the basic feature vector to generate an expert time-weighted feature vector, and sum all expert time-weighted feature vectors according to the feature dimension to generate an expert cumulative time-weighted feature vector. S43. Read the state sequence, action sequence and action duration sequence of the simulated trajectory in the same way as S41 and S42, construct the simulation basic feature vector sequence and the simulation time weighted feature vector sequence, and accumulate them according to the feature dimension to generate the simulation cumulative time weighted feature vector. S44. Set the expert cumulative time weighted feature vector as the expert expected feature, set the simulated cumulative time weighted feature vector as the model expected feature, and combine the expert expected feature and the model expected feature as the time-weighted reward expression.

[0011] Optionally, S5 specifically includes: S51. Initialize the reward function parameters and set the reward function parameter dimensions corresponding to the time-weighted reward expression; S52. Based on the time-weighted reward expression, calculate the logarithmic probability of the expert trajectory, multiply the cumulative time-weighted feature vector of the expert in the time-weighted reward expression by the reward function parameters component by component, and sum them to generate the expected reward value of the expert. S53. Calculate the logarithmic probability of the simulated trajectory based on the time-weighted reward expression, multiply the cumulative time-weighted feature vector of the model by the reward function parameters component by component, and sum them to generate the expected reward value of the model. S54. Perform a difference operation between the expert's expected reward value and the model's expected reward value according to the preset solution structure to generate the gradient components of the reward function parameters, and perform gradient iteration on the reward function parameters according to the iteration step size to update the reward function parameters. S55. Repeat S52 to S54 until the reward function parameters meet the convergence condition, thus forming the current reward function.

[0012] Optionally, S6 specifically includes: S61. Receive the newly added expert maintenance trajectory, read the state sequence, action sequence and action duration sequence, concatenate the state feature and action code corresponding to each time index to form the basic feature vector of the new expert, and multiply the components according to the action duration to generate the time-weighted feature vector of the new expert. S62. Set time tags for the maintenance trajectory of newly added experts, convert the time tags into attenuation coefficients, multiply the attenuation coefficients by the time-weighted feature vector of the newly added experts component by component, and accumulate them according to the feature dimension to generate the cumulative time-weighted feature vector of the newly added experts. S63. Add the newly added expert cumulative time weighted feature vector to the expert cumulative time weighted feature vector according to the feature dimension to form the superimposed expert cumulative time weighted feature vector, and arrange the superimposed expert cumulative time weighted feature vector and the model cumulative time weighted feature vector according to the feature dimension to form a new time weighted reward expression; S64. Update the reward function parameters according to the new time-weighted reward expression. Subtract the cumulative time-weighted feature vector of the experts from the cumulative time-weighted feature vector of the model according to the feature dimension to form the gradient vector of the reward function parameters. Add the gradient vector to the reward function parameters one by one according to the iteration step size to update the reward function parameters. S65. Repeat S61 to S64 until the difference between two consecutive updates of the reward function parameters is lower than the set threshold, and the updated reward function is obtained.

[0013] Optionally, S7 specifically includes: S71. Write the reward function into the state evolution model, read each state element in the state space, calculate the reward value of all actions in the corresponding action space, and form a state-action-reward set. S72. Based on the state-action reward set, read the state transition rules, combine the action reward value corresponding to each state element with the state transition probability, and generate a state-action value expression. S73. Based on the expression of state-action value, the constraint of action duration is referenced, and the action duration is combined with the state-action value to form a state-action value sequence. S74. According to the state-action value sequence, perform action selection for each state element in the state space, generate the corresponding maintenance action, and arrange the maintenance actions corresponding to all state elements to form a cable accessory maintenance strategy.

[0014] The beneficial effects of this invention are: (1) By constructing a degradation stage sequence of a hidden semi-Markov model and combining it with the duration distribution, this invention realizes the identification of hidden stages and expression of temporal characteristics of the degradation state of cable accessories. Compared with the traditional method based on threshold or trend judgment, it can obtain a more accurate and stable degradation stage inference effect, providing a reliable basis for the refined formulation of maintenance strategies.

[0015] (2) This invention uses a semi-Markov decision process to model the degeneracy stage, state characteristics, action space and action duration constraints in a unified manner. It can express the time influence of maintenance actions on degeneracy evolution, realize the state evolution mechanism consistent with the actual maintenance process, overcome the defect that traditional Markov models cannot handle the difference in action duration, and make the maintenance strategy more temporally reasonable and feasible.

[0016] (3) This invention utilizes maximum entropy inverse reinforcement learning to construct a time-weighted reward expression and introduces a time-label-based decay mechanism to perform incremental reward function updates on newly added expert maintenance trajectories. This realizes the dynamic integration of expert experience and continuous optimization of maintenance knowledge. Compared with existing static models, the maintenance strategy can be adjusted in real time according to the latest running data, thereby improving the adaptability and effectiveness of the strategy. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an intelligent maintenance strategy optimization method for cable accessories based on reinforcement learning proposed in this invention; Figure 2 This is a schematic diagram of a semi-Markov process for an intelligent maintenance strategy optimization method for cable accessories based on reinforcement learning proposed in this invention. Figure 3 The diagram shows the maximum entropy inverse reinforcement learning optimization loop structure of the intelligent maintenance strategy optimization method for cable accessories based on reinforcement learning proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figure 1-3 A method for optimizing intelligent maintenance strategies for cable accessories based on reinforcement learning includes the following steps: S1. Acquire cable accessory operation data, environmental parameters, monitoring records and expert maintenance trajectories, construct state characteristic sequences and action sequences, and form a degradation stage observation sequence; S2. Train a hidden semi-Markov model based on the observation sequence of the degradation stage, set the set of degradation stages, the stage transition probability matrix and the stage duration distribution, and obtain the hidden state transition relationship and duration expression. S3. Construct a semi-Markov decision process based on the hidden state transition relationship and duration expression, set the state space, action space, state transition rules and action duration constraints, form a state evolution model, and generate a simulated trajectory based on the state evolution model; S4. Define a time-weighted feature set containing state, action, and action duration; extract time-weighted feature information from expert-maintained trajectory and simulated trajectory; and construct a maximum entropy inverse reinforcement learning optimization objective. S5. Initialize the reward function parameters, calculate the expert trajectory probability distribution based on maximum entropy inverse reinforcement learning, perform gradient iteration to solve for the reward function parameters, and generate the current reward function. S6. Receive newly added expert-maintained trajectories, introduce trajectory time labels and data weight decay factors, extract newly added time-weighted feature information, perform incremental updates of reward function parameters based on maximum entropy inverse reinforcement learning, and generate updated reward functions. S7. Input the reward function into the state evolution model, use the reinforcement learning algorithm to execute the strategy iterative solution, and output the cable accessory maintenance strategy.

[0020] In this embodiment, S1 specifically includes: S11. Obtain cable accessory operation data, environmental parameters and monitoring records, and align current amplitude, load level, conductor temperature, ambient temperature, ambient humidity, sheath current and partial discharge detection data in chronological order to form a state characteristic sequence; S12. Obtain the expert maintenance trajectory and arrange the inspection actions, replacement actions, maintenance actions and pre-test actions according to the maintenance start time and maintenance end time to construct the action sequence; S13. Based on partial discharge detection data, insulation resistance test data, conductor temperature and ambient humidity, perform degradation-related feature extraction, arrange the extracted features in chronological order to form a degradation stage observation sequence.

[0021] In this embodiment, S3 specifically includes: S21. Based on the observation sequence of the degradation stage, a set of degradation stages for the hidden semi-Markov model is set, and the degradation stages are divided into initial degradation stage, slow degradation stage, accelerated degradation stage and critical degradation stage according to a preset order to form a set of degradation stages. S22. Set a state duration distribution function for each degradation stage in the degradation stage set. Based on the statistical distribution of degradation characteristics in each time period in the degradation stage observation sequence, construct the duration distribution of duration characteristics of each degradation stage to form a duration distribution set. In this embodiment, the Weibull distribution is used as the state duration distribution function form, and the parameter estimation method is maximum likelihood estimation. S23. Based on the changes in degradation characteristics between adjacent time periods in the degradation stage observation sequence, calculate the state transition frequency between each degradation stage, and generate a degradation stage transition probability matrix based on the transition frequency. S24. Input the degradation stage observation sequence, degradation stage set, duration distribution set, and degradation stage transition probability matrix into the hidden semi-Markov model, perform model training, and obtain the hidden semi-Markov model parameters. S25. Based on the trained hidden semi-Markov model, perform hidden state inference on the observation sequence of the degradation stage to generate the hidden degradation stage sequence. S26. The sequence of hidden degeneration stages and their corresponding durations are expressed as hidden state transition relations and durations.

[0022] In this embodiment, S21 specifically includes: During the initial degradation stage, the discharge amplitude is <5pC, the discharge frequency is ≤20 times / second, the insulation resistance is ≥500MΩ, the temperature rise is ≤8℃, and the humidity is ≤70%RH; During the slow degradation stage, 5pC≤discharge amplitude≤20pC, 20≤discharge frequency≤60 times / second, 100MΩ≤insulation resistance≤500MΩ, 8℃≤temperature rise≤15℃, 70%RH≤humidity≤85%RH; Accelerated degradation stage: 20pC≤discharge amplitude≤60pC, 60≤discharge frequency≤200 times / second, 20MΩ≤insulation resistance≤100MΩ, 15℃≤temperature rise≤25℃, 85%RH≤humidity≤95%RH; Critical degradation stage: discharge amplitude > 60 pC; discharge frequency > 200 times / second; insulation resistance < 20 MΩ; temperature rise > 25℃; humidity > 95%RH; sheath current > 50 mA; During implementation, the above thresholds are compared segment by segment according to time period. If three or more consecutive sampling segments meet the judgment conditions of a certain stage, the time interval is classified into the corresponding degradation stage, forming a set of degradation stages.

[0023] In this embodiment, the calculation process of the degradation stage transition probability matrix in S23 specifically includes: S231. Read the degradation stage observation sequence according to the time index, identify the degradation stage to which adjacent time periods belong, and form a stage transition pair for any two adjacent time periods; count all transition pairs in the form of "starting stage - target stage", and record the count values ​​from the initial degradation stage to the slow degradation stage, from the slow degradation stage to the accelerated degradation stage, from the accelerated degradation stage to the critical degradation stage, and the unchanged count values ​​of each stage, forming a stage transition frequency array. S232. Taking a certain degradation stage as the starting stage, sum all the transition frequencies of it to obtain the total number of transitions in the starting stage; divide the transition frequency of this degradation stage to each target stage by the total number of transitions to obtain the corresponding transition probability; perform the above calculation on all degradation stages in the set of degradation stages in sequence, and fill all the transition probabilities from the starting stage to the target stage into the matrix in a fixed order to form the degradation stage transition probability matrix. S233. Write all the transition probabilities of the initial degradation stage, the slow degradation stage, the accelerated degradation stage, and the critical degradation stage into the same matrix, so that each row in the matrix corresponds to an initial stage and each column corresponds to a target stage, thus completing the generation of the degradation stage transition probability matrix.

[0024] In this embodiment, the hidden state inference process in S25 specifically includes: S251. Input the observation sequence of the degradation stage into the hidden semi-Markov model in chronological order, and calculate the observation probability value of the observation feature at each time index in the sequence according to the observation probability distribution in the model. S252. Based on the degradation stage transition probability matrix and duration distribution set obtained from model training, perform joint calculation of duration probability and transition probability for adjacent time periods to generate the state maintenance probability and state switching probability of each latent degradation stage under different durations. S253. Based on the forward-backward recursive state probability update method, the forward probability calculation is performed in chronological order, and the current observation probability, the stage maintenance probability of the previous moment, and the stage transition probability are combined to obtain the forward probability that the current time period belongs to each degeneration stage; then the backward probability calculation is performed in reverse chronological order, and the subsequent observation probability is combined with the probability of possible future duration to obtain the backward probability that the current time period belongs to each degeneration stage. S254. Multiply the forward and backward probabilities of the same time period component by component and normalize them to obtain the maximum posterior probability of the degeneration stage corresponding to that time period; use the degeneration stage number corresponding to the maximum posterior probability as the degeneration stage identifier of the time period. Perform the above inference process sequentially on all time indices, and arrange the obtained degeneration stages in chronological order to form a degeneration stage sequence.

[0025] In this embodiment, S3 specifically includes: S31. Combine the temporally corresponding degeneracy stages and state characteristics into state elements, arrange all state elements in chronological order to form the state space of a semi-Markov decision process; add inspection actions, maintenance actions, pre-test actions and replacement actions into the action space according to preset action numbers. S32. Based on the hidden state transition relationship, establish a state transition connection between the starting state element and the target state element corresponding to each stage, and write the transition probability of the corresponding stage in the hidden state transition relationship into the state transition connection to form a state transition rule. S33. Based on the duration expression, read each degenerate stage in the degenerate stage sequence, write the duration distribution corresponding to the degenerate stage in the duration expression into the state element associated with the degenerate stage, and form an action duration constraint for all actions in the action space by referencing the duration distribution. S34. Integrate the state space, action space, state transition rules and action duration constraints according to the preset structure to construct a state evolution model. Call the state evolution model to execute state reading, action selection, state transition and action duration allocation in sequence from any starting time index to generate state sequence, action sequence and action duration sequence, forming a simulation trajectory.

[0026] In this embodiment, S32 specifically includes: S321. Scan the degenerate stage sequence segment by segment according to the time index, and form a stage transition pair by forming two adjacent degenerate stages corresponding to two time indices. For each stage transition pair, read the transition probability value corresponding to the stage transition pair from the hidden state transition relationship, and record it as the stage transition probability from the starting stage to the target stage in the current time period. S322. Read the state feature sequence according to the same time index, extract the state element corresponding to the current time period from the state space as the starting state element, and extract the state element from the next time period of the state feature sequence as the target state element; establish a state transition connection between the starting state element and the target state element in the form of "starting state element → target state element", and write the stage transition probability into the probability field of the state transition connection. S323. Repeat the above steps for all adjacent time periods in the degenerate phase sequence, generate each "starting state element - target state element" state transition connection in sequence, and write the corresponding phase transition probability into the state transition connection structure in a fixed position to form a complete set of state transition connections. S324. Arrange all state transition connection sets according to the order of the state space, establish an independent record for all reachable target state elements corresponding to each state element, and write a fixed target state element number and the corresponding stage transition probability into each record to form a state transition rule.

[0027] In this embodiment, S33 specifically includes: S331. For each degenerate stage in the degenerate stage sequence, read the stage number of the degenerate stage in sequence according to the time index, and obtain the duration distribution parameter pair corresponding to the stage number from the duration expression; record the parameter pair as the duration distribution associated with the current degenerate stage, and write the duration distribution into the duration field of all state elements corresponding to the stage. S332. Read the inspection actions, maintenance actions, pre-test actions and replacement actions in the action space in sequence, and establish an action duration constraint field for each action; for each action in the action space, write the duration distribution parameter pair corresponding to the current degeneration stage into the action duration constraint field according to the action number, so that the action duration constraint field contains a fixed correspondence of "action number - duration distribution parameter pair". S333. For each time index of the degeneracy phase sequence, read the degeneracy phase number corresponding to the index, extract the duration distribution parameter pair associated with the phase from the action duration constraint field, and write it into the duration record of the action under the corresponding time index. S334. Arrange the action duration distributions corresponding to all time indices in chronological order to form an action duration constraint set; the action duration constraint set contains the duration distribution of each action under different degeneration stages.

[0028] In this embodiment, the state evolution model in S34 is constructed based on a semi-Markov decision process, specifically including: S341. Based on the aforementioned degeneracy stage sequence and state characteristics, combine the state characteristics corresponding to each time index with the degeneracy stage number to form a state element, and assign a unique number to each state element to form a state space. S342. Arrange the inspection actions, maintenance actions, pre-test actions, and replacement actions according to the action number, and write the duration distribution parameter pairs corresponding to the degradation stage to form the action space. S343. Establish the transition relationship between the initial state element and the target state element one by one according to the hidden state transition relationship, and write the corresponding transition probability into the state transition field to form the state transition rule. S344. Write the duration distribution parameters in the duration expression into the action duration constraint according to the degeneracy stage number; when generating the simulation trajectory, the state evolution model starts from any initial state and sequentially executes state reading, action selection, duration extraction and state transition to obtain the state sequence, action sequence and action duration sequence, and arranges them in time order to form the simulation trajectory.

[0029] In this embodiment, S4 specifically includes: S41. Set a time-weighted feature template, determine the state feature dimension and action coding dimension, and concatenate the state feature and action coding corresponding to each time index to generate a basic feature vector; read the state sequence, action sequence and action duration sequence in sequence according to the time index for the expert maintenance trajectory to construct the expert basic feature vector sequence; the time-weighted feature template determines the length of the state feature vector according to the feature dimension of the state feature sequence, and determines the length of the action coding vector according to the coding method of the action number in the action space; S42. Read the corresponding action duration for each basic feature vector in the expert basic feature vector sequence, multiply the action duration as a weight with each component of the basic feature vector to generate an expert time-weighted feature vector, and sum all expert time-weighted feature vectors according to the feature dimension to generate an expert cumulative time-weighted feature vector. S43. Read the state sequence, action sequence and action duration sequence of the simulated trajectory in the same way as S41 and S42, construct the simulation basic feature vector sequence and the simulation time weighted feature vector sequence, and accumulate them according to the feature dimension to generate the simulation cumulative time weighted feature vector. S44. Set the expert cumulative time weighted feature vector as the expert expected feature, set the simulated cumulative time weighted feature vector as the model expected feature, and combine the expert expected feature and the model expected feature as the time-weighted reward expression.

[0030] In this embodiment, S44 specifically includes: S441. The expert cumulative time weighted feature vector is used as the expert expected feature, and the state feature component and action coding component represented by each feature dimension are kept in the cumulative numerical form. S442. Use the simulated cumulative time-weighted feature vector as the model's expected feature, and record the model's time-weighted results in the simulated trajectory with the same feature dimension structure. S443. Arrange the expected features of experts and the expected features of the model in a fixed feature dimension order to form a feature pair consisting of two sets of feature vectors. The first component of each feature pair is the corresponding dimension value of the expected features of experts, and the second component is the corresponding dimension value of the expected features of the model. Combine all feature pairs to form a time-weighted reward expression.

[0031] In this embodiment, S5 specifically includes: S51. Initialize the reward function parameters and set the reward function parameter dimensions corresponding to the time-weighted reward expression; S52. Based on the time-weighted reward expression, calculate the logarithmic probability of the expert trajectory, multiply the cumulative time-weighted feature vector of the expert in the time-weighted reward expression by the reward function parameters component by component, and sum them to generate the expected reward value of the expert. S53. Calculate the logarithmic probability of the simulated trajectory based on the time-weighted reward expression, multiply the cumulative time-weighted feature vector of the model by the reward function parameters component by component, and sum them to generate the expected reward value of the model. S54. Perform a difference operation between the expert's expected reward value and the model's expected reward value according to the preset solution structure to generate the gradient components of the reward function parameters, and perform gradient iteration on the reward function parameters according to the iteration step size to update the reward function parameters. S55. Repeat S52 to S54 until the reward function parameters meet the convergence condition, thus forming the current reward function.

[0032] In this embodiment, the preset solution structure in S54 specifically includes: S541. Construct a parameter storage unit to record the initial values ​​of the reward function parameter vector, and set independent parameter components for each feature dimension to keep the parameter vector consistent with the feature dimensions of the time-weighted reward expression. S542. Set a fixed solution step size and write the solution step size as a scalar into the parameter update unit; S543. Construct a gradient calculation unit, subtract components one by one according to a fixed feature dimension to generate a gradient vector; pass the generated gradient vector into the parameter update unit, which adds the gradient vector component by component to the current reward function parameters according to the solution step size to form new reward function parameters; after each update, the parameter update unit writes the updated parameter vector back to the parameter storage unit to complete one iteration; S544. After each parameter vector update, the iterative control unit reads the new parameter vector and the previous parameter vector, subtracts each component of the corresponding feature dimension, and calculates the maximum absolute value of the difference vector. The maximum component is compared with a set threshold. If the maximum component is less than the set threshold, the iteration stops; otherwise, gradient calculation and parameter update continue until the threshold condition is met. In this embodiment, the threshold is 0.001.

[0033] In this embodiment, S6 specifically includes: S61. Receive the newly added expert maintenance trajectory, read the state sequence, action sequence and action duration sequence, concatenate the state feature and action code corresponding to each time index to form the basic feature vector of the new expert, and multiply the components according to the action duration to generate the time-weighted feature vector of the new expert. S62. Set time tags for the maintenance trajectory of newly added experts, convert the time tags into attenuation coefficients, multiply the attenuation coefficients by the time-weighted feature vector of the newly added experts component by component, and accumulate them according to the feature dimension to generate the cumulative time-weighted feature vector of the newly added experts; In this implementation, -0.05 multiplied by time is used as the exponent of e, and this term is used as the attenuation coefficient. S63. Add the newly added expert cumulative time weighted feature vector to the expert cumulative time weighted feature vector according to the feature dimension to form the superimposed expert cumulative time weighted feature vector, and arrange the superimposed expert cumulative time weighted feature vector and the model cumulative time weighted feature vector according to the feature dimension to form a new time weighted reward expression; S64. Update the reward function parameters according to the new time-weighted reward expression. Subtract the cumulative time-weighted feature vector of the experts from the cumulative time-weighted feature vector of the model according to the feature dimension to form the gradient vector of the reward function parameters. Add the gradient vector to the reward function parameters one by one according to the iteration step size to update the reward function parameters. S65. Repeat S61 to S64 until the difference between two consecutive updates of the reward function parameters is lower than the set threshold, and the updated reward function is obtained; in this embodiment, the threshold is the same as that in S544, that is, 0.001.

[0034] In this embodiment, S7 specifically includes: S71. Write the reward function into the state evolution model, read each state element in the state space, calculate the reward value of all actions in the corresponding action space, and form a state-action-reward set. S72. Read the transition probability of the state element to all reachable target states from the state transition rules, and multiply the reward value of each action by the corresponding transition probability in a fixed order to form the transition benefit component of the action in different target states; add the transition benefit components of the same action in all reachable target states to obtain the state action value corresponding to the action; repeat the above operation for all actions in the action space; write all state action value values ​​into the state action value recording unit in the order of state number and action number to form the state action value expression. S73. For each action corresponding to each state element in the state-action value expression, read the state-action value value and extract the duration distribution parameter pair corresponding to the action in the current degeneracy stage from the action duration constraint; according to the parameter pair, extract the action duration once to obtain the duration value in the current state; combine the action duration value with the corresponding state-action value in a fixed order, use the duration as a weighting factor, and perform the above calculation on all actions in the state in sequence to form a state-action value sequence. S74. According to the state-action value sequence, perform action selection for each state element in the state space to generate the corresponding state maintenance action, and arrange the maintenance actions corresponding to all state elements to form a cable accessory maintenance strategy; in this embodiment, the action selection is determined based on the maximum state action value.

[0035] Example 1: To verify the feasibility of this invention in practice, it was applied to a cable line that operates continuously, experiences significant load fluctuations, has high ambient humidity, and has a history of multiple minor anomalies. During long-term operation, some accessories of this cable line gradually exhibited phenomena such as increased partial discharge amplitude, decreased insulation resistance, and elevated conductor temperature. Traditional periodic maintenance methods could not promptly identify these degradation trends, resulting in delayed maintenance actions. The degradation stage identification based on a hidden semi-Markov model, the action duration representation based on a semi-Markov decision process, and the reward function learning and incremental update method based on maximum entropy inverse reinforcement learning proposed in this invention were fully validated in this scenario.

[0036] During operation, the working status of cable accessories is affected by factors such as load fluctuations, changes in ambient temperature and humidity, and multiple operation switches, resulting in a phased degradation process. Traditional methods, relying mainly on manual inspections or periodic tests, struggle to identify degradation phase changes in a timely manner, leading to a lack of effective measures in the early stages of degradation. This invention constructs a state feature sequence, aligning current amplitude, load level, conductor temperature, ambient humidity, and partial discharge detection data in chronological order, and combines expert maintenance trajectories with monitoring records to construct input data for degradation identification and strategy optimization. In this embodiment, approximately 24,000 time points of continuous operational data were collected, and over 200 expert maintenance trajectories were extracted from historical maintenance records.

[0037] Monitoring data and maintenance records are extracted from the application. The monitoring data is processed in chronological order of sampling time to construct a state feature sequence. Then, degradation-related feature sequences are constructed using partial discharge amplitude, insulation resistance, and humidity change rate. Degradation stage boundary thresholds are set based on feature change trends, and the duration distribution and transition relationship of degradation stages are obtained after training. A semi-Markov decision process is constructed using the degradation stage sequence, state feature sequence, and expert maintenance action sequence, ensuring that action duration is synchronously constrained with state changes. Subsequently, time-weighted features of expert maintenance are extracted, and an optimization objective of maximum entropy inverse reinforcement learning is constructed and the reward function is solved. When a new maintenance trajectory appears, a decay coefficient is generated based on the time label, and the reward function is incrementally updated, allowing the strategy to dynamically adjust with the operating status.

[0038] After updating the reward function, it is written into the state evolution model. By comparing the time-weighted value of each action in each state, a final maintenance strategy is generated and applied to cable accessory state assessment and maintenance decisions. The resulting maintenance strategy can distinguish critical degradation nodes and allocate action durations that are more consistent with the state stage, making maintenance actions more precise.

[0039] To compare the improved maintenance performance brought by this invention, traditional periodic maintenance, traditional state maintenance, maintenance methods based on degradation identification, maintenance methods based on action duration, and the final strategy of this invention were tested under the same operating scenario. The comparison metrics included the pre-failure degradation index, action duration, and cumulative maintenance benefit score. Test results show that the final maintenance strategy of this invention can reduce the average pre-failure degradation index to 0.31, significantly advance the maintenance timing, shorten the maintenance action duration to 7.2 hours, and increase the cumulative maintenance benefit score to 88.9, which is significantly better than other methods. This verifies the effectiveness of this invention in predicting degradation stages, allocating reasonable action durations, and generating high-quality maintenance strategies.

[0040] Table 1: Comparison of Cable Accessory Maintenance Strategy Optimization Data As can be seen from the table data, traditional periodic maintenance performs poorly in terms of both pre-failure degradation index and maintenance benefits, with a degradation index of 0.72 and an operation duration of 14.6 hours, indicating that maintenance actions are often delayed and require high resource input. Traditional condition-based maintenance can make judgments based on some condition characteristics, reducing the degradation index to 0.65, but it still cannot capture phased degradation characteristics, resulting in limited improvement in benefits.

[0041] After introducing a hidden semi-Markov model, the degradation stages can be effectively distinguished, reducing the degradation index to 0.54 and decreasing the duration of maintenance actions, indicating that degradation identification has a significant effect on early intervention. Further introducing a semi-Markov decision process to express the action duration makes the action allocation closer to the actual degradation stage requirements, reducing the degradation index to 0.47 and achieving a cumulative gain of 76.8.

[0042] This invention introduces maximum entropy inverse reinforcement learning, constructs a time-weighted reward representation, and performs incremental reward updates, enabling the policy to dynamically adapt to new maintenance trajectories. Ultimately, the degradation index is reduced to 0.31, the action duration is reduced to 7.2 hours, and the cumulative return reaches 88.9. All three indicators are superior to other methods, fully demonstrating the overall superiority of this invention in degradation understanding, decision-making, and policy updating.

[0043] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for optimizing intelligent maintenance strategies for cable accessories based on reinforcement learning, characterized in that, Includes the following steps: S1. Acquire cable accessory operation data, environmental parameters, monitoring records and expert maintenance trajectories, construct state characteristic sequences and action sequences, and form a degradation stage observation sequence; S2. Train a hidden semi-Markov model based on the observation sequence of the degradation stage, set the set of degradation stages, the stage transition probability matrix and the stage duration distribution, and obtain the hidden state transition relationship and duration expression. S3. Construct a semi-Markov decision process based on the hidden state transition relationship and duration expression, set the state space, action space, state transition rules and action duration constraints, form a state evolution model, and generate a simulated trajectory based on the state evolution model; S4. Define a time-weighted feature set containing state, action, and action duration; extract time-weighted feature information from expert-maintained trajectory and simulated trajectory; and construct a maximum entropy inverse reinforcement learning optimization objective. S5. Initialize the reward function parameters, calculate the expert trajectory probability distribution based on maximum entropy inverse reinforcement learning, perform gradient iteration to solve for the reward function parameters, and generate the current reward function. S6. Receive newly added expert-maintained trajectories, introduce trajectory time labels and data weight decay factors, extract newly added time-weighted feature information, perform incremental updates of reward function parameters based on maximum entropy inverse reinforcement learning, and generate updated reward functions. S7. Input the reward function into the state evolution model, use the reinforcement learning algorithm to execute the strategy iterative solution, and output the cable accessory maintenance strategy.

2. The method for optimizing intelligent maintenance strategies for cable accessories based on reinforcement learning according to claim 1, characterized in that, S1 specifically includes: S11. Obtain cable accessory operation data, environmental parameters and monitoring records, and align current amplitude, load level, conductor temperature, ambient temperature, ambient humidity, sheath current and partial discharge detection data in chronological order to form a state characteristic sequence; S12. Obtain the expert maintenance trajectory and arrange the inspection actions, replacement actions, maintenance actions and pre-test actions according to the maintenance start time and maintenance end time to construct the action sequence; S13. Based on partial discharge detection data, insulation resistance test data, conductor temperature and ambient humidity, perform degradation-related feature extraction, arrange the extracted features in chronological order to form a degradation stage observation sequence.

3. The method for optimizing intelligent maintenance strategies for cable accessories based on reinforcement learning according to claim 2, characterized in that, S3 specifically includes: S21. Based on the observation sequence of the degradation stage, a set of degradation stages for the hidden semi-Markov model is set, and the degradation stages are divided into initial degradation stage, slow degradation stage, accelerated degradation stage and critical degradation stage according to a preset order to form a set of degradation stages. S22. Set a state duration distribution function for each degradation stage in the degradation stage set. Based on the statistical distribution of degradation characteristics in each time period in the degradation stage observation sequence, construct the duration distribution of duration characteristics of each degradation stage to form a duration distribution set. S23. Based on the changes in degradation characteristics between adjacent time periods in the degradation stage observation sequence, calculate the state transition frequency between each degradation stage, and generate a degradation stage transition probability matrix based on the transition frequency. S24. Input the degradation stage observation sequence, degradation stage set, duration distribution set, and degradation stage transition probability matrix into the hidden semi-Markov model, perform model training, and obtain the hidden semi-Markov model parameters. S25. Based on the trained hidden semi-Markov model, perform hidden state inference on the observation sequence of the degradation stage to generate the hidden degradation stage sequence. S26. The sequence of hidden degeneration stages and their corresponding durations are expressed as hidden state transition relations and durations.

4. The method for optimizing intelligent maintenance strategies for cable accessories based on reinforcement learning according to claim 3, characterized in that, S3 specifically includes: S31. Combine the temporally corresponding degeneracy stages and state characteristics into state elements, arrange all state elements in chronological order to form the state space of a semi-Markov decision process; add inspection actions, maintenance actions, pre-test actions and replacement actions into the action space according to preset action numbers. S32. Based on the hidden state transition relationship, establish a state transition connection between the starting state element and the target state element corresponding to each stage, and write the transition probability of the corresponding stage in the hidden state transition relationship into the state transition connection to form a state transition rule. S33. Based on the duration expression, read each degenerate stage in the degenerate stage sequence, write the duration distribution corresponding to the degenerate stage in the duration expression into the state element associated with the degenerate stage, and form an action duration constraint for all actions in the action space by referencing the duration distribution. S34. Integrate the state space, action space, state transition rules and action duration constraints according to the preset structure to construct a state evolution model. Call the state evolution model to execute state reading, action selection, state transition and action duration allocation in sequence from any starting time index to generate state sequence, action sequence and action duration sequence, forming a simulation trajectory.

5. The method for optimizing intelligent maintenance strategies for cable accessories based on reinforcement learning according to claim 4, characterized in that, S4 specifically includes: S41. Set a time-weighted feature template, determine the state feature dimension and action coding dimension, and concatenate the state feature and action coding corresponding to each time index to generate a basic feature vector; read the state sequence, action sequence and action duration sequence of the expert maintenance trajectory according to the time index in sequence, and construct the expert basic feature vector sequence. S42. Read the corresponding action duration for each basic feature vector in the expert basic feature vector sequence, multiply the action duration as a weight with each component of the basic feature vector to generate an expert time-weighted feature vector, and sum all expert time-weighted feature vectors according to the feature dimension to generate an expert cumulative time-weighted feature vector. S43. Read the state sequence, action sequence and action duration sequence of the simulated trajectory in the same way as S41 and S42, construct the simulation basic feature vector sequence and the simulation time weighted feature vector sequence, and accumulate them according to the feature dimension to generate the simulation cumulative time weighted feature vector. S44. Set the expert cumulative time weighted feature vector as the expert expected feature, set the simulated cumulative time weighted feature vector as the model expected feature, and combine the expert expected feature and the model expected feature as the time-weighted reward expression.

6. The method for optimizing intelligent maintenance strategies for cable accessories based on reinforcement learning according to claim 5, characterized in that, S5 specifically includes: S51. Initialize the reward function parameters and set the reward function parameter dimensions corresponding to the time-weighted reward expression; S52. Based on the time-weighted reward expression, calculate the logarithmic probability of the expert trajectory, multiply the cumulative time-weighted feature vector of the expert in the time-weighted reward expression by the reward function parameters component by component, and sum them to generate the expected reward value of the expert. S53. Calculate the logarithmic probability of the simulated trajectory based on the time-weighted reward expression, multiply the cumulative time-weighted feature vector of the model by the reward function parameters component by component, and sum them to generate the expected reward value of the model. S54. Perform a difference operation between the expert's expected reward value and the model's expected reward value according to the preset solution structure to generate the gradient components of the reward function parameters, and perform gradient iteration on the reward function parameters according to the iteration step size to update the reward function parameters. S55. Repeat S52 to S54 until the reward function parameters meet the convergence condition, thus forming the current reward function.

7. The method for optimizing intelligent maintenance strategies for cable accessories based on reinforcement learning according to claim 6, characterized in that, S6 specifically includes: S61. Receive the newly added expert maintenance trajectory, read the state sequence, action sequence and action duration sequence, concatenate the state feature and action code corresponding to each time index to form the basic feature vector of the new expert, and multiply the components according to the action duration to generate the time-weighted feature vector of the new expert. S62. Set time tags for the maintenance trajectory of newly added experts, convert the time tags into attenuation coefficients, multiply the attenuation coefficients by the time-weighted feature vector of the newly added experts component by component, and accumulate them according to the feature dimension to generate the cumulative time-weighted feature vector of the newly added experts. S63. Add the newly added expert cumulative time weighted feature vector to the expert cumulative time weighted feature vector according to the feature dimension to form the superimposed expert cumulative time weighted feature vector, and arrange the superimposed expert cumulative time weighted feature vector and the model cumulative time weighted feature vector according to the feature dimension to form a new time weighted reward expression; S64. Update the reward function parameters according to the new time-weighted reward expression. Subtract the cumulative time-weighted feature vector of the experts from the cumulative time-weighted feature vector of the model according to the feature dimension to form the gradient vector of the reward function parameters. Add the gradient vector to the reward function parameters one by one according to the iteration step size to update the reward function parameters. S65. Repeat S61 to S64 until the difference between two consecutive updates of the reward function parameters is lower than the set threshold, and the updated reward function is obtained.

8. The method for optimizing intelligent maintenance strategies for cable accessories based on reinforcement learning according to claim 7, characterized in that, Specifically, S7 includes: S71. Write the reward function into the state evolution model, read each state element in the state space, calculate the reward value of all actions in the corresponding action space, and form a state-action-reward set. S72. Based on the state-action reward set, read the state transition rules, combine the action reward value corresponding to each state element with the state transition probability, and generate a state-action value expression. S73. Based on the expression of state-action value, the constraint of action duration is referenced, and the action duration is combined with the state-action value to form a state-action value sequence. S74. According to the state-action value sequence, perform action selection for each state element in the state space, generate the corresponding maintenance action, and arrange the maintenance actions corresponding to all state elements to form a cable accessory maintenance strategy.

Images