Series resonance whole process autonomous decision control system based on deep reinforcement learning

Abstract

The invention discloses a series resonance whole-process autonomous decision control system based on deep reinforcement learning, which relates to the technical field of intelligent control of power equipment and comprises a state sensing module, a Markov decision modeling module, a deep Q network decision module, a strategy optimization module and a control execution module. The state sensing module is used for collecting and preprocessing operation state parameters of the series resonance system, the Markov decision modeling module is used for modeling a series resonance control process into a Markov decision process, and the deep Q network decision module is used for outputting Q value estimation of a control action according to a current state. The strategy optimization module is used for carrying out training optimization on the deep Q network based on an experience playback mechanism, the control execution module is used for executing a control instruction output by a decision and acting on the series resonance device, autonomous intelligent control over the whole process of a series resonance test is achieved through deep reinforcement learning, and the control precision and the automation level are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for power equipment, specifically to a series resonant full-process autonomous decision-making control system based on deep reinforcement learning. Background Technology

[0002] Series resonant withstand voltage testing is an important test method in power systems for detecting the insulation performance of power equipment. It is widely used in factory testing and field testing of power equipment such as cables, GIS equipment, and transformers. The series resonant test system consists of a frequency converter, excitation transformer, reactor, and the test object. By adjusting the excitation frequency, the system reaches a resonant state, thereby generating a high voltage on the test object for withstand voltage testing. Traditional series resonant control systems mainly use manual control, with operators manually adjusting voltage and frequency parameters based on experience. This control method has the following shortcomings: First, it relies heavily on the professional experience of the operators, and the control effect varies greatly among different operators; Second, manual adjustment has a slow response speed, making it difficult to achieve precise parameter control; Third, under complex operating conditions, human judgment is prone to errors, posing safety hazards. To address the shortcomings of manual control, existing technologies employ PID-based automatic control methods. PID controllers achieve closed-loop control through a combination of proportional, integral, and derivative control actions. However, the PID control method has the following limitations: First, PID parameters need to be tuned according to the characteristics of different test samples, lacking adaptive adjustment capability; Second, PID control is a linear control based on error, which is difficult to handle complex control problems that are nonlinear and time-varying. Third, PID control is difficult to make intelligent decisions throughout the entire experiment and cannot autonomously select control strategies based on the system state.

[0003] With the development of artificial intelligence technology, deep reinforcement learning, as an intelligent decision-making method that combines deep learning and reinforcement learning, has shown superior performance in complex control problems. Through the interaction between the agent and the environment, deep reinforcement learning learns the mapping strategy from state to action, and can achieve autonomous decision-making control without relying on precise mathematical models. Therefore, applying deep reinforcement learning technology to series resonant control systems to achieve autonomous intelligent control of the entire process has important practical significance. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a series resonance full-process autonomous decision-making control system based on deep reinforcement learning. It achieves autonomous intelligent control of the entire series resonance experiment process through a deep Q-network, thereby improving control accuracy and automation level.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A series resonant autonomous decision-making control system based on deep reinforcement learning includes: a state perception module, a Markov decision modeling module, a deep Q-network decision module, a policy optimization module, and a control execution module.

[0006] The state sensing module is used to collect the operating state parameters of the series resonant system, and output a state vector after preprocessing the collected state parameters. The operating state parameters include the output voltage value, output current value, resonant frequency value, power factor value, and equipment temperature value.

[0007] The Markov decision modeling module is used to receive the state vector, model the series resonant control process as a Markov decision process, and define the state space, action space and reward function. The Markov decision process is the mathematical foundation of reinforcement learning and has the Markov property, that is, the next state is only related to the current state and the current action, and is independent of the historical state.

[0008] The deep Q-network decision module is used to receive the state vector as input, calculate the Q value corresponding to each control action through a deep neural network, and select the control action according to the Q value. The Q value represents the cumulative expected reward that can be obtained by performing a certain action in the current state.

[0009] The strategy optimization module is used to store state transition experience samples, and to sample training data from the stored experience samples based on the experience replay mechanism to optimize and update the network parameters in the deep Q network decision module. The experience replay mechanism improves the stability and efficiency of training by breaking the temporal correlation between samples.

[0010] The control execution module is used to receive the control actions output by the deep Q-network decision module, convert the control actions into control commands, and apply them to the series resonant device.

[0011] Furthermore, the state perception module includes a sensor acquisition unit and a data preprocessing unit.

[0012] The sensor acquisition unit includes a voltage sensor, a current sensor, a frequency detector, a power factor detector, and a temperature sensor, which are used to acquire the output voltage value, output current value, resonant frequency value, power factor value, and equipment temperature value of the series resonant system, respectively. The voltage sensor adopts a high-precision voltage divider and has a measurement range of 0 to 500kV. The current sensor uses a current transformer and has a measurement range of 0 to 1000A; The frequency detector has a measurement range of 20Hz to 300Hz; The temperature sensor uses a PT100 resistance temperature detector (RTD) and has a measurement range of -20℃ to 150℃.

[0013] The data preprocessing unit is used to normalize the collected parameters, mapping each parameter to a numerical range of 0 to 1 to form a five-dimensional state vector. The normalization process uses a linear mapping method, and the calculation formula is as follows: ; in, The value is the normalized value. These are the original collected values. This is the lower limit of the range of this parameter. This is the upper limit of the range of this parameter.

[0014] Furthermore, the state space defined in the Markov decision modeling module is a five-dimensional continuous space, which includes the normalized output voltage value, output current value, resonant frequency value, power factor value, and device temperature value.

[0015] Furthermore, the action space defined in the Markov decision modeling module includes seven discrete control actions, namely: voltage increase action, voltage decrease action, frequency increase action, frequency decrease action, maintain current state action, start resonance action, and stop resonance action.

[0016] The voltage increase action increases the output voltage of the voltage regulator by a fixed step, which is set to 1% of the rated voltage. The voltage decrease action decreases the output voltage of the voltage regulator by a fixed step. The frequency increase action increases the excitation frequency by a fixed step, which is set to 0.1Hz. The frequency decrease action decreases the excitation frequency by a fixed step. The current state maintenance action keeps the current voltage and frequency parameters unchanged. The resonant action control system enters the resonant operation state from the standby state. The resonant action stop action control system exits the resonant operation state and returns to the standby state.

[0017] Furthermore, the reward function defined in the Markov decision modeling module is: ; in, Total reward value; As a stability bonus; Reward points for efficiency; To protect the reward amount.

[0018] Stability bonus Used to evaluate the degree of deviation between the output voltage and the target voltage. When the output voltage deviation is within 2% of the target value, A value of 1 indicates that the system output is stable. When the output voltage deviation exceeds 2% of the target value, A value of -1 indicates that the system output is unstable.

[0019] Efficiency Reward Quantity Used to evaluate the resonant efficiency of the system, when the power factor is greater than 0.95. A value of 1 indicates that the system is operating in a high-efficiency state; when the power factor is less than 0.95, The value is the difference between the power factor value and 0.95. A negative difference indicates that the efficiency needs to be improved.

[0020] Protect reward portion Used to assess the safety status of the system; when the system is in a safe operating state. The value is 0, which means that when the system triggers a protection action, A value of -10 serves as a significant penalty signal, guiding the strategy to avoid triggering protection.

[0021] Furthermore, the deep Q-network decision module includes a main network and a target network, which have the same network structure. The purpose of adopting a dual-network structure is to solve the problem of Q-value overestimation and improve the stability of training.

[0022] The network structure includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, and an output layer. The input layer has 5 neurons, corresponding to a five-dimensional state vector. The first hidden layer has 128 neurons, and the ReLU activation function is used. The second hidden layer has 64 neurons, and the ReLU activation function is used. The third hidden layer has 32 neurons, and the ReLU activation function is used. The output layer has 7 neurons, corresponding to the Q values ​​of the seven control actions, and no activation function is used in the output layer.

[0023] The mathematical expression for the ReLU activation function is: ; in, The output of the activation function; The input value for the neuron; This means taking the larger of the two values.

[0024] Furthermore, the deep Q-network decision module adopts an ε-greedy strategy to select control actions. The ε-greedy strategy balances exploration and utilization, ensuring the exploration of unknown actions while utilizing learned knowledge to select better actions.

[0025] During the training phase, a random exploration action is performed with probability ε, that is, an action is randomly and uniformly sampled from the seven control actions, and the action whose Q value is the first in the Q value ranking among all actions in the current state is selected with probability 1-ε, that is, the action whose Q value output is the first in the Q value ranking.

[0026] The initial value of ε is set to 1.0, indicating completely random exploration in the early stage of training. Subsequently, it decays exponentially by 0.995 per training round, gradually reducing the exploration ratio. When ε decays to the lower limit of 0.01, it remains unchanged to ensure that a small amount of exploration is still retained in the later stage of training. In the testing phase, ε is set to 0, and the actions are executed completely according to the learned strategy.

[0027] Furthermore, the strategy optimization module includes an experience replay pool and a network update unit.

[0028] The experience replay pool is used to store state transition experience samples. Each experience sample contains five elements: the current state. Execution of actions Receive rewards Next state and termination mark The termination flag is used to indicate whether the current round has ended. The experience replay pool has a capacity of 100,000 experience samples. When the experience replay pool is full, new experience samples will overwrite the oldest samples that have been stored for the longest time.

[0029] The network update unit is used to train and update the main network. First, it randomly and uniformly samples 64 empirical samples from the empirical replay pool. Then, it calculates the target Q-value and the loss function value based on the temporal difference method. The loss function uses the mean squared error, and the calculation formula is as follows: ; in, The value of the loss function. This represents the batch size, with a value of 64. For the first The temporal difference target value of each sample. Main network for the first Each sample is in state Next action Q-value estimation.

[0030] The network update uses the Adam optimizer for gradient descent updates with a learning rate of 0.001. The Adam optimizer combines the advantages of the momentum method and the adaptive learning rate method, and has the characteristics of fast convergence speed and insensitivity to hyperparameters.

[0031] The target network parameters are synchronously copied from the main network every 1000 training steps. The target network parameters remain fixed between the two synchronizations to calculate a stable target Q value, thus avoiding training instability caused by drastic changes in the target value during the training process.

[0032] Furthermore, the deep Q-network decision module uses the DoubleDQN algorithm to calculate the temporal difference target value to solve the Q-value overestimation problem in the standard DQN algorithm. The calculation method for the temporal difference target value is as follows: ; in, The target value for time-series difference is... The reward value obtained in the current step. This is a discount factor with a value of 0.99, used to balance immediate rewards and long-term returns. The new state after the action is performed. These are candidate actions for the new state. The output function for the Q-value of the target network. The Q-value output function of the main network, This indicates selecting the action that makes the Q-value output function of the main network obtain the first Q-value among all candidate actions in the current state.

[0033] The core idea of ​​the DoubleDQN algorithm is to decouple action selection and action evaluation: the main network selects the action with the highest Q value in the new state, and the target network evaluates the Q value of that action. This method effectively reduces Q value overestimation and improves the accuracy of policy learning.

[0034] Furthermore, the control execution module includes an action analysis unit, a voltage regulation control unit, a frequency regulation control unit, and a protection control unit.

[0035] The action parsing unit is used to receive the control action number output by the deep Q network decision module, and parse the control action number into a specific control type and control parameters. The control action number is an integer from 0 to 6, which corresponds to seven discrete control actions.

[0036] The voltage regulation control unit is used to generate a voltage regulation control signal based on the voltage increase and voltage decrease actions, and to control the voltage regulator of the series resonant device. The voltage regulation control signal is an analog output with a voltage range of 0V to 10V, corresponding to 0% to 100% of the voltage regulator output.

[0037] The frequency modulation control unit is used to generate frequency modulation control signals based on frequency increase and frequency decrease actions, adjust the excitation frequency of the series resonant device, and output the frequency modulation control signal in pulse width modulation mode, thereby controlling the output frequency of the frequency converter by changing the pulse frequency.

[0038] The protection control unit is used to generate a protection control signal based on the stop resonance action, and cut off the power supply of the series resonant device in abnormal situations. The protection control signal is output by relay dry contacts to realize the emergency stop function.

[0039] This invention also provides a series resonant full-process autonomous decision-making control method based on deep reinforcement learning, applied to the above-mentioned system, comprising the following steps: Step S1: Collect the operating status parameters of the series resonant system through the state sensing module, including output voltage value, output current value, resonant frequency value, power factor value and equipment temperature value, perform normalization preprocessing on the operating status parameters, map each parameter to the value range of 0 to 1, and generate a five-dimensional current state vector.

[0040] Step S2: Input the current state vector into the main network of the deep Q-network decision module. The main network calculates through forward propagation, passing through the input layer, three hidden layers and the output layer in sequence, and outputs seven values, which correspond to the Q-value estimates of the seven control actions in the current state.

[0041] Step S3: According to the ε-greedy strategy, select a control action based on the calculated Q value, generate a random number between 0 and 1. If the random number is less than the current ε value, select an action randomly and evenly from the seven actions. If the random number is greater than the current ε value, select the action with the first Q value in the order.

[0042] Step S4: The selected control action is converted into a control command by the control execution module and applied to the series resonant device. The action parsing unit parses the action number into a specific control type. The voltage regulation control unit, frequency regulation control unit and protection control unit execute the corresponding control operations respectively.

[0043] Step S5: After the control action is completed, collect the new state vector after the control action is executed, calculate the reward value obtained according to the reward function, calculate the stability reward component, efficiency reward component and protection reward component respectively, and sum them up to obtain the total reward value.

[0044] Step S6: The current state vector, control action, reward value, new state vector and termination flag are used to form an experience sample, which is then stored in the experience replay pool of the policy optimization module. If the experience replay pool is not full, it is directly appended to the storage. If the experience replay pool is full, the sample with the longest storage time is overwritten.

[0045] Step S7: Determine if the number of samples in the experience replay pool is greater than the batch size of 64. If it is less than 64, skip the training step. If it is greater than 64, randomly and uniformly sample 64 experience samples from the experience replay pool. For each sampled experience sample, use the main network to calculate the action with the first-ranked Q value in the new state, use the target network to calculate the Q value corresponding to the action, calculate the target Q value according to the temporal difference formula, calculate the mean squared error loss between the Q value estimate of the main network and the target Q value, and use the Adam optimizer to update the gradient of the main network parameters.

[0046] Step S8: Determine whether the current training step has reached the target network update cycle of 1000 steps. If the update cycle has been reached, then all parameters of the main network are synchronously copied to the target network. If the update cycle has not been reached, then the parameters of the target network remain unchanged.

[0047] Step S9: Use the new state vector as the current state vector, return to step S2 to continue execution, and form a control closed loop. When the series resonance test reaches the predetermined time, the system triggers the protection action. When the stop command is received, the test process ends and the control loop is exited.

[0048] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows: First, this invention uses deep reinforcement learning technology to achieve autonomous decision-making control of the entire series resonance process. The intelligent agent can autonomously select control actions according to the real-time state of the system, without relying on a precise mathematical model of the system, and has strong adaptive capabilities.

[0049] Second, this invention models the series resonant control problem as a Markov decision process, defining a reasonable state space, action space, and reward function, enabling deep reinforcement learning to effectively learn control strategies.

[0050] Third, this invention uses a deep Q-network as a decision-making model and leverages the function approximation capability of deep neural networks to process high-dimensional continuous state spaces, thus solving the curse of dimensionality problem in complex problems in traditional Q-learning.

[0051] Fourth, this invention employs the DoubleDQN algorithm and the experience replay mechanism, which effectively solves the problems of Q-value overestimation and sample correlation, thereby improving the stability of training and the accuracy of the strategy.

[0052] Fifth, the reward function designed in this invention comprehensively considers the three aspects of system stability, efficiency, and security, guiding the strategy to learn a control strategy with excellent overall performance.

[0053] Sixth, this invention achieves automated control of the entire series resonance test process, including the stages of startup, adjustment, steady-state operation and shutdown, reducing the reliance on the professional experience of operators and improving test efficiency and safety. Attached Figure Description

[0054] Figure 1 This is an overall architecture diagram of a series resonant autonomous decision-making control system based on deep reinforcement learning, provided for an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the structure of a deep Q-network provided in an embodiment of the present invention.

[0056] Figure 3 A flowchart illustrating the workflow of the strategy optimization module provided in this embodiment of the invention.

[0057] Figure 4 The overall flowchart of the control method provided in the embodiment of the present invention.

[0058] Figure 5 This is a schematic diagram of the state transition of series resonance control provided in an embodiment of the present invention. Detailed Implementation

[0059] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0060] Example 1 like Figure 1 As shown, this embodiment provides an autonomous decision-making control system for the entire series resonance process based on deep reinforcement learning, which is applied to the series resonance withstand voltage test of 110kV cables. The system includes a state perception module, a Markov decision modeling module, a deep Q-network decision module, a strategy optimization module, and a control execution module.

[0061] The status sensing module is responsible for collecting the operating status of the series resonant system. In this embodiment, the voltage sensor uses a voltage divider with an accuracy class of 0.2 and a measurement range of 0 to 200kV. The current sensor uses a current transformer with an accuracy class of 0.5 and a measurement range of 0 to 500A. The frequency detector uses a digital frequency meter with a measurement range of 30Hz to 300Hz and a resolution of 0.01Hz. The power factor detector uses a digital power analyzer with a measurement accuracy of 0.1%. The temperature sensor uses a PT100 resistance temperature detector (RTD) and has a measurement range of 0 to 100°C.

[0062] The data preprocessing unit normalizes the five collected parameters. Taking the 110kV cable test in this embodiment as an example, the normalization parameters are set as follows: the voltage normalization range is 0 to 150kV. The current normalization range is 0 to 300A; The frequency normalization range is 30Hz to 300Hz; The power factor normalization range is 0 to 1; The temperature normalization range is 0 to 80°C.

[0063] The Markov decision modeling module models the series resonant control process as a Markov decision process. The state space is a normalized five-dimensional continuous vector, and the action space contains seven discrete actions: action 0 is to increase the voltage by 5%. Action 1 is to reduce the voltage by 5%; Action 2 is to increase the frequency by 0.5Hz; Action 3 is to reduce the frequency by 0.5Hz; Action 4 is to maintain the current state; Action 5 initiates resonance; Action 6 is to stop resonance.

[0064] The reward function is designed as follows: the stability reward weight is 0.4, the efficiency reward weight is 0.3, the protection reward weight is 0.3, the stability criterion is whether the deviation between the output voltage and the target voltage of 110kV is within 2.2kV, the efficiency criterion is whether the power factor is greater than 0.95, and the protection criterion is whether the system triggers overvoltage protection, overcurrent protection, and overtemperature protection.

[0065] like Figure 2 As shown, the deep Q-network decision module adopts a dual-network structure including a main network and a target network. The network structure is a fully connected neural network, containing 5 neurons in the input layer, 128 neurons in the first hidden layer, 64 neurons in the second hidden layer, 32 neurons in the third hidden layer, and 7 neurons in the output layer. The activation function of the hidden layers is the ReLU function, and the network parameters are initialized using the Xavier initialization method.

[0066] like Figure 3 As shown, the policy optimization module includes an experience replay pool and a network update unit. The capacity of the experience replay pool is set to 100,000. The network update uses the Adam optimizer with a learning rate of 0.001 and a batch size of 64. The target network is updated synchronously every 1,000 training steps. The initial value of ε for the ε-greedy policy is 1.0, the decay coefficient is 0.995, the lower bound is 0.01, and the discount factor γ is set to 0.99.

[0067] The control execution module includes an action analysis unit, a voltage regulation control unit, a frequency regulation control unit, and a protection control unit. The voltage regulation control unit outputs a 0 to 10V analog signal to control the voltage regulator with an accuracy of 0.1%. The frequency regulation control unit outputs a PWM signal to control the frequency converter with a frequency resolution of 0.1Hz. The protection control unit achieves emergency shutdown through relay output.

[0068] Before practical application, the system in this embodiment needs to be trained offline. During the training process, a simulation model of a series resonant system is used as the environment. The agent interacts and learns with the simulation environment. The simulation model is established based on the equivalent circuit parameters of the series resonant circuit and can simulate the system response under different test objects and different operating conditions.

[0069] The training settings are as follows: a total of 5000 training rounds, 500 steps per round, and a total of 2,500,000 training steps. After training is completed, the parameters of the main network are saved for actual control.

[0070] Training results show that after 5000 rounds of training, the agent's average round reward increased from the initial -50 to a stable value of around 180. The control strategy can adjust the system to a resonant state within 30 steps, the output voltage is stable within 2% deviation of the target value, and the power factor is greater than 0.95.

[0071] Example 2 like Figure 4 As shown, this embodiment provides a series resonance full-process autonomous decision-making control method based on deep reinforcement learning, which is applied to the system described in Embodiment 1 to perform full-process control of the 110kV cable series resonance withstand voltage test.

[0072] Before the experiment began, the operator set the following parameters: target voltage of 110kV, experiment time of 60 minutes, sampling period of 100ms, the system loaded the trained deep Q network parameters, set ε to 0, and executed control according to a deterministic strategy.

[0073] Step S1: The state perception module collects the initial state of the system. The initial state is standby state, the output voltage is 0kV, the output current is 0A, the frequency is 50Hz, the power factor is 0, the equipment temperature is 25℃, and the normalized state vector is [0,0,0.074,0,0.3125].

[0074] Step S2: Input the state vector into the main network of the deep Q network. The main network forward propagates to calculate the Q values ​​of the seven actions. An example of the Q value output is [-2.1, -2.3, -1.8, -1.9, -0.5, 5.2, -3.0].

[0075] Step S3: Since ε is 0, directly select the action with the first Q value in the numerical ranking. In this example, the Q value of action 5 (start resonance) is 5.2, which is the first in the numerical ranking of all actions, so action 5 is selected.

[0076] Step S4: The control execution module performs the start resonance action, the voltage regulation control unit sets the voltage regulator output to the initial value of 10%, and the frequency regulation control unit starts frequency scanning to find the resonance point.

[0077] Step S5: After a sampling period of 100ms, acquire the new state. An example of the new state is: output voltage 15kV, output current 50A, frequency 35Hz, power factor 0.3, temperature 26℃. The normalized new state vector is [0.1,0.167,0.019,0.3,0.325]. Calculate the reward value: stability reward is -1 (deviation exceeds 2%), efficiency reward is -0.65 (0.3-0.95), protection reward is 0, and the total reward is 0.4×(-1)+0.3×(-0.65)+0.3×0=-0.595.

[0078] Step S6: Store the experience samples in the experience replay pool. Since this is the testing phase and no training is being conducted, this step can be skipped.

[0079] Steps S7 to S8: No network updates are performed during the testing phase; skip these two steps.

[0080] Step S9: Set the new state as the current state and return to step S2 to continue execution.

[0081] The control process continues, and after about 50 control cycles, the system enters the resonance state, the output voltage reaches the target value of around 110kV, and the power factor increases to 0.98. At this time, the deep Q network mainly outputs the action to maintain the current state (Action 4) to keep the system running stably.

[0082] like Figure 5 As shown, the entire control process can be divided into three stages: the start-up stage (0 to 5 seconds), the adjustment stage (5 to 10 seconds), and the steady-state stage (10 seconds to 60 minutes). The start-up stage mainly performs the start-up resonance action and frequency adjustment action to find the resonance point. The adjustment stage mainly performs the voltage adjustment action to gradually increase the output voltage to the target value. The steady-state stage mainly performs the action to maintain the current state and keep the output voltage stable.

[0083] During the test, if the system detects an abnormal situation, the deep Q network can promptly output a stop resonance action. Abnormal situations include: output voltage exceeding the protection setting of 132kV, output current exceeding the protection setting of 300A, and equipment temperature exceeding the protection setting of 70℃.

[0084] After the test, the system recorded the test data. The statistical results of the test data in this embodiment are as follows: the average output voltage is 109.8kV, and the voltage deviation range is -1.5% to 1.8%. The average power factor is 0.97; Statistics on the number of control actions executed: voltage increase action executed 15 times, voltage decrease action executed 12 times, frequency increase action executed 8 times, frequency decrease action executed 6 times, maintain current state action executed 35,900 times, start resonance action executed 1 time, stop resonance action executed 1 time.

[0085] Example 3 This embodiment provides a detailed description of the training process for deep Q-networks.

[0086] The training environment uses a series resonant system simulator developed based on the Python language. The simulator is established based on the differential equation of the series resonant circuit and can simulate the dynamic response of the system under different control inputs. The simulation step size of the simulator is set to 1ms and the control period is set to 100ms.

[0087] The training parameters are set as follows: the experience replay pool capacity is 100,000; Batch size is 64; The learning rate is 0.001; The discount factor is 0.99; The initial value of ε is 1.0; The ε attenuation coefficient is 0.995; The lower limit of ε is 0.01; The target network update cycle is 1000 steps; The total number of training rounds is 5000. The number of steps per round is 500.

[0088] The following strategies are employed during the training process to improve training efficiency and strategy quality: First, a course-based learning strategy is adopted, using simple scenarios (fixed test sample parameters, fixed target voltage) in the early stages of training, and gradually increasing the complexity of the scenarios (random test sample parameters, random target voltage) in the later stages of training.

[0089] Second, a reward-shaping technique is used to add intermediate state rewards to the basic reward function, guiding the agent to learn to control the trajectory correctly.

[0090] Third, a priority experience replay mechanism is adopted, which allocates sampling priority according to the size of the temporal difference error of the experience samples. Samples with larger temporal difference errors have a higher sampling probability, which accelerates the learning of difficult samples.

[0091] During training, the following metrics were monitored: average round reward, average round steps, Q-value estimate, and loss function value. The training curve showed that the average round reward increased rapidly in the first 1000 rounds, increased slowly from 1000 to 3000 rounds, and stabilized after 3000 rounds.

[0092] After training, the learned strategy was evaluated and tested in the following scenarios: standard scenario (the parameters of the test subject are the same as those in training), generalization scenario (the parameters of the test subject are different from those in training), and perturbation scenario (random perturbations are added during operation). The test results show that the learned strategy can achieve stable control in all scenarios, which proves the effectiveness and generalization ability of the strategy.

[0093] In summary, this invention achieves autonomous decision-making control of the entire series resonance process through deep reinforcement learning technology. It features strong adaptability, high control accuracy, and high degree of automation, and can effectively improve the efficiency and safety of series resonance withstand voltage tests.

Claims

1. A series resonant fully autonomous decision-making control system based on deep reinforcement learning, characterized in that: include: The module consists of a state awareness module, a Markov decision modeling module, a deep Q-network decision module, a policy optimization module, and a control execution module. The state sensing module is used to collect the operating state parameters of the series resonant system, and output a state vector after preprocessing the collected state parameters. The operating state parameters include the output voltage value, output current value, resonant frequency value, power factor value, and equipment temperature value. The Markov decision modeling module is used to receive the state vector, model the series resonant control process as a Markov decision process, and define the state space, action space and reward function. The deep Q-network decision module is used to receive the state vector as input, calculate the Q value corresponding to each control action through a deep neural network, and select the control action based on the Q value. The strategy optimization module is used to store state transition experience samples, and to sample training data from the stored experience samples based on the experience replay mechanism to optimize and update the network parameters in the deep Q network decision module. The control execution module is used to receive the control actions output by the deep Q-network decision module, convert the control actions into control commands, and apply them to the series resonant device.

2. The series resonant fully autonomous decision-making control system based on deep reinforcement learning according to claim 1, characterized in that: The state perception module includes a sensor acquisition unit and a data preprocessing unit; The sensor acquisition unit includes a voltage sensor, a current sensor, a frequency detector, a power factor detector, and a temperature sensor, which are used to acquire the output voltage value, output current value, resonant frequency value, power factor value, and equipment temperature value of the series resonant system, respectively. The data preprocessing unit is used to normalize the collected parameters, mapping each parameter to a numerical range of 0 to 1, forming a five-dimensional state vector.

3. The series resonant fully autonomous decision-making control system based on deep reinforcement learning according to claim 1, characterized in that: The action space defined in the Markov decision modeling module includes seven discrete control actions: voltage increase action, voltage decrease action, frequency increase action, frequency decrease action, maintain current state action, start resonance action, and stop resonance action.

4. The series resonant fully autonomous decision-making control system based on deep reinforcement learning according to claim 1, characterized in that: The reward function defined in the Markov decision modeling module is: ； in, The total reward value, As a stability bonus component, when the output voltage deviation is within 2% of the target value... The value is set to 1 when the output voltage deviation exceeds 2% of the target value. The value is -1. As an efficiency bonus component, when the power factor is greater than 0.95 The value is 1 when the power factor is less than 0.

95. The value is the difference between the power factor value and 0.

95. To protect reward amounts, when the system is in a safe operating state... The value is 0 when the system triggers a protection action. The value is -10.

5. The series resonant fully autonomous decision-making control system based on deep reinforcement learning according to claim 1, characterized in that: The deep Q-network decision module includes a main network and a target network, which have the same network structure. The network structure includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, and an output layer. The input layer has 5 neurons, corresponding to a five-dimensional state vector. The first hidden layer has 128 neurons, and the activation function is ReLU. The second hidden layer has 64 neurons, and the activation function is ReLU. The third hidden layer has 32 neurons, and the activation function is ReLU. The output layer has 7 neurons, corresponding to the Q-values ​​of seven control actions.

6. The series resonant fully autonomous decision-making control system based on deep reinforcement learning according to claim 1, characterized in that: The deep Q-network decision module adopts an ε-greedy strategy to select control actions. During the training phase, random exploration actions are executed with probability ε, and the action with probability 1-ε is selected as the action with the highest Q value among all actions in the current state. The initial value of ε is set to 1.0, and it decays at a rate of 0.995 per training round until it decays to the lower limit of 0.01 and remains unchanged.

7. The series resonant fully autonomous decision-making control system based on deep reinforcement learning according to claim 1, characterized in that: The strategy optimization module includes an experience replay pool and a network update unit; The experience replay pool is used to store state transition experience samples. Each experience sample contains five elements: current state, executed action, reward obtained, next state, and termination flag. The capacity of the experience replay pool is 100,000 experience samples. The network update unit is used to randomly sample 64 empirical samples from the empirical replay pool, calculate the loss function value based on the temporal difference method, use the Adam optimizer to update the main network parameters with gradients, and set the learning rate to 0.

001. The target network parameters are synchronously copied from the main network every 1000 training steps.

8. The series resonant fully autonomous decision-making control system based on deep reinforcement learning according to claim 1, characterized in that: The control execution module includes an action analysis unit, a voltage regulation control unit, a frequency regulation control unit, and a protection control unit; The action parsing unit is used to receive the control action number output by the deep Q network decision module and parse the control action number into a specific control type and control parameters. The voltage regulation control unit is used to generate voltage regulation control signals based on voltage increase and voltage decrease actions, and to control the voltage regulator of the series resonant device. The frequency modulation control unit is used to generate a frequency modulation control signal based on the frequency increase and frequency decrease actions, and to adjust the excitation frequency of the series resonant device. The protection control unit is used to generate a protection control signal based on the stop resonance action, and to cut off the power supply of the series resonant device in abnormal situations.

Images