Distributed reinforcement learning conductivity sensor parameter optimization method

Abstract

The invention discloses a conductivity sensor parameter optimization method based on distributed reinforcement learning. The method can solve the problem of multi-target parameter optimization in the design process of a high-performance conductivity sensor. The method is composed of a distributed system and reinforcement learning. Different objective functions are optimized through a plurality of reinforcement learning agents, and reinforcement learning training is accelerated and an optimal strategy is quickly made through sharing a parameter optimal solution and minimum total loss among the agents. The optimal solution set of each parameter is obtained through distributed reinforcement learning, the optimal design parameter of the conductivity sensor is determined by calculating the total loss to judge whether the minimum value is reached, and the high-performance conductivity sensor is designed according to the optimal design parameter.

Description

Technical Field

[0001] The present invention belongs to the intersection of intelligent sensor technology and machine learning, and relates to a distributed reinforcement learning conductivity sensor parameter optimization method suitable for the design of an efficient conductivity sensor for a valve cooling system. Background Art

[0002] With the development of smart grid and energy Internet, micro sensors have the characteristics of small size, low energy consumption, high sensitivity and high precision, and fast response speed. Micro sensors are widely used in real-time monitoring of smart grid data to improve the intelligence level of power systems. Conductivity sensors have the characteristics of high sensitivity, high precision, real-time monitoring, wide measurement range, temperature compensation, and high reliability. High-performance conductivity sensors are the key to ensuring the safe and efficient operation of valve cooling systems. In recent years, artificial intelligence methods have developed rapidly in the field of sensor design, and they play a vital role in sensor parameter optimization and big data processing. Reinforcement learning continuously optimizes the decision-making process through interaction with the environment, and is suitable for conductivity sensor control and optimization in dynamic systems. The goal of reinforcement learning is to enable the agent to maximize its cumulative reward after training. The agent continuously acquires knowledge in the cycle of action and feedback, gradually updates its strategy to better adapt to the environment, and finds the optimal decision path in various scenarios.

[0003] The distributed system adopts different reinforcement learning strategies for different independent variables by establishing multiple reinforcement learning agents. Distributed reinforcement learning involves the mutual cooperation between multiple agents. Each agent learns in its own task and cooperates through sharing strategies, thereby accelerating the overall learning process. Therefore, the present invention combines the distributed system and the reinforcement learning method. The proposed method can shorten the optimization time of the conductivity sensor parameters and improve the overall performance of the conductivity sensor. Summary of the invention

[0004] The present invention proposes a distributed reinforcement learning conductivity sensor parameter optimization method, which applies a reinforcement learning method and a distributed system to the conductivity sensor parameter optimization to achieve the optimal design parameters of the conductivity sensor.

[0005] To achieve the above object, the present invention provides the following technical solution: a distributed reinforcement learning conductivity sensor parameter optimization method, comprising the following steps:

[0006] Step 1: Establish the total loss model of the conductivity sensor;

[0007] Step 2: Set the initial parameters of distributed reinforcement learning;

[0008] Step 3: Determine the objective function of distributed reinforcement learning;

[0009] Step 4: Calculate the reward value of distributed reinforcement learning, and output the corresponding reinforcement learning state and strategy when the maximum reward value is reached;

[0010] Step 5: Calculate the reward values ​​of the three distributed reinforcement learnings and obtain the states and strategies of the three distributed reinforcement learnings;

[0011] If the three distributed reinforcement learnings do not obtain the maximum reward value, repeat step 4;

[0012] If the distributed reinforcement learning obtains the maximum reward value, execute step 6;

[0013] Step 6: The agents use a consensus protocol to exchange optimal parameter solutions and minimum total loss to achieve distributed training of multiple agents.

[0014] Step 7: Get the optimal solution set of each parameter;

[0015] Step 8: Calculate the core loss and winding loss of the conductivity sensor;

[0016] Step 9: Obtain the optimal design parameters of the conductivity sensor.

[0017] Preferably, the specific method for establishing the total loss model of the conductivity sensor is:

[0018] The total loss of the conductivity sensor includes core loss and winding loss. The total core loss under non-sinusoidal excitation is:

[0019]

[0020] Among them, P core is the core loss under non-sinusoidal excitation; T is the magnetization period; k i is the coefficient related to material properties; is the derivative of magnetic flux density with respect to time; ΔB is the change in magnetic flux density; dt is the time increment; α is the constant obtained by fitting the core loss test data under sinusoidal excitation, reflecting the change of core loss with frequency; β is the exponent of magnetic flux density, reflecting the change of core loss with magnetic flux density;

[0021] The total winding loss under non-sinusoidal excitation is:

[0022]

[0023] Among them, P windings is the total winding loss; is the square of the effective value of the current of the nth harmonic; N is the number of current harmonics;

[0024] Preferably, the method for determining the objective function of distributed reinforcement learning is:

[0025] Taking the core loss, winding loss and leakage inductance of the conductivity sensor as the optimization objective function, the total loss of the conductivity sensor is:

[0026] f1(x)=min{P core +P windings}

[0027] Among them, f1(x) is the objective function 1; min{} is the minimization function;

[0028] Control conductivity sensor leakage inductance L δ is the set value L const , the conductivity sensor leakage inductance optimization function is:

[0029]

[0030] Where f2(x) is the objective function 2; L δ is the leakage inductance of the conductivity sensor.

[0031] Preferably, the specific method of calculating the reward value of reinforcement learning and outputting the corresponding reinforcement learning state and strategy when the maximum reward value is output is:

[0032] The reinforcement learning agent takes maximizing the cumulative reward as the training goal. The cumulative reward at time t can be expressed as:

[0033]

[0034] Among them, R t is the reward value obtained at time t; γ is the discount factor, γ∈[0,1]; it reflects that the importance of current feedback decreases over time; r t+1 represents the reward value obtained by the agent when performing an action at time t+1; r t+2 represents the reward value obtained by the agent when performing the action at time t+2; r t+3 represents the reward value obtained by the agent when performing the action at time t+3; r t+k+1 Represents the reward value obtained by the agent when performing an action at time t+k+1.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows: the method can solve the multi-objective parameter optimization problem faced in the design process of a high-performance conductivity sensor. The method is composed of a distributed system and reinforcement learning. Different objective functions are optimized by multiple reinforcement learning agents. Each agent accelerates the training of reinforcement learning and quickly makes the optimal strategy by sharing the optimal solution of parameters and the lowest total loss. Distributed reinforcement learning obtains the optimal solution set of each parameter. The optimal design parameters of the conductivity sensor are determined by calculating the total loss to determine whether it reaches the minimum, and a high-performance conductivity sensor is designed according to the optimal design parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 1 is a diagram of a conductivity sensor parameter optimization method using distributed reinforcement learning according to the method of the present invention.

[0037] Figure 2 Graph showing a distributed reinforcement learning method of the method of the present invention. DETAILED DESCRIPTION

[0038] A distributed reinforcement learning conductivity sensor parameter optimization method proposed in the present invention is described in detail as follows in conjunction with the accompanying drawings: Figure 1 1 is a diagram of the conductivity sensor parameter optimization method of distributed reinforcement learning of the method of the present invention. First, the conductivity sensor design problem is modeled and real number encoded. Initialize the reinforcement learning parameters, construct the objective function of the conductivity sensor core loss and winding loss and add constraints. Distributed reinforcement learning is used to train the design parameters of the conductivity sensor, obtain the maximum output reward value of the distributed reinforcement learning, and output the state and strategy of the distributed reinforcement learning. The optimal solution set of each parameter of the conductivity sensor is obtained by the distributed reinforcement learning method, and the core loss and winding loss of the conductivity sensor under the optimized parameters are calculated. By judging whether the total loss is the lowest, if the total loss is the lowest, the optimal design parameters of the conductivity sensor are output. If the total loss does not reach the minimum value, the optimal design parameters of the conductivity sensor when the total loss is the lowest are obtained through continuous training through distributed reinforcement learning. A high-performance conductivity sensor is designed according to the optimal design parameters.

[0039] Figure 2 This is a diagram of the distributed reinforcement learning method of the method of the present invention. The present invention uses three reinforcement learning agents to build a distributed architecture. During the parameter optimization process of reinforcement learning, agent 1 and agent 2 use the optimal parameter solution and the lowest total loss to exchange consistency protocols; agent 2 and agent 3 use the optimal parameter solution and the lowest total loss to exchange consistency protocols; agent 3 and agent 1 use the optimal electrical parameter solution and the lowest total loss to exchange consistency protocols; a distributed training network for reinforcement learning is formed.

[0040] The specific scheme is as follows: A distributed reinforcement learning conductivity sensor parameter optimization method. This method applies reinforcement learning methods and distributed systems to conductivity sensor parameter optimization to achieve the optimal design parameters of the conductivity sensor.

[0041] In the design process of conductivity sensor, the most important thing to consider is the loss of internal material of conductivity sensor. The internal loss of conductivity sensor mainly includes core loss and winding loss.

[0042] The core loss under sinusoidal excitation is calculated using the Steinmetz equation:

[0043]

[0044] Among them, P v is the core loss; K is the constant obtained by fitting the core loss test data under sinusoidal excitation; f is the sinusoidal excitation frequency; α is the constant obtained by fitting the core loss test data under sinusoidal excitation, reflecting that the core loss changes with frequency; B m is the core magnetic flux density amplitude; β is the index of magnetic flux density, which reflects the change of core loss with magnetic flux density.

[0045] The core loss under non-sinusoidal excitation is not only related to the rate of change of magnetic flux, but also to the magnetization period. The core loss under non-sinusoidal excitation is:

[0046]

[0047] Among them, P core is the core loss under non-sinusoidal excitation; T is the magnetization period; k i is the coefficient related to material properties; is the time derivative of magnetic flux density; ΔB is the change in magnetic flux density; dt is the time increment.

[0048] k i The calculation formula is:

[0049]

[0050] Among them, cosθ is the cosine function, which describes the component of the magnetic field in a certain direction; θ is the angle variable; and dθ is the differential of the angle θ.

[0051] The eddy current field distribution of the conductor is complex at high frequencies, and the winding loss is related to the winding method of the winding in the core window. The conductivity sensor winding loss is based on a one-dimensional electromagnetic field model. The winding conductor is copper, the magnetic permeability of the core is infinite, and the winding height is equal to the core window height. The conductor AC resistance factor is:

[0052]

[0053] Among them, F r,n is the AC resistance factor under the nth harmonic current; R ac is the AC resistance of the winding conductor; R dc is the DC resistance of the winding conductor; m is the number of winding layers; d w is the copper skin thickness; δ is the skin depth; It is the correction factor due to the skin effect of the conductor under AC conditions; is the correction factor of the conductor due to the proximity effect under AC conditions; Δ is a frequency-related parameter that reflects the influence of the skin effect.

[0054] When the winding height is not equal to the core window height, the conductor AC resistance factor is:

[0055]

[0056] Among them, h w is the winding height; h c is the core window height.

[0057] When the winding is a circular conductor, the conductor AC resistance factor is corrected by the circular conductor equivalent method:

[0058]

[0059] Among them, d r is the diameter of the circular conductor.

[0060] When the primary excitation current of the conductivity sensor is a non-sinusoidal wave, the AC resistance factor at different harmonic frequencies is calculated respectively, and the total winding loss is:

[0061]

[0062] Among them, P windings is the total winding loss; is the square of the effective value of the nth harmonic current; N is the number of current harmonics.

[0063] The leakage inductance of the conductivity sensor is calculated by the leakage magnetic energy. The leakage inductance is calculated as:

[0064]

[0065] Among them, W leakage is leakage inductance; μ0 is relative magnetic permeability; H is magnetic field intensity; L δ It is the total leakage inductance of the secondary side converted to the primary side and the primary side leakage inductance; I1 is the peak value of the primary side excitation current.

[0066] The distribution of magnetic field intensity in the primary and secondary windings with distance x is:

[0067]

[0068] Wherein, H1(x) is the magnetic field strength at a certain distance x inside the winding; H0 is the reference magnetic field strength; ρ is a complex parameter; and j is an imaginary unit.

[0069] The magnitude of the field strength between winding layers is related to the current and the number of winding layers. The field strength between winding layers is:

[0070]

[0071] Among them, H2(x) is the field strength between winding layers.

[0072] The winding structure of the conductivity sensor, when there is no cross winding between the primary and secondary windings, the total leakage magnetic energy of the conductivity sensor is:

[0073] W leakage =W pri +W sec +W isolation +W ins(pri) +W ins(sec)

[0074] Among them, W pri W is the leakage magnetic energy in the primary winding of the conductivity sensor; sec W is the leakage magnetic energy in the secondary winding of the conductivity sensor; isolation is the leakage magnetic energy between the primary winding and the secondary winding; W ins(pri) is the leakage magnetic energy between the primary winding layers; W ins(sec) is the leakage magnetic energy between the secondary winding layers.

[0075] The core loss, winding loss and leakage inductance of the conductivity sensor are related to the conductivity sensor winding structure parameters. When the number of turns of the secondary winding increases, the magnetic flux density amplitude of the conductivity sensor core decreases, the core loss decreases, and the winding loss increases. Therefore, the optimization goals of the core loss and winding loss are contradictory.

[0076] When the distance between the primary winding layers and the distance between the secondary winding layers of the conductivity sensor increase, the equivalent length of the winding increases, and the winding loss increases. When the conductivity sensor winding is not cross-wound, and the insulation distance between the primary and secondary windings of the conductivity sensor remains unchanged, the leakage inductance of the conductivity sensor increases. The distance between the primary winding layers and the distance between the secondary winding layers of the conductivity sensor will affect the winding loss and leakage inductance of the conductivity sensor, and there is a contradiction between these two optimization goals.

[0077] Therefore, in order to design a high-performance conductivity sensor, each parameter must be reasonably designed. Under the insulation constraint, the value range of the primary and secondary winding parameters of the conductivity sensor is limited. In order to prevent the conductivity sensor core from saturating, when the internal core ratio of the conductivity sensor is determined, the number of turns of the secondary winding of the conductivity sensor is also constrained.

[0078] The design of high-frequency conductivity sensors needs to comprehensively consider many factors such as conductivity sensor loss, weight, and volume. There are often conflicts between the optimization targets of conductivity sensor parameters, and multiple optimization targets cannot be optimized at the same time. Therefore, a distributed reinforcement learning method is used to optimize each optimization target, and the optimized parameters are obtained through information sharing. The distributed reinforcement learning method can obtain the optimal solution for multiple target optimizations, and share the optimal solution of multiple targets to each reinforcement learning agent to obtain the optimal output.

[0079] The reinforcement learning agent takes the maximization of cumulative rewards as the training goal, the environment state at time t, A is the action that the reinforcement learning agent can perform, and a t ∈A represents the action performed by the agent at time t, R is the reward value of reinforcement learning, and r t ∈R represents the reward value obtained by the agent when performing an action at time t, P is the state transition probability of reinforcement learning, and represents the agent performing action a t From the state t Transition to next state s t+1 probability.

[0080] The reinforcement learning agent takes maximizing the cumulative reward as the training goal. The cumulative reward at time t can be expressed as:

[0081]

[0082] Among them, R t is the reward value obtained at time t; γ is the discount factor, γ∈[0,1]; it reflects that the importance of current feedback decreases over time; r t+1 represents the reward value obtained by the agent when performing an action at time t+1; r t+2 represents the reward value obtained by the agent when performing the action at time t+2; r t+3 represents the reward value obtained by the agent when performing the action at time t+3; r t+k+1 Represents the reward value obtained by the agent when performing an action at time t+k+1.

[0083] State value function V π (s) is used to evaluate the quality of the status. The status value is defined as follows:

[0084]

[0085] Among them, V π (s) represents the cumulative reward obtained by the agent according to strategy π starting from state s until the end of the interaction.

[0086] In each action of reinforcement learning, this model requires obtaining a current state s and action a from the set S, and updating the state s from s t-1 to t The model's prediction of the test sample and the label selected by the model in the next step are determined by S t and t Decide.

[0087] Optimal Reward Value for Reinforcement Learning The definition of is as follows:

[0088]

[0089] in, is the optimal reward value; E π is the policy variable of the environment state; R t |S t =s is the reward value at time t.

[0090] Behavior value function Q π (s,a) is used to evaluate the quality of an action and is defined as follows:

[0091]

[0092] Among them, Q π (s,a) is the cumulative reward obtained by the agent when it performs a series of actions a in state s according to strategy π until the interaction ends; A t = a is the action performed at time t; R t |S t =s,A t =a is the reward value obtained by performing action a under state s at time t.

[0093] Maximum reward corresponding to for:

[0094]

[0095] in, It is the maximum cumulative reward obtained by the agent when it performs a series of actions a in state s according to the strategy π until the interaction ends.

[0096] Update the Q value using the Bellman formula:

[0097] Q i+1 (s,a)=E π [R t+γmaxQ i (s t+1 ,a t+1 )|S t =s,A t =a]

[0098] Among them, Q i+1 (s,a) is the strategy i+1 that performs a series of actions a in state s and updates the cumulative reward obtained from the interaction; Q i (s t+1 ,a t+1 ) is the cumulative reward obtained by strategy i in state s+1 after executing a series of actions a+1 and updating the end of the interaction; maxQ i (s t+1 ,a t+1 ) is the strategy i that performs a series of actions a+1 in state s+1 and updates the maximum cumulative reward value obtained at the end of the interaction.

[0099] Among them, when i→∞,Q i Gradually converge to the optimal behavior value function and obtain the optimal strategy:

[0100] π * = argmax a∈A Q π * (s,a)

[0101] Among them, π * is the optimal strategy; arg() is the inverse trigonometric function; max a∈A Q π * (s,a) is the maximum cumulative reward value obtained when executing action a under the π strategy.

[0102] Conductivity sensor loss calculation model and leakage inductance calculation model. The conductivity sensor optimization design goal is to minimize the total loss and control the leakage inductance at the set value.

[0103] The total loss of the conductivity sensor includes the core loss and winding loss. The total loss of the conductivity sensor is:

[0104] f1(x)=min{P core +P windings}

[0105] Control conductivity sensor leakage inductance L δ is the set value L const , the conductivity sensor leakage inductance optimization function is:

[0106]

[0107] The constraints of the conductivity sensor objective function are that the distance between windings, the distance between winding layers, and the distance from the winding end to the magnetic core meet the insulation requirements; the ratio of the winding height to the core window height is within a certain range; and the conductivity sensor magnetic core is saturated.

[0108] The distributed system optimizes different objective functions through multiple reinforcement learning agents. Each agent shares the optimal solution of parameters and the lowest total loss to speed up the training of reinforcement learning and quickly make the best strategy. Distributed reinforcement learning obtains the optimal solution set of each parameter, and determines the optimal design parameters of the conductivity sensor by calculating the total loss to determine whether it reaches the minimum. Based on the optimal design parameters, a high-performance conductivity sensor is designed.

Claims

1. A conductivity sensor parameter optimization method based on distributed reinforcement learning, characterized in that: The following steps are involved: Step 1: Establish the total loss model of the conductivity sensor; Step 2: Set the initial parameters of distributed reinforcement learning; Step 3: Determine the objective function of distributed reinforcement learning; Step 4: Calculate the reward value of distributed reinforcement learning, and output the corresponding reinforcement learning state and strategy when the maximum reward value is reached; Step 5: Calculate the reward values ​​of the three distributed reinforcement learnings and obtain the states and strategies of the three distributed reinforcement learnings; If the three distributed reinforcement learnings do not obtain the maximum reward value, repeat step 4; If the distributed reinforcement learning obtains the maximum reward value, execute step 6; Step 6: The agents use a consensus protocol to exchange optimal parameter solutions and minimum total loss to achieve distributed training of multiple agents. Step 7: Get the optimal solution set of each parameter; Step 8: Calculate the core loss and winding loss of the conductivity sensor; Step 9: Obtain the optimal design parameters of the conductivity sensor.

2. The method for optimizing conductivity sensor parameters based on distributed reinforcement learning according to claim 1, characterized in that: The specific method for establishing the total loss model of the conductivity sensor is: The total loss of the conductivity sensor includes core loss and winding loss. The total core loss under non-sinusoidal excitation is: Among them, P core is the core loss under non-sinusoidal excitation; T is the magnetization period; k i is a coefficient related to material properties; is the derivative of magnetic flux density with respect to time; ΔB is the change in magnetic flux density; dt is the time increment; α is the constant obtained by fitting the core loss test data under sinusoidal excitation, reflecting the change of core loss with frequency; β is the exponent of magnetic flux density, reflecting the change of core loss with magnetic flux density; The total winding loss under non-sinusoidal excitation is: Among them, P windings is the total winding loss; is the square of the effective value of the nth harmonic current; N is the number of current harmonics.

3. The conductivity sensor parameter optimization method based on distributed reinforcement learning according to claim 1, characterized in that: The method for determining the objective function of distributed reinforcement learning is: Taking the core loss, winding loss and leakage inductance of the conductivity sensor as the optimization objective function, the total loss of the conductivity sensor is: f1(x)=min{P core +P windings } Among them, f1(x) is the objective function 1; min{} is the minimization function; Control conductivity sensor leakage inductance L δ is the set value L const , the conductivity sensor leakage inductance optimization function is: Where f2(x) is the objective function 2; L δ is the leakage inductance of the conductivity sensor.

4. The conductivity sensor parameter optimization method based on distributed reinforcement learning according to claim 1, characterized in that: The specific method of calculating the reward value of reinforcement learning and outputting the corresponding reinforcement learning state and strategy when the maximum reward value is output is: The reinforcement learning agent takes maximizing the cumulative reward as the training goal. The cumulative reward at time t can be expressed as: Among them, R t is the reward value obtained at time t; γ is the discount factor, γ∈[0,1]; it reflects that the importance of current feedback decreases over time; r t+1 represents the reward value obtained by the agent when performing an action at time t+1; r t+2 represents the reward value obtained by the agent when performing the action at time t+2; r t+3 represents the reward value obtained by the agent when performing the action at time t+3; r t+k+1 Represents the reward value obtained by the agent when performing an action at time t+k+1.

Images