DDPG-based switching affine system ring domain finite time control method and system

Abstract

The application discloses a switching affine system ring domain finite time control method and system based on DDPG. The method and system convert the continuous action decision result in reinforcement learning into an executable discrete switching signal by constructing a deep deterministic policy gradient learning framework, a target ring domain, a core ring, a segmented state reward, an instant reward and a reward mapping, so as to realize accurate control of a Buck-Boost converter. The application reduces the dependence on an accurate system model, can directly express a ring domain constraint control target, realizes effective combination of continuous action reinforcement learning and a discrete switching actuator, is favorable for reducing unnecessary frequent switching, and takes into account control performance and a safety margin.

Description

Technical Field

[0001] This invention belongs to the field of power electronic control and intelligent control technology, specifically relating to a ring-domain finite-time control method and system for Buck-Boost converters based on DDPG. Background Technology

[0002] Currently, common methods for controlling switched systems or switched affine systems include switching stability analysis methods based on common Lyapunov functions or multiple Lyapunov functions, switching control methods based on dwell time constraints, model predictive control methods, and methods based on optimal control or dynamic programming.

[0003] The above methods can address the stability control issues of some switching systems to a certain extent, but they still have the following shortcomings in practical applications: 1) It relies heavily on accurate models.

[0004] Many existing methods require accurate system matrices, load parameters, input power supply parameters, and external disturbance information to complete control law design and stability analysis. However, in practical power electronic devices, device parameters drift with temperature, aging, and operating point changes, and the load may also have significant uncertainties, making accurate modeling quite difficult.

[0005] 2) The design of the target for the ring domain constraint is not direct enough.

[0006] Traditional control methods often aim to stabilize the equilibrium point or converge the tracking error. However, for the goal of "keeping the state within a certain loop domain," additional constraints or complex optimization processes are often required. Especially when simultaneously ensuring upper and lower boundary constraints within a finite time interval, controller design and feasibility analysis become quite complex.

[0007] 3) Switching strategies can easily become conservative or complex.

[0008] Existing handover control strategies typically require pre-constructing analytical handover surfaces, dwell time rules, or mode selection logic. When the system dimension is high, the operating state is complex, or the target region has a special shape, the difficulty of manually designing handover strategies increases significantly, and conservatism is easily introduced.

[0009] 4) It is difficult to balance control performance and switching frequency constraints.

[0010] For power electronic systems, frequent switching increases device losses, electromagnetic interference, and the difficulty of control implementation. While some methods can achieve stable control, they do not fully consider the number of switching operations, the rate of change of action, or the smoothness of switching, which can easily lead to high-frequency chattering in practice and even cause engineering risks similar to the Zeno phenomenon.

[0011] 5) Insufficient adaptability to unknown environments and data-driven scenarios.

[0012] With the increasing demand for complex power electronic systems and intelligent control, traditional analytical model-based methods have significant limitations in handling unknown models, online adaptation, and self-learning, making it difficult to meet the application requirements of data-driven control scenarios.

[0013] The development of reinforcement learning methods and their problems in switching control: Reinforcement learning is a data-driven approach that uses an agent to interact with its environment and iteratively learn control strategies based on reward signals. In recent years, with the development of deep neural networks, deep reinforcement learning has gained widespread attention in fields such as robot control, autonomous driving, energy management, and industrial optimization. For dynamic systems with unknown or difficult-to-model models, reinforcement learning methods can directly learn control laws through state transition samples, thereby reducing reliance on analytical models of the system.

[0014] Among numerous reinforcement learning algorithms, the Deep Deterministic Policy Gradient (DDPG) algorithm is an Actor-Critic type method suitable for continuous action spaces. This algorithm outputs continuous control actions through an Actor network, evaluates the value function of state-action pairs through a Critic network, and improves training stability by utilizing experience replay and a target network mechanism. Therefore, DDPG has good application potential in continuous control problems.

[0015] However, when applying continuous action reinforcement learning methods such as DDPG directly to switched affine systems, especially power electronic switching systems, the following key issues still exist: 1) The action space and the execution signal form do not match.

[0016] DDPG outputs are typically continuous values, while the execution inputs of systems like Buck-Boost converters are usually discrete switching modes or binary switching signals. Continuous actions cannot directly drive actual switching devices. Therefore, how to smoothly and effectively map continuous actions to discrete switching signals is a problem that must be solved when reinforcement learning is applied to switching control.

[0017] 2) The design of the reward function directly affects the expression of the control objective.

[0018] If the reward function only adopts the form of general tracking error or state penalty, it is difficult to accurately express the control requirement of "staying within the ring domain," let alone reflect the engineering preference of "staying away from the boundary and prioritizing the core area." Therefore, it is necessary to design a reward mechanism that is tightly coupled with the ring domain objective in order to guide the learning algorithm to obtain a control strategy that conforms to safety constraints.

[0019] 3) Frequent switching may occur during training.

[0020] If rewards are optimized solely based on immediate states, reinforcement learning strategies may tend to switch rapidly to obtain higher returns, but such strategies are impractical for engineering implementation in real-world circuits. Therefore, constraints on the rate of change, switching frequency, or action smoothness need to be introduced into the reward design or action generation mechanism.

[0021] 4) There is a lack of a unified implementation scheme for ring domain constraint control.

[0022] Existing reinforcement learning research mainly focuses on trajectory tracking, equilibrium point stability, or general optimal control. However, there is still a relative lack of complete technical solutions for finite-time control of switching affine systems in the ring domain, especially those that take into account discrete switching implementation, reward design, and engineering feasibility.

[0023] To more clearly illustrate the mismatch between "continuous action output" and "discrete switch execution" in the existing scheme, Figure 1 A schematic flowchart is provided. As shown in the diagram, the reinforcement learning controller outputs continuous action quantities, while the actual controlled object requires binary or finite mode switching signals. There is usually a lack of an effective interface mechanism between the two. Furthermore, state constraint objectives, switching smoothness requirements, and system security requirements must also be reflected simultaneously in the reward design and control implementation.

[0024] In summary, existing technologies still suffer from problems such as strong model dependence, indirect expression of control objectives, difficulty in switching implementation, and insufficient engineering feasibility when dealing with the loop-domain constrained control problem of switched affine systems, especially power electronic converters. Particularly in scenarios where the system model is unknown or incompletely known, how to automatically learn the switching control strategy using data-driven methods while simultaneously meeting the following requirements remains a pressing technical problem to be solved: 1) Keep the system state within the target ring domain within a given finite time interval.

[0025] 2) Keep the state as far away from the boundary of the ring domain as possible to increase the safety margin.

[0026] 3) Convert the continuous control output of reinforcement learning into discrete switching signals that can actually be executed.

[0027] 4) Reduce drastic changes in control actions and unnecessary frequent switching.

[0028] 5) Improve the method's adaptability to model uncertainty and complex environments.

[0029] Definitions: Mapping Module: The mapping module is used to convert the continuous control quantity output by the reinforcement learning network into a discrete switching signal that can be executed by the converter. Combined with the internal accumulated variable and threshold decision mechanism, it reduces switching jitter and improves the stability and engineering feasibility of the control strategy. Summary of the Invention

[0030] To address the aforementioned problems, this invention discloses a finite-time control method and system for Buck-Boost converters in the ring domain based on DDPG.

[0031] To achieve the above objectives, the technical solution of the present invention is as follows: A finite-time control method for a switched affine system based on DDPG in the ring domain includes the following steps: S1. Obtain the system state of the switching affine system. Constructing a quadratic form index ; S2, Set the target ring area and core ring area ; S3, according to , and Constructing a piecewise state reward function ; S4. Segmented State Reward Function Construct an instant reward that includes motion smoothing. ; S5. Construct a deep deterministic policy gradient learning framework, which includes an Actor network, a Critic network, a target Actor network, a target Critic network, an experience replay buffer, a reward calculation module, and a switching mapping module. S6. Input the current state of the affine system into the Actor network in real time. The Actor network outputs continuous motion control quantities. The switching mapping module maps continuous motion control quantities into discrete switching signals. Discrete switching signal As the control signal for switching the affine system to the next moment, the state of the next moment is obtained. ; S7 and Critic networks are calculated based on instantaneous reward functions. The reward value is used to construct state transition samples and store them in the experience replay buffer. S8. Update the parameters of the Actor network and Critic network: Repeat steps S6 and S7 until the number of state transition samples in the experience replay buffer reaches a preset number. Then, randomly select a small batch of samples from the experience replay buffer to construct a training sample set. Update the deep deterministic policy gradient learning framework based on the training sample set to obtain the trained deep deterministic policy gradient learning framework. S9, Switching the Affine System State at Current Moment Input a pre-trained deep deterministic policy gradient learning framework Actor network, and the Actor network outputs... , Mapped to discrete switching signals As the control input signal for switching affine systems.

[0032] Further improvements, the specific steps of step S1 are as follows: Establish a state model for the switching affine system: in, Let be the system state vector. Let n represent an n-dimensional real vector space, where n represents the dimension of the system state. for The first derivative with respect to time t; For discrete switching mode, For a finite set of patterns, for System matrix under the model, for Affine terms in the pattern; Constructing a state quadratic index based on the state model of a switched affine system : in, It is a symmetric positive definite matrix; Indicates time The system state vector, T represents the matrix transpose; Select Represents the identity matrix. Right now Indicates inductor current. This represents the capacitor voltage.

[0033] A further improvement is that the switching affine system is a Buck-Boost converter; in, Indicates time The inductor current, Indicates time The capacitor voltage; : Where 1 indicates that the affine system is in mode one, and the affine system is in the energy charging process; 2 indicates that the affine system is in mode two, and the affine system is in the energy release process. The subsystem model is as follows: as well as in, Indicates the switching of the load resistance of the affine system; This indicates the capacitor of the switching affine system. This indicates switching the input power supply of the affine system. This indicates the switching of the inductance in the affine system.

[0034] Further improvements include step S2, which involves the target ring domain. as follows: in, The lower boundary of the ring domain, It is the upper boundary of the ring domain, and satisfies Construct a quadratic state index for a switched affine system; Core ring area as follows: As the lower boundary of the core ring domain, The upper boundary of the core ring domain.

[0035] Further improvements, step S3 is as follows: For state quadratic index Construct a piecewise state reward function, let This represents the performance index value of the quadratic form. Represents the system state vector. Represents the system state vector transpose; State reward function The construction is as follows: in, All of these are adjustable parameters; And it satisfies: ; In step S4, the immediate reward function as follows: in, and These are the state reward weights and the action change rate penalty weights, respectively. The continuous motion control quantity output by the Actor network; in, For the first The continuous motion control quantity output by the Actor network at each sampling time. The sampling period.

[0036] In a further improvement, in S5, the Actor network is used to receive the current state. And output continuous motion control quantity , After sampling, it is discretized into the first... Actions at each sampling time ; Represents the parameters of the Actor network. This represents the Actor's policy function. Represents a deterministic policy mapping. Indicates the current time The state vector; Critic networks are used to evaluate the value function of state-action pairs. ,in, Indicates the parameters of the Critic network; The target Actor network and the target Critic network are delayed-update copies of the online Actor network and the online Critic network, respectively. Their input-output structures are consistent with the corresponding Actor network and Critic network. They are used to generate target actions and estimate target values ​​to improve training stability. An experience replay pool is used to store historical state transition samples to break the temporal correlation of samples. The reward calculation module is used to generate instant rewards based on step S4. ; The switching mapping module is used to convert discrete actions Converted to discrete switching signal .

[0037] A further improvement is made: the historical state transition samples stored in the experience replay pool are written as follows: ; in, Indicates the first Instant reward at each sampling moment; Critic network target value Represented as: in, As a discount factor, Represents the target Actor network. Represents the target Critic network; Loss function of Critic network Represented as: Indicates the parameters of the Critic network. For small batch sample size The update direction of the Actor network is given by the deterministic policy gradient, expressed as: in, This indicates information about the Actor network parameters. gradient, Describe the objective function. Indicates about actions gradient, Indicates the state of the Actor network. The output is a continuous motion control quantity; The target Actor network and the target Critic network use a soft update method for parameter updates: Indicates the target Critic network parameters. Represents the target Critic network. Indicates the parameters of the online Critic network. Indicates an online Critic network. Represents the network parameters of the target Actor. Represents the target Actor network. Indicates the parameters of the online Actor network. Indicates an online Actor network. This is the soft update coefficient.

[0038] In a further improvement, in step S6, the mapping module is switched via... The modulation mechanism maps continuous motion control quantities to discrete switching signals. : Set internal cumulative variable And update as follows: in, Indicates the first Internal cumulative variables at each time point Indicates the first The output at each time step, where k represents the discrete time step index; According to cumulative variables The sign of the variable or the comparison result with a preset threshold are used to generate a discrete control signal. and according to Confirm switching mode: .

[0039] Further improvements include the following steps for training a deep deterministic policy gradient learning framework: Step A1: Initialize parameters: Initialize the parameters of the Actor network, Critic network, target Actor network, and target Critic network; initialize the experience replay pool; set the discount factor, learning rate, sampling period, number of training rounds, reward parameter, and ring boundary parameter; Step A2: Set the initial state: At the beginning of each training round, the system is initialized to its initial state. Set it within the predetermined initial area and bring the environment into its initial operating state; Step A3: Generate continuous actions: Current state Inputting into an Actor network yields continuous action outputs. ; Step A4: Generate discrete switching modes: Continuous action enter The mapping module generates the corresponding discrete switching signals. and according to Determine the operating mode of the Buck-Boost converter at the current moment; Step A5: Execute control and sample: Discrete switching signal Acting on the controlled object, causing the Buck-Boost converter to change from state Run to the next time step state Calculate instant rewards and samples Store in the experience replay pool; Step A6: Network parameter update: Once the number of samples in the experience replay pool reaches the preset requirement, a small batch of samples is randomly selected from it. First, the Critic network parameters are updated, then the Actor network parameters are updated according to the Critic network output, and soft updates are performed on the target Actor network and the target Critic network. Step A7: Determine if training is complete: If the preset number of training rounds is reached, or the cumulative reward converges, or the changes in network parameters meet the termination threshold, then training ends; otherwise, return to step A3 to continue iterating. Step A8: Deploy control policies: After training is complete, the Actor network is retained as the final controller.

[0040] A DDPG-based switched affine system ring domain finite-time control system is provided, wherein the DDPG-based switched affine system ring domain finite-time control system is used to run the above-mentioned DDPG-based switched affine system ring domain finite-time control method.

[0041] The advantages of this invention are as follows: 1) It reduces the reliance on accurate system models.

[0042] This invention employs a deep reinforcement learning framework, which learns control strategies through interactive sampling with the environment. It does not rely on a precise analytical model of the controlled object, and is therefore more suitable for real-world scenarios where parameters are uncertain, models are difficult to establish accurately, or working conditions vary greatly.

[0043] 2) It can directly express the control objectives of the cyclic domain constraint.

[0044] This invention constructs a piecewise reward function corresponding to the quadratic state index, directly embedding the control requirements of "keeping the state within the target domain" and "staying as close as possible to the core domain" into the learning objective, enabling the reinforcement learning strategy to be optimized around the domain safety constraints.

[0045] 3) It achieves an effective combination of continuous motion reinforcement learning and discrete switching actuators.

[0046] This invention establishes a mapping mechanism from continuous actions to discrete switching modes, preferably employing... The modulation mechanism converts the continuous actions output by the Actor network into discrete switching signals that can actually be executed, thereby solving the problem that continuous action algorithms are difficult to directly drive switching devices.

[0047] 4) It helps reduce unnecessary and frequent switching.

[0048] This invention introduces a penalty term for the rate of change of action into the instant reward, which suppresses drastic changes in the control action, thereby helping to reduce switching frequency, reduce switching losses, mitigate electromagnetic interference, and improve the stability of engineering implementation.

[0049] 5) Applicable to practical switching systems such as power electronics.

[0050] This invention can be applied to Buck-Boost converters and other DC-DC converters, switching power supplies, energy conversion devices, and industrial objects that can be abstracted as switching affine systems, and has strong engineering promotion value.

[0051] 6) Balancing control performance with safety margin.

[0052] By setting target and core ring domains, this invention not only keeps the system state within acceptable limits but also guides the state away from boundary regions, thereby improving the safety and robustness of system operation. Attached Figure Description

[0053] Figure 1 This is a Buck-Boost converter structure.

[0054] Figure 2 This is a schematic diagram of the process of the present invention.

[0055] Figure 3 A schematic diagram of the target ring domain and the core ring domain design.

[0056] Figure 4 This is a schematic diagram of the segmented state reward function.

[0057] Figure 5 This is a schematic diagram of the DDPG learning framework.

[0058] Figure 6 for A schematic diagram of the mapping mechanism.

[0059] Figure 7 The system state phase plane trajectory diagram for different stages.

[0060] Figure 8 This is a signal timing diagram.

[0061] Figure 9 This is a graph showing the cumulative reward changes. Detailed Implementation

[0062] Example 1 like Figure 1As shown, this embodiment uses a DC-DC Buck-Boost converter as the controlled object to illustrate the proposed finite-time control method for switching affine systems based on deep deterministic policy gradients. This embodiment is applicable when input voltage, load, or other parameters change, and learns a switching control strategy that meets the requirements of the loop domain constraints through a data-driven approach.

[0063] 1) Controlled object and circuit structure The Buck-Boost converter structure used in this embodiment can be referred to Figure 1. This circuit includes at least an input power supply. Switching devices and ,inductance ,capacitance and load resistance Among them, switching devices and By switching the system on and off in a complementary or controlled manner, the system can switch between different operating modes to achieve energy storage and release.

[0064] In this embodiment, inductor current and capacitor voltage are selected as system state variables, and the state vector is defined as follows: (16) Because the circuit topology differs depending on the switching device's conduction state, the system exhibits different continuous dynamic equations in different modes. By unifying the dynamic equations across modes, the following switching affine model can be obtained: (17) in, Indicates the current switching mode.

[0065] In one specific implementation, when the system is in mode 1, it represents the energy charging process; when the system is in mode 2, it represents the energy releasing process. Accordingly, the following subsystem model can be obtained: (18) as well as (19) The specific switch conduction relationship corresponding to Mode 1 and Mode 2 can be set according to the actual hardware connection method. This invention does not impose strict limitations on this, as long as a switching affine dynamic relationship under a finite set of modes can be formed.

[0066] 2) Setting the control objectives for the ring domain In this embodiment, instead of using the traditional asymptotic convergence of the equilibrium point as the sole control objective, the objective is to maintain the state within a predetermined loop domain. To this end, a quadratic form index of the system state is constructed. (20) In a specific implementation, one can select (twenty one) Right now (twenty two) Furthermore, the target ring domain is set as (twenty three) in, The lower boundary of the ring domain, It is the upper boundary of the ring domain, and satisfies .

[0067] To keep the system state as far away from the boundary as possible within the target ring domain, a core ring domain can be further defined within the target ring domain. (twenty four) in, (25) In this embodiment, the target ring domain Used to define the overall security zone where state-permitted activities are allowed, core ring domain This is used to define the preferred maintenance region. Through subsequent reward design, the learned strategy can be guided to prioritize maintaining the system state near the core ring region, thereby improving the safety margin of system operation.

[0068] For a more intuitive representation of the target ring domain and the core ring domain, please refer to [reference needed]. Figure 3 In the diagram, the outer solid line represents the boundary of the target annular domain, and the inner dashed line represents the boundary of the core annular domain. The state trajectory is guided to run within the annular domain, rather than simply converging to the origin.

[0069] 3) Construction of the piecewise reward function To ensure that the reinforcement learning algorithm accurately reflects the control objective in the loop domain, this embodiment focuses on the state index. Construct the piecewise state reward function. Let (26) In a preferred implementation, the state reward function Constructed as (27) in, This is an adjustable parameter.

[0070] The above reward function has the following effect: 1) When When this occurs, it indicates that the system state falls outside the lower boundary of the target ring domain, and a negative reward is given to encourage the state to return to the allowed region. 2) When When the state is within the target ring domain but close to the lower boundary, the reward increases as the state moves closer to the core ring domain; 3) When When the state is within the core ring domain, a relatively high constant reward is given; 4) When When the state is within the target ring domain but close to the upper boundary, the reward increases as the state reverts to the core ring domain; 5) When If the system state exceeds the upper boundary of the target ring domain, a negative reward is given to cause the state to fall back to the allowed region.

[0071] To avoid significant abrupt changes at the connection points of reward segments, parameters can be optimally set to meet certain conditions. (28) To further reduce unnecessary switching, an immediate reward function is constructed based on the state reward. (29) in, and These are the state reward weights and the action change rate penalty weights, respectively. The continuous actions output by the Actor network.

[0072] In discrete sampling implementation It can be expressed in a difference approximation form, for example, it can be written as (30) in, For the first The continuous action output at each sampling time point The sampling period.

[0073] Figure 4 The piecewise structure of the state reward function is illustrated schematically. Through this reward structure, the system state is not only required to remain within the target domain. Moreover, they are preferably guided into and remain in the core ring domain. Inside.

[0074] 4) Construction of the DDPG learning framework This embodiment employs a deep deterministic policy gradient algorithm to learn the switching control strategy. DDPG is an Actor-Critic reinforcement learning algorithm suitable for continuous action spaces, and its overall structure is as follows: Figure 5 As shown.

[0075] Specifically, the learning framework includes: 1) Actor network, used to receive the current state. And output continuous actions ; 2) Critic network, used to evaluate the value function of state-action pairs. ; 3) Target Actor network and Target Critic network, used to generate target values ​​and improve training stability; 4) Experience replay pool, used to store historical state transition samples to break the temporal correlation of samples; 5) Reward calculation module, used to generate instant rewards according to formula (29); 6) Switching mapping module, used to switch continuous actions Converted to discrete switching signal .

[0076] In one specific implementation, the samples stored in the experience replay pool can be written as (31) in, Indicates the first Instant reward for each sampling moment.

[0077] The objective value of the Critic network can be expressed as: (32) in, As a discount factor, Represents the target Actor network. This represents the target Critic network.

[0078] The loss function of the Critic network can be expressed as: (33) in, This refers to the size of the small batch of samples.

[0079] The update direction of the Actor network is given by the deterministic policy gradient, which can be expressed as: (34) The target network uses a soft update method to update parameters, which can be represented as follows: (35) (36) in, This is the soft update coefficient.

[0080] 5) Mapping from continuous operation to discrete switching signal Since the Actor network outputs continuous actions, while the Buck-Boost converter's execution control input is a discrete switching mode, a mapping mechanism from continuous actions to discrete switching signals needs to be designed.

[0081] In this embodiment, it is preferred to use This mapping is achieved through a modulation mechanism. Its purpose is to output a binary or finite-mode switching signal suitable for switching devices while preserving continuous operation trend information. A schematic diagram of this mapping mechanism is shown below. Figure 6 As shown.

[0082] In one exemplary implementation, an internal cumulative variable can be set. And update as follows: (37) in, This indicates the quantized output from the previous time step.

[0083] Subsequently, based on cumulative variables The sign of the input signal or the result of comparison with a preset threshold is used to output a quantized signal. Then, a switching mode is generated based on the quantized signal. For example, in a two-mode switching scenario, the following rules can be used: (38) It should be noted that the above The mapping method is only one preferred implementation, and the present invention is not limited to it. For multi-mode switching systems, multi-threshold quantization, segmented quantization, probability quantization, or other continuous action discretization methods can also be used, as long as the conversion from continuous actions to finite switching modes can be achieved.

[0084] By introducing the above mapping mechanism, the Actor network does not need to directly output discrete switch states, but instead outputs continuous action variables that are more conducive to neural network training; then, it is converted into actual executable switching signals through the mapping module, thus balancing learning efficiency and engineering feasibility.

[0085] 6) Control strategy training process Based on the above system model, reward design, learning framework, and action mapping mechanism, the control policy training process in this embodiment can be executed according to the following steps: Step A1: Initialize parameters.

[0086] Initialize the parameters of the Actor network, Critic network, target Actor network, and target Critic network; initialize the experience replay pool; set the discount factor, learning rate, sampling period, number of training rounds, reward parameters, and ring boundary parameters.

[0087] Step A2: Set the initial state.

[0088] At the beginning of each training round, the system is initialized to its initial state. Set it within the predetermined initial area and bring the environment into its initial operating state.

[0089] Step A3: Generate continuous actions.

[0090] Current state Inputting into an Actor network yields continuous action outputs. During the training phase, exploratory noise can be superimposed on the action to enhance the strategy exploration capability.

[0091] Step A4: Generate discrete switching modes.

[0092] Continuous action enter The mapping module generates the corresponding discrete switching signals. This determines the operating mode of the Buck-Boost converter at the current moment.

[0093] Step A5: Perform control and sample.

[0094] Switch mode Acting on the controlled object, causing the system to change from state Run to the next time step state Calculate instant rewards and samples Store in the experience replay pool.

[0095] Step A6: Network parameter update.

[0096] Once the number of samples in the experience replay pool reaches the preset requirement, a small batch of samples is randomly selected from it. First, the Critic network parameters are updated, then the Actor network parameters are updated according to the Critic network output, and the target network is softly updated.

[0097] Step A7: Determine whether the training is complete.

[0098] If the preset number of training rounds is reached, or the cumulative reward converges, or the changes in network parameters meet the termination threshold, then training ends; otherwise, return to step A3 to continue iterating.

[0099] Step A8: Deploy control policies.

[0100] After training, the Actor network is retained as the final controller. During the online operation phase, the real-time state is input into the trained Actor network, and its output is converted into a switching signal via a mapping module to control the Buck-Boost converter to operate within the target loop domain.

[0101] A set of exemplary parameter settings: In a set of exemplary experiments in this embodiment, the Buck-Boost converter parameters can be set as shown in Figure 1. It should be understood that these parameter settings are only used to illustrate the feasibility of the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0102] Table 1 System Parameters Furthermore, some hyperparameters in DDPG training can be set as shown in Figure 2.

[0103] Table 2 DDPG Training Parameters The parameters described above are merely illustrative settings. In other implementations, the number of network layers, number of neurons, number of training rounds, sampling period, ring boundary, and reward weight can be adjusted according to the specific hardware platform, control objectives, response speed requirements, learning stability requirements, and system scale.

[0104] Control effect description: After training using the above method, the resulting Actor network can output continuous actions that meet the requirements of loop domain control, and is converted into actual switching signals by the mapping module to drive the Buck-Boost converter to switch between different operating modes.

[0105] Under optimal conditions, as the number of training rounds increases, the system state trajectory gradually converges from initial large fluctuations to a restricted trajectory within the target ring domain. Furthermore, the state trajectory is preferably maintained near the core ring domain, thereby keeping the system operation away from the target ring domain boundary and improving the safety margin.

[0106] Meanwhile, because a penalty term for the rate of change of action is introduced into the immediate reward, the trained switching policy will not blindly pursue frequent switching to improve instantaneous rewards, but will take into account the smoothness of action while satisfying the cyclic domain constraint objective. Therefore, the resulting switching signal usually exhibits a control sequence with a certain structure, limited switching frequency, and is more suitable for actual hardware execution.

[0107] In some implementations, the control effect of the method of the present invention can be further illustrated by combining simulation results. The simulation results may include state phase plane trajectory diagrams, switching signal timing diagrams, and training cumulative reward change curves, etc., to demonstrate the policy convergence characteristics of the method of the present invention during the training process and the closed-loop control effect after training is completed.

[0108] Figure 7The figure illustrates the phase plane trajectory of the system state during training. For example, the figure shows the changes in the state trajectory after 500, 1000, and 1500 rounds of training. It can be seen that in the early stages of training, such as around 500 rounds, the learned control policy has not yet fully converged, and the system state trajectory is still distributed over a relatively large area around the target region. As training continues, for example, around 1000 rounds, the state trajectory begins to shrink significantly and gradually approaches the predetermined loop region. When training reaches approximately 1500 rounds, the closed-loop state trajectory is essentially confined to the target loop region. (39) The results show that the deep reinforcement learning-based switching control method proposed in this invention can gradually form an effective control strategy through training and constrain the system state within a preset loop domain.

[0109] Figure 8 The switching signal applied to the system is shown. The changes over time are shown in the figure. As can be seen from the figure, the trained switching control strategy produces a switching sequence with a certain regularity, rather than completely random switching. Furthermore, the frequency of the switching signal remains within a bounded range, indicating that the method of the present invention, while achieving the ring domain control objective, helps to suppress excessively frequent switching behavior, thereby reducing switching losses, reducing electromagnetic interference, and avoiding the risk of unlimited switching in practical applications of switching systems. When the system state enters and remains within the target ring domain, the switching action tends to be relatively smooth, indicating that the method of the present invention can reduce unnecessary switching operations after satisfying the constraint objective, thus contributing to the steady-state operation of the system.

[0110] Figure 9 The cumulative reward curve during the training process is shown. In one exemplary embodiment, the training process can last for 300 rounds. Initially, the cumulative reward is relatively low because the policy is in a random exploration phase; as the number of training rounds increases, the cumulative reward gradually increases and eventually stabilizes. This change indicates that the Actor network and Critic network gradually learn effective switching control policies during interaction with the environment, and the training process exhibits good convergence. The stabilization of the cumulative reward also demonstrates that the training termination condition proposed in this invention is feasible, and the obtained control policy can stably achieve the predetermined domain control objective.

[0111] comprehensive Figures 7 to 9The results show that the method proposed in this invention can gradually improve control performance during training and form an effective control strategy suitable for switched affine systems after training is completed. This strategy not only constrains the system state within the target loop domain but also keeps the switching behavior within a reasonable range, thus balancing control effectiveness and engineering feasibility. Therefore, the method proposed in this invention can achieve loop domain constrained control for switched affine systems such as Buck-Boost converters under unknown or incompletely known model conditions.

[0112] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. Other modifications can be easily made by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A finite-time control method for a switched affine system in the ring domain based on DDPG, characterized in that, Includes the following steps: S1. Obtain the system state of the switching affine system. Constructing a quadratic form index ; S2, Set the target ring area and core ring area ; S3, according to , and Constructing a piecewise state reward function ; S4. Segmented State Reward Function Construct an instant reward that includes motion smoothing. ; S5. Construct a deep deterministic policy gradient learning framework, which includes an Actor network, a Critic network, a target Actor network, a target Critic network, an experience replay buffer, a reward calculation module, and a switching mapping module. S6. Input the current state of the affine system into the Actor network in real time. The Actor network outputs continuous motion control quantities. The switching mapping module maps continuous motion control quantities into discrete switching signals. Discrete switching signal As the control signal for switching the affine system to the next moment, the state of the next moment is obtained. ; S7 and Critic networks are calculated based on instantaneous reward functions. The reward value is used to construct state transition samples and store them in the experience replay buffer. S8. Update the parameters of the Actor network and Critic network: Repeat steps S6 and S7 until the number of state transition samples in the experience replay buffer reaches the preset number. Then, randomly select a small batch of samples from the experience replay buffer to construct a training sample set. The deep deterministic policy gradient learning framework is updated based on the training sample set to obtain the trained deep deterministic policy gradient learning framework. S9, Switching the Affine System State at Current Moment Input a pre-trained deep deterministic policy gradient learning framework Actor network, and the Actor network outputs... , Mapped to discrete switching signals As the control input signal for switching affine systems.

2. The finite-time control method for a switched affine system based on DDPG as described in claim 1, characterized in that, The specific steps of step S1 are as follows: Establish the state model of the switching affine system: in, Let be the system state vector. Let n represent an n-dimensional real vector space, where n represents the dimension of the system state. for The first derivative with respect to time t; For discrete switching mode, For a finite set of patterns, for System matrix under the model, for Affine terms in the pattern; Constructing a state quadratic index based on the state model of a switched affine system : in, It is a symmetric positive definite matrix; Indicates time The system state vector, T represents the matrix transpose; Select Represents the identity matrix. Right now Indicates inductor current. This represents the capacitor voltage.

3. The finite-time control method for Buck-Boost converter in the loop domain based on DDPG as described in claim 2, characterized in that, The switching affine system is a Buck-Boost converter; in, Indicates time The inductor current, Indicates time The capacitor voltage; ： Where 1 indicates that the affine system is in mode one, and the affine system is in the energy charging process; 2 indicates that the affine system is in mode two, and the affine system is in the energy release process. The subsystem model is as follows: as well as in, Indicates the switching of the load resistance of the affine system; This indicates the capacitor of the switching affine system. This indicates switching the input power supply of the affine system. This indicates the switching of the inductance in the affine system.

4. The finite-time control method for a switched affine system based on DDPG as described in claim 2, characterized in that, In step S2, the target ring domain as follows: in, The lower boundary of the ring domain, It is the upper boundary of the ring domain, and satisfies Construct a quadratic state index for a switched affine system; Core ring area as follows: As the lower boundary of the core ring domain, It is the upper boundary of the core ring domain.

5. The finite-time control method for a switched affine system based on DDPG as described in claim 4, characterized in that, Step S3 is as follows: For state quadratic index Construct a piecewise state reward function, let This represents the performance index value of the quadratic form. Represents the system state vector. Represents the system state vector Transpose of; State reward function The construction is as follows: in, All of these are adjustable parameters; And it satisfies: ； In step S4, the immediate reward function as follows: in, and These are the state reward weights and the action change rate penalty weights, respectively. The continuous motion control quantity output by the Actor network; in, For the first The continuous motion control quantity output by the Actor network at each sampling time. The sampling period.

6. The finite-time control method for a switched affine system based on DDPG as described in claim 1, characterized in that, In S5, the Actor network is used to receive the current state. And output continuous motion control quantity , After sampling, it is discretized into the first... Actions at each sampling time ; Represents the parameters of the Actor network. This represents the Actor's policy function. Represents a deterministic policy mapping. Indicates the current time The state vector; Critic networks are used to evaluate the value function of state-action pairs. ,in, Indicates the parameters of the Critic network; The target Actor network and the target Critic network are delayed-update copies of the online Actor network and the online Critic network, respectively. Their input-output structures are consistent with the corresponding Actor network and Critic network. They are used to generate target actions and estimate target values ​​to improve training stability. An experience replay pool is used to store historical state transition samples to break the temporal correlation of samples. The reward calculation module is used to generate instant rewards based on step S4. ; The switching mapping module is used to convert discrete actions Converted to discrete switching signal .

7. The finite-time control method for a switched affine system based on DDPG as described in claim 6, characterized in that, The historical state transition samples stored in the experience replay pool are written as: ； in, Indicates the first Instant reward at each sampling moment; Critic network target value Represented as: in, As a discount factor, Represents the target Actor network. Represents the target Critic network; Loss function of Critic network Represented as: Indicates the parameters of the Critic network. For small batch sample size The update direction of the Actor network is given by the deterministic policy gradient, expressed as: in, This indicates information about the Actor network parameters. gradient, Describe the objective function. Indicates about actions gradient, Indicates the state of the Actor network. The output is a continuous motion control quantity; The target Actor network and the target Critic network use a soft update method for parameter updates: Indicates the target Critic network parameters. Represents the target Critic network. Indicates the parameters of the online Critic network. Indicates an online Critic network. Represents the network parameters of the target Actor. Represents the target Actor network. Indicates the parameters of the online Actor network. Indicates an online Actor network. This is the soft update coefficient.

8. The finite-time control method for a switched affine system based on DDPG as described in claim 1, characterized in that, In step S6, the mapping module is switched via... The modulation mechanism maps continuous motion control quantities to discrete switching signals. : Set internal cumulative variable And update as follows: in, Indicates the first Internal cumulative variables at each time point Indicates the first The output at each time step, where k represents the discrete time step index; According to cumulative variables The sign of the variable or the comparison result with a preset threshold are used to generate a discrete control signal. and according to Confirm switching mode: 。 9. The finite-time control method for a switched affine system based on DDPG as described in claim 1, characterized in that, The steps for training a deep deterministic policy gradient learning framework are as follows: Step A1: Initialize parameters: Initialize the parameters of the Actor network, Critic network, target Actor network, and target Critic network; initialize the experience replay pool; set the discount factor, learning rate, sampling period, number of training rounds, reward parameter, and ring boundary parameter; Step A2: Set the initial state: At the beginning of each training round, the system is initialized to its initial state. Set it within the predetermined initial area and bring the environment into its initial operating state; Step A3: Generate continuous actions: Current state Inputting into an Actor network yields continuous action outputs. ; Step A4: Generate discrete switching modes: Continuous action enter The mapping module generates the corresponding discrete switching signals. and according to Determine the operating mode of the Buck-Boost converter at the current moment; Step A5: Execute control and sample: Discrete switching signal Acting on the controlled object, causing the Buck-Boost converter to change from state Run to the next time step state Calculate instant rewards and samples Store in the experience replay pool; Step A6: Network parameter update: Once the number of samples in the experience replay pool reaches the preset requirement, a small batch of samples is randomly selected from it. First, the Critic network parameters are updated, then the Actor network parameters are updated according to the Critic network output, and soft updates are performed on the target Actor network and the target Critic network. Step A7: Determine if training is complete: Training ends when the preset number of training rounds is reached, the cumulative reward converges, or the changes in network parameters meet the termination threshold. Otherwise, return to step A3 and continue the iteration; Step A8: Deploy control policies: After training is complete, the Actor network is retained as the final controller.

10. A finite-time system in a ring domain based on a DDPG-based switched affine system, characterized in that, The DDPG-based switched affine system ring domain finite-time control system is used to run the DDPG-based switched affine system ring domain finite-time control method according to any one of claims 1-9.

Images