Satellite on-board PWM-MDP combined high-precision temperature control system and method thereof

Abstract

The present application belongs to the field of satellite thermal control technology in the field of aerospace, aiming at overcoming the shortcomings of the existing satellite thermal control mode, solving the problem that it cannot meet the high-precision and high-stability temperature control requirements of the next generation of satellites, and putting forward a kind of on-board PWM-MDP combined high-precision temperature control system and its method. The system comprises a physical model, a control decision, a state management, an experience management and a learning training module; the physical model module simulates the temperature of the controlled object and outputs data, the control decision module receives the data and outputs instructions combined with the four-stage progressive control framework, the state management module collects the temperature data feedback after control and processes, the experience management module stores the control experience, and the learning training module trains the MDP control unit offline. The system is operated in four stages, realizes the smooth transition from improved PWM-Bang-Bang control to MDP high-precision control, stabilizes the temperature in a higher precision range, and makes the application suitable for satellite thermal control requirements.

Description

Technical Field

[0001] This invention belongs to the field of satellite thermal control technology in the aerospace industry, specifically relating to an onboard PWM-MDP combined high-precision temperature control system and method. Background Technology

[0002] The thermal control subsystem is one of the core functional modules of the satellite platform. Its core task is to maintain the operation of various onboard devices within their permissible operating temperature ranges. High-precision measurement components, such as star sensors and gyroscopes, are particularly sensitive to temperature changes—even minute temperature fluctuations can cause drift in their measurement outputs, directly weakening the stability and accuracy of satellite attitude determination. Therefore, ensuring high temperature stability for these devices during on-orbit operation and controlling temperature fluctuations within a minimal range is a crucial foundation for guaranteeing the overall satellite performance and lifespan.

[0003] In previous satellite missions, common active thermal control methods mainly included Bang-Bang control and PID control. While Bang-Bang control is simple in structure and easy to implement in engineering, its inherent switching characteristics easily lead to significant overshoot and oscillations, resulting in large fluctuations and insufficient stability in the temperature control curve. Traditional PID control, relying on precise mathematical models, often exhibits lag in response to dynamic stimuli such as transient external heat flux and changes in equipment power consumption encountered by the satellite in orbit, and its adaptive capability is limited, making it difficult to meet the high-precision and high-stability temperature control requirements of next-generation satellites. Therefore, it is necessary to explore more advanced and intelligent temperature control methods.

[0004] In summary, the Bang-Bang control used in existing satellite active thermal control is prone to overshoot and oscillation, resulting in poor temperature control stability; while traditional PID control relies on precise mathematical models, which lags in response to dynamic stimuli such as transient external heat flow and changes in equipment power consumption in orbit, and has limited adaptive capabilities. Both are difficult to meet the high-precision and high-stability temperature control requirements of next-generation satellites. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing satellite thermal control methods, which fail to meet the high-precision and high-stability temperature control requirements of next-generation satellites. It proposes an onboard PWM-MDP joint high-precision temperature control system and method. This invention achieves a smooth transition from traditional Bang-Bang control to Markov decision process control through a four-stage progressive control architecture. At the same time, it designs a discrete action space for the characteristics of PWM control and adopts a core region enhanced exploration mechanism to solve the problem of insufficient exploration.

[0006] To achieve the above objectives, the present invention provides the following technical solution: This invention proposes an on-board PWM-MDP combined high-precision temperature control system, which includes a physical model module, a control decision module, a state management module, an experience management module, and a learning and training module. The physical model module is used to simulate the dynamic temperature characteristics of the controlled object and output the current temperature data of the controlled object. The control decision module is used to receive the current temperature data output by the physical model module, and in accordance with the requirements of the four-stage progressive control framework, output corresponding control commands to control the temperature of the controlled object. The control decision module includes a traditional control unit and a Markov decision process control unit; the traditional control unit is used to execute an improved PWM-Bang-Bang controller to provide basic temperature control support for the system; the Markov decision process control unit is built based on the Q-learning algorithm to achieve high-precision temperature control, and the two units work together to complete four-stage control switching and temperature regulation. The state management module is used to collect the temperature data of the controlled object after it has been processed by the control decision module, calculate the temperature error, store the control actions output by the control decision module, and perform state discretization processing on the temperature-related data to form empirical data, which is then output to the empirical management module. The experience management module uses a circular buffer to store the experience data and supports experience playback; The learning and training module is used to perform offline training on the Markov decision process control unit. After the training is completed, the optimized control strategy is fed back to the Markov decision process control unit through the experience management module.

[0007] Furthermore, the four-stage progressive control framework described in this invention specifically includes: Phase 1: Data collection and basic control phase, starting at time t1. The traditional control unit runs an improved PWM-Bang-Bang controller to control the temperature of the controlled object within the range of ±ΔT1 of the target temperature T_target. At the same time, comprehensive control experience data is collected through an active exploration mechanism. The second stage is the offline training stage, which starts at time t2. Based on the control experience data collected in the first stage, the Markov decision process control unit is trained in the simulation environment. In this stage, the improved PWM-Bang-Bang controller is still used for the controlled object to control the temperature within the range of ±ΔT1 of the target temperature T_target. The third stage is the gradual switching stage, which starts at time t3. A time-varying hybrid control strategy is used to achieve a smooth transition from traditional control to Markov decision process control. The fourth stage is the pure Markov decision process control stage, which starts at time t4. The trained Markov decision process control unit takes over the control completely to achieve high-precision temperature control. The control objective is to keep the temperature within the range of T_target ±ΔT2, where ΔT2 < ΔT1.

[0008] Furthermore, the improved PWM-Bang-Bang controller of the present invention includes a core region active exploration mechanism, which specifically includes: When the system temperature is within the core control region, an exploratory action is performed with a preset probability p_explore, where the core region is defined as the region where |ΔT|≤ΔT_core; Represented as: ΔT = T_current - T_target Where T_current is the current temperature of the controlled object, and ΔT_core is the preset core region boundary value.

[0009] Furthermore, the time-varying hybrid control strategy described in this invention is expressed as follows: The control output duty(t) is obtained by weighting the output duty_bb(t) of the traditional Bang-Bang controller and the output duty_mdp(t) of the Markov decision process control unit, and is expressed as: duty(t) = w(t)·duty_bb(t) + [1-w(t)]·duty_mdp(t) Where w(t) is a time-varying hybrid weight function, whose value monotonically decreases from 1 to 0 during the switching time period [t_switch_start, t_switch_end], thereby achieving a smooth transition from traditional Bang-Bang control to Markov decision control.

[0010] Furthermore, the Markov decision process control unit described in this invention is designed based on a discrete state space and a discrete action space, specifically as follows: The state space S contains Ns discrete states, which are obtained by discretizing the continuous temperature error ΔT∈[-ΔTmax, ΔTmax] at equal intervals, where ΔTmax is the maximum permissible temperature error; The action space A contains Na discrete actions, corresponding to Na discrete levels of PWM duty cycle, with the duty cycle range covering 0% to 100%; The reward function of the Markov decision process control unit is designed solely based on the temperature control accuracy of the controlled object, and is specifically defined as a monotonically decreasing function of the temperature error. The state-action value function Q(s,a) of the Markov decision process control unit is stored in tabular form with dimensions Ns×Na. Each cell in the table describes the benefit obtained by taking action a in state s.

[0011] Furthermore, the offline training described in this invention includes experience replay, batch training, and improved exploration strategies; The improved exploration strategy includes: The core region enhanced exploration mechanism increases the exploration rate to ε_core = k_core × ε_base when the system is in a discrete state corresponding to the core control region, where k_core>1 is the enhancement coefficient and ε_base is the base exploration rate. The action preference mechanism prioritizes actions with intermediate duty cycles based on probability p_mid when exploring the core control area. The experience preference replay mechanism prioritizes the experience data of the core control area during experience sampling with probability p_prefer.

[0012] Based on the same inventive concept, this invention also provides a high-precision temperature control method implemented by an on-board PWM-MDP combined high-precision temperature control system, the high-precision temperature control method comprising the following steps: Step A: System initialization, set the target temperature T_target, initialize the physical model parameters, set the four-stage time parameters t1, t2, t3, t4 and the control accuracy targets ΔT1, ΔT2, where ΔT2 < ΔT1; Step B: Perform the first stage of data collection and control, using an improved PWM-Bang-Bang controller with an active exploration mechanism in the core area to control the temperature within the range of T_target±ΔT1 within time t1, while collecting complete control experience including state, action, reward, and next state. Step C: Perform the second stage of offline training. Based on the control experience collected in step B, train the Markov decision process control unit. The training process adopts the core region enhancement exploration strategy and the preference experience playback mechanism. During this stage, the control method of the controlled object within time t2 is still the improved PWM-Bang-Bang controller. Step D: Execute the third-stage gradual switching, and within time t3, achieve a smooth transition from traditional control to Markov decision process control through time-varying hybrid control; Step E: Execute the fourth stage of pure Markov decision process control. Within time t4, the trained Markov decision process controller will precisely control the temperature within the range of T_target±ΔT2.

[0013] Furthermore, the improved PWM-Bang-Bang controller operation method in step B of the present invention includes: Step B1: Read the current temperature T_current and calculate the temperature error ΔT = T_current - T_target; Step B2: Determine if the system is in the core control region. If |ΔT|≤ΔT_core, proceed to step B3; otherwise, proceed to step B4. Step B3: Execute an exploration action with probability p_explore, randomly selecting from a preset set of intermediate duty cycles; execute a basic control action with probability 1-p_explore. Step B4: Execute the basic PWM-Bang-Bang control logic: if ΔT < -ΔT1, output duty cycle 100%; if ΔT > ΔT1, output duty cycle 0%; otherwise, output the duty cycle of the previous moment. Step B5: Execute control actions and update system status; Step B6: Record control experience, including state s=ΔT, action a=output duty cycle, reward r, and next state s'=ΔT_next.

[0014] Furthermore, the training method for the Markov decision process control unit in step C of the present invention includes: Step C1: State-space design, discretize the continuous temperature error ΔT∈[-ΔT_max, ΔT_max] into N_s states at equal intervals, where ΔT_max is the maximum temperature deviation allowed by the physical system; Step C2: Action space design, discretize the PWM output duty cycle into N_a levels, duty cycle duty ∈ {duty_0, duty_1, ..., duty_{N_a-1}}, where duty_i = i / (N_a-1); Step C3: Design the reward function. Based on the control accuracy, design a monotonically decreasing function: R(ΔT) = f(ΔT), where f(·) is a monotonically decreasing function of ΔT; Step C4: Experience preprocessing, converting the control experience collected in step B into Markov decision process experience tuples (s,a,r,s'), where s and s' are the discretized states; Step C5: Initialize the Q-table, which has dimensions N_s × N_a, and initialize all elements to 0; Step C6: Execute the training loop, performing N_episode rounds of training; Step C7: At the start of each training round, randomly initialize the environment state; Step C8: For each time step, calculate the current discrete state s; Step C9: Select action a using an improved ε-greedy strategy, and apply an enhanced exploration rate to the state of the core control area; Step C10: Perform action a, interact with the environment to obtain reward r and the next discrete state s'; Step C11: Store the experience (s,a,r,s') into the experience replay buffer; Step C12: Sample N_batch of experience samples from the buffer to form a training batch, using a preference sampling mechanism; Step C13: For each experience in the batch, calculate the target Q value: target = r + γ·max_{a'}Q(s',a'); Step C14: Update Q value: Q(s,a) = Q(s,a) + α·(target - Q(s,a)), where α is the learning rate and γ is the discount factor; Step C15: Decrease the exploration rate, decrease ε according to the predetermined decay law; Step C16: Determine if the termination condition has been met; if so, end the current round of training. Step C17: Determine whether all training rounds have been completed. If so, end the training and obtain the Q-table of completed training.

[0015] Furthermore, the improved ε-greedy strategy described in this invention is specifically as follows: For the current state s, determine whether it is a discrete state corresponding to the core control region; If it is a core control area state, then the enhanced exploration rate ε_enhanced = min(1.0, ε_base ×k_enhance) is used, where k_enhance>1 is the enhancement coefficient; If the state is not a core control area, the base exploration rate ε_base is used; Randomly select an action with probability ε, and select the action with the largest current Q value with probability 1-ε; When randomly exploring the core control area, the action with the middle duty cycle is selected with probability p_mid_action.

[0016] Furthermore, the specific method for the gradual switching in step D of the present invention is as follows: Set the switching time period to [t_switch_start, t_switch_end], where t_switch_start = t3 and t_switch_end = t4; During the switching time period, the time-varying hybrid weight w(t) is calculated according to the following formula: w(t) = 1- (t-t_switch_start) / (t_switch_end-t_switch_start) The control output duty(t) is a convex combination of the traditional controller output duty_bb(t) and the Markov decision process controller output duty_mdp(t); Monitor temperature fluctuations during the switchover process to ensure a smooth transition.

[0017] The beneficial effects of this invention are as follows: 1. High Control Precision: This invention optimizes the control strategy through Markov decision process learning, ultimately achieving high-precision temperature control at the ±0.1℃ level, significantly outperforming traditional control methods. The discrete motion space design perfectly matches the characteristics of PWM control, avoiding errors introduced by continuous motion quantization.

[0018] 2. Efficient Exploration: This invention designs an enhanced exploration mechanism for the core control region, solving the key problem of insufficient exploration near the target value in reinforcement learning. By increasing the exploration rate of the core region, favoring intermediate actions, and prioritizing the replay of core experiences, the quality and efficiency of experience data are significantly improved.

[0019] 3. Smooth transition: This invention adopts a time-varying hybrid control strategy to achieve a smooth transition from traditional control to intelligent control. The hybrid weights change linearly with time, and the control output changes continuously, avoiding temperature oscillations and overshoot caused by abrupt changes in the control strategy.

[0020] 4. Strong Adaptability: The Markov Decision Process Controller can autonomously learn the optimal control strategy through interaction with the environment, exhibiting strong adaptability to changes in system parameters and environmental disturbances. The tabular Q-learning algorithm has good interpretability, facilitating engineering implementation and debugging.

[0021] This invention is not only applicable to on-orbit temperature control of high-precision temperature-sensitive measurement components such as star sensors and gyroscopes, but can also be extended to other physical quantities requiring high-precision control, such as pressure, flow rate, and concentration. The four-stage progressive framework and core region enhancement exploration mechanism are universal and can provide a reference paradigm for the application of reinforcement learning in practical control. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a block diagram of the overall architecture of the on-board PWM-MDP combined high-precision temperature control system described in this invention. Figure 2 This is a timeline diagram of the four-stage progressive control framework described in this invention. Figure 3 This is a flowchart of the improved PWM-Bang-Bang controller described in this invention. Figure 4 This is a training flowchart for the Markov decision process controller described in this invention. Figure 5 This is a schematic diagram of the hybrid control principle for the gradual switching stage described in this invention; Figure 6 This is a schematic diagram of the core region enhancement exploration mechanism described in this invention. Detailed Implementation

[0024] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0025] Implementation Method 1: Combination Figures 1 to 6 This invention addresses the shortcomings of existing satellite thermal control methods, which fail to meet the high-precision and high-stability temperature control requirements of next-generation satellites. It proposes an onboard PWM-MDP combined high-precision temperature control system. The following description, combined with... Figure 1 The PWM-MDP combined high-precision temperature control system described in this invention will be specifically explained; The overall architecture of the system is as follows: Figure 1 As shown, it includes: a physical model module, a control decision module, a state management module, an experience management module, and a learning and training module; The physical model module is used to simulate the dynamic temperature characteristics of the controlled object and output the current temperature data of the controlled object. The control decision module is used to receive the current temperature data output by the physical model module, and in accordance with the requirements of the four-stage progressive control framework, output corresponding control commands to control the temperature of the controlled object. The control decision module includes a traditional control unit and a Markov decision process control unit; the traditional control unit is used to execute an improved PWM-Bang-Bang controller to provide basic temperature control support for the system; the Markov decision process control unit is built based on the Q-learning algorithm to achieve high-precision temperature control, and the two units work together to complete four-stage control switching and temperature regulation. The state management module is used to collect the temperature data of the controlled object after it has been processed by the control decision module, calculate the temperature error, store the control actions output by the control decision module, and perform state discretization processing on the temperature-related data to form empirical data, which is then output to the empirical management module. The experience management module uses a circular buffer to store the experience data and supports experience playback; The learning and training module is used to perform offline training on the Markov decision process control unit. After the training is completed, the optimized control strategy is fed back to the Markov decision process control unit through the experience management module.

[0026] Furthermore, in this embodiment, the functions of each of the above modules are further explained; The physical model module simulates the temperature dynamics of the controlled object based on a simplified first-order thermodynamic equation subjected to periodic external heat flow disturbance; the model equations are as follows: C·(dT / dt)=P_in–Q_loss+Q(t) Where C is the heat capacity of the controlled object (J / ℃), T is the current temperature (℃), t is the time (s), P_in is the heater input heat power (W), and Q_loss is the radiation loss power (W). The input heat power is proportional to the PWM duty cycle: P_in = η·P_rated·duty, where η is the heater efficiency, P_rated is the heater rated power (W), and duty is the PWM output duty cycle (0-1). The radiation loss power is calculated according to the Stephen-Boltzmann radiation formula: P_loss = σ·A·((T+273.15)⁴-T_env⁴), where σ is the Stephen-Boltzmann constant (W / (m²·K⁴)), A is the heat dissipation area (m²), and T_env is the ambient temperature (K), typically 4K, the temperature of a cold, black cosmic background. The periodic external heat flow disturbance is given by a sine function Q(t) = Q₀. (sin(2π / Tn t)+1), Q0 is the peak external heat flux magnitude (W / m²), and Tn is the disturbance external heat flux period (s).

[0027] The control decision module comprises a traditional control unit and a Markov decision process control unit. The traditional control unit executes an improved PWM-Bang-Bang controller, adding a core region active exploration mechanism to the basic Bang-Bang control logic. The Markov decision process control unit is implemented based on a tabular Q-learning algorithm and includes functions such as state awareness, action selection, and value function update.

[0028] Specifically, the core area active exploration mechanism is as follows: When the system temperature is within the core control region, an exploratory action is performed with a preset probability p_explore. The core control region is defined as the region where |ΔT|≤ΔT_core, where ΔT=T_current-T_target, T_current is the current temperature of the controlled object, and ΔT_core is the preset core region boundary value.

[0029] The Markov Decision Process (MDP) described in this invention is a mathematical framework for sequential decision-making. It achieves autonomous learning and prediction of the optimal control strategy by modeling the environmental state and evaluating the long-term benefits of different actions. This invention applies the MDP algorithm to a satellite thermal control system, enabling the control system to dynamically adjust the heating strategy based on real-time thermal conditions and historical data, providing more timely and smoother compensation for external and internal thermal disturbances. This method is expected to significantly improve the temperature stability of onboard units, especially temperature-sensitive critical units, achieving a leap from "no over-temperature" to "precise constant temperature," providing key technical support for the development of next-generation high-precision satellite platforms.

[0030] The Markov decision process control unit described in this invention is designed based on discrete state space and discrete action space, wherein: The state space S contains N_s discrete states, which are obtained by discretizing the continuous temperature error ΔT∈[-ΔT_max,ΔT_max] at equal intervals, where ΔT_max is the maximum allowable temperature error. The action space A contains N_a discrete actions, corresponding to N_a discrete levels of PWM duty cycle, with the duty cycle range covering 0% to 100%; The reward function R(s,a) is designed based solely on the temperature control accuracy of the controlled object. It is defined as a monotonically decreasing function of the temperature error, meaning that the smaller the difference between the temperature of the controlled object and the target temperature, the higher the reward. The state-action value function Q(s,a) is stored in tabular form with dimensions N_s×N_a. Each cell in the table describes the benefit gained by taking action a in state s.

[0031] The learning and training module implements the offline training function of the Markov decision process controller. It includes sub-modules such as experience replay buffer management, batch training, and exploration strategy management. An improved exploration strategy is employed, specifically optimized for the core control region.

[0032] The improved exploration strategy includes: The core region enhanced exploration mechanism increases the exploration rate to ε_core = k_core × ε_base when the system is in a discrete state corresponding to the core control region, where k_core > 1 is the enhancement coefficient and ε_base is the base exploration rate. The action preference mechanism prioritizes actions with intermediate duty cycles based on probability p_mid when exploring the core control area. The experience preference replay mechanism prioritizes the experience data of the core control area during experience sampling with probability p_prefer.

[0033] The state management module is responsible for acquiring temperature data, calculating the error with the target temperature, and discretizing the state. State discretization uses an equal-interval discretization method to map the continuous temperature error to a finite discrete state space. The PWM output duty cycle action is discretized into a finite discrete action space.

[0034] Experience management module: Uses a first-in-first-out circular buffer to store control experience; the buffer capacity is configurable. Supports experience playback function, and can employ uniform random sampling or preferred sampling strategies.

[0035] Furthermore, in this embodiment, the four-stage progressive control framework is described in detail; like Figure 2 As shown, the system operates according to a four-stage progressive control framework, with the time parameters and control objectives for each stage as follows: Phase 1 (0-t1 seconds): Data collection and basic control phase. An improved PWM-Bang-Bang controller is employed, with the control objective being to keep the temperature within the range of T_target ± ΔT1, where ΔT1 = 1.0℃. During this phase, while ensuring basic control performance, the system actively collects comprehensive control experience data through an exploratory mechanism.

[0036] Phase 2 (t2~t3 seconds): Offline training phase. Based on the data collected in Phase 1, the Markov decision process controller is trained in a simulation environment. The training process is conducted entirely offline and does not affect the normal operation of the actual system.

[0037] The third stage (t3~t4 seconds): gradual transition stage. A time-varying hybrid control strategy is used to gradually transition from traditional control to Markov decision process control. The hybrid weight w(t) decays linearly with time, ensuring a smooth control transition.

[0038] Fourth stage (after t4 seconds): Pure Markov decision process control stage. The trained Markov decision process controller completely takes over the control, and the control objective is to precisely control the temperature within the range of T_target ± ΔT2, where ΔT2 = 0.1 °C. The system operates continuously in this stage to achieve high-precision temperature control.

[0039] In the progressive switching stage, a time-varying hybrid control strategy is adopted. The control output duty(t) is obtained by weighting the output duty_bb(t) of the traditional Bang-Bang controller and the output duty_mdp(t) of the Markov decision process controller, and the formula is as follows: duty(t)=w(t)·duty_bb(t)+[1-w(t)]·duty_mdp(t) where w(t) is a time-varying hybrid weight function, and its value monotonically decreases from 1 to 0 within the switching time period [t_switch_start, t_switch_end], so as to achieve a smooth transition from traditional Bang-Bang control to MDP control.

[0040] As Figure 3 shown, the working process of the improved PWM-Bang-Bang controller includes: S301: Read the current temperature sensor data T_current, and calculate the temperature error ΔT = T_current - T_target, where T_target is the target temperature; S302: At the beginning of each round of training, randomly initialize the environmental state. The initial temperature is randomly selected uniformly within the range of [T_target - 0.8, T_target + 0.8] °C to ensure that the training covers various initial conditions.

[0041] S303: Determine whether it is the core control area. The core area is defined as |ΔT| ≤ ΔT_core, where ΔT_core = 0.2 °C. If it is the core area, enter S304, otherwise enter S307; S304: Generate a random number rand ∈ [0, 1), and determine whether to execute an exploration action. The exploration probability p_explore is recommended to be set to 0.2; S305: If rand < p_explore, execute an exploration action. Select an exploration action set according to the error direction: if ΔT < 0, randomly select a duty ratio from the heating action set {0.6, 0.7, 0.8, 0.9}; if ΔT > 0, randomly select a duty ratio from the cooling action set {0.1, 0.2, 0.3, 0.4}; if ΔT ≈ 0, randomly select a duty ratio from the maintaining action set {0.3, 0.4, 0.5, 0.6, 0.7}; S306: If rand≥p_explore, execute the basic control action: output the heater switching status under basic Bang-Bang control.

[0042] S307: Non-core areas execute basic PWM-Bang-Bang control logic: if ΔT < -ΔT1, output duty = 1.0; if ΔT > ΔT1, output duty = 0.0; otherwise, output the duty of the previous state. S308: Outputs a PWM control signal to control the heater's operation; S309: Record control experience, including state s=ΔT, action a=output duty cycle, immediate reward r, and next state s'=ΔT_next. The reward r is calculated based on the control accuracy ΔT; see step C3 for the specific formula. like Figure 4 As shown, the training process for the Markov decision process controller includes: S401: Initialize training parameters. The learning rate α is set to 0.001 to control the Q-value update step size; the discount factor γ is set to 0.95 to balance immediate rewards and long-term rewards; the initial exploration rate ε_initial is set to 1.0 to encourage full exploration; the final exploration rate ε_final is set to 0.01 to ensure that the learned knowledge is fully utilized in the later stages; the number of training epochs N_episode is set to 2000 to ensure sufficient training.

[0043] S402: Design the state space. Discretize the continuous temperature error ΔT∈[-1.0,1.0]℃ into N_s=20 states at equal intervals. The state index calculation formula is: state_idx=floor((ΔT+1.0) / (2.0 / (N_s-1))), ensuring that state_idx∈{0,1,...,19}, with the core control region corresponding to states 8-11.

[0044] S403: Design the action space. Discretize the PWM duty cycle into N_a = 11 actions, corresponding to duty cycles of 0%, 10%, 20%, ..., 100%. The mapping relationship between the action index and the duty cycle is: duty = action_idx / 10.0.

[0045] S404: Design the reward function. Design a monotonically decreasing function based on control precision. If |ΔT|≤0.1℃ and r=2.0, precise control is encouraged; If 0.1 < |ΔT| ≤ 0.3℃, r = 1.0, encouraging the approach to the target; If |ΔT|>0.3℃, r=-|ΔT|×3.0, which penalizes a large error.

[0046] S405: Initialize Q-table. Create a two-dimensional array Q_table of size N_s × N_a, and initialize all elements to 0.

[0047] S406: Experience Preprocessing. The control experience collected in the first stage is converted into Markov decision process experience tuples (s, a, r, s'). This requires discretizing the continuous temperature error ΔT into state indices s and s', and quantizing the continuous duty cycle into action index a.

[0048] S407: Start the training loop. Perform N_episode=2000 training rounds, each round is called an episode.

[0049] S408: At the start of each training round, the environment state is randomly initialized. The initial temperature is uniformly and randomly selected within the range of [T_target-0.8, T_target+0.8]℃ to ensure that training covers various initial conditions.

[0050] S409: Calculate the current discrete state s. Read the current temperature T_current, calculate ΔT, and obtain the state index s using the state discretization formula.

[0051] S410: Select action a using an improved ε-greedy strategy. For the current state s, determine if it is a core control region state (s∈{8,9,10,11}). If it is a core region state, use the enhanced exploration rate ε_enhanced=min(1.0,ε×1.5); otherwise, use the base exploration rate ε. Randomly select an action with probability ε, where during random exploration in the core region, intermediate actions (action indices 2-8) are selected with a 70% probability; the action with the largest current Q value is selected with a probability of 1-ε.

[0052] S411: Execute action a, interacting with the environment. Convert action index a to duty cycle (duty), input the physical model to calculate the temperature T_next at the next moment, calculate ΔT_next and discretize it into state s', and calculate the immediate reward r for this state according to the reward function.

[0053] S412: Stores experience (s,a,r,s') into the experience replay buffer. The buffer uses first-in-first-out management, with a capacity of 10000. When the buffer is full, the oldest experience is automatically evicted.

[0054] S413: Sample training batches from the buffer. The batch size N_batch is set to 32. A preference sampling mechanism is adopted: core region experience is sampled with a 50% probability. Specifically, the experience of state s or s' as the core region is selected from the buffer, and half of the batches are randomly sampled from it; the other half of the batches are uniformly and randomly sampled from the entire buffer.

[0055] S414: Batch training updates Q-value. For each experience (s,a,r,s') in the batch, calculate the target value according to the following formula, calculate the time difference error δ=target-Q(s,a), and update the Q-value: Q(s,a)=Q(s,a)+α·δ.

[0056] target=r+γ·max_{a'}Q(s',a') S415: Decaying exploration rate. The exploration rate is updated according to an exponential decay law: ε=max(ε_final,ε×decay_rate), where decay_rate=(ε_final / ε_initial)^{1 / N_episode}, ensuring that ε is close to ε_final at the end of training.

[0057] S416: Determine if the termination conditions have been met. Termination conditions include: 1) The maximum number of steps has been executed (e.g., 200 steps); 2) The temperature has stabilized near the target (|ΔT| < 0.05℃ for 10 consecutive steps). If the termination conditions are met, the current round of training ends.

[0058] S417: Determine if all training rounds have been completed. If N_episode rounds of training have been completed, end the training and obtain the Q-table of the completed training; otherwise, return to S408 to start the next round of training.

[0059] like Figure 5 As shown, the hybrid control principle during the gradual switching phase is as follows: During the switching time interval [t3, t4], the control output duty(t) is a weighted sum of the output duty_bb(t) of the traditional controller and the output duty_mdp(t) of the Markov decision process controller. The hybrid weight w(t) changes linearly with time. w(t) = 1 - (t - t3) / (t4 - t3) When t=t3, w=1, and the system is completely controlled by a conventional controller; when t=t4, w=0, and the system is completely controlled by a Markov decision process controller; the intermediate time points represent a mixture of both. This linear transition ensures continuous change in the control output, avoiding temperature oscillations caused by abrupt changes.

[0060] like Figure 6 As shown, the enhanced exploration mechanism in the core area includes three levels: 1. Active exploration of the core area during the data collection phase: When the system is in the core control area (|ΔT|≤0.2℃), an exploration action is performed with a probability p_explore=0.2 to try various intermediate duty cycles and collect rich experience in the core area.

[0061] 2. Enhanced exploration of core regions during the training phase: For the discrete states (states 8-11) corresponding to the core regions, the exploration rate is increased by 50%, encouraging more actions to be attempted in these key states.

[0062] 3. Core area movement preference during training: When exploring the core area, there is a 70% probability of choosing intermediate movements (duty cycle 20%-80%). These movements are more suitable for fine adjustment and avoid overshoot caused by extreme movements.

[0063] 4. Core region preference sampling for experience replay: When sampling from the experience replay buffer, core region experiences are selected with a 50% probability to increase the learning frequency of these key experiences.

[0064] Implementation Method 2: This implementation method provides a detailed description of the high-precision temperature control method implemented by the on-board PWM-MDP combined high-precision temperature control system proposed in this invention. The specific implementation steps are as follows: Step 1: System Initialization Step 1.1: Set control targets: target temperature T_target = 25.0℃, first stage control accuracy ΔT1 = 1.0℃, fourth stage control accuracy ΔT2 = 0.1℃, core area boundary ΔT_core = 0.2℃.

[0065] Step 1.2: Set time parameters: first stage duration T1 = 1200 seconds, second stage duration T2 = 600 seconds, third stage duration T3 = 600 seconds, fourth stage duration T4 is set as needed, at least 1200 seconds to evaluate steady-state performance.

[0066] Step 1.3: Initialize physical model parameters: heat capacity C = 10.0 J / ℃, heater efficiency η = 1.0, rated heating power P_rated = 3.0 W, disturbance external heat flow peak value 0.5 W / m², disturbance external heat flow period 200 s, ambient temperature T_env = 4 K. Control period Δt = 1.0 s.

[0067] Step 1.4: Initialize control parameters: Improve the exploration probability p_explore of the PWM-Bang-Bang controller to 0.2; the learning rate α = 0.001, discount factor γ = 0.95, initial exploration rate ε_initial = 1.0, final exploration rate ε_final = 0.01, number of training epochs N_episode = 2000, batch size N_batch = 32, and experience buffer capacity = 10000.

[0068] Step 2: First Phase Data Collection and Control Step 2.1: The system starts and begins the first stage of control. Current time t=0.

[0069] Step 2.2: Perform the following operations in each control cycle: Step 2.2.1: Read the current temperature T_current. In the actual system, this is collected by a temperature sensor; in the simulation system, it is calculated using a physical model.

[0070] Step 2.2.2: Calculate the temperature error ΔT = T_current - 25.0.

[0071] Step 2.2.3: Execute improved PWM-Bang-Bang control: If |ΔT|≤0.2, enter the core region control logic: generate a random number rand∈[0,1); if rand<0.2, perform an exploration action: if ΔT<-0.1, randomly select a duty from {0.6,0.7,0.8,0.9}; if ΔT>0.1, randomly select a duty from {0.1,0.2,0.3,0.4}; if -0.1≤ΔT≤0.1, randomly select a duty from {0.3,0.4,0.5,0.6,0.7}. If rand≥0.2, perform the basic control action: duty=0.5.

[0072] If |ΔT|>0.2, enter the basic control logic: if ΔT<-1.0, duty=1.0; if ΔT>1.0, duty=0.0; if -1.0≤ΔT≤1.0, duty=duty', where duty' is the duty cycle of the previous state.

[0073] Step 2.2.4: Output PWM control signal to control the heater to work.

[0074] Step 2.2.5: Calculate the temperature at the next moment using a physical model.

[0075] Step 2.2.6: Calculate the error at the next time step ΔT_next = T_next - 25.0.

[0076] Step 2.2.7: Calculate the immediate reward r: if |ΔT|≤0.1, r=2.0; otherwise if |ΔT|≤0.3, r=1.0; otherwise r=-|ΔT|×3.0.

[0077] Step 2.2.8: Record control experience: Store the tuple (ΔT,duty,r,ΔT_next) into the experience dataset.

[0078] Step 2.2.9: Update the state: T_current=T_next.

[0079] Step 2.3: When t≥t1, the first stage ends. At this point, 1200 control experiences have been collected.

[0080] Step 3: Second Stage Offline Training Step 3.1: Prepare training data. Convert the 1200 experiences collected in the first stage into Markov decision process format. For each experience (ΔT, duty, r, ΔT_next), perform the following transformations: State discretization: s = floor((ΔT + 1.0) / 0.1053), ensuring s ∈ {0, 1, ..., 19}; Action quantization: a = round(duty × 10), ensuring a ∈ {0, 1, ..., 10}; Next state discretization: s' = floor((ΔT_next + 1.0) / 0.1053); thus obtaining the Markov decision process experience (s, a, r, s').

[0081] Step 3.2: Initialize the Q-table. Create a 20×11 two-dimensional array Q_table, and initialize all elements to 0.

[0082] Step 3.3: Initialize the experience replay buffer. Create an empty list `experience_buffer` with a maximum capacity of 10,000. Store the 1,200 transformed Markov decision process experiences into the buffer.

[0083] Step 3.4: Initialize training parameters. The current exploration rate ε = 1.0, and the decay coefficient decay_rate = (0.01 / 1.0)^{1 / 2000} ≈ 0.9985.

[0084] Step 3.5: Perform the training cycle. Run 2000 training cycles, with each cycle proceeding as follows: Step 3.5.1: Initialize the environment. Randomly select an initial temperature T_init∈[24.2,25.8]℃, calculate the initial error ΔT_init=T_init-25.0, and discretize to obtain the initial state s.

[0085] Step 3.5.2: The current training step number step=0, and the current cumulative reward episode_reward=0.

[0086] Step 3.5.3: When step < 200, execute the following loop: Step 3.5.3.1: Select Action. Determine if the current state s is a core region state (s∈{8,9,10,11}). If yes, enhance the exploration rate ε_enhanced=min(1.0,ε×1.5); otherwise, ε_enhanced=ε. Generate a random number rand∈[0,1). If rand<ε_enhanced, perform exploration: if it is a core region state and rand<0.7, randomly select a from the intermediate actions {2,3,4,5,6,7,8}; otherwise, randomly select a from all actions {0,1,...,10}. If rand≥ε_enhanced, perform exploitation: a=argmax_{a'}Q_table[s,a'].

[0087] Step 3.5.3.2: Execute the action. Convert the action index a to a duty cycle of duty = a / 10.0. Calculate the next temperature T_next using the physical model. Calculate the next error ΔT_next = T_next - 25.0, and discretize to obtain the next state s'.

[0088] Step 3.5.3.3: Calculate the reward r: If |ΔT|≤0.1, r=2.0; otherwise, if |ΔT|≤0.3, r=1.0; otherwise, r=-|ΔT|×3.0.

[0089] Step 3.5.3.4: Store experiences. Add the experience (s,a,r,s') to the experience_buffer. If the buffer is full (more than 10,000 entries), remove the oldest experience.

[0090] Step 3.5.3.5: Experience replay learning. If the buffer size is ≥32, perform batch training: Sampling strategy: Preferred sampling is used with a 50% probability, prioritizing core region experiences. Implementation: Experiences where state s or s' is a core region state are selected from the buffer. If the number is ≥16, 16 are randomly selected; otherwise, all are selected, and the remaining experiences are randomly selected uniformly from the entire buffer, for a total of 32 experiences. Uniform random sampling is used with a 50% probability, randomly selecting 32 experiences from the entire buffer.

[0091] For each experience (s_i, a_i, r_i, s'i) in the batch, calculate the target value target_i = r_i + 0.95·max{a'}Q_table[s'_i, a']. Calculate the update amount δ_i = target_i - Q_table[s_i, a_i]. Update the Q value: Q_table[s_i, a_i] = Q_table[s_i, a_i] + 0.001·δ_i.

[0092] Step 3.5.3.6: Update the status and statistics. s=s', T_current=T_next, step=step+1, episode_reward=episode_reward+r.

[0093] Step 3.5.3.7: Determine the early termination condition. If step > 30 and |ΔT| < 0.05 for 10 consecutive steps, terminate the current round of training early.

[0094] Step 3.5.4: Decay detection rate. ε=max(0.01,ε×0.9985).

[0095] Step 3.5.5: Record training statistics. Save the cumulative reward (episode_reward) for the current round for analyzing training progress.

[0096] Step 3.6: Training complete. After 2000 rounds of training, the trained Q-table, Q_table, is obtained. At this point, the exploration rate ε has decayed to approximately 0.01.

[0097] Step 4: Phase 3 Gradual Switching Step 4.1: The system enters the third stage. The current time t = 1800 seconds, the switch start time t3 = 1800 seconds, and the switch end time t4 = 2400 seconds.

[0098] Step 4.2: Perform the following operations in each control cycle: Step 4.2.1: Read the current temperature T_current and calculate ΔT = T_current - 25.0.

[0099] Step 4.2.2: Calculate the mixed weight w. w = 1 - (t - 1800) / 600. When t = 1800, w = 1. When t = 2400, w = 0. The weight changes linearly in between.

[0100] Step 4.2.3: Calculate the traditional controller output duty_bb. Use improved PWM-Bang-Bang control logic; Step 4.2.4: Calculate the Markov decision process controller output, duty_mdp. Discretize the state: s = floor((ΔT + 1.0) / 0.1053). Select the optimal action: a* = argmax_{a}Q_table[s,a]. Calculate the duty cycle: duty_mdp = a* / 10.0.

[0101] Step 4.2.5: Calculate the mixed output duty = w·duty_bb + (1-w)·duty_mdp.

[0102] Step 4.2.6: Output PWM control signal to control the heater to work.

[0103] Step 4.2.7: Calculate the temperature T_next at the next moment and update the state T_current=T_next.

[0104] Step 4.3: When t≥2400, the third stage ends and the fourth stage begins.

[0105] Step 5: Fourth-stage pure Markov decision process control Step 5.1: The system enters the fourth stage. Current time t ≥ 2400 seconds.

[0106] Step 5.2: Perform the following operations in each control cycle: Step 5.2.1: Read the current temperature T_current and calculate ΔT = T_current - 25.0.

[0107] Step 5.2.2: Discretize the state. s=floor((ΔT+1.0) / 0.1053), ensuring that s∈{0,1,...,19}.

[0108] Step 5.2.3: Select the optimal action. a*=argmax_{a}Q_table[s,a]. If multiple actions have the same maximum Q value, select the action with the duty cycle closest to 0.5.

[0109] Step 5.2.4: Calculate the duty cycle. duty = a * / 10.0.

[0110] Step 5.2.5: Output PWM control signal to control the heater to work.

[0111] Step 5.2.6: Calculate the temperature T_next at the next moment, update the state T_current=T_next, t=t+1.

[0112] Step 5.3: Continuous operation to achieve high-precision temperature control.

[0113] In summary, the on-board PWM-MDP combined high-precision temperature control system and method proposed in this invention have good maintainability, and the Q-table can be viewed and adjusted online, facilitating engineers' understanding and optimization of control strategies. Simultaneously, the following effects are achieved: High control precision: By learning and optimizing the control strategy through a Markov decision process, high-precision temperature control at the ±0.1℃ level is achieved, significantly outperforming traditional control methods. The discrete motion space design perfectly matches the characteristics of PWM control, avoiding errors introduced by continuous motion quantization.

[0114] Efficient Exploration: An enhanced exploration mechanism is designed for the core control region, addressing the key issue of insufficient exploration near the target value in reinforcement learning. By increasing the exploration rate in the core region, favoring intermediate actions, and prioritizing the replay of core experiences, the quality and efficiency of experience data are significantly improved.

[0115] Smooth transition: A time-varying hybrid control strategy is adopted to achieve a smooth transition from traditional control to intelligent control. The hybrid weight changes linearly with time, and the control output changes continuously, avoiding temperature oscillations and overshoot caused by abrupt changes in the control strategy.

[0116] Strong adaptability: The Markov decision process controller can autonomously learn the optimal control strategy through interaction with the environment, exhibiting strong adaptability to changes in system parameters and environmental disturbances. The tabular Q-learning algorithm has good interpretability, facilitating engineering implementation and debugging.

[0117] This invention is not only applicable to temperature control, but can also be extended to other physical quantities requiring high-precision control, such as pressure, flow rate, and concentration. The four-stage progressive framework and core region enhancement exploration mechanism are universal and can provide a reference paradigm for the application of reinforcement learning in practical control.

[0118] The above description of the technical solution provided by the present invention through several specific embodiments is intended to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, reasonable combinations of implementation methods and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A satellite-based PWM-MDP combined high-precision temperature control system, characterized in that, The system includes a physical model module, a control decision module, a state management module, an experience management module, and a learning and training module; The physical model module is used to simulate the dynamic temperature characteristics of the controlled object and output the current temperature data of the controlled object. The control decision module is used to receive the current temperature data output by the physical model module, and in accordance with the requirements of the four-stage progressive control framework, output corresponding control commands to control the temperature of the controlled object. The control decision module includes a traditional control unit and a Markov decision process control unit; the traditional control unit is used to execute an improved PWM-Bang-Bang controller to provide basic temperature control support for the system; the Markov decision process control unit is built based on the Q-learning algorithm to achieve high-precision temperature control, and the two units work together to complete four-stage control switching and temperature regulation. The state management module is used to collect the temperature data of the controlled object after it has been processed by the control decision module, calculate the temperature error, store the control actions output by the control decision module, and perform state discretization processing on the temperature-related data to form empirical data, which is then output to the empirical management module. The experience management module uses a circular buffer to store the experience data and supports experience playback; The learning and training module is used to perform offline training on the Markov decision process control unit. After the training is completed, the optimized control strategy is fed back to the Markov decision process control unit through the experience management module.

2. The on-board PWM-MDP combined high-precision temperature control system according to claim 1, characterized in that, The four-stage progressive control framework specifically includes: Phase 1: Data collection and basic control phase, starting at time t1. The traditional control unit runs an improved PWM-Bang-Bang controller to control the temperature of the controlled object within the range of ±ΔT1 of the target temperature T_target. At the same time, comprehensive control experience data is collected through an active exploration mechanism. The second stage is the offline training stage, which starts at time t2. Based on the control experience data collected in the first stage, the Markov decision process control unit is trained in the simulation environment. In this stage, the improved PWM-Bang-Bang controller is still used for the controlled object to control the temperature within the range of ±ΔT1 of the target temperature T_target. The third stage is the gradual switching stage, which starts at time t3. A time-varying hybrid control strategy is used to achieve a smooth transition from traditional control to Markov decision process control. The fourth stage is the pure Markov decision process control stage, which starts at time t4. The trained Markov decision process control unit takes over the control completely to achieve high-precision temperature control. The control objective is to keep the temperature within the range of T_target ±ΔT2, where ΔT2 < ΔT1.

3. A satellite-based PWM-MDP combined high-precision temperature control system according to claim 1 or 2, characterized in that, The improved PWM-Bang-Bang controller includes a core region active exploration mechanism, which specifically includes: When the system temperature is within the core control region, an exploratory action is performed with a preset probability p_explore, where the core region is defined as the region where |ΔT|≤ΔT_core; Represented as: ΔT = T_current - T_target Where T_current is the current temperature of the controlled object, and ΔT_core is the preset core region boundary value.

4. The on-board PWM-MDP combined high-precision temperature control system according to claim 2, characterized in that, The time-varying hybrid control strategy is expressed as follows: The control output duty(t) is obtained by weighting the output duty_bb(t) of the traditional Bang-Bang controller and the output duty_mdp(t) of the Markov decision process control unit, and is expressed as: duty(t) = w(t)·duty_bb(t) + [1-w(t)]·duty_mdp(t) Where w(t) is a time-varying hybrid weight function, whose value monotonically decreases from 1 to 0 during the switching time period [t_switch_start, t_switch_end], thereby achieving a smooth transition from traditional Bang-Bang control to Markov decision control.

5. A satellite-based PWM-MDP combined high-precision temperature control system according to claim 1 or 2, characterized in that, The Markov decision process control unit is designed based on discrete state space and discrete action space, specifically as follows: The state space S contains Ns discrete states, which are obtained by discretizing the continuous temperature error ΔT∈[-ΔTmax, ΔTmax] at equal intervals, where ΔTmax is the maximum permissible temperature error; The action space A contains Na discrete actions, corresponding to Na discrete levels of PWM duty cycle, with the duty cycle range covering 0% to 100%; The reward function of the Markov decision process control unit is designed solely based on the temperature control accuracy of the controlled object, and is specifically defined as a monotonically decreasing function of the temperature error. The state-action value function Q(s,a) of the Markov decision process control unit is stored in tabular form with dimensions Ns×Na. Each cell in the table describes the benefit obtained by taking action a in state s.

6. A satellite-based PWM-MDP combined high-precision temperature control system according to claim 1 or 2, characterized in that, The offline training includes experience replay, batch training, and an improved exploration strategy; the improved exploration strategy includes: The core region enhanced exploration mechanism increases the exploration rate to ε_core = k_core × ε_base when the system is in a discrete state corresponding to the core control region, where k_core>1 is the enhancement coefficient and ε_base is the base exploration rate. The action preference mechanism prioritizes actions with intermediate duty cycles based on probability p_mid when exploring the core control area. The experience preference replay mechanism prioritizes the experience data of the core control area during experience sampling with probability p_prefer.

7. A high-precision temperature control method based on the on-board PWM-MDP combined high-precision temperature control system according to any one of claims 1-6, characterized in that, The methods include: Step A: System initialization, set the target temperature T_target, initialize the physical model parameters, set the four-stage time parameters t1, t2, t3, t4 and the control accuracy targets ΔT1, ΔT2, where ΔT2 < ΔT1; Step B: Perform the first stage of data collection and control, using an improved PWM-Bang-Bang controller with an active exploration mechanism in the core area to control the temperature within the range of T_target±ΔT1 within time t1, while collecting complete control experience including state, action, reward, and next state. Step C: Perform the second stage of offline training. Based on the control experience collected in step B, train the Markov decision process control unit. The training process adopts the core region enhancement exploration strategy and the preference experience playback mechanism. During this stage, the control method of the controlled object within time t2 is still the improved PWM-Bang-Bang controller. Step D: Execute the third-stage gradual switching, and within time t3, achieve a smooth transition from traditional control to Markov decision process control through time-varying hybrid control; Step E: Execute the fourth stage of pure Markov decision process control. Within time t4, the trained Markov decision process controller will precisely control the temperature within the range of T_target±ΔT2.

8. The high-precision temperature control method according to claim 7, characterized in that, The improved operation method of the PWM-Bang-Bang controller in step B includes: Step B1: Read the current temperature T_current and calculate the temperature error ΔT = T_current - T_target; Step B2: Determine whether it is in the core control area. If |ΔT|≤ΔT_core, proceed to step B3; otherwise, proceed to step B4. Step B3: Execute an exploration action with probability p_explore, randomly selecting from a preset set of intermediate duty cycles; execute a basic control action with probability 1-p_explore. Step B4: Execute the basic PWM-Bang-Bang control logic: if ΔT < -ΔT1, output duty cycle 100%; if ΔT > ΔT1, output duty cycle 0%; otherwise, output the duty cycle of the previous moment. Step B5: Execute control actions and update system status; Step B6: Record control experience, including state s=ΔT, action a=output duty cycle, reward r, and next state s'=ΔT_next.

9. The high-precision temperature control method according to claim 8, characterized in that, The training method for the Markov decision process control unit in step C includes: Step C1: State-space design, discretize the continuous temperature error ΔT∈[-ΔT_max, ΔT_max] into N_s states at equal intervals, where ΔT_max is the maximum temperature deviation allowed by the physical system; Step C2: Action space design, discretize the PWM output duty cycle into N_a levels, duty cycle duty ∈ {duty_0, duty_1, ..., duty_{N_a-1}}, where duty_i = i / (N_a-1); Step C3: Design the reward function, based on the control accuracy, design a monotonically decreasing function: R(ΔT) = f(ΔT), where f(·) is a monotonically decreasing function of ΔT; Step C4: Experience preprocessing, converting the control experience collected in step B into Markov decision process experience tuples (s,a,r,s'), where s and s' are the discretized states; Step C5: Initialize the Q-table, which has dimensions N_s × N_a, and initialize all elements to 0; Step C6: Execute the training loop, performing N_episode rounds of training; Step C7: At the start of each training round, randomly initialize the environment state; Step C8: For each time step, calculate the current discrete state s; Step C9: Select action a using an improved ε-greedy strategy, and apply an enhanced exploration rate to the state of the core control area; Step C10: Perform action a, interact with the environment to obtain reward r and the next discrete state s'; Step C11: Store the experience (s,a,r,s') into the experience replay buffer; Step C12: Sample N_batch of experience samples from the buffer to form a training batch, using a preference sampling mechanism; Step C13: For each experience in the batch, calculate the target Q value: target = r + γ·max_{a'}Q(s',a'); Step C14: Update Q value: Q(s,a) = Q(s,a) + α·(target - Q(s,a)), where α is the learning rate and γ is the discount factor; Step C15: Decrease the exploration rate, decrease ε according to the predetermined decay law; Step C16: Determine if the termination condition has been met; if so, end the current round of training. Step C17: Determine whether all training rounds have been completed. If so, end the training and obtain the Q-table of completed training.

10. The high-precision temperature control method according to claim 9, characterized in that, The improved ε-greedy strategy is specifically as follows: For the current state s, determine whether it is a discrete state corresponding to the core control region; If it is a core control area state, then the enhanced exploration rate ε_enhanced = min(1.0, ε_base × k_enhance) is used, where k_enhance>1 is the enhancement coefficient; If the state is not a core control area, the base exploration rate ε_base is used; Randomly select an action with probability ε, and select the action with the largest current Q value with probability 1-ε; When randomly exploring the core control area, prioritize actions with intermediate duty cycles based on probability p_mid_action; The specific method for the gradual switching in step D is as follows: Set the switching time period to [t_switch_start, t_switch_end], where t_switch_start = t3 and t_switch_end = t4; During the switching time period, the time-varying hybrid weight w(t) is calculated according to the following formula: w(t) = 1- (t-t_switch_start) / (t_switch_end-t_switch_start) The control output duty(t) is a convex combination of the traditional controller output duty_bb(t) and the Markov decision process controller output duty_mdp(t); Monitor temperature fluctuations during the switchover process to ensure a smooth transition.

Images