Motor micro-channel heat dissipation online optimization method based on deep reinforcement learning

Abstract

The invention discloses a motor micro-channel heat dissipation online optimization method based on deep reinforcement learning. The method comprises the following steps: firstly, modeling a control problem of a motor micro-channel heat dissipation system as a Markov decision process; secondly, a double-delay depth deterministic strategy gradient algorithm is adopted, a deep neural network based on an actor-commentator framework is constructed to serve as a strategy network, and mapping from a state vector to an optimal control action is achieved; then, in a digital twinborn simulation environment, performing security pre-training on the strategy network to obtain an initial agent model; and finally, deploying the model to an embedded controller of an actual motor system, carrying out online fine adjustment with security constraints through online acquisition of interaction data, and adding a security fault-tolerant mechanism to ensure stable operation of the system. According to the method, self-learning can be carried out, a global optimal balance point of heat dissipation efficiency and energy consumption can be found, on-demand heat dissipation and prospective control are realized, local hot spots are effectively inhibited, and the temperature control precision and the energy-saving effect under complex working conditions are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of motor thermal management, intelligent control and reinforcement learning, and specifically relates to an online optimization method for motor microchannel heat dissipation based on deep reinforcement learning. Background Technology

[0002] In recent years, with the advancement of microfabrication technology, microchannel cooling technology has been introduced into the field of motor thermal management due to its advantages such as small channel size, large specific surface area, and short heat transfer path. However, existing microchannel cooling systems mostly lack intelligent temperature sensing and closed-loop control capabilities, and still mostly adopt a simple "all on when one is on, all off when one is off" control strategy, which cannot adaptively adjust according to the real-time thermal state of the motor. Existing intelligent control methods such as PID and fuzzy control still rely on manual experience for parameter tuning, and cannot achieve globally optimal energy efficiency balance under complex and variable operating conditions. Furthermore, the on-demand, intermittent supply of cooling medium has not been achieved, its energy-saving potential has not been fully explored, and its response to sudden load changes is not rapid enough.

[0003] Conventional liquid cooling systems often operate in continuous mode, with the cooling pump running continuously regardless of whether the motor is in a high-heat state, resulting in a large amount of ineffective energy consumption and serious energy waste.

[0004] Existing microchannel cooling systems often lack closed-loop control strategies based on real-time temperature sensing, making it impossible to achieve on-demand, intermittent cooling. Their energy-saving potential has not been fully explored, and they are slow to respond to sudden load changes, lacking intelligent adaptive control capabilities.

[0005] Therefore, developing a new type of motor cooling system that can deeply integrate high-precision temperature sensing, efficient heat transfer through microchannels, and intelligent adaptive control technologies to achieve on-demand cooling, eliminate energy waste, and ensure the temperature safety of the motor under various operating conditions has become a key technical challenge for promoting the development of the next generation of high-performance motors. Summary of the Invention

[0006] To address the issues of lag and insufficient temperature control accuracy in existing DRL microfluidic cooling systems, this invention employs a lightweight improved DRL model architecture to achieve precise adaptation of dynamic heat generation in the motor region, balancing temperature control efficiency and energy saving. This invention models the heat dissipation control problem as a Markov decision process, defining states, actions, and reward functions that align with engineering realities. An online optimization architecture for a dual-delay deep deterministic policy gradient algorithm is designed, resolving the issues of efficient learning and stability in continuous control spaces.

[0007] This invention proposes a hybrid deployment mode of "simulation pre-training + online fine-tuning," which significantly shortens the learning cycle and ensures system safety. A safety layer and exploration constraint mechanism are introduced to ensure that the learning process never triggers dangerous conditions such as motor overheating.

[0008] This invention provides an online optimization method for heat dissipation of motor microchannels based on deep reinforcement learning, comprising the following steps:

[0009] S1: The control problem of the motor microfluidic cooling system is modeled as a Markov decision process. A state space containing multi-dimensional states of motor thermal, electrical, and fluid dynamics is defined, as well as a continuous action space containing pump speed and valve opening. A multi-objective weighted reward function integrating temperature tracking, energy consumption, safety, and control smoothness is also defined. S2: A deep neural network is trained as a policy network using a deep reinforcement learning algorithm based on an actor-critic framework. This policy network maps state vectors to optimal control actions. S3: The policy network is safely pre-trained in a digital twin simulation environment to obtain an initial agent model. S4: The initial agent model is deployed to the embedded controller of the actual motor system. During actual operation, online fine-tuning with safety constraints is performed based on online collected interaction data to achieve adaptive optimization control of the system.

[0010] As one embodiment of the present invention, in step S1, the variables included in the state space include at least six of the following: motor core temperature T_core, core temperature change rate dT_core / dt, load current I_load, motor speed ω_motor, ambient temperature T_ambient, pump power P_pump_prev and coolant flow rate Q_flow_prev of the cooling system at the previous moment, and time since the last cooling t_since_last_cool.

[0011] In one embodiment of the present invention, in step S1, the control command output by the action space is a normalized continuous value, including: the speed setting value of the variable frequency pump Pump_Speed, the inlet proportional valve opening Valve_In, and the outlet proportional valve opening Valve_Out.

[0012] In one embodiment of the present invention, in step S1, the multi-objective weighted reward function r_t is calculated by the following formula: r_t = w1 * R_temperature + w2 * R_energy + w3 * R_safety + w4 * R_smooth; where R_temperature = -α * |T_core - T_optimal|², is the temperature tracking reward, driving the temperature to approach the optimal set value; R_energy = -β * (P_pump + P_valve), is the energy consumption penalty, encouraging the reduction of system power consumption; R_safety = -γ * max(0, T_core - T_safe)³, is the safety penalty, imposing a severe penalty on temperatures exceeding the safety threshold; R_smooth = -δ * |a_t - a_{t-1}|², is the action smoothing reward, suppressing drastic fluctuations in control commands; w1, w2, w3, w4 are adjustable weight coefficients, and α, β, γ, δ are scaling coefficients.

[0013] In one embodiment of the present invention, in step S2, the deep reinforcement learning algorithm is a dual-delay deep deterministic policy gradient algorithm, whose network structure includes: a policy network for generating deterministic actions based on the current state; two commentator networks with identical structures but independent parameters for evaluating the value of state-action pairs and overcoming overestimation of value by taking the minimum value; and a target actor network and a target commentator network corresponding to the above networks, which stabilize the training process through a soft update mechanism.

[0014] As one embodiment of the present invention, in step S3, the digital twin simulation environment is a high-fidelity model constructed based on the "electric-thermal-fluid" multi-physics field coupling principle of the motor, which can simulate the dynamic response between the core temperature of the motor and the microchannel heat dissipation system under different loads, speeds and ambient temperatures.

[0015] In this model, the TD3 algorithm is run for large-scale, secure pre-training until the policy converges. The trained policy network parameters θ^μ are then exported as the initial controller.

[0016] As one embodiment of the present invention, the "online fine-tuning with safety constraints" in step S4 specifically includes:

[0017] S4.1: Run an online learning thread in parallel within the actual controller, using a learning rate much lower than that of the pre-training phase; S4.2: Add an action constraint layer after the output layer of the policy network to map the original actions output by the network to preset safe operating boundaries for pumps and valves; S4.3: Store the system operation data collected online into a fixed-capacity experience playback buffer and periodically sample it to fine-tune the policy network parameters.

[0018] In one specific embodiment of the present invention, online deployment and safe fine-tuning (online learning phase) involve deploying a pre-trained policy network to an embedded controller. When the system is running on an actual motor, a low-learning-rate, strongly constrained online fine-tuning thread is started in parallel. A safe exploration boundary is set: action outputs must pass through a safety filter to ensure they do not exceed preset pump speeds and valve maximum operating ranges. Continuous learning: New data collected online is continuously added to the experience pool, periodically triggering fine-tuning updates, allowing the policy to continuously adapt to slow drifts such as motor aging and changes in coolant performance.

[0019] As one embodiment of the present invention, it also includes a safety fault-tolerant step: real-time monitoring of the output actions of the deep reinforcement learning controller, the key states of the system, and the reward function value; when abnormal action output, sensor data failure, or abnormal sharp drop in reward value is detected, the control system is immediately switched from deep reinforcement learning mode to rule-based backup controller mode.

[0020] Compared with the prior art, the present invention has achieved the following beneficial effects:

[0021] Global Optimum: By maximizing long-term cumulative rewards, it automatically finds the globally optimal balance between heat dissipation efficiency and energy consumption, surpassing any rule-based or PID-based method.

[0022] Fully adaptive: It does not require any precise thermal model of the motor, but learns autonomously through interactive data, making it highly adaptable.

[0023] Proactive control: Based on the state history sequence, it can learn complex dynamics such as thermal inertia, enabling early intervention and avoiding temperature overshoot.

[0024] Highly scalable: This framework can be easily extended to complex scenarios such as multi-motor collaborative heat dissipation and multiple cooling loops.

[0025] Highly efficient and precise heat dissipation: By combining microfluidic structure with high-precision temperature sensing, rapid and directional cooling of the core heat-generating area of ​​the motor is achieved, significantly improving heat dissipation efficiency and effectively suppressing local hot spots. Attached Figure Description

[0026] Figure 1 This is an overview of the architecture of the microfluidic motor cooling system based on deep reinforcement learning in this invention.

[0027] Figure 2 This is a diagram of the overall control architecture of the microfluidic motor cooling system based on deep reinforcement learning in this invention.

[0028] Figure 3 This is a flowchart of the hybrid deployment mode of the microfluidic motor heat dissipation system based on deep reinforcement learning in this invention, which combines "simulation pre-training + online fine-tuning".

[0029] Figure 4 This is a comparison curve of the temperature response of heat dissipation control based on deep reinforcement learning and traditional PID control methods under sudden load changes. Detailed Implementation Plan

[0030] Figure 1 This is an overview of the architecture of the microfluidic motor cooling system based on deep reinforcement learning in this invention. Figure 2 This is a diagram of the overall control architecture of the microfluidic motor cooling system based on deep reinforcement learning in this invention. Figure 3 This is a flowchart of the hybrid deployment mode of the microfluidic motor heat dissipation system based on deep reinforcement learning in this invention, which combines "simulation pre-training + online fine-tuning".

[0031] Example 1: System Modeling and Data Preprocessing;

[0032] First, perform Markov decision process modeling:

[0033] (1) The state space S is a multidimensional observation that can comprehensively reflect the thermal, electrical and fluid states of the motor. s_t = [ T_core (motor core temperature (°C)), dT_core / dt (core temperature change rate (°C / s)), I_load (load current (A)), ω_motor (motor speed (rpm)), T_ambient (ambient temperature (°C)), P_pump_prev (pump power at the previous moment (W)), Q_flow_prev (coolant flow rate at the previous moment (L / min)), t_since_last_cool (time since the last cooling (s)) .

[0034] (2) In action space A, the agent outputs continuous and refined control commands:

[0035] a_t = [ Pump_Speed(Variable frequency pump speed setting value [0, 1]), Valve_In(Inlet proportional valve opening [0, 1]), Valve_Out(Outlet proportional valve opening [0, 1])].

[0036] (3) Reward function R

[0037] The reward function is key to guiding the agent's learning. This invention designs a reward function that balances multiple objectives:

[0038] r_t = w1 * R_temperature + w2 * R_energy + w3 * R_safety + w4 * R_smooth;

[0039] Where: R_temperature = -α * |T_core - T_optimal|² / / Temperature tracking reward;

[0040] R_energy = -β * (P_pump + P_valve) / / Energy penalty;

[0041] R_safety = -γ * max(0, T_core - T_safe)³ / / Safety penalty (over-temperature cubic penalty, severe);

[0042] R_smooth = -δ * |a_t - a_{t-1}|² / / Smooth motion reward;

[0043] By adjusting the weights w1, w2, w3, and w4, different control preferences such as "prioritizing efficient heat dissipation" or "prioritizing extreme energy saving" can be flexibly defined. Figure 1 The definitions of the state s_t, action a_t, and reward r_t of the deep reinforcement learning agent, as shown in the overview of the heat dissipation architecture, are as above.

[0044] Example 2: Core algorithms for deep learning;

[0045] The preferred algorithm is the dual-delay deep deterministic policy gradient (TD3) algorithm. The core reason is that it "adapts to the continuous action space and improves training stability", which solves the problems of "training oscillation and slow convergence" in traditional reinforcement learning in engineering scenarios. Figure 2 The policy network and value assessment network in the overall system control architecture diagram are shown below, and their structures are as follows:

[0046] [Actor Network] μ(s|θ^μ) → Output action a; where a_t=μ(s_t∣θ^μ)+ϵ where ϵ is exploration noise (following Clip Gaussian distribution), used to balance exploration and exploitation.

[0047] [Critic Network] Q(s, a|θ^Q) → Evaluate the Q-value; Input state s_t and action a_t, output action value Q, two parallel network parameters are...

[0048]

[0049] [Target Network] Actor', Critic1', Critic2' → Stable training; Target policy network μ′ and target Q networks Q1′ and Q2′, with parameters θ^μ′, θ^Q1′, and θ^Q2′ respectively, are used for stable training.

[0050] Training process:

[0051] 1. Data Collection: The agent performs actions in the environment, storing and transferring (s_t, a_t, r_t, s_{t+1}) to the experience pool. That is... Figure 1 , Figure 2 , Figure 3 The process referred to by the experience replay buffer module.

[0052]

[0053] Where γ∈[0,1] is the discount factor, balancing immediate reward and long-term reward; at+1′=μ′(st+1∣θ^μ′)+ϵ′, ϵ′ is the target action noise (used to smooth the target value); the min operation takes the minimum value of the output of the two target Q network to alleviate overestimation bias.

[0054] 2. Sampling Update: Randomly sample small batches of data from the experience pool.

[0055] 3. Critic Update: The two Critic networks are updated by minimizing the temporal difference error. The Critic network updates its parameters by minimizing the temporal difference error, and the loss function is:

[0056]

[0057] Where N is the batch sampling data volume, (si, ai, ri, si+1) are the sampled samples in the experience pool, and yi is the target Q value of the corresponding sample.

[0058] 4. Actor Update: The Actor network is updated using the policy gradient ascent method to maximize the expected reward. The Actor network updates its parameters by maximizing the expected Q-value. The objective function is:

[0059]

[0060] Gradient ascent update formula (gradient calculated via backpropagation):

[0061]

[0062] Where J is the target profit of the Actor network.

[0063] 5. Target Network Soft Update: To ensure training stability, the target network parameters adopt a soft update strategy. The update formula is as follows:

[0064]

[0065] Where τ∈(0,1] is the soft update coefficient (usually τ=0.005), which realizes the slow iteration of the target network parameters.

[0066] Example 3: Simulation environment setup and offline pre-training;

[0067] To address the issues of high safety risks, high debugging costs, and long training cycles associated with direct training of real motor systems, this invention first constructs a high-fidelity simulation environment for offline pre-training, and then adapts it to the real-world scenario through online fine-tuning. Figure 3 The second step in the flowchart involves running the TD3 algorithm pre-training in the simulation environment. The specific steps are as follows:

[0068] (1) Construction of multiphysics simulation model

[0069] Based on finite element analysis (FEA) and computational fluid dynamics (CFD), a multiphysics coupled simulation model of the motor heat dissipation system is built: thermal simulation submodule: input motor load current and speed, calculate heat generation power based on copper loss / iron loss formula, and simulate the spatiotemporal distribution and rate of change (dT_core / dt) of core temperature (T_core) by combining motor body material parameters (thermal conductivity, specific heat capacity).

[0070] Fluid simulation submodule: Input pump speed and valve opening to simulate the flow rate (Q_flow) and pressure loss of the heat dissipation medium (such as coolant), as well as the convective heat transfer coefficient between the medium and the motor heat exchange surface;

[0071] Energy consumption simulation submodule: Based on the efficiency curves of variable frequency pumps and proportional valves, calculate pump power (P_pump) and valve energy consumption (P_valve) under different action commands.

[0072] Environmental Coupling Submodule: Supports customizable external interference parameters such as ambient temperature (T_ambient) and wind speed to simulate heat dissipation effects under different operating conditions.

[0073] (2) Offline pre-training strategy

[0074] 1. Operating Condition Library Construction: Covering the entire operating range of the motor, generating over 100,000 typical operating condition samples, including:

[0075] Steady-state operating conditions: rated load (50A), rated speed (3000rpm), ambient temperature (25℃);

[0076] Transient operating conditions: sudden increase in load (10A→80A), sudden change in speed (1000rpm→5000rpm), high temperature environment (45℃);

[0077] Extreme operating conditions: approaching the safe temperature threshold (T_safe=120℃), long-term low-flow cooling (Q_flow_prev=5L / min).

[0078] 2. Pre-training process:

[0079] Deploy the TD3 algorithm to the simulation environment and set the initial weights (w1=0.4, w2=0.3, w3=0.2, w4=0.1, prioritizing temperature control + energy balance).

[0080] The experience pool capacity is set to 1 million, the sampling batch size is 64, the learning rate is 0.001, and the target network soft update coefficient τ=0.005;

[0081] The training process involves 500,000 iterations, with the control effect of key operating conditions verified every 10,000 steps. When the temperature deviation is ≤2℃, energy consumption is reduced by ≥15%, and there is no risk of overheating, the pre-training is stopped and the pre-trained model parameters are saved.

[0082] Example 4: Online Deployment and Adaptive Fine-tuning;

[0083] After the pre-trained model is completed, it is deployed to the embedded hardware of the motor control system (such as ARM Cortex-A72), and the model deviation is adapted to the real system through online fine-tuning. Figure 3 The execution flow following the judgment process shown in the flowchart of the hybrid deployment mode of the heat dissipation system "simulation pre-training + online fine-tuning" is as follows:

[0084] (1) Hardware deployment optimization

[0085] Model lightweighting: The Actor / Critic network of TD3 is pruned (removing redundant neurons) and quantized (FP32→FP16), reducing the number of model parameters from 1.2 million to 300,000 and the inference latency from 20ms to 5ms, meeting the real-time control requirements (control cycle 10ms).

[0086] Data acquisition module: integrates temperature sensor (accuracy ±0.5℃), current sensor, speed encoder, and flow sensor, and acquires state vector s_t in real time with a sampling frequency of 100Hz;

[0087] Execution drive module: Converts the continuous action values ​​(0~1) output by the Actor network into control signals for the actuator (such as pump speed 0.6→1800rpm, valve opening 0.8→80%), driving the variable frequency pump and proportional valve to perform the operation.

[0088] (2) Online adaptive fine-tuning

[0089] Decay rate of online fine-tuning learning:

[0090]

[0091] ηt is the online fine-tuning learning rate at time t; η0 is the initial learning rate (usually η0 = 10^−5).

[0092] λ is the decay coefficient (usually λ=10^−4); t is the number of online running steps, realizing the learning rate decays slowly over time, balancing adaptability and stability; Deviation detection: real-time comparison between "the real T_core collected by the sensor" and "the T_core predicted by the simulation model", when the deviation is ≥3℃ for 5 consecutive seconds, the fine-tuning mechanism is triggered;

[0093] Incremental learning: Only effective experience (s_t, a_t, r_t, s_{t+1}) from real systems is collected and stored in an "online experience pool" (capacity 10,000) to avoid massive amounts of invalid data; "small batch incremental update" is adopted: 32 real data points are sampled each time and mixed with the data in the pre-trained experience pool at a ratio of 1:4 to update the Actor / Critic network parameters, and the learning rate is reduced to 0.0001 to reduce oscillations; Adaptive weight adjustment: Fuzzy logic is introduced to dynamically adjust w1~w4 according to real-time operating conditions: when T_core ≥ 110℃ (close to the safety threshold): w3 is increased to 0.4 and w1 is increased to 0.3, prioritizing safety; when the ambient temperature ≤ 15℃ (low heat dissipation requirements): w2 is increased to 0.4 and w1 is reduced to 0.2, prioritizing energy saving. Figure 4 This is a comparison curve of the temperature response of heat dissipation control based on deep reinforcement learning and traditional PID control methods under sudden load changes.

Claims

1. A method for online optimization of heat dissipation in motor microchannels based on deep reinforcement learning, characterized in that, Includes the following steps: S1: The control problem of the motor microfluidic cooling system is modeled as a Markov decision process. A state space containing multi-dimensional states of motor thermal, electrical, and fluid dynamics is defined, along with a continuous action space containing pump speed and valve opening. A multi-objective weighted reward function integrating temperature tracking, energy consumption, safety, and control smoothness is also defined. S2: A deep reinforcement learning algorithm based on an actor-critic framework is used to train a deep neural network as a policy network, which maps state vectors to optimal control actions. S3: The policy network is safely pre-trained in a digital twin simulation environment to obtain an initial agent model. S4: The initial agent model is deployed to the embedded controller of the actual motor system, and online fine-tuning with safety constraints is performed based on online collected interaction data during actual operation to achieve adaptive optimization control of the system.

2. The online optimization method for heat dissipation of motor microchannels based on deep reinforcement learning according to claim 1, characterized in that, In step S1, the variables included in the state space include at least six of the following: motor core temperature T_core, core temperature change rate dT_core / dt, load current I_load, motor speed ω_motor, ambient temperature T_ambient, pump power P_pump_prev and coolant flow rate Q_flow_prev of the cooling system at the previous moment, and time since the last cooling t_since_last_cool.

3. The online optimization method for heat dissipation of motor microchannels based on deep reinforcement learning according to claim 1, characterized in that, In step S1, the control commands output by the action space are normalized continuous values, including: the speed setting value of the variable frequency pump Pump_Speed, the inlet proportional valve opening Valve_In, and the outlet proportional valve opening Valve_Out.

4. The online optimization method for heat dissipation of motor microchannels based on deep reinforcement learning according to claim 1, characterized in that, In step S1, the multi-objective weighted reward function r_t is calculated by the following formula: r_t = w1 * R_temperature + w2 * R_energy + w3 * R_safety + w4 * R_smooth; where R_temperature = -α * |T_core - T_optimal|², is the temperature tracking reward, driving the temperature to approach the optimal set value; R_energy = -β * (P_pump + P_valve), is the energy consumption penalty, encouraging the reduction of system power consumption; R_safety = -γ * max(0, T_core - T_safe)³, is the safety penalty, imposing a severe penalty on temperatures exceeding the safety threshold; R_smooth = -δ * |a_t - a_{t-1}|², is the action smoothing reward, suppressing drastic fluctuations in control commands; w1, w2, w3, w4 are adjustable weight coefficients, and α, β, γ, δ are scaling coefficients.

5. The online optimization method for heat dissipation of motor microchannels based on deep reinforcement learning according to claim 1, characterized in that, In step S2, the deep reinforcement learning algorithm is a dual-delay deep deterministic policy gradient algorithm, whose network structure includes: a policy network for generating deterministic actions based on the current state; two commentator networks with identical structures but independent parameters for evaluating the value of state-action pairs and overcoming overestimation of value by taking the minimum value; and a target actor network and a target commentator network corresponding to the above networks, which stabilize the training process through a soft update mechanism.

6. The online optimization method for heat dissipation of motor microchannels based on deep reinforcement learning according to claim 1, characterized in that, The digital twin simulation environment is a high-fidelity model built on the principle of multi-physics coupling of "electricity-heat-fluid" in motors, which can simulate the dynamic response between the core temperature of the motor and the microchannel heat dissipation system under different loads, speeds and ambient temperatures.

7. The online optimization method for heat dissipation of motor microchannels based on deep reinforcement learning according to claim 1, characterized in that, The "online fine-tuning with safety constraints" mentioned in step S4 specifically includes: S4.1: running an online learning thread in parallel in the actual controller, with the thread using a learning rate much lower than that in the pre-training stage; S4.2: adding an action constraint layer after the output layer of the policy network to map the original actions output by the network to the preset safe operating boundaries of the pump and valve; S4.3: storing the system operation data collected online into a fixed-capacity experience playback buffer, and periodically sampling to fine-tune the policy network parameters.

8. The online optimization method for heat dissipation of motor microchannels based on deep reinforcement learning according to claim 1, characterized in that, It also includes safety and fault-tolerant steps: real-time monitoring of the output actions of the deep reinforcement learning controller, the critical state of the system, and the reward function value; when abnormal action output, sensor data failure, or abnormally sharp drop in reward value is detected, the control system is immediately switched from deep reinforcement learning mode to rule-based backup controller mode.

Images