Satellite plate thermal control method based on deep reinforcement learning
By applying a deep reinforcement learning model in the temperature control of satellite boards, an optimized temperature control strategy is generated, which solves the problems of uneven temperature control and high-frequency vibration in the existing technology, and achieves stable and accurate temperature control.
Patent Information
- Application Number
- CN202411950840.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-05-02
AI Technical Summary
The existing satellite plate temperature control strategy can easily lead to high-frequency vibration and uneven heat from the plate in complex scenarios, affecting the stability and service life of the components.
Using a deep reinforcement learning method, a model including Actor network and Critic network is established, a temperature control strategy is generated through temperature sensor information, and the opening and closing of the heat dissipation element is optimized to achieve stable temperature control.
It significantly reduces material consumption and implementation difficulty, can achieve stable and precise temperature control with less heat dissipation resources, and extends the service life of the components.
Smart Images
Figure CN119916861A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of satellite temperature control technology, and in particular to a satellite panel thermal control method based on deep reinforcement learning. Background Art
[0002] As the importance of space power grows, aerospace technology occupies an increasingly important position in my country's strategic layout. When the heating element of a satellite is working, the temperature of the board where the element is located will continue to rise because there is no heat convection cooling assistance in a vacuum state. When the temperature is higher than a certain level, it will have an adverse effect on the accuracy and stability of the element, as well as the service life of the element.
[0003] Existing simple temperature control strategies often set corresponding thresholds based on human experience to control the opening and closing of the heat sink. However, when applied to more complex and diverse actual application scenarios, they are prone to high-frequency vibrations and uneven heating of panels. Summary of the invention
[0004] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides a satellite panel thermal control method based on deep reinforcement learning.
[0005] The technical solution of the present invention is as follows:
[0006] A satellite panel thermal control method based on deep reinforcement learning is provided, the method comprising:
[0007] Establish a satellite panel temperature field simulation model, including the panel and the heating elements, cooling elements and temperature sensors arranged on it;
[0008] Construct a deep reinforcement learning model including an Actor network and a Critic network, wherein the Actor network is used to generate a temperature control strategy based on the information of the temperature sensor to control the opening and closing of the heat dissipation element, and the Critic network is used to evaluate the impact of the temperature control strategy on the temperature control effect;
[0009] Use the policy gradient optimization algorithm to train the deep reinforcement learning model in a simulation environment to obtain the optimal temperature control strategy;
[0010] The trained deep reinforcement learning model is applied to the actual satellite panel thermal control system to control the opening and closing of the cooling elements and achieve stable temperature control.
[0011] In one embodiment of the present invention, both the Actor network and the Critic network are multi-layer fully connected neural network architectures.
[0012] In one embodiment of the present invention, the policy gradient optimization algorithm adopts the Proximal Policy Optimization algorithm.
[0013] In one embodiment of the present invention, a Proximal Policy Optimization algorithm is used to train a deep reinforcement learning model, including:
[0014] Initialization: Initialize the Actor network and Critic network, and set training parameters;
[0015] Data collection: Perform the following steps in the simulation environment:
[0016] Status acquisition: Get the current status Status from the environment Environment;
[0017] Action generation: Use the Actor network to generate action Actors based on the current state to control the opening and closing of the cooling element;
[0018] Action execution: Apply the action to the environment and control the panel temperature by opening and closing the heat dissipation element;
[0019] Reward acquisition: Get rewards from actions in the environment;
[0020] Status update: Update the environment status to prepare for the next iteration;
[0021] Model training: Use the collected data to train the Actor network and Critic network:
[0022] Critic network training: Use state, action, and reward data to train the Critic network and learn to evaluate actions
[0023] Value;
[0024] Actor network training: Use the state, action, reward, and action value data provided by the Critic network to train the Actor network and learn to generate better action strategies;
[0025] Iteration: Repeat the data collection and model training steps until the preset training objectives or convergence conditions are achieved.
[0026] In one embodiment of the present invention, the temperature control strategy includes controlling the switching state and switching time of the heat dissipation element.
[0027] In one embodiment of the present invention, the heat dissipation element is a heat pipe heat sink or a semiconductor heat sink.
[0028] In one embodiment of the present invention, there are multiple heat dissipation elements, and the multiple heat dissipation elements are evenly arranged on the plate in multiple rows and columns.
[0029] In one embodiment of the present invention, there are multiple temperature sensors, and the multiple temperature sensors are evenly arranged on the plate in multiple rows and columns.
[0030] In one embodiment of the present invention, the temperature sensor is arranged at a key position of the plate, and the key position is an area where high temperature frequently occurs.
[0031] In one embodiment of the present invention, the satellite panel temperature field simulation model is solved by a second-order implicit format of the heat diffusion equation.
[0032] The main advantages of the technical solution of the present invention are as follows:
[0033] The satellite panel thermal control method based on deep reinforcement learning of the present invention uses an agent to learn the temperature control strategy in a simulation environment, which significantly reduces material consumption and reduces the difficulty of implementation. In the simulation environment, the deep learning reinforcement model is trained using policy gradient optimization, and the neural network parameters therein are adjusted. When the trained deep learning reinforcement model is applied to the thermal control of satellite panels, stable and accurate temperature control can be achieved with less heat dissipation resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0035] Figure 1 It is a flow chart of a satellite panel thermal control method based on deep reinforcement learning according to an embodiment of the present invention;
[0036] Figure 2 It is a schematic diagram of a satellite panel temperature field simulation model in a satellite panel thermal control method based on deep reinforcement learning according to an embodiment of the present invention;
[0037] Figure 3 A schematic diagram of the structure of the Actor network and the Critic network in a satellite panel thermal control method based on deep reinforcement learning according to an embodiment of the present invention;
[0038] Figure 4 This is a schematic diagram of model training based on the PPO strategy in a satellite panel thermal control method based on deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0040] The technical solution provided by the embodiments of the present invention is described in detail below with reference to the accompanying drawings.
[0041] The embodiment of the present invention provides a satellite panel thermal control method based on deep reinforcement learning, as shown in the attached Figure 1 As shown, including:
[0042] S1, establish a satellite panel temperature field simulation model, including the panel and the heating elements, heat dissipation elements and temperature sensors arranged thereon.
[0043] This step constructs a satellite panel temperature field simulation model, which facilitates the subsequent simulation of the panel heating and cooling process in the simulation environment, significantly reduces material consumption, is easier to implement, and is easier to operate. The layout of the panels in the simulation model can be set according to the actual working conditions, production environment, or the structure of the satellite panel to be controlled.
[0044] S2, build a deep reinforcement learning model including Actor network and Critic network, where the Actor network is used to generate a temperature control strategy based on the information of the temperature sensor to control the opening and closing of the cooling element, and the Critic network is used to evaluate the impact of the temperature control strategy on the temperature control effect.
[0045] Introducing a deep reinforcement learning model to control panel temperature can fully utilize the advantages of reinforcement learning and achieve more accurate temperature control.
[0046] S3, uses the policy gradient optimization algorithm to train the deep reinforcement learning model in a simulation environment to obtain the optimal temperature control strategy.
[0047] It is understandable that the deep learning model needs to be pre-trained before use. This step uses the policy gradient optimization algorithm PPO (Proximal Policy Optimization) to train the deep reinforcement learning model in a simulation environment, so that the reinforcement learning model can learn the impact of different temperature control strategies on the temperature control effect, and finally obtain the best temperature control strategy. As a reinforcement learning optimization algorithm, PPO can solve the sample efficiency problem in the policy gradient method while maintaining the simplicity and easy implementation of the algorithm. The trained model can select the appropriate temperature control strategy according to the actual working conditions to achieve precise control of the temperature of the satellite panels.
[0048] S4, applies the trained deep reinforcement learning model to the actual satellite panel thermal control system to control the opening and closing of the cooling elements to achieve stable temperature control.
[0049] The satellite panel thermal control method based on deep reinforcement learning provided by the embodiment of the present invention uses an agent to learn the temperature control strategy in a simulation environment, which significantly reduces material consumption and reduces the difficulty of implementation. In the simulation environment, the deep learning reinforcement model is trained using policy gradient optimization, and the neural network parameters are adjusted. When the trained deep learning reinforcement model is applied to the thermal control of satellite panels, stable and accurate temperature control can be achieved with less heat dissipation resources.
[0050] The following is a detailed description of the steps and principles involved in the satellite panel thermal control method based on deep reinforcement learning provided in an embodiment of the present invention.
[0051] In the temperature field of the board, there is first a basic plate. Its total power is 0 for the entire temperature field, and it only plays the role of transferring heat. On the basis of the plate, there are heating elements and heat dissipation elements. The heating element is a functional element, and the goal of heat dissipation control is to keep it at a suitable temperature. The heat dissipation element is a controlled element, which generates negative power when working and absorbs the heat generated by the heating element to achieve the purpose of heat dissipation.
[0052] The heat conduction differential equation in the temperature field is:
[0053]
[0054] Since the problem to be solved is a discrete problem, we can differentiate it and get:
[0055]
[0056] And let:
[0057]
[0058] The representation in the matrix is:
[0059] u x,y (t n+1 )=A×u xy (t n )
[0060] The letters in the above formula mean: ρ: density; c: specific heat capacity; k: thermal conductivity; t: time step; x: x-axis coordinate, y: y-axis coordinate; u: temperature; r x r y : Discretized thermal conductivity.
[0061] (I) Satellite panel modeling and simulation
[0062] Based on the above heat diffusion equation, the definitions of heating elements, temperature sensors, heat dissipation elements and other components are added to build a simulation environment. The layout of the board is as follows Figure 2 As shown. Among them, the sizes of the rectangles representing the heating elements are different, and the size, shape and position of the heating elements are set according to the actual working conditions, and multiple heating elements are scattered at different positions on the plate. In one embodiment, the heat dissipation elements are evenly arranged on the plate in multiple rows and columns. Similarly, the temperature sensors are evenly arranged on the plate in multiple rows and columns. In one embodiment, the heat dissipation elements and temperature sensors can be irregularly arranged on the plate, showing a distribution method with more key positions and fewer non-key positions. Among them, the key position is the area where the heat dissipation elements are located and high temperature occurs frequently. In this way, key monitoring and key heat dissipation of key positions can be achieved. In the embodiment of the present invention, the positions of the heat dissipation elements, the heating elements and the temperature sensors are not specifically limited, and those skilled in the art can flexibly adjust them according to actual needs.
[0063] In one embodiment, the heat dissipation element may be a heat pipe heat sink or a semiconductor heat sink, or may be a heat sink in other forms, which is not specifically limited in the embodiment of the present invention.
[0064] In one embodiment, the satellite panel temperature field simulation model is solved by using a second-order implicit format of the heat diffusion equation.
[0065] (II) Building a network model
[0066] Construct a deep reinforcement model including Actor network and Critic network. The shapes of Actor network and Critic network are the same, both of which are multi-layer perceptron MLP structures. See Appendix Figure 3 , the multilayer perceptron network consists of an input layer, a hidden layer, and an output layer. The representation of each layer is as follows:
[0067] Input Layer:
[0068] Hidden Layer 1:
[0069] Hidden Layer 2:
[0070] Output layer:
[0071] The letters in the above formula have the following meanings:
[0072] x: initial input information;
[0073] N i : The i-th layer of neural network;
[0074] R N : state space with dimension N;
[0075] d in : The number of state space dimensions of the input information;
[0076] d out : The number of state space dimensions of the output information;
[0077] σ: activation function;
[0078] W i : The weight matrix of the i-th layer neural network;
[0079] b i : The bias matrix of the i-th layer neural network.
[0080] Among them, the Actor network can output the cooling element switch signal according to the input measurement point temperature information, that is, the start and stop of the cooling element in the next period of time is determined according to the real-time temperature of the measurement point. The Critic network can output the action value according to the input environmental information.
[0081] (III) Training model parameters
[0082] A schematic diagram of using the policy gradient optimization algorithm to train a deep reinforcement learning model in a simulation environment can be found in the attached Figure 4 ,include:
[0083] S31, Initialization: Initialize the Actor network and Critic network, and set training parameters.
[0084] S32, Data Collection: Perform the following steps in the simulation environment:
[0085] S321, status acquisition: obtain the current status Status from the environment Environment;
[0086] S322, action generation: using the Actor network to generate action Actors according to the current state to control the opening and closing of the cooling element;
[0087] S323, action execution: applying the action to the environment, and controlling the temperature of the panel by opening and closing the heat dissipation element;
[0088] S324, reward acquisition: obtaining the reward brought by the action from the environment;
[0089] S325, status update: Update the environment status to prepare for the next iteration.
[0090] S33, model training: Use the collected data to train the Actor network and Critic network:
[0091] S331, Critic network training: Use state, action and reward data to train the Critic network and learn to evaluate the action value;
[0092] S332, Actor network training: Use the state, action, reward and action value data provided by the Critic network to train the Actor network, update the network parameters in the direction of larger reward values, and learn to generate better action strategies;
[0093] S34, repeated iteration: Repeat the data collection and model training steps until the preset training purpose or convergence condition is achieved.
[0094] (IV) Use Model
[0095] Use Pytorch to build a deep reinforcement learning model and read the previously stored neural network parameters. Use it to control the switch of the heat dissipation element when running in the simulation environment. Compared with the benchmark method, the thermal control effect obtained by this method can control the temperature more stably, so that the loss of the chip caused by thermal stress is reduced. In one embodiment, the total temperature difference can be reduced by 25%.
[0096] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, "front", "back", "left", "right", "upper" and "lower" in this article are all referenced to the placement state shown in the accompanying drawings.
[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A satellite panel thermal control method based on deep reinforcement learning, characterized in that: include: Establish a satellite panel temperature field simulation model, including the panel and the heating elements, cooling elements and temperature sensors arranged on it; Construct a deep reinforcement learning model including an Actor network and a Critic network, wherein the Actor network is used to generate a temperature control strategy based on the information of the temperature sensor to control the opening and closing of the heat dissipation element, and the Critic network is used to evaluate the impact of the temperature control strategy on the temperature control effect; Use the policy gradient optimization algorithm to train the deep reinforcement learning model in a simulation environment to obtain the optimal temperature control strategy; The trained deep reinforcement learning model is applied to the actual satellite panel thermal control system to control the opening and closing of the cooling elements and achieve stable temperature control.
2. The satellite panel thermal control method based on deep reinforcement learning according to claim 1 is characterized in that: The Actor network and the Critic network are both multi-layer fully connected neural network architectures.
3. The satellite panel thermal control method based on deep reinforcement learning according to claim 2 is characterized in that: The policy gradient optimization algorithm adopts the Proximal Policy Optimization algorithm.
4. The satellite panel thermal control method based on deep reinforcement learning according to claim 3 is characterized in that: The Proximal Policy Optimization algorithm is used to train the deep reinforcement learning model, including: Initialization: Initialize the Actor network and Critic network, and set training parameters; Data collection: Perform the following steps in the simulation environment: Status acquisition: Get the current status Status from the environment Environment; Action generation: Use the Actor network to generate action Actors based on the current state to control the opening and closing of the cooling element; Action execution: Apply the action to the environment and control the panel temperature by opening and closing the heat dissipation element; Reward acquisition: Get rewards from actions in the environment; Status update: Update the environment status to prepare for the next iteration; Model training: Use the collected data to train the Actor network and Critic network: Critic network training: Use state, action, and reward data to train the Critic network and learn to evaluate the action value; Actor network training: Use the state, action, reward, and action value data provided by the Critic network to train the Actor network and learn to generate better action strategies; Iteration: Repeat the data collection and model training steps until the preset training objectives or convergence conditions are achieved.
5. The satellite panel thermal control method based on deep reinforcement learning according to claim 4 is characterized in that: The temperature control strategy includes controlling the switching state and switching time of the heat dissipation element.
6. The satellite panel thermal control method based on deep reinforcement learning according to claim 1, characterized in that: The heat dissipation element is a heat pipe heat sink or a semiconductor heat sink.
7. The satellite panel thermal control method based on deep reinforcement learning according to claim 1, characterized in that: There are multiple heat dissipation elements, and the multiple heat dissipation elements are evenly arranged on the plate in multiple rows and columns.
8. The satellite panel thermal control method based on deep reinforcement learning according to claim 1, characterized in that: There are multiple temperature sensors, and the multiple temperature sensors are evenly arranged on the plate in multiple rows and columns.
9. The satellite panel thermal control method based on deep reinforcement learning according to claim 1, characterized in that: The temperature sensor is arranged at a key position of the plate, and the key position is an area where high temperature frequently occurs.
10. The satellite panel thermal control method based on deep reinforcement learning according to claim 1, characterized in that: The satellite panel temperature field simulation model is solved by the second-order implicit format of the heat diffusion equation.