DDPG-based spacecraft storage battery system charging and discharging optimization control method

Abstract

The invention provides a spacecraft storage battery system charging and discharging optimization control method based on a DDPG, and the method comprises the steps: constructing a battery state model in a spacecraft power system, and obtaining an input current, a temperature and a load voltage; estimating the key unmeasurable state of the battery by adopting an extended Kalman filter; the method comprises the following steps: establishing an agent based on a DDPG algorithm, introducing a target network to periodically update an Actor network and a Critic network, and introducing an experience playback mechanism to train the agent based on the DDPG algorithm, performing multi-objective optimization on an intelligent agent based on a DDPG algorithm; training the optimized agent based on the DDPG algorithm by using the key unmeasurable state of the battery; inputting the input current, the temperature, the load voltage, the estimated state of charge (SOC) and the estimated state of health (SOH) into the trained and optimized DDPG algorithm-based intelligent agent to obtain charging / discharging current. The method provided by the invention not only improves the use efficiency of the battery, but also ensures the stable operation of the spacecraft under various conditions.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control of a spacecraft power system, and in particular to a DDPG-based charge and discharge optimization control method for a spacecraft battery system. Background Art

[0002] With the rapid development of aerospace technology, the reliability and efficiency of spacecraft power systems have become particularly important. When performing missions, spacecraft require a stable power supply to support the normal operation of various instruments and equipment. Traditional spacecraft power systems usually use a periodic sampling method to monitor and control the power status. This method samples battery parameters such as voltage, current, and temperature at fixed time intervals, and then adjusts the charging and discharging strategy based on these sampled data.

[0003] However, the periodic sampling method has some obvious shortcomings. First, this method does not take into account the actual system operating status and environmental changes, resulting in sampling even when frequent updates are not needed, resulting in a waste of resources. In modern spacecraft power systems, batteries are key energy storage units, and their state of health (SOH) and remaining useful life (RUL) have a direct impact on the safety and performance of the entire system. The performance of the battery will gradually decline with the increase of usage time and the number of charge and discharge cycles. Therefore, accurately estimating the battery's SoH and predicting the RUL are crucial to formulating effective charge and discharge strategies. However, traditional periodic sampling methods often ignore this dynamic change in battery performance and cannot provide sufficient information to achieve battery health management and life prediction.

[0004] In addition, when performing missions, spacecraft may encounter various extreme environments, such as temperature changes, radiation effects, etc., which may affect the performance and life of the battery. Therefore, an intelligent control system that can respond to environmental changes in real time and optimize the charging and discharging strategy is needed to improve the adaptability and reliability of the spacecraft power system. Summary of the invention

[0005] The purpose of the present invention is to provide a DDPG-based spacecraft battery system charge and discharge optimization control method. This method can not only improve energy utilization efficiency and extend battery life by learning and adjusting the charge and discharge strategy in real time, but also adapt to the complex space environment and ensure the stable operation of the spacecraft power system.

[0006] To achieve the above object, the present invention provides a DDPG-based spacecraft battery system charge and discharge optimization control method, comprising the following steps:

[0007] Construct a battery state model in the spacecraft power system, where the battery model is a second-order RC model, and obtain the input current, temperature, and load voltage;

[0008] An extended Kalman filter is used to estimate the key unmeasurable states of the battery, where the key unmeasurable states of the battery include the state of charge (SOC) and the state of health (SOH);

[0009] Create an intelligent agent based on the DDPG algorithm. The intelligent agent includes an actor network Actor and a critic network Critic. Then introduce a target network to regularly update the Actor network and the Critic network. Introduce an experience replay mechanism to design a training process for the intelligent agent based on the DDPG algorithm.

[0010] Design loss functions to achieve multi-objective optimization of the DDPG-based agent, where the multi-objectives include ensuring load satisfaction, controlling battery temperature, and extending battery life;

[0011] Interact with the environment and use the training process to train the DDPG-based agent so that it can output the best strategy based on the measured state and the estimated key unmeasurable state of the battery;

[0012] The input current, temperature, load voltage, estimated state of charge SOC and state of health SOH are input into the trained and optimized DDPG algorithm-based agent to obtain the charge / discharge current.

[0013] Preferably, the SOC estimate is updated based on the charge accumulation of the battery, calculated by current integration:

[0014]

[0015] Among them, Q nom is the rated capacity of the battery, t k represents the kth time point in a discrete time series; ΔQ represents the change in battery charge from time step k-1 to time step k; SOC(k) represents the state of charge of the battery at time step k;

[0016] For SOH estimation, based on the change of battery internal resistance:

[0017]

[0018] In the formula, R 0,new is the initial internal resistance of a new battery, R 0,EOL is the internal resistance at the end of the battery life, and R0 is the current internal resistance value.

[0019] Preferably, the agent based on the DDPG algorithm specifically includes

[0020] The Actor network is responsible for proposing action suggestions based on the current state. Its output action is generated by the state through a specific neural network structure. The forward propagation of the Actor network is expressed by the formula

[0021]

[0022] Represented as follows, where a is the action, s is the state, and θ μ is the parameter of the Actor network, μ is the Actor network;

[0023] The Critic network evaluates the action value under the current strategy, inputs the current state and action, and outputs a Q value, which is the expected utility of the state-action pair. The forward propagation is expressed as

[0024]

[0025] Among them, θ Q and θ′ Q are the parameters of the Critic network.

[0026] Preferably, a target network is introduced into the agent based on the DDPG algorithm, i.e., a slowly updated version of the Actor and Critic. The parameters of the target network are regularly updated through a soft update strategy, which is

[0027] θ′ μ ←τθ μ +(1-τ)θ′ μ

[0028] θ′ Q ←τθ Q +(1-τ)θ′ Q

[0029] Among them, τ is the learning rate, which controls the speed of updating the target network parameters; θ′ μ Parameters for the strategy actor.

[0030] Preferably, the agent based on the DDPG algorithm also introduces an experience replay mechanism, which stores the experience data samples generated by the interaction between the Actor and the environment into the experience pool, and randomly extracts batch data samples from the experience pool for training;

[0031] During the training process, the Critic network is first trained to evaluate the current strategy, with the goal of minimizing the mean square error loss between the target Q value and the current Q value, which is

[0032]

[0033] In the formula, y i is the target Q value, is the current Q value, s i is the state at time step i; a i is the action at time step i; N is the number of samples, which means the total number of samples extracted from the experience replay buffer;

[0034] Next, the Actor network updates its strategy based on the evaluation of the Critic network, with the goal of maximizing the expected Q value. Its strategy gradient is updated as

[0035]

[0036] In the formula, is the strategy function J with respect to the Actor network parameter θ μ The policy function J is represented as the predicted return obtained under a given policy, N is the number of samples drawn from the experience replay buffer, It refers to the gradient of the Q-value function output by the Critic network with respect to action a, which means that in state s i Take action a i The rate of change of Q value when . Similarly, is the Actor network with respect to its parameters θ μ The gradient of . It means that in state s i In the following example, we show how the actions output by an Actor network change as its parameters change.

[0037] Preferably, multi-objective optimization is performed through the Actor-Critic structure of the DDPG algorithm:

[0038]

[0039] Among them, C load , C temp , C life They represent the metrics of load satisfaction, battery temperature control and battery life extension respectively, while ω1, ω2 and ω3 are the corresponding weight coefficients.

[0040] Therefore, the present invention adopts the above-mentioned DDPG-based spacecraft battery system charge and discharge optimization control method, and the technical effects are as follows:

[0041] The present invention can realize a more intelligent charging and discharging control strategy, dynamically adjust the charging and discharging process according to the change of solar current and the actual demand of the load, thereby significantly improving the energy utilization efficiency and reducing the ineffective charging and discharging caused by the mismatch between supply and demand.

[0042] In addition, the present invention effectively delays the battery aging process and extends its service life by real-time monitoring of the battery's health status and performing optimization control.

[0043] At the same time, the real-time learning and adjustment capabilities of the intelligent agent enable the system to adapt to the complex space environment and meet dynamically changing mission requirements, enhancing the safety and reliability of the system. The application of the DDPG algorithm further enhances the intelligence level of charge and discharge control, and can dynamically adjust the strategy according to real-time data to adapt to different mission requirements and environmental conditions.

[0044] In summary, the present invention not only optimizes the charging and discharging process of the spacecraft power system and improves the system performance, but also improves the efficiency and life of the battery. It has important practical value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Diagram of the different phases of a spacecraft in an orbital cycle.

[0046] Figure 2 The figure is a flow chart of the algorithm of the present invention.

[0047] Figure 3 This is the training result diagram. DETAILED DESCRIPTION

[0048] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0049] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0050] Embodiment 1

[0051] The present invention provides a DDPG-based spacecraft battery system charge and discharge optimization control method, comprising the following steps:

[0052] Step 1: Construct a battery state model in the spacecraft power system, where the battery model is a second-order RC model, and obtain the input current, temperature, and load voltage;

[0053] When modeling the battery state in a spacecraft power system, a comprehensive approach was used to ensure that the model can adapt to the spacecraft's operating cycle in orbit and the effects of changes in solar radiation. Figure 1 As shown, this process first involves orbit determination, which is achieved by calculating the orbital inclination and period, ensuring that the battery model can adapt to changes in solar radiation due to the rotation and revolution of the Earth.

[0054] Orbital inclination i s It is determined by the following formula:

[0055]

[0056] in, is the rate of change of orbital inclination, h is the orbital altitude, and RE is the radius of the Earth.

[0057] The orbital period T is given by Kepler's third law:

[0058]

[0059] Where T is the orbital period, G is the gravitational constant, and M is the mass of the Earth.

[0060] Solar radiation has a direct impact on battery performance. We simulate its impact on battery charging and discharging behavior by considering the change in solar radiation intensity. The solar radiation intensity S can be expressed as follows:

[0061] S=S0·cos(θ)

[0062] Where S0 is the solar constant (about 1367W / m 2 ), θ is the solar incidence angle.

[0063] Solar radiation acts on solar panels to generate photocurrent, which acts on the spacecraft power system through a series of processes. The formula is as follows:

[0064]

[0065] In the formula, I ph Represents the photocurrent. Among the parameters of the photovoltaic array, I scr Indicates the short-circuit current of the photovoltaic array at reference temperature and radiation. This is the current value measured under standard test conditions, and the unit is ampere (A). k is the temperature coefficient of the short-circuit current, which describes the sensitivity of the short-circuit current to temperature changes, and the unit is ampere per Kelvin (A / K). T n is the actual operating temperature of the battery, in Kelvin (K). r It is the reference temperature of the battery, usually a standard temperature, such as 25℃ or 298K, and is also measured in Kelvin. S represents the intensity of solar radiation, measured in watts per square meter (W / m 2 ), which reflects the amount of light energy that hits the solar cell.

[0066] At the same time, we established an electrothermal model for lithium-ion batteries (LIBs), which consists of a second-order RC model and a two-state thermal model. The model can describe the changes in battery parameters such as voltage, current, internal resistance, and temperature during the charge and discharge process. The battery state of charge SoC change is given by the following formula:

[0067]

[0068] Where I(t) is the current at time t, Cn is the rated SoC(t) capacity of the battery, where SoC(t) represents the state of charge.

[0069] Battery terminal voltage V t By the open circuit voltage V oc , polarization voltage (V p1 ,V p2 ) and internal resistance R s Decide:

[0070] V t (t) = V oc (SoC,t)+V p1 (t)+V p2 (t)+R s (t)·I(t)

[0071] The thermal energy conservation equation is used to describe the thermal effect of the battery during the charging and discharging process:

[0072]

[0073] Among them, T s (t) and T a (t) represent the surface and internal temperature of the battery, R u and R c is the thermal resistance of battery surface convection and internal conduction, C s and C c is the heat capacity of the battery surface and interior, T f is the ambient temperature, and H(t) is the heat generated by the chemical reaction inside the battery.

[0074] Among them, the resistance increase in the battery aging model is summarized by the following formula:

[0075]

[0076] Among them, R age is the increase in terminal resistance due to calendar aging, and R0 is the initial internal resistance of the battery. r is the aging factor related to temperature and open circuit voltage is the i-th storage temperature. is the open circuit voltage at the ith storage temperature. is the storage time at time point i, and n is the total number of storage temperatures and time intervals.

[0077] Through these models, the operation of batteries in different orbital cycles is simulated, thus providing a simulation environment support for the optimization of spacecraft power systems. These models not only help improve the performance and life of batteries, but also help adapt to the complex space environment and ensure the stable operation of spacecraft power systems.

[0078] Step 2: Using an extended Kalman filter to estimate the key unmeasurable states of the battery, where the key unmeasurable states of the battery include the state of charge SOC and the state of health SOH;

[0079] The Extended Kalman Filter (EKF) is a powerful tool for estimating the internal state of a battery, including the state of charge (SOC) and state of health (SOH).

[0080] In the prediction step, the state prediction is performed through the state transfer function f, which predicts the current state based on the state estimate at the previous moment and the current control input (current). At the same time, the error covariance prediction takes into account the state transfer matrix and the process noise covariance matrix Q noise , to predict the uncertainty of the state estimate.

[0081] In the update step, the Kalman gain K is first calculated, which uses the predicted error covariance, the observation matrix H, and the measurement noise covariance R to determine how to weight the new measurement data. This gain is then used to update the state estimate, which involves applying the difference between the predicted state and the observed value to the predicted state. Finally, the error covariance is updated to account for the uncertainty of the new measurement data.

[0082] The update of SOC depends on the battery model and current integration. By calculating the current integral from the previous moment to the current moment, the change in battery charge is obtained, thereby updating SOC. The update of SOH depends on the measurement or estimation of the battery internal resistance. By comparing the current internal resistance with the new battery internal resistance and the end-of-life internal resistance, the health status of the battery is evaluated.

[0083] First, in the prediction step, the changes in SOC and internal temperature are predicted based on the battery state model. The prediction is based on the state estimate at the previous moment and the current control input (current):

[0084]

[0085] Where f represents the state transfer function, is the state estimate at the previous moment, and u(k) is the current control input.

[0086] Next, the prediction error covariance P(k|k-1) is calculated, taking into account the state transition matrix F and the process noise covariance matrix Q noise :

[0087] P(k|k-1)=FP(k-1)F T +Q noise

[0088] Here, F describes the linear effect of state on state, while Q noise Represents model uncertainty and external disturbances.

[0089] In the update step, the Kalman gain K is first calculated, which uses the predicted error covariance, the observation matrix H, and the measurement noise covariance R to determine how to incorporate new measurement data into the state estimate:

[0090] K=P(k|k-1)H T (HP(k|k-1)H T +R) -1

[0091] This gain is then used to update the state estimate:

[0092]

[0093] Where z(k) is the battery voltage and temperature measured from the sensor, and H is the observation matrix that maps the state space to the observation space.

[0094] Updating the error covariance is the last part of the update step:

[0095] P(k|k)=(I-KH)P(k|k-1)

[0096] Where P(k|k) is the updated covariance error.

[0097] This reflects how new measurement data affects the uncertainty of the state estimate.

[0098] The SOC is updated based on the battery charge accumulation, calculated by current integration:

[0099]

[0100] Among them, Q nom is the rated capacity of the battery, t k represents the kth time point in a discrete time series; ΔQ represents the change in battery charge from time step k-1 to time step k; SOC(k) represents the state of charge of the battery at time step k.

[0101] The update of SOH is based on the change of battery internal resistance:

[0102]

[0103] In the formula, R 0,new is the initial internal resistance of a new battery, R 0,EOL is the internal resistance at the end of the battery life, and R0 is the current internal resistance value.

[0104] Through these steps, EKF can effectively estimate the battery's SOC and SOH, providing key status information for the battery management system to ensure the reliability and performance of the battery. These calculations require not only accurate model parameters, but also sufficient computing speed and stability to ensure real-time execution in actual battery management systems.

[0105] Step 3: Create an intelligent agent based on the DDPG algorithm. The intelligent agent includes an actor network Actor and a critic network Critic. Then introduce a target network to regularly update the Actor network and the Critic network. Introduce an experience replay mechanism to design a training process for the intelligent agent based on the DDPG algorithm.

[0106] The intelligent agent based on the DDPG algorithm can learn the optimal charging and discharging strategy through interaction with the environment. The intelligent agent includes an actor network and a critic network, and adjusts the charging and discharging strategy by continuously optimizing network parameters.

[0107] The Deep Deterministic Policy Gradient (DDPG) algorithm is an advanced model-free reinforcement learning algorithm that combines the Actor-Critic framework and deep learning technology, and is particularly suitable for solving problems with continuous action spaces. The core of the DDPG algorithm is to use deep neural networks to approximate the policy function and value function, and to train the parameters of these networks through stochastic gradient descent.

[0108] In the DDPG algorithm, the Actor network is responsible for proposing action suggestions based on the current state. Its output action is generated by the state through a specific neural network structure. The forward propagation of the network can be expressed as

[0109]

[0110] Represented as follows, where a is the action, s is the state, and θ μ is the parameter of the Actor network, and μ is the Actor network. The Critic network evaluates the action value under the current strategy. Its input includes the current state and action, and outputs a Q value, which is the expected utility of the state-action pair. Its forward propagation can be expressed as

[0111]

[0112] Among them, θ Q and θ′ Q are the parameters of the Critic network.

[0113] In order to improve the stability and speed of learning, the DDPG algorithm introduces a target network, which is a slowly updated version of the Actor and Critic. The parameters of these target networks are regularly updated through a soft update strategy. The soft update is

[0114] θ′ μ ←τθ μ +(1-τ)θ′ μ

[0115] θ′ Q ←τθ Q +(1-τ)θ′ Q

[0116] Among them, τ is the learning rate, which controls the speed of updating the target network parameters, and θ′ μ Parameters for the strategy actor.

[0117] The DDPG algorithm also introduces an experience replay mechanism, which stores the experience data samples generated by the interaction between the Actor and the environment in the experience pool and randomly extracts batch data samples from the experience pool for training. This helps to remove the time correlation and dependency between samples, making the algorithm easier to converge.

[0118] During the training process, the Critic network is first trained to evaluate the current strategy, with the goal of minimizing the mean square error loss between the target Q value and the current Q value, which is

[0119]

[0120] In the formula, y i is the target Q value, is the current Q value, s i is the state at time step i; a i is the action at time step i; N is the number of samples, which represents the total number of samples extracted from the experience replay buffer.

[0121] Next, the Actor network updates its strategy based on the evaluation of the Critic network, with the goal of maximizing the expected Q value. Its strategy gradient is updated as

[0122]

[0123] In the formula, is the strategy function J with respect to the Actor network parameters θ μ The policy function J is represented as the predicted return obtained under a given policy, N is the number of samples drawn from the experience replay buffer, It refers to the gradient of the Q-value function output by the Critic network with respect to action a, which means that in state s i Take action a i The rate of change of Q value when is the Actor network with respect to its parameters θ μ The gradient of iIn the following example, we show how the actions output by an Actor network change as its parameters change.

[0124] In this way, the DDPG algorithm can learn an effective strategy to optimize the charging and discharging strategy of the spacecraft power system and improve the overall performance and efficiency of the system, such as Figure 2 shown.

[0125] Step 4: Design a loss function to achieve multi-objective optimization of the DDPG-based agent, where the multi-objectives include ensuring load satisfaction, controlling battery temperature, and extending battery life.

[0126] When designing the optimal control method for charging and discharging of the spacecraft power system, the Deep Deterministic Policy Gradient (DDPG) algorithm was used to solve a multi-objective optimization problem. The core of this problem is to develop a charging and discharging strategy that not only ensures the normal operation of the spacecraft load, but also controls the temperature of the battery and extends its service life as much as possible. This goal is achieved through the Actor-Critic structure of the DDPG algorithm, where the Actor network is responsible for proposing charging and discharging actions, while the Critic network is responsible for evaluating the utility of these actions.

[0127] The optimization problem can be formalized as a problem of minimizing a loss function which is a weighted sum of three different objectives.

[0128] The first goal is to ensure load satisfaction, that is, the power supplied by the battery needs to meet the needs of various instruments and equipment on the spacecraft. The second goal is to control the battery temperature and prevent the battery from overheating to avoid performance degradation and potential safety risks. The third goal is to extend the battery life by optimizing the charging and discharging process to reduce the wear on the battery. These goals can be formalized as the following optimization problem:

[0129]

[0130] Among them, C load , C temp , C life They represent the metrics of load satisfaction, battery temperature control and battery life extension, respectively, while ω1, ω2 and ω3 are the corresponding weight coefficients used to balance the importance of different objectives.

[0131] Step 5: Interact with the environment and use the training process to train the DDPG-based agent so that it can output the best strategy based on the measured state and the estimated key unmeasurable state of the battery;

[0132] The DDPG algorithm learns a strategy through its Actor-Critic structure The strategy outputs the optimal action a (i.e., charging and discharging strategy) given the current battery state s. The Critic network evaluates the utility of these actions and guides the training of the Actor network through a loss function. Specifically, the goal of the Critic network is to minimize the mean square error loss between the target Q value and the current Q value.

[0133] When implementing the charging and discharging strategy, the system monitors the battery status and changes in the external environment in real time, such as the battery's state of charge (SOC), temperature, and charging and discharging current. This information is used as state s to input into the Actor network, generating corresponding charging and discharging actions a. At the same time, the Critic network evaluates the utility of these actions and feeds back to the Actor network for adjustment. In this way, the system is able to dynamically adjust the charging and discharging strategy to adapt to changing operating conditions and ensure the stability of the power system and the health of the battery. The training results are as follows: Figure 3 shown.

[0134] Step 6: Input the input current, temperature, load voltage, estimated state of charge SOC and state of health SOH into the trained and optimized DDPG algorithm-based agent to obtain the charge / discharge current.

[0135] Through this design, we can implement an intelligent charging and discharging strategy that takes multiple objectives into consideration and responds to environmental changes in real time, thereby optimizing the performance and reliability of the spacecraft power system. This strategy not only improves the efficiency of battery use, but also ensures the stable operation of the spacecraft under various conditions.

[0136] Therefore, the present invention adopts the above-mentioned DDPG-based spacecraft battery system charging and discharging optimization control method to solve the problems existing in the existing spacecraft power system control method, such as the waste of resources caused by periodic sampling, the inability of fixed sampling frequency to adapt to the dynamic changes in battery performance, and the lack of effective management of battery health status and remaining service life; by introducing deep reinforcement learning technology, it can optimize the charging and discharging strategy in real time according to the actual working state of the battery and changes in the external environment, thereby improving the efficiency, reliability and adaptability of the spacecraft power system.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A DDPG-based spacecraft battery system charging and discharging optimization control method, characterized in that: The following steps are involved: Construct a battery state model in the spacecraft power system, where the battery model is a second-order RC model, and obtain the input current, temperature, and load voltage; An extended Kalman filter is used to estimate the key unmeasurable states of the battery, where the key unmeasurable states of the battery include the state of charge (SOC) and the state of health (SOH); Create an intelligent agent based on the DDPG algorithm. The intelligent agent includes an actor network Actor and a critic network Critic. Then introduce a target network to regularly update the Actor network and the Critic network. Introduce an experience replay mechanism to design a training process for the intelligent agent based on the DDPG algorithm. Design loss functions to achieve multi-objective optimization of the DDPG-based agent, where the multi-objectives include ensuring load satisfaction, controlling battery temperature, and extending battery life; Interact with the environment and use the training process to train the DDPG-based agent so that it can output the best strategy based on the measured state and the estimated key unmeasurable state of the battery; The input current, temperature, load voltage, estimated state of charge SOC and state of health SOH are input into the trained and optimized DDPG algorithm-based agent to obtain the charge / discharge current.

2. The DDPG-based spacecraft battery system charge and discharge optimization control method according to claim 1, characterized in that: The SOC estimate is updated based on the battery charge accumulation, calculated by current integration: Among them, Q nom is the rated capacity of the battery, t k represents the kth time point in the discrete time series; ΔQ represents the change in battery charge from time step k-1 to time step k; SOC(k) represents the state of charge of the battery at time step k; For SOH estimation, based on the change of battery internal resistance: In the formula, R 0,new is the initial internal resistance of a new battery, R 0,EOL is the internal resistance at the end of the battery life, and R0 is the current internal resistance value.

3. The DDPG-based spacecraft battery system charge and discharge optimization control method according to claim 1, characterized in that: The intelligent agent based on the DDPG algorithm includes an Actor network that is responsible for proposing action suggestions based on the current state. Its output action is generated by the state through a specific neural network structure. The forward propagation of the Actor network is expressed by the formula Represented as follows, where a is the action, s is the state, and θ μ is the parameter of the Actor network, μ is the Actor network; The Critic network evaluates the action value under the current strategy, inputs the current state and action, and outputs a Q value, which is the expected utility of the state-action pair. The forward propagation is expressed as Among them, θ Q and θ′ Q are the parameters of the Critic network.

4. The DDPG-based spacecraft battery system charge and discharge optimization control method according to claim 3, characterized in that: The target network, i.e. the slowly updated version of Actor and Critic, is introduced into the agent based on the DDPG algorithm. The parameters of the target network are regularly updated through the soft update strategy. The soft update is θ′ μ ←tth μ +(1-τ)θ′ μ θ′ Q ←tth Q +(1-τ)θ′ Q Among them, τ is the learning rate, which controls the speed of updating the target network parameters; θ′ μ Parameters for the strategy actor.

5. The DDPG-based spacecraft battery system charge and discharge optimization control method according to claim 3, characterized in that: The DDPG-based agent also introduces an experience replay mechanism, which stores the experience data samples generated by the interaction between the Actor and the environment into the experience pool, and randomly extracts batch data samples from the experience pool for training; During the training process, the Critic network is first trained to evaluate the current strategy, with the goal of minimizing the mean square error loss between the target Q value and the current Q value, which is In the formula, y i is the target Q value, is the current Q value, s i is the state at time step i; a i is the action at time step i; N is the number of samples, which means the total number of samples extracted from the experience replay buffer; Next, the Actor network updates its strategy based on the evaluation of the Critic network, with the goal of maximizing the expected Q value. Its strategy gradient is updated as In the formula, is the strategy function J with respect to the Actor network parameter θ μ The policy function J is represented as the predicted return obtained under a given policy, N is the number of samples drawn from the experience replay buffer, It refers to the gradient of the Q-value function output by the Critic network with respect to action a, which means that in state s i Take action a i The rate of change of Q value when is the Actor network with respect to its parameters θ μ The gradient of i In the following example, we show how the actions output by an Actor network change as its parameters change.

6. The DDPG-based spacecraft battery system charge and discharge optimization control method according to claim 1, characterized in that: Multi-objective optimization is performed through the Actor-Critic structure of the DDPG algorithm: Among them, C load , C temp , C life They represent the metrics of load satisfaction, battery temperature control and battery life extension respectively, while ω1, ω2 and ω3 are the corresponding weight coefficients.

Images