Satellite exploration control system and method based on deep reinforcement learning

Abstract

The application belongs to the technical field of aircraft navigation, guidance and control, and provides a satellite exploration control system and method based on deep reinforcement learning, which comprises a perception layer, a decision layer and an execution layer; the perception layer is used for integrating various device data and performing preliminary processing; the decision layer is used for deploying a deep reinforcement learning model and regularly performing data training and model updating; and the execution layer is used for controlling satellite attitude and orbit adjustment, and performing autonomous fault diagnosis and recovery operation; through continuous learning and optimization, the application can realize effective control under different environments and sudden situations, has strong adaptability and autonomous control ability, and through gradual optimization and iteration, can realize efficient and accurate control strategy in complex and changeable space environment, greatly improving the autonomous operation ability and task execution efficiency of the satellite.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of aircraft navigation, guidance and control, and in particular to a satellite exploration control system and method based on deep reinforcement learning. BACKGROUND

[0002] In modern space exploration, satellites serve as important detection tools, undertaking multiple tasks such as data collection, environmental monitoring and scientific experiments. With the advancement of technology, the autonomous control capability of satellites has gradually become a research hotspot. Traditional satellite control systems rely heavily on pre-set control algorithms and rules, which often fail to adapt to changes in complex and dynamic space environments, resulting in low efficiency or failure of tasks;

[0003] Deep reinforcement learning (DRL) is an emerging artificial intelligence technology that learns optimal strategies through interaction with the environment, demonstrating strong capabilities in complex decision-making problems. Its core idea is to guide agents to explore and learn in a specific environment through a reward mechanism, thereby achieving autonomous decision-making. Applying deep reinforcement learning to satellite exploration control systems can enable satellites to adjust their behavior in real time in unknown or changing environments to achieve the best exploration results;

[0004] In recent years, with the increase in the number of satellites and the complexity of tasks, control methods based on deep reinforcement learning have gradually gained attention. This approach not only improves the autonomy and flexibility of satellites, but also demonstrates significant advantages in multi-task coordination, resource optimization allocation, etc. Therefore, the present inventors propose a satellite exploration control system and method based on deep reinforcement learning, aiming to enhance system adaptability, achieve autonomous control of satellites and reduce dependence on ground control systems. SUMMARY

[0005] To solve the above technical problems, the present application provides a satellite exploration control system and method based on deep reinforcement learning to solve the problems raised in the background art.

[0006] According to the disclosed first aspect, a satellite exploration control system based on deep reinforcement learning is proposed, comprising a perception layer, a decision layer and an execution layer; the perception layer is used for integrating device data and performing preliminary processing; the decision layer is used for deploying a deep reinforcement learning model and regularly performing data training and model updating; the execution layer is used for controlling satellite attitude and orbit adjustment, and performing autonomous fault diagnosis and recovery operations;

[0007] The perception and data processing unit integrates multi-source data to provide accurate environmental models and dynamic information for the decision layer, and performs real-time data processing;

[0008] A decision and learning unit, which makes real-time decisions on orbit for the deep reinforcement learning model and updates the model through data interaction between the ground station and the satellite;

[0009] An execution and autonomous control unit, which executes specific operation commands issued by the decision layer and automatically detects and responds to various fault conditions;

[0010] An optimization unit, which is used to extract high-level features from raw sensor data and decompose complex control tasks into simple subtasks, with each layer focusing on different control objectives.

[0011] Preferably, the deep learning model is constructed using a soft actor-critic algorithm, in which the input of the actor network for generating the policy network of the action is the state s, the output is the probability distribution of the action a, the parameters θ of the network are optimized through the policy gradient, and the combination of the reward and the entropy is maximized; the critic network is used to evaluate the value of the selected action, the input is also the state s, and the output is the action value Q(s, a), the parameters ω of the network are optimized through gradient descent, and the difference between the actual return is minimized;

[0012] An entropy term H(.) is added to the objective function to accelerate the learning process, which is expressed as:

[0013] J(θ)=E[r(s,a)+αH(.)]

[0014] Where r(s, a) represents the immediate reward, α represents the temperature parameter for adjusting the importance of entropy, H(.) represents the entropy of the action distribution output by the policy; the optimization of the policy is to maximize the objective function J(θ) by adjusting the parameters θ, by calculating the policy gradient and applying gradient ascent, the temperature parameter α is adjusted during training to balance the trade-off between exploration and exploitation; the optimization of the value function is to minimize the mean square error loss by adjusting the parameters ω, which is expressed as:

[0015]

[0016] Where V(s'; ω') represents the target value network; during training, the action is sampled according to the current policy network and a noise distribution; and the algorithm uses a double delayed replay mechanism to store and update state transition samples.

[0017] Preferably, the perception and data processing unit comprises a multi-source data fusion module and a real-time data processing module, the multi-source data fusion module is deployed in the on-board computer to receive data from different sensors, pre-process and fuse the data to form a comprehensive environmental perception; the real-time data processing module is also deployed in the on-board computer to perform preliminary processing of the data using edge computing technology, reduce the data transmission demand for the ground station and reduce the communication delay.

[0018] Preferably, the pre-processing step in the multi-source data fusion module comprises: S101, geometric correction of data from different sensors to align them in the same spatial coordinate system; S102, atmospheric correction to eliminate the influence of the atmosphere on the sensor data; S103, radiation correction of the sensor data to make the data reflect the actual ground reflectivity; S104, spatial registration of images obtained by different sensors through feature matching and spatial transformation, with a registration accuracy of 1-2 pixels;

[0019] The fusion step in the multi-source data fusion module comprises: S201, extracting terrain, weather conditions and ground target features from the pre-processed data; S202, stitching, weighting or combining the extracted features to form a rich feature vector; S203, integrating the outputs of multiple independent decision modules through weighted averaging; S204, integrating the prediction results of multiple models through Boosting ensemble learning to obtain the final prediction.

[0020] Preferably, the decision and learning unit is to deploy a DRL model in the on-board computer, where a lightweight deep reinforcement learning model makes real-time decisions on orbit, and the multi-source data fusion module fuses features from different sources to form a comprehensive feature representation. The fused features are converted into state vectors for the DRL model, each state vector containing comprehensive environmental information. The encoded state vectors are input into the Actor network and Critic network of the soft actor-critic algorithm. Finally, the model is updated through data interaction between the ground station and the satellite. The ground station regularly uploads improved model parameters to the satellite to introduce the latest training results.

[0021] The Actor network outputs an action probability distribution based on the current state, representing the likelihood of different actions. A specific action is sampled from the action probability distribution for the satellite to execute. The satellite control system receives the selected action and converts it into specific control instructions for execution. Based on the satellite state change after action execution and the task target, the immediate reward is calculated, and the current state of the satellite is updated as the input for the next decision.

[0022] The state, action, reward and subsequent state of each decision are stored in an experience replay buffer for offline training of the model; the Critic network estimates the value of the action according to the current state and the immediate reward, and calculates the time difference error; using the time difference error, the parameters of the Actor network are updated through a gradient descent optimization algorithm to optimize the policy; and using the time difference error, the parameters of the Critic network are updated to optimize the value function.

[0023] Preferably, the execution and autonomous control unit receives control instructions from the decision layer through the control execution module, which are generated based on the deep learning model and the soft actor-critic algorithm, and contain specific action targets and optimized control parameters. By decoding the instructions, they are converted into operational control signals. In the decoding process, the instruction type, target action and parameters are identified. According to the decoded instructions, corresponding control signals are generated, and different types of instructions require different combinations of control signals. After the action is executed, the sensors on the satellite collect new state data in real time, including position, velocity and attitude, and feed back to the execution and autonomous control unit. By analyzing the feedback data, it is determined whether the current state is consistent with the instruction target. If there is a deviation, adjust the control signal to eliminate the error.

[0024] Preferably, the execution and autonomous control unit also collects various types of running data from sensors on the satellite, as well as collects running state information of each subsystem and device and log information during network operation, and classifies and analyzes the information; by comparing historical normal data and threshold values, abnormal data that exceeds the normal range is identified and marked as potential failure; based on a predefined rule base, the identified abnormal data is matched with rules to automatically provide preliminary diagnostic results. When a fault is diagnosed, an alarm message is immediately generated through the autonomous fault handling module, including the fault location, type and urgency. The alarm message is transmitted in real time to the ground control center through the satellite communication system. According to the diagnostic results, repair operations are performed, and after repair, relevant data is re-collected to verify whether the satellite state has returned to normal.

[0025] Preferably, the decision and learning unit also constructs a plurality of state value functions and action value functions, each function optimized for a different target, i.e. the overall task target is decomposed into a number of specific sub-task targets, and a state value function V π (s) or action value function Q π (s,a) is constructed for each sub-task target, which is optimized according to the specific task target. In each step, the system receives a plurality of reward signals r i from the contribution of its behavior to each sub-task target, and normalizes the reward signals, and then updates the parameters of the Actor network through a gradient descent optimization algorithm to optimize the policy. iThe weighted sum is performed to form a final comprehensive reward signal: r = ∑ i ω i ·r i , wherein the weight ω i is dynamically adjusted according to the task requirements; and the PBRS technology is utilized to increase or decrease the immediate reward of certain states without changing the optimal strategy by assigning a "potential energy" value to each state; in multi-objective optimization, the Pareto optimal frontier is sought, that is, a balance is reached among all objectives, at which time no objective is further improved without deteriorating other objectives; meanwhile, the weights ω i of the subtask objectives are dynamically adjusted by real-time monitoring of different task requirements and environmental changes.

[0026] Preferably, the optimization unit utilizes deep learning technology to extract high-level features from raw sensor data and compress data transmission, reducing bandwidth requirements; at the same time, the continuous action space is discretized to simplify the decision-making process, and a hierarchical control framework is adopted to decompose complex control tasks into simple subtasks, with each layer focusing on different control objectives to simplify the learning process.

[0027] According to the disclosed second aspect, a satellite exploration control method based on deep reinforcement learning is proposed, specifically comprising the following steps:

[0028] S1, specific task objectives of satellite exploration control are determined, and a simulation environment is constructed, which simulates the physical state of the satellite in space, including position, velocity and attitude;

[0029] S2, a state space and an action space are designed, the state space should contain all necessary satellite state information, and the action space defines all possible actions that the satellite can take;

[0030] S3, a plurality of state value functions and action value functions are constructed, each function is optimized for different objectives, that is, a dynamic weight multi-objective reward function is designed, and the parameters of the reward function are dynamically adjusted according to the data collected during task execution;

[0031] S4, the deep reinforcement learning model is trained using the training environment, the model selects an action according to the current strategy, the environment responds to the action and gives a new state and reward, and the model updates the strategy accordingly;

[0032] S5, the trained deep reinforcement learning model is deployed to the actual satellite control system, and the model autonomously decides and executes the corresponding control action according to the real-time acquired satellite state information.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] 1、The present application realizes effective control in different environments and unexpected situations through continuous learning and optimization, does not need to rely on accurate satellite physical parameters and prior knowledge of orbit parameters, and has strong adaptability and autonomous control ability.

[0035] 2、The present application can autonomously perform fault diagnosis and recovery operation under the instability of the space environment and possible faults of the satellite, guarantee the normal operation of the satellite, and significantly improve the efficiency of task execution through optimization of task planning and scheduling strategy. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The satellite exploration control system based on deep reinforcement learning of the present application is a block diagram;

[0037] Figure 2 The satellite exploration control method based on deep reinforcement learning of the present application is a flowchart. DETAILED DESCRIPTION

[0038] The embodiments of the present application will be further described in detail below in combination with the drawings and examples. The following examples are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0039] As shown in the accompanying Figure 1 drawings:

[0040] Example 1: The present application provides a satellite exploration control system based on deep reinforcement learning, which includes a perception layer, a decision layer and an execution layer; the perception layer is used to integrate device data, including data of high-resolution cameras, infrared sensors, weather radars and other devices, for preliminary processing; the decision layer is used to deploy a deep reinforcement learning model on a satellite computer, and regularly perform data training and model updating; the execution layer is used to control satellite attitude and orbit adjustment, and integrate fault detection and autonomous recovery mechanisms, so that the satellite can autonomously perform fault diagnosis and recovery operation when facing unexpected situations.

[0041] The perception and data processing unit provides accurate environment models and dynamic information for the decision layer by integrating multi-source data, and performs real-time data processing to reduce data transmission volume and delay and improve response speed; the decision and learning unit makes real-time decisions on the orbit through a deep reinforcement learning model, and updates the model through data interaction between the ground station and the satellite; the execution and autonomous control unit executes specific operation commands issued by the decision layer, and automatically detects and responds to various fault conditions; the optimization unit is used to extract high-level features from raw sensor data, and decomposes complex control tasks into simple subtasks, with each layer focusing on different control objectives.

[0042] The deep learning model is constructed using a soft actor-critic algorithm. The input of the actor network for generating the policy network of the action is the state s, and the output is the probability distribution of the action a. The parameters θ of the network are optimized by the policy gradient to maximize the combination of the reward and the entropy. The critic network is used to evaluate the value of the selected action. The input of the network is also the state s, and the output is the action value Q(s, a). The parameters ω of the network are optimized by gradient descent to minimize the difference between the actual return.

[0043] An entropy term H(.) is added to the objective function to accelerate the learning process and improve the robustness of the final policy. The objective function is represented as:

[0044] J(θ)=E[r(s,a)+αH(.)]

[0045] where r(s, a) represents the immediate reward, α represents the temperature parameter for adjusting the importance of the entropy, and H(.) represents the entropy of the action distribution output by the policy. The optimization of the policy is achieved by adjusting the parameters θ to maximize the objective function J(θ). The temperature parameter α is adjusted during training to balance the trade-off between exploration and exploitation. The optimization of the value function is achieved by adjusting the parameters ω to minimize the mean square error loss, which is represented as:

[0046]

[0047] where V(s'; ω') represents the target value network. During training, actions are sampled according to the current policy network and a noise distribution to ensure sufficient exploration. The algorithm uses a double delayed replay mechanism to store and update state transition samples, improving sample utilization efficiency and enhancing algorithm stability. By optimizing the policy by maximizing the trade-off between reward and entropy, more stable and robust learning is achieved, solving the problem of complex action decision-making in satellite exploration, enabling satellites to autonomously learn and adapt in a changing space environment, thereby improving the efficiency and reliability of task execution.

[0048] The perception and data processing unit includes a multi-source data fusion module and a real-time data processing module. The multi-source data fusion module is deployed in the on-board computer and is used to receive data from different sensors, pre-process and fuse the data, and form a comprehensive environmental perception. The real-time data processing module is also deployed in the on-board computer and uses edge computing technology to perform preliminary processing of the data, reducing the need for data transmission to the ground station and reducing communication delay.

[0049] The preprocessing step in the multi-source data fusion module includes: S101, geometric correction of data from different sensors to align them in the same spatial coordinate system; S102, atmospheric correction to eliminate the influence of the atmosphere on sensor data; S103, radiation correction of sensor data to reflect the actual ground reflectivity; S104, spatial registration by feature matching and spatial transformation to align the images obtained by different sensors in space, with a registration accuracy of 1-2 pixels;

[0050] The fusion step includes: S201, extracting terrain, weather conditions and ground target features from the preprocessed data; S202, stitching, weighting or combining the extracted features to form a rich feature vector, which helps to integrate information from different sensors and improve the accuracy of data analysis; S203, integrating the outputs of multiple independent decision modules through weighted averaging; S204, integrating the prediction results of multiple models through Boosting ensemble learning to obtain the final prediction. The multi-source data fusion module ensures the effective integration of data from different sensors through the above preprocessing and fusion steps, thereby providing comprehensive environmental perception for the satellite exploration system, enabling the system to more accurately understand and interpret complex data from different sensors.

[0051] The decision and learning unit is to deploy the DRL model in the on-board computer, where the lightweight deep reinforcement learning model makes real-time decisions on orbit, and the multi-source data fusion module integrates features from different sources to form a comprehensive feature representation. The fused features are converted into state vectors for the DRL model, each state vector containing comprehensive environmental information. The encoded state vectors are then input into the Actor network and Critic network of the soft actor-critic algorithm. Finally, the model is updated through data interaction between the ground station and the satellite. The ground station regularly uploads improved model parameters to the satellite, introducing the latest training results;

[0052] The Actor network outputs an action probability distribution based on the current state, representing the likelihood of different actions. A specific action is sampled from the action probability distribution for the satellite to execute. The satellite control system receives the selected action and converts it into specific control instructions, such as thruster ignition, attitude adjustment, etc., and executes them. Based on the satellite state change after action execution and the task target, the immediate reward is calculated, such as positive reward for moving towards the target and negative reward for deviating from the orbit. At the same time, the current state of the satellite is updated as the input for the next decision;

[0053] The state, action, reward and subsequent state of each decision are stored in an experience replay buffer for offline training of the model; the Critic network estimates the value of the action based on the current state and the immediate reward, and calculates the time difference error; using the time difference error, the parameters of the Actor network are updated through a gradient descent optimization algorithm to optimize the policy; and using the time difference error, the parameters of the Critic network are updated to optimize the value function. Through the above manner, the deep reinforcement learning model can make real-time decisions on the orbit, so that the satellite can adapt to the complex space environment and effectively perform tasks.

[0054] The execution and autonomous control unit receives control instructions from the decision layer through the control execution module, which are generated based on the deep learning model and the soft actor-critic algorithm, and contain specific action targets and optimized control parameters. By decoding the instructions, they are converted into operational control signals. In the decoding process, the instruction type, target action and parameters are identified. According to the decoded instructions, corresponding control signals are generated, including clock signals, read / write signals, address selection signals, etc., which are used to control various execution mechanisms on the satellite such as thrusters, attitude control systems, etc. Different types of instructions require different combinations of control signals. For example, a thruster ignition instruction may require ignition time, thrust size, etc. signals, while an attitude adjustment instruction requires offset and action time signals for each attitude control actuator. After executing the action, the sensors on the satellite collect new state data in real time, including position, velocity and attitude, and feed back to the execution and autonomous control unit. By analyzing the feedback data, it is determined whether the current state is consistent with the instruction target. If there is a deviation, adjust the control signal to eliminate the error and ensure accurate execution. Through the above steps, the operation commands issued by the decision layer are accurately and accurately executed.

[0055] The execution and autonomous control unit also collects various types of running data from sensors on the satellite in real time, such as temperature, voltage, current, attitude, etc., as well as collecting the running state information of each subsystem and device and the log information in the network running process, and classifying and analyzing the information. By comparing historical normal data and threshold values, abnormal data that exceeds the normal range is identified as a potential fault. Based on a predefined rule base, the identified abnormal data is matched with rules to automatically provide preliminary diagnostic results, and the detailed mathematical models of each system of the satellite are used to compare telemetry data with model expectations to further confirm the fault location and cause. When a fault is diagnosed, an alarm message is immediately generated through the autonomous fault handling module, including the fault location, type and urgency. The alarm message is transmitted in real time to the ground control center through the satellite communication system. According to the diagnostic results, repair operations are performed, and after repair, relevant data is re-collected to verify whether the satellite state has returned to normal, ensuring that the fault has been effectively solved.

[0056] The decision and learning unit also builds multiple state value functions and action value functions, each function is optimized for different goals, i.e. the overall task goal is decomposed into several specific sub-task goals, and a state value function V π (s) or action value function Q π (s, a) is built for each sub-task goal, which is optimized according to the specific task goal; in each step, the system receives multiple reward signals r i according to its contribution to each sub-task goal, and the reward signals are normalized and then weighted and summed through dynamic weights ω i to form the final comprehensive reward signal: r = ∑ i ω i · r i , wherein the weight ω i is dynamically adjusted according to the task requirements; and by using the PBRS technology, by assigning a "potential" value to each state, the immediate reward of some states is increased or decreased without changing the optimal strategy; in multi-objective optimization, a Pareto optimal frontier is sought, i.e. a balance between all objectives is reached, at this time no objective can be further improved without deteriorating other objectives; at the same time, by monitoring different task requirements and environmental changes in real time, the weights ω i of each sub-task goal are dynamically adjusted to ensure that the reward signal can accurately reflect the most important task goal. At the same time, a human intervention interface is reserved to allow ground control personnel to adjust the reward function when necessary to guide the satellite to make more reasonable decisions and increase the controllability of the system.

[0057] The optimization unit uses deep learning technology to extract high-level features from raw sensor data, reduces dimension disaster, improves decision efficiency, and reduces consumption of computing resources, and uses an efficient data compression algorithm to reduce the bandwidth demand of state information transmission to the ground station, reduce communication cost, and improve data transmission efficiency; at the same time, the continuous action space is discretized, the decision process is simplified, the control accuracy is guaranteed, the accuracy requirement of task execution is met, and a hierarchical control framework is adopted to decompose complex control tasks into simple sub-tasks, each layer focuses on different control targets, simplifies the learning process, and improves control efficiency. Ensure that the satellite exploration control system can efficiently and accurately complete the task in a complex and variable space environment, while having the ability to learn and adapt to new challenges.

[0058] As shown in the accompanying Figure 2 :

[0059] Embodiment two: the application also provides a satellite exploration control method based on deep reinforcement learning, specifically comprising the following steps:

[0060] S1, determine the specific task goal of satellite exploration control, build a simulation environment that simulates the physical state of the satellite in space, including position, velocity, attitude;

[0061] S2, design state space and action space, the state space should contain all necessary satellite state information, and the action space defines all possible actions that the satellite can take;

[0062] S3, build multiple state value functions and action value functions, each function optimizes for different goals, i.e. design a multi-objective reward function with dynamic weights, and dynamically adjust the parameters of the reward function according to the data collected during task execution;

[0063] S4, use the training environment to train the deep reinforcement learning model, the model selects an action according to the current policy, the environment responds to the action and gives a new state and reward, and the model updates the policy accordingly;

[0064] S5, deploy the trained deep reinforcement learning model to the actual satellite control system, the model autonomously decides and executes the corresponding control action according to the real-time satellite state information.

[0065] In addition, in order to provide a brief description of the exemplary embodiments, all the features of the actual embodiments (i.e. those unrelated to the best mode of carrying out the invention currently considered, or those unrelated to the implementation of the invention) can not be described.

[0066] It should be understood that during the development of any actual implementation, numerous implementation-specific decisions can be made. Such development efforts, while possibly complex and time-consuming, would nevertheless be routine undertaking for those of ordinary skill in the art having the benefit of this disclosure.

[0067] It should be noted that the above examples are only used to illustrate the technical solutions of the present application and not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A satellite exploration control system based on deep reinforcement learning, characterized in that: It includes a perception layer, a decision layer, and an execution layer. The perception layer is used to integrate data from various devices and perform preliminary processing. The decision layer is used to deploy deep reinforcement learning models and regularly conduct data training and model updates. The execution layer is used to control satellite attitude and orbit adjustment, and perform autonomous fault diagnosis and recovery operations. The perception and data processing unit integrates multi-source data to provide accurate environmental models and dynamic information to the decision-making layer and performs real-time data processing; The decision-making and learning unit uses the deep reinforcement learning model to make real-time decisions on orbit and updates the model through data interaction between the ground station and the satellite; Execution and autonomous control unit, which executes specific operation commands issued by the decision-making layer and automatically detects and responds to various fault conditions; The optimization unit uses deep learning technology to extract high-level features from raw sensor data and compress data transmission to reduce bandwidth requirements; at the same time, it discretizes the continuous action space to simplify the decision-making process, and adopts a hierarchical control framework to decompose complex control tasks into simple subtasks, with each layer focusing on different control objectives to simplify the learning process.

2. The satellite exploration and control system based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning model is constructed using the soft actor-critic algorithm. The Actor network is used to generate the action policy network input, which is the state s and outputs the probability distribution of action a. The network parameters θ are optimized through policy gradient to maximize the combination of reward and entropy. The Critic network is used to evaluate the value of the selected action. Its input is also the state s and the output is the action value Q(s, a). The network parameters ω are optimized through gradient descent to minimize the difference between the actual reward and the actual reward. The learning process is accelerated by adding an entropy term H(.) to the objective function, which is expressed as: J(θ)=E[r(s,a)+αH(.)] Among them, r(s,a) represents the immediate reward, α represents the temperature parameter, which is used to adjust the importance of entropy, and H(.) represents the entropy of the policy output action distribution; its policy optimization is to maximize the objective function J(θ) by adjusting the parameter θ, and by calculating the policy gradient and applying gradient ascent. The temperature parameter α is adjusted during training to balance the trade-off between exploration and development; the value function is optimized by adjusting the parameter ω to minimize the mean square error loss, which is expressed as: L(ω)=E[(Q(s,a;ω)-(r(s,a)+V(s′;ω′)) 2 )] where V(s′;ω′) denotes the target-value network; during training, actions are sampled from the current policy network and a noise distribution; and the algorithm uses a double-delayed replay mechanism to store and update state transition samples.

3. The satellite exploration and control system based on deep reinforcement learning according to claim 1, characterized in that: The perception and data processing unit includes a multi-source data fusion module and a real-time data processing module. The multi-source data fusion module is deployed in the onboard computer and is used to receive data from different sensors and perform pre-processing and fusion to form comprehensive environmental perception; the real-time data processing module is also deployed in the onboard computer and uses edge computing technology to perform preliminary data processing, reducing the data transmission requirements of the ground station and reducing communication delays.

4. The satellite exploration and control system based on deep reinforcement learning according to claim 3, characterized in that: The preprocessing steps in the multi-source data fusion module include: S101, geometrically correcting the data from different sensors so that they are aligned in the same spatial coordinate system; S102, performing atmospheric correction to eliminate the influence of the atmosphere on the sensor data; S103, performing radiometric correction on the sensor data so that the data reflects the actual surface reflectivity; S104, spatially aligning the images acquired by different sensors through feature matching and spatial transformation, and performing spatial registration, with the registration accuracy within 1 to 2 pixels; The fusion steps in the multi-source data fusion module include: S201, extracting terrain, meteorological conditions and ground target features from the preprocessed data; S202, splicing, weighting or combining the extracted features to form a rich feature vector; S203, integrating the outputs of multiple independent decision modules by weighted averaging; S204, integrating the prediction results of multiple models through Boosting ensemble learning to obtain the final prediction.

5. The satellite exploration and control system based on deep reinforcement learning according to claim 1, characterized in that: The decision-making and learning unit deploys a deep reinforcement learning model in an onboard computer, wherein a lightweight deep reinforcement learning model makes real-time decisions on orbit. The multi-source data fusion module fuses features from different sources to form a comprehensive feature representation, converting the fused features into state vectors processed by the deep reinforcement learning model. Each state vector contains comprehensive environmental information. The encoded state vector is then input into the actor network and critic network of the soft actor-critic algorithm. Finally, the model is updated through data interaction between the ground station and the satellite. The ground station regularly uploads improved model parameters to the satellite to introduce the latest training results. Based on the current state, the Actor network outputs an action probability distribution, representing the likelihood of different actions. A specific action is sampled from the action probability distribution for the satellite to execute. The satellite control system receives the selected action, converts it into a specific control instruction, and executes it. Based on the satellite state change and mission objectives after the action is executed, an immediate reward is calculated and the current state of the satellite is updated, which serves as input for the next decision. The state, action, reward, and subsequent state of each decision are stored in the experience replay buffer for offline training of the model. The Critic network estimates the value of the action based on the current state and immediate reward, and calculates the time difference error. The time difference error is used to update the parameters of the Actor network through the gradient descent optimization algorithm for policy optimization. The time difference error is then used to update the parameters of the Critic network for value function optimization.

6. The satellite exploration and control system based on deep reinforcement learning according to claim 1, characterized in that: The execution and autonomous control unit receives control instructions from the decision-making layer through the control execution module. The instructions are generated based on a deep reinforcement learning model and a soft actor-critic algorithm and include specific action targets and optimized control parameters. The instructions are decoded and converted into operable control signals. During the decoding process, the instruction type, target action, and parameters are identified. Based on the decoded instructions, corresponding control signals are generated. Different types of instructions require different control signal combinations. After the action is executed, sensors on the satellite collect new status data in real time, including position, speed, and attitude, and feed it back to the execution and autonomous control unit. By analyzing the feedback data, it is determined whether the current state is consistent with the instruction target. If there is a deviation, the control signal is adjusted to eliminate the error.

7. A satellite exploration and control system based on deep reinforcement learning as claimed in claim 6, characterized in that: The execution and autonomous control unit also collects various types of operating data in real time from sensors on the satellite, as well as operating status information of each subsystem and equipment and log information during network operation, and classifies and analyzes the information; by comparing historical normal data and thresholds, it identifies abnormal data that exceeds the normal range and marks it as a potential fault; based on a predefined rule base, it matches the identified abnormal data with rules and automatically gives preliminary diagnostic results. When a fault is diagnosed, an alarm information is immediately generated through the autonomous fault processing module, including the fault location, type, and urgency; the alarm information is transmitted to the ground control center in real time through the satellite communication system; according to the diagnostic results, a repair operation is performed, and after the repair, the relevant data is re-collected to verify whether the satellite status has returned to normal.

8. The satellite exploration and control system based on deep reinforcement learning according to claim 5, characterized in that: The decision-making and learning unit also constructs multiple state value functions and action value functions, each function is optimized for different goals, that is, the overall task goal is decomposed into several specific sub-task goals, and a state value function V is constructed for each sub-task goal. π (s) or action-value function Q π (s, a), which is optimized according to a specific task goal; at each step, the system receives multiple reward signals r according to the contribution of its behavior to each subtask goal i , and normalize the reward signal, and then pass the dynamic weight ω i Perform weighted summation to form the final comprehensive reward signal: r = Σ i ω i ·r i , where the weight ω i Dynamically adjust according to task requirements; and use PBRS technology to increase or decrease the immediate rewards of certain states without changing the optimal strategy by assigning a "potential energy" value to each state; in multi-objective optimization, seek to achieve the Pareto optimal frontier, that is, to achieve a balance between all goals, at this time no goal can be further improved without deteriorating the others; at the same time, by real-time monitoring of different task requirements and environmental changes, dynamically adjust the weights of each sub-task goal i .

9. A satellite exploration control method based on deep reinforcement learning, applied to the satellite exploration control system based on deep reinforcement learning according to any one of claims 1 to 8, characterized in that: The specific steps include: S1. Define the specific mission objectives of satellite exploration and control, and build a simulation environment that simulates the physical state of the satellite in space, including position, velocity, and attitude; S2. Design the state space and action space. The state space should contain all necessary satellite state information, and the action space defines all possible actions that the satellite can take. S3. Construct multiple state-value functions and action-value functions, each optimized for a different objective. This means designing a multi-objective reward function with dynamic weights and dynamically adjusting the reward function parameters based on the data collected during task execution. S4. Use the training environment to train the deep reinforcement learning model. The model selects an action to execute based on the current policy. The environment responds to the action and gives a new state and reward. The model updates its policy accordingly. S5. Deploy the trained deep reinforcement learning model to the actual satellite control system. The model makes autonomous decisions and executes corresponding control actions based on the real-time satellite status information.

Images