Satellite exploration control system and method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of aircraft navigation, guidance and control, and provides a satellite exploration control system and method based on deep reinforcement learning, and the system comprises a sensing layer, a decision layer, and an execution layer. The sensing layer is used for integrating equipment data and performing primary processing; the decision-making layer is used for deploying a deep reinforcement learning model and regularly carrying out data training and model updating; the execution layer is used for controlling satellite attitude and orbit adjustment and performing autonomous fault diagnosis and recovery operation; according to the method, effective control under different environments and emergencies is realized through continuous learning and optimization, the adaptive capacity and the autonomous control capacity are relatively high, and an efficient and accurate control strategy can be realized in a complex and changeable space environment through gradual optimization and iteration, so that the method is suitable for large-scale popularization and application. And the autonomous operation capability and the task execution efficiency of the satellite are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of aircraft navigation, guidance and control, and specifically relates to a satellite exploration control system and method based on deep reinforcement learning. Background Art

[0002] In modern space exploration, satellites, as important detection tools, undertake multiple tasks such as data collection, environmental monitoring and scientific experiments. With the advancement of technology, the autonomous control capability of satellites has gradually become a research hotspot. Traditional satellite control systems mostly rely on preset control algorithms and rules, which are often difficult to adapt to changes in complex and dynamic space environments, resulting in low efficiency or failure of missions;

[0003] As an emerging artificial intelligence technology, Deep Reinforcement Learning (DRL) demonstrates its powerful ability in complex decision-making problems by learning optimal strategies through interaction with the environment. Its core idea is to guide the agent to explore and learn in a specific environment through a reward mechanism, thereby achieving autonomous decision-making. Applying deep reinforcement learning to satellite exploration control systems can enable satellites to adjust their behavior in real time in unknown or changing environments to achieve the best exploration effect;

[0004] In recent years, with the increase in the number of satellites and the increase in mission complexity, control methods based on deep reinforcement learning have gradually attracted attention. This method can not only improve the autonomy and flexibility of satellites, but also show significant advantages in multi-task collaboration and resource optimization allocation. Therefore, those skilled in the art have proposed a satellite exploration control system and method based on deep reinforcement learning, aiming to enhance the adaptability of the system, realize autonomous control of satellites, and reduce dependence on ground measurement and control systems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a satellite exploration control system and method based on deep reinforcement learning to solve the problems raised in the background technology.

[0006] According to the first aspect disclosed, a satellite exploration control system based on deep reinforcement learning is proposed, comprising a perception layer, a decision layer and an execution layer; the perception layer is used to integrate data from various devices and perform preliminary processing; the decision layer is used to deploy a deep reinforcement learning model, and regularly perform data training and model updates; the execution layer is used to control satellite attitude and orbit adjustment, and perform autonomous fault diagnosis and recovery operations;

[0007] The perception and data processing unit integrates multi-source data to provide the decision-maker with accurate environmental models and dynamic information, and performs real-time data processing;

[0008] The decision-making and learning unit uses the deep reinforcement learning model to make real-time decisions on orbit and updates the model through data interaction between the ground station and the satellite;

[0009] Execution and autonomous control unit, which executes specific operation commands issued by the decision-making layer and automatically detects and responds to various fault conditions;

[0010] An optimization unit is used to extract high-level features from raw sensor data and decompose complex control tasks into simple subtasks, with each layer focusing on a different control objective.

[0011] Preferably, the deep learning model is constructed using a soft actor-critical algorithm, wherein the Actor network is used to generate the action strategy network input, which is the state s, and the output is the probability distribution of the action a, and the network parameter θ is optimized by the policy gradient to maximize the combination of reward and entropy; the Critic network is used to evaluate the value of the selected action, and its input is also the state s, and the output is the action value Q(s, a), and the network parameter ω is optimized by gradient descent to minimize the difference with the actual reward;

[0012] The learning process is accelerated by adding an entropy term H(.) to the objective function, which is expressed as:

[0013] J(θ)=E[r(s,a)+αH(.)]

[0014] Among them, r(s,a) represents the immediate reward, α represents the temperature parameter, which is used to adjust the importance of entropy, and H(.) represents the entropy of the policy output action distribution; the optimization of its strategy is to maximize the objective function J(θ) by adjusting the parameter θ, and the temperature parameter α is adjusted with training to balance the trade-off between exploration and development by calculating the policy gradient and applying gradient ascent; the optimization of the value function is to minimize the mean square error loss by adjusting the parameter ω, which is expressed as:

[0015]

[0016] where V(s′;ω′) denotes the target value network; during training, actions are sampled from the current policy network and a noise distribution; and the algorithm uses a double delayed replay mechanism to store and update state transition samples.

[0017] Preferably, the perception and data processing unit includes a multi-source data fusion module and a real-time data processing module. The multi-source data fusion module is deployed in the onboard computer, and is used to receive data from different sensors, and perform pre-processing and fusion to form a comprehensive environmental perception; the real-time data processing module is also deployed in the onboard computer, and uses edge computing technology to perform preliminary processing of the data, thereby reducing the data transmission requirements for the ground station and reducing communication delays.

[0018] Preferably, the preprocessing steps in the multi-source data fusion module include: S101, geometrically correcting the data from different sensors so that they are aligned in the same spatial coordinate system; S102, performing atmospheric correction to eliminate the influence of the atmosphere on the sensor data; S103, performing radiation correction on the sensor data so that the data reflects the actual surface reflectivity; S104, aligning the images obtained by different sensors in space through feature matching and spatial transformation, and performing spatial registration, with the registration accuracy within 1 to 2 pixels;

[0019] The fusion steps in the multi-source data fusion module include: S201, extracting terrain, meteorological conditions and ground target features from the preprocessed data; S202, splicing, weighting or combining the extracted features to form a rich feature vector; S203, integrating the outputs of multiple independent decision modules by weighted averaging; S204, integrating the prediction results of multiple models through Boosting integrated learning to obtain the final prediction.

[0020] Preferably, the decision-making and learning unit deploys the DRL model in an onboard computer, wherein a lightweight deep reinforcement learning model makes real-time decisions on orbit, and a multi-source data fusion module fuses features from different sources to form a comprehensive feature representation, converts the fused features into state vectors processed by the DRL model, each state vector contains comprehensive environmental information, and then inputs the encoded state vectors into the Actor network and Crit ic network of the soft actor-crit ic algorithm. Finally, the model is updated through data interaction between the ground station and the satellite, and the ground station regularly uploads the improved model parameters to the satellite to introduce the latest training results;

[0021] Based on the current state, the Actor network outputs an action probability distribution, which represents the possibility of different actions. A specific action is sampled from the action probability distribution for the satellite to execute. The satellite control system receives the selected action, converts it into a specific control instruction, and executes it. Based on the satellite state change and mission objectives after the action is executed, the immediate reward is calculated, and the current state of the satellite is updated as input for the next decision.

[0022] The state, action, reward and subsequent state of each decision are stored in the experience replay buffer for offline training of the model; the Critical network estimates the value of the action based on the current state and immediate reward, and calculates the time difference error; the time difference error is used to update the parameters of the Actor network through the gradient descent optimization algorithm for strategy optimization; and the time difference error is used to update the parameters of the Critical network for value function optimization.

[0023] Preferably, the execution and autonomous control unit receives control instructions from the decision-making layer through the control execution module. The instructions are generated based on a deep learning model and a soft actor-critical algorithm, and include specific action targets and optimized control parameters. The instructions are decoded and converted into operable control signals. During the decoding process, the instruction type, target action, and parameters are identified. According to the decoded instructions, corresponding control signals are generated, and different types of instructions require different control signal combinations. After the action is executed, the sensors on the satellite collect new status data in real time, including position, speed, and attitude, and feed it back to the execution and autonomous control unit. By analyzing the feedback data, it is determined whether the current status is consistent with the instruction target. If there is a deviation, the control signal is adjusted to eliminate the error.

[0024] Preferably, the execution and autonomous control unit also collects various types of operating data in real time from sensors on the satellite, as well as operating status information of each subsystem and equipment and log information during network operation, and classifies and analyzes the information; identifies abnormal data that exceeds the normal range by comparing historical normal data and thresholds, and marks them as potential faults; performs rule matching on the identified abnormal data based on a predefined rule base, and automatically gives preliminary diagnostic results. When a fault is diagnosed, an alarm information is immediately generated through an autonomous fault processing module, including the fault location, type, and urgency; the alarm information is transmitted to the ground control center in real time through the satellite communication system; based on the diagnostic results, a repair operation is performed, and after the repair, relevant data is re-collected to verify whether the satellite status has returned to normal.

[0025] Preferably, the decision-making and learning unit further constructs a plurality of state value functions and action value functions, each function is optimized for a different goal, that is, the overall task goal is decomposed into a number of specific sub-task goals, and a state value function V is constructed for each sub-task goal. π (s) or action-value function Q π (s, a), which is optimized according to a specific task goal; at each step, the system receives multiple reward signals r according to the contribution of its behavior to each subtask goal i , and normalize the reward signal, and then pass the dynamic weight ω iPerform weighted summation to form the final comprehensive reward signal: r = ∑ i ω i ·r i , where the weight ω i Dynamically adjust according to task requirements; and use PBRS technology to increase or decrease the immediate rewards of certain states without changing the optimal strategy by assigning a "potential energy" value to each state; in multi-objective optimization, seek to reach the Pareto optimal frontier, that is, to achieve a balance between all goals, at which time no goal can be further improved without deteriorating other goals; at the same time, by real-time monitoring of different task requirements and environmental changes, dynamically adjust the weight of each sub-task goal ω i .

[0026] Preferably, the optimization unit utilizes deep learning technology to extract high-level features from raw sensor data and compresses data transmission to reduce bandwidth requirements; at the same time, the continuous action space is discretized to simplify the decision-making process, and a hierarchical control framework is adopted to decompose complex control tasks into simple subtasks, with each layer focusing on different control objectives to simplify the learning process.

[0027] According to the second aspect disclosed, a satellite exploration control method based on deep reinforcement learning is proposed, which specifically includes the following steps:

[0028] S1. Define the specific mission objectives of satellite exploration and control, and build a simulation environment that simulates the physical state of the satellite in space, including position, velocity, and attitude;

[0029] S2. Design the state space and action space. The state space should contain all necessary satellite state information, and the action space defines all possible actions that the satellite can take.

[0030] S3. Construct multiple state value functions and action value functions, each of which is optimized for different objectives, that is, design a multi-objective reward function with dynamic weights, and dynamically adjust the parameters of the reward function based on the data collected during the task execution;

[0031] S4. Use the training environment to train the deep reinforcement learning model. The model selects an action to execute based on the current strategy. The environment responds to the action and gives a new state and reward. The model updates the strategy accordingly.

[0032] S5. Deploy the trained deep reinforcement learning model to the actual satellite control system. The model makes autonomous decisions and executes corresponding control actions based on the real-time satellite status information.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention realizes effective control in different environments and emergencies through continuous learning and optimization. It does not need to rely on precise prior knowledge such as satellite physical parameters and orbital parameters, and has strong adaptability and autonomous control capabilities.

[0035] 2. In the event of an unstable space environment and possible satellite failures, the present invention can autonomously perform fault diagnosis and recovery operations to ensure the normal operation of the satellite, and significantly improve the efficiency of mission execution by optimizing mission planning and scheduling strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a block diagram of a satellite exploration control system based on deep reinforcement learning of the present invention;

[0037] Figure 2 This is a flow chart of the satellite exploration control method based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0038] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0039] As attached Figure 1 As shown:

[0040] Embodiment 1: The present invention provides a satellite exploration control system based on deep reinforcement learning, including a perception layer, a decision layer and an execution layer; the perception layer is used to integrate data from various devices, including data from high-resolution cameras, infrared sensors, weather radars and other devices, and perform preliminary processing; the decision layer is used to deploy a deep reinforcement learning model on an onboard computer, and regularly perform data training and model updates; the execution layer is used to control satellite attitude and orbit adjustment, and integrates fault detection and autonomous recovery mechanisms, so that the satellite can autonomously perform fault diagnosis and recovery operations when facing unexpected situations;

[0041] The perception and data processing unit integrates multi-source data to provide the decision-making layer with accurate environmental models and dynamic information, and performs real-time data processing to reduce data transmission volume and delay and improve response speed. The decision-making and learning unit makes real-time decisions based on the deep reinforcement learning model in orbit and updates the model through data interaction between the ground station and the satellite. The execution and autonomous control unit executes specific operation commands issued by the decision-making layer and automatically detects and responds to various fault conditions. The optimization unit is used to extract high-level features from raw sensor data and decompose complex control tasks into simple subtasks, with each layer focusing on different control objectives.

[0042] The deep learning model is constructed using the soft actor-critic algorithm. The Actor network is used to generate the action policy network input, which is the state s, and the output is the probability distribution of action a. The network parameters θ are optimized through policy gradient to maximize the combination of reward and entropy. The Critic network is used to evaluate the value of the selected action. Its input is also the state s, and the output is the action value Q(s, a). The network parameters ω are optimized through gradient descent to minimize the difference with the actual reward.

[0043] By adding an entropy term H(.) to the objective function, the learning process is accelerated and the robustness of the final strategy is improved. The objective function is expressed as:

[0044] J(θ)=E[r(s,a)+αH(.)]

[0045] Among them, r(s,a) represents the immediate reward, α represents the temperature parameter, which is used to adjust the importance of entropy, and H(.) represents the entropy of the policy output action distribution; the optimization of its strategy is to maximize the objective function J(θ) by adjusting the parameter θ, and the temperature parameter α is adjusted with training to balance the trade-off between exploration and development by calculating the policy gradient and applying gradient ascent; the optimization of the value function is to minimize the mean square error loss by adjusting the parameter ω, which is expressed as:

[0046]

[0047] Among them, V(s′; ω′) represents the target value network; during training, actions are sampled according to the current policy network and a noise distribution to ensure the adequacy of exploration; and the algorithm uses a double delayed playback mechanism to store and update state transition samples, improve sample utilization efficiency and enhance the stability of the algorithm. By optimizing the strategy by maximizing the trade-off between reward and entropy, more stable and robust learning is achieved, thus solving the complex action decision-making problem in satellite exploration, enabling satellites to autonomously learn and adapt in a changing space environment, thereby improving the efficiency and reliability of mission execution.

[0048] The perception and data processing unit includes a multi-source data fusion module and a real-time data processing module. The multi-source data fusion module is deployed in the onboard computer and is used to receive data from different sensors, perform pre-processing and fusion, and form a comprehensive environmental perception. The real-time data processing module is also deployed in the onboard computer and uses edge computing technology to perform preliminary data processing, reducing the data transmission requirements of the ground station and reducing communication delays.

[0049] The preprocessing steps in the multi-source data fusion module include: S101, geometrically correcting the data from different sensors so that they are aligned in the same spatial coordinate system; S102, performing atmospheric correction to eliminate the influence of the atmosphere on the sensor data; S103, performing radiation correction on the sensor data so that the data reflects the actual surface reflectivity; S104, aligning the images obtained by different sensors in space through feature matching and spatial transformation, and performing spatial registration, with the registration accuracy within 1 to 2 pixels;

[0050] The fusion steps include: S201, extracting terrain, meteorological conditions and ground target features from the preprocessed data; S202, splicing, weighting or combining the extracted features to form a rich feature vector, which helps to integrate the information of different sensors and improve the accuracy of data analysis; S203, integrating the outputs of multiple independent decision modules by weighted average; S204, integrating the prediction results of multiple models through Boosting ensemble learning to obtain the final prediction. The multi-source data fusion module ensures the effective integration of different sensor data through the above preprocessing and fusion steps, thereby providing a comprehensive environmental perception for the satellite exploration system, enabling the system to more accurately understand and interpret complex data from different sensors.

[0051] The decision-making and learning unit deploys the DRL model in the onboard computer, where the lightweight deep reinforcement learning model makes real-time decisions on orbit. The multi-source data fusion module fuses features from different sources to form a comprehensive feature representation, and converts the fused features into state vectors processed by the DRL model. Each state vector contains comprehensive environmental information, and then the encoded state vector is input into the Actor network and Cr itic network of the soft actor-critic algorithm. Finally, the model is updated through data interaction between the ground station and the satellite. The ground station regularly uploads the improved model parameters to the satellite to introduce the latest training results.

[0052] Based on the current state, the Actor network outputs an action probability distribution, which represents the possibility of different actions. A specific action is sampled from the action probability distribution for the satellite to execute. The satellite control system receives the selected action, converts it into specific control instructions, such as thruster ignition, attitude adjustment, etc., and executes it. According to the satellite state changes and mission objectives after the action is executed, the immediate reward is calculated, such as moving toward the target may get a positive reward, and deviating from the orbit may get a negative reward. At the same time, the current state of the satellite is updated as input for the next decision.

[0053] The state, action, reward and subsequent state of each decision are stored in the experience replay buffer for offline training of the model; the Critical network estimates the value of the action based on the current state and immediate reward, and calculates the time difference error; the time difference error is used to update the parameters of the Actor network through the gradient descent optimization algorithm to optimize the strategy; and the time difference error is used to update the parameters of the Critical network to optimize the value function. In this way, the deep reinforcement learning model can make real-time decisions on orbit, so that the satellite can autonomously adapt to the complex space environment and effectively perform its mission.

[0054] The execution and autonomous control unit receives control instructions from the decision layer through the control execution module. The instructions are generated based on the deep learning model and the soft actor-critical algorithm, and contain specific action targets and optimized control parameters. By decoding the instructions, they are converted into control signals that can be operated; during the decoding process, the instruction type, target action and parameters are identified; according to the decoded instructions, corresponding control signals are generated, including clock signals, read / write signals, address selection signals, etc., which are used to control various actuators on the satellite, such as thrusters, attitude control systems, etc. Different types of instructions require different control signal combinations. For example, the thruster ignition instruction may require signals such as ignition time and thrust size, while the attitude adjustment instruction requires the offset and action time signals of each attitude control actuator; after the action is executed, the sensors on the satellite collect new status data in real time, including position, speed and attitude, and feed it back to the execution and autonomous control unit; by analyzing the feedback data, it is determined whether the current state is consistent with the instruction target. If there is a deviation, the control signal is adjusted to eliminate the error and ensure accurate execution. The above steps ensure that the operation commands issued by the decision layer are executed accurately.

[0055] The execution and autonomous control unit also collects various operating data in real time from sensors on the satellite, such as temperature, voltage, current, attitude, etc., as well as operating status information of each subsystem and equipment and log information during network operation, and classifies and analyzes the information; by comparing historical normal data and thresholds, it identifies abnormal data that exceeds the normal range and marks it as a potential fault; based on the predefined rule base, it matches the identified abnormal data with rules, automatically gives preliminary diagnostic results, and uses detailed mathematical models of each satellite system to compare telemetry data with model expectations to further confirm the location and cause of the fault; when a fault is diagnosed, an alarm message is immediately generated through the autonomous fault handling module, including the fault location, type, and urgency; the alarm information is transmitted to the ground control center in real time through the satellite communication system; according to the diagnostic results, the repair operation is performed, and after the repair, the relevant data is re-collected to verify whether the satellite status has returned to normal to ensure that the fault has been effectively resolved.

[0056] The decision-making and learning unit also constructs multiple state value functions and action value functions, each of which is optimized for different goals, that is, the overall task goal is decomposed into several specific sub-task goals, and a state value function V is constructed for each sub-task goal. π (s) or action-value function Q π (s, a), which is optimized according to a specific task goal; at each step, the system receives multiple reward signals r according to the contribution of its behavior to each subtask goal i , and normalize the reward signal, and then pass the dynamic weight ω i Perform weighted summation to form the final comprehensive reward signal: r = ∑ i ω i ·r i , where the weight ω i Dynamically adjust according to task requirements; and use PBRS technology to increase or decrease the immediate rewards of certain states without changing the optimal strategy by assigning a "potential energy" value to each state; in multi-objective optimization, seek to reach the Pareto optimal frontier, that is, to achieve a balance between all goals, at which time no goal can be further improved without deteriorating other goals; at the same time, by real-time monitoring of different task requirements and environmental changes, dynamically adjust the weight of each sub-task goal ω i , ensuring that the reward signal can accurately reflect the most important mission goal at the moment. At the same time, the human intervention interface is retained, allowing ground control personnel to adjust the reward function when necessary to guide the satellite to make more reasonable decisions and increase the controllability of the system.

[0057] The optimization unit uses deep learning technology to extract high-level features from raw sensor data, reduce dimensional disasters, improve decision-making efficiency, and reduce computing resource consumption. It also uses efficient data compression algorithms to reduce bandwidth requirements for state information transmission to ground stations, reduce communication costs, and improve data transmission efficiency. At the same time, it discretizes continuous action space, simplifies the decision-making process, ensures sufficient control accuracy, and meets the accuracy requirements of task execution. It also uses a hierarchical control framework to decompose complex control tasks into simple subtasks, with each layer focusing on different control objectives, simplifying the learning process and improving control efficiency. This ensures that the satellite exploration control system can complete tasks efficiently and accurately in a complex and changing space environment, and has the ability to self-learn and adapt to new challenges.

[0058] As attached Figure 2 As shown:

[0059] Embodiment 2: The present invention also provides a satellite exploration control method based on deep reinforcement learning, which specifically includes the following steps:

[0060] S1. Define the specific mission objectives of satellite exploration and control, and build a simulation environment that simulates the physical state of the satellite in space, including position, velocity, and attitude;

[0061] S2. Design the state space and action space. The state space should contain all necessary satellite state information, and the action space defines all possible actions that the satellite can take.

[0062] S3. Construct multiple state value functions and action value functions, each of which is optimized for different objectives, that is, design a multi-objective reward function with dynamic weights, and dynamically adjust the parameters of the reward function based on the data collected during the task execution;

[0063] S4. Use the training environment to train the deep reinforcement learning model. The model selects an action to execute based on the current strategy. The environment responds to the action and gives a new state and reward. The model updates the strategy accordingly.

[0064] S5. Deploy the trained deep reinforcement learning model to the actual satellite control system. The model makes autonomous decisions and executes corresponding control actions based on the real-time satellite status information.

[0065] Additionally, in order to provide a concise description of exemplary embodiments, all features of an actual embodiment (ie, those features that are not relevant to the best mode presently contemplated for carrying out the invention or those that are not relevant to implementing the invention) may not be described.

[0066] It should be understood that in the development of any actual implementation, as in any engineering or design project, numerous implementation-specific decisions may be made. Such a development effort may be complex and time-consuming, but for those of ordinary skill having the benefit of this disclosure, without undue experimentation, the development effort will be a routine task of design, fabrication, and production.

[0067] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A satellite exploration control system based on deep reinforcement learning, characterized in that: It includes a perception layer, a decision layer and an execution layer; the perception layer is used to integrate the data of each device and perform preliminary processing; the decision layer is used to deploy a deep reinforcement learning model and regularly perform data training and model updates; the execution layer is used to control the satellite attitude and orbit adjustment, and perform autonomous fault diagnosis and recovery operations; The perception and data processing unit integrates multi-source data to provide the decision-maker with accurate environmental models and dynamic information, and performs real-time data processing; The decision-making and learning unit uses the deep reinforcement learning model to make real-time decisions on orbit and updates the model through data interaction between the ground station and the satellite; Execution and autonomous control unit, which executes specific operation commands issued by the decision-making layer and automatically detects and responds to various fault conditions; An optimization unit is used to extract high-level features from raw sensor data and decompose complex control tasks into simple subtasks, with each layer focusing on a different control objective.

2. A satellite exploration control system based on deep reinforcement learning as claimed in claim 1, characterized in that: The deep learning model is constructed using a soft actor-critic algorithm, in which the Actor network is used to generate the action policy network input, which is the state s, and the output is the probability distribution of the action a. The network parameter θ is optimized by policy gradient to maximize the combination of reward and entropy; the Critic network is used to evaluate the value of the selected action. The input is also the state s, and the output is the action value Q(s, a). The network parameter ω is optimized by gradient descent to minimize the difference with the actual reward; The learning process is accelerated by adding an entropy term H(.) to the objective function, which is expressed as: J(θ)=E[r(s,a)+αH(.)] Among them, r(s,a) represents the immediate reward, α represents the temperature parameter, which is used to adjust the importance of entropy, and H(.) represents the entropy of the policy output action distribution; the optimization of its strategy is to maximize the objective function J(θ) by adjusting the parameter θ, and the temperature parameter α is adjusted with training to balance the trade-off between exploration and development by calculating the policy gradient and applying gradient ascent; the optimization of the value function is to minimize the mean square error loss by adjusting the parameter ω, which is expressed as: L(ω)=E[(Q(s,a;ω)-(r(s,a)+V(s′;ω′)) 2 )] where V(s′;ω′) denotes the target value network; during training, actions are sampled from the current policy network and a noise distribution; and the algorithm uses a double delayed replay mechanism to store and update state transition samples.

3. A satellite exploration control system based on deep reinforcement learning as claimed in claim 1, characterized in that: The perception and data processing unit includes a multi-source data fusion module and a real-time data processing module. The multi-source data fusion module is deployed in the onboard computer, and is used to receive data from different sensors, and perform pre-processing and fusion to form a comprehensive environmental perception; the real-time data processing module is also deployed in the onboard computer, and uses edge computing technology to perform preliminary data processing, thereby reducing the data transmission requirements for the ground station and reducing communication delays.

4. A satellite exploration control system based on deep reinforcement learning as claimed in claim 3, characterized in that: The preprocessing steps in the multi-source data fusion module include: S101, geometrically correcting the data from different sensors so that they are aligned in the same spatial coordinate system; S102, performing atmospheric correction to eliminate the influence of the atmosphere on the sensor data; S103, performing radiation correction on the sensor data so that the data reflects the actual surface reflectivity; S104, aligning the images obtained by different sensors in space through feature matching and spatial transformation, and performing spatial registration, with the registration accuracy within 1 to 2 pixels; The fusion steps in the multi-source data fusion module include: S201, extracting terrain, meteorological conditions and ground target features from the preprocessed data; S202, splicing, weighting or combining the extracted features to form a rich feature vector; S203, integrating the outputs of multiple independent decision modules by weighted averaging; S204, integrating the prediction results of multiple models through Boosting ensemble learning to obtain the final prediction.

5. A satellite exploration control system based on deep reinforcement learning as claimed in claim 1, characterized in that: The decision-making and learning unit deploys the DRL model in the onboard computer, wherein the lightweight deep reinforcement learning model makes real-time decisions on orbit, and the multi-source data fusion module fuses features from different sources to form a comprehensive feature representation, and converts the fused features into state vectors processed by the DRL model. Each state vector contains comprehensive environmental information, and then the encoded state vector is input into the Actor network and Critic network of the soft actor-critic algorithm. Finally, the model is updated through data interaction between the ground station and the satellite. The ground station regularly uploads the improved model parameters to the satellite to introduce the latest training results. Based on the current state, the Actor network outputs an action probability distribution, which represents the possibility of different actions. A specific action is sampled from the action probability distribution for the satellite to execute. The satellite control system receives the selected action, converts it into a specific control instruction, and executes it. Based on the satellite state change and mission objectives after the action is executed, the immediate reward is calculated, and the current state of the satellite is updated as input for the next decision. The state, action, reward and subsequent state of each decision are stored in the experience replay buffer for offline training of the model. The Critic network estimates the value of the action based on the current state and immediate reward, and calculates the time difference error. The time difference error is used to update the parameters of the Actor network through the gradient descent optimization algorithm for strategy optimization. The time difference error is also used to update the parameters of the Critic network for value function optimization.

6. A satellite exploration control system based on deep reinforcement learning as claimed in claim 1, characterized in that: The execution and autonomous control unit receives control instructions from the decision layer through the control execution module. The instructions are generated based on the deep learning model and the soft actor-critic algorithm, and contain specific action targets and optimized control parameters. The instructions are decoded and converted into control signals that can be operated. During the decoding process, the instruction type, target action and parameters are identified. According to the decoded instructions, corresponding control signals are generated. Different types of instructions require different control signal combinations. After the action is executed, the sensors on the satellite collect new status data in real time, including position, speed and attitude, and feed them back to the execution and autonomous control unit. By analyzing the feedback data, it is determined whether the current state is consistent with the instruction target. If there is a deviation, the control signal is adjusted to eliminate the error.

7. A satellite exploration control system based on deep reinforcement learning as claimed in claim 6, characterized in that: The execution and autonomous control unit also collects various types of operation data in real time from sensors on the satellite, collects operation status information of each subsystem and equipment, and log information during network operation, and classifies and analyzes the information; identifies abnormal data that exceeds the normal range by comparing historical normal data and thresholds, and marks them as potential faults; based on a predefined rule base, matches rules on the identified abnormal data, and automatically gives preliminary diagnosis results. When a fault is diagnosed, an alarm information is immediately generated through an autonomous fault processing module, including the fault location, type, and urgency; the alarm information is transmitted to the ground control center in real time through the satellite communication system; based on the diagnosis results, a repair operation is performed, and after the repair, relevant data is re-collected to verify whether the satellite status has returned to normal.

8. A satellite exploration control system based on deep reinforcement learning as claimed in claim 5, characterized in that: The decision-making and learning unit also constructs multiple state value functions and action value functions, each function is optimized for different goals, that is, the overall task goal is decomposed into several specific sub-task goals, and a state value function V is constructed for each sub-task goal. π (s) or action-value function Q π (s, a), which is optimized according to a specific task goal; at each step, the system receives multiple reward signals r according to the contribution of its behavior to each subtask goal i , and normalize the reward signal, and then pass the dynamic weight ω i Perform weighted summation to form the final comprehensive reward signal: r = ∑ i ω i ·r i , where the weight ω i Dynamically adjust according to task requirements; and use PBRS technology to increase or decrease the immediate rewards of certain states without changing the optimal strategy by assigning a "potential energy" value to each state; in multi-objective optimization, seek to reach the Pareto optimal frontier, that is, to achieve a balance between all goals, at which time no goal can be further improved without deteriorating other goals; at the same time, by real-time monitoring of different task requirements and environmental changes, dynamically adjust the weights of each sub-task goal i .

9. A satellite exploration control system based on deep reinforcement learning as claimed in claim 1, characterized in that: The optimization unit uses deep learning technology to extract high-level features from raw sensor data and compresses data transmission to reduce bandwidth requirements; at the same time, it discretizes the continuous action space to simplify the decision-making process, and adopts a hierarchical control framework to decompose complex control tasks into simple subtasks, with each layer focusing on different control objectives to simplify the learning process.

10. A satellite exploration control method based on deep reinforcement learning, characterized in that: The specific steps include: S1. Define the specific mission objectives of satellite exploration and control, and build a simulation environment that simulates the physical state of the satellite in space, including position, velocity, and attitude; S2. Design the state space and action space. The state space should contain all necessary satellite state information, and the action space defines all possible actions that the satellite can take. S3. Construct multiple state value functions and action value functions, each of which is optimized for different objectives, that is, design a multi-objective reward function with dynamic weights, and dynamically adjust the parameters of the reward function based on the data collected during the task execution; S4. Use the training environment to train the deep reinforcement learning model. The model selects an action to execute based on the current strategy. The environment responds to the action and gives a new state and reward. The model updates the strategy accordingly. S5. Deploy the trained deep reinforcement learning model to the actual satellite control system. The model makes autonomous decisions and executes corresponding control actions based on the real-time satellite status information.

Images