Multi-satellite earth observation hidden risk scheduling method based on risk perception reinforcement learning

Abstract

The application discloses a multi-satellite earth observation hidden risk scheduling method based on risk perception reinforcement learning, and belongs to the technical field of intelligent task scheduling of earth observation satellites and space resource collaborative planning. By constructing a hidden risk scheduling environment containing cloud layer obstruction, grid electromagnetic interference and satellite health degradation, a dynamic observation opportunity of a multi-satellite and a ground observation task is generated; based on satellite local observation and a historical execution feedback sequence, potential risk belief is inferred and current task imaging success probability is predicted; the risk belief is introduced into a multi-agent reinforcement learning scheduling model, and a risk perception correction reward is constructed according to the predicted success probability, so that the imaging benefits, resource consumption and failure risk are collaboratively balanced. After training is completed, on-orbit satellites are deployed in a light weight mode, and autonomous decision-making of the satellites under the hidden risk state of the satellites is realized. According to the application, each satellite can avoid high-risk observation tasks in an uncertain environment, and the comprehensive observation benefits and imaging resource utilization rate of a low-orbit remote sensing satellite constellation are improved.

Description

Technical Field

[0001] This invention relates to the fields of intelligent mission scheduling for Earth observation satellites, multi-agent reinforcement learning, and collaborative planning of space resources, and particularly to a method for scheduling hidden risks in multi-satellite Earth observation based on risk perception reinforcement learning. Background Technology

[0002] Earth observation remote sensing satellites are core spatial information acquisition carriers for disaster emergency monitoring, natural resource surveys, and national security management. With the large-scale networking of low-Earth orbit satellite constellations, satellite scheduling has shifted from traditional single-satellite offline static planning to an online scheduling mode that involves multi-satellite collaboration and real-time response to dynamic mission opportunities. Multi-satellite observation scheduling is a typical resource-constrained temporal combinatorial optimization problem, with constraints including multiple coupled conditions such as mission priority, satellite transit observation window, imaging energy consumption, attitude maneuver limitations, and maximum number of imaging attempts per cycle.

[0003] Real-world on-orbit observation scenarios present multiple hidden uncertainties and risks, and these risks are difficult for satellites to perceive directly during the scheduling and decision-making phase. Feedback can only be generated post-event based on the effectiveness of the imaging operation: 1) Optical satellite imaging is susceptible to cloud cover, potentially leading to invalid or degraded observation data; 2) Ground / space electromagnetic interference reduces the reliability of payload imaging and satellite-to-ground data transmission; 3) After multiple imaging operations, the payload, energy, and attitude control systems continuously degrade, and the mission success rate gradually decreases with each imaging attempt. These three types of risks collectively determine whether a single imaging operation yields effective observational benefits, and current scheduling technologies still have shortcomings. (1) Traditional mathematical programming and heuristic scheduling algorithms rely on fixed preset rules, static prior risk information or relatively certain task models, making it difficult to make full use of on-orbit real-time imaging feedback to dynamically infer hidden risks; facing dynamically changing cloud fields, electromagnetic hotspots and satellite health status, they have weak adaptive capabilities, and high-risk grid areas may generate a large number of invalid images, consuming the satellite's limited imaging resources. (2) Existing multi-agent reinforcement learning satellite scheduling schemes (standard MAPPO, deep Q network, etc.): rely solely on local satellite observations at the current moment to make decisions, do not model historical imaging feedback time series, make it difficult to effectively mine the spatial neighborhood correlation of clouds and electromagnetic interference and the time series pattern of satellite health degradation, lack risk perception optimization mechanism, and cannot actively weigh the mission benefits against the resource consumption cost caused by imaging failure. (3) There is still a gap between the existing risk modeling scheme and the real on-orbit scheduling constraints: Some existing risk modeling methods usually use cloud cover, electromagnetic intensity or mission success probability as external known information to directly participate in scheduling, or treat risk factors as static constraints. This is still different from the scenario in real on-orbit scheduling where the risk state is difficult to observe directly and can only be judged indirectly through imaging feedback. Summary of the Invention

[0004] The purpose of this invention is to propose a hidden risk scheduling method for multi-satellite Earth observation based on risk-aware reinforcement learning. This method constructs a hidden risk environment integrating real ERA5 cloud cover, gridded electromagnetic interference, and satellite health degradation. It unifies the online mission-acceptance type multi-satellite scheduling problem into a Hidden-Risk Dec-POMDP distributed partially observable Markov decision model. A self-supervised BeliefNet network is designed to infer potential risk belief representations and predict the current mission's imaging success probability from the satellite's historical execution feedback time-series. Based on the risk belief representation, the MAPPO multi-agent reinforcement learning framework is improved to achieve Actor-Critic network belief conditionation, and a risk-aware correction reward is constructed based on the predicted imaging success probability. The model is trained offline using a centralized training and decentralized execution CTDE paradigm. During the on-orbit phase, each satellite independently outputs imaging / skip decisions based on local observations and its own historical feedback. This maximizes the long-term observational total benefit and reduces satellite resource consumption caused by ineffective imaging without directly obtaining the actual hidden risk values.

[0005] To achieve the above objectives, this invention provides a multi-satellite Earth observation hidden risk scheduling method based on risk-aware reinforcement learning, comprising the following steps: Step S1: Construct a scheduling environment that integrates hidden risks such as cloud cover, gridded electromagnetic interference, and satellite health degradation, and generate time-by-time satellite-mission observation opportunities based on DQG unified gridded mission representation and SGP4 orbit propagation model; Step S2: The opportunity-accepting scheduling problem of multi-satellite online missions under hidden risks is formalized into a Hidden-Risk Dec-POMDP distributed partially observable Markov decision model, defining the global state, local satellite observations, binary action space, hidden risk-driven state transition rules, and multi-satellite cooperative global reward function. Step S3: Construct a fixed-window-length execution feedback historical time series for each satellite. The sequence stores historical mission DQG grid spatial structure features, historical scheduling actions, and historical imaging success and failure feedback, serving as temporal evidence input for inferring hidden risks. Step S4: Construct a self-supervised BeliefNet risk inference network, using the current local observations of the satellite and the historical sequence of its own execution feedback as joint inputs, and output a low-dimensional potential risk belief representation and a predicted value of the current mission imaging success probability; construct a binary cross-entropy self-supervised loss based on the binary success and failure feedback after the imaging action to complete the self-supervised update of BeliefNet parameters; Step S5: Construct the risk perception belief conditional RAB-MAPPO multi-agent reinforcement learning framework, input the potential risk belief representation output by BeliefNet into the Actor policy network and Critic value network respectively to realize belief conditionalization; construct a modified reward with risk constraints based on the predicted imaging success probability; Step S6: Adopt the centralized training and decentralized execution CTDE paradigm to jointly optimize the parameters of the three types of networks, BeliefNet, Actor, and Critic, and iterate the training until the scheduling model converges; Step S7: Prune the converged model, retaining only the lightweight BeliefNet inference module and the Actor policy network, and distribute them to the onboard embedded processing units of each satellite. During the on-orbit phase, each satellite relies on local real-time observations and its own independently maintained execution feedback history cache to distribute output imaging or skip mission decisions, thereby achieving collaborative online observation and scheduling of hidden risks in the multi-satellite constellation.

[0006] Preferably, in step S1, constructing the hidden risk scheduling environment specifically includes: S11: DQG grid is used to uniformly represent ground point, line and area observation targets. Based on the USM unified scheduling model, various original observation targets are transformed into standardized grid task units and a standardized task attribute set is generated. S12: Read the two lines of orbital elements of the satellite TLE, calculate the spatial position of the satellite at discrete scheduling time through the SGP4 orbital propagation model, transform it to the ground-fixed coordinate system, and then combine it with the satellite field of view constraint to match the DQG grid task to generate time-by-time satellite-mission observation opportunities. S13: Hidden risks are modeled in three categories. All true and effective imaging probabilities are calculated only within the hidden risk scheduling environment and are not disclosed to satellites. ① Cloud cover risk: Import the ERA5 global reanalysis cloud field and perform spatial interpolation on the task DQG grid location to obtain the real-time total cloud cover of the grid. Linear mapping to calculate the effective imaging probability of clouds ; ② Mesh-based electromagnetic interference risk: Set up a spatially distributed set of electromagnetic interference hotspots, calculate the electromagnetic interference intensity of the task grid based on the Gaussian distance attenuation function, and map it to obtain the electromagnetic effective imaging probability. ; ③ Satellite health degradation risk: A linear degradation model is established based on the cumulative number of satellite imaging attempts, and a random health perturbation term is superimposed to calculate the probability of effective imaging of a single satellite in good health. ; S14. Calculate the success probability of true imaging in a single task using multiplicative fusion. Hidden risk scheduling environment based on Bernoulli distribution Sampling generates imaging success or failure results Hidden risk status ,reality It is not included in local satellite observations, but maintains the observable constraints of the scene.

[0007] Preferably, in step S2, the Hidden-Risk Dec-POMDP model is defined as follows: Unified interpretation of all parameters: Hidden risk scheduling environment involves participants in collaborative scheduling A collection of satellite intelligent agents; : Global state space, time step Global state ; To conceal the completion status of all grid tasks in the risk scheduling environment, This is a vector representing the cumulative number of images captured by each satellite. For the current discrete scheduling time, To conceal the unobservable hidden risk states generated by the risk scheduling environment; :satellite In the binary action space, action 0 represents skipping the current observation opportunity, and action 1 represents performing imaging. The action output is sent to the hidden risk scheduling environment to complete the state transition. : Hidden risk-driven state transition function; after the satellite performs imaging actions, it calls the hidden risk scheduling environment. Sampling imaging results Update task completion status Cumulative number of satellite imaging times Generate the global state for the next time step. ; :satellite Local observation space; local observation vector Includes mission DQG grid spatial features, mission priority, satellite-mission observation geometric imaging quality, imaging energy consumption, current cumulative number of satellite imaging attempts, scheduling progress, and remaining scheduling time; local observation vectors. It does not include any real-world risk probability parameters related to cloud cover, electromagnetic fields, or satellite health. : Multi-satellite cooperative global reward function; single-satellite basic reward distinguishes between three scenarios: successful imaging, failed imaging, and skipped mission; global single-step reward is the arithmetic mean of the single-step basic rewards of all satellites; : Reinforcement learning discount factor, used to weight long-term time-series returns.

[0008] Preferably, in step S3, the satellite At any moment The execution feedback history expression is: ; Unified explanation of all parameters: : The execution feedback history has a fixed window length, which is a preset hyperparameter; A historic moment The spatial structure features derived from the DQG grid for the corresponding observation task include grid hierarchy, neighborhood topology, and spatial adjacency relationships, which are used to characterize the spatial correlation of risks. A historic moment satellite Selected scheduling action; Historical imaging feedback markers This indicates successful imaging. This indicates imaging failure; if no imaging action was performed at a historical moment, there will be no effective imaging feedback. When the cumulative interactive sample length of the satellite is less than the window At that time, zero vector padding is used to fill in the sequence length to ensure that the input dimension of BeliefNet is fixed and uniform.

[0009] Preferably, in step S4, BeliefNet uses joint features As input, output a low-dimensional representation of potential risk beliefs. Predicted probability of successful imaging for the current mission The BeliefNet self-supervised binary cross-entropy loss function is: ; In the formula: For the training batch sample set; To effectively provide feedback on indicator variables, the satellite at time... When performing imaging actions and obtaining imaging success or failure feedback If imaging is not performed... It is used to filter out invalid training samples that do not perform imaging actions.

[0010] Preferably, in step S5, the belief-conditional Actor-Critic network is constructed as follows: Actor strategy network input splicing satellite local observations Compared with BeliefNet's output of risk belief representation Output satellite action probability distribution , These are the parameters of the Actor network; Critic value network input global state Local satellite observations Risk belief representation Output global state value estimate , These are the parameters for the Critic network.

[0011] Preferably, in step S5, the formula for calculating the risk perception correction reward is: ; In the formula: The satellite single-step reward is corrected based on risk perception. As a reward for the original satellite; To preset the risk perception coefficient; It is a binary indicator function, which is only used when the satellite selects an imaging action. Overlapping risk correction items; Output the predicted imaging success probability for BeliefNet; when When the time correction term is positive, it incentivizes the satellite to perform the current low-risk imaging mission; when The time correction term is negative to suppress the satellite from performing high-risk imaging missions.

[0012] Preferably, in step S5, the overall joint optimization loss function of RAB-MAPPO is: ; In the formula: For PPO shearing strategy loss; The mean squared error loss of the Critic value; The BeliefNet self-supervised cross-entropy loss is used. This is a policy entropy regularization term, used to maintain policy exploration capabilities; These are the value loss weight, belief loss weight, and entropy regularization term weight coefficients, respectively.

[0013] Preferably, in step S6, the complete offline model training process is as follows: S61. Initialize all parameters of the Actor network, Critic network, and BeliefNet network, and initialize the global interactive experience cache and the independent execution feedback history cache for each satellite. S62, Episode-by-Episode training: Reset the completion status of all tasks, the cumulative imaging resource status of each satellite, and the historical feedback cache of all satellites within the hidden risk scheduling environment; traverse all discrete moments within the scheduling cycle. ; S63, Single-moment interaction: Reading local observations for each satellite. Read its own execution feedback history Risk beliefs are obtained by inputting BeliefNet inference. Predicting the probability of success Actor networks are based on Sampling satellites' independent actions; integrating all satellite actions to form a joint action input hidden risk scheduling environment; S64, Hidden Risk Scheduling Environment Returns Satellite to Original Basic Reward Imaging success or failure markers The global state at the next moment; based on Calculate risk perception correction reward The state transition samples are stored in the global experience cache; the satellite's own execution feedback history sequence is updated synchronously. S65. After a single episode interaction is completed, the reward is adjusted based on risk perception. Calculate the GAE advantage function and discounted return; update the Actor policy network, Critic value network, and BeliefNet risk inference network in sequence; clear the global experience cache and proceed to the next episode iteration; S66. Repeat the iteration until the global average reward fluctuation of multiple consecutive episodes is less than the preset threshold, then determine that the model has converged and end the offline training.

[0014] Preferably, in step S7, the on-orbit online scheduling and deployment method is as follows: after the offline model converges, the Critic value network used only in the centralized training phase is pruned, and only the lightweight BeliefNet inference module and Actor policy network are retained, and distributed to the on-board embedded processing unit of each satellite; when the satellite is running in orbit, it independently calculates its own and ground grid mission's observation opportunities in real time, maintains a fixed-length execution feedback history cache locally, autonomously calls BeliefNet inference risk beliefs and outputs imaging or skip decisions through Actor; the entire process does not require a centralized on-board control node, and multiple satellites coordinate in a distributed manner to complete the scheduling of Earth observations under hidden risks.

[0015] Therefore, the multi-satellite Earth observation hidden risk scheduling method based on risk perception reinforcement learning described above has the following advantages: (1) This invention integrates three types of real-world scenarios: cloud layer, gridded electromagnetic field, and satellite health degradation. It constructs an environment based on the measured cloud amount of ERA5. The risks have spatial neighborhood and temporal degradation rules. The simulation scenario fits the actual satellite on-orbit engineering and solves the defects of traditional simulation risks that are completely random and detached from the real observation environment. (2) This invention is strictly based on the Hidden-Risk Dec-POMDP modeling and scheduling problem, fully distinguishes between observable satellite mission information and unobservable hidden risk status, and better matches the real constraints that risks cannot be read in advance during the on-orbit stage and can only be obtained after imaging feedback. The theoretical framework of the model is consistent with the logic of the engineering scenario. (3) Design an execution feedback historical time series with spatial grid features, and match it with self-supervised BeliefNet to complete risk reasoning by relying only on binary feedback of imaging success and failure. The actual risk probability inside the environment is not input throughout the process, which effectively explores the spatial risk distribution of the grid area and the temporal degradation law of satellite health, and predicts the current mission imaging success rate. (4) A belief-conditional RAB-MAPPO multi-agent reinforcement learning framework is proposed, which embeds risk belief representations into a two-layer network of policy and value, and constructs risk perception correction rewards to actively balance the task benefits and the resource consumption cost of high-risk imaging, reduce the number of invalid imaging attempts, and save the limited imaging resources of the satellite. (5) Adopting a centralized training and distributed execution CTDE architecture, the training of multi-satellite collaborative strategies is completed offline in a unified manner. During the on-orbit phase, each satellite makes independent distributed decisions. Only a lightweight inference network is deployed on the satellite, resulting in low computational overhead. It is suitable for the embedded computing power limitations of low-orbit satellites and can flexibly adapt to remote sensing satellite constellations of different sizes, ranging from 3 to 20.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is an overall flowchart of the multi-satellite Earth observation hidden risk scheduling method based on risk perception reinforcement learning in this embodiment of the invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0020] Example like Figure 1 As shown in the figure, this embodiment proposes a multi-satellite Earth observation hidden risk scheduling method based on risk perception reinforcement learning. The specific steps are as follows: S1. Construct a scheduling environment that integrates hidden risks such as cloud cover, gridded electromagnetic interference, and satellite health degradation, and generate time-by-time satellite-mission observation opportunities based on a unified gridded task representation and satellite orbit propagation model. The process of building a hidden risk scheduling environment includes complete sub-steps: S11. DQG grid is used to complete the unified gridded representation of ground point targets, linear targets and area observation areas. Based on the USM unified scheduling model, various original observation targets are transformed into standardized grid task units, generating a set of standardized task attributes including priority, imaging quality, energy consumption and observation window. S12. Read the orbital elements of the satellite's two TLE lines, calculate the spatial position of all satellites at discrete scheduling times using the SGP4 orbital propagation model; transform the satellite's inertial coordinate system position to the ground-fixed coordinate system, and combine the satellite payload field of view constraint to match the DQG grid task to generate time-by-time satellite-mission observation opportunities. S13. Hidden risks are modeled in three categories. The true and effective imaging probabilities corresponding to all risks are only calculated within the environment and are not disclosed to the satellite agent. ① Cloud cover risk: Import the ERA5 global reanalysis cloud field and perform spatial interpolation based on the task DQG grid location to obtain the real-time total cloud cover of the grid. The effective imaging probability of clouds is calculated using a linear mapping formula. ; ② Mesh-based electromagnetic interference risk: Set up a spatially distributed set of electromagnetic interference hotspots, calculate the electromagnetic interference intensity experienced by the task grid based on the Gaussian distance attenuation function, and map it to obtain the electromagnetic effective imaging probability. ; ③ Satellite health degradation risk: A linear degradation model is established based on the cumulative number of satellite imaging attempts, and a random health perturbation term is superimposed to calculate the probability of effective imaging of a single satellite in good health. ; S14. Calculate the success probability of true imaging in a single task using a multiplicative fusion method. ; The environment is based on Bernoulli distribution pairs Sampling, generating imaging success or failure results Hidden risk status ,reality The entire process is not included in local satellite observations, ensuring that some observable constraints of the scheduling scenario are maintained.

[0021] S2. The opportunity-acceptance scheduling problem of multi-satellite online missions under hidden risks is formalized into a Hidden-RiskDec-POMDP distributed partially observable Markov decision model, which fully defines the model's global state, satellite local observations, binary action space, hidden risk-driven state transition rules, and multi-satellite cooperative global reward function. The complete definition of the Hidden-Risk Dec-POMDP model is... All parameters are explained uniformly: : Participating in coordinated scheduling A collection of satellite intelligent agents; : Global state space, at any time Global state ; This is a set of completion status markers for all grid tasks. This is a vector representing the cumulative number of images captured by each satellite. For the current discrete scheduling time, This represents an unobservable, globally hidden risk state. :satellite In the binary action space, action 0 represents skipping the current observation opportunity and not consuming imaging resources, while action 1 represents performing imaging and consuming the corresponding imaging resources. : Hidden risk-driven state transition function; after the satellite performs imaging actions, the environment is based on Sampling imaging results Synchronously update task completion status Cumulative number of satellite imaging times Generate the global state for the next time step. ; :satellite Local observation space; local observation vector It includes the mission DQG grid spatial features, mission priority, imaging quality, imaging energy consumption, current cumulative number of satellite imaging attempts, scheduling progress, and remaining scheduling time; the local observation vector does not contain any real risk probability parameters related to clouds, electromagnetic fields, or satellite health. : Multi-satellite cooperative global reward function; single-satellite basic reward distinguishes between three scenarios: successful imaging, failed imaging, and skipped mission; global single-step reward is the arithmetic mean of the single-step basic rewards of all satellites; : Reinforcement learning discount factor, used to weight long-term time-series returns.

[0022] S3. Construct a fixed-window-length execution feedback historical time series for each satellite. The sequence uniformly stores historical mission DQG grid spatial structure features, historical scheduling actions, and historical imaging success and failure feedback, serving as the core temporal evidence input for BeliefNet to infer hidden risks.

[0023] satellite At any moment The unified expression for the execution feedback history is: ; Unified explanation of all parameters: : The execution feedback history has a fixed window length, which is a preset hyperparameter; A historic moment The spatial structure features derived from the DQG grid for the corresponding observation task include grid hierarchy, neighborhood topology, and spatial adjacency relationships, which are used to characterize the spatial correlation of risks. A historic moment satellite Selected scheduling action; Historical imaging feedback markers This indicates successful imaging. This indicates imaging failure; if no imaging action was performed at a historical moment, there will be no effective imaging feedback. When the cumulative interactive sample length of the satellite is less than the window At that time, zero vector padding is used to fill in the sequence length to ensure that the input dimension of BeliefNet is fixed and uniform.

[0024] S4. Construct a self-supervised BeliefNet risk inference network, using the current local observations of the satellite and the historical sequence of its own execution feedback as joint inputs, and output a low-dimensional potential risk belief representation and a predicted value of the current mission imaging success probability; construct a binary cross-entropy self-supervised loss based on the binary success and failure feedback obtained after the imaging action is executed, and independently complete the BeliefNet parameter update.

[0025] The input to the self-supervised BeliefNet network is joint features. The network output consists of two parts: a low-dimensional representation of potential risk beliefs. Predicted probability of successful imaging for the current mission BeliefNet uses a recurrent sequence encoder to process the execution feedback historical time-series information, and constructs a self-supervised binary cross-entropy loss based on the binary feedback after the imaging action. The loss function has a unified formula: ; Unified explanation of parameters: For the training batch sample set; To effectively provide feedback on indicator variables, the satellite at time... When performing imaging actions and obtaining imaging success or failure feedback If imaging is not performed... It is used to filter out invalid training samples without imaging actions to avoid interfering with risk inference learning.

[0026] S5. Construct the risk-perception belief-conditionalized RAB-MAPPO multi-agent reinforcement learning framework, input the potential risk belief representations output by BeliefNet into the Actor policy network and the Critic value network respectively, and complete the risk belief conditionalization of the policy and value networks; based on the predicted imaging success probability output by BeliefNet, construct a modified reward with risk perception constraints.

[0027] The risk perception belief conditionalization RAB-MAPPO framework includes a three-layer coupling improvement and unified contextual logic: (1) Belief-conditional Actor-Critic network structure: Actor strategy network input splicing satellite local observations Compared with BeliefNet's output of risk belief representation Output satellite action probability distribution , These are the parameters of the Actor network; Critic value network input global state Local satellite observations Risk belief representation Output global state value estimate , These are the parameters for the Critic network. (2) Formula for constructing risk perception-adjusted rewards: ; Unified explanation of parameters: The satellite single-step reward is corrected based on risk perception. As a reward for the original satellite; To preset the risk perception coefficient; It is a binary indicator function, which is only used when the satellite selects an imaging action. Overlapping risk correction items; Output the predicted imaging success probability for BeliefNet; The correction items are logically consistent: When the time correction term is positive, it incentivizes the satellite to perform the current low-risk imaging mission. The time correction term is negative to suppress the satellite from performing high-risk imaging missions; (3) RAB-MAPPO performs overall joint optimization of the loss function, and simultaneously completes the training of the policy, value, and risk belief networks: ; Unified explanation of parameters: For PPO shearing strategy loss; The mean squared error loss of the Critic value; The BeliefNet self-supervised cross-entropy loss in step S4; This is a policy entropy regularization term, used to maintain policy exploration capabilities; These are the value loss weight, belief loss weight, and entropy regularization term weight coefficients, respectively.

[0028] S6. Adopt the CTDE paradigm of centralized training and decentralized execution, jointly optimize the parameters of the three types of networks: BeliefNet, Actor, and Critic, and iterate the training until the scheduling model converges.

[0029] Step S6: Complete offline model training process with seamless logical loop: S61. Initialize all parameters of the Actor network, Critic network, and BeliefNet network, and initialize the global interactive experience cache and the independent execution feedback history cache for each satellite. S62, Episode-by-Episode training: Reset the completion status of all tasks in the scheduling environment, the cumulative imaging resource status of each satellite, and the historical feedback cache of all satellites; traverse all discrete moments within the scheduling cycle. ; S63, Single-moment interaction process: Read local observations for each satellite. Read its own execution feedback history Risk beliefs are obtained by inputting BeliefNet inference. Predicting the probability of success Actor networks are based on Sampling satellites operate independently; integrating all satellite actions to form a joint action input into a hidden risk environment; S64, Environmental Return: Original Base Reward for Each Satellite Imaging success or failure markers The global state at the next moment; based on Calculate risk perception correction reward The complete state transition sample is stored in the global experience cache; the satellite's own execution feedback history sequence is updated based on the current task grid features, satellite actions, and imaging feedback. S65. After a complete interaction with a single episode, the reward is adjusted based on risk perception. Calculate the GAE advantage function and discounted reward; sequentially update network parameters: update the Actor policy network, update the Critic value network, update the BeliefNet risk inference network; clear the global experience cache and proceed to the next episode iteration; S66. Repeat the iteration until the global average reward fluctuation of multiple consecutive episodes is less than the preset threshold, then determine that the model has converged and end the offline training.

[0030] Step S7: Trim the offline trained model, retaining only the BeliefNet lightweight inference module and the Actor policy network, and distribute them to the on-board embedded processing units of each satellite. During the on-orbit operation phase, each satellite relies solely on local real-time observations and its own independently maintained execution feedback history cache to distribute output imaging / skip task decisions, thereby achieving collaborative online observation and scheduling of hidden risks in the multi-satellite constellation.

[0031] The entire on-orbit online scheduling and deployment process is unified with the offline training context: After the offline model converges, the Critic value network used only in the centralized training phase is pruned, retaining only the lightweight BeliefNet inference module and the Actor policy network, and the lightweight model is distributed to the on-board embedded processing unit of each satellite; when the satellite is in orbit, it independently completes three operations: real-time calculation of its own and ground grid mission observation opportunities, local independent maintenance of a fixed-length execution feedback history cache, and autonomous invocation of BeliefNet inference risk beliefs and output of imaging / skip decisions through Actor; the entire process does not require a centralized on-board control node, and multiple satellites coordinate in a distributed manner to complete the scheduling of Earth observations under hidden risks.

[0032] One specific implementation method is as follows: This embodiment employs a collaborative observation constellation composed of six low-orbit optical remote sensing satellites, with a global scheduling cycle of discrete duration. Seconds, maximum number of imaging cycles per satellite Execution feedback history window length Offline training hardware uses Intel Xeon servers, while on-orbit inference hardware uses onboard ARM embedded processors.

[0033] 1. Setting up a hidden risk environment: The global land observation area is divided into three categories using a 6-level DQG grid. The observation tasks include point targets, linear rivers, and area farmland. Based on the TLE orbital elements of 6 satellites, the satellite positions are propagated second by second using the SGP4 model. Combined with the satellite field of view constraints, the DQG grid is matched to generate time-by-time satellite-mission observation opportunities.

[0034] Cloud risk: Import ERA5 global hourly reanalysis cloud cover data, and obtain the cloud cover at each time point for each DQG grid through grid interpolation. Minimum effective probability of cloud cover , ; Electromagnetic interference: 30 electromagnetic interference hotspots are set globally, and the interference intensity of a single hotspot is... Affecting distance variance km, calculating grid interference intensity based on Gaussian distance attenuation formula, electromagnetic effective probability range. ; Satellite health degradation: initial health level of the satellite Degradation coefficient , superimposed Interval random perturbation, lower bound of healthy and effective probability ; Multiplying these three factors yields the true internal environment. Based on Bernoulli distribution sampling imaging results ,reality The probability of the three categories of sub-risks will not be disclosed to the satellite intelligent agent throughout the entire process.

[0035] 2. Hidden-Risk Dec-POMDP Unified Parameter Configuration: Discount factor Satellite binary action space The satellite's local observation vector has a dimension of 128, fully encompassing grid features, task priority, imaging quality, imaging energy consumption, cumulative imaging count, and scheduling progress. Unified calculation rules for basic rewards: Basic reward for successful imaging Weight parameters Imaging failure penalty coefficient Skipping a task always results in a base reward of 0; the global single-step reward is the average of the single-step rewards for 6 satellites.

[0036] 3. Execution feedback history and BeliefNet network configuration: Historical Window Each historical unit feature vector is 64-dimensional, and zero-vector padding is used when the sequence length is less than H. BeliefNet uses a two-layer LSTM recurrent encoder to process the historical time series sequences, and the back-end fully connected layer outputs a 64-dimensional risk belief representation. One-dimensional prediction probability Self-supervised cross-entropy loss for training batch sample size .

[0037] 4. Unified configuration of RAB-MAPPO network and optimized hyperparameters: Both Actor and Critic employ a three-layer fully connected network; risk perception coefficient PPO shear coefficient Joint loss weights Predicted imaging probability cutoff interval This avoids extreme probability disturbances that could affect training stability.

[0038] 5. Complete process of offline training and on-orbit deployment: The offline training iteration spanned 8000 episodes, with each episode collecting 3600 interaction samples. After each round of sample collection, the Actor, Critic, and BeliefNet networks were updated sequentially. The model convergence criterion was that the global average reward fluctuation was less than 1% over 200 consecutive episodes. After the model converges, the Critic network is removed and only the lightweight BeliefNet+Actor model is distributed to the embedded units of 6 satellites; the satellites complete observation opportunity matching, historical cache update, risk belief inference, and imaging decision output once per second in orbit.

[0039] Comparison of implementation results: Three baseline comparison schemes were set up simultaneously: standard MAPPO multi-agent scheduling, genetic algorithm offline scheduling, and greedy heuristic online scheduling. The method in this embodiment outperformed the baselines in all metrics. (1) Global long-term average cumulative observation gains increased by 27.3%; (2) The total number of invalid imaging attempts decreased by 41.6%, and the utilization rate of the satellite's limited imaging resources was greatly improved; (3) The success rate of imaging missions in high-risk grid areas with high cloud cover and strong electromagnetic interference was improved by 32.1%; (4) Resource conflicts of multi-satellite missions were reduced by 19.2%, and the balance of constellation collaborative scheduling was significantly optimized.

[0040] Implementation and expansion instructions: The parameters of this method can be flexibly adjusted according to the business scenario: satellite constellation size supports 3 to 20 low-Earth orbit satellites; historical window. Risk perception coefficient The joint loss weight can be adjusted according to the type of optical / SAR satellite payload and the needs of routine monitoring / emergency observation operations; it can be adapted to radar satellite scheduling by simply replacing the cloud risk modeling module, and can be adapted to multiple types of Earth observation business scenarios.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for scheduling hidden risks in multi-satellite Earth observation based on risk perception reinforcement learning, characterized in that, Includes the following steps: Step S1: Construct a scheduling environment that integrates hidden risks such as cloud cover, gridded electromagnetic interference, and satellite health degradation, and generate time-by-time satellite-mission observation opportunities based on DQG unified gridded mission representation and SGP4 orbit propagation model; Step S2: Model the multi-satellite online mission opportunity-acceptance scheduling problem under hidden risk as a Hidden-Risk Dec-POMDP distributed partially observable Markov decision model, defining the global state, satellite local observations, binary action space, hidden risk-driven state transition rules, and multi-satellite cooperative global reward function. Step S3: Construct a fixed-window-length execution feedback historical time series for each satellite. The sequence stores historical mission DQG grid spatial structure features, historical scheduling actions, and historical imaging success and failure feedback, serving as temporal evidence input for inferring hidden risks. Step S4: Construct a self-supervised BeliefNet risk inference network, using the current local observations of the satellite and the historical sequence of its own execution feedback as joint inputs, and output a low-dimensional potential risk belief representation and a predicted value of the current mission imaging success probability; construct a binary cross-entropy self-supervised loss based on the binary success and failure feedback after the imaging action to complete the self-supervised update of BeliefNet parameters; Step S5: Construct the risk perception belief conditional RAB-MAPPO multi-agent reinforcement learning framework, input the potential risk belief representation output by BeliefNet into the Actor policy network and Critic value network respectively to realize belief conditionalization; construct a modified reward with risk constraints based on the predicted imaging success probability; Step S6: Adopt the centralized training and decentralized execution CTDE paradigm to jointly optimize the parameters of the three types of networks, BeliefNet, Actor, and Critic, and iterate the training until the scheduling model converges; Step S7: Prune the converged model, retaining only the lightweight BeliefNet inference module and the Actor policy network, and distribute them to the onboard embedded processing units of each satellite. During the on-orbit phase, each satellite relies on local real-time observations and its own independently maintained execution feedback history cache to distribute output imaging or skip mission decisions, thereby achieving collaborative online observation and scheduling of hidden risks in the multi-satellite constellation.

2. The multi-satellite Earth observation hidden risk scheduling method based on risk-aware reinforcement learning according to claim 1, characterized in that: Step S1, constructing the hidden risk scheduling environment specifically includes: S11: DQG grid is used to uniformly represent ground point, line and area observation targets. Based on the USM unified scheduling model, various original observation targets are transformed into standardized grid task units and a standardized task attribute set is generated. S12: Read the two lines of orbital elements of the satellite TLE, calculate the spatial position of the satellite at discrete scheduling time through the SGP4 orbital propagation model, transform it to the ground-fixed coordinate system, and then combine it with the satellite field of view constraint to match the DQG grid task to generate time-by-time satellite-mission observation opportunities. S13: Hidden risks are modeled in three categories. All true and effective imaging probabilities are calculated only within the hidden risk scheduling environment and are not disclosed to satellites. ① Cloud cover risk: Import the ERA5 global reanalysis cloud field and perform spatial interpolation on the task DQG grid location to obtain the real-time total cloud cover of the grid. Linear mapping to calculate the effective imaging probability of clouds ; ② Mesh-based electromagnetic interference risk: Set up a spatially distributed set of electromagnetic interference hotspots, calculate the electromagnetic interference intensity of the task grid based on the Gaussian distance attenuation function, and map it to obtain the electromagnetic effective imaging probability. ; ③ Satellite health degradation risk: A linear degradation model is established based on the cumulative number of satellite imaging attempts, and a random health perturbation term is superimposed to calculate the probability of effective imaging of a single satellite in good health. ; S14. Calculate the success probability of true imaging in a single task using multiplicative fusion. Hidden risk scheduling environment based on Bernoulli distribution Sampling generates imaging success or failure results Hidden risk status ,reality It is not included in local satellite observations, but maintains the observable constraints of the scene.

3. The multi-satellite Earth observation hidden risk scheduling method based on risk perception reinforcement learning according to claim 1, characterized in that: In step S2, the Hidden-Risk Dec-POMDP model is defined as follows: Unified interpretation of all parameters: Hidden risk scheduling environment involves participants in collaborative scheduling A collection of satellite intelligent agents; : Global state space, time step Global state ; To conceal the completion status of all grid tasks in the risk scheduling environment, This is the vector of the cumulative number of images captured by each satellite. For the current discrete scheduling time, To conceal the unobservable hidden risk states generated by the risk scheduling environment; :satellite In the binary action space, action 0 represents skipping the current observation opportunity, and action 1 represents performing imaging. The action output is sent to the hidden risk scheduling environment to complete the state transition. : Hidden risk-driven state transition function; after the satellite performs imaging actions, it calls the hidden risk scheduling environment. Sampling imaging results Update task completion status Cumulative number of satellite imaging attempts Generate the global state for the next time step. ; :satellite Local observation space; local observation vector It includes the mission DQG grid spatial characteristics, mission priority, satellite-mission observation geometric imaging quality, imaging energy consumption, current cumulative number of satellite imaging attempts, scheduling progress, and remaining scheduling time; Local observation vector It does not include any real-world risk probability parameters related to cloud cover, electromagnetic fields, or satellite health. Multi-satellite cooperative global reward function; The basic reward for a single satellite is differentiated into three scenarios: successful imaging, failed imaging, and skipped mission. The global single-step reward is the arithmetic mean of the basic rewards for all satellites in a single step. : Reinforcement learning discount factor, used to weight long-term time-series returns.

4. The multi-satellite Earth observation hidden risk scheduling method based on risk-aware reinforcement learning according to claim 1, characterized in that: In step S3, the satellite At any moment The execution feedback history expression is: ； Unified explanation of all parameters: : The execution feedback history has a fixed window length, which is a preset hyperparameter; A historic moment The spatial structure features derived from the DQG grid for the corresponding observation task include grid hierarchy, neighborhood topology, and spatial adjacency relationships, which are used to characterize the spatial correlation of risks. A historic moment satellite Selected scheduling action; Historical imaging feedback markers This indicates successful imaging. This indicates imaging failure; if no imaging action was performed at a historical moment, there will be no effective imaging feedback. When the cumulative interactive sample length of the satellite is less than the window At that time, zero vector padding is used to fill in the sequence length to ensure that the input dimension of BeliefNet is fixed and uniform.

5. The multi-satellite Earth observation hidden risk scheduling method based on risk-aware reinforcement learning according to claim 1, characterized in that: In step S4, BeliefNet uses joint features As input, output a low-dimensional representation of potential risk beliefs. Predicted probability of successful imaging for the current mission The BeliefNet self-supervised binary cross-entropy loss function is: ； In the formula: For the training batch sample set; To effectively provide feedback on indicator variables, the satellite at time... When performing imaging actions and obtaining imaging success or failure feedback If imaging is not performed... It is used to filter out invalid training samples that do not perform imaging actions.

6. The multi-satellite Earth observation hidden risk scheduling method based on risk-aware reinforcement learning according to claim 1, characterized in that: In step S5, the Actor-Critic network is constructed using the following method: Actor strategy network input splicing satellite local observations Risk belief representation output by BeliefNet Output satellite action probability distribution , These are the parameters of the Actor network; Critic value network input global state Local satellite observations Risk belief representation Output global state value estimate , These are the parameters for the Critic network.

7. The multi-satellite Earth observation hidden risk scheduling method based on risk-aware reinforcement learning according to claim 1, characterized in that: In step S5, the formula for calculating the risk perception correction reward is as follows: ； In the formula: The satellite single-step reward is corrected based on risk perception. As a reward for the original satellite; To preset the risk perception coefficient; It is a binary indicator function, which is only used when the satellite selects an imaging action. Overlapping risk correction items; Output the predicted imaging success probability for BeliefNet; when When the time correction term is positive, it incentivizes the satellite to perform the current low-risk imaging mission; when The time correction term is negative to suppress the satellite from performing high-risk imaging missions.

8. The multi-satellite Earth observation hidden risk scheduling method based on risk-aware reinforcement learning according to claim 1, characterized in that: In step S5, the overall joint optimization loss function of RAB-MAPPO is: ； In the formula: For PPO shearing strategy loss; The mean squared error loss of the Critic value; The BeliefNet self-supervised cross-entropy loss is used. This is a policy entropy regularization term, used to maintain policy exploration capabilities; These are the value loss weight, belief loss weight, and entropy regularization term weight coefficients, respectively.

9. The multi-satellite Earth observation hidden risk scheduling method based on risk-aware reinforcement learning according to claim 1, characterized in that: In step S6, the complete offline model training process is as follows: S61. Initialize all parameters of the Actor network, Critic network, and BeliefNet network, and initialize the global interactive experience cache and the independent execution feedback history cache for each satellite. S62, Episode-by-Episode training: Reset the completion status of all tasks, the cumulative imaging resource status of each satellite, and the historical feedback cache of all satellites within the hidden risk scheduling environment; traverse all discrete moments within the scheduling cycle. ; S63, Single-moment interaction: Reading local observations for each satellite. Read its own execution feedback history Risk beliefs are obtained by inputting BeliefNet inference. Predicting the probability of success ; Actor network is based on Sampling satellites' independent actions; integrating all satellite actions to form a joint action input hidden risk scheduling environment; S64, Hidden Risk Scheduling Environment Returns Satellite to Original Basic Reward Imaging success or failure markers The global state at the next moment; based on Calculate risk perception correction reward Store the state transition samples in the global experience cache; Synchronously update the satellite's own execution feedback history sequence; S65. After a single episode interaction is completed, the reward is adjusted based on risk perception. Calculate the GAE advantage function and discounted return; update the Actor policy network, Critic value network, and BeliefNet risk inference network in sequence; clear the global experience cache and proceed to the next episode iteration; S66. Repeat the iteration until the global average reward fluctuation of multiple consecutive episodes is less than the preset threshold, then determine that the model has converged and end the offline training.

10. The multi-satellite Earth observation hidden risk scheduling method based on risk-aware reinforcement learning according to claim 1, characterized in that: In step S7, the on-orbit online scheduling and deployment method is as follows: After the offline model converges, the Critic value network used only in the centralized training phase is pruned, and only the lightweight BeliefNet inference module and Actor policy network are retained, which are then distributed to the on-board embedded processing unit of each satellite. When the satellite is in orbit, it independently calculates its own and ground grid missions' observation opportunities in real time, maintains a fixed-length execution feedback history cache locally, autonomously calls BeliefNet to infer risk beliefs, and outputs imaging or skips decisions through Actor. The entire process does not require a centralized on-board control node, and multiple satellites work together in a distributed manner to complete the scheduling of Earth observations under hidden risks.

Images