Satellite multi-point target imaging deep reinforcement learning scheduling method and device

Abstract

The application discloses a kind of satellite multi-point target imaging deep reinforcement learning scheduling method and system, by the forecast of target visible window to the whole visible time window of task target, establish four-dimensional matrix to store the attitude conversion time of each satellite each time window to any other satellite other time window, the process of remote sensing satellite task execution imaging task is modeled as agent Markov decision process, define state space, action space, reward, state transition probability and discount factor, in each training step constraint expression is carried out, strategy network and value network are established, PPO algorithm training model is executed, and convergence obtains optimization strategy, input test data and initialize environment, read strategy and input feasible target observation start time and end time in each time step, and output task planning strategy.The application can be effectively applied to remote sensing satellite multi-point target imaging task planning.

Description

Technical Field

[0001] This invention relates to the field of satellite remote sensing technology, and in particular to a deep reinforcement learning scheduling method and apparatus for satellite multi-point target imaging. Background Technology

[0002] Remote sensing satellites have wide applications in environmental monitoring, disaster early warning, agricultural management, urban planning, and resource exploration. Remote sensing satellite point target mission planning involves abstracting mission requirements into point targets, analyzing mission requirements and satellite resource constraints, abstracting the mission plan into a mathematical model, and solving it. This process rationally arranges satellite observation tasks on ground targets within a specific timeframe to maximize observation efficiency and scientific value. With the increasing number of satellites and observation demands, mission planning becomes increasingly complex and challenging. The sources of remote sensing satellite observation demands are diverse and highly dynamic in terms of time series, making traditional "offline" manual mission planning methods unsuitable. With the dramatic increase in the number of demands, the dynamism of those demands will also increase significantly. If traditional "offline" planning methods are still used, it will be difficult to adapt to future application scenarios with large-scale mission growth. On the other hand, research on single-satellite point target mission planning model solutions still mainly focuses on solving "static" planning models for conventional missions, primarily using deterministic algorithms and (meta)heuristic algorithms to obtain optimal or near-optimal mission planning schemes, which are difficult to adapt to dynamic and increasingly large-scale mission demands.

[0003] For dynamic scheduling algorithms of imaging satellites, existing research has proposed heuristic algorithms for multi-satellite dynamic scheduling models, and designed multi-objective optimization algorithms, dynamic programming algorithms, and other methods for solving the problem. However, the solution methods are still mainly deterministic algorithms and (meta)heuristic algorithms, with limited search effectiveness and low intelligence levels. While these methods have strong descriptive capabilities for application scenarios and relatively mature solution algorithms, they still have limitations. On the one hand, these methods require customized rules for different scheduling scenarios, resulting in weak generalization ability and difficulty in adapting to the needs of large-scale task scheduling modeling.

[0004] Deep reinforcement learning (DRM) is a machine learning method suitable for dynamic task planning problems, aiming to maximize cumulative rewards. It has been widely applied in solving intelligent optimization control problems, possessing advantages such as strong generalization ability and good scalability. It is also one of the hot topics in research on algorithms for solving remote sensing satellite mission planning models in recent years. DRM possesses adaptive decision-making capabilities, long-term policy optimization, and the ability to handle uncertainty. Its autonomous decision-making ability is well-adaptable to environments with high uncertainty. It can effectively handle uncertainties in tasks by exploring and utilizing them, making efficient adaptive adjustments to different tasks without relying on large amounts of pre-labeled data. This makes it a current hot topic in satellite mission planning solutions and one of the key areas of machine learning. In satellite mission scheduling, current research on reinforcement learning applications mainly focuses on two aspects: first, combining reinforcement learning algorithms with deterministic algorithms and (meta)heuristic optimization algorithms to collaboratively solve satellite mission planning models; second, designing and using reinforcement learning methods to solve scheduling models based on Markov processes to obtain optimal mission scheduling schemes. In single-satellite mission scheduling, Huang et al. proposed a deep deterministic policy gradient algorithm (DDPG) to solve the mission planning model for time-series mission scheduling needs (Huang et al., 2021); Wang et al. proposed a scheduling method based on deep reinforcement learning (DRL) for agile satellite multi-autonomous mission planning problems (Wang et al., 2022). However, in the development stage from single-satellite small-scale mission scheduling to multi-mission scheduling, these solving algorithms still use relatively traditional deep reinforcement learning algorithms, with a high degree of abstraction of satellite mission constraints and relatively simple mission scenarios. On the other hand, when the mission scale increases, there are still problems such as a dramatic increase in action space dimension and high training cost that need to be solved. Given the current limitations of traditional single-satellite multi-mission satellite mission planning methods in adapting to the needs of large-scale mission growth and the shortcomings of existing methods, new technical solutions are urgently needed in this field.

[0005] The relevant references are as follows:

[0006] [1]Huang,Y.,Mu,Z.,Wu,S.,Cui,B.,and Duan,Y.,2021.Revising theObservation Satellite Scheduling Problem Based on Deep Reinforcement Learning[J].Remote Sensing,13(12),2377.

[0007] [2]Wang, Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a deep reinforcement learning scheduling method for satellite multi-point target imaging, comprising the following steps:

[0009] Step 1: Preprocess the scene data to obtain the visible time window w between satellite i and ground target j. ij =(starttime) ij ,endtime ij ), starttime and endtime represent the start time and end time, respectively;

[0010] Step 2, Point target task constraint construction: Calculate the attitude transition time from each time window of each satellite to any other time window of any other satellite, and establish a four-dimensional matrix for storage;

[0011] Step 3, Markov decision environment construction: This includes modeling the satellite imaging mission execution process as a multi-agent Markov decision process, defining the state space, action space, reward, state transition probability, and discount factor, and expressing them in a constrained manner at each training step.

[0012] Step 4: Initialize the Markov decision environment and the PPO algorithm, and establish the policy network and value network; update the policy network parameters according to the objective function of the PPO algorithm, and update the value network at the same time to minimize the mean square error of the value function estimation. Through continuous iterative training, the optimal scheduling policy is obtained when the training converges.

[0013] Step 5: Set different learning rates, discount factors, pruning coefficients, value function coefficients, and entropy coefficients, and set different training batch sizes and training periods. Start training and store the algorithm training results when training converges.

[0014] Step 6: Load the PPO algorithm, input test data and initialize the environment. At each time step, read the strategy and input the observation start and end times of feasible targets, and output the imaging task planning strategy.

[0015] Furthermore, the preprocessing of the scene data in step 1 includes: predicting the target visibility window based on the orbit and the maximum field of view of the remote sensing satellite; assuming the remote sensing satellite moves in a two-body model, predicting the visibility window of the satellite to each target point using the six orbital elements of the satellite, and obtaining the visible time window w of satellite i to ground target j. ij =(starttime) ij ,endtime ij ).

[0016] Furthermore, the specific implementation of step 1 includes the following sub-steps:

[0017] Step 1.1: Initialize the remote sensing satellite information. The known orbital parameters are: semi-major axis a, eccentricity e, inclination i, right ascension of the ascending node Ω, argument of perigee ω, and mean anomaly M0. The epoch time is t0, and the remote sensing satellite's field of view is θ. FOV ;

[0018] Step 1.2, calculate the satellite's mean anomaly angle:

[0019] M(t) = M0 + n(t - t0)

[0020] Where t is the current time and t0 is the epoch time;

[0021] n is the average velocity of the satellite, calculated using the following formula:

[0022]

[0023] Where G is the gravitational constant, M is the mass of the central celestial body (i.e., the mass of the Earth), k is the Kepler constant, and a is the semi-major axis of the satellite orbit;

[0024] Solving the Kepler equations yields the anomalous angle E:

[0025] Ee sin(E)=M(t)

[0026] E is the anomalous angle, e is the orbital eccentricity, M is the mean anomalous angle, and t is the current time.

[0027] Calculate the true anterior angle v:

[0028]

[0029] Where E is the asymmetry angle and e is the orbital eccentricity;

[0030] Calculate the satellite's geocentric coordinates:

[0031]

[0032] Calculate the coordinates (x, y) of the remote sensing satellite in the geocentric inertial coordinate system based on its orbital parameters. s ,y s ,z s ):

[0033] x s =r(cos(Ω)cos(ω+v)-sin(Ω)sin(ω+v)cos(i))

[0034] y s =r(sin(Ω)cos(ω+v)+cos(Ω)sin(ω+v)cos(i))

[0035] z s =r(sin(ω+v)sin(i))

[0036] Step 1.3, calculate the geocentric coordinates (x, y) of the ground target. t ,y t ,z t R is the Earth's radius;

[0037] x t = (R+h)cos(φ)cos(λ)

[0038] y t = (R+h)cos(φ)sin(λ)

[0039] z t =(R+h)sin(φ)

[0040] Where R is the Earth's radius, h is the altitude of the ground target, φ represents the longitude of the ground target, and λ represents the longitude of the ground target;

[0041] Step 1.4, calculate the distance between the satellite and the Earth's center:

[0042]

[0043] Step 1.5, calculate the distance between the target and the Earth's center:

[0044]

[0045] Step 1.6: Determine whether the line connecting the satellite and the target passes through the Earth;

[0046]

[0047] Where, θ st The angle between the satellite and the target is calculated using the following formula:

[0048]

[0049] If the above formula is true, then the target is not blocked by the Earth; otherwise, the target is blocked by the Earth.

[0050] Step 1.7: Calculate the angle between the remote sensing satellite and the ground target, and compare it with the remote sensing satellite's field of view. When θ ≤ θ FOV At that time, the ground target is within the field of view of the remote sensing satellite;

[0051]

[0052] Step 1.8: Starting from the epoch, repeat steps 1.1-1.7 every Δt to calculate the visibility of the mission target, obtaining the visible time window w of each remote sensing satellite i for each ground target j. ij =(starttime) ij ,endtime ij ).

[0053] Furthermore, the specific implementation of step 2 includes the following sub-steps:

[0054] Step 2.1: Create a remote sensing satellite imaging mission switching duration matrix, calculate the maximum number of observation windows for the mission target, create a matrix with a size of number of satellites × maximum number of observation windows × number of satellites × maximum number of observation windows, and save it as a .npy file;

[0055] Step 2.2: Calculate the imaging target p within the time window w using the rotation matrix method. px The attitude transition time of target q in each time slice Δt within the visible duration was observed; let r A Let r be the position vector of target p in the geocentric inertial frame at a specified time. B Given the position vector of target q in the geocentric inertial frame at a specified time, calculate the target direction vector d using the position vectors of remote sensing satellites and the target. AB d AB =r B -r A ; will d AB Normalization yields the unit direction vector.

[0056] Construct the target direction cosine matrix (DCM), and use the orthogonal basis matrix of the direction vectors as the target direction vector DCM. target Assuming the initial attitude matrix points to target p, calculate the attitude error matrix. DCM initial This refers to the satellite's attitude at the initial moment; the trace (DCM) is obtained using the rotation matrix. error Calculate the maximum values ​​of pitch angle, roll angle, and rotation angle. The attitude transition time between two target windows is calculated every Δt. Where ω is the angular velocity ω during attitude transition, the attitude transition duration sequence within this time window is obtained. The start and end times of satellite i relative to ground target j;

[0057] Step 2.3: Perform hypercube sampling on the attitude transition time sequence calculated in the previous step to obtain an estimated attitude transition time and assign it to martix[p][x][q][y], which represents the estimated task switching time when task target p observes task target q in the x-th window and in the y-th window. When task target p or q does not have a visible window, assign a value of -1.

[0058] Step 2.4: Due to differences in maneuvering conditions among remote sensing satellites, if the imaging mission target p or q does not have a visible time window, does not meet maneuvering attitude constraints, or is not within the time window of either target after the attitude transition ends, it is impossible to switch from mission p to q. In this case, the matrix is ​​assigned a value of -1.

[0059] Furthermore, in step 3, the remote sensing satellite is abstracted as an intelligent agent. A remote sensing satellite has N imaging tasks to be completed. The imaging tasks of the remote sensing satellite are regarded as point targets, and each target i has a fixed geographical location Loc. i =(lon) i ,lat i ), with multiple visible observation time windows TW i ={tw i1 ,tw i2 ,...,tw ij Each visible time window contains a visible start time and an visible end time w. ij =(starttime) ij ,endtime ij );

[0060] The process of a remote sensing satellite performing imaging tasks on multiple ground point targets is modeled as a multi-agent Markov decision process, determined by a quintuple (S, A, r, P, γ), where S, A, r, P, and γ represent the state space, action space, reward, state transition probability, and discount factor, respectively. The state S = s I ×s T ×s Sat Describes the joint state of the imaging mission target state and remote sensing satellite resources, s I Includes the observation status of the imaging target, s Sat Describes the satellite status, s T This includes a list of imaging missions completed by remote sensing satellites and their current operational status.

[0061] Furthermore, the specific implementation method of step 3 is as follows;

[0062] Actions of remote sensing satellites A sat ∈A includes two types: observed and unobserved, which are represented by different indices 0 and 1;

[0063] Rewards in the Markov process are constructed using imaging task priorities: Each imaging task i has a priority prio. i The urgency and importance of the task are indicated by the reward r, which is determined by the current state S and the action A. The reward r describes the benefit gained from taking an action in the current state. The reward r varies depending on the remote sensing satellite. j There are different calculation methods; the total reward is the sum of all remote sensing satellite rewards. Convergence occurs when the reward value is maximized and no longer changes, yielding the optimal strategy;

[0064] The time is segmented according to the target requirement refresh cycle. The remote sensing satellite status and imaging task observation status of each time slice are used as status information. Different rewards are assigned to different actions in response to environmental feedback. The remote sensing satellite observation sequence is obtained with the goal of maximizing rewards. After the remote sensing satellite completes the imaging task of a point target, a certain task switching time is required before the next target can be observed. The task switching time depends on the satellite's attitude maneuverability. Attitude conversion needs to be performed according to the attitude requirements of the two imaging tasks. The matrix value martix[p][t] calculated in step 2 is read. x ][q][t y When the value is less than 0, it is considered that the current target p is at t. x The window cannot be in the t of the q target. y After the window performs attitude transformation, target q is observed. When the value is greater than 0, the current target p is at t. x The window can be on the t of the q target. y After the window performs an attitude transition, target q is observed; if the maneuver conditions for attitude transition are not met, the next target cannot be observed, and a new target that satisfies the attitude transition constraints is selected.

[0065] The problem of scheduling multi-target imaging tasks for remote sensing satellites is transformed into a problem of solving optimal policies for an agent: the reward state transition probability function P(s′|s,a) describes the probability that the imaging task and the remote sensing satellite will transition to the new state s′ after the remote sensing satellite performs action a in the current state s. The policy π θ (a|s) represents the probability distribution of a remote sensing satellite taking action a in state s, where π θ (a|s) represents a random policy; the neural network applies the policy function π. θBy fitting the data, the probability distribution of action a in state s is obtained. The goal is to determine the action sequence of each remote sensing satellite and find the current optimal strategy π. θ To maximize the overall expected return of the constellation, the objective function is defined as follows:

[0066]

[0067] In the above formula, θ represents the strategy π. θ The parameters, This represents the expected value of the initial state s0. Let be the value function, representing the outcome of following policy π in the initial state s0. θ The expected total return of execution; specifically, the value function is defined as the return calculated from state s0 in future time steps according to policy π. θ The expected value of the cumulative discounted return that can be obtained from the action; In terms of strategy π θ The expected value of sampling is given below; γ is a discount factor, the closer the value is to 1, the greater the impact of future rewards on the current decision; the closer the value is to 0, the more attention is paid to recent rewards; r(s) t ,a t ) indicates that at time step t, state s t and action a t The following rewards; This indicates starting from the initial state s0, following strategy π θ Execution, the expectation of the value function under all possible initial states; Indicates according to strategy π θ Execution, receiving an immediate reward r(s) at each time step t. t ,a t ), and then perform a weighted summation of future rewards according to the discount factor γ; in solving the strategy to maximize the objective function J(θ), find θ * =argmax θ J(θ) is used to obtain the optimal action strategy for the time sequence.

[0068] Furthermore, in step 4, the state is defined as including the current time and the observation state of the target, and the action is the decision to select the target for observation, which is expressed in a constrained manner in each step; the policy is represented by the policy network Actor, with the current state as the input and the probability distribution of each action as the output; the value of the state is estimated by the value network Critic, with the current state as the input and the corresponding value as the output; samples are collected under the current policy to record the state, action, reward, and next state information.

[0069] Step 4.1: Construct a policy network, which is used to generate remote sensing satellite action policies π under a given state. θ(a|s) Action probability distribution, including input layer, hidden layer and output layer; input state s t The probability distribution of the output action π(a) t |s t );

[0070] Step 4.2: Construct a value network to estimate the observation value of all remote sensing satellite agents using a fully connected neural network, including an input layer, hidden layers, and an output layer; input state s t The value of the output state V(s) t );

[0071] The observational value of all remote sensing satellite agents is estimated using a fully connected neural network. The Critic value network minimizes the mean square error L. MSE (φ) is used for updating, and the calculation method is as follows, where r t It is the reward for time step t.

[0072]

[0073] Where MSE is the mean squared error. It is the expected value of empirical sampling over time step t, r t It is the reward for time step t. Let φ be the value network's prediction of the reward at time step t, and let φ be the parameters of the value network.

[0074] Furthermore, the specific implementation of step 5 includes the following sub-steps:

[0075] Step 5.1: Begin training. During each training time step, the agent interacts with the environment and collects trajectory data during the training process. Each trajectory consists of a state, an action, a reward, and a next state;

[0076] Step 5.2: Collect training experience through the policy network (Actor) to compute the advantage estimate at each time step. The generalized advantage estimate (GAE) is used to estimate advantage. GAE can smooth the advantage estimate by balancing bias and variance. The formula is as follows:

[0077]

[0078] Where γ is the discount factor, λ represents the guiding coefficient (0 ≤ λ ≤ 1), controlling the decay rate of the guiding trace in the dominance function estimation; l is a count variable representing the l-th future time step starting from the current time step t, δ t This refers to the TD residual, and the formula for the TD residual is:

[0079] δ t =rt +γV(s t+1 )-V(s t )

[0080] During training, GAE accumulates data over a finite trajectory length. Let T be the step size at the end of training. The specific calculation process is as follows:

[0081]

[0082] Step 5.3, in each time step update, the probability ratio r of the remote sensing satellite adopting the old strategy and the new strategy before and after the update. t (θ) is used to update the policy:

[0083]

[0084] Where, π θ (a t |s t ) is the new strategy in state s t Take action a t The probability, It represents the probability of the old strategy under the same state and action;

[0085] Define the policy loss function L based on the shear probability ratio. CLIP (θ) to limit the update range of the policy:

[0086]

[0087] Among them, E t r is the expected value of empirical sampling at time step t. t (θ) represents the new strategy π at time step t. θ and old strategies The probability ratio between them This is an estimate of the dominance function; clip(r) t (θ), 1-∈, 1+∈) represent the probability ratio r. t (θ) is used to clip the data so that it is within the interval (1-∈, 1+∈), where ∈ is the clipping constant, representing the range of allowed strategy changes;

[0088] Step 5.4, the value network estimates the value V(s) of each state. t The mean squared error (MSE) is used to define the value loss function L. value (θ)=E t [(V θ (s t )-V t target ) 2 [, where E tThe expected value, V, obtained by empirical sampling over time step t. θ (s t ) is the value network in state s t The predicted value, Vt t target It is the target value for time step t, which is obtained through actual observation during training;

[0089] To encourage strategy exploration, an entropy penalty term H(π) is added. θ ), calculate the entropy of the strategy:

[0090]

[0091] Where, π θ (a|s t ) is strategy π θ In state s t The probability of taking action 'a';

[0092] The total loss function L(θ) includes policy loss, value loss, and entropy penalty:

[0093] L(θ)=L CLIP (θ)-c1L value (θ)+c2H(π θ )

[0094] Where c1 and c2 are weight hyperparameters;

[0095] Step 5.5: Update the parameters of the policy network and value network using the gradient of the total loss function, and use the Adam optimizer for parameter updates:

[0096]

[0097] Where α is the learning rate. This represents the steepest ascent direction of the gradient loss function in the parameter space. In each iteration, the loss function L(θ) is increased by updating the parameters along the gradient direction, thereby improving the strategy.

[0098] Step 5.6: In each time step, the Actor layer learns the policy π. θ (a|s), in each iteration of the training process, the policy network Actor outputs the probability distribution of the remote sensing satellite action space. Based on the generated probability distribution, the decoder selects remote sensing satellite actions and then updates the policy by maximizing the entropy of each action to minimize the mean squared error. Calculate and update the value network Critic;

[0099] Step 5.7: Update the policy network parameters according to the objective function of the PPO algorithm, use the pruning method to ensure the stability of the policy update, and update the value network at the same time to minimize the mean square error of the value function estimation; through continuous iterative training, obtain the optimal scheduling policy when the training converges; repeat the above steps 5.1-5.6 until the predetermined number of training rounds is reached or all objectives are observed and the reward no longer increases.

[0100] Furthermore, in step 6, for different imaging tasks, the saved models are loaded. During training, the PPO algorithm learns a policy π. θ (a|s), this strategy selects the optimal action a based on the current state s, strategy π θ (a|s) is represented by a neural network, whose output is the probability distribution of taking different actions in each state;

[0101] Step 6.1: Initialize the initial state of the system, including the initial state information of all remote sensing satellites and targets;

[0102] Step 6.2: Iterate through each time step, and at time step t, based on the current state s... t Use strategy π θ (a|s) Select an action a t Action a t =arg max a π θ (a|s t This specifies which remote sensing satellite should observe;

[0103] Step 6.3, execute action a t Then, update the system status s t+1 The updates include reducing the remaining visible time window for the corresponding targets and updating the status of remote sensing satellites;

[0104] Step 6.4: Record the tasks performed at each time step, including the time, remote sensing satellite, target, and imaging task results;

[0105] Step 6.5: Repeat steps 6.2 to 6.4 until the visible time window of all targets ends or the predetermined imaging task time range is reached. After completion, obtain the remote sensing satellite, start observation time, and end observation time for each point target imaging task, and obtain the imaging task planning scheme of the remote sensing satellite for different point targets.

[0106] The present invention also provides a deep reinforcement learning scheduling device for satellite multi-point target imaging, comprising:

[0107] One or more processors;

[0108] A storage device is provided for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement a deep reinforcement learning scheduling method for satellite multi-point target imaging as described in the above technical solution.

[0109] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0110] Traditional single-satellite target task constraint optimization modeling typically involves directly constructing a quantitative relationship between decision variables and constraints. This method, however, takes into account the characteristics of Markov decision processes and expresses the constraints of the task switching process using matrices. During training, it quantitatively evaluates and judges the uniqueness of actions and task switching constraints related to task execution. This method abstracts the process of satellites performing multiple tasks and simplifies the action space to observation and non-observation, significantly reducing the computational cost caused by the large scale and high dimensionality of the action space that changes over time. Furthermore, this method uses the PPO algorithm for solution, balancing the stability and performance of the update step size. Compared to other deep reinforcement learning algorithms, it exhibits higher stability, faster convergence speed, and robustness, improving the efficiency of solving satellite mission planning strategies. Attached Figure Description

[0111] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0112] Figure 2 This is a schematic diagram illustrating the switching of remote sensing satellite imaging missions according to the present invention.

[0113] Figure 3 : This is a schematic diagram of the PPO algorithm training of the present invention.

[0114] Figure 4 : This is a schematic diagram of the target distribution in an embodiment of the present invention. Detailed Implementation

[0115] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0116] The following is combined Figure 1 This invention provides a deep reinforcement learning scheduling method for satellite multi-point target imaging, comprising the following steps:

[0117] Step 1, Preprocess scene data. The target visibility window is predicted by using the six orbital data of remote sensing satellites and the maximum field of view of each satellite to forecast the entire visible time window of the mission target;

[0118] Step 2, constructing point target task constraints. Calculate the attitude transition time from each time window of each remote sensing satellite to any other time window of any other satellite, establish a four-dimensional matrix for storage, and assign a value of -1 if the attitude transition conditions are not met;

[0119] Step 3, Markov Decision Environment Setup. The execution process of remote sensing satellite imaging mission is modeled as a multi-agent Markov decision process, defining the state space, action space, reward, state transition probability, and discount factor, which are expressed in a constrained manner at each training step.

[0120] Step 4: Initialize the environment, initialize the Proximity Policy Optimization (PPO) algorithm, and establish the policy network (Actor) and value network (Critic).

[0121] Step 5: Execute the PPO algorithm to train the model and obtain the optimization strategy upon convergence;

[0122] Step 6: Load the PPO algorithm, input test data and initialize the environment. At each time step, read the strategy and input the observation start and end times of feasible targets, and output the imaging task planning strategy.

[0123] Step 1 describes the preprocessing of scene data, including predicting the target visibility window based on the orbit and the maximum field of view of the remote sensing satellite. Assuming the remote sensing satellite moves in a two-body model, the visibility window of the satellite relative to each target point is predicted using the six orbital elements, resulting in the visible time window w of satellite i relative to ground target j. ij =(starttime) ij ,endtime ij );

[0124] Step 1.1: Initialize the remote sensing satellite information. The known orbital parameters are: semi-major axis a, eccentricity e, inclination i, right ascension of the ascending node Ω, argument of perigee ω, and mean anomaly M0. The epoch time is t0, and the remote sensing satellite's field of view is θ. FOV .

[0125] Step 1.2, calculate the satellite's mean anomaly angle:

[0126] M(t) = M0 + n(t - t0)

[0127] Where t is the current time and t0 is the epoch time.

[0128] n is the average velocity of the satellite, calculated using the following formula:

[0129]

[0130] Where G is the gravitational constant, M is the mass of the central celestial body (i.e., the mass of the Earth), k is the Kepler constant, and a is the semi-major axis of the satellite orbit.

[0131] Solving the Kepler equations yields the anomalous angle E:

[0132] E-esin(E)=M(t)

[0133] E is the anomalous angle, e is the orbital eccentricity, M is the mean anomalous angle, and t is the current time.

[0134] Calculate the true anterior angle v:

[0135]

[0136] Where E is the asymmetry angle and e is the orbital eccentricity;

[0137] Calculate the satellite's geocentric coordinates:

[0138]

[0139] Calculate the coordinates (x, y) of the remote sensing satellite in the geocentric inertial coordinate system based on its orbital parameters. s ,y s ,z s ):

[0140] x s =r(cos(Ω)cos(ω+ν)-sin(Ω)sin(ω+ν)cos(i))

[0141] y s =r(sin(Ω)cos(ω+ν)+cos(Ω)sin(ω+ν)cos(i))

[0142] z s =r(sin(ω+ν)sin(i))

[0143] Step 1.3, calculate the geocentric coordinates (x, y) of the ground target. t ,y t ,z t R is the Earth's radius;

[0144] x t = (R+h)cos(φ)cos(λ)

[0145] y t = (R+h)cos(φ)sin(λ)

[0146] z t=(R+h)sin(φ)

[0147] Where R is the Earth's radius, h is the altitude of the ground target, φ represents the longitude of the ground target, and λ represents the longitude of the ground target;

[0148] Step 1.4, calculate the distance between the satellite and the Earth's center:

[0149]

[0150] Step 1.5, calculate the distance between the target and the Earth's center:

[0151]

[0152] Step 1.6: Determine whether the line connecting the satellite and the target passes through the Earth;

[0153]

[0154] Where, θ st The angle between the satellite and the target is calculated using the following formula:

[0155]

[0156] If the above formula is true, then the target is not obscured by the Earth; otherwise, the target is obscured by the Earth.

[0157] Step 1.7: Calculate the angle between the remote sensing satellite and the ground target, and compare it with the remote sensing satellite's field of view. When θ ≤ θ FOV At that time, the ground target is within the field of view of the remote sensing satellite.

[0158]

[0159] Step 1.8: Starting from the epoch, repeat steps 1.1-1.7 every Δt to calculate the visibility of the mission target, obtaining the visible time window w of each remote sensing satellite i for each ground target j. ij =(starttime) ij ,endtime ij ).

[0160] As described in step 2, this method segments the time according to the target requirement refresh cycle. The remote sensing satellite status and mission observation status of each time slice are used as status information. During the training process of each time step, the execution constraints of the imaging mission are judged. After the satellite completes the imaging mission of a point target, a certain mission switching time is required before the next target can be observed. This invention uses a four-dimensional matrix to store the satellite imaging mission switching time. The imaging mission switching time depends on the satellite's attitude maneuverability. Attitude transformation is required according to the attitude requirements of two imaging missions. If the maneuverability conditions for attitude transformation are not met, the next target cannot be observed, and a new target that meets the attitude transformation constraints must be selected.

[0161] Step 2.1: Create a remote sensing satellite imaging mission switching duration matrix. Calculate the maximum number of observation windows for the mission objective, create a matrix with a size of (number of satellites × maximum number of observation windows × number of satellites × maximum number of observation windows), and save it as a .npy file;

[0162] Step 2.2: Calculate the imaging target p within the time window w using the rotation matrix method. pc The attitude transition time of target q in each time slice Δt is observed within the visible duration. Let r... A Let r be the position vector of target p in the geocentric inertial frame at a specified time. B Given the position vector of target q in the geocentric inertial frame at a specified time, calculate the target direction vector d using the position vectors of remote sensing satellites and the target. AB d AB =r B -r A ; will d AB Normalization yields the unit direction vector.

[0163] Construct the target direction cosine matrix (DCM) and use the orthogonal basis matrix of the direction vectors as the target direction vector DCM. target Assuming the initial attitude matrix points to target p, calculate the attitude error matrix. DCM initial This refers to the satellite's attitude at the initial moment; the trace (DCM) is obtained using the rotation matrix. error Calculate the maximum values ​​of pitch angle, roll angle, and rotation angle. The attitude transition time between two target windows is calculated every Δt. Where ω is the angular velocity ω during attitude transition, the attitude transition duration sequence within this time window is obtained.

[0164] Step 2.3: Perform hypercube sampling on the attitude transition time sequence calculated in the previous step to obtain an estimated attitude transition time and assign it to martix[p][x][q][y], which represents the estimated task switching time when task target p observes task target q in the x-th window and in the y-th window. When task target p or q does not have a visible window, assign a value of -1.

[0165] Step 2.4: Due to the differences in maneuvering conditions of remote sensing satellites, this method assumes that the imaging mission target p or q cannot switch from mission p to q if there is no visible time window, does not meet the maneuvering attitude constraints, or the time after the attitude transition is not within the time window of either. In this case, the matrix is ​​assigned a value of -1.

[0166] As described in step 3, this method abstracts remote sensing satellites as intelligent agents. A remote sensing satellite has N imaging tasks to be completed, and these tasks are considered as point targets. Each target i (i = 1, 2, ..., N) has a fixed geographical location Loc. i =(lon) i ,lat i ), with multiple visible observation time windows TW i ={tw i1 ,tw i2 ,...,tw ij Each visible time window contains a visible start time and an visible end time w. ij =(starttime) ij ,endtime ij ).

[0167] Step 3.1: The process of a remote sensing satellite performing multiple ground point target imaging tasks is modeled as a multi-agent Markov decision process, determined by a quintuple (S, A, r, P, γ), where S, A, r, P, and γ represent the state space, action space, reward, state transition probability, and discount factor, respectively. State S = s I ×s T ×s Sat Describes the joint state of the imaging mission target state and remote sensing satellite resources, s I Includes the observation status (observed, unobserved) of the imaging target, including s Sat Describes the satellite's status (observation, attitude transition, idle), s T This includes information such as a list of imaging tasks completed by remote sensing satellites and their current operational status.

[0168] Step 3.2, the action A of the remote sensing satellite in this invention sat ∈A includes two types: observed and unobserved, which are represented by different indices 0 and 1.

[0169] Step 3.3: This invention uses imaging task priorities to construct the reward in the Markov process. Each imaging task objective i has a priority prio. i This indicates the urgency and importance of the task. The reward *r* is determined by the current state *S* and action *A*, describing the benefit gained from taking an action in the current state. The reward *r* varies depending on the remote sensing satellite. j They have different calculation methods. For example, the reward for an imaging action is the priority of the corresponding observation target, while emergency imaging tasks have a higher priority; the reward for idle actions is set to r. j =0. The total reward is the sum of all remote sensing satellite rewards. Convergence occurs when the reward value is maximized and no longer changes, yielding the optimal strategy.

[0170] Step 3.4: The time is segmented according to the target requirement refresh cycle. The remote sensing satellite status and imaging task observations of each time slice are used as status information. Different rewards are assigned to different actions based on the environmental feedback, with the goal of maximizing rewards to obtain the remote sensing satellite observation sequence. After the remote sensing satellite completes an imaging task of a point target, a certain task switching time is required before observing the next target. The task switching time depends on the satellite's attitude maneuverability and requires attitude conversion based on the attitude requirements of the two imaging tasks. The matrix value `martix[p][t]` calculated in Step 2 is then read. x ][q][t y When the value is less than 0, it is considered that the current target p is at t. x The window cannot be in the t of the q target. y After the window performs attitude transformation, target q is observed. When the value is greater than 0, the current target p is at t. x The window can be on the t of the q target. y After the window performs an attitude transition, target q is observed; if the maneuver conditions for attitude transition are not met, the next target cannot be observed, and a new target that satisfies the attitude transition constraints must be selected.

[0171] Step 3.5: This invention transforms the remote sensing satellite multi-target imaging task scheduling problem into an optimal policy solution problem for an intelligent agent. The reward state transition probability function P(s′|s,a) involved in this invention describes the probability that, under the current imaging task and remote sensing satellite state s, after the remote sensing satellite performs action a, both the imaging task and the remote sensing satellite will transition to a new state s′. Policy π θ (a|s) represents the probability distribution of a remote sensing satellite taking action a in state s, where π θ (a|s) represents a random policy. This method uses a neural network to handle the policy function π. θ By fitting the data, the probability distribution of action a in state s is obtained. The goal is to determine the action sequence of each remote sensing satellite and find the current optimal strategy π.θ To maximize the overall expected return of the constellation, the objective function is defined as follows:

[0172]

[0173] In the above formula, This represents the expected value of the initial state s0. Let be the value function, representing the outcome of following policy π in the initial state s0. θ The expected total return of execution. Specifically, the value function is defined as the return calculated in future time steps according to policy π, starting from state s0. θ The expected value of the cumulative discounted return that can be obtained from the action; In terms of strategy π θ The expected value of the sample is γ; γ is a discount factor, the closer the value is to 1, the greater the impact of future rewards on the current decision; the closer the value is to 0, the more attention is paid to recent rewards. t ,a t ) indicates that at time step t, state s t and action a t The reward is as follows. This indicates starting from the initial state s0, following strategy π θ Execute the expected value function of all possible initial states. Indicates according to strategy π θ Execution, receiving an immediate reward r(s) at each time step t. t ,a t ), and then sum the future rewards according to the discount factor γ.

[0174] θ represents strategy π θ The parameters, Let the initial state be the value of the value function. It is an indicator used by intelligent agents to evaluate the expected future returns of remote sensing satellite intelligent agents. Indicates from state s t Begin, in all future time steps, follow strategy π θ The expected value of the cumulative discount reward that can be obtained during the action; the value function is expressed by the Bellman equation. Calculate r(s) t ,a t ) represents state s t Action a t The reward is given by T, where T is the overall step size and t is the current time step size. In solving the policy to maximize the objective function J(θ), we find θ... * =argmax θ J(θ) is used to obtain the optimal action strategy for the time sequence.

[0175] Step 4 involves initializing the environment by defining it based on the observation target and cooling matrix, initializing the PPO algorithm, and establishing a policy network (Actor) and a value network (Critic). First, the state is defined, including the current time and the observation state of the target. Actions are decisions made to select targets for observation, expressed with constraints in each Step. The policy network (Actor) represents the policy, with the current state as input and the probability distribution of each action as output. The value network (Critic) estimates the value of each state, with the current state as input and the corresponding value as output. Samples are collected under the current policy, recording information such as state, action, reward, and next state. The policy network parameters are updated according to the objective function of the PPO algorithm, using a pruning method to ensure the stability of the policy update. Simultaneously, the value network is updated to minimize the mean squared error of the value function estimation. Through continuous iterative training, the optimal scheduling policy is obtained upon training convergence.

[0176] Step 4.1: Construct the policy network (Actor Network), which is used to generate remote sensing satellite action policies π under given states. θ The (a|s) action probability distribution is constructed as follows:

[0177] 1. Construct the input layer and store the state s t .

[0178] 2. Construct a hidden layer consisting of n fully connected layers, each using the ReLU activation function.

[0179] First hidden layer: h1 = ReLU(W1s) t +b1)

[0180] Second hidden layer: h2 = ReLU(W2h1 + b2)

[0181] ...

[0182] The nth hidden layer: h n =ReLU(W n h n-1 +b n )

[0183] 3. Probability distribution π(a) of output actions from the output layer t |s t For discrete action spaces, this method uses the softmax function.

[0184] π(a t |s t = softmax(W) out h n +b out )

[0185] Step 4.2: Construct a Critic Network to estimate the observation value of all remote sensing satellite agents using a fully connected neural network;

[0186] 1. Construct the input layer and store the state s t .

[0187] 2. Construct a hidden layer consisting of n fully connected layers, each using the ReLU activation function.

[0188] First hidden layer: h1 = ReLU(W1s) t +b1)

[0189] Second hidden layer: h2 = ReLU(W2h1 + b2)

[0190] ...

[0191] The nth hidden layer: h n =ReLU(W n h n-1 +b n )

[0192] 3. The value of the output state V(s) of the output layer's output action t ).

[0193] V(s t ) = W o ′ ut h′ n +b o ′ ut

[0194] The Critic network estimates the observational value of all remote sensing satellite agents using a fully connected neural network, by minimizing the mean squared error (MSE). MsE (φ) is used for updating, and the calculation method is as follows:

[0195]

[0196] in, It is the expected value of empirical sampling over time step t, r t It is the reward for time step t. Let φ be the value network's prediction of the reward at time step t, and let φ be the parameters of the value network.

[0197] As described in step 5, different learning rates, discount factors, pruning coefficients, value function coefficients, and entropy coefficients are set, along with different training batch sizes and training periods. Training begins, and the algorithm training results are stored when training converges.

[0198] Step 5.1: Begin training. During each training time step, the agent interacts with the environment and collects trajectory data during the training process. Each trajectory consists of a state, an action, a reward, and a next state.

[0199] Step 5.2: Collect training experience using Actors to calculate the advantage estimate at each time step. The generalized advantage estimate (GAE) is used to estimate advantage. GAE can smooth the advantage estimate by weighing bias and variance. The formula is as follows:

[0200]

[0201] Where γ is the discount factor, λ represents the guiding coefficient (0 ≤ λ ≤ 1), which controls the decay rate of the guiding trace in the dominance function estimation; and l is a count variable representing the l-th future time step δ starting from the current time step t. t This refers to the TD residual, and the formula for the TD residual is:

[0202] δ t =r t +γV(s t+1 )-V(s t )

[0203] During training, GAE accumulates data over a finite trajectory length. Let T be the step size at the end of training. The specific calculation process is as follows:

[0204]

[0205] Step 5.3, in each time step update, the probability ratio r of the remote sensing satellite adopting the old strategy and the new strategy before and after the update. t (θ) is used to update the policy:

[0206]

[0207] Where, π θ (a t |s t ) is the new strategy in state s t Take action a t The probability, It represents the probability of the old strategy under the same state and action.

[0208] Define the policy loss function L based on the shear probability ratio. CLIP (θ) to limit the update range of the policy:

[0209]

[0210] Among them, E tr is the expected value of empirical sampling at time step t. t (θ) represents the new strategy π at time step t. θ and old strategies The probability ratio between them This is an estimate of the dominance function. clip(r) t (θ), 1-∈, 1+∈) represent the probability ratio r. t (θ) is used to clip the data so that it falls within the interval (1-∈, 1+∈), where ∈ is the clipping constant, representing the range of allowed strategy variations.

[0211] Step 5.4, the value network estimates the value V(s) of each state. t The mean squared error (MSE) is used to define the value loss function L. value (θ)=E t [(V θ (s t )-V t target ) 2 ], where E t The expected value, V, obtained by empirical sampling over time step t. θ (s t ) is the value network in state s t The predicted value, V t target It is the target value of time step t, which is obtained through actual observation during training.

[0212] To encourage strategy exploration, an entropy penalty term H(π) is added. θ ), calculate the entropy of the strategy:

[0213]

[0214] Where, π θ (a|s t ) is strategy π θ In state s t The probability of taking action a.

[0215] The total loss function L(θ) includes policy loss, value loss, and entropy penalty:

[0216] L(θ)=L CLIP (θ)-c1L value (θ)+c2H(π θ )

[0217] Here, c1 and c2 are weight hyperparameters.

[0218] Step 5.5: Update the parameters of the policy network and value network using the gradient of the total loss function, and use the Adam optimizer for parameter updates:

[0219]

[0220] Where α is the learning rate. This represents the steepest ascent direction of the gradient loss function in the parameter space. In each iteration, updating the parameters along the gradient direction increases the loss function L(θ), thereby improving the strategy.

[0221] Step 5.6: In each time step, the Actor layer learns the policy π. θ (a|s). In each iteration of the training process, the Actor network outputs the probability distribution of the remote sensing satellite action space. Based on the generated probability distribution, the decoder selects remote sensing satellite actions and then updates the policy by maximizing the entropy (KL divergence) of each action. This minimizes the mean squared error. Calculate and update the Critic network.

[0222] Step 5.7: Update the policy network parameters according to the objective function of the PPO algorithm, using a pruning method to ensure the stability of the policy update, and simultaneously update the value network to minimize the mean squared error of the value function estimation; through continuous iterative training, obtain the optimal scheduling policy when training converges. Repeat steps 5.1-5.6 above until the predetermined number of training rounds is reached or all objectives are observed and the reward no longer increases.

[0223] As described in step 6, for different imaging tasks, the saved model is loaded. During training, the PPO algorithm learns a policy π. θ (a|s), this policy selects the optimal action a based on the current state s. Policy π θ (a|s) is usually represented by a neural network, whose output is the probability distribution of taking different actions in each state.

[0224] Step 6.1: Initialize the initial state of the system, including the initial state information of all remote sensing satellites and targets.

[0225] Step 6.2: Iterate through each time step, and at time step t, based on the current state s... t Use strategy π θ (a|s) Select an action a t Action a t =arg max a π θ (a|s t This specifies which remote sensing satellite should observe;

[0226] Step 6.3, execute action a t Then, update the system status s t+1 The updates include reducing the remaining visible time window for the corresponding target and updating the status of remote sensing satellites (such as the start of observation, the remaining duration after the observation is completed, etc.).

[0227] Step 6.4: Record the tasks performed at each time step, including the time, remote sensing satellite, target, and imaging task results (successful observation or cooling, etc.).

[0228] Step 6.5: Repeat steps 6.2 to 6.4 until the visible time window for all targets ends or the predetermined imaging task time range is reached. After completion, obtain the executing remote sensing satellite, start observation time, and end observation time for each point target imaging task, thus obtaining the remote sensing satellite imaging task planning scheme for different point targets.

[0229] On the other hand, embodiments of the present invention also provide a deep reinforcement learning scheduling device for satellite multi-point target imaging, comprising:

[0230] One or more processors;

[0231] A storage device is provided for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement a deep reinforcement learning scheduling method for satellite multi-point target imaging as described in the above technical solution.

[0232] For ease of implementation, an example is presented here. This invention uses one remote sensing satellite and 50 ground target tasks for algorithm training, and another 50 targets are selected for experimentation to illustrate the specific implementation of this invention. The experiment was developed based on a Python environment, with the following settings: learning rate of 0.1, discount factor of 0.99, pruning factor of 0.2, entropy coefficient of 0.01, value function coefficient of 0.1, environmental interaction steps of 2048, and training batch size of 100,000. The orbital parameters of the remote sensing satellite are shown in Table 1, and the geographic location visibility window and priority of the point targets are shown in Table 2.

[0233] Table 1 Satellite orbital parameters

[0234]

[0235] Table 2 Task Objective Information

[0236]

[0237] After modeling the mission scenario of a single-point target remote sensing satellite through steps 1, 2, 3, and 4, the experiment adopted the PPO algorithm for solution, and the final satellite mission scheduling scheme is shown in Table 3.

[0238] Table 3 Satellite Mission Scheduling Scheme

[0239]

[0240]

[0241] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A deep reinforcement learning scheduling method for satellite multi-point target imaging, characterized in that, Includes the following steps: Step 1: Preprocess the scene data to obtain the visible time window of satellite i to ground target j. , and These represent the start time and the end time, respectively. Step 2, Point target task constraint construction: Calculate the attitude transition time from each time window of each satellite to any other time window of any other satellite, and establish a four-dimensional matrix for storage; Step 2 includes the following sub-steps: Step 2.1: Create a remote sensing satellite imaging mission switching duration matrix, calculate the maximum number of observation windows for the mission target, create a matrix with a size of number of satellites × maximum number of observation windows × number of satellites × maximum number of observation windows, and save it as a .npy file. Step 2.2: Calculate the imaging target using the rotation matrix method. In the time window Target observed Each time slice within the visible duration Attitude transition time; let For the goal The position vector of the geocentric inertial frame at a specified time. For the goal Given the position vector in the geocentric inertial frame at a specified time, the target direction vector is calculated using the position vectors of the remote sensing satellite and the target. , ;Will Normalization yields the unit direction vector. ; Construct the target direction cosine matrix (DCM) and use the orthogonal basis matrix of the direction vectors as the target direction vectors. Assuming the initial attitude matrix points towards the target Calculate the attitude error matrix , The satellite's attitude at the initial moment; Using the trace of the rotation matrix Calculate the maximum values ​​of pitch angle, roll angle, and rotation angle. For the pose transition time between two target windows, every [time period]... Calculate the duration of one attitude transition ,in Given the angular velocity of the attitude transition, the attitude transition duration sequence within this time window is obtained. The start and end times of satellite i relative to ground target j; Step 2.3: Perform hypercube sampling on the attitude transition time sequence calculated in the previous step to obtain an estimated attitude transition time and assign it to... [ ][ ][ ][ ] indicates the task objective In the The window observed the mission objective. In the Estimated task switching time for each window, when the task target or If no visible window exists, assign a value of -1; Step 2.4, due to differences in maneuverability among remote sensing satellites, the imaging mission target... or In cases where there is no visible time window, the maneuver attitude constraints are not met, or the moment after the attitude transition ends is not within either of the time windows, it is impossible to complete the task. Switch to At this point, the matrix is ​​assigned the value -1; Step 3, Markov decision environment construction: This includes modeling the satellite imaging mission execution process as a multi-agent Markov decision process, defining the state space, action space, reward, state transition probability, and discount factor, and expressing them in a constrained manner at each training step. Step 4: Initialize the Markov decision environment and the PPO algorithm, and establish the policy network and value network; update the policy network parameters according to the objective function of the PPO algorithm, and update the value network at the same time to minimize the mean square error of the value function estimation. Through continuous iterative training, the optimal scheduling policy is obtained when the training converges. Step 5: Set different learning rates, discount factors, pruning coefficients, value function coefficients, and entropy coefficients, and set different training batch sizes and training periods. Start training and store the algorithm training results when training converges. Step 6: Load the PPO algorithm, input test data and initialize the environment. At each time step, read the strategy and input the observation start and end times of feasible targets, and output the imaging task planning strategy.

2. The deep reinforcement learning scheduling method for satellite multi-point target imaging as described in claim 1, characterized in that: Step 1, the preprocessing of scene data, includes: predicting the target visibility window based on the orbit and the maximum field of view of the remote sensing satellite; assuming the remote sensing satellite moves in a two-body model, predicting the visibility window of the satellite to each target point using the six orbital elements of the satellite, thus obtaining the visible time window of satellite i to ground target j. .

3. A deep reinforcement learning scheduling method for satellite multi-point target imaging as described in claim 1 or 2, characterized in that: Step 1 includes the following sub-steps: Step 1.1: Initialize the remote sensing satellite information. The six elements of the known satellite orbit are the semi-major axis. eccentricity ,inclination Right ascension of ascending node Perigeal argument , and the near point angle epoch time is The field of view of the remote sensing satellite is ; Step 1.2, calculate the satellite's mean anomaly angle: in, For the current time, This refers to an epochal time. The average velocity of the satellite is calculated using the following formula: in, For gravitational constant, Let k be the mass of the central celestial body, i.e., the mass of the Earth, and k be the Kepler constant. For the semi-major axis of the satellite orbit; Solving the Kepler equations yields the aperitone angle. : For the near-point angle, For orbital eccentricity, For the near-point angle, The current time; Calculate the true anterior angle : in, For the near-point angle, The orbital eccentricity; Calculate the satellite's geocentric coordinates: Calculate the coordinates of the remote sensing satellite in the geocentric inertial coordinate system based on its orbital parameters. : Step 1.3: Calculate the geocentric coordinates of the ground target. R is the Earth's radius; in, For the Earth's radius, The altitude of the ground target. Indicates the longitude of the ground target. Indicates the longitude of the ground target; Step 1.4, calculate the distance between the satellite and the Earth's center: Step 1.5, calculate the distance between the target and the Earth's center: Step 1.6: Determine whether the line connecting the satellite and the target passes through the Earth; in, The angle between the satellite and the target is calculated using the following formula: If the above formula is true, then the target is not blocked by the Earth; otherwise, the target is blocked by the Earth. Step 1.7: Calculate the angle between the remote sensing satellite and the ground target, and compare it with the remote sensing satellite's field of view. When... At that time, the ground target is within the field of view of the remote sensing satellite; Step 1.8, starting from the epoch, every... Repeat steps 1.1-1.7 to calculate the visibility of the mission target once, and obtain the visibility of each remote sensing satellite. For each ground target Visible time window .

4. The deep reinforcement learning scheduling method for satellite multi-point target imaging as described in claim 1, characterized in that: In step 3, remote sensing satellites are abstracted into intelligent agents. A remote sensing satellite has... Each imaging task to be completed is considered as a point target, with each target... It has a fixed geographical location It has multiple visible observation time windows. Each visible time window contains the visible start and end times. ; The process of remote sensing satellites performing imaging tasks on multiple ground point targets is modeled as a multi-agent Markov decision process, consisting of a quintuple. Sure, These are the state space, action space, reward, state transition probability, and discount factor. Describe the combined state of the imaging mission target and remote sensing satellite resources. It includes the observation status of the imaging target. Describe the satellite status. This includes a list of imaging missions completed by remote sensing satellites and their current operational status.

5. The deep reinforcement learning scheduling method for satellite multi-point target imaging as described in claim 4, characterized in that: The specific implementation method of step 3 is as follows; Actions of remote sensing satellites It includes two types: observed and unobserved, represented by different indices 0 and 1; Rewards in the Markov process are constructed using imaging task priorities: for each imaging task It has a priority This indicates the urgency and importance of the task, and the reward. Determined by the current state S and action A, this describes the benefit gained from taking an action in the current state, and the action rewards for different remote sensing satellites. There are different calculation methods; the total reward is the sum of all remote sensing satellite rewards. The strategy converges when the reward value is maximized and no longer changes, thus yielding the optimal strategy. The time is segmented according to the target requirement refresh cycle. The remote sensing satellite status and imaging mission observations in each time slice are used as status information. Different rewards are assigned to different actions in response to environmental feedback, with the goal of maximizing rewards to obtain the remote sensing satellite observation sequence. After the remote sensing satellite completes the imaging mission of a point target, a certain task switching time is required before the next target can be observed. The task switching time depends on the satellite's attitude maneuverability and requires attitude conversion according to the attitude requirements of the two imaging missions. The matrix values ​​calculated in step 2 are then read. [ ][ ][ ][ When the value is less than 0, it is considered the current target. exist The window cannot be in target After the window undergoes attitude transformation, it is observed that The target, when the value is greater than 0, is the current target. exist The window can target After the window undergoes attitude transformation, it is observed that If the maneuvering conditions for attitude transition are not met, the next target cannot be observed, and a new target that satisfies the attitude transition constraints is selected. The problem of scheduling multi-target imaging missions for remote sensing satellites is transformed into a problem of solving the optimal policy for an intelligent agent: reward state transition probability function. Describes the current imaging mission and the status of remote sensing satellites. The remote sensing satellites then performed the following actions. Subsequently, imaging missions and remote sensing satellites were transferred to a new state. probability, strategy For remote sensing satellites in status Next, take action The probability distribution, For stochastic policies; neural networks for policy functions Fitting to obtain the state Next action The goal is to determine the probability distribution of each remote sensing satellite and find the optimal strategy. To maximize the overall expected return of the constellation, the objective function is defined as follows: In the above formula, For strategy The parameters, Represents the initial state Expected value Let be the value function, representing the initial state. Next, according to the strategy The expected total return from execution; specifically, the value function is defined as the return from the state. Begin, in future time steps according to strategy The expected value of the cumulative discounted return that can be obtained from the action; Indicating in strategy The expected value of sampling below; As a discount factor, the closer the value is to 1, the greater the impact of future rewards on current decisions; the closer the value is to 0, the more attention is paid to recent rewards. Indicates at time step At that time, state and actions The following rewards; Indicates starting from the initial state Begin, according to strategy Execution, the expectation of the value function under all possible initial states; Indicates according to strategy Execution, at each time step Get instant rewards And future rewards will be calculated based on a discount factor. Perform weighted summation; maximize the objective function in the solution strategy. ,turn up Thus, the optimal action strategy for the timing is obtained.

6. The deep reinforcement learning scheduling method for satellite multi-point target imaging as described in claim 5, characterized in that: In step 4, the state is defined as including the current time and the observation state of the target, and the action is the decision to select the target for observation, which is expressed in a constrained manner in each step; the policy is represented by the policy network Actor, with the current state as the input and the probability distribution of each action as the output; the value of the state is estimated by the value network Critic, with the current state as the input and the corresponding value as the output; samples are collected under the current policy to record the state, action, reward, and next state information. Step 4.1: Construct a policy network, which is used to generate remote sensing satellite action policies under given conditions. Action probability distribution, including input layer, hidden layer, and output layer; input state The probability distribution of output actions ; Step 4.2: Construct a value network by estimating the observation value of all remote sensing satellite agents using a fully connected neural network, including the input layer, hidden layer, and output layer; input state Output the value of the state ; The observational value of all remote sensing satellite agents is estimated using a fully connected neural network. The Critic value network minimizes the mean square error. To update, the calculation method is as follows, where It is the time step The reward. in, Mean square error, time step The expected value obtained by empirical sampling. It is the time step The reward For value networks in time step The predicted value of the reward at that time, parameters These are parameters of the value network.

7. The deep reinforcement learning scheduling method for satellite multi-point target imaging as described in claim 5, characterized in that: Step 5 is implemented in the following sub-steps: Step 5.1: Begin training. During each training time step, the agent interacts with the environment and collects trajectory data during the training process. Each trajectory consists of a state, an action, a reward, and a next state; Step 5.2: Collect training experience through the policy network (Actor) to compute the advantage estimate at each time step. The generalized advantage estimation (GAE) is used to estimate advantage. GAE can smooth the advantage estimate by balancing bias and variance. The formula is as follows: in It is a discount factor. λ represents the guiding coefficient, which ranges from 0 to 1 and controls the decay rate of the guiding trace in the dominance function estimation. It is a counting variable, representing the current time step. The beginning of the first A future time step, This refers to the TD residual, and the formula for the TD residual is: During training, GAE accumulates data over a finite trajectory length, with the step size at training termination set to 1. The specific calculation process is as follows: Step 5.3: In each time step update, the probability ratio of the old strategy and the new strategy adopted by the remote sensing satellite before and after the update. Update the strategy: in, Is this a new strategy in the state? Take action below The probability, It represents the probability of the old strategy under the same state and action; Define the policy loss function based on the shearing probability ratio. To limit the update range of the policy: in, For time step The expected value obtained by empirical sampling. To show the time step At that time, a new strategy and old strategies The probability ratio between them This is an estimate of the dominance function; Represents the probability ratio Edit it to fit within the range Inside, This is the pruning constant, representing the range of allowed policy variations; Step 5.4, the value network estimates the value of each state. The mean squared error (MSE) is used to define the value loss function. ,in For time step The expected value obtained by empirical sampling. The value network is in state The predicted value below, It is the time step The target value is obtained through actual observation during training; To encourage strategy exploration, an entropy penalty term is added. Calculate the entropy of the strategy: in, It is a strategy In state Take action below The probability of; Total loss function Includes policy loss, value loss, and entropy penalty: in, and It is a weight hyperparameter; Step 5.5: Update the parameters of the policy network and value network using the gradient of the total loss function, and use the Adam optimizer for parameter updates: in, For learning rate, The gradient loss function has the steepest ascent direction in the parameter space. In each iteration, the parameters are updated along the gradient direction to make the loss function... Increase, thereby improving the strategy; Step 5.6, at each time step, the Actor layer learns the strategy. In each iteration of the training process, the policy network (Actor) outputs the probability distribution of the remote sensing satellite action space. Based on the generated probability distribution, the decoder selects remote sensing satellite actions and then updates the policy by maximizing the entropy of each action to minimize the mean squared error. Calculate and update the value network Critic; Step 5.7: Update the policy network parameters according to the objective function of the PPO algorithm, use the pruning method to ensure the stability of the policy update, and update the value network at the same time to minimize the mean square error of the value function estimation; through continuous iterative training, obtain the optimal scheduling policy when the training converges; repeat the above steps 5.1-5.6 until the predetermined number of training rounds is reached or all objectives are observed and the reward no longer increases.

8. The deep reinforcement learning scheduling method for satellite multi-point target imaging as described in claim 5, characterized in that: In step 6, for different imaging tasks, the saved models are loaded. During training, the PPO algorithm learns a strategy. This strategy is based on the current state. Select the optimal action ,Strategy Represented by a neural network, its output is a probability distribution of taking different actions in each state; Step 6.1: Initialize the initial state of the system, including the initial state information of all remote sensing satellites and targets; Step 6.2, iterate through each time step, and at each time step... Based on the current state Use strategy Choose an action ,action It specifies which remote sensing satellite should observe; Step 6.3, perform the action Then, update the system status. The updates include reducing the remaining visible time window for the corresponding targets and updating the status of remote sensing satellites; Step 6.4: Record the tasks performed at each time step, including the time, remote sensing satellite, target, and imaging task results; Step 6.5: Repeat steps 6.2 to 6.4 until the visible time window of all targets ends or the predetermined imaging task time range is reached. After completion, obtain the remote sensing satellite, start observation time, and end observation time for each point target imaging task, and obtain the imaging task planning scheme of the remote sensing satellite for different point targets.

9. A deep reinforcement learning scheduling device for satellite multi-point target imaging, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a deep reinforcement learning scheduling method for satellite multi-point target imaging as described in any one of claims 1 to 8.

Images