A satellite resource scheduling optimization method based on federated reinforcement learning

Abstract

The present invention provides a satellite resource scheduling optimization method based on federated reinforcement learning, which abstracts the Earth observation satellite resource scheduling optimization problem into a discrete Markov decision problem, and applies a federated reinforcement learning algorithm to solve the optimal solution for Earth observation satellite resource scheduling. The present invention utilizes federated learning and reinforcement learning technologies to explore the implicit intrinsic correlation between Earth observation tasks and satellite resources, fully explores various related characteristic indicators of each intelligent agent during the training process, forms an efficient and high-quality global scheduling optimization model, and ultimately generates an optimal and conflict-free Earth observation satellite resource scheduling optimization solution. In addition, the optimization method effectively reduces the dependence on scheduling heuristic rules and artificial historical allocation experience in satellite resource scheduling optimization, improves the effectiveness and accuracy of the resource scheduling method, and can significantly improve the level of intelligent management and control of Earth observation satellite resource scheduling.

Description

Technical Field

[0001] The present invention relates to the technical field of earth observation satellite resource planning, and in particular to a satellite resource scheduling optimization method based on federated reinforcement learning. Background Art

[0002] The resource scheduling optimization problem for Earth observation satellites is a complex combinatorial optimization problem with time window constraints and resource constraints. It requires comprehensive consideration of the characteristics of various satellite resources and observation missions, as well as the various constraints between them, in order to maximize the utilization of satellite resources or the completion rate of missions as the scheduling optimization goal. It also requires full consideration of the characteristics of various satellite resources, mission characteristics, and various constraints between missions and resources. It also requires a reasonable arrangement of satellite observation resource scheduling plans based on mission objectives to generate the optimal scheduling plan for Earth observation satellites.

[0003] The traditional method to solve such problems is based on constrained planning problems, using heuristic algorithms or super-meta-heuristic algorithms and machine learning algorithms, and adopting empirical rules to seek the optimal and worst-case satellite resource scheduling solutions within an acceptable time frame. However, this type of intelligent algorithm is highly dependent on empirical rules, and the design of such empirical rules requires a lot of professional knowledge and rich industry experience, which is difficult and costly to build. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to address the shortcomings of the existing technology and provide a satellite resource scheduling optimization method based on federated reinforcement learning. This method can effectively solve the problem proposed in the background technology that the traditional method relies heavily on empirical rules, and the design of such empirical rules requires a lot of professional knowledge and rich industry experience, which is difficult and costly to build.

[0005] The present invention specifically provides a satellite resource scheduling optimization method based on federated reinforcement learning. The problem of optimizing Earth observation satellite resource scheduling is abstracted into a discrete Markov decision problem. The federated reinforcement learning algorithm is applied to solve the optimal solution for Earth observation satellite resource scheduling. The present invention specifically includes the following steps:

[0006] Step 1: Establish a reinforcement learning DQN (Deep Q-learning, DQN) model for each agent in the federated reinforcement learning algorithm, set the state space of the agent in the environment, the behavior space that the agent can decide, and the environment's behavioral rewards for the agent;

[0007] Step 2: Based on the reinforcement learning DQN algorithm, a corresponding neural network is established for each agent, and the target neural network (Target-Q) is used to obtain the approximate value function;

[0008] Step 3: The agent uses an ∈-greedy strategy to decide the next action based on the assigned goal and its current state, interacts with the environment, obtains its next state, stores the decision experience in the replay memory unit, and updates the target neural network (Target-Q) model parameters in a timely manner according to the gradient of the error function.

[0009] Step 4: After the loop reaches the set number of times, the local target neural network model parameters are passed to the agent DQN model used for parameter aggregation. The agent DQN model is recorded as the joint virtual model for subsequent federated learning.

[0010] Step 5: Aggregate the parameters uploaded by all agents and return the results to each agent for parameter update. The joint virtual model aggregates the parameters of the agents and returns the corresponding aggregation results.

[0011] Step 6: Each agent soft-updates the received return results and the target neural network (Target-Q) model parameters to obtain the latest local reinforcement learning model parameters;

[0012] Step 7: Repeat steps 3 to 6 until the target task is completed and the optimal reinforcement learning model parameters are obtained;

[0013] Step 8: Use the optimal reinforcement learning model parameters to build an enhanced reinforcement learning model (E-Agent DQN) to obtain the optimal satellite resource scheduling solution;

[0014] In step 5, the deep reinforcement learning DQN model parameter sample uploaded by the i-th agent is recorded as θ i (such as memory capacity N, initial weight ω, etc.), and at the same time build a deep reinforcement learning DQN model for fusion learning, denoted as a joint virtual agent, and the parameter sample set of the joint virtual agent is Θ={θ i}, 1≤i≤N, the center point θ of the sample is obtained by calculating the average value avg :

[0015]

[0016] θ avg This is the aggregated result returned by the joint virtual model.

[0017] In step 6, the agent receives the aggregated result θ returned by the joint virtual model. avg After that, the local deep reinforcement learning DQN model is updated in a soft update manner, that is, θ is updated with a proportion τ. avg Add model parameter sample θ i After updating, the neural network parameters θ′ of the deep reinforcement learning DQN model arei for:

[0018] θ′ i =(1-τ)θ i +τ·θ avg

[0019] At this point, a federated learning process is completed, where τ∈[0,1]. When τ is 0, it means that the parameters will not be updated. avg The local deep reinforcement learning DQN model is not integrated. When τ is 1, it means that the local deep reinforcement learning DQN model directly copies the update parameters θ. avg ;

[0020] The formula for updating parameters of the joint virtual agent is:

[0021]

[0022]

[0023] The superscript v represents the serial number of the joint virtual agent, θ t (v) is the neural network parameter of the joint virtual agent deep reinforcement learning DQN model, θ t (i) is the neural network training parameter of the deep reinforcement learning DQN model of the i-th agent at time t, v t (v) is the parameter change value of the deep reinforcement learning DQN model of the vth joint virtual agent, l t is the learning rate, N t is the number of active agents at time t, Loss(.) is the loss function, and ρ is the system weight, which is generally 0.5.

[0024] In step 8, the satellite resource scheduling optimization problem is modeled using a Markov decision process (MDP). The three elements that constitute the MDP are the environment state s, the decision action a, and the reward r.

[0025] The decision-making process is to select the corresponding action according to the strategy based on the current state, obtain the corresponding decision reward, and use the Q-value function to describe the expected reward return of the entire Markov decision process;

[0026] In the Markov decision process, the agent selects a decision action a based on a certain strategy in the environment state s. The decision action a acts on the external environment in which the agent interacts, causing the environment state s to change accordingly, thereby obtaining a corresponding reward r. The goal is to obtain a strategy with the optimal reward based on this interaction process.

[0027] The satellite resource scheduling optimization problem is modeled using a Markov decision process. This essentially uses stochastic processes to formally describe the application scenario of Earth observation satellite resource scheduling, extracts the three elements of the Markov decision process, and transforms it into a resource scheduling optimization model that can be described and solved using reinforcement learning.

[0028] Specifically, for the Earth observation satellite resource scheduling application scenario, the model including the external environment state, decision-making actions, and reward return evaluation indicators is abstracted as follows:

[0029] The state set of each satellite resource and observation task in the Earth observation mission is abstracted as the state of the Markov decision process, recorded as the environment state; the satellite resource decision action variable is abstracted as the action of the Markov decision process, recorded as the decision action, and the satellite resource scheduling performance evaluation index is used as the decision reward in the Markov decision process.

[0030] In step 8, the environment state for Earth observation satellite resource scheduling is a description of the application scenario for Earth observation satellite resource debugging, including the description of the attribute characteristics of Earth observation satellite resources and the characteristics of observation tasks. The entire environment state includes the observation state and the task state.

[0031] When an idle time window of satellite observation resources is visible and available to a task, the state of the corresponding observation state matrix position is set to 1, otherwise it is set to 0;

[0032] For different satellite resources in each time window, the number 0 or 1 is used to mark them according to whether the satellite resources can meet the observation requirements in the current time window, and it is determined whether various types of satellite observation resources are idle relative to the observation task in each time window;

[0033] The state matrix of satellite observation resources in a given satellite resource scheduling scenario is represented by a 0-1 matrix, thereby determining the availability of satellite observation resources relative to the observation task at each moment, thereby determining the observation state matrix and constructing the state of satellite resources in the time dimension;

[0034] The observation state matrix and task state matrix in the same time window are integrated to form the environmental state of the Earth observation satellite resource scheduling in the current time window. The designed environmental state matrix S [TaskS,TaskE] In the following form:

[0035]

[0036] Among them, TaskS, TaskE represent the start time and end time of the current time window respectively, and the environment state matrix S [TaskS,TaskE]The first column is the serial number of each task, and each of the other columns is the corresponding task start time, task end time, task priority, imaging time, total number of observation tasks, number of targets to be observed, task observation time requirement, ..., equipment conversion time, storage capacity, ...;

[0037] Environmental state matrix S [TaskS,TaskE] The first part, TaskS, is the task status, and the second part, TaskE, is the resource status. Their values ​​are related to the actual observation task scenario. The above data is only used to represent the matrix form. The matrices of different time windows form the state space of the MDP model of Earth observation satellite resources.

[0038] In step 8, regarding decision actions: The Earth observation satellite resource scheduling problem is essentially an optimization problem with multiple constraints, and is an NP-hard problem. In specific application scenarios, it is necessary to comprehensively consider the constraints of various satellite resources and mission characteristics, and determine the reach of the corresponding decision-making behavior while satisfying various resource constraints in the current state;

[0039] The Earth satellite observation task scheduling problem is expressed as a five-tuple<E,S,T,C,F> Description, where E is the observation period, generally defined as 24 hours, is a collection of observation satellites, Indicates the Nth S observation satellites, is the set of observation tasks, Indicates the Nth T observation tasks, C is the set of constraints, and F is the objective function;

[0040] An observation mission can be imaged by more than two satellites. Each observation satellite has several visible time windows for the observation mission. The jth observation satellite S is j For the i-th observation task T i The kth visible time window of ijk =[ws ijk ,we ijk ] is the visible time window, ws ijk is the start time of the visible window, we ijk is the end time of the visibility window. Within a specific time window, the observation satellite can define the mission T i The visible time window set O i :

[0041]

[0042] Among them, N ij is the jth observation satellite S j For the i-th observation task T i Time window data, NS is the total number of satellites available for selection;

[0043] Assume that the i-th observation task T i The priority p i The required imaging time is d i , the jth observation satellite S j The sensor sway rate is r j , the stabilization time after side swing is h j , the storage space required per unit time for imaging is α j , the maximum storage capacity is M j , the maximum number of side glances allowed is R j , x ijk is the decision variable, where

[0044]

[0045] In step 8, the Earth satellite observations satisfy the following constraints:

[0046] The observation task uniqueness constraint means that the observation task is only observed once by the observation satellite and cannot be interrupted, which is expressed as follows:

[0047]

[0048] Transition constraints between satellite observation activities: There must be enough time between two consecutive imaging activities of the observation satellite to ensure that the onboard remote sensor can perform attitude transformation, including the side swing rotation time of the remote sensor. ikj -g i'jk' | and the stabilization time after side swing h j , which is expressed as follows:

[0049] we ijk +|g ikj -g i'jk' |+h j ≤ws i'jk'

[0050] And we ijk ≤ws i'jk'

[0051] Among them, g ikj ,g i'jk' Respectively represent the start and end time of posture conversion;

[0052] Satellite memory capacity constraints: The onboard memory capacity is limited, and the data acquired by satellite imaging is i Cannot exceed the storage capacity limit M j , which is expressed as follows:

[0053]

[0054] where α j is the number of satellite imaging times;

[0055] Satellite side-viewing times constraint: Due to satellite resource and maneuverability limitations, a satellite can only complete a limited number of R j The side-view imaging action is expressed as follows:

[0056]

[0057] For a satellite observation task, the decision action of satellite resource scheduling is transformed into whether the satellite observation resource accepts the current observation task at a time t, which is determined by the variable a. i To describe:

[0058]

[0059] The entire satellite observation resource scheduling strategy π is described as The value of i ranges from 1 to N S .

[0060] In step 8, for reward returns: a comprehensive evaluation index of satellite observation scheduling performance is constructed by comprehensively considering the completion of observation tasks, the priority of observation targets, and the utilization rate of satellite observation resources, where:

[0061] Sub-goal 1: Maximize the priority y1 of the observed target:

[0062]

[0063] Where a is the weight parameter, p i is the priority of target i;

[0064] Sub-goal 2: The priority y2 of maximizing the number of target observations is:

[0065]

[0066] Where b is the weight parameter and max is the maximum value function;

[0067] Sub-goal 3: Minimize resource consumption y3 as:

[0068]

[0069] Among them C i is the number of resources consumed by task i observation, c is the weight parameter;

[0070] The objective function of satellite resource scheduling is a multi-objective planning problem. To simplify the calculation, the ideal point method is used to transform the multi-objective planning problem into a single-objective planning problem, that is, to first find the optimal solution of a single objective, that is, the optimal solution of the three sub-objectives. and and the worst solution for the three sub-goals and Then calculate the relative closeness h between the target value and the optimal solution and the relative closeness H between the target value and the worst solution under any scheme:

[0071]

[0072]

[0073] Where η, ρ, and γ are the weights of targets y1, y2, and y3, respectively, satisfying: η + ρ + γ = 1. They are set according to the actual requirements of the observation task, such as 0.5 for η, 0.3 for ρ, and 0.2 for γ. According to the above formula, the objective function (i.e., reward) is transformed into maximizing the reward r, which is expressed as follows:

[0074]

[0075] The instantaneous return r in the Markov decision process model of observation satellite resource scheduling is set as:

[0076]

[0077] In step 8, based on the abstraction and description of each element in the satellite observation resource scheduling problem, the satellite observation resource scheduling strategy, that is, the description of the actions of the reinforcement learning network, specifically includes: determining the visible time window of the satellite observation task, determining whether to accept the measurement and control task, and determining the observation time window and observation resources for the satellite observation task.

[0078] In step 8, determining the visible time window of the satellite observation task includes: based on the discretization of the observation time period and the design of the observation state, by judging whether the start time and end time of the observation task to be assigned are within the visible time window range of each satellite observation resource, thereby determining the set of time windows in which different satellite observation resources may complete the same observation task.

[0079] The determination of whether to accept the measurement and control task includes: judging whether to accept the current observation task based on the satellite visibility time window for completing the observation task and the constraint conditions. If an observation task does not have an observation visibility time window that can complete the task, it can be determined as an observation task that is temporarily impossible to complete.

[0080] The determination of the observation time window and observation resources for the satellite observation task includes: according to the federated reinforcement learning algorithm, based on the set of visible time windows for completing the observation task, deciding to obtain a visible time window uniquely corresponding to the satellite observation resource. The satellite observation multi-agent federated reinforcement learning algorithm can determine the satellite observation resources and observation visible time window for completing the observation task.

[0081] The method of the present invention is to use federated learning and reinforcement learning technology to explore the implicit intrinsic correlation between earth observation tasks and satellite resources, use a single intelligent agent to autonomously learn and optimize the model parameters in the satellite resource scheduling process, use federated learning, and fuse the model parameters of each single intelligent agent based on reinforcement learning to construct an earth observation satellite resource scheduling optimization model; secondly, use the federated learning method with adaptive weights to fully explore the available features in the intelligent agent training process to form a higher quality global model.

[0082] The present invention has the following beneficial effects:

[0083] 1. The present invention utilizes federated learning and reinforcement learning technologies to explore the implicit intrinsic correlation between Earth observation tasks and satellite resources, utilizes a single intelligent agent to autonomously learn and optimize the model parameters in the satellite resource scheduling process, utilizes federated learning, and fuses the model parameters of each single intelligent agent based on reinforcement learning to construct an Earth observation satellite resource scheduling optimization model. It fully explores the various related characteristic indicators of each intelligent agent during the training process, forms an efficient and high-quality global scheduling optimization model, and ultimately generates the optimal and conflict-free Earth observation satellite resource scheduling optimization plan. This optimization method effectively reduces the dependence on scheduling heuristic rules and manual historical allocation experience in satellite resource scheduling optimization, improves the effectiveness and accuracy of the resource scheduling method, and can significantly improve the level of intelligent management and control of Earth observation satellite resource scheduling.

[0084] 2. The aggregation method adopted by the present invention is an adaptive weighted federated learning method. The joint virtual agent will calculate its objective function index based on the local training model received from each agent. If the agent weight update condition is met, the contribution of its model accuracy index to the model training accuracy is calculated for each agent, and the corresponding weighted average is generated to generate the global model parameters;

[0085] The joint virtual agent sends the updated global model parameters to the local agents. After receiving them, each local agent retains the local model parameter characteristics, performs model training on local training data, and uploads the local model and training accuracy indicators to the joint virtual agent again after the training is completed. This fully exploits the available features in the agent training process and forms a higher-quality global model, thereby improving model accuracy and convergence efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0087] Figure 1 is a flow chart of the steps of the method of the present invention;

[0088] Figure 2 It is a flow chart for constructing a satellite observation resource scheduling strategy of the present invention;

[0089] Figure 3 It is a schematic diagram of the framework of the multi-agent federated reinforcement learning satellite resource scheduling algorithm of the present invention. DETAILED DESCRIPTION

[0090] Example: Figure 1 、 Figure 2 、 Figure 3 As shown, the present invention provides a technical solution, a satellite resource scheduling optimization method based on federated reinforcement learning, which abstracts the Earth observation satellite resource scheduling optimization problem into a discrete Markov decision problem and applies the federated reinforcement learning algorithm to solve the optimal solution for Earth observation satellite resource scheduling;

[0091] The imaging satellite resource scheduling method specifically includes the following steps:

[0092] Step 1: Establish a reinforcement learning DQN model for each agent, set the state space S of the agent in the environment, the action space A that the agent can decide, and the environment's reward R for the agent's behavior;

[0093] Step 2: Based on the reinforcement learning DQN algorithm, a corresponding neural network is established for each agent, and the target neural network (Target-Q) is used to approximate the value function;

[0094] Step 3: The agent uses the ∈-greedy strategy to decide the next action based on the assigned goal and its current state, interacts with the environment, obtains its next state, stores the decision experience in the replay memory unit, and updates the target neural network (Target-Q) model parameters in a timely manner according to the gradient of the error function.

[0095] Step 4: After the loop reaches a set number of times, such as 100,000 times, the local target neural network (Target-Q) model parameters are transferred to the joint virtual model for subsequent federated learning;

[0096] Step 5: Aggregate the parameters uploaded by all agents and return the results to each agent for parameter update. The joint virtual model aggregates the parameters of the agents and returns the corresponding aggregation results.

[0097] Step 6: Each agent soft-updates the received return results and the target neural network (Target-Q) model parameters to obtain the latest local reinforcement learning model parameters;

[0098] Step 7: Repeat steps 3 to 6 until the target task is completed and the optimal reinforcement learning model parameters are obtained;

[0099] Step 8: Finally, the obtained optimal reinforcement learning model parameters are used to construct an enhanced reinforcement learning model (E-AgentDQN) to obtain the optimal satellite resource scheduling solution;

[0100] This parameter aggregation algorithm uses federated learning to aggregate the parameters uploaded by agents on a joint virtual model, remove abnormal and outlier parameters, and perform weighted averaging on the remaining parameters. The results are then returned to each agent for parameter update. The details are as follows:

[0101] Let the model parameter sample uploaded by the i-th agent be θ i , the model parameter sample set of the joint virtual agent is Θ={θ i}, 1≤i≤N, the center point θ of the sample is obtained by calculating the average value avg :

[0102]

[0103] The agent soft-updates the received results and the local neural network parameters, that is, adding the results to the local parameters with a certain proportion;

[0104] The agent receives the updated parameter θ returned by the joint virtual model avg After that, the local neural network model is updated in a soft update manner, that is, θ is updated with a certain proportion τ. avg Add the local current neural model parameters θ i In the above example, the updated neural network parameters θ′ are i for:

[0105] θ′ i =(1-τ)θ i +τ·θ avg

[0106] At this point, a federated learning process is completed, where τ∈[0,1]. When τ is 0, it means that the parameters will not be updated. avg Not integrated into the local model, when τ is 1, it means that the local model directly copies the updated parameters θ avg ;

[0107] Based on experience, the weight τ is set to a relatively large value at the beginning to speed up the training pace. Then, during the training iteration, the weight is gradually reduced to ensure convergence stability. In this way, each agent can learn from the experience of other agents, optimize the local model of the agent, form a good collaborative effect, and jointly complete the task;

[0108] The way to update parameters of the joint virtual agent is:

[0109]

[0110]

[0111] where θ t (v) is the neural network parameter of the virtual agent, θ t (i) is the neural network training parameter of the i-th agent, l t is the learning rate.

[0112] Based on the above technical solution, the three elements that constitute the Markov decision process (MDP) are the environment state (s), the decision action (a), and the reward return (r);

[0113] The decision-making process is to select the corresponding action according to the strategy based on the current state, obtain the corresponding decision reward, and use the Q-value function to describe the expected reward return of the entire Markov decision process;

[0114] In the process of optimizing and solving MDPs using reinforcement learning, the agent selects a corresponding decision action (a) based on a certain strategy under the environment state (s). The decision action (a) acts on the external environment in which the agent interacts, thereby causing the environment state (s) to change accordingly, thereby obtaining a corresponding reward return (r). The goal is to obtain the optimal reward return strategy based on this interaction process.

[0115] The modeling of the MDP model for optimizing Earth observation satellite resource scheduling essentially involves using stochastic processes to formally describe the application scenario of Earth observation satellite resource scheduling, extracting the three elements of the MDP, and then transforming it into a resource scheduling optimization model that can be described and solved using reinforcement learning.

[0116] Specifically, the application scenario of Earth observation satellite resource scheduling includes the abstraction of a model including the external environment state, decision actions, and reward return evaluation indicators;

[0117] The state set of related objects in the Earth observation mission is abstracted into the state of the MDP, recorded as the environment state, including satellite resources and observation tasks. The satellite resource decision action variables are abstracted into the actions of the MDP, recorded as decision actions. The satellite resource scheduling performance evaluation index is used as the decision reward in the MDP.

[0118] Based on the above technical solution, the environmental state: The environmental state of Earth observation satellite resource scheduling is a description of the application scenario of Earth observation satellite resource debugging, including the description of the attribute characteristics of Earth observation satellite resources and the characteristics of observation tasks. The entire environmental state consists of two parts: observation state and task state;

[0119] On the one hand, the satellite observation scheduling time is discretized periodically, the time interval division scale is determined according to the specific requirements of the observation mission, and the scheduling time is divided into time windows of different scales to further clarify whether the Earth observation satellite resources are visible and available time windows relative to the observation mission;

[0120] In a given time window, the observation state matrix of satellite resources relative to the observation task is constructed based on whether the satellite resources are visible and available to the task. Specifically, when a certain idle time window of satellite observation resources is visible and available to a task, the state of the corresponding position in the observation state matrix is ​​set to 1, otherwise it is set to 0.

[0121] For each time window, different satellite resources, such as satellite payload type and memory capacity, are marked with a number 0 or 1 according to whether the satellite resources can meet the observation requirements in the current time window, to determine whether various types of satellite observation resources are idle relative to the observation task in each time window;

[0122] That is, a 0-1 matrix is ​​used to represent the state matrix of satellite observation resources in a given satellite resource scheduling scenario, thereby determining the availability of satellite observation resources relative to the observation task at each moment. In this way, the state of satellite resources in the time dimension can be constructed by determining the observation state matrix.

[0123] On the other hand, the task status based on the observation task dimension is mainly defined by static indicators such as task sequence number, observation task start time, observation task end time, task priority, image resolution requirement, imaging time, minimum observation time of observation task, total observation time of completed observation task, number of completed observation tasks, total priority of completed observation tasks, total actual observation time of completed observation tasks, etc., to define the task status of the measurement and control task in different time windows, thereby constructing a task status matrix;

[0124] The observation state matrix and the task state matrix in the same time window are integrated to form the environmental state of the Earth observation satellite resource scheduling in the current time window. The designed environmental state matrix S is as follows:

[0125]

[0126] Among them, [TaskS, TaskE] represents the start time and end time of the current time window. The first column is the sequence number of each task, and each of the other columns is the corresponding task start time, task end time, task priority, imaging time, total number of observation tasks, number of targets to be observed, task observation time requirement, ..., equipment conversion time, storage capacity, ...;

[0127] The first part of the environment state matrix S is the task state, and the second part is the resource state. Its value is related to the actual observation mission scenario. The above data is only used to represent the form of the matrix. The matrices of different time windows form the state space of the MDP model of Earth observation satellite resources.

[0128] Based on the above technical solution, the decision-making action is: The resource scheduling problem of Earth observation satellites is essentially an optimization problem with multiple constraints. It is an NP-hard problem. In specific application scenarios, it is necessary to comprehensively consider the constraints of various satellite resources and mission characteristics, and determine the reachable range of corresponding decision-making actions while satisfying various resource constraints in the current state;

[0129] Select decision actions based on actual observation application scenarios, that is, select appropriate satellite observation resources to meet the observation requirements of the corresponding tasks, to ensure that the current observation resources can meet the various constraints of the current observation tasks, and ensure that the scheduling of various satellite resources in actual applications is feasible and has greater practicality;

[0130] Satellites orbit the Earth as it rotates. Satellites can only observe ground targets during specific time periods. Observation targets are all "point targets," meaning each observation mission can be completed by a single satellite in a single attempt. Each mission has imaging time constraints, payload type, and image resolution requirements. Satellite resource scheduling involves selecting multiple satellites that meet these requirements to observe the target. The decision-making process involves selecting the best satellite observation resource within the different visible time windows of multiple satellite observation resources to achieve the optimal scheduling objective function.

[0131] The Earth satellite observation task scheduling problem is expressed as a five-tuple<E,S,T,C,F> Description, where E is the observation period, generally defined as 24 hours, is a collection of observation satellites, is the set of observation tasks, C is the set of constraints, and F is the objective function;

[0132] An observation mission can be imaged by multiple satellites, and each observation satellite has several visible time windows for it, denoted by O ijk =[ws ijk ,we ijk ] is the visible time window. Within a specific time window, the observation satellite can define the task T i The set of visible time windows of :

[0133]

[0134] Among them, N ij Satellite S j For Task T i Time window data, O ijk Satellite S j For Task T i The kth time window, N S is the total number of satellites available for selection.

[0135] Assume task T i The priority p i , the required imaging time d i , satellite S j The sensor sway rate r j , the stabilization time after side swing h j , the storage space required per unit time for imaging is α j , maximum storage capacity M j , the maximum number of side glances allowed R j , x ijk is the decision variable, where

[0136]

[0137] Based on the above technical solution, Earth satellite observation must meet the following constraints:

[0138] Mission uniqueness constraint: a mission is only observed once by the satellite and cannot be interrupted;

[0139]

[0140] Transition constraints between satellite observation activities: There must be enough time between two consecutive imaging activities of the satellite to ensure that the onboard remote sensor can perform attitude transitions, including the sensor's side swing rotation time and the stabilization time after the side swing;

[0141] we ijk +|g ikj -g i'jk' |+h j ≤ws i'jk'

[0142] And we ijk ≤ws i'jk'

[0143] Satellite memory capacity constraints: The onboard memory capacity is limited, and the data acquired by satellite imaging cannot exceed its storage capacity limit;

[0144]

[0145] Satellite side-viewing limit: Due to satellite resource and maneuverability limitations, a satellite can only perform a limited number of side-viewing maneuvers.

[0146]

[0147] For a satellite observation task, the decision action of satellite resource scheduling is transformed into whether the satellite observation resource accepts the current observation task and which satellite observation resource accepts the observation task at a certain time t;

[0148]

[0149] The entire satellite observation resource scheduling strategy π is described as The value of i ranges from 1 to N S .

[0150] Based on the above technical solution, the reward reward is: a comprehensive evaluation index of satellite observation scheduling performance constructed by comprehensively considering the completion of observation tasks, the priority of observation targets, and the utilization rate of satellite observation resources, among which:

[0151] Sub-goal 1: Maximize the priority of the observed target:

[0152]

[0153] Sub-goal 2: Prioritize the number of target observations:

[0154]

[0155] Sub-goal 3: Minimize resource consumption:

[0156]

[0157] Among them C i is the number of resources consumed by task i observation, c is the weight parameter;

[0158] The objective function of satellite resource scheduling is a multi-objective programming problem. To simplify the calculation, the ideal point method is used to transform it into a single-objective programming problem, that is, to first find the optimal solution of a single objective. and and the worst solution and Then calculate the relative closeness between the target value and the optimal solution and the worst solution under any scheme, let

[0159]

[0160]

[0161] Where η, ρ and γ are the weights of targets y1, y2 and y3 respectively, which can be set according to the actual requirements of the observation task. According to the above formula, the objective function can be transformed into

[0162]

[0163] The immediate reward in the MDP model of observation satellite resource scheduling is set as:

[0164]

[0165] like Figure 2 As shown in the figure, based on the above technical solution and the abstraction and description of each element in the satellite observation resource scheduling problem, the satellite observation resource scheduling strategy, that is, the description of the action of the reinforcement learning network, specifically includes: determining the visible time window of the satellite observation task, determining whether to accept the measurement and control task, and determining the observation time window and observation resources for the satellite observation task.

[0166] Based on the above technical solution, the visible time window of the satellite observation task is determined: based on the discretization of the observation time period and the design of the observation state, by judging whether the start time and end time of the observation task to be assigned are within the visible time window range of each satellite observation resource, the set of time windows that different satellite observation resources may complete for the same observation task is determined.

[0167] Based on the above technical solution, determine whether to accept the measurement and control task: based on the satellite visibility time window for completing the observation task, determine whether to accept the current observation task according to the constraints. If an observation task does not have an observation visibility time window that can complete the task, it can be determined as an observation task that is temporarily impossible to complete.

[0168] Based on the above technical solution, the observation time window and observation resources for the satellite observation mission are determined: according to the multi-agent deep reinforcement learning satellite resource scheduling algorithm, based on the set of visible time windows for completing the observation mission, the decision can obtain the visible time window that uniquely corresponds to the satellite observation resources. The satellite observation multi-agent deep reinforcement learning satellite resource scheduling algorithm decision can determine the satellite observation resources and observation visible time window for completing the observation mission.

[0169] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium is capable of storing a computer program that, when executed by the data processing unit, can execute the invention content of the satellite resource scheduling optimization method based on federated reinforcement learning provided by the present invention and some or all of the steps in each embodiment. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0170] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. The computer program software product can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0171] This invention provides a method for optimizing satellite resource scheduling based on federated reinforcement learning. There are numerous methods and approaches for implementing this technical solution. The above is merely a preferred embodiment of the invention. It should be noted that those skilled in the art may make improvements and modifications without departing from the principles of the invention, and such improvements and modifications are also within the scope of protection of the invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A satellite resource scheduling optimization method based on federated reinforcement learning, characterized in that: The following steps are involved: Step 1: Establish a deep reinforcement learning (DQN) model for each agent in the federated reinforcement learning algorithm, set the state space of each agent in the environment, the decision-making behavior space of the agent, and the environment's behavioral rewards for the agent; Step 2: Based on the reinforcement learning DQN algorithm, a corresponding neural network is established for each agent, and the target neural network is used to obtain the approximate value function; Step 3: The agent uses the ϵ-greedy strategy to decide the next action based on the assigned goal and its current state, interacts with the environment, obtains its next state, stores the decision experience in the replay memory unit, and updates the target neural network model parameters based on the gradient of the error function. Step 4: After the loop reaches the set number of times, the local target neural network model parameters are passed to the agent DQN model used for parameter aggregation. The agent DQN model is recorded as the joint virtual model for subsequent federated learning. Step 5: Aggregate the parameters uploaded by all agents and return the results to each agent for parameter update. The joint virtual model aggregates the parameters of the agents and returns the corresponding aggregation results. Step 6: Each agent soft-updates the received aggregation results and the target neural network model parameters to obtain the latest local reinforcement learning model parameters; Step 7: Repeat steps 3 to 6 until the target task is completed and the optimal reinforcement learning model parameters are obtained; Step 8: Using the optimal reinforcement learning model parameters to build an enhanced reinforcement learning model to obtain the optimal satellite resource scheduling solution; In step 5, the deep reinforcement learning DQN model parameter sample uploaded by the i-th agent is recorded as At the same time, a deep reinforcement learning DQN model for fusion learning is constructed, which is recorded as a joint virtual agent. The parameter sample set of the joint virtual agent is , the center point of the sample is obtained by calculating the average value : ， This is the aggregated result returned by the joint virtual model.

2. The method according to claim 1, characterized in that In step 6, the agent receives the aggregated results returned by the joint virtual model. After that, the local deep reinforcement learning DQN model is updated in a soft update manner, that is, the proportion Will Add model parameter samples After updating, the neural network parameters of the deep reinforcement learning DQN model are for: ， So far, a federated learning process has been completed, in which ,when When it is 0, it means that the parameters will not be updated. Without integrating the local deep reinforcement learning DQN model, when When it is 1, it means that the local deep reinforcement learning DQN model directly copies the updated parameters ; The formula for updating parameters of the joint virtual agent is: ， in, is the neural network parameter of the joint virtual agent deep reinforcement learning DQN model, is the neural network training parameter of the deep reinforcement learning DQN model of the i-th agent at time t, is the parameter change value of the deep reinforcement learning DQN model of the vth joint virtual agent, is the learning rate, is the number of active agents at time t, is the loss function, is the system weight.

3. The method according to claim 2, characterized in that In step 8, the satellite resource scheduling optimization problem is modeled using a Markov decision process. The three elements that constitute the Markov decision process are the environment state s, the decision action a, and the reward r. The decision-making process is to select the corresponding action according to the strategy based on the current state, obtain the corresponding decision reward, and use the Q-value function to describe the expected reward return of the entire Markov decision process; In the Markov decision process, the agent selects a decision action a based on the strategy in the environment state s. The decision action a acts on the external environment in which the agent interacts, causing the environment state s to change accordingly, thereby obtaining a corresponding reward r. The goal is to obtain the optimal reward strategy based on the interaction process. Modeling the satellite resource scheduling optimization problem using a Markov decision process is a process that uses stochastic processes to formally describe the application scenario of Earth observation satellite resource scheduling. The three elements of the Markov decision process are extracted and converted into a resource scheduling optimization model that can be described and solved using reinforcement learning. The model including the external environment state, decision-making actions and reward return evaluation indicators is abstracted as follows: The state set of each satellite resource and observation task in the Earth observation mission is abstracted as the state of the Markov decision process, recorded as the environment state; the satellite resource decision action variable is abstracted as the action of the Markov decision process, recorded as the decision action, and the satellite resource scheduling performance evaluation index is used as the decision reward in the Markov decision process.

4. The method according to claim 3, characterized in that In step 8, the environment state for Earth observation satellite resource scheduling is a description of the application scenario for Earth observation satellite resource debugging, including the description of the attribute characteristics of Earth observation satellite resources and the characteristics of observation tasks. The entire environment state includes the observation state and the task state. When an idle time window of satellite observation resources is visible and available to a task, the state of the corresponding observation state matrix position is set to 1, otherwise it is set to 0; For different satellite resources in each time window, the number 0 or 1 is used to mark them according to whether the satellite resources can meet the observation requirements in the current time window, and it is determined whether various types of satellite observation resources are idle relative to the observation task in each time window; The state matrix of satellite observation resources in a given satellite resource scheduling scenario is represented by a 0-1 matrix, thereby determining the availability of satellite observation resources relative to the observation task at each moment, thereby determining the observation state matrix and constructing the state of satellite resources in the time dimension; The observation state matrix and the task state matrix in the same time window are integrated to form the environmental state of the Earth observation satellite resource scheduling in the current time window. The designed environmental state matrix In the following form: ， Among them, TaskS, TaskE represent the start time and end time of the current time window respectively, and the environment state matrix The first column is the sequence number of each task.

5. The method according to claim 4, characterized in that In step 8, for the decision action: the Earth satellite observation task scheduling problem is represented by a five-tuple Description, where is the observation period, is a collection of observation satellites, Indicates the observation satellites, is the set of observation tasks, Indicates the Observation missions, is a set of constraints, is the objective function; An observation mission can be imaged by more than two satellites. Each observation satellite has a visible time window for the observation mission. The jth observation satellite is recorded as For the i-th observation task No. The visible time window is is the visible time window, is the visible window start time, The end time of the visibility window. Within a specific time window, the observation satellite can define the mission The set of visible time windows : ， in, is the jth observation satellite For the i-th observation task Time window data, is the total number of satellites available for selection; Assume that the i-th observation task Priority The imaging time required is , the jth observation satellite The sensor sway rate is , the stabilization time after side swing is , the storage space required per unit time for imaging is , the maximum storage capacity is , the maximum number of side glances allowed is , is the decision variable, where 。 6. The method according to claim 5, characterized in that In step 8, the Earth satellite observations satisfy the following constraints: The observation task uniqueness constraint means that the observation task is only observed once by the observation satellite and cannot be interrupted, which is expressed as follows: ， Transition constraints between satellite observation activities: There must be enough time between two consecutive imaging activities of the observation satellite to ensure that the onboard remote sensor can perform attitude transformation, including the side swing rotation time of the remote sensor. and stabilization time after side swing , which is expressed as follows: ， in, Respectively represent the start and end time of posture conversion; Satellite memory capacity constraints: The onboard memory capacity is limited, and the data acquired by satellite imaging Cannot exceed storage capacity limit , which is expressed as follows: ， in is the number of satellite imaging times; Satellite side-view limit: Due to satellite resource and maneuverability limitations, the satellite can only complete a limited number of side-views. The side-view imaging action is expressed as follows: ， For a satellite observation task, the decision action of satellite resource scheduling is transformed into whether the satellite observation resource accepts the current observation task at a time t, which is determined by the variable To describe: ， The entire satellite observation resource scheduling strategy Described as , i takes the value of 1~ .

7. The method according to claim 6, characterized in that In step 8, for reward returns: a comprehensive evaluation index of satellite observation scheduling performance is constructed by comprehensively considering the completion of observation tasks, the priority of observation targets, and the utilization rate of satellite observation resources, where: Sub-goal 1: Maximize the priority of the observed target for: ， Where a is the weight parameter, p i is the priority of target i; Sub-goal 2: Prioritize the number of target observations for: ， Where b is the weight parameter and max is the maximum value function; Sub-Goal 3: Minimize resource consumption for: ， Among them C i is the number of resources consumed by task i observation, c is the weight parameter; The objective function of satellite resource scheduling is a multi-objective planning problem. The ideal point method is used to transform the multi-objective planning problem into a single-objective planning problem, that is, to first find the optimal solution of the three sub-objectives. 、 and and the worst solution for the three sub-goals 、 and , and then calculate the relative closeness between the target value and the optimal solution under any scheme The relative closeness between the target value and the worst solution : ， in 、 and Target 、 and The weights satisfy: + + =1, set according to the actual needs of the observation task, and transform the objective function into maximizing the reward r, which is expressed as follows: ， Instantaneous returns in a Markov decision process model for satellite resource scheduling Set to: 。 8. The method according to claim 7, characterized in that In step 8, based on the abstraction and description of each element in the satellite observation resource scheduling problem, the visible time window of the satellite observation task is determined, whether to accept the measurement and control task is determined, and the observation time window and observation resources for the satellite observation task are determined.

9. The method according to claim 8, characterized in that In step 8, determining the visible time window of the satellite observation task includes: based on the discretization of the observation time period and the design of the observation state, by judging whether the start time and end time of the observation task to be assigned are within the visible time window range of each satellite observation resource, thereby determining the set of time windows in which different satellite observation resources may complete the same observation task; The determining whether to accept the measurement and control task includes: judging whether to accept the current observation task according to the satellite visibility time window for completing the observation task and the constraint conditions; if an observation task does not have an observation visibility time window that can complete the task, then it is determined to be an observation task that is temporarily impossible to complete; The determination of the observation time window and observation resources for the satellite observation task includes: according to the federated reinforcement learning algorithm, based on the set of visible time windows for completing the observation task, deciding to obtain a visible time window uniquely corresponding to the satellite observation resource. The satellite observation multi-agent federated reinforcement learning algorithm can determine the satellite observation resources and observation visible time window for completing the observation task.

Images