Deep reinforcement learning based heterogeneous payload scheduling method for earth observation satellite

Abstract

The application provides a kind of heterogeneous load scheduling method of earth observation satellite based on deep reinforcement learning, it is related to satellite observation technical field, including obtaining satellite parameters, task parameters and visible time window information based on the scheduled task timeline sequence of target satellite;Satellite scheduling state and candidate joint action space are constructed;Action probability is generated according to the current satellite scheduling state and the preset constraint condition;In the candidate joint action space, it is judged whether there is a candidate joint action identified as valid by action mask;If there is, the target joint action is selected in the candidate joint action space based on the action mask and joint action probability;If not, terminate iteration and output the current satellite scheduling scheme;After decoding target joint action, it is inserted into the scheduled task timeline sequence of target satellite to obtain a new satellite scheduling scheme.The application improves the decision accuracy of heterogeneous load collaborative scheduling in satellite observation.

Description

Technical Field

[0001] This invention relates to the field of satellite observation technology, and specifically to a method for scheduling heterogeneous payloads of Earth observation satellites based on deep reinforcement learning. Background Technology

[0002] Earth Observation Satellites (EOS) are artificial satellites that operate in fixed orbits, carry imaging payloads, and provide space-based information services, primarily through remote sensing imagery. EOS applications cover multiple fields, including military reconnaissance, environmental governance, agricultural yield estimation, and intelligent transportation, and are of significant strategic importance for safeguarding national security and promoting economic development. Optical payloads and Synthetic Aperture Radar (SAR) payloads are the two most common types of imaging payloads on EOS.

[0003] Existing EOS scheduling methods are mainly designed for single loads, including exact algorithms (such as dynamic programming and branch and bound), heuristic algorithms (such as greedy search and tabu search), and reinforcement learning methods (such as deep Q-learning and pointer networks).

[0004] However, the above method only analyzes each satellite carrying only one type of payload, does not consider the interaction between multiple types of payloads, and cannot handle the cooperative timing constraints between heterogeneous payloads. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a method for scheduling heterogeneous payloads for Earth observation satellites based on deep reinforcement learning. This method solves the problem that existing methods do not consider the interactive effects between multiple types of payloads and cannot handle the collaborative scheduling of heterogeneous payloads.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: Firstly, this application provides a method for scheduling heterogeneous payloads of Earth observation satellites based on deep reinforcement learning, including: S1, obtain satellite parameters, mission parameters and visible time window information based on the timeline sequence of scheduled missions of the target satellite; S2, Based on the satellite parameters, mission parameters, and visible time window information, construct the satellite scheduling state and candidate joint action space; the satellite scheduling state includes satellite state, mission state, and time window state; each candidate joint action in the candidate joint action space includes the mission number to be observed, payload type, satellite number, and visible time window number; S3, Generate an action mask based on the current satellite scheduling status and preset constraints; Process the satellite scheduling status through a pre-trained policy network to obtain joint action probabilities; The policy network includes a task encoder, a satellite encoder, a time window encoder, a hierarchical cross-attention aggregation module, a global context encoder, and a conditional layer-by-layer scoring network; The constraints include visible time window constraints, task priority constraints, satellite energy constraints, storage capacity constraints, and payload usage constraints. S4, determine whether there is a valid candidate joint action identified by the action mask in the candidate joint action space; if there is, select the target joint action in the candidate joint action space based on the action mask and the joint action probability; if there is no, terminate the iteration and output the current satellite scheduling scheme. S5, decode the target joint action and insert it into the timeline sequence of the scheduled tasks of the target satellite to obtain a new satellite scheduling scheme, update the satellite scheduling status according to the target joint action, and return to step S3.

[0007] Preferably, the step of processing the satellite scheduling state through a pre-trained policy network to obtain joint action probabilities includes: The task state is processed by the task encoder of the policy network to obtain the task embedding sequence, the satellite state is processed by the satellite encoder of the policy network to obtain the satellite embedding sequence, and the time window state is processed by the time window encoder of the policy network to obtain the time window embedding sequence. The hierarchical cross-attention aggregation module of the policy network processes the task embedding sequence, satellite embedding sequence, and time window embedding sequence to obtain an aggregated embedding sequence; The aggregated embedding sequence is processed by the global context encoder of the policy network to obtain a global context vector; The global context vector, task embedding sequence, satellite embedding sequence, and time window embedding sequence are processed by the conditional layer-by-layer scoring network of the policy network to obtain joint action probabilities; wherein, the conditional layer-by-layer scoring network includes an action scalar scoring branch and a task implicit scoring branch.

[0008] Preferably, the step of processing the global context vector, task embedding sequence, satellite embedding sequence, and time window embedding sequence through the conditional layer-by-layer scoring network of the policy network to obtain the joint action probability includes: The action scalar scoring branch processes the task embedding sequence, global context vector, satellite embedding sequence, and time window embedding sequence to obtain a joint action scalar score; the task implicit scoring branch processes the global context vector and multiplies the processing result with the task embedding sequence to obtain a task implicit score; wherein, the action scalar scoring branch includes a three-level multilayer perceptron. The implicit task score is added to the scalar score of the joint action to obtain the joint action probability.

[0009] Preferably, the training method for the policy network includes: The satellite scheduling scheme of randomly sampled Earth observation satellites is used, and the satellite scheduling status is iteratively updated based on the policy network to record the complete sampling trajectory of the target joint actions; The cumulative revenue is calculated based on the sampling trajectory, where the cumulative revenue is the sum of the task value of all payloads in the sampling trajectory completing the observation; The initial satellite scheduling state of the aforementioned scheduling scheme is cloned, and the joint action with the highest probability is selected step by step in a greedy mode of the policy network to deduce the baseline benefit. Calculate the advantage value for each sampling trajectory and normalize the advantage value within a batch; the advantage value is the cumulative gain minus the baseline gain. The initial satellite scheduling state of the scheduling scheme is cloned, and each step of the joint action recorded along the sampling trajectory is replayed step by step from the initial satellite scheduling state to perform state transition. The log probability and action entropy of each step of the action are recalculated with the current policy network parameters. The loss is calculated based on the logarithmic probability, action entropy, and dominance value. The gradient is calculated based on the backpropagation of the loss, and the gradient is clipped to a preset maximum value; The Adam optimizer is used to update the policy network parameters based on gradients. Repeat the above training steps, evaluate the performance of the policy network with different policy network parameters in the validation set, and save the best-performing policy network as the completed policy network when the preset total number of training rounds is reached.

[0010] Preferably, the target joint action decoding includes: The action index is divided by the product of the total number of satellites and the maximum visible time window for each mission. The quotient is the mission number, and the remainder is the local offset. The action index is the representation of the target joint action in the candidate joint action space. When the local offset is less than the number of optical satellites multiplied by the maximum number of visible time windows per mission, the payload type is determined to be an optical payload, and the local offset is used as the adjusted offset; when the local offset is greater than or equal to the number of optical satellites multiplied by the maximum number of visible time windows per mission, the payload type is determined to be a synthetic aperture radar payload, and the difference between the local offset and the number of optical satellites multiplied by the maximum number of visible time windows per mission is used as the adjusted offset. Divide the adjusted offset by the maximum number of visible time windows for each mission. The quotient is the satellite number, and the remainder is the time window number.

[0011] Preferably, the objective function of the policy network is: In the formula, The number of ground targets to be observed. Number the ground targets; The benefits of completing observations of ground targets; The number of satellites carrying synthetic aperture radar payloads, The satellite designation for the satellite carrying the synthetic aperture radar payload; The number of satellites carrying optical payloads. The satellite number is used to designate the satellite carrying optical payloads. This represents the maximum number of time windows during which a satellite can be visible to ground targets. Number the visible time window; This represents the decision variable indicating whether a satellite observes a ground target within the corresponding visible time window.

[0012] Preferably, the constraints include visible time window constraints, mission priority constraints, satellite energy constraints, storage capacity constraints, and payload usage constraints; The expression for the visible time window constraint is: In the formula, Set the start time of the selected visible time window; The end time of the selected visible time window; This is the start time of the observation; For the duration of the observation; The set of ground targets to be observed. Number the target; For Earth observation satellites, Number the satellite; This represents the set of visible time windows for observing ground targets from satellites. This represents the maximum number of time windows during which a satellite can be visible to ground targets. Number the visible time window; The expression for the task priority constraint is: In the formula, The earliest observable time for ground targets; This is the latest observable time for ground targets; For the same satellite targeting ground targets and Indicator of whether sequential or preceding observations are required; For attitude transition time, It is a pre-defined, very large positive integer; Total number of satellites; The expression for the satellite energy constraint is: In the formula, Energy consumed for observation; For the first The energy limit of a satellite; The expression for the storage capacity constraint is: In the formula, Storage space consumed for observation, For the first The maximum storage space limit for each satellite; The expression for the load using constraints is: In the formula, For ground targets Load observation sequence identifier; These are decision variables.

[0013] Secondly, this application also provides a heterogeneous payload scheduling system for Earth observation satellites based on deep reinforcement learning, comprising: The acquisition module is used to acquire satellite parameters, mission parameters, and visible time window information based on the timeline sequence of scheduled missions of the target satellite; The scheduling status and action space construction module is used to construct the satellite scheduling status and candidate joint action space based on the satellite parameters, mission parameters and visible time window information; the satellite scheduling status includes satellite status, mission status and time window status; each candidate joint action in the candidate joint action space includes the mission number to be observed, payload type, satellite number and visible time window number; The iterative loop module is used to execute iterative steps S3-S5 to output a satellite scheduling scheme, including: S3, Generate an action mask based on the current satellite scheduling status and preset constraints; Process the satellite scheduling status through a pre-trained policy network to obtain joint action probabilities; The policy network includes a task encoder, a satellite encoder, a time window encoder, a hierarchical cross-attention aggregation module, a global context encoder, and a conditional layer-by-layer scoring network; The constraints include visible time window constraints, task priority constraints, satellite energy constraints, storage capacity constraints, and payload usage constraints. S4, determine whether there is a valid candidate joint action identified by the action mask in the candidate joint action space; if there is, select the target joint action in the candidate joint action space based on the action mask and the joint action probability; if there is no, terminate the iteration and output the current satellite scheduling scheme. S5, decode the target joint action and insert it into the timeline sequence of the scheduled tasks of the target satellite to obtain a new satellite scheduling scheme, update the satellite scheduling status according to the target joint action, and return to step S3.

[0014] Thirdly, this application also provides a computer-readable storage medium storing a computer program for scheduling heterogeneous payloads of Earth observation satellites based on deep reinforcement learning, wherein the computer program causes a computer to execute the heterogeneous payload scheduling method for Earth observation satellites based on deep reinforcement learning as described above.

[0015] Fourthly, this application also provides an electronic device, comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for executing the deep reinforcement learning-based heterogeneous payload scheduling method for Earth observation satellites as described above.

[0016] (III) Beneficial Effects This invention provides a method for scheduling heterogeneous payloads for Earth observation satellites based on deep reinforcement learning. Compared with existing technologies, it has the following advantages: 1. This application constructs a candidate joint action space based on satellite parameters, mission parameters, and visible time window information. Each candidate joint action consists of the mission number to be observed, payload type, satellite number, and visible time window number, enabling joint decision-making for mission selection, payload selection, satellite selection, and time window selection within the same probability distribution. This avoids the propagation of errors from previous layers to subsequent layers and improves the decision-making accuracy of heterogeneous payload collaborative scheduling.

[0017] 2. The policy network of this application encodes the task state, satellite state, and time window state through a task encoder, a satellite encoder, and a time window encoder, respectively. A hierarchical cross-attention aggregation module then progressively merges the numerous time window embeddings into an aggregated embedding sequence commensurate with the task and satellite scales. Finally, a conditional layer-by-layer scoring network fuses task-level, satellite-level, and window-level information step-by-step using the global context vector as a condition. Compared to a high-dimensional one-time scoring method for all candidate joint actions, this application reduces the computational complexity of scoring while maintaining scoring expressiveness, eliminating the memory bottleneck in high-dimensional joint scoring. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating the steps of the heterogeneous payload scheduling method for Earth observation satellites provided in the embodiments of this application; Figure 2 A schematic diagram illustrating the processing flow of the policy network provided in this application embodiment; Figure 3 A schematic diagram of the training gain as a function of the number of training rounds, provided for an embodiment of this application; Figure 4 This is a schematic diagram showing the change in the number of completed tasks with the number of training rounds, provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This application provides a method for scheduling heterogeneous payloads for Earth observation satellites based on deep reinforcement learning. This method solves the problem that existing methods do not consider the interaction between multiple types of payloads and cannot handle the collaborative scheduling of heterogeneous payloads. It realizes the collaborative scheduling and reasonable task scheduling of optical payloads and synthetic aperture radar payloads under multiple constraints such as visible time window, energy limit, storage limit, attitude transition time, dual payload sequence constraint, and non-repeating observation, thereby maximizing the total weighted benefit of completing the observation task.

[0022] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows: Optical payloads and Synthetic Aperture Radar (SAR) payloads are the two most common imaging payloads on EOS. Their imaging mechanisms and image effects differ significantly, and their application scenarios also differ. Optical payloads can acquire intuitive, high-resolution imagery, but are easily limited by clouds, fog, and day / night conditions. SAR payloads possess all-weather observation capabilities and can acquire 3D target positioning information, but image interpretation is more challenging. In practical applications, the two types of payloads need to work together to achieve complementary performance. Taking topographic mapping as an example, relying solely on optical imagery can lead to low positioning accuracy, ineffective imaging due to terrain obstruction and limited viewing angles. SAR payloads can compensate for optical blind spots, provide all-weather 3D positioning information, and assist optical imagery in high-precision geometric orientation. Together, they can achieve rapid and accurate positioning of satellite imagery.

[0023] In terms of multi-load scheduling, although there are methods such as genetic algorithms and multi-objective evolutionary algorithms based on deep reinforcement learning that attempt to extend the problem to multi-load scenarios, these methods still treat different loads as independent scheduling objects and do not model the core collaborative constraint of multiple loads being observed sequentially. The temporal coupling relationship between preceding and subsequent loads is ignored, so the global optimality of heterogeneous load collaborative observation cannot be guaranteed.

[0024] Therefore, how to model the coordinated scheduling of optical and SAR payloads under multiple constraints such as visible time window, energy limit, storage limit, attitude transition time, dual payload sequence constraint, and non-repeating observation, and how to automatically learn the optimal scheduling strategy through deep reinforcement learning, has become an urgent problem to be solved.

[0025] This application aims to provide a method for scheduling heterogeneous payloads of Earth observation satellites based on deep reinforcement learning. Under multiple constraints such as visible time window, energy limit, storage limit, attitude transition time, dual payload sequence constraint, and non-repeating observation, the method can achieve collaborative scheduling of heterogeneous payloads and reasonable task scheduling through deep reinforcement learning, thereby maximizing the total weighted benefit of completing the observation tasks.

[0026] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0027] like Figure 1 As shown in the figure, this application provides a method for scheduling heterogeneous payloads of Earth observation satellites based on deep reinforcement learning, including: S1, obtain satellite parameters, mission parameters and visible time window information based on the timeline sequence of scheduled missions of the target satellite; S2, Based on the satellite parameters, mission parameters, and visible time window information, construct the satellite scheduling state and candidate joint action space; the satellite scheduling state includes satellite state, mission state, and time window state; each candidate joint action in the candidate joint action space includes the mission number to be observed, payload type, satellite number, and visible time window number; S3, Generate an action mask based on the current satellite scheduling state and preset constraints; input the satellite scheduling state into a trained policy network to obtain joint action probabilities; the policy network includes a task encoder, a satellite encoder, a time window encoder, a hierarchical cross-attention aggregation module, a global context encoder, and a conditional layer-by-layer scoring network; the task encoder processes the task state to obtain a task embedding sequence, the satellite encoder processes the satellite state to obtain a satellite embedding sequence, and the time window encoder processes the time window state to obtain a time window embedding sequence; the hierarchical cross-attention aggregation module processes the task embedding sequence, the satellite embedding sequence, and the time window embedding sequence to obtain an aggregated embedding sequence; the global context encoder processes the aggregated embedding sequence to obtain a global context vector; the conditional layer-by-layer scoring network processes the global context vector, the task embedding sequence, the satellite embedding sequence, and the time window embedding sequence to obtain joint action probabilities; S4, determine whether there is a valid candidate joint action identified by the action mask in the candidate joint action space; if there is, select the target joint action in the candidate joint action space based on the action mask and the joint action probability; if there is no, terminate the iteration and output the current satellite scheduling scheme; wherein, the candidate joint actions identified as invalid in the action mask do not participate in the selection. S5, decode the target joint action and insert it into the timeline sequence of the scheduled tasks of the target satellite to obtain a new satellite scheduling scheme, update the satellite scheduling status according to the target joint action, and return to step S3.

[0028] The heterogeneous payload scheduling method for Earth observation satellites in this embodiment constructs a candidate joint action space based on satellite parameters, mission parameters, and visible time window information. Each candidate joint action is composed of the mission number to be observed, payload type, satellite number, and visible time window number, enabling joint decision-making for mission selection, payload selection, satellite selection, and time window selection within the same probability distribution. This avoids the propagation of errors from previous layers to subsequent layers and improves the decision-making accuracy of heterogeneous payload collaborative scheduling.

[0029] In one embodiment, step S1 involves obtaining satellite parameters, mission parameters, and visible time window information based on the timeline sequence of the scheduled missions of the target satellite.

[0030] Understandably, the scheduled task timeline sequence is used to record the observation tasks already scheduled on the target satellite and their time arrangement, serving as the basis for subsequently determining whether new observation tasks can be inserted and their insertion positions. Specifically, satellite parameters include the payload type, energy limit, storage capacity limit, and attitude transition time of each satellite; task parameters include the observation value of each ground target to be observed, the energy consumption, storage consumption, and observation duration corresponding to each payload, as well as the observation order constraints of the dual payloads; the visible time window information includes the set of observable time intervals for each ground target by each satellite, and the start and end times of each observable time interval. In this embodiment, the above parameters can be obtained through real measurement data or through simulation software.

[0031] In one embodiment, step S2 involves constructing a satellite scheduling state and a candidate joint action space based on the satellite parameters, mission parameters, and visible time window information.

[0032] In this embodiment, the satellite scheduling status includes satellite status, mission status, and time window status. The satellite status records the remaining energy, remaining storage capacity, attitude transition time, and number of scheduled missions for each satellite. The mission status records the observation value, dual-payload energy consumption and storage consumption, observation duration, dual-payload observation order constraints, and execution completion status of each payload for each mission. The time window status records the time information and execution selection status of each satellite for each mission within each visible time window.

[0033] It should be clarified that the candidate joint action space is jointly defined by the mission status, satellite status, and time window status in the satellite scheduling state. Specifically, each candidate joint action consists of a quadruple consisting of the mission number to be observed, payload type, satellite number, and visible time window number, enabling joint decision-making of mission selection, payload type selection, satellite number selection, and time window number selection within the same probability distribution. The total dimension of the candidate joint action space is the product of the number of missions to be observed, the total number of satellites, and the maximum visible time window number per mission. The total number of satellites is the sum of the number of optical satellites and the number of synthetic aperture radar satellites.

[0034] For ease of description, the quadruple is referred to as follows in this embodiment. ;in Number the task to be observed. For load type, Number the satellite. Number the visible time window.

[0035] In this embodiment, the number of tasks to be observed is denoted as... The number of optical satellites is denoted as The number of synthetic aperture radar satellites is denoted as The maximum number of visible time windows per task is denoted as The candidate joint action space is denoted as Then the total dimension of the candidate joint action space satisfies .

[0036] This embodiment merges the selection of mission number, payload type, satellite number, and time window number into a unified joint action space, enabling the above four selections to be jointly decided within the same probability distribution. This avoids the problem of errors from previous layer decisions being passed on to subsequent layers and accumulating layer by layer, which is common in hierarchical decision-making methods.

[0037] During encoding, the quadruple is mapped to a one-dimensional action index as follows: the segment start offset corresponding to the task is obtained by multiplying the task number by the total number of satellites and then by the maximum visible time window per task; the satellite number and the time window number are combined into a local offset, wherein the local offset corresponding to optical satellites is located in the segment from zero to the number of optical satellites multiplied by the maximum visible time window per task, and the local offset corresponding to synthetic aperture radar satellites is located in the segment from the number of optical satellites multiplied by the maximum visible time window per task to the total number of satellites multiplied by the maximum visible time window per task; the action index is the sum of the segment start offset and the local offset.

[0038] Specifically, the encoding of a quadruple can be represented as: in, For action indexing, Number the task to be observed. This is the product of the total number of satellites and the maximum visible time window for each mission. This is the local offset after merging the satellite number and the visible time window number.

[0039] In one embodiment, step S3 involves generating an action mask based on the current satellite scheduling state and preset constraints; processing the satellite scheduling state through a pre-trained policy network to obtain joint action probabilities; the policy network includes a task encoder, a satellite encoder, a time window encoder, a hierarchical cross-attention aggregation module, a global context encoder, and a conditional layer-by-layer scoring network.

[0040] In this embodiment, the constraints include visible time window constraints, mission priority constraints, satellite energy constraints, storage capacity constraints, and payload usage constraints; the specific meanings and formulas of each constraint are explained below.

[0041] It should be noted that, for ease of expression, the following text will use the term "foreign language". , , All of these represent satellite numbers, among which The satellite designation for carrying a synthetic aperture radar payload. ; The satellite number is used to designate the satellite carrying the optical payload. That is, satellite collection Center front One is a satellite carrying a synthetic aperture radar payload, and the next... Each satellite carries an optical payload; the total number of satellites is [number missing]. ; The number of ground targets to be observed. Number the ground target.

[0042] Specifically, the visible time window constraint is used to limit satellite observations of ground targets to be conducted within the visible time window during which the satellite can observe the ground target. That is, the observation start time is no earlier than the start time of the selected visible time window, and the sum of the observation start time and the observation duration does not exceed the end time of the selected visible time window. The expression for the visible time window constraint is: In the formula, Set the start time of the selected visible time window; The end time of the selected visible time window; This is the start time of the observation; For the duration of the observation; The set of ground targets to be observed. The number of ground targets to be observed. Number the target; It is a collection of Earth Observation Satellites (EOS), among which the first... One satellite carrying a SAR payload, later One carrying an optical payload, Number the satellite; This represents the set of visible time windows for observing ground targets from satellites. This represents the maximum number of time windows during which a satellite can be visible to ground targets. Number the visible time window.

[0043] The task priority constraint is used to limit the observation of ground targets to their observable time range, and to reserve attitude transition time when the same satellite performs sequential observations of multiple ground targets. Specifically, the observation start time is no earlier than the earliest observable time of the ground target, the observation end time is no later than the latest observable time of the ground target, and the end time of the preceding observation, after attitude transition time extension, does not exceed the start time of the subsequent observation. The expression for the task priority constraint is: In the formula, The earliest observable time for ground targets; This is the latest observable time for ground targets; For the same satellite targeting ground targets and Indicator of whether sequential observations are required. To distinguish from Another ground target number, if A value of 1 indicates that sequential observations are performed; otherwise, a value of 0 indicates that observations are performed. For satellite ground targets and The attitude transition time required for preceding and subsequent observations. It is a pre-defined, very large positive integer; This represents the total number of satellites.

[0044] The satellite energy constraint is used to limit the total energy consumed by each satellite in observing various ground targets to not exceed the satellite's energy limit; the expression for the satellite energy constraint is: In the formula, Energy consumed for observation; For the first The energy limit of a satellite; The storage capacity constraint is used to limit the total storage space consumed by each satellite in observing various ground targets to not exceed the upper limit of the satellite's storage space; the formula for the storage capacity constraint is: In the formula, Storage space consumed for observation, For the first The maximum storage space limit for each satellite; The payload usage constraint is used to limit the ground target to be observed sequentially by the two types of payloads in a prescribed observation order. The ground target is either observed by both types of payloads or not observed by either, and the same ground target is observed at most once by the synthetic aperture radar payload and at most once by the optical payload. The expression for the payload usage constraint is: In the formula, For ground targets The load observation sequence identifier, if A value of 1 indicates that the synthetic aperture radar (SAR) payload observes before the optical payload, while a value of -1 indicates that the optical payload observes before the SAR payload. In the formula, To represent the decision variable for whether a satellite observes a ground target within the corresponding visible time window, a value of 1 indicates observation, and a value of 0 indicates no observation. , , These represent the satellite number, ground target number, and visible time window number, respectively.

[0045] The satellite designation for carrying a synthetic aperture radar payload. ; The satellite number is used to designate the satellite carrying the optical payload. That is, the first in the satellite set One is a satellite carrying a synthetic aperture radar payload, and the next... Each of these is a satellite carrying optical payloads.

[0046] In this embodiment, the policy network includes: The task encoder is used to encode the raw features of each task into a task embedding through normalization and a feedforward network. The raw features of each task include 14 dimensions of information, including task value, energy and storage consumption of dual payloads, observation duration, observation order constraints, and current execution state.

[0047] Specifically, the task The original features are denoted as The task embedding after normalization and feedforward network encoding is as follows: = in Indicates normalization, This represents the feedforward network of the task encoder. and This is the weight matrix. and For bias vectors, It is a linear rectified function; the above task embeddings of all tasks constitute a task embedding sequence: In the formula, This represents the number of tasks to be observed.

[0048] The satellite encoder is used to encode the raw features of each optical satellite and each synthetic aperture radar satellite into satellite embeddings through normalization and feedforward networks. The raw features of each satellite include four-dimensional information: remaining energy, remaining storage capacity, attitude transition time, and the number of scheduled tasks.

[0049] Specifically, optical satellites The original features are denoted as Synthetic Aperture Radar Satellite The original features are denoted as The satellite embeddings after normalization and feedforward network encoding are respectively and The satellite embeddings of the two types of satellites are stitched together along the satellite dimensions to form a satellite embedding sequence: in, For the number of optical satellites, This represents the number of synthetic aperture radar satellites.

[0050] The time window encoder is used to normalize and encode the original features of each candidate time window into a time window embedding through a feedforward network. The original features of each candidate time window include six dimensions of information: window start time, window end time, duration, whether it has been selected for execution, actual execution start time, and actual execution end time.

[0051] Specifically, the original features of the candidate time windows are denoted as... Its time window embedding after normalization and feedforward network encoding is The above time window embeddings of all candidate time windows constitute a time window embedding sequence. .

[0052] The hierarchical cross-attention aggregation module is used to first aggregate multiple time windows of the same mission-satellite pair into a mission-window aggregated embedding by window-to-mission-level cross-attention, and then aggregate multiple mission-window aggregated embeddings corresponding to the same satellite into a satellite-window aggregated embedding by mission-to-satellite-level cross-attention.

[0053] Specifically, window-to-task level cross-attention uses learnable query vectors. To query, using the same satellite and mission pair corresponding to Each time window is embedded as a key and value. A residual connection is made between the multi-head attention output and the learnable query vector, and then a layer of normalization is performed to obtain the task-window aggregation embedding. Subsequently, the mission employs satellite-level cross-attention with learnable query vectors. To obtain the satellite-window aggregation embedding, which uses the task-window aggregation embedding corresponding to the same satellite as the key and value, a residual connection is made between the multi-head attention output and the learnable query vector, followed by layer normalization. ,in and For multi-headed cross-attention, The satellites are numbered. The satellite-window aggregation embeddings of all satellites constitute a satellite-window aggregation embedding sequence. .

[0054] In one embodiment, the task embedding, satellite embedding, time window embedding, and each aggregate embedding have the same dimension, denoted as the embedding dimension. The number of heads in the multi-head cross-attention is denoted as the attention head number, and the dimension of each attention head is equal to the embedding dimension divided by the attention head number. In a specific example, the embedding dimension is 128, the attention head number is 8, and correspondingly, the dimension of each attention head is 16. Both the window-to-task level cross-attention and the task-to-satellite level cross-attention adopt a query-key-value format, and the dimensions of the query, key, and value are all equal to the embedding dimension. To avoid information loss during the hierarchical aggregation process and to improve training stability, both the window-to-task level cross-attention and the task-to-satellite level cross-attention set a residual connection between the output of the multi-head attention and the learnable query vector, and the result of the residual connection is layer-normalized, as shown in the formula above.

[0055] It is important to clarify that, since the number of visible time windows between different tasks and satellites may vary, and not every task and every satellite has a visible time window, this embodiment aligns the set of visible time windows corresponding to the same task-satellite pair according to the maximum number of visible time windows per task. For positions with fewer than this number, placeholder embeddings are used, and a negative infinity attention mask is applied to the filled positions during cross-attention calculation, ensuring that the learnable query vector interacts only with the actual visible time window embeddings. For task-satellite pairs without visible time windows, their task-window aggregation embedding is set to zero and skipped when constructing the satellite-window aggregation embedding sequence. Similarly, the number of tasks associated with different satellites may also vary, and the task-to-satellite level cross-attention is handled using the same alignment and masking method as described above. Therefore, the hierarchical cross-attention aggregation module can adapt to situations where the number of tasks to be observed is inconsistent with the total number of satellites, and the number of visible time windows corresponding to each task-satellite pair is unequal.

[0056] Specifically, the dimensions of the task embedding sequence, satellite embedding sequence, time window embedding sequence, task-window aggregation embedding sequence, and satellite-window aggregation embedding sequence are respectively: number of tasks to be observed × embedding dimension, total number of satellites × embedding dimension, number of tasks to be observed × total number of satellites × maximum visible time window per task × embedding dimension, number of tasks to be observed × total number of satellites × embedding dimension, and total number of satellites × embedding dimension. That is, the hierarchical cross-attention aggregation module, through two-level aggregation from window to task and from task to satellite, gradually merges the time window embeddings of the scale of number of tasks to be observed × total number of satellites × maximum visible time window per task into the satellite-window aggregation embedding sequence of the scale of the total number of satellites. This reduces the computational complexity of the subsequent scoring network while preserving the hierarchical dependency relationship between satellite, task, and time window.

[0057] The global context encoder is used to perform global attention computation after concatenating the task embedding, satellite embedding, and satellite-window aggregation embedding along the sequence dimension, and outputs a global context vector.

[0058] Specifically, the global context encoder uses a learnable global query vector. For querying, multi-head attention is performed on the concatenated sequence obtained by concatenating the task embedding sequence, satellite embedding sequence, and satellite-window aggregated embedding along the sequence dimension, and the global context vector is output after layer normalization and linear projection: in, For the attention of the bulls, For layer normalization, It is a linear projection.

[0059] The conditional layer-by-layer scoring network is used to generate a scalar score for each candidate joint action by fusing task-level, satellite-level, and window-level information sequentially through a three-level hierarchical multilayer perceptron, using the global context vector as a condition.

[0060] It should be clarified that the hierarchical cross-attention aggregation module adopts a two-level aggregation of window to task and task to satellite. Its function is to merge the large number of candidate time windows into an embedding sequence that is comparable to the scale of the task and the scale of the satellite, thereby reducing the computational complexity of the subsequent scoring network while preserving the hierarchical dependency relationship between satellite, task and time window.

[0061] In this embodiment, the step of processing the satellite scheduling state through a pre-trained policy network to obtain joint action probabilities includes: S31, the task state is processed by the task encoder to obtain a task embedding sequence, the satellite state is processed by the satellite encoder to obtain a satellite embedding sequence, and the time window state is processed by the time window encoder to obtain a time window embedding sequence; the task embedding sequence includes a task embedding vector. S32, the hierarchical cross-attention aggregation module processes the task embedding sequence, satellite embedding sequence, and time window embedding sequence to obtain an aggregated embedding sequence; S33, The aggregated embedding sequence is processed by the global context encoder to obtain a global context vector; S34, the global context vector, task embedding sequence, satellite embedding sequence and time window embedding sequence are processed by the conditional layer-by-layer scoring network to obtain the joint action probability; wherein, the conditional layer-by-layer scoring network includes an action scalar scoring branch and a task implicit scoring branch.

[0062] The process of processing the global context vector, task embedding sequence, satellite embedding sequence, and time window embedding sequence through a conditional layer-by-layer scoring network to obtain the joint action probability includes: S341, the task embedding sequence, global context vector, satellite embedding sequence, and time window embedding sequence are processed through the action scalar scoring branch to obtain a joint action scalar score; the global context vector is processed through the task implicit scoring branch, and the processing result is multiplied by the task embedding sequence to obtain a task implicit score; wherein, the action scalar scoring branch includes a three-level multilayer perceptron, the first-level multilayer perceptron processes the task embedding sequence and global context vector to obtain a first fused sequence; the second-level multilayer perceptron processes the first fused sequence and satellite embedding sequence to obtain a second fused sequence; the third-level multilayer perceptron processes the second fused sequence and time window embedding sequence to obtain a joint action scalar score; S342, add the implicit task score to the joint action scalar score to obtain the joint action probability.

[0063] Specifically, using the global context vector As a condition, the three-level multilayer perceptron is as follows: Each level of the multilayer perceptron consists of a two-layer feedforward network; the network for the task implicit scoring branch is denoted as... The task's implicit rating is The combined action scalar score is The sum of the two yields the probability of the joint action: Joint actions that do not meet the constraints The mask is This ensures that sampling is only performed within the available space.

[0064] See Figure 2 , Figure 2The processing flow of the policy network provided in this application embodiment is illustrated. First, the task features, satellite features, and time window features are encoded by the task encoder, satellite encoder, and time window encoder, respectively, to obtain the corresponding task embedding sequence, satellite embedding sequence, and time window embedding sequence; wherein, the internal structure of the task encoder, satellite encoder, and time window encoder is a normalization layer (Norm) followed by a linear layer (Linear), a rectified linear unit (ReLU), and another linear layer (Linear) in sequence, i.e., Norm→Linear→ReLU→Linear; The time window embedding sequence undergoes two levels of cross-attention processing: window-to-task level aggregation followed by task-to-satellite level aggregation, to obtain the satellite-window aggregated embedding. The task embedding sequence, satellite embedding sequence, and satellite-window aggregated embedding are concatenated along the sequence dimension and input into the global context encoder to obtain the global context vector. The global context vector is then processed sequentially through context-task fusion, task-satellite fusion, and joint action scoring to output a three-level multilayer perceptron. The task embedding vector, satellite embedding vector, and time window embedding vector are fused level by level to output the joint action score for each candidate joint action. The joint action score is normalized to obtain the joint action probability. After setting the probability of invalid candidate joint actions to zero according to the action mask, the target joint action is output.

[0065] In one embodiment, step S4 involves determining whether there exists a candidate joint action that is identified as valid by the action mask in the candidate joint action space; if it exists, a target joint action is selected in the candidate joint action space based on the action mask and the joint action probability; if it does not exist, the iteration is terminated and the current satellite scheduling scheme is output.

[0066] Understandably, the action mask is used to identify whether each candidate joint action in the candidate joint action space meets preset constraints under the current satellite scheduling state. For any candidate joint action, if its corresponding observation task, payload type, satellite, and time window all meet the constraints in terms of visible time window, satellite energy, storage capacity, payload usage, and task priority, it is marked as valid in the action mask; otherwise, it is marked as invalid. Candidate joint actions marked as invalid in the action mask do not participate in the selection.

[0067] The time complexity of generating the action mask once is ,in The total dimension of the candidate joint action space is the product of the number of tasks to be observed, the total number of satellites, and the maximum visible time window for each task. The action mask corresponds to the satellite scheduling state and can be cached; only when the satellite scheduling state changes is the mask component corresponding to the changed satellite and task recalculated, and the remaining mask components reuse the cached results from the previous round; when entering a new scheduling round, the action mask is invalidated as a whole and regenerated.

[0068] The action mask is a one-dimensional vector with the same total dimension as the candidate joint action space. It is generated by performing constraint verification on each candidate joint action in the candidate joint action space. If the candidate joint action satisfies all constraints simultaneously, the value in the action mask is 1; otherwise, it is 0. In one embodiment, the constraint verification is performed sequentially in the order of visible time window constraints, task priority constraints, payload usage constraints, satellite energy constraints, and storage capacity constraints. If any constraint is not satisfied, the candidate joint action is set to 0 in the action mask, and no further constraint verification is performed, thereby reducing verification overhead while ensuring the correctness of the judgment result.

[0069] The verification criteria for the visible time window constraint are as follows: whether the satellite has a visible time window that covers the selected visible time window for the mission; whether the mission priority constraint is verified after insertion and still meets the earliest observable time, latest observable time, and attitude transition time required for preceding and following observations; whether the payload usage constraint is verified based on the execution completion status of the two types of payloads in the mission status and the payload observation sequence identifier; and whether the satellite energy constraint and storage capacity constraint are verified based on whether the satellite's remaining energy and remaining storage capacity are not less than the energy consumption and storage consumption required for this observation.

[0070] Specifically, it is determined whether there are any valid candidate joint actions identified by the action mask in the candidate joint action space, that is, whether there are still executable candidate joint actions in the current satellite scheduling state; if there are, the target joint action is selected in the candidate joint action space based on the action mask and the joint action probability, wherein the probability corresponding to the candidate joint action identified as invalid is set to zero, so that the selection is only performed in the feasible action space; if there are no, it indicates that there are no executable candidate joint actions in the current satellite scheduling state, then the iteration is terminated and the current satellite scheduling scheme is output.

[0071] In one embodiment, step S5 involves decoding the target joint action and inserting it into the timeline sequence of the scheduled tasks of the target satellite to obtain a new satellite scheduling scheme, updating the satellite scheduling status according to the target joint action, and returning to step S3.

[0072] In this embodiment, the target joint action is first decoded to obtain the target mission, target payload, target satellite, and target time window.

[0073] The target joint action decoding in this embodiment includes: S51, divide the product of the total number of satellites and the maximum visible time window for each mission by the action index, the quotient is the mission number, and the remainder is the local offset; the action index is the representation of the target joint action in the candidate joint action space. S52, when the local offset is less than the number of optical satellites multiplied by the maximum number of visible time windows per mission, the payload type is determined to be an optical payload, and the local offset is used as the adjusted offset; when the local offset is greater than or equal to the number of optical satellites multiplied by the maximum number of visible time windows per mission, the payload type is determined to be a synthetic aperture radar payload, and the difference between the local offset and the number of optical satellites multiplied by the maximum number of visible time windows per mission is used as the adjusted offset. S53. Divide the adjusted offset by the maximum number of visible time windows for each mission. The quotient is the satellite number, and the remainder is the time window number.

[0074] Specifically, the decoding of the joint action of the target can be represented as: when At that time, it was determined to be an optical payload. and ; when At that time, it was determined to be a SAR payload. and ; , .

[0075] in Indicates divisibility. Indicates the remainder; For load type, the value is [value to be filled in]. Indicates optical payload, with a value of Indicates the synthetic aperture radar payload; This is a local offset. This is the adjusted offset. Number the satellite. Number the visible time window.

[0076] Understandably, inserting the decoded target joint action into the scheduled task timeline sequence of the target satellite means determining the insertion position of the target task within the scheduled task timeline sequence of the target satellite according to the decoded target task, target payload, target satellite, and target time window. Specifically, the scheduled task timeline sequence of the target satellite is obtained, and candidate insertion gaps are constructed at the beginning, end, and between adjacent scheduled tasks of the scheduled task timeline sequence. For each candidate insertion gap, the larger of the start time of the visible time window corresponding to the candidate insertion gap and the start time of the candidate insertion gap is taken as the candidate actual start time. It is verified whether the candidate actual end time corresponding to the candidate actual start time exceeds the end time of the visible time window, and whether it exceeds the end time of the candidate insertion gap after attitude transition time extension. Among the candidate insertion gaps that meet the above verification conditions, the earliest time is selected as the insertion position. Subsequently, the energy and storage capacity of the target satellite are deducted, the mission execution information is updated, the target mission is inserted into the timeline sequence of the scheduled missions and reordered according to the start time, the termination flag is updated and the mask cache is cleared, and the satellite scheduling status is updated according to the target joint action before returning to step S3. It should be noted that if multiple candidate insertion gaps have the same actual start time and are all the earliest, the earliest candidate insertion gap in the timeline sequence of the scheduled missions is selected as the insertion position to ensure that the insertion result is unique and the decoding process is reproducible.

[0077] If none of the candidate insertion gaps meet the above verification conditions, the target joint action is determined to be infeasible; at this time, the target joint action is marked as invalid in the action mask, and the process returns to step S4 to reselect the target joint action; during the training phase of the policy network, the current sampling trajectory is marked as the termination trajectory.

[0078] In addition, the action mask components corresponding to the target satellite and its associated missions are set to invalid so that they can be recalculated when returning to step S3, while the remaining mask components continue to be reused.

[0079] After insertion, the observation start time of the target task is set to the candidate actual start time of the selected insertion gap, and the observation end time is set to the sum of the observation start time and the observation duration. The energy consumption corresponding to the target payload is deducted from the remaining energy of the target satellite, and the storage consumption corresponding to the target payload is deducted from the remaining storage capacity of the target satellite. The execution completion status of the target payload is recorded in the task status; when both types of payloads have completed observations, the task is marked as completed and included in the revenue. The target task is then inserted into the timeline sequence of the scheduled tasks in order of observation start time, and the corresponding visible time window status is updated. This insertion method in this embodiment, by performing dual verification of the visible time window and attitude transition time for each candidate insertion gap and selecting the earliest feasible gap, ensures that the inserted task sequence does not conflict with the already scheduled tasks in the time dimension, and eliminates the need to rearrange the already scheduled tasks, thus reducing the computational overhead of the scheduling process.

[0080] In one embodiment, the training method for the policy network includes: S101, randomly sample the satellite scheduling scheme of Earth observation satellites, and iteratively update the satellite scheduling status based on the policy network, and record the complete sampling trajectory of the target joint actions; S102, calculate the cumulative benefit based on the sampling trajectory, where the cumulative benefit is the sum of the task value of all payloads in the sampling trajectory completing the observation; S103, clone the initial satellite scheduling state of the satellite scheduling scheme, and use the greedy mode of the policy network to gradually select the joint action with the highest probability at present to deduce the baseline benefit; S104, calculate the advantage value of each sampling trajectory, and normalize the advantage value within the batch; the advantage value is the cumulative gain minus the baseline gain; S105, clone the initial satellite scheduling state of the satellite scheduling scheme, and starting from the initial satellite scheduling state, replay each step of the joint action recorded along the sampling trajectory to perform state transition, and recalculate the logarithmic probability of each step of the action using the current policy network parameters. With action entropy ; S106, Calculate the loss based on the logarithmic probability, dominance value, and action entropy. ; S107, Calculate the gradient based on the backpropagation of the loss, and clip the gradient to a preset maximum value; S108, the Adam optimizer is used to update the policy network parameters based on gradients; S109, repeat the above training steps, evaluate the performance of policy networks with different policy network parameters in the validation set, and when the preset total number of training rounds is reached, save the best performing policy network as the policy network that has been trained.

[0081] Understandably, state transition refers to starting from the current satellite scheduling state, executing the joint actions recorded in the sampling trajectory at that moment, and updating the satellite's scheduled task timeline sequence, visible time window set, and energy / storage state quantities according to the satellite scheduling rules to obtain the satellite scheduling state at the next moment. Thus, starting from the cloned initial satellite scheduling state and replaying the recorded joint actions step by step along the sampling trajectory, the satellite scheduling state at each moment can be reconstructed, which is used to recalculate the logarithmic probability and action entropy of each action step with the current policy network parameters.

[0082] Specifically, the policy network is trained using a policy gradient algorithm with baseline, and its gradient is: In the formula, For sampling trajectory The cumulative benefit is the sum of the mission value of all payloads in the sampling trajectory completing the observation; The baseline return is the result of a greedy deduction under the same initial conditions; For policy networks, For policy network parameters; This represents the expectation of the sampling trajectory; For the first The selected action; For the first Step-by-step satellite scheduling status; The step number in the sampling trajectory; Let be the training objective function for the policy network.

[0083] In this embodiment, the loss for: In the formula, For batch size, For the first The dominance value of the trajectory, For the first Trajectory number A valid indicator of a step For the first Trajectory number The logarithmic probability of a step. For the first Trajectory number The motion entropy of a step, is the entropy regularization coefficient. Number the trajectory; Number the steps in the trajectory; For the selected combined action, The satellite is in a scheduling state.

[0084] In this embodiment, the objective function of the policy network is: In the formula, The number of ground targets to be observed; Number the ground targets; Benefits from ground target observation; The number of satellites carrying synthetic aperture radar payloads, The satellite designation for the satellite carrying the synthetic aperture radar payload; The number of satellites carrying optical payloads. The satellite number is used to designate the satellite carrying optical payloads. This represents the maximum number of time windows during which a satellite can be visible to ground targets. Number the visible time window; The decision variable represents whether the satellite observes the ground target within the corresponding visible time window. A value of 1 indicates that observation is performed, and a value of 0 indicates that observation is not performed.

[0085] The objective function in this embodiment aims to maximize the gains of the mission observation sequence, and ground targets will only generate gains after being observed by both SAR and visible light payloads.

[0086] See Figure 3 , Figure 3 This is a schematic diagram illustrating the change in training reward with the number of training epochs, as provided in an embodiment of this application. The horizontal axis represents the number of training epochs, and the vertical axis represents the training reward. As shown in the diagram, the training reward increases rapidly with the number of training epochs in the early stages of training, indicating that under the policy gradient training method with baseline, the policy network can quickly learn scheduling policies that result in higher total weighted rewards for completing the observation task. When the number of training epochs continues to increase, the training reward tends to stabilize and fluctuates only within a small range, indicating that the parameters of the policy network have converged and the training process is stable.

[0087] See Figure 4 , Figure 4 This is a schematic diagram illustrating the change in the number of completed tasks with the number of training rounds, as provided in an embodiment of this application. The horizontal axis represents the number of training rounds, and the vertical axis represents the number of completed tasks. As shown in the figure, the trend of the number of completed tasks is consistent with the training gains, rising rapidly in the early stages of training, then stabilizing, and finally settling at a relatively high level. This indicates that the policy network gradually learns to schedule two types of heterogeneous payloads under multiple constraints during training, such as the visible time window, energy limit, storage limit, attitude transition time, and payload usage order, enabling more observation tasks to be completed while satisfying the constraints.

[0088] This invention also provides a heterogeneous payload scheduling system for Earth observation satellites based on deep reinforcement learning, comprising: The acquisition module is used to acquire satellite parameters, mission parameters, and visible time window information based on the timeline sequence of scheduled missions of the target satellite; The scheduling status and action space construction module is used to construct the satellite scheduling status and candidate joint action space based on the satellite parameters, mission parameters and visible time window information; The iterative loop module is used to execute iterative steps S3-S5 to output a satellite scheduling scheme, including: S3, Generate an action mask based on the current satellite scheduling state and preset constraints; Process the satellite scheduling state through a pre-trained policy network to obtain joint action probabilities; The policy network includes a task encoder, a satellite encoder, a time window encoder, a hierarchical cross-attention aggregation module, a global context encoder, and a conditional layer-by-layer scoring network; S4, determine whether there is a valid candidate joint action identified by the action mask in the candidate joint action space; if there is, select the target joint action in the candidate joint action space based on the action mask and the joint action probability; if there is no, terminate the iteration and output the current satellite scheduling scheme. S5, decode the target joint action and insert it into the timeline sequence of the scheduled tasks of the target satellite to obtain a new satellite scheduling scheme, update the satellite scheduling status according to the target joint action, and return to step S3.

[0089] It is understood that the heterogeneous payload scheduling system for Earth observation satellites based on deep reinforcement learning provided in this embodiment of the invention corresponds to the heterogeneous payload scheduling method for Earth observation satellites based on deep reinforcement learning described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the heterogeneous payload scheduling method for Earth observation satellites based on deep reinforcement learning, and will not be repeated here.

[0090] This invention also provides a computer-readable storage medium storing a computer program for a deep reinforcement learning-based method for scheduling heterogeneous payloads of Earth observation satellites, wherein the computer program causes a computer to execute the deep reinforcement learning-based method for scheduling heterogeneous payloads of Earth observation satellites as described above.

[0091] This application also provides an electronic device, including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the deep reinforcement learning-based heterogeneous payload scheduling method for Earth observation satellites as described above.

[0092] In summary, compared with existing technologies, it has the following beneficial effects: 1. This application constructs a candidate joint action space based on satellite parameters, mission parameters, and visible time window information. Each candidate joint action consists of the mission number to be observed, payload type, satellite number, and visible time window number, enabling joint decision-making for mission selection, payload selection, satellite selection, and time window selection within the same probability distribution. This avoids the propagation of errors from previous layers to subsequent layers and improves the decision-making accuracy of heterogeneous payload collaborative scheduling.

[0093] 2. The policy network of this application encodes the task state, satellite state, and time window state through a task encoder, a satellite encoder, and a time window encoder, respectively. A hierarchical cross-attention aggregation module then progressively merges the numerous time window embeddings into an aggregated embedding sequence commensurate with the task and satellite scales. Finally, a conditional layer-by-layer scoring network fuses task-level, satellite-level, and window-level information step-by-step using the global context vector as a condition. Compared to a high-dimensional one-time scoring method for all candidate joint actions, this application reduces the computational complexity of scoring while maintaining scoring expressiveness, eliminating the memory bottleneck in high-dimensional joint scoring.

[0094] 3. This application integrates visible time window constraints, satellite energy constraints, storage capacity constraints, payload usage constraints, and mission priority constraints into unified constraint conditions, and generates action masks based on these constraints. Among these, payload usage constraints and mission priority constraints jointly characterize the temporal coupling relationship where two types of heterogeneous payloads must be observed sequentially according to a prescribed order, with reserved attitude transition time between preceding and succeeding observations. Compared to scheduling different payloads as independent objects, this application can accurately characterize the cross-payload constraints of the completion time of preceding payloads on the feasible scheduling interval of subsequent payloads, thus ensuring the feasibility of the output scheduling scheme under multiple constraint conditions. This solves the problems of existing methods that do not consider the interactive effects between multiple types of payloads and cannot handle the collaborative scheduling of heterogeneous payloads.

[0095] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0096] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for scheduling heterogeneous payloads of Earth observation satellites based on deep reinforcement learning, characterized in that, include: S1, obtain satellite parameters, mission parameters and visible time window information based on the timeline sequence of scheduled missions of the target satellite; S2, Based on the satellite parameters, mission parameters, and visible time window information, construct the satellite scheduling state and candidate joint action space; the satellite scheduling state includes satellite state, mission state, and time window state; each candidate joint action in the candidate joint action space includes the mission number to be observed, payload type, satellite number, and visible time window number; S3, Generate an action mask based on the current satellite scheduling status and preset constraints; Process the satellite scheduling status through a pre-trained policy network to obtain joint action probabilities; The policy network includes a task encoder, a satellite encoder, a time window encoder, a hierarchical cross-attention aggregation module, a global context encoder, and a conditional layer-by-layer scoring network; The constraints include visible time window constraints, task priority constraints, satellite energy constraints, storage capacity constraints, and payload usage constraints. S4, determine whether there is a valid candidate joint action identified by the action mask in the candidate joint action space; if there is, select the target joint action in the candidate joint action space based on the action mask and the joint action probability; if there is no, terminate the iteration and output the current satellite scheduling scheme. S5, decode the target joint action and insert it into the timeline sequence of the scheduled tasks of the target satellite to obtain a new satellite scheduling scheme, update the satellite scheduling status according to the target joint action, and return to step S3.

2. The method for scheduling heterogeneous payloads of Earth observation satellites according to claim 1, characterized in that, The process of processing the satellite scheduling state through a pre-trained policy network to obtain joint action probabilities includes: The task state is processed by the task encoder of the policy network to obtain the task embedding sequence, the satellite state is processed by the satellite encoder of the policy network to obtain the satellite embedding sequence, and the time window state is processed by the time window encoder of the policy network to obtain the time window embedding sequence. The hierarchical cross-attention aggregation module of the policy network processes the task embedding sequence, satellite embedding sequence, and time window embedding sequence to obtain an aggregated embedding sequence; The aggregated embedding sequence is processed by the global context encoder of the policy network to obtain a global context vector; The global context vector, task embedding sequence, satellite embedding sequence, and time window embedding sequence are processed by the conditional layer-by-layer scoring network of the policy network to obtain joint action probabilities; wherein, the conditional layer-by-layer scoring network includes an action scalar scoring branch and a task implicit scoring branch.

3. The method for scheduling heterogeneous payloads of Earth observation satellites according to claim 2, characterized in that, The process of processing the global context vector, task embedding sequence, satellite embedding sequence, and time window embedding sequence through the conditional layer-by-layer scoring network of the policy network to obtain the joint action probability includes: The action scalar scoring branch processes the task embedding sequence, global context vector, satellite embedding sequence, and time window embedding sequence to obtain a joint action scalar score; the task implicit scoring branch processes the global context vector and multiplies the processing result with the task embedding sequence to obtain a task implicit score; wherein, the action scalar scoring branch includes a three-level multilayer perceptron. The implicit task score is added to the scalar score of the joint action to obtain the joint action probability.

4. The method for scheduling heterogeneous payloads of Earth observation satellites according to claim 1, characterized in that, The training method for the policy network includes: The satellite scheduling scheme of randomly sampled Earth observation satellites is used, and the satellite scheduling status is iteratively updated based on the policy network to record the complete sampling trajectory of the target joint actions; The cumulative revenue is calculated based on the sampling trajectory, where the cumulative revenue is the sum of the task value of all payloads in the sampling trajectory completing the observation; The initial satellite scheduling state of the satellite scheduling scheme is cloned, and the joint action with the highest probability is selected step by step in the greedy mode of the policy network to obtain the baseline benefit. Calculate the advantage value for each sampling trajectory and normalize the advantage value within a batch; the advantage value is the cumulative gain minus the baseline gain. The initial satellite scheduling state of the satellite scheduling scheme is cloned, and each step of the joint action recorded along the sampling trajectory is replayed step by step from the initial satellite scheduling state to perform state transition. The log probability and action entropy of each step of the action are recalculated with the current policy network parameters. The loss is calculated based on the logarithmic probability, action entropy, and dominance value. The gradient is calculated based on the backpropagation of the loss, and the gradient is clipped to a preset maximum value; The Adam optimizer is used to update the policy network parameters based on gradients. Repeat the above training steps, evaluate the performance of the policy network with different policy network parameters in the validation set, and save the best-performing policy network as the completed policy network when the preset total number of training rounds is reached.

5. The method for scheduling heterogeneous payloads of Earth observation satellites according to claim 1, characterized in that, The target joint action decoding includes: The action index is divided by the product of the total number of satellites and the maximum visible time window for each mission. The quotient is the mission number, and the remainder is the local offset. The action index is the representation of the target joint action in the candidate joint action space. When the local offset is less than the number of optical satellites multiplied by the maximum number of visible time windows per mission, the payload type is determined to be an optical payload, and the local offset is used as the adjusted offset; when the local offset is greater than or equal to the number of optical satellites multiplied by the maximum number of visible time windows per mission, the payload type is determined to be a synthetic aperture radar payload, and the difference between the local offset and the number of optical satellites multiplied by the maximum number of visible time windows per mission is used as the adjusted offset. Divide the adjusted offset by the maximum number of visible time windows for each mission. The quotient is the satellite number, and the remainder is the time window number.

6. The method for scheduling heterogeneous payloads of Earth observation satellites according to claim 1, characterized in that, The objective function of the policy network is: In the formula, The number of ground targets to be observed. Number the ground targets; The benefits of completing observations of ground targets; The number of satellites carrying synthetic aperture radar payloads, The satellite designation for the satellite carrying the synthetic aperture radar payload; The number of satellites carrying optical payloads. The satellite number is used to designate the satellite carrying optical payloads. This represents the maximum number of time windows during which a satellite can be visible to ground targets. Number the visible time window; This represents the decision variable indicating whether a satellite observes a ground target within the corresponding visible time window.

7. The method for scheduling heterogeneous payloads of Earth observation satellites according to claim 6, characterized in that, The expression for the visible time window constraint is: In the formula, Set the start time of the selected visible time window; The end time of the selected visible time window; This is the start time of the observation; For the duration of the observation; The set of ground targets to be observed. Number the target; For the collection of Earth observation satellites, Number the satellite; This represents the set of visible time windows for observing ground targets from satellites. This represents the maximum number of time windows during which a satellite can be visible to ground targets. Number the visible time window; The expression for the task priority constraint is: In the formula, The earliest observable time for ground targets; This is the latest observable time for ground targets; For the same satellite targeting ground targets and Indicator of whether sequential or preceding observations are required; For attitude transition time, It is a pre-defined, very large positive integer; Total number of satellites; The expression for the satellite energy constraint is: In the formula, Energy consumed for observation; For the first The energy limit of a satellite; The expression for the storage capacity constraint is: In the formula, Storage space consumed for observation, For the first The maximum storage space limit for each satellite; The expression for the load using constraints is: In the formula, For ground targets Load observation sequence identifier; These are decision variables.

8. A heterogeneous payload scheduling system for Earth observation satellites based on deep reinforcement learning, characterized in that, include: The acquisition module is used to acquire satellite parameters, mission parameters, and visible time window information based on the timeline sequence of scheduled missions of the target satellite; The scheduling status and action space construction module is used to construct the satellite scheduling status and candidate joint action space based on the satellite parameters, mission parameters and visible time window information; the satellite scheduling status includes satellite status, mission status and time window status; each candidate joint action in the candidate joint action space includes the mission number to be observed, payload type, satellite number and visible time window number; The iterative loop module is used to execute iterative steps S3-S5 to output a satellite scheduling scheme, including: S3 generates an action mask based on the current satellite scheduling status and preset constraints; The satellite scheduling state is processed by a pre-trained policy network to obtain joint action probabilities; the policy network includes a task encoder, a satellite encoder, a time window encoder, a hierarchical cross-attention aggregation module, a global context encoder, and a conditional layer-by-layer scoring network; the constraints include visible time window constraints, task priority constraints, satellite energy constraints, storage capacity constraints, and payload usage constraints. S4, determine whether there is a valid candidate joint action identified by the action mask in the candidate joint action space; if there is, select the target joint action in the candidate joint action space based on the action mask and the joint action probability; if there is no, terminate the iteration and output the current satellite scheduling scheme. S5, decode the target joint action and insert it into the timeline sequence of the scheduled tasks of the target satellite to obtain a new satellite scheduling scheme, update the satellite scheduling status according to the target joint action, and return to step S3.

9. A computer-readable storage medium, characterized in that, It stores a computer program for scheduling heterogeneous payloads of Earth observation satellites based on deep reinforcement learning, wherein the computer program causes a computer to execute the method for scheduling heterogeneous payloads of Earth observation satellites based on deep reinforcement learning as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the deep reinforcement learning-based heterogeneous payload scheduling method for Earth observation satellites as described in any one of claims 1 to 7.

Images