Satellite task planning method based on graph attention reinforcement learning and storage medium

Abstract

The present application relates to the technical field of satellite task planning and modeling, and particularly relates to a satellite task planning method based on graph attention reinforcement learning and a storage medium. The satellite task planning method based on graph attention reinforcement learning comprises: constructing a to-be-planned directed graph based on a satellite-based meta-task set; defining a state space, an action space, a state transition rule and a reward function of an agent based on the to-be-planned directed graph, a graph attention network and a policy network, so as to model a path planning problem of the to-be-planned directed graph as a Markov decision process; jointly training the graph attention network and the policy network to maximize an expected cumulative discounted reward of the Markov decision process; planning an execution path of the meta-task by using the trained graph attention network and the policy network; and converting the execution path of the meta-task into an observation instruction sequence executable by the satellite. The present application meets the requirement of dynamic task changes, and has the advantages of strong adaptability and fast decision-making speed.

Description

Technical Field

[0001] This invention relates to the field of satellite mission planning and modeling technology, specifically to a satellite mission planning method and storage medium based on graph attention reinforcement learning. Background Technology

[0002] With the rapid development of aerospace technology, the application scenarios of satellites in orbit continue to expand, gradually extending from traditional land resource surveys, meteorological monitoring, and environmental remote sensing to complex fields such as emergency disaster relief, military reconnaissance, and deep space exploration. On the one hand, satellite payload capacity is constantly improving, and the number of meta-missions that can be undertaken per day has increased significantly; on the other hand, multiple constraints such as satellite orbital maneuvers, payload timing, and energy consumption make satellite mission planning increasingly difficult.

[0003] Currently, mainstream satellite mission planning technologies generally rely on traditional optimization methods such as manually designed heuristic rules, genetic algorithms, and particle swarm optimization. When the mission scale is large or the constraints are complex, the planning and solution efficiency is low, making it difficult to meet the real-time or rapid response mission scheduling requirements. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this invention proposes a satellite mission planning method and storage medium based on graph attention reinforcement learning, which improves decision-making speed.

[0005] In one aspect, the present invention proposes a satellite mission planning method based on graph attention reinforcement learning, the method comprising: Constructing a directed graph to be planned based on the set of satellite-based meta-tasks; Based on the directed graph to be planned, the graph attention network, and the policy network, the state space, action space, state transition rules, and reward function of the agent are defined, thereby modeling the path planning problem of the directed graph to be planned as a Markov decision process. The graph attention network and the policy network are jointly trained to maximize the expected cumulative discounted reward of the Markov decision process; The trained graph attention network and policy network are used to plan the execution path of the meta-task; The meta-task execution path is converted into a sequence of observation instructions that can be executed by the satellite.

[0006] Preferably, the steps of "constructing a directed graph to be planned based on a satellite-based meta-task set" include: Map each metatask in the metatask set to a node in the directed graph to be planned; Establish directed edges between nodes that satisfy time feasibility constraints and attitude maneuver constraints; Calculate the weight of each edge or node in the directed graph to be planned based on the planning objective; Virtual start node and virtual end node are introduced into the directed graph to be planned to describe the start and end of the meta-task sequence; The meta-task set includes: point meta-tasks, region meta-tasks, and / or merged meta-tasks.

[0007] Preferably, the step of "calculating the weight of each edge or each node in the directed graph to be planned according to the planning objective" includes: If the planning objective is to minimize the task switching cost, then directed edges will be... The weights are defined as a function of the time required for switching or the attitude maneuvering: ; in, For the edge The cost; For the meta-task Switch to meta-task Time required; For the meta-task Switch to meta-task Required change in lateral yaw angle; and These are weighting coefficients used to balance the time cost and the attitude maneuver cost; If the planning objective is to maximize task benefits, then it is to optimize the nodes. The weight is defined as the reward of the corresponding meta-task of that node. The total path revenue is expressed as: ; in, The total revenue of the path; The selected task execution path in the directed graph to be planned; To execute the meta-task The gains obtained.

[0008] Preferably, the step of "defining the agent's state space, action space, state transition rules, and reward function based on the directed graph to be planned, the graph attention network, and the policy network, thereby modeling the path planning problem of the directed graph to be planned as a Markov decision process" includes: The current executing node and its reachable neighbor information in the directed graph to be planned are used as the state for reinforcement learning: ; ; in, express The state of the agent at any given moment; This indicates the current graph node, corresponding to the meta-task that is about to be executed or is currently being executed; Represents a node The set of outgoing neighbors; The node representation matrix obtained by encoding the directed graph to be planned using the graph attention network is used to provide global structure and context information for decision-making. , For nodes of 3D vector representation, i = 1, 2,..., N ; This represents the total number of nodes in the directed graph to be planned; Indicates the node feature embedding dimension; The state of an agent is defined by the following formula. The following actions will be taken: ; in, Indicates the state The actions of the agent described below are used to... A node is selected from the set of outgoing neighbors to determine the next meta-task to be executed; Indicates the state The following is a set of available actions; Indicates the current node The set of outgoing neighbors; Define the state transition rules as follows: The intelligent agent in Always follow the action probability distribution to select actions This causes the execution node to... Transferred to ,and ; The corresponding next state is: ; in, Indicates the next node; Represents a node The set of outgoing neighbors; The node represents a matrix; Define the reward function: ; in, This indicates that the intelligent agent is from the node Transfer to node The instant reward received.

[0009] Preferably, the reward function is as follows: ; ; in, This indicates that the intelligent agent is from the node Transfer to node The instant reward received; Indicates the execution node The task rewards obtained from the corresponding meta-task; Indicates from node Transfer to node The cost; This indicates the time consumed from the current meta-task to the next meta-task; This represents the change in lateral angle from the current meta-task to the next meta-task; and The weighting coefficients represent the balance time and attitude cost.

[0010] Preferably, if the output layer of the policy network uses the softmax function to normalize the selection probabilities of candidate nodes, then the action probability distribution is: in, The parameter is The policy function in the state The following action probability distribution; Let the node represent the current node in the matrix. Embedded representation; Indicates the candidate next node Embedded representation; A scoring function implemented by a trainable network is used to score the merits of candidate nodes. Indicates the current node The set of out neighbors.

[0011] Preferably, the steps for constructing the node representation matrix include: For each meta-task node in the directed graph to be planned, an initial feature vector is constructed; the initial feature vector includes task time window parameters, attitude constraint parameters, task reward parameters, and orbital cycle identifiers; Based on the initial feature vector, the graph attention network is used to encode the entire directed graph to be planned, obtaining the embedding representation of each node, and then constructing the node representation matrix.

[0012] Preferably, the step of "jointly training the graph attention network and the policy network to maximize the expected cumulative discounted return of the Markov decision process" includes: The parameters of the graph attention network and the policy network are initialized respectively; The graph attention network is used to encode the entire training sample graph to obtain the embedding representation of each node, and then a node representation matrix is ​​constructed. Starting from the virtual starting node, the following steps are executed in a loop: constructing the current state, obtaining the set of outgoing neighbors of the current node, calculating the action probability distribution through the policy network, selecting the next node according to the action probability distribution, calculating the instantaneous reward from the current node to the next node, and recording the state-action-reward triplet information in the trajectory, until the virtual ending node is reached or the preset maximum number of steps is reached. Based on the entire trajectory, the cumulative discount reward for each step is calculated from back to front; for the state and action of each step in the trajectory, the policy gradient term is calculated, and the gradient of the policy network parameters and the gradient of the graph attention network parameters are updated in combination with the cumulative discount reward. Update the parameters of the graph attention network and the policy network according to the gradient ascent direction and the set learning rate; Repeat the process of encoding the training sample graph, selecting nodes and recording trajectories, accumulating gradient values ​​and updating parameters until the preset number of training rounds is completed.

[0013] Preferably, the step of "planning the execution path of the meta-task using the trained graph attention network and the policy network" includes: The initial feature vectors of all meta-task nodes in the directed graph to be planned are encoded using the trained graph attention network to obtain optimized node embedding representations and form a node representation matrix. The node representation matrix is ​​input into the trained policy network, and the action probability distribution is output to guide the agent to select the next meta-task node step by step from the virtual starting node until the virtual ending node is reached, thus obtaining a meta-task execution path that satisfies the constraints.

[0014] In another aspect, the present invention provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute the methods described above.

[0015] The present invention has the following beneficial effects: This invention enables intelligent agents to quickly generate near-optimal or highly satisfactory task execution paths in dynamic task environments through reinforcement learning training. Compared with traditional methods such as manual design heuristics or genetic algorithms, this invention can achieve rapid planning and solution, meet the dynamic task change requirements, and has the advantages of strong adaptability and fast decision-making speed.

[0016] By using graph attention networks to encode complex task constraints and graph structures, node embedding representations with global context information are obtained.

[0017] By using a policy network to progressively select the next meta-task on a directed graph, satellite mission planning is transformed into a sequential decision problem. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the main steps of an embodiment of the satellite mission planning method based on graph attention reinforcement learning in this invention. Detailed Implementation

[0019] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] It should be noted that in the description of this invention, the terms "first" and "second" are used merely for ease of description and do not indicate or imply the relative importance of the described devices, elements, or parameters, and therefore should not be construed as limiting the invention. Furthermore, the term "and / or" in this invention merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0022] The "meta-task set" mentioned in this embodiment of the invention can be constructed through the following steps A10-A70: Step A10: Receive the task set submitted by the user. The task set includes one or more task objectives, each of which can be a point objective or an area objective.

[0023] Point targets in the task set can be identified by latitude and longitude coordinates. The mission-allowed time window indicates that area targets can be identified through the ground boundary polygon. And the allowed time window for the task.

[0024] Step A20: For each point target, obtain the first observable time period set for that point target each time the satellite passes over, the set including one or more first consecutive time periods.

[0025] Specifically, based on the constraints of the satellite imaging environment (satellite orbit parameters, sensor field of view, and illumination conditions), the first set of observable time periods for each point target within the planned time range can be obtained through orbit propagation and geometric relationship calculations.

[0026] The first set of observable time periods here is already within the time window allowed by the user for the task. That is, the first set of observable time periods is obtained by intersecting the time window allowed by the task with the original observable time range when a satellite passes through a certain orbit.

[0027] Step A30: Divide one or more executable time windows within each first continuous time period, and construct a corresponding point element task based on each executable time window and the corresponding available lateral swing angle range and observation duration.

[0028] Specifically, within each first consecutive time period, one or more executable time windows are segmented based on satellite imaging attitude constraints (satellite attitude maneuverability, available side angle range, and imaging duration).

[0029] For example, for the first Given a single point objective, several meta-tasks can be constructed as shown in formula (1) below: (1) in, Indicates the first The target point is in its first j The first generation generated within the consecutive time period k Individual task; This indicates that the target point is at the [number]th [location]. The first [time period] within which imaging of the target point can be completed k One executable time window; Indicates that in the first k The range of available lateral tilt angles within an executable time window that can complete imaging of the target point; For the first k The observation duration required to perform one imaging operation within an executable time window. This satisfies the time window and attitude constraints. Collected into the point meta task set middle.

[0030] A meta-task (including the "point meta-task" mentioned above and the "region meta-task" to be mentioned below) refers to an "executable observation unit" that is abstracted from a certain task target (a sub-region of a point target or a regional target) under given orbit, attitude and load constraints, combined with its observable time window and available attitude range.

[0031] Typically, a meta-task corresponds to a complete imaging activity, with well-defined triplet parameters: the executable time window (start / end time), the available lateral angle range (minimum / maximum lateral angle), and the observation duration.

[0032] Step A40: For each regional target, obtain a second set of observable time periods for the regional target each time the satellite passes over, the set including one or more second consecutive time periods.

[0033] Specifically, based on the constraints of the satellite imaging environment, a second set of observable time periods for each regional target is obtained.

[0034] Step A50: For each second consecutive time period, calculate the range of available side swing angles that allow the payload field of view to at least partially overlap with the target in that region. Treat the second consecutive time period as an executable time window and construct a corresponding regional meta-task by combining the corresponding available side swing angle range and observation duration.

[0035] It should be noted that, for each regional target, the present invention does not perform spatial grid or strip discretization, but for each second consecutive time period, it calculates the range of available lateral swing angles that allow the load field of view to at least partially overlap with the regional target based on orbital geometry and imaging field of view.

[0036] The second set of observable time periods here is already within the time window allowed by the user for the mission. It is obtained by intersecting the mission's allowed time window with the original observable time range when a satellite passes through a certain orbit.

[0037] Assume the first Regional Targets Boundary polygon The description states that the allowed time window for the task is... When a satellite passes over a certain orbit, the second set of observable time periods for that region includes several second consecutive time periods, as shown in formula (2): (2) in, Indicates regional targets The second set of observable time periods within this orbital period; This indicates the number of second consecutive time intervals for the target in this region within the orbital period, with each second consecutive time interval serving as an executable time window; k Indicates the sequence number of the executable time window. ; Indicates the first The start and end times of each executable time window (already related to the task's allowed time window) (Find the intersection).

[0038] Assuming the satellite payload's permissible global yaw angle range during this orbital transit is: ,For example .

[0039] In regional objectives The Within an executable time window, the satellite sub-points and the region boundary change over time. With different relative positions, do the ground field projections corresponding to different lateral tilt angles match the actual ground field projections? The intersections are also different. This invention uses geometric calculations to determine the entire executable time window. Within, this makes the load's field of view and the region The set of at least partially overlapping lateral sway angle values ​​is shown in formula (3): (3) in, Indicates the time window during execution. Inside, making the field of view and area The set of all overlapping lateral sway angle values; Indicates the load-side sway angle; Indicates at time Side swing angle is At that time, the area covered by the load's field of view on the ground; This indicates that the field of view has a non-empty intersection with the target in the region, meaning that it covers at least a part of the region.

[0040] In engineering implementation, It is usually approximated as a second continuous time interval or the union of several second continuous time intervals. To simplify modeling, this invention uses its main interval as an approximation as follows: (4) (4) For example: during a transit, for regional targets The k A set of executable time windows are used to calculate the available lateral sway angle range. This means that within the executable time window, as long as the satellite side swing angle remains within the specified range... Within the range, the field of view will cover a portion of the target area.

[0041] Based on the available lateral angle range, a regional meta-task can be constructed around this executable time window. If a specific execution time interval is selected within this executable time window... Satisfy the following formulas (5) and (6): (5) (6) The corresponding regional meta-task can then be expressed as formula (7): (7) in, Indicates regional targets The first under the transit of this orbit One executable time window; Indicates that in the first Within an executable time window, the available yaw angle range allows the field of view to align with the target area as long as the yaw angle falls within this range. The overlap occurs, thus enabling a single effective observation of the region; Indicates that in the first The duration of an imaging operation for a regional target within an executable time window (which can be determined by planning parameters such as transit length and required coverage ratio).

[0042] For the r Regional Targets The set of regional meta-tasks is shown in formula (8): (8) in, This indicates the number of the second consecutive time intervals. Using the above method, this invention directly calculates the effective lateral swing angle range within each executable time window, instead of first discretizing the region spatially and then generating multiple time windows for each sub-block. The resulting regional meta-task naturally includes three sets of key parameters: executable time window, available lateral swing angle range, and observation duration. This facilitates unified modeling with the point-based meta-task and its use in subsequent construction of the directed graph to be planned.

[0043] Step A60: Generate the complete set of first-dimensional tasks based on all point-level tasks and all region-level tasks, as shown in formulas (9)-(11): (9) (10) (11) in, This represents the complete set of first-dimensional tasks; This represents the set of point meta-tasks corresponding to all point targets; This represents the set of regional meta-tasks corresponding to all regional objectives. i and j These represent the number of the point target and the number of the area target, respectively; Indicates the first i The set of point-based tasks corresponding to each point target; Indicates the first r The set of regional meta-tasks corresponding to each regional objective.

[0044] Step A70: Based on the complete set of first-level meta-tasks, merge multiple meta-tasks that can be completed by continuous imaging with the same side angle during a single satellite pass, thereby obtaining a meta-task set.

[0045] After the meta-tasks are generated, executing them one by one on the complete set of first meta-tasks often results in the satellite serving only a single target or a single sub-region in each imaging operation, failing to fully utilize the imaging swath width and continuous observation capabilities. To improve the overall utilization rate of a single observation and increase the task combination method of "completing multiple targets in one imaging," this invention, under the premise of satisfying time windows, attitude maneuvers, and resource constraints, merges some compatible meta-tasks to form a larger-granularity "merged meta-task," used to describe multi-target observation activities that can be completed simultaneously or continuously during a single imaging process. In this invention, the purpose of meta-task merging is to merge multiple targets that can be continuously observed under the same orbital transit and the same side angle conditions into a single "continuous observation meta-task" without increasing intermediate attitude maneuvers, thereby indicating that a single imaging operation can simultaneously or sequentially cover multiple targets.

[0046] Specifically, step A70 may include steps A71-A74: Step A71: Select multiple meta-tasks from the complete set of first meta-tasks that can be completed by continuous imaging with the same side angle during a single satellite pass and use them as meta-tasks to be merged.

[0047] For example, two meta-tasks to be merged were selected. and As shown in formulas (12) and (13) below: (12) (13) in, and Representing meta-tasks and The executable time window; and Representing meta-tasks and The range of available lateral swing angles; and Representing meta-tasks and The duration of the observation.

[0048] In real-world scenarios, multiple groups of meta-tasks may be selected to be merged, with each group containing two or more meta-tasks. A merge operation needs to be performed on each group of meta-tasks.

[0049] This invention proposes that only a pair of meta-tasks that meet the following conditions (a)-(c) can be merged into a single meta-task of "single-attitude single-pass continuous observation" (i.e., continuous imaging with the same lateral tilt angle during a single satellite pass): (a) Passing over the same orbit or the same observation strip: The targets (point targets or regional sub-blocks) corresponding to the two meta-tasks are located on the same or adjacent strips during the same orbital transit, allowing the satellite to sequentially scan the meta-tasks without interrupting observations during a single transit. and Corresponding area.

[0050] In implementation, constraints can be imposed using discrete identifiers such as "orbit cycle number" and "strip number," for example, requiring the meta-task to... and The conditions shown in formulas (14) and (15) must be met: (14) (15) in, Indicates the orbital revolution number. Indicates the strip or track number.

[0051] (b) Same lateral sway angle (no mid-way adjustment allowed): The merged observations require the use of the same target side swing angle value throughout the entire continuous observation period. That is, in executing the meta-task and No intermediate attitude adjustment is performed at this time. Therefore, a certain... It simultaneously satisfies the conditions shown in formulas (16) and (17) below: (16) (17) That is, it needs to satisfy formula (18): (18) If the available lateral swing angle ranges of the two do not overlap, it means that the meta-task cannot be completed simultaneously under the same lateral swing angle. and At this point, they are not merged; they remain as two independent meta-tasks.

[0052] (c) It can be covered by a single continuous observation in time: Because this embodiment requires a single pass and continuous imaging in a single burst, it does not allow for meta-task... and The imaging process involves stopping and restarting the camera or moving between these two points, so they must be covered by a continuous imaging sequence over time.

[0053] Assuming the satellites pass over the same orbit and tilt at the same angle on the same side... Perform continuous imaging, from time Start, until time The entire continuous imaging duration is shown in formula (19): (19) The continuous observation interval must be able to cover and Each has its own observation interval, and the whole exists within its own executable time window, i.e., there is... and , so that: Meta-task and The conditions shown in formulas (20)-(22) are satisfied respectively: (20) (twenty one) (twenty two) in, and Representing meta-tasks and The executable time window; Representing meta-tasks The duration of the observation; Indicates the swing angle on the same side Below, the satellite starts from the meta-mission. Ground projection of the coverage area is scanned to the meta-task. The time required for the ground projection of the covered area (determined by the geometric relationship between the trajectory and the ground projection).

[0054] In engineering implementation, the above condition can be simplified to: the existence of a continuous observation interval. It can simultaneously satisfy formulas (23)-(25): (twenty three) (twenty four) (25) in, Representing meta-tasks The duration of the observation.

[0055] If the above conditions are met, it means that in a continuous imaging operation, the meta-task can be covered sequentially without changing the side-swing angle. and In the corresponding region, the meta-task can be performed at this time. and Merge.

[0056] For a pair of meta-tasks that satisfy the above three conditions (a)-(c) and This invention combines them into a new "continuous observation meta-task". .

[0057] Step A72: Calculate the executable time window, observation duration, and available lateral swing angle range of the merged meta-task based on the meta-tasks to be merged.

[0058] Specifically, the executable time window of the merged meta-task is calculated according to the following formula (26): (26) in, This indicates that the same lateral swing angle is used when passing through the same orbit. The time range for continuous observation can be initiated, i.e., the merged meta-task. The executable time window.

[0059] Merged Metatask The observation duration is shown in formula (27): (27) in, Represents the merged continuous observation meta-task The duration of the observation; Indicates the swing angle on the same side Below, from the coverage From area to coverage The sweeping time of the area.

[0060] Because the swing angle within the entire continuous observation interval is required to remain constant, the merged meta-task The available range of side swing angles can be taken as the intersection of the two, as shown in formula (28): (28) In actual execution, a specific execution lateral angle can be selected within this range. Throughout the entire observation period Maintain this posture during the period.

[0061] Merged Metatask This can be expressed as formula (29): (29) Meanwhile, meta-task The target set is the union of the targets corresponding to the original two meta-tasks, i.e. Complete the imaging process in a single continuous imaging operation , Overall coverage of both objectives.

[0062] If a group of metatasks to be merged contains more than two metatasks, the above method can be used to merge them one by one.

[0063] Step A73: Calculate the total revenue and total resource consumption of the merged meta-tasks.

[0064] For the merged meta-task The total revenue is shown in formula (30): (30) in, and Meta-tasks He Yuan Mission The benefits, Total revenue; For the merged meta-task, the total resource consumption is shown in formula (31): (31) in, and Meta-tasks He Yuan Mission resource consumption, This indicates a continuous sweeping meta-task under a fixed attitude. He Yuan Mission Additional resource consumption in the area between corresponding targets (such as increased data volume due to imaging time). This represents the total resource consumption.

[0065] Situations where meta-tasks cannot be merged: If any of the conditions (d)-(f) below are not met, then no merging will be performed, especially in the following cases: (d) An intermediate attitude adjustment is required before starting from the meta-task. Required observation attitude change to meta-task The required observation attitude, i.e., there is no single one (e) The two meta-tasks do not overlap in the same transit or the same strip, and cannot be naturally swept over in a single continuous imaging operation; (f) Although there is overlap in the time window, it is not possible to allocate enough time for continuous observation to complete the meta-task. and Add the time spent sweeping in between.

[0066] In the above circumstances, and They are kept as two independent meta-tasks, serving only as two independent nodes in the directed graph to be planned, for subsequent planning decisions to select the execution order or to discard them.

[0067] Step A74: Combine the meta-tasks that were not selected from the first set of meta-tasks and the merged meta-tasks into a meta-task set.

[0068] In the set of metatasks, each metatask is uniquely characterized by a triplet of parameters consisting of the corresponding executable time window, available lateral angle range, and observation duration.

[0069] Figure 1 This is a schematic diagram illustrating the main steps of an embodiment of the satellite mission planning method based on graph attention reinforcement learning in this invention. Figure 1 As shown, the planning method in this embodiment includes steps S10-S50: Step S10: Construct a directed graph to be planned based on the satellite-based meta-task set.

[0070] The meta-task set includes: point meta-tasks, region meta-tasks, and / or merged meta-tasks.

[0071] Specifically, step S10 may include steps S11-S14: Step S11: Map each metatask in the metatask set to a node in the directed graph to be planned.

[0072] Step S12: At the node that satisfies the time feasibility constraint and attitude maneuver constraint and nodes Establish directed edges between them This indicates that after the node has been executed... Corresponding meta-task Then it can switch to executing nodes. Corresponding meta-task .

[0073] (1) Time feasibility constraints: Meta-task The execution end time to the time when the meta-task is ready to be executed The required time is denoted as (Mainly determined by attitude maneuvering, system preparation time, etc.), then the time feasibility constraint can be written as formula (32): (32) in, For meta-task The end time; To complete Then converted to executable Time required; For meta-task The start time of execution.

[0074] at the same time, It also needs to satisfy its own time window constraints, as shown in formulas (33) and (34): (33) (34) in, Representing meta-tasks The executable time window; Representing meta-tasks The duration of the observation.

[0075] (2) Attitude maneuver constraints (side roll angle variation): Let the execution meta-task be set The representative side swing angle at that time is Execute meta-task The representative side swing angle at that time is Then from Switch to The required attitude change can be expressed as formula (35): (35) in, Indicates from the meta-task Switch to meta-task Required change in lateral yaw angle; Indicates the execution of a meta-task Typical or target lateral swing angle (e.g., lateral swing angle at the median of the time window); Indicates the execution of a meta-task Typical or target side swing angle.

[0076] If the satellite's maximum yaw rate is Therefore, theoretically, the minimum time required to complete this attitude maneuver is as shown in formula (36): (36) in, This indicates the maximum angular velocity of the satellite's side yaw angle; This represents the lower bound of the time consumed for attitude maneuvers and related preparations (a safety factor can be added in practice).

[0077] Therefore, in constructing edges At that time, it should be based on and Calculate or estimate And in conjunction with the aforementioned time window constraints, determine whether there are any conditions that are met. If both time and attitude maneuver constraints are satisfied, then it is considered that from... arrive The transformation is feasible by adding directed edges to the graph. in, This represents the set of edges in the directed graph to be planned.

[0078] Step S13: Calculate the weight of each edge or node in the directed graph to be planned according to the planning objective.

[0079] If the planning objective is to minimize the cost of task switching, then there will be directed edges. The weight is defined as a function of the time required for switching or the attitude maneuver, as shown in formula (37): (37) in, For the edge The cost; For the meta-task Switch to meta-task Time required; For the meta-task Switch to meta-task Required change in lateral yaw angle; and is a weighting coefficient used to balance the time cost and the attitude maneuver cost.

[0080] If the planning objective is to maximize task benefits, then the nodes will be... The weight is defined as the reward of the corresponding meta-task of that node. The total path revenue is expressed as shown in formula (38): (38) in, The total revenue of the path; The selected task execution path in the directed graph to be planned; To execute the meta-task The benefits obtained (such as weight, coverage area, benefit function value, etc.).

[0081] Step S14: Introduce a virtual starting node into the directed graph to be planned. and virtual endpoint nodes This is used to describe the start and end of the meta-task sequence, thus unifying the entire task planning problem into a single representation starting from... arrive The path search problem.

[0082] (1) Starting node : Virtual starting point This represents the initial state of the plan, which can be connected to the first meta-task that is feasible in all timeframes: If the meta-task If the operation can be performed after the planning start time and the constraints are satisfied, then an edge is established. .in, Virtual starting node Representation and Meta-task The corresponding node.

[0083] (2) End point : Virtual endpoint This indicates the final state of the plan, for all terminal meta-tasks that can be completed before the plan's deadline. Establish edges .in, This represents a virtual endpoint node.

[0084] By introducing and The entire task planning problem can be uniformly represented as starting from... arrive This addresses the path search problem, facilitating subsequent solution using reinforcement learning-based sequence decision-making methods.

[0085] Through the above steps, this invention will set up a meta-task set. Mapped to a directed graph to be planned , where the node set Corresponding to all meta-tasks; edge set Feasible transformation relationships between meta-tasks are expressed through constraints such as time and attitude; edge weights or node rewards are used to characterize the cost and benefit of task execution.

[0086] Based on the directed graph model to be planned, the satellite mission planning problem can be formalized as finding a path from the graph... arrive The feasible paths are identified and solved under a given optimization objective (such as maximum benefit or minimum cost), providing a structured state space and transition relationships for the subsequent introduction of reinforcement learning agents.

[0087] Step S20: Define the state space, action space, state transition rules, and reward function of the agent based on the directed graph to be planned, the graph attention network, and the policy network, thereby modeling the path planning problem of the directed graph to be planned as a Markov decision process.

[0088] Specifically, step S20 may include steps S21-S24: Step S21: Take the current execution node and its reachable neighbors in the directed graph to be planned as the state of reinforcement learning, as shown in formulas (39)-(40): (39) (40) in, express The state of the agent at any given time; This indicates the current graph node, corresponding to the meta-task that is about to be executed or is currently being executed; Represents a node The out-neighbor set (i.e. from) (The set of candidate meta-tasks that can be directly reached from the starting point). This is the node representation matrix obtained after encoding the directed graph to be planned using a graph attention network, which is used to provide global structural and contextual information for decision-making. , For nodes of 3D vector representation, i = 1, 2,..., N ; This represents the total number of nodes in the directed graph to be planned; This indicates the dimension of node feature embedding.

[0089] In this embodiment, the node representation matrix is ​​constructed according to the following steps (1)-(2): (1) Construct an initial feature vector for each metatask node in the directed graph of the planning.

[0090] For example, for nodes Construct the initial feature vector This may include: task time window parameters (such as...) ), attitude constraint parameters (such as ), task reward parameters (such as ), and other constraints or attributes (orbit cycle identifier, target type identifier, etc.).

[0091] By constructing an initial feature vector for each node, we can obtain the node feature set, as shown in formula (41): (41) (2) Based on the initial feature vector, a graph attention network is used to encode the entire directed graph to be planned, and the embedding representation of each node is obtained, and then the node representation matrix is ​​constructed.

[0092] Taking a single-layer graph attention network (GAT) as an example, for each node... The new embedding representation is calculated as shown in Equation (42): (42) in, Represents a node The updated embedded representation; Represents a non-linear activation function, such as ReLU; Represents a node The set of neighboring nodes; Represents a trainable linear transformation matrix; Indicates from neighboring nodes To the central node Attention weights.

[0093] Attention weight The calculation can be written as formula (43): (43) in, Indicates neighbors For nodes Attention weights; This represents a trainable attention vector; This represents a vector concatenation operation; and Representing nodes respectively and The representation of the initial feature vector after a linear transformation.

[0094] After several layers of GAT updates, the final embedding representation of each node can be obtained. This, in turn, constitutes the aforementioned node representation matrix. .

[0095] Step S22: Define the agent in state according to formula (44). The following actions will be taken: (44) in, Indicates the state The actions of the agent are used in the node Select a node from the set of outgoing neighbors to determine the next meta-task to be executed; Indicates the state The following is a set of available actions; Indicates the current node The set of outgoing neighbors. When the next node selected is the virtual endpoint. When this occurs, it indicates that the task sequence has terminated. Step S23: Define state transition rules.

[0096] Specifically, intelligent agents in Always follow the action probability distribution to select actions This causes the execution node to... Transferred to ,and .

[0097] The corresponding next state is shown in formula (45): (45) in, Indicates the next node; Represents a node The set of outgoing neighbors; The node represents a matrix that can be considered as being updated periodically or once by the graph attention network during training or planning, or it can be updated dynamically as needed (e.g., recalculated only when the graph structure changes).

[0098] If the output layer of the policy network uses the softmax function to normalize the selection probability of candidate nodes, then the action probability distribution is as shown in formula (46): (46) in, The parameter is The policy function in the state The probability distribution of actions under the following conditions; Let node represent the current node in the matrix. Embedded representation; Indicates the candidate next node Embedded representation; A scoring function implemented by a trainable network is used to score the merits of candidate nodes. Indicates the current node The set of out neighbors.

[0099] To guide the agent in selecting a path according to the planned objective, this invention uses the following step S24 for each step of the transition. Design instant rewards.

[0100] Step S24: Define the reward function, as shown in formula (47): (47) in, This indicates that the intelligent agent is from the node Transfer to node The instant reward received.

[0101] A typical design of the reward function is shown in formulas (48)-(49): (48) (49) in, Indicates that the agent starts from the node Transfer to node The immediate rewards obtained are used to guide the agent to choose a path that is both high-yield and low-cost; Indicates the execution node The task rewards obtained from the corresponding meta-task; Indicates from node Transfer to node The cost; This indicates the time consumed from the current meta-task to the next meta-task; This represents the change in lateral angle from the current meta-task to the next meta-task; and The weighting coefficients represent the balance time and attitude cost.

[0102] Step S30: Jointly train the graph attention network and the policy network to maximize the expected cumulative discounted reward of the Markov decision process.

[0103] This embodiment can use policy gradient methods (such as REINFORCE, Actor-Critic, or PPO) to apply the parameters of the policy network. The parameters of the graph attention network are jointly trained.

[0104] Taking turn-based task planning as an example, a complete path can be represented as shown in formula (50): (50) The corresponding cumulative discount return is shown in formula (51): (51) in, This represents the cumulative discount reward for the entire path from the starting point to the end point; Indicates the discount factor; Indicates the first Instant rewards gained from step transfers; This indicates the number of decision steps before training ends.

[0105] The objective of policy gradient updates is to maximize the expected cumulative discount return as shown in equation (52): (52) The policy gradient is approximated by formula (53): (53) in, Indicates the number of sampling paths (rounds); and They represent the first The path in the first Step actions, status, and cumulative discount rewards; Represents baseline terms (such as value function estimates), used to reduce variance; This represents the gradient with respect to the policy parameters.

[0106] Through the above updates, the policy network and GAT parameters will be gradually adjusted so that the agent's decisions on the graph tend to generate high-return paths, that is, to select the meta-task sequence with high returns and low costs while satisfying constraints.

[0107] In the training process example below, the input data includes: a training sample graph (this graph has the same feature space as the directed graph to be planned, i.e., similar topology and consistent node feature definitions), the initial feature vector x_i of each node v_i, the number of training episodes N_episodes, the discount factor γ, and the learning rate η. The output data includes: the trained GAT parameters θ_GAT and the trained policy network parameters θ_π. The training process is as follows: Initialize θ_GAT Initialize θ_π For episode = 1 to N_episodes, do: 1. Encode the entire graph using the current GAT to obtain the embedding vector H for each node. H = GAT_Encode(G, {x_i}, θ_GAT) # 2. Generate a path starting from the virtual starting node v_s. v = v_s Trajectory Traj = empty list while v is not a virtual endpoint node v_t and the maximum number of steps has not been reached do # Construct the current state (such as the embedding vector of the current node). s = BuildState(v, H) # The current set of possible next-next neighbor nodes A = OutNeighbors(G, v) If A is empty, then the current round ends. # Calculate the selection probability of each candidate node based on the policy network π_θ π = PolicyNetwork(s, A, θ_π) # Select the next node v_next based on probability distribution π v_next = SampleFrom(π) # Calculate the instantaneous reward r from v to v_next r = ComputeReward(v, v_next) Add (s, a, r) ​​to the trajectory Traj Where a = v_next # State Transition v = v_next end while # 3. Update network parameters based on the reward for the entire trajectory (REINFORCE approach) # Calculate the cumulative discount reward G_k for each step (cumulative from back to front) Starting from the last step of Traj and working backwards, calculate G_k at each step. grad_θ_π = 0 For each step (s_k, a_k, r_k, G_k) in Traj, execute: # Calculate the policy gradient term for this step θ_π log π_θ(a_k | s_k) g = Grad_log_Policy(s_k, a_k, θ_π) # Accumulate the gradients of the policy parameters and GAT parameters grad_θ_π+= g*G_k # 4. Update parameters according to the gradient ascent direction θ_π = θ_π + η * grad_θ_π The graph attention network parameter θ_GAT is updated simultaneously through backpropagation of the policy network. end for Return θ_GAT, θ_π Specifically, step S30 may include steps S31-S37: Step S31: Initialize the parameters of the graph attention network and the policy network respectively.

[0108] Step S32: Encode the entire training sample graph using a graph attention network to obtain the embedding representation of each node, and then construct the node representation matrix.

[0109] Step S33: Starting from the virtual starting node, execute the following in a loop: construct the current state, obtain the set of out neighbors of the current node, calculate the action probability distribution through the policy network, select the next node according to the action probability distribution, calculate the instant reward from the current node to the next node, and record the state-action-reward triplet information in the trajectory until the virtual ending node is reached or the preset maximum number of steps is reached.

[0110] Step S34: Based on the entire trajectory, calculate the cumulative discount reward for each step from back to front.

[0111] Step S35: For each state and action in the trajectory, calculate the policy gradient term, and update the gradient of the policy network parameters and the gradient of the graph attention network parameters by combining the cumulative discount reward.

[0112] Step S36: Update the parameters of the graph attention network and the policy network with the set learning rate according to the gradient ascent direction.

[0113] Step S37: Repeat steps S33-S36 until the preset number of training rounds is completed.

[0114] Step S40: Use the trained graph attention network and policy network to plan the execution path of the meta-task.

[0115] Specifically, step S40 may include steps S41-S42: Step S41: Encode the initial feature vectors of all meta-task nodes in the directed graph to be planned using the trained graph attention network to obtain the optimized node embedding representation and construct the node representation matrix.

[0116] Step S42: Input the node representation matrix into the trained policy network, output the action probability distribution, guide the agent to start from the virtual starting node, gradually select the next meta-task node, until the virtual ending node is reached, and obtain the meta-task execution path that satisfies the constraints.

[0117] Step S50: Convert the meta-task execution path into a sequence of observation instructions that can be executed by the satellite.

[0118] Although the steps in the above embodiments are described in the above order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not need to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple variations are all within the protection scope of this invention.

[0119] Based on the above method embodiments, the present invention also provides an embodiment of a computer-readable storage medium, wherein the storage medium of this embodiment stores a computer program that can be loaded by a processor and execute the methods described above.

[0120] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] Those skilled in the art will recognize that the method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

[0122] The technical solution of the present invention has now been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A satellite mission planning method based on graph attention reinforcement learning, characterized in that, The method includes: Constructing a directed graph to be planned based on a set of satellite-based meta-tasks; Based on the directed graph to be planned, the graph attention network, and the policy network, the state space, action space, state transition rules, and reward function of the agent are defined, thereby modeling the path planning problem of the directed graph to be planned as a Markov decision process. The graph attention network and the policy network are jointly trained to maximize the expected cumulative discounted reward of the Markov decision process; The trained graph attention network and policy network are used to plan the execution path of the meta-task; The meta-task execution path is converted into a sequence of observation instructions that can be executed by the satellite.

2. The satellite mission planning method based on graph attention reinforcement learning according to claim 1, characterized in that, The steps for "constructing a directed graph to be planned based on a set of satellite-based meta-tasks" include: Map each metatask in the metatask set to a node in the directed graph to be planned; Establish directed edges between nodes that satisfy time feasibility constraints and attitude maneuver constraints; Calculate the weight of each edge or node in the directed graph to be planned based on the planning objective; Virtual start node and virtual end node are introduced into the directed graph to be planned to describe the start and end of the meta-task sequence; The meta-task set includes: point meta-tasks, region meta-tasks, and / or merged meta-tasks.

3. The satellite mission planning method based on graph attention reinforcement learning according to claim 2, characterized in that, The steps of "calculating the weight of each edge or each node in the directed graph to be planned according to the planning objective" include: If the planning objective is to minimize the task switching cost, then directed edges will be... The weights are defined as a function of the time required for switching or the attitude maneuvering: ； in, For the edge The cost; For the meta-task Switch to meta-task Time required; For the meta-task Switch to meta-task Required change in lateral yaw angle; and These are weighting coefficients used to balance the time cost and the attitude maneuver cost; If the planning objective is to maximize task benefits, then it is to optimize the nodes. The weight is defined as the reward of the corresponding meta-task of that node. The total path revenue is expressed as: ； in, The total revenue of the path; The selected task execution path in the directed graph to be planned; To execute the meta-task The gains obtained.

4. The satellite mission planning method based on graph attention reinforcement learning according to claim 2, characterized in that, The steps of "defining the agent's state space, action space, state transition rules, and reward function based on the directed graph to be planned, the graph attention network, and the policy network, thereby modeling the path planning problem of the directed graph to be planned as a Markov decision process" include: The current executing node and its reachable neighbor information in the directed graph to be planned are used as the state for reinforcement learning: ； ； in, express The state of the agent at any given moment; This indicates the current graph node, corresponding to the meta-task that is about to be executed or is currently being executed; Represents a node The set of outgoing neighbors; The node representation matrix obtained by encoding the directed graph to be planned using the graph attention network is used to provide global structure and context information for decision-making. , For nodes of 3D vector representation, i=1,2,…,N ; This represents the total number of nodes in the directed graph to be planned; Indicates the node feature embedding dimension; The state of an agent is defined by the following formula: The following actions will be taken: ； in, Indicates the state The actions of the agent described below are used to... A node is selected from the set of outgoing neighbors to determine the next meta-task to be executed; Indicates the state The following is a set of available actions; Indicates the current node The set of out-neighbors; Define the state transition rules as follows: The intelligent agent in Always follow the action probability distribution to select actions This causes the execution node to... Transferred to ,and ; The corresponding next state is: ； in, Indicates the next node; Represents a node The set of outgoing neighbors; The node represents a matrix; Define the reward function as follows: ； in, This indicates that the intelligent agent is from the node Transfer to node The instant reward received.

5. The satellite mission planning method based on graph attention reinforcement learning according to claim 4, characterized in that, The reward function is specifically as follows: ； ； in, This indicates that the intelligent agent is from the node Transfer to node The instant reward received; Indicates the execution node The task rewards obtained from the corresponding meta-task; Indicates from node Transfer to node The cost; This indicates the time consumed from the current meta-task to the next meta-task; This represents the change in lateral sway angle from the current meta-task to the next meta-task; and The weighting coefficients represent the balance time and attitude cost.

6. The satellite mission planning method based on graph attention reinforcement learning according to claim 4, characterized in that, If the output layer of the policy network uses the softmax function to normalize the selection probabilities of candidate nodes, then the action probability distribution is: in, The parameter is The policy function in the state The following action probability distribution; Let the node represent the current node in the matrix. Embedded representation; Indicates the candidate next node Embedded representation; A scoring function implemented by a trainable network is used to score the merits of candidate nodes. Indicates the current node The set of out neighbors.

7. The satellite mission planning method based on graph attention reinforcement learning according to claim 4, characterized in that, The steps for constructing the node representation matrix include: For each meta-task node in the directed graph to be planned, an initial feature vector is constructed; the initial feature vector includes task time window parameters, attitude constraint parameters, task reward parameters, and orbital cycle identifiers; Based on the initial feature vector, the graph attention network is used to encode the entire directed graph to be planned, obtaining the embedding representation of each node, and then constructing the node representation matrix.

8. The satellite mission planning method based on graph attention reinforcement learning according to claim 4, characterized in that, The step of "jointly training the graph attention network and the policy network to maximize the expected cumulative discounted return of the Markov decision process" includes: The parameters of the graph attention network and the policy network are initialized respectively; The graph attention network is used to encode the entire training sample graph to obtain the embedding representation of each node, and then a node representation matrix is ​​constructed. Starting from the virtual starting node, the following steps are executed in a loop: constructing the current state, obtaining the set of outgoing neighbors of the current node, calculating the action probability distribution through the policy network, selecting the next node according to the action probability distribution, calculating the instantaneous reward from the current node to the next node, and recording the state-action-reward triplet information in the trajectory, until the virtual ending node is reached or the preset maximum number of steps is reached. Based on the entire trajectory, the cumulative discount reward for each step is calculated from back to front; for the state and action of each step in the trajectory, the policy gradient term is calculated, and the gradient of the policy network parameters and the gradient of the graph attention network parameters are updated in combination with the cumulative discount reward. Update the parameters of the graph attention network and the policy network according to the gradient ascent direction and the set learning rate; Repeat the process of encoding the training sample graph, selecting nodes and recording trajectories, accumulating gradient values ​​and updating parameters until the preset number of training rounds is completed.

9. The satellite mission planning method based on graph attention reinforcement learning according to claim 8, characterized in that, The step of "planning the execution path of the meta-task using the trained graph attention network and policy network" includes: The initial feature vectors of all meta-task nodes in the directed graph to be planned are encoded using the trained graph attention network to obtain optimized node embedding representations and form a node representation matrix. The node representation matrix is ​​input into the trained policy network, and the action probability distribution is output to guide the agent to select the next meta-task node step by step from the virtual starting node until the virtual ending node is reached, thus obtaining a meta-task execution path that satisfies the constraints.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and execute the method as described in any one of claims 1-9.

Images