Constellation design method, device and equipment based on evolutionary reinforcement learning and memory

Abstract

The application relates to a constellation design method, device and equipment based on evolutionary reinforcement learning and a memory. The method comprises the following steps: converting a constellation design task problem into a multi-satellite earth observation task planning problem; modeling the multi-satellite earth observation task planning problem as a multi-objective combinatorial optimization problem; solving the multi-objective combinatorial optimization problem by using a double-layer circulation model with a double-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm, and optimizing a constellation deployment design scheme; the double-layer circulation model is used for realizing task planning of dynamic target observation through an inner-layer circulation based on multi-agent reinforcement learning, feeding rewards and losses of an action network to an outer-layer circulation, and optimizing the constellation deployment design scheme by using the outer-layer circulation based on the multi-objective genetic algorithm to find a group of satellite constellation parameters with maximum return, minimum loss and meeting constraint conditions as a target. The method improves the scheme design efficiency and the representation ability of time window data.

Description

Technical Field

[0001] The present application relates to the field of constellation design technology, and in particular to a constellation design method, apparatus, device, and memory based on evolutionary reinforcement learning. Background Art

[0002] With the rapid development of satellite communication technology, low-orbit Internet constellations represented by Starlink and Kuiper can achieve seamless global coverage. Such constellations can not only observe ground areas, but also be converted into celestial observation constellations. Among them, converting such constellations into celestial observation constellations can achieve the integration of real-time astronomical observation and large-area celestial coverage advantages, which may give rise to new economic and industrial forms in the field of astronomical observation, such as popular astronomical entertainment, consumption, and education industries.

[0003] Constellation design is the process of planning, designing, and configuring a group of satellites to meet specific mission and coverage requirements. This process involves deciding on key parameters such as the number of satellites, their orbital parameters, relative positions, and communication frequencies to ensure the constellation can effectively perform its mission.

[0004] The design of a satellite constellation deployment plan involves multiple core steps, such as launch preparation, launch time selection, actual launch, orbit insertion, data collection, processing, and transmission. The primary goal of constellation deployment design is to optimize resource allocation and workflows to rapidly establish a constellation network and meet specific objectives. The responsiveness of a constellation is influenced by the constellation layout and satellite orbital parameters. To meet mission requirements, the constellation layout and orbital parameters must be determined first. To this end, constellation design strategies primarily focus on optimizing these parameters, conducting quantitative analysis and problem-solving based on the specific mission requirements. Because different missions have different deployment requirements, constellation optimization and decision-making must be tailored to specific scenarios. Currently, to achieve multi-satellite networking, rapid on-orbit response, and continuous coverage, a decision-making framework applicable to various scenarios is still needed to assist decision-makers in resolving problems subject to multiple constraints and conflicting constraints and finding appropriate solutions.

[0005] Currently, most approaches to constellation deployment design use evolutionary computation, employing a single-layer iterative optimization approach to solve orbital parameters based on regional coverage performance to achieve constellation design. In scenarios typically associated with dynamic target observation, constellation design cannot be implemented without completing intelligent mission planning, thereby achieving higher efficiency and reducing resource consumption. To address this issue, this paper proposes a two-layer framework for satellite constellation morphology evolution. This two-layer algorithm implements dynamic target observation mission planning in the inner loop, while the outer loop optimizes the constellation deployment design. Summary of the Invention

[0006] Therefore, it is necessary to provide a constellation design method, device and equipment based on evolutionary reinforcement learning and a memory to solve the above technical problems.

[0007] A constellation design method based on evolutionary reinforcement learning, the method comprising:

[0008] Converting the constellation design task problem into a multi-satellite earth observation task planning problem.

[0009] Modeling the multi-satellite earth observation task planning problem as a multi-objective combinatorial optimization problem; wherein the optimization objective of the multi-objective combinatorial optimization problem is to maximize the performance and benefit of the earth observation satellite under given resources and constraint conditions.

[0010] Solving the multi-objective combinatorial optimization problem by using a double-layer circulation model with a double-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm, and optimizing the constellation deployment design scheme; the double-layer circulation model is used to realize dynamic target observation task planning through an inner-layer circulation based on multi-agent reinforcement learning, and feed rewards and losses of the action network to an outer-layer circulation, and through the outer-layer circulation based on the multi-objective genetic algorithm, a set of satellite constellation parameters with maximum return, minimum loss and satisfying constraint conditions is found as a target, and the constellation deployment design scheme is optimized.

[0011] In one of the embodiments, the constraints of the multi-satellite earth observation task planning model include:

[0012]

[0013]

[0014]

[0015]

[0016]

[0017] wherein, the number of satellites is , the set of , the number of observation tasks is , the set of , the task and the task are continuously observed by the same satellite, , the observation time and the end time of the task , respectively, , the attitude conversion time between the task and the task , respectively, , the observation time and the end time of the task , respectively.The start time, end time, duration and observed pendulum angle of the visible time window, Indicates a task Subtasks In satellite No. Whether the time window is observed, Indicates a task In satellite No. Whether the time window is observed; Represents satellites Memory and power, Represents satellites The memory and power consumption per unit time of observation, 、 、 They are the number of satellites that continuously observe three target tasks, the number of satellites that observe one target task, and the number of satellites that are assigned all observation tasks.

[0018] In one embodiment, the optimization objectives of the multi-objective combination optimization problem include: maximizing mission benefits, minimizing attitude maneuvering angles, minimizing the coefficient of variation of satellite mission completion, and minimizing the sum of observation times for selected mission completion;

[0019] Among them, the expression for maximizing task benefits is:

[0020]

[0021] in, is the maximum value of the task benefit;

[0022] The expression for minimizing the attitude maneuver angle is:

[0023]

[0024] in, is the minimum value of the attitude maneuver angle, For the task The average pitch angle, For the task The average measured pendulum angle, is the number of all satellites assigned observation tasks;

[0025] The expression for minimizing the coefficient of variation of the satellite's mission completion is:

[0026]

[0027] in, is the minimum value of the coefficient of variation for the satellite to complete its mission, For satellite working hours, is the average working time of the satellite, The number of satellites for observing a target mission;

[0028] The expression for minimizing the sum of the observation time to complete the selected tasks is:

[0029]

[0030] in, is the minimum value of the sum of the observation time for the selected tasks to complete, which is 、 、 They are the number of satellites that continuously observe two target tasks, the number of satellites that observe one target task, and the number of satellites that are assigned all observation tasks. Represents tasks Observation time and end time, Indicates a task In satellite No. Whether a time window is observed.

[0031] In one embodiment, a two-layer recurrent model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm is used to solve a multi-objective combinatorial optimization problem and optimize the constellation deployment design, including:

[0032] Multiple agents are built in the inner layer; each agent includes an Actor network; all agents share a critic network.

[0033] A centralized training distributed execution framework is used to train the agent. During the centralized training process, the global situation information and the action information of all agents are used to train the agent. As the input of the Actor network, it outputs action strategies and reward values; all action strategies interact with the environment to obtain the global observation state; the global observation state and all action strategies are input into the critic network to obtain the value estimate; the network parameters are adjusted according to the reward value and value estimate, and the action strategy and corresponding reward are stored in the experience replay library, entering the next round of training until the preset conditions are met to obtain a trained intelligent agent.

[0034] Each trained agent is deployed on a satellite, and the local information observed by itself is input into the agent's Actor network to obtain the decision action of each satellite.

[0035] In the outer layer, a multi-objective optimization process based on a multi-objective genetic algorithm is used to evaluate the satellite's decision-making actions, and the constraints of the reward values ​​and target parameters are calculated. The constraints of the reward values ​​and target parameters are used as the basis for selection, crossover, and mutation operations. By optimizing the rewards and losses, the constellation orbit parameters are obtained and the constellation deployment design is realized.

[0036] In one embodiment, the Actor network includes a Transformer network, and the Critic network includes a Transformer network.

[0037] In one embodiment, the multi-objective genetic algorithm is a multi-objective genetic algorithm NSGA-II.

[0038] In one embodiment, the multi-agent deep reinforcement learning algorithm is the MAPPO algorithm of the CTDE architecture.

[0039] A constellation design device based on evolutionary reinforcement learning, comprising:

[0040] The constellation design mission modeling module is used to transform the constellation design mission problem into a multi-satellite Earth observation mission planning problem; the multi-satellite Earth observation mission planning problem is modeled as a multi-objective combinatorial optimization problem; the optimization goal of the multi-objective combinatorial optimization problem is to maximize the performance and benefits of the Earth observation satellite under given resources and constraints.

[0041] The constellation deployment design module is used to solve multi-objective combinatorial optimization problems using a two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm, thereby optimizing the constellation deployment design scheme. The two-layer loop model is used to implement task planning for dynamic target observation through an inner loop based on multi-agent reinforcement learning, and to feed back the rewards and losses of the action network to the outer loop. The outer loop based on the multi-objective genetic algorithm aims to find a set of satellite constellation parameters that maximizes the reward, minimizes the loss, and satisfies the constraints, thereby optimizing the constellation deployment design scheme.

[0042] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned constellation design method based on evolutionary reinforcement learning when executing the computer program.

[0043] A computer-readable memory having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned constellation design method based on evolutionary reinforcement learning.

[0044] The aforementioned constellation design method, apparatus, device, and memory based on evolutionary reinforcement learning include: transforming the constellation design task problem into a multi-satellite Earth observation mission planning problem; modeling the multi-satellite Earth observation mission planning problem as a multi-objective combinatorial optimization problem; and solving the multi-objective combinatorial optimization problem using a two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm to optimize the constellation deployment design. The two-layer loop model implements dynamic target observation mission planning through an inner loop based on multi-agent reinforcement learning, and feeds the rewards and losses of the action network into an outer loop. The outer loop, based on a multi-objective genetic algorithm, optimizes the constellation deployment design with the goal of finding a set of satellite constellation parameters that maximizes reward, minimizes loss, and satisfies constraints. This method utilizes a two-layer loop algorithm to implement dynamic target observation mission planning in the inner loop and optimize the constellation deployment design in the outer loop, thereby improving design efficiency and the ability to represent time window data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagrams of observation methods for non-agile Earth observation satellites and agile Earth observation satellites, where (a) is a schematic diagram of the observation method for agile satellites, and (b) is a schematic diagram of the observation method for non-agile Earth observation satellites;

[0046] Figure 2 1 is a flow chart of a constellation design method based on evolutionary reinforcement learning in one embodiment;

[0047] Figure 3 Schematic diagram of a double-layer loop model structure based on a double-layer architecture of a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm in one embodiment;

[0048] Figure 4 A schematic diagram of a centralized training-distributed execution structure in another embodiment;

[0049] Figure 5 A structural block diagram of a constellation design device based on evolutionary reinforcement learning in one embodiment;

[0050] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] Constellation deployment involves detailed research and planning of the layout and configuration of satellite constellations to meet specific communication, navigation, or observation requirements. During constellation deployment, multiple factors must be considered to develop an effective and adaptable constellation deployment plan. First, the satellite constellation's objectives and requirements must be clearly defined, including the type, intensity, and quality of services required. These services include communication, navigation, and observation. Second, during the problem analysis phase, the constellation's coverage and connectivity must be considered, including the number of satellites, satellite layout, orbital altitude, and resource management and utilization. A reasonable resource allocation strategy must be determined, taking into account constraints such as onboard storage space, to maximize the constellation's mission completion rate and avoid unnecessary resource waste. Finally, the cost and feasibility of constellation deployment must be considered. Manufacturing, launch, maintenance, and management costs are key considerations. A comprehensive consideration of all costs and performance factors is crucial to developing an economically viable constellation deployment plan that meets mission requirements.

[0053] In the process of Earth observation by satellite Earth observation systems, onboard sensors and sub-satellite point trajectories are used to determine the observation range. The following is an introduction to the basic concepts of satellite Earth observation:

[0054] Definition 1: Subsatellite point trajectory

[0055] Also known as a ground track or sub-satellite path, it is the projected path of a satellite onto Earth's surface, representing the satellite's path across the planet. The sub-satellite path is the visible trajectory of a satellite's position on Earth. It reflects how the satellite's orbit moves relative to the Earth's surface and is often used to describe the time and location at which a satellite can be observed from a specific location or area on Earth.

[0056] Definition 2: Ground Coverage

[0057] The visible range of a satellite or satellite constellation on the Earth's surface. It indicates the area or geographic location where a satellite can provide communications, navigation, Earth remote sensing, or other application services. Earth coverage is often used in satellite system design and planning to ensure that satellites can cover communications, navigation, and observation needs within a specific region or globally.

[0058] Definition 3: Earth Observation Strip

[0059] A narrow strip of Earth's surface imagery captured or observed by a remote sensing satellite as it passes over a specific area. This observation mode is a common method used in remote sensing technology, allowing a satellite to capture images or data each time it passes over a specific area, thereby covering the entire Earth's surface. The width of the strip is related to the satellite's flight altitude and field of view.

[0060] Definition 4: Satellite imaging side view and satellite access time window

[0061] For ground targets not on the sub-satellite trajectory, satellite observation requires adjusting the satellite platform or sensor attitude to observe them at a certain angle, which is called the satellite imaging side view. Generally, satellite observation of ground targets cannot exceed its maximum side view angle.

[0062] The set of times a satellite observes a target is calculated based on the satellite sub-satellite point trajectory and the target's geographical location on the ground. This is called the satellite's access time window to the target.

[0063] Definition 5: Earth Observation Planning Meta-Task

[0064] A satellite has multiple access time windows for a given observation target. Each access time window, along with the satellite and ground observation target, constitutes a ground observation planning meta-task. The three core elements of a meta-task are the satellite, the ground target, and the access time window. The set of planning meta-tasks serves as the input for satellite mission planning, and the planning algorithm determines which planning meta-tasks will be executed.

[0065] Definition 6: Multi-satellite Observation Conflict Area

[0066] It refers to areas where simultaneous observation or data collection by multiple satellites or observation platforms within specific time and space coordinates may lead to conflict or competition, and often requires coordination and supervision to ensure efficient use of resources and interoperability.

[0067] Definition 7: Conflict Observation Task

[0068] Earth observation missions that are in the conflict area of ​​multi-satellite observations are called conflict observation missions. For observations that only need to be observed once, multiple observations will waste precious satellite resources.

[0069] Definition 8: Observation method of agile earth observation satellite: Agile earth observation satellites and non-agile earth observation satellites are two different types of satellite systems. Agile satellites usually have high flexibility and maneuverability, and can quickly adjust tasks and execute strategies to adapt to different mission requirements and environmental changes. They are more suitable for flexible and changeable mission requirements, such as target tracking, data acquisition and rapid mission response. Their observation methods are as follows: Figure 1 As shown in (a) in the figure. Non-agile satellites are relatively lacking in flexibility and maneuverability, and have fixed mission modes and mission plans. They are more suitable for long-term stable missions and fixed data acquisition needs, such as scientific observation, navigation, and communication. Their observation methods are as follows: Figure 1 To better adapt to the flexible and changing mission requirements, this application uses agile earth observation satellites to implement operations.

[0070] In one embodiment, Figure 2As shown, a constellation design method based on evolutionary reinforcement learning is provided, which includes the following steps:

[0071] Step 100: Convert the constellation design task problem into a multi-satellite Earth observation mission planning problem.

[0072] Specifically, the Earth observation satellite mission planning problem is structured as follows: Based on a given observation objective, the objective is first pre-processed and adaptively adjusted. This involves classifying the objective based on the complexity and priority of the mission requirements, matching available satellites based on their capabilities and characteristics, and rationally adjusting the mission objectives. Secondly, based on the mission pre-processing, the observation capabilities of the available satellites are uniformly described and parameterized. This generates observation tasks that can be planned and scheduled. Finally, collaborative mission planning is performed to assign appropriate observation satellites to the generated observation tasks and customize the task sequence. Factors such as satellite availability, observation resource allocation, task priority, and timing must be comprehensively considered and decided.

[0073] Step 102: Model the multi-satellite Earth observation mission planning problem as a multi-objective combination optimization problem; wherein the optimization goal of the multi-objective combination optimization problem is to maximize the performance and benefits of the Earth observation satellite under given resources and constraints.

[0074] Specifically, the Earth observation satellite mission planning problem can be composed of two stages: mission planning and task scheduling. This application does not specifically study the mission planning problem, but focuses on the Earth observation satellite mission planning and scheduling model and algorithm. At present, traditional satellite scheduling methods only consider the total imaging benefit as a scheduling objective. However, due to the different observation objectives of the satellite constellation of the Earth observation satellite system, in order to meet various types of requirements and balance the interests of multiple parties, it is necessary to evaluate the scheduling plan based on multiple objectives under certain circumstances, and provide the optimal solution to assist decision-making. Therefore, the Earth observation satellite scheduling problem is defined as a multi-objective combinatorial optimization problem.

[0075] Step 104: A two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm is used to solve the multi-objective combinatorial optimization problem and optimize the constellation deployment design scheme. The two-layer loop model is used to implement task planning for dynamic target observation through an inner loop based on multi-agent reinforcement learning, and the reward and loss Loss of the action network are fed back to the outer loop. The outer loop based on the multi-objective genetic algorithm is used to find a set of satellite constellation parameters that maximizes the reward, minimizes the loss, and satisfies the constraints, thereby optimizing the constellation deployment design scheme.

[0076] Specifically, it aims to achieve mission planning based on dynamic target observation scenarios through the idea of ​​a double-layer loop. That is, the inner layer of the double-layer architecture interacts with the environment through a reinforcement learning algorithm, and realizes mission planning based on dynamic target observation scenarios. The outer layer uses a multi-objective evolutionary algorithm to achieve target mission planning and constellation morphology evolution based on the parameters learned in the inner loop.

[0077] The inner loop of the two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm introduces the Transformer network to improve the representation ability of time window data; the outer layer uses a distributed computing framework to improve the computational efficiency of the algorithm.

[0078] Taking the observed target as input, the performance of the MAPPO algorithm is improved through the Transformer network, and the network parameters are updated in real time during the reinforcement learning training process to obtain the Reward (reward for completing the task) and Loss (satellite repetition time window) of the Actor network. The goal of evolutionary computation is to find a set of satellite constellation parameters with maximum reward, minimum loss and satisfying constraints. In the optimization process of the upper-level multi-objective evolutionary algorithm, the candidate solutions are first evaluated, and the constraints of the reward value and the target parameters are calculated. These evaluation results are used to determine the fitness of the solution and serve as the basis for selection, crossover and mutation operations. Finally, the constellation orbit parameters are obtained by optimizing the Reward and Loss values, thereby realizing the constellation deployment design. The two-layer loop model with a two-layer architecture based on the multi-objective genetic algorithm and the multi-agent deep reinforcement learning algorithm is as follows. Figure 3 shown.

[0079] The aforementioned constellation design method based on evolutionary reinforcement learning includes: transforming the constellation design task problem into a multi-satellite Earth observation mission planning problem; modeling the multi-satellite Earth observation mission planning problem as a multi-objective combinatorial optimization problem; and solving the multi-objective combinatorial optimization problem using a two-layer loop model based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm to optimize the constellation deployment design. The two-layer loop model implements dynamic target observation mission planning through an inner loop based on multi-agent reinforcement learning, and feeds the rewards and losses of the action network into an outer loop. The outer loop based on the multi-objective genetic algorithm optimizes the constellation deployment design with the goal of finding a set of satellite constellation parameters that maximizes reward, minimizes loss, and satisfies constraints. This method utilizes a two-layer loop algorithm to implement dynamic target observation mission planning in the inner loop and optimize the constellation deployment design in the outer loop, thereby improving design efficiency and the ability to represent time window data.

[0080] In one embodiment, the constraints of the multi-satellite Earth observation mission planning problem in step 104 include:

[0081] (1)

[0082] (2)

[0083] (3)

[0084] (4)

[0085] (5)

[0086] wherein, denotes a set of satellites, denotes a set of observation tasks, denotes that task is observed by the same satellite as task , denotes the start time, end time, respectively, of task , denotes the attitude transition time between task and task , denotes the start time, end time, duration, and observation swing angle, respectively, of the visibility window of task , denotes whether subtask of task is observed in the th time window of satellite , subtask of task is observed in the th time window of satellite is 1, subtask of task is not observed in the th time window of satellite is 0, denotes whether task is observed in the th time window of satellite , task is observed in the th time window of satellite is 1, task is not observed in the th time window of satellite is 0; denotes a satellite Memory and power, Represents satellites The memory and power consumption per unit time of observation, 、 、 They are the number of satellites that continuously observe three target tasks, the number of satellites that observe one target task, and the number of satellites that are assigned all observation tasks.

[0087] Specifically, the problem of agile Earth observation satellite mission planning involves rationally planning and arranging satellite observation missions within the constraints of limited resources and time, maximizing the satellite's observation capabilities and mission effectiveness. Mission planning requires comprehensive consideration of factors such as satellite resource limitations, mission priorities, mission conflicts, and target accessibility, enabling the rational allocation of observation missions within limited resources and time constraints.

[0088] (1) Observation time window constraints

[0089] The observation start time is constrained to be selected within a certain range. The same target will have multiple observation windows to choose from within the scheduling cycle of the same satellite. The description of the constraint is shown in formula (1).

[0090] (2) Posture transition time constraint

[0091] The satellite attitude transition time depends on the satellite attitude between the end of the previous mission and the start of the current mission. The length of the transition time largely determines the quality of the solution. In satellite Earth observation missions, the attitude transition time of agile satellites is an important factor in determining the satellite's observation capability. The description of the attitude transition time constraint is shown in Formula (2).

[0092] (3) Onboard storage capacity constraints

[0093] Because the on-board resource space is limited, when the stored observation data reaches the storage capacity, the previous results will be overwritten by the next task. To avoid this situation, the formal description of the memory is shown in formula (3).

[0094] (4) Satellite energy constraints

[0095] To ensure the normal operation of the satellite, the energy on board must be within a certain range. The energy is controlled by factors such as the number of attitude transitions within a certain period of time and the total power-on time. The formal description of the energy constraint is shown in Formula (4).

[0096] (5) Task uniqueness constraint

[0097] A target can only be observed once at most. The formal description of the task uniqueness constraint is shown in formula (5).

[0098] For the design of earth observation satellite task planning, the following reasonable assumptions and simplifications are made for earth observation satellite task planning:

[0099] (1) There is an observation window for each observation task corresponding to the satellite, and the task order is random;

[0100] (2) The observation window intervals of different tasks may overlap;

[0101] (3) Each task has different observation time;

[0102] (4) Each task has different reward weights according to importance;

[0103] (5) Each satellite has a fixed storage capacity, and each execution of a task will have a capacity overhead;

[0104] (6) At any time, a satellite can only execute one task at a time, and once a certain observation task is executed, it cannot be preempted by other requests.

[0105] In one embodiment, the optimization objective of the multi-objective combinatorial optimization problem in step 102 includes: maximizing task revenue, minimizing attitude maneuver angle, minimizing coefficient of variation of satellite task completion, and minimizing the sum of observation time of selected tasks.

[0106] Among them, the task revenue of the satellite in the observation process depends on the revenue management, through the calculation of priority and importance, the task strategy is adjusted, and the expression of maximizing the task revenue is:

[0107] (6)

[0108] Among them, is the maximum value of the task revenue, and represents a set of observation targets with .

[0109] In the process of earth observation satellite task scheduling, the satellite attitude angle will directly affect the observation quality, excessive attitude adjustment will cause waste of satellite and orbit resources, and will affect the effective observation time, therefore, in the scheduling scheme, the angle of attitude maneuver is one of the evaluation methods for the pros and cons of the scheme. The expression of minimizing the attitude maneuver angle is:

[0110] (7)

[0111] (8)

[0112] (9)

[0113] Among them, is the minimum value of the attitude maneuver angle, For the task The average pitch angle, For the task Subtasks In satellite No. Whether the time window is observed, For the task The observation pitch angle of the observation time window, is the number of satellites that continuously observe three target missions, For the task The average measured pendulum angle, For the task The observed pendulum angle of the visible time window is is the number of all satellites assigned observation tasks; is the number of satellites that continuously observe two target missions.

[0114] If satellite load balancing is not considered during satellite planning and scheduling, it may result in uneven distribution of solutions, leading to reduced efficiency. In order to distribute many tasks to each satellite as evenly as possible, the coefficient of variation is used to represent the degree of balance of each solution. The expression for minimizing the coefficient of variation for satellites to complete tasks is:

[0115] (10)

[0116] (11)

[0117] (12)

[0118] in, is the minimum value of the coefficient of variation for the satellite to complete its mission, For satellite working hours, is the average working time of the satellite, The number of satellites that observe a target mission, For the task The duration of the visible time window; For the task Subtasks In satellite No. Whether the time window is observed, the task Subtasks In satellite No. time windows are observed is 1.

[0119] In an agile satellite constellation, missions will have different observation opportunities. However, within the same cycle, the difference between different observation opportunities can range from a few minutes to dozens of hours. Therefore, completing the observation mission as early as possible will greatly improve efficiency. The expression for minimizing the sum of the observation time of the selected missions is:

[0120] (13)

[0121] in, is the minimum value of the sum of the observation time for the selected tasks to complete, which is 、 、 They are the number of satellites that continuously observe two target tasks, the number of satellites that observe one target task, and the number of satellites that are assigned all observation tasks. Represents tasks Observation time and end time, Indicates a task In satellite No. Whether the time window is observed, the task In satellite No. time windows are observed 1, task In satellite No. Time windows are not observed is 0.

[0122] In one embodiment, step 104 includes: constructing multiple agents in the inner layer; each agent includes an Actor network; all agents share a critic network; using a centralized training distributed execution framework to train the agents, and using global situation information and action information of all agents in the centralized training process. As the input of the Actor network, it outputs action strategies and reward values; all action strategies interact with the environment to obtain a global observation state; the global observation state and all action strategies are input into the critic network to obtain a value estimate; the network parameters are adjusted according to the reward value and value estimate, and the action strategy and corresponding reward are stored in the experience replay library, entering the next round of training until the preset conditions are met to obtain a trained intelligent agent; each trained intelligent agent is deployed on a satellite, and the local information observed by itself is input into the Actor network of the intelligent agent to obtain the decision action of each satellite; in the outer multi-objective optimization process based on a multi-objective genetic algorithm, the satellite's decision action is evaluated, and the constraints of the reward value and target parameters are calculated. The constraints of the reward value and target parameters are used as the basis for selection, crossover and mutation operations. By optimizing the rewards and losses, the constellation orbit parameters are obtained and the constellation deployment design is realized.

[0123] Specifically, a centralized training and distributed execution architecture is adopted to carry out multi-agent reinforcement learning.

[0124] The "Centralized Training with DecentralizedExecution" (CTDE) training architecture is adopted. In this way, the agent can prevent the change of the strategy of a single agent from affecting the selection strategy of the rest of the agents globally. In the process of single-agent reinforcement learning, the action of the agent determines the state transition of the environment of the agent system, while for the multi-agent system, the actions of all agents determine the state transition of the multi-agent environment. In addition, in the multi-agent system, the rewards obtained by each agent in the learning process depend not only on its own choice of action, but also on the choices of other agents. In a multi-agent system, when the strategy of one of the agents is changed, the optimal selection strategy of other agents will also change. Choosing the CTDE architecture can avoid the occurrence of these problems between multiple agents. The centralized training-distributed execution structure is as follows. Figure 4 shown.

[0125] Centralized training refers to the use of joint action value functions to train the agent during the training process. Its input is the global situation information and all agents’ action information The advantage of this is that all agents can share a set of network parameters, and the strong coupling relationship between agents will be taken into account when making action choices. As a result, the generated reward functions are all related to the joint action, which can prevent the action strategy of a single agent from affecting the strategies of other agents, thus solving the problem of difficult algorithm convergence. "Distributed execution" means that since each individual agent can only receive local observed information during the execution process and cannot obtain global information, the observation information of each agent is used as input and the decision action made by the agent is used as output. By adopting this distributed decision-making method, the shortcomings of small action differentiation between different agents and weak model exploratory ability can be compensated.

[0126] The CTDE architecture adopted in this application uses a Critic network to learn a central value function that can observe global agent and environment information, and then through the Actor network, each agent calculates the strategy through its own local observation.

[0127] In one embodiment, the Actor network includes a Transformer network, and the Critic network includes a Transformer network.

[0128] Specifically, the inner layer of the double-layer loop model based on the multi-objective genetic algorithm and the multi-agent deep reinforcement learning algorithm uses the Actor-Critic network architecture to train the Transformer network. The training architecture is as follows: Figure 4 As shown, there are two networks involved in the training algorithm:

[0129] 1) Actor Network: Based on the input state, the Transformer network generates the probability for determining the next action, thereby selecting the optimal action strategy;

[0130] 2) Critic Network: It is a value function network that evaluates the value estimate of a given state through the Transformer network, thereby evaluating the reward of a specific state.

[0131] In one embodiment, the multi-objective genetic algorithm in step 104 is a multi-objective genetic algorithm NSGA-II.

[0132] In one embodiment, the multi-agent deep reinforcement learning algorithm in step 104 is a MAPPO algorithm of the CTDE architecture.

[0133] Specifically, the two-layer loop model, based on a two-layer architecture combining a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm, uses the MAPPO algorithm in the inner layer and the NSGA II algorithm in the outer layer. In the inner layer, the reward received by each agent (corresponding to a satellite) is related not only to its own behavior but also to that of other agents. The outer loop utilizes a distributed computing framework.

[0134] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0135] In one embodiment, Figure 5 As shown, a constellation design device based on evolutionary reinforcement learning is provided, including: a constellation design task modeling module and a constellation deployment design module, wherein:

[0136] The constellation design mission modeling module is used to transform the constellation design mission problem into a multi-satellite Earth observation mission planning problem; the multi-satellite Earth observation mission planning problem is modeled as a multi-objective combinatorial optimization problem; the optimization goal of the multi-objective combinatorial optimization problem is to maximize the performance and benefits of the Earth observation satellite under given resources and constraints.

[0137] The constellation deployment design module is used to solve multi-objective combinatorial optimization problems using a two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm, thereby optimizing the constellation deployment design scheme. The two-layer loop model is used to implement task planning for dynamic target observation through an inner loop based on multi-agent reinforcement learning, and to feed back the rewards and losses of the action network to the outer loop. The outer loop based on the multi-objective genetic algorithm aims to find a set of satellite constellation parameters that maximizes the reward, minimizes the loss, and satisfies the constraints, thereby optimizing the constellation deployment design scheme.

[0138] In one embodiment, the constraints of the multi-satellite Earth observation mission planning model are shown in formulas (1) to (5).

[0139] In one of the embodiments, the optimization objectives of the multi-objective combinatorial optimization problem include maximizing the task revenue, minimizing the attitude maneuver angle, minimizing the coefficient of variation of the satellite completing the task, and minimizing the sum of observation time of the selected tasks; wherein the expression of maximizing the task revenue is shown in formula (6); the expression of minimizing the attitude maneuver angle is shown in formula (7); the expression of minimizing the coefficient of variation of the satellite completing the task is shown in formula (10); and the expression of minimizing the sum of observation time of the selected tasks is shown in formula (14).

[0140] In one of the embodiments, the constellation deployment design module is further configured to construct a plurality of agents in the inner layer; wherein each agent includes an Actor network; all the agents share a critic network; the agents are trained by using a centralized training and distributed execution framework, and in the centralized training process, the global situation information and the action information of all the agents are used to adjust the network parameters. As the input of the Actor network, the output action strategy and the reward value are obtained; all the action strategies are interacted with the environment to obtain the global observation state; the global observation state and all the action strategies are input into the critic network to obtain the value estimation; the network parameters are adjusted according to the reward value and the value estimation, and the action strategy and the corresponding reward are stored in the experience replay library, and the next round of training is entered until the preset condition is met, and the trained agent is obtained; each trained agent is deployed on a satellite, and the local information observed by itself is input into the Actor network of the agent to obtain the decision action of each satellite; in the multi-objective optimization process based on the multi-objective genetic algorithm in the outer layer, the decision action of the satellite is evaluated, and the reward value and the constraint of the target parameter are calculated, the reward value and the constraint of the target parameter are used as the basis of the selection, crossover and mutation operations, and the constellation orbit parameters are obtained by optimizing the reward and loss, thereby realizing the constellation deployment design.

[0141] In one of the embodiments, the Actor network in the constellation deployment design module includes a Transformer network, and the Critic network includes a Transformer network.

[0142] In one of the embodiments, the multi-objective genetic algorithm in the constellation deployment design module is a multi-objective genetic algorithm NSGA-II.

[0143] In one of the embodiments, the multi-agent deep reinforcement learning algorithm in the constellation deployment design module is a MAPPO algorithm based on the CTDE architecture.

[0144] The specific definitions of the constellation design device based on evolutionary reinforcement learning can be found in the definitions of the constellation design method based on evolutionary reinforcement learning above and will not be repeated here. Each module in the aforementioned constellation design device based on evolutionary reinforcement learning can be implemented in whole or in part via software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.

[0145] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a constellation design method based on evolutionary reinforcement learning is implemented. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0146] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0147] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0148] In one embodiment, a computer readable memory is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0149] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable memory. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0150] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A constellation design method based on evolutionary reinforcement learning, characterized in that: The method comprises: Convert the constellation design task problem into a multi-satellite Earth observation mission planning problem; The multi-satellite Earth observation mission planning problem is modeled as a multi-objective combinatorial optimization problem. The optimization goal of the multi-objective combinatorial optimization problem is to maximize the performance and benefits of the Earth observation satellite under given resources and constraints. A two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm is used to solve the multi-objective combinatorial optimization problem and optimize the constellation deployment design scheme. The two-layer loop model is used to implement task planning for dynamic target observation through an inner loop based on multi-agent reinforcement learning, and the reward and loss of the action network are fed back to the outer loop. The outer loop based on the multi-objective genetic algorithm aims to find a set of satellite constellation parameters that maximizes the reward, minimizes the loss, and satisfies the constraints, thereby optimizing the constellation deployment design scheme.

2. The constellation design method based on evolutionary reinforcement learning according to claim 1, characterized in that: The constraints of the multi-satellite Earth observation mission planning model include: in, The number of expressions is A collection of satellites, Indicates the quantity A collection of observation tasks, Indicates a task and tasks Continuously observed by the same satellite, Represents tasks Observation time and end time, , indicating the task With the task The attitude transition time between Represents tasks The start time, end time, duration and observed pendulum angle of the visible time window, Indicates a task Subtasks In satellite No. Whether the time window is observed, Indicates a task In satellite No. Whether the time window is observed, Represents satellites Memory and power, Represents satellites The memory and power consumption per unit time of observation, 、 、 、 They are the number of satellites that continuously observe two target tasks, the number of satellites that continuously observe three target tasks, the number of satellites that observe one target task, and the number of all satellites assigned observation tasks.

3. The constellation design method based on evolutionary reinforcement learning according to claim 1, characterized in that: The optimization objectives of the multi-objective combination optimization problem include: maximizing mission benefits, minimizing attitude maneuvering angles, minimizing the coefficient of variation of satellite mission completion, and minimizing the sum of observation times for selected mission completion. Among them, the expression for maximizing task benefits is: in, is the maximum value of the task benefit; The expression for minimizing the attitude maneuver angle is: in, is the minimum value of the attitude maneuver angle, For the task The average pitch angle, For the task The average measured pendulum angle, is the number of all satellites assigned observation tasks; The expression for minimizing the coefficient of variation of the satellite's mission completion is: in, is the minimum value of the coefficient of variation for the satellite to complete its mission, For satellite working hours, is the average working time of the satellite, The number of satellites for observing a target mission; The expression for minimizing the sum of the observation time to complete the selected tasks is: in, is the minimum value of the sum of the observation time for the selected tasks to complete, which is 、 、 They are the number of satellites that continuously observe two target tasks, the number of satellites that observe one target task, and the number of satellites that are assigned all observation tasks. Represents tasks Observation time and end time, Indicates a task In satellite No. Whether a time window is observed.

4. The constellation design method based on evolutionary reinforcement learning according to claim 1, characterized in that: A two-layer recurrent model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm is used to solve the multi-objective combinatorial optimization problem and optimize the constellation deployment design, including: Build multiple agents in the inner layer; each agent includes an Actor network; all agents share a critic network; A centralized training distributed execution framework is used to train the agent. During the centralized training process, the global situation information and the action information of all agents are used to train the agent. As input to the Actor network, it outputs action strategies and reward values; all of the action strategies interact with the environment to obtain a global observation state; the global observation state and all of the action strategies are input into the critic network to obtain a value estimate; the network parameters are adjusted based on the reward value and value estimate, and the action strategies and corresponding rewards are stored in the experience replay library, entering the next round of training until the preset conditions are met, resulting in a trained agent; Each trained agent is deployed on a satellite, and its observed local information is input into the agent's Actor network to obtain the decision action of each satellite; In the outer layer, a multi-objective optimization process based on a multi-objective genetic algorithm is used to evaluate the satellite's decision-making actions, and the constraints of the reward values ​​and target parameters are calculated. The constraints of the reward values ​​and target parameters are used as the basis for selection, crossover, and mutation operations. By optimizing the rewards and losses, the constellation orbit parameters are obtained and the constellation deployment design is realized.

5. The constellation design method based on evolutionary reinforcement learning according to claim 4, characterized in that: The Actor network includes the Transformer network, and the Critic network includes the Transformer network.

6. The constellation design method based on evolutionary reinforcement learning according to claim 1, characterized in that: The multi-objective genetic algorithm is the multi-objective genetic algorithm NSGA-II.

7. The constellation design method based on evolutionary reinforcement learning according to claim 1, characterized in that: The multi-agent deep reinforcement learning algorithm is the MAPPO algorithm of the CTDE architecture.

8. A constellation design device based on evolutionary reinforcement learning, characterized in that: The device comprises: The constellation design mission modeling module is used to transform the constellation design mission problem into a multi-satellite Earth observation mission planning problem. The multi-satellite Earth observation mission planning problem is modeled as a multi-objective combinatorial optimization problem. The optimization goal of the multi-objective combinatorial optimization problem is to maximize the performance and benefits of Earth observation satellites under given resources and constraints. The constellation deployment design module is used to solve the multi-objective combinatorial optimization problem using a two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm, thereby optimizing the constellation deployment design scheme. The two-layer loop model is used to implement task planning for dynamic target observation through an inner loop based on multi-agent reinforcement learning, and to feed back the rewards and losses of the action network to the outer loop. The outer loop based on the multi-objective genetic algorithm is used to optimize the constellation deployment design scheme with the goal of finding a set of satellite constellation parameters that maximizes the reward, minimizes the loss, and satisfies the constraints.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the constellation design method based on evolutionary reinforcement learning according to any one of claims 1 to 7 is implemented.

10. A computer readable memory having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the constellation design method based on evolutionary reinforcement learning according to any one of claims 1 to 7 is implemented.

Images