Constellation design method and device based on evolutionary reinforcement learning, equipment and memory

Abstract

The invention relates to a constellation design method and device based on evolutionary reinforcement learning, equipment and a memory. The method comprises the following steps: converting a constellation design task problem into a multi-satellite earth observation task planning problem; modeling a multi-satellite earth observation task planning problem into a multi-target combinatorial optimization problem; a double-layer circulation model of a double-layer architecture based on a multi-target genetic algorithm and a multi-agent deep reinforcement learning algorithm is adopted to solve a multi-target combinatorial optimization problem, and a constellation deployment design scheme is optimized; the double-layer circulation model is used for realizing task planning of dynamic target observation through inner-layer circulation based on multi-agent reinforcement learning and feeding back awards and loss of a mobile network to outer-layer circulation; through outer layer circulation based on a multi-target genetic algorithm, a group of satellite constellation parameters which are maximum in return and minimum in loss and meet constraint conditions are searched as a target, and a constellation deployment design scheme is optimized. According to the method, the scheme design efficiency and the representation capability of the time window data are improved.

Description

Technical Field

[0001] The present application relates to the technical field of constellation design, and in particular to a constellation design method, apparatus, device and memory based on evolutionary reinforcement learning. Background Art

[0002] With the rapid development of satellite communication technology, low-orbit Internet constellations represented by Starlink and Kuiper can achieve seamless global coverage. Such constellations can not only observe ground areas, but also be transformed into celestial observation constellations. Specifically, transforming such constellations into celestial observation constellations can achieve the real-time nature of astronomical observation and the coverage advantages of a large area of ​​the celestial sphere, which may give birth to new economic and industrial forms in the field of astronomical observation, such as popular astronomical entertainment, consumption, and education industries.

[0003] Constellation deployment design refers to the process of planning, designing and configuring a group of satellites to meet specific mission and coverage requirements. This process involves deciding on key parameters such as the number of satellites, orbital parameters, relative positions, communication frequencies, etc. to ensure that the satellite constellation can effectively perform its mission.

[0004] The design of satellite constellation deployment involves multiple core links, such as preparation for launch, selection of launch time, actual launch, orbit entry, data collection, processing and transmission. The main purpose of constellation deployment design is to optimize resource allocation and workflow to quickly establish a constellation network and meet the needs of specific goals. The responsiveness of the constellation is affected by the constellation layout and satellite orbit parameters. In order to meet the mission requirements, it is usually necessary to determine the constellation layout and orbit parameters first. To this end, the constellation design strategy mainly focuses on optimizing these parameters and conducting quantitative analysis and problem solving in combination with the specific requirements of the mission. Since different missions have different requirements for deployment, constellations must be optimized and decided according to specific scenarios. At present, in order to achieve multi-satellite networking, rapid on-orbit response and continuous coverage, a decision-making framework applicable to various scenarios is still needed to assist decision makers in solving problems subject to multiple restrictions and conflicting constraints and finding appropriate solutions.

[0005] At present, most of the solutions to constellation deployment design problems use evolutionary computing, which uses a single-layer iterative optimization method to solve orbital parameters based on regional coverage performance to achieve constellation design. In scenarios with dynamic target observation as a typical application, it is impossible to implement constellation design on the basis of completing mission intelligent planning, thereby achieving higher efficiency and less resource consumption. To solve such problems, this paper proposes a two-layer framework for satellite constellation morphology evolution. Through a two-layer loop algorithm, the inner loop realizes the mission planning of dynamic target observation, and the outer loop optimizes the constellation deployment design scheme. Summary of the invention

[0006] Based on this, it is necessary to provide a constellation design method, device, equipment and memory based on evolutionary reinforcement learning to address the above technical problems.

[0007] A constellation design method based on evolutionary reinforcement learning, the method comprising: The constellation design task problem is transformed into a multi-satellite Earth observation mission planning problem.

[0008] The multi-satellite Earth observation mission planning problem is modeled as a multi-objective combinatorial optimization problem; the optimization goal of the multi-objective combinatorial optimization problem is to maximize the performance and benefits of the Earth observation satellite under given resources and constraints.

[0009] A two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm is used to solve the multi-objective combinatorial optimization problem and optimize the constellation deployment design scheme. The two-layer loop model is used to realize the task planning of dynamic target observation through an inner loop based on multi-agent reinforcement learning, and the reward and loss Loss of the action network are fed back to the outer loop. The outer loop based on the multi-objective genetic algorithm aims to find a set of satellite constellation parameters with the maximum reward, the minimum loss and satisfying the constraints, so as to optimize the constellation deployment design scheme.

[0010] In one embodiment, the constraints of the multi-satellite earth observation mission planning model include:

[0011]

[0012]

[0013]

[0014]

[0015] in, The number of expressions is A collection of satellites, Indicates the quantity A collection of observation tasks, Representation Task and tasks Continuously observed by the same satellite, Respectively represent tasks The observation time and end time of , indicating the task With the task The attitude transition time between Respectively represent tasks The start time, end time, duration and observed pendulum angle of the visible time window, Representation Task Subtasks In satellite No. Whether the time window is observed, Representation Task In satellite No. Whether a time window is observed; Satellite Memory and power, Satellite The memory and power consumption per unit time of observation, , , They are the number of satellites continuously observing three target tasks, the number of satellites observing one target task, and the number of satellites assigned all observation tasks.

[0016] In one embodiment, the optimization objectives of the multi-objective combined optimization problem include: maximizing mission benefits, minimizing attitude maneuvering angles, minimizing the coefficient of variation of satellite mission completion, and minimizing the sum of observation times for selected mission completion; Among them, the expression for maximizing task benefits is:

[0017] in, is the maximum value of the task benefit; The expression for minimizing the attitude maneuver angle is:

[0018] in, is the minimum value of the attitude maneuvering angle, For the task The average pitch angle, For the task The average measured swing angle, is the number of all satellites assigned observation tasks; The expression for minimizing the coefficient of variation of the satellite's mission completion is:

[0019] in, is the minimum value of the coefficient of variation for the satellite to complete its mission, For satellite Working hours, is the average working time of the satellite, The number of satellites for observing a target mission; The expression for minimizing the sum of the observation time to complete the selected tasks is:

[0020] in, is the minimum value of the sum of observation time for the selected tasks to complete, , , They are the number of satellites that continuously observe two target tasks, the number of satellites that observe one target task, and the number of satellites that are assigned observation tasks. Respectively represent tasks The observation time and end time of Representation Task In satellite No. Whether a time window is observed.

[0021] In one embodiment, a two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm is used to solve a multi-objective combinatorial optimization problem and optimize the constellation deployment design scheme, including: Multiple agents are built in the inner layer; each agent includes an Actor network; all agents share a critic network.

[0022] The centralized training distributed execution framework is used to train the intelligent agent. During the centralized training process, the global situation information and the action information of all intelligent agents are used. As the input of the Actor network, it outputs action strategies and reward values; all action strategies interact with the environment to obtain the global observation state; the global observation state and all action strategies are input into the critic network to obtain the value estimate; the network parameters are adjusted according to the reward value and value estimate, and the action strategy and the corresponding reward are stored in the experience replay library to enter the next round of training until the preset conditions are met to obtain a trained intelligent agent.

[0023] Each trained agent is deployed on a satellite, and the local information observed by itself is input into the agent's Actor network to obtain the decision action of each satellite.

[0024] In the outer multi-objective optimization process based on the multi-objective genetic algorithm, the decision actions of the satellite are evaluated, and the constraints of the reward values ​​and target parameters are calculated. The constraints of the reward values ​​and target parameters are used as the basis for selection, crossover and mutation operations. The constellation orbit parameters are obtained by optimizing the rewards and losses to realize the constellation deployment design.

[0025] In one embodiment, the Actor network includes a Transformer network, and the Critic network includes a Transformer network.

[0026] In one embodiment, the multi-objective genetic algorithm is a multi-objective genetic algorithm NSGA-II.

[0027] In one embodiment, the multi-agent deep reinforcement learning algorithm is a MAPPO algorithm based on the CTDE architecture.

[0028] A constellation design device based on evolutionary reinforcement learning, the device comprising: The constellation design task modeling module is used to transform the constellation design task problem into a multi-satellite Earth observation mission planning problem; the multi-satellite Earth observation mission planning problem is modeled as a multi-objective combinatorial optimization problem; the optimization goal of the multi-objective combinatorial optimization problem is to maximize the performance and benefits of the Earth observation satellite under given resources and constraints.

[0029] The constellation deployment design module is used to solve the multi-objective combinatorial optimization problem by adopting a two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm, and optimize the constellation deployment design scheme; the two-layer loop model is used to realize the task planning of dynamic target observation through an inner loop based on multi-agent reinforcement learning, and feed back the reward and loss Loss of the action network to the outer loop. Through the outer loop based on the multi-objective genetic algorithm, the goal is to find a set of satellite constellation parameters with the maximum reward, the minimum loss and satisfying the constraints, so as to optimize the constellation deployment design scheme.

[0030] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned constellation design method based on evolutionary reinforcement learning when executing the computer program.

[0031] A computer-readable memory having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned constellation design method based on evolutionary reinforcement learning.

[0032] The above-mentioned constellation design method, device, equipment and memory based on evolutionary reinforcement learning, the method includes: converting the constellation design task problem into a multi-satellite earth observation task planning problem; modeling the multi-satellite earth observation task planning problem as a multi-objective combination optimization problem; using a double-layer loop model with a double-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm to solve the multi-objective combination optimization problem and optimize the constellation deployment design scheme; the double-layer loop model is used to realize the task planning of dynamic target observation through an inner loop based on multi-agent reinforcement learning, and the reward and loss Loss of the action network are fed back to the outer loop, and the outer loop based on the multi-objective genetic algorithm is used to find a set of satellite constellation parameters with the maximum reward, the minimum loss and satisfying the constraint conditions as the goal, and optimize the constellation deployment design scheme. This method realizes the task planning of dynamic target observation in the inner loop through the double-layer loop algorithm, and optimizes the constellation deployment design scheme in the outer loop, thereby improving the scheme design efficiency and the representation ability of time window data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagrams of observation methods for non-agile earth observation satellites and agile earth observation satellites, wherein (a) is a schematic diagram of the observation method for an agile satellite, and (b) is a schematic diagram of the observation method for a non-agile earth observation satellite; Figure 2 is a flowchart of a constellation design method based on evolutionary reinforcement learning in one embodiment; Figure 3 A schematic diagram of a double-layer loop model structure of a double-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm in one embodiment; Figure 4 A schematic diagram of a centralized training-distributed execution structure in another embodiment; Figure 5 is a structural block diagram of a constellation design device based on evolutionary reinforcement learning in one embodiment; Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] Constellation deployment refers to the detailed study and planning of the layout and configuration of satellite constellations to meet specific communication, navigation or observation needs. In the process of constellation deployment, many factors need to be considered to develop an effective and adaptable constellation deployment plan. First, it is necessary to clarify the goals and needs of the satellite constellation, including the types of services, service intensity and quality requirements provided by the required constellation, where the service types include communication, navigation and observation; secondly, in the problem analysis stage, the coverage and connectivity of the constellation should be considered, including the number of satellites, satellite layout, orbital altitude, resource management and utilization, etc., and a reasonable resource allocation strategy should be determined, taking into account limiting factors such as on-board storage space, so as to maximize the constellation's mission completion rate and avoid unnecessary waste of resources; finally, the cost and feasibility analysis of constellation deployment should also be considered, among which the costs of manufacturing, launch, maintenance and management are important considerations. It is necessary to comprehensively consider the costs and performance of various aspects to generate an economically feasible constellation deployment plan that meets mission requirements.

[0036] In the process of earth observation by satellite earth observation system, satellite-borne sensors and sub-satellite point trajectories are needed to determine the observation range. Next, the basic concepts of satellite earth observation process will be defined and introduced: Definition 1: Subsatellite point trajectory Also called ground track or subsatellite path, it is the projection of a satellite on the Earth's surface, representing the path of the satellite's movement on the Earth. Subsatellite track is the visible position track of a satellite on Earth, which reflects how the satellite's orbit moves relative to the Earth's surface. It is often used to describe the time and position at which a satellite can be observed from a specific location or area on Earth.

[0037] Definition 2: Ground Coverage Refers to the visible range of a satellite or satellite constellation on the Earth's surface. It represents the area or geographic location where the satellite's communications, navigation, Earth remote sensing or other application services can be provided. Earth coverage is often used in satellite system design and planning to ensure that satellites can cover communications, navigation and observation needs in a specific area or globally.

[0038] Definition 3: Earth Observation Strip Refers to the narrow strip of surface images taken or observed by a remote sensing satellite when it passes over a specific area on the earth. This observation mode is a commonly used method in remote sensing technology. It allows the satellite to capture images or data every time it flies over a specific area, thereby covering the entire surface of the earth. Its width is related to the satellite's flight altitude and field of view.

[0039] Definition 4: Satellite imaging side view and satellite access time window For ground targets that are not on the sub-satellite track, the satellite needs to adjust the satellite platform or sensor attitude to observe them at a certain angle, which is called the satellite imaging side angle. Generally, the satellite's observation of ground targets cannot exceed its maximum side angle.

[0040] The set of times when a satellite observes a target is calculated through the satellite sub-satellite trajectory and the target's geographical location on the ground, which is called the satellite's visit time window to the target.

[0041] Definition 5: Earth Observation Planning Meta-Task The satellite has multiple access time windows for a certain observation target. Each access time window, together with the satellite and the ground observation target, constitutes a ground observation planning meta-task. The three core elements of the meta-task are: satellite, ground target and access time window. The planning meta-task set is the input of satellite mission planning, and the planning algorithm determines which planning meta-tasks will be executed.

[0042] Definition 6: Multi-satellite Observation Conflict Area It refers to areas where simultaneous observation or data collection by multiple satellites or observation platforms within specific time and space coordinates may lead to conflict or competition, and often requires coordination and supervision to ensure efficient use of resources and interoperability.

[0043] Definition 7: Conflict Observation Task Earth observation missions that are in the multi-satellite observation conflict area are called conflict observation missions. For observation requirements that only need to be observed once, multiple observations will waste precious satellite resources.

[0044] Definition 8: Observation methods of agile earth observation satellites: Agile earth observation satellites and non-agile earth observation satellites are two different types of satellite systems. Agile satellites usually have high flexibility and mobility, and can quickly adjust tasks and execute strategies to adapt to different mission requirements and environmental changes. They are more suitable for flexible and changeable mission requirements, such as target tracking, data acquisition and rapid mission response. Their observation methods are as follows: Figure 1 As shown in (a) in the figure, non-agile satellites are relatively lacking in flexibility and maneuverability, have fixed mission modes and mission plans, and are more suitable for long-term stable missions and fixed data acquisition needs, such as scientific observation, navigation, and communication. Their observation methods are as follows: Figure 1 In order to better adapt to the flexible and changing mission requirements, this application uses agile earth observation satellites to implement operations.

[0045] In one embodiment, Figure 2 As shown, a constellation design method based on evolutionary reinforcement learning is provided, and the method includes the following steps: Step 100: Convert the constellation design task problem into a multi-satellite Earth observation mission planning problem.

[0046] Specifically, the structure of the Earth observation satellite mission planning problem is as follows: Based on the given observation targets, the targets are first preprocessed and adaptively adjusted, including classification according to the complexity and priority of the mission requirements, matching the available satellites according to the capabilities and characteristics of the satellites, and rationally adjusting the mission targets. Secondly, based on the mission preprocessing, the observation capabilities of the called satellites are uniformly described and parameterized. This forms observation tasks that can be planned and scheduled. Finally, collaborative mission planning is carried out to assign appropriate observation satellites to the generated observation tasks and customize the task sequence. It is necessary to comprehensively consider and make decisions on factors such as satellite availability, observation resource allocation, task priority and timing.

[0047] Step 102: Model the multi-satellite earth observation mission planning problem as a multi-objective combinatorial optimization problem; wherein the optimization goal of the multi-objective combinatorial optimization problem is to maximize the performance and benefits of the earth observation satellite under given resources and constraints.

[0048] Specifically, the mission planning problem of earth observation satellites can be composed of two stages: mission planning and task scheduling. This application does not specifically study the mission planning problem, but focuses on the mission planning and scheduling model and algorithm for earth observation satellites. At present, traditional satellite scheduling methods only consider the total imaging benefit as a scheduling goal. However, due to the different observation targets of the satellite constellation of the earth observation satellite system, in order to meet various types of requirements and weigh the interests of multiple parties, it is necessary to evaluate the scheduling plan according to multiple objectives under certain circumstances, and give the optimal solution to assist decision-making. Therefore, the earth observation satellite scheduling problem is defined as a multi-objective combinatorial optimization problem.

[0049] Step 104: A two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm is used to solve the multi-objective combinatorial optimization problem and optimize the constellation deployment design scheme; the two-layer loop model is used to realize the task planning of dynamic target observation through an inner loop based on multi-agent reinforcement learning, and feed back the reward and loss Loss of the action network to the outer loop, and optimize the constellation deployment design scheme through an outer loop based on a multi-objective genetic algorithm with the goal of finding a set of satellite constellation parameters with maximum reward, minimum loss and satisfying constraints.

[0050] Specifically, it aims to achieve task planning based on dynamic target observation scenarios through the idea of ​​a double-layer loop, that is, the inner layer of the double-layer architecture interacts with the environment through a reinforcement learning algorithm, and the outer layer uses a multi-objective evolutionary algorithm to achieve target task planning and constellation morphology evolution based on the parameters learned in the inner loop.

[0051] The inner loop of the two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm introduces the Transformer network to improve the representation ability of time window data; the outer layer uses a distributed computing framework to improve the computational efficiency of the algorithm.

[0052] Taking the observed target as input, the performance of the MAPPO algorithm is improved through the Transformer network. The network parameters are updated in real time during the reinforcement learning training process, and the Reward (reward for completing the task) and Loss (satellite repetition time window) of the Actor network are obtained. The goal of evolutionary computation is to find a set of satellite constellation parameters with maximum reward, minimum loss and satisfying constraints. In the optimization process of the upper-level multi-objective evolutionary algorithm, the candidate solutions are first evaluated, and the constraints of the reward value and the target parameters are calculated. These evaluation results are used to determine the fitness of the solution and serve as the basis for selection, crossover and mutation operations. Finally, the constellation orbit parameters are obtained by optimizing the Reward and Loss values, thereby realizing the constellation deployment design. The two-layer loop model with a two-layer architecture based on the multi-objective genetic algorithm and the multi-agent deep reinforcement learning algorithm is as follows. Figure 3 shown.

[0053] In the above-mentioned constellation design method based on evolutionary reinforcement learning, the method includes: converting the constellation design task problem into a multi-satellite earth observation task planning problem; modeling the multi-satellite earth observation task planning problem as a multi-objective combination optimization problem; using a double-layer loop model with a double-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm to solve the multi-objective combination optimization problem and optimize the constellation deployment design scheme; the double-layer loop model is used to realize the task planning of dynamic target observation through an inner loop based on multi-agent reinforcement learning, and the reward and loss Loss of the action network are fed back to the outer loop, and the outer loop based on the multi-objective genetic algorithm is used to find a set of satellite constellation parameters with the maximum reward, the minimum loss and satisfying the constraints as the goal, and optimize the constellation deployment design scheme. This method realizes the task planning of dynamic target observation in the inner loop through the double-layer loop algorithm, and optimizes the constellation deployment design scheme in the outer loop, thereby improving the scheme design efficiency and the representation ability of time window data.

[0054] In one embodiment, the constraints of the multi-satellite earth observation mission planning problem in step 104 include: (1) (2) (3) (4) (5) in, The number of expressions is A collection of satellites, Indicates the quantity A collection of observation tasks, Representation Task and tasks Continuously observed by the same satellite, Respectively represent tasks The observation time and end time of , indicating the task With the task The attitude transition time between Respectively represent tasks The start time, end time, duration and observed pendulum angle of the visible time window, Representation Task Subtasks In satellite No. Whether the time window is observed, the task Subtasks In satellite No. time windows are observed 1, task Subtasks In satellite No. Time windows not observed is 0, Representation Task In satellite No. Whether the time window is observed, the task In satellite No. time windows are observed 1, task In satellite No. Time windows not observed is 0; Satellite Memory and power, Satellite The memory and power consumption per unit time of observation, , , They are the number of satellites continuously observing three target tasks, the number of satellites observing one target task, and the number of satellites assigned all observation tasks.

[0055] Specifically, the agile Earth observation satellite mission planning problem refers to the reasonable planning and arrangement of satellite observation missions under the constraints of limited resources and time, so as to maximize the satellite's observation capabilities and mission benefits. When planning missions, factors such as satellite resource limitations, mission priorities, mission conflicts, and observation target accessibility should be comprehensively considered, so as to reasonably arrange observation tasks under limited resources and time conditions.

[0056] (1) Observation time window constraints The observation start time is constrained to be selected within a certain range, and the same target will have multiple observation windows to choose from within the scheduling cycle of the same satellite. The description of the constraint is shown in formula (1).

[0057] (2) Posture transition time constraint The satellite attitude conversion time depends on the satellite attitude at the end of the previous mission and the start of the current mission. The conversion time largely determines the quality of the scheme. In the satellite earth observation mission, the attitude conversion time of agile satellites is an important factor that determines the satellite observation capability. The description of the attitude conversion time constraint is shown in formula (2).

[0058] (3) Onboard storage capacity constraints Because the on-board resource space is limited, when the stored observation data reaches the storage capacity, the previous results will be overwritten by the next task. To avoid this situation, the formal description of memory is shown in formula (3).

[0059] (4) Satellite energy constraints To ensure the normal operation of the satellite, the energy on the satellite must have a corresponding applicable range. Its energy is controlled by factors such as the number of attitude conversions within a certain period of time and the total power-on time. The formal description of the energy constraint is shown in formula (4).

[0060] (5) Task uniqueness constraint A target can be observed at most once. The formal description of the task uniqueness constraint is shown in formula (5).

[0061] In order to design the Earth observation satellite mission planning, the following reasonable assumptions and simplifications are made for the Earth observation satellite mission planning: (1) Each observation task corresponds to a satellite with an observation window, and the order of tasks is random; (2) The observation windows of different tasks may overlap with each other; (3) The observation time for each task is different; (4) Each task has different reward weights according to its importance; (5) Each satellite has a fixed storage capacity, and each mission will incur capacity overhead; (6) At any time, a satellite can only perform one mission at the same time, and once it is performing an observation mission, it cannot be preempted by other requests.

[0062] In one embodiment, the optimization objectives of the multi-objective combined optimization problem in step 102 include: maximizing mission benefits, minimizing attitude maneuvering angles, minimizing the coefficient of variation of satellite mission completion, and minimizing the sum of observation times for selected mission completion.

[0063] Among them, the mission benefit of the satellite during the observation process depends on the benefit management. By calculating the priority and importance, the mission strategy is adjusted, and the expression for maximizing the mission benefit is: (6) in, is the maximum value of the task benefit, indicating the quantity is The set of observation targets.

[0064] In the process of scheduling Earth observation satellite missions, the satellite attitude angle will directly affect the observation quality. Excessive attitude adjustment will cause waste of satellite and orbital resources and affect the effective observation time. Therefore, the attitude maneuver angle is used as one of the evaluation methods for the scheduling scheme. The expression for minimizing the attitude maneuver angle is: (7) (8) (9) in, is the minimum value of the attitude maneuvering angle, For the task The average pitch angle, For the task Subtasks In satellite No. Whether the time window is observed, For the task The observation pitch angle of the observation time window, The number of satellites that continuously observe the three target missions, For the task The average measured swing angle, For the task The observed pendulum angle in the visible time window is is the number of all satellites assigned observation tasks; is the number of satellites that continuously observe two target missions.

[0065] If the load balance of satellites is not considered during satellite planning and scheduling, it may cause uneven distribution of solutions, resulting in reduced efficiency. In order to distribute many tasks to each satellite as evenly as possible, the coefficient of variation is used to represent the degree of balance of each solution. The expression for minimizing the coefficient of variation of satellites completing tasks is: (10) (11) (12) in, is the minimum value of the coefficient of variation for the satellite to complete its mission, For satellite Working hours, is the average working time of the satellite, The number of satellites that observe a target mission, For the task The duration of the visible time window; For the task Subtasks In satellite No. Whether the time window is observed, the task Subtasks In satellite No. time windows are observed is 1.

[0066] In an agile satellite constellation, tasks will have different observation opportunities, but in the same cycle, the difference between different observation opportunities can range from a few minutes to dozens of hours, so completing the observation task as early as possible will greatly improve efficiency. The expression that minimizes the sum of the observation time of the selected tasks is: (13) in, is the minimum value of the sum of observation time for the selected tasks to complete, , , They are the number of satellites that continuously observe two target tasks, the number of satellites that observe one target task, and the number of satellites that are assigned observation tasks. Respectively represent tasks The observation time and end time of Representation Task In satellite No. Whether the time window is observed, the task In satellite No. time windows are observed 1, task In satellite No. Time windows not observed is 0.

[0067] In one embodiment, step 104 includes: constructing multiple agents in the inner layer; each agent includes an Actor network; all agents share a critic network; using a centralized training distributed execution framework to train the agents, using global situation information and action information of all agents in the centralized training process As the input of the Actor network, the action strategy and reward value are output; all action strategies interact with the environment to obtain the global observation state; the global observation state and all action strategies are input into the critic network to obtain the value estimate; the network parameters are adjusted according to the reward value and the value estimate, and the action strategy and the corresponding reward are stored in the experience replay library to enter the next round of training until the preset conditions are met to obtain a trained agent; each trained agent is deployed on a satellite, and the local information observed by itself is input into the Actor network of the agent to obtain the decision action of each satellite; in the outer multi-objective optimization process based on the multi-objective genetic algorithm, the decision action of the satellite is evaluated, and the constraints of the reward value and the target parameter are calculated. The constraints of the reward value and the target parameter are used as the basis for selection, crossover and mutation operations, and the constellation orbit parameters are obtained by optimizing the reward and loss to realize the constellation deployment design.

[0068] Specifically, a centralized training and distributed execution architecture is adopted to carry out multi-agent reinforcement learning.

[0069] The "Centralized Training with Decentralized Execution" (CTDE) training architecture is adopted. In this way, the agent can prevent the change of the strategy of a single agent from affecting the selection strategy of other agents globally. In the single-agent reinforcement learning process, the action of the agent determines the state transition of the agent system environment, while for the multi-agent system, the actions of all agents determine the state transition of the multi-agent environment. In addition, in the multi-agent system, the rewards obtained by each agent in the learning process depend not only on its own choice of action, but also on the choices of other agents. In a multi-agent system, when the strategy of one of the agents is changed, the optimal selection strategies of other agents will also change accordingly. Choosing the CTDE architecture can avoid the occurrence of these problems between multi-agents. The centralized training-distributed execution structure is as follows: Figure 4 shown.

[0070] Centralized training refers to the use of joint action value functions to train the agent during the training process. Its input is the global situation information and the action information of all agents . The advantage of this is that all agents can share a set of network parameters, and the strong coupling relationship between agents will be considered when making action choices. The resulting reward functions are all related to the joint action, which can prevent the action strategy of a single agent from affecting the strategies of other agents, thus solving the problem of the algorithm's difficulty in convergence. "Distributed execution" means that during the execution process, each individual agent can only receive observed local information and cannot obtain global information. Therefore, the observation information of each agent is used as input, and the decision action made by the agent is used as output. By adopting this distributed decision-making method, the shortcomings of small action differentiation between different agents and weak model exploratory ability can be compensated.

[0071] The CTDE architecture adopted in this application uses a Critic network to learn a central value function that can observe global agent and environmental information, and then through the Actor network, each agent calculates the strategy through its own local observations.

[0072] In one embodiment, the Actor network includes a Transformer network, and the Critic network includes a Transformer network.

[0073] Specifically, the inner layer of the double-layer loop model based on the double-layer architecture of the multi-objective genetic algorithm and the multi-agent deep reinforcement learning algorithm uses the Actor-Critic network architecture to train the Transformer network. The training architecture is as follows Figure 4 As shown, there are two networks involved in the training algorithm: 1) Actor network: Based on the input state, the Transformer network generates the probability for determining the next action, thereby selecting the optimal action strategy; 2) Critic network: It is a value function network that evaluates the value estimate of a given state through the Transformer network, thereby evaluating the reward of a specific state.

[0074] In one embodiment, the multi-objective genetic algorithm in step 104 is a multi-objective genetic algorithm NSGA-II.

[0075] In one embodiment, the multi-agent deep reinforcement learning algorithm in step 104 is a MAPPO algorithm of the CTDE architecture.

[0076] Specifically, the inner layer of the double-layer loop model based on the double-layer architecture of the multi-objective genetic algorithm and the multi-agent deep reinforcement learning algorithm adopts the MAPPO algorithm, and the outer layer adopts the NSGA Ⅱ algorithm. In the inner layer, the reward obtained by each agent (corresponding to a satellite) is not only related to its own behavior, but also to other agents. The outer loop adopts a distributed computing framework.

[0077] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0078] In one embodiment, Figure 5 As shown, a constellation design device based on evolutionary reinforcement learning is provided, including: a constellation design task modeling module and a constellation deployment design module, wherein: The constellation design task modeling module is used to transform the constellation design task problem into a multi-satellite Earth observation mission planning problem; the multi-satellite Earth observation mission planning problem is modeled as a multi-objective combinatorial optimization problem; the optimization goal of the multi-objective combinatorial optimization problem is to maximize the performance and benefits of the Earth observation satellite under given resources and constraints.

[0079] The constellation deployment design module is used to solve the multi-objective combinatorial optimization problem by adopting a two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm, and optimize the constellation deployment design scheme; the two-layer loop model is used to realize the task planning of dynamic target observation through an inner loop based on multi-agent reinforcement learning, and feed back the reward and loss Loss of the action network to the outer loop. Through the outer loop based on the multi-objective genetic algorithm, the goal is to find a set of satellite constellation parameters with the maximum reward, the minimum loss and satisfying the constraints, so as to optimize the constellation deployment design scheme.

[0080] In one embodiment, the constraints of the multi-satellite earth observation mission planning model are shown in formulas (1) to (5).

[0081] In one embodiment, the optimization objectives of the multi-objective combined optimization problem include: maximizing mission benefits, minimizing attitude maneuvering angles, minimizing the coefficient of variation of satellite mission completion, and minimizing the sum of observation times for selected mission completion; wherein, the expression for maximizing mission benefits is shown in formula (6); the expression for minimizing attitude maneuvering angles is shown in formula (7); the expression for minimizing the coefficient of variation of satellite mission completion is shown in formula (10); and the expression for minimizing the sum of observation times for selected mission completion is shown in formula (14).

[0082] In one embodiment, the constellation deployment design module is further used to construct multiple agents in the inner layer; each agent includes an Actor network; all agents share a critic network; a centralized training distributed execution framework is used to train the agents, and global situation information and action information of all agents are used in the centralized training process. As the input of the Actor network, the action strategy and reward value are output; all action strategies interact with the environment to obtain the global observation state; the global observation state and all action strategies are input into the critic network to obtain the value estimate; the network parameters are adjusted according to the reward value and the value estimate, and the action strategy and the corresponding reward are stored in the experience replay library to enter the next round of training until the preset conditions are met to obtain a trained agent; each trained agent is deployed on a satellite, and the local information observed by itself is input into the Actor network of the agent to obtain the decision action of each satellite; in the outer multi-objective optimization process based on the multi-objective genetic algorithm, the decision action of the satellite is evaluated, and the constraints of the reward value and the target parameter are calculated. The constraints of the reward value and the target parameter are used as the basis for selection, crossover and mutation operations, and the constellation orbit parameters are obtained by optimizing the reward and loss to realize the constellation deployment design.

[0083] In one of the embodiments, the Actor network in the constellation deployment design module includes a Transformer network, and the Critic network includes a Transformer network.

[0084] In one of the embodiments, the multi-objective genetic algorithm in the constellation deployment design module is a multi-objective genetic algorithm NSGA-II.

[0085] In one embodiment, the multi-agent deep reinforcement learning algorithm in the constellation deployment design module is the MAPPO algorithm of the CTDE architecture.

[0086] For the specific definition of the constellation design device based on evolutionary reinforcement learning, please refer to the definition of the constellation design method based on evolutionary reinforcement learning above, which will not be repeated here. Each module in the above-mentioned constellation design device based on evolutionary reinforcement learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0087] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a constellation design method based on evolutionary reinforcement learning is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0088] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0089] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0090] In one embodiment, a computer readable memory is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0091] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable memory. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0092] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A constellation design method based on evolutionary reinforcement learning, characterized in that: The method comprises: Transform the constellation design task problem into a multi-satellite Earth observation mission planning problem; The multi-satellite Earth observation mission planning problem is modeled as a multi-objective combinatorial optimization problem. The optimization goal of the multi-objective combinatorial optimization problem is to maximize the performance and benefits of the Earth observation satellite under given resources and constraints. A double-layer loop model with a double-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm is used to solve the multi-objective combinatorial optimization problem and optimize the constellation deployment design scheme; the double-layer loop model is used to realize task planning of dynamic target observation through an inner loop based on multi-agent reinforcement learning, and feed back the reward and loss Loss of the action network to the outer loop. The outer loop based on the multi-objective genetic algorithm aims to find a set of satellite constellation parameters with maximum reward, minimum loss and satisfying constraints, so as to optimize the constellation deployment design scheme.

2. The constellation design method based on evolutionary reinforcement learning according to claim 1, characterized in that: The constraints of the multi-satellite Earth observation mission planning model include: in, The number of expressions is A collection of satellites, Indicates the quantity A collection of observation tasks, Indicates the task and tasks Continuously observed by the same satellite, Respectively represent tasks The observation time and end time of , indicating the task With the task The attitude transition time between Respectively represent tasks The start time, end time, duration and observed pendulum angle of the visible time window, Indicates the task Subtasks In satellite No. Whether the time window is observed, Indicates the task In satellite No. Whether the time window is observed, Satellite Memory and power, Satellite The memory and power consumption per unit time of observation, , , , They are the number of satellites continuously observing two target tasks, the number of satellites continuously observing three target tasks, the number of satellites observing one target task, and the number of all satellites assigned observation tasks.

3. The constellation design method based on evolutionary reinforcement learning according to claim 1, characterized in that: The optimization objectives of the multi-objective combination optimization problem include: maximizing mission benefits, minimizing attitude maneuvering angles, minimizing the coefficient of variation of satellite mission completion, and minimizing the sum of observation times for selected mission completion; Among them, the expression for maximizing task benefits is: in, is the maximum value of the task benefit; The expression for minimizing the attitude maneuver angle is: in, is the minimum value of the attitude maneuvering angle, For the task The average pitch angle, For the task The average measured swing angle, is the number of all satellites assigned observation tasks; The expression for minimizing the coefficient of variation of the satellite's mission completion is: in, is the minimum value of the coefficient of variation for the satellite to complete its mission, For satellite Working hours, is the average working time of the satellite, The number of satellites for observing a target mission; The expression for minimizing the sum of the observation time to complete the selected tasks is: in, is the minimum value of the sum of observation time for the selected tasks to complete, , , They are the number of satellites that continuously observe two target tasks, the number of satellites that observe one target task, and the number of satellites that are assigned observation tasks. Respectively represent tasks The observation time and end time of Indicates the task In satellite No. Whether a time window is observed.

4. The constellation design method based on evolutionary reinforcement learning according to claim 1, characterized in that: A two-layer loop model with a two-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm is used to solve the multi-objective combinatorial optimization problem and optimize the constellation deployment design scheme, including: Build multiple agents in the inner layer; each agent includes an Actor network; all agents share a critic network; The centralized training distributed execution framework is used to train the intelligent agent. During the centralized training process, the global situation information and the action information of all intelligent agents are used. As the input of the Actor network, output action strategies and reward values; interact all the action strategies with the environment to obtain a global observation state; input the global observation state and all the action strategies into the critic network to obtain a value estimate; adjust the network parameters according to the reward value and the value estimate, and store the action strategy and the corresponding reward in the experience replay library, and enter the next round of training until the preset conditions are met to obtain a trained intelligent agent; Deploy each trained agent on a satellite, input the local information observed by itself into the agent's Actor network, and obtain the decision action of each satellite; In the outer multi-objective optimization process based on the multi-objective genetic algorithm, the decision actions of the satellite are evaluated, and the constraints of the reward values ​​and target parameters are calculated. The constraints of the reward values ​​and target parameters are used as the basis for selection, crossover and mutation operations. The constellation orbit parameters are obtained by optimizing the rewards and losses to realize the constellation deployment design.

5. The constellation design method based on evolutionary reinforcement learning according to claim 3, characterized in that: The Actor network includes the Transformer network, and the Critic network includes the Transformer network.

6. The constellation design method based on evolutionary reinforcement learning according to claim 1, characterized in that: The multi-objective genetic algorithm is the multi-objective genetic algorithm NSGA-II.

7. The constellation design method based on evolutionary reinforcement learning according to claim 1, characterized in that: The multi-agent deep reinforcement learning algorithm is the MAPPO algorithm of the CTDE architecture.

8. A constellation design device based on evolutionary reinforcement learning, characterized in that: The device comprises: The constellation design task modeling module is used to transform the constellation design task problem into a multi-satellite earth observation task planning problem; the multi-satellite earth observation task planning problem is modeled as a multi-objective combinatorial optimization problem; the optimization goal of the multi-objective combinatorial optimization problem is to maximize the performance and benefits of the earth observation satellite under given resources and constraints; The constellation deployment design module is used to solve the multi-objective combinatorial optimization problem by adopting a double-layer loop model with a double-layer architecture based on a multi-objective genetic algorithm and a multi-agent deep reinforcement learning algorithm, and optimize the constellation deployment design scheme; the double-layer loop model is used to realize the task planning of dynamic target observation through an inner loop based on multi-agent reinforcement learning, and feed back the reward and loss Loss of the action network to the outer loop, and optimize the constellation deployment design scheme through an outer loop based on a multi-objective genetic algorithm with the goal of finding a set of satellite constellation parameters with maximum reward, minimum loss and satisfying constraints.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the constellation design method based on evolutionary reinforcement learning described in any one of claims 1 to 7 is implemented.

10. A computer readable memory having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the constellation design method based on evolutionary reinforcement learning described in any one of claims 1 to 7 is implemented.

Images