Observation satellite multi-target planning method and system based on mixed action space

Abstract

The invention provides an observation satellite multi-target planning method and system based on a mixed action space, and the method comprises the steps: describing an agile satellite comprehensive task planning problem, and constructing a satellite comprehensive task planning model based on continuous decision making; providing a multi-target deep reinforcement learning method based on a mixed space, and training a satellite comprehensive task planning model based on continuous decision by using the method; and performing task planning by using the trained satellite comprehensive task planning model based on the continuous decision. According to the method, aiming at the complex and emergent task condition of a target and the characteristics of large solving space and low search speed of an agile satellite task planning comprehensive problem, a multi-target problem model is established based on reinforcement learning, and a reinforcement learning method based on a gradient strategy mixed space is introduced to solve the satellite comprehensive task planning problem; therefore, discrete selection in continuous time is made, and data transmission and observation tasks can be arranged at the same time due to the existence of the mixed action space.

Description

Technical Field

[0001] The present invention relates to the technical field of satellite mission planning, and in particular to a method and system for multi-objective planning of an observation satellite based on a hybrid action space, and also to a corresponding computer terminal and a computer-readable storage medium. Background Art

[0002] After years of development, satellites have evolved from being able to transmit signals only through the radio transmitters they carry to being able to conduct autonomous ground dispatch observations through their own attitude controllers. Satellite mission planning has also become increasingly complex. With the emergence of agile satellites, various latest technologies in the computer field have begun to be used in aerospace dispatch to adapt to large-scale and complex satellite mission planning. Agile satellites have higher spatial freedom and are more flexible and autonomous than non-agile satellites. With the development of aerospace technology, agile satellites have become the target of research and development in various countries due to their better imaging capabilities. Agile satellites also introduce more operations, so their mission planning is more difficult than that of traditional non-agile satellites.

[0003] The problem of integrated remote sensing satellite planning is to maximize the benefits by reasonably arranging observation tasks while meeting the constraints. However, traditional heuristic methods are not flexible enough when facing the problem of integrated satellite scheduling, and many complex heuristic rules need to be customized to achieve integrated task planning. Moreover, existing technologies for deep reinforcement learning (DRL) applications do not consider the integrated scheduling of data transmission tasks. The action space in these DRL methods is single, making it difficult to comprehensively schedule multiple tasks. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and system for multi-objective planning of an observation satellite based on a hybrid action space, and also provides a corresponding computer terminal and a computer-readable storage medium.

[0005] According to one aspect of the present invention, a method for multi-objective planning of an observation satellite based on a hybrid action space is provided, comprising:

[0006] Describe the problem of agile satellite integrated mission planning and build a satellite integrated mission planning model based on continuous decision-making;

[0007] A multi-objective deep reinforcement learning method based on mixed space is provided, and the method is used to train a satellite integrated mission planning model based on continuous decision-making;

[0008] The trained satellite integrated mission planning model based on continuous decision-making is used for mission planning.

[0009] Preferably, the description of the agile satellite integrated mission planning problem and the construction of a satellite integrated mission planning model based on continuous decision-making include:

[0010] Define the elements of the agile satellite integrated mission planning problem;

[0011] Setting constraints for remote sensing satellite planning problems;

[0012] Based on the elements in the defined agile satellite integrated mission planning problem and the set constraints of the remote sensing satellite planning problem, a satellite integrated mission planning based on continuous decision-making is constructed.

[0013] Preferably, the elements of the agile satellite integrated mission planning problem are defined, including:

[0014] set up:

[0015] [ST,ET] is the start time and end time of the scenario planned for this task;

[0016] GT={gt,|GT|=n gt} is the set of all observed targets, where n gt is the number of observed targets, and each observed target gt can be expressed as:

[0017] gt={No,lat,lon,alt,alt,p,d,vtw}

[0018] Among them, No is the number of the observed target, gt.{lat,lon,alt} is the location of the observed target, gt.lat, gt.lon, gt.alt are the longitude, latitude and altitude of the target respectively, gt.p is the priority of the observed target, gt.d is the duration of the observation, which is generally specified by the user, and vtw is the visible time window.

[0019] ObTask={ob,|Obtask|=n ob} is the set of all observation tasks generated within the planning time, where n ob is the total number of observation tasks, and each observation task ob can be expressed as:

[0020] ob={No,gt,s,e,θ,d,m} (3)

[0021] Among them, No is the number of the observed target, gt is the target observed in this task, ob.s is the start time of the observation, ob.e is the end time of the observation, ob.θ is the measured pendulum angle required for the observation, ob.d is the duration of the observation, which is determined by the priority of the observed target, and ob.m is the storage space consumed by the observation task.

[0022] Preferably, the setting of constraints on the remote sensing satellite planning problem includes:

[0023] The conflict constraint between data transmission and observation tasks is used to ensure that the satellite can still observe the observation target when transmitting data to a data transmission station whose distance is greater than the set distance threshold;

[0024] Time window constraints are used to ensure that the task execution time always stays within the time window;

[0025] Attitude transfer time constraint is used to limit the attitude transfer time to ensure that the satellite can maneuver to the set position to perform observation or data transmission tasks after completing one mission and executing the next mission;

[0026] Storage constraints and energy constraints are used to limit the size of temporary data on board to no more than the total storage space of the satellite.

[0027] Preferably, the constructing of satellite integrated mission planning based on continuous decision-making includes:

[0028] A time-continuous resource allocation model is provided to solve the problem of agile satellite integrated mission planning. A satellite integrated mission planning model based on continuous decision-making is established, and continuous decision variables are introduced to represent the task start time within the executable time window of each task.

[0029] The optimization objectives of the satellite integrated mission planning model based on continuous decision-making are constructed, including:

[0030] Objective 1: The sum of the priorities of the observed targets is the largest; the sum of the priorities of all observed and transmitted tasks is defined as:

[0031] Max(∑ dt∈DtTask ∑ ob∈dt ob.gt.p)

[0032] Among them, dt is the corresponding time of the data transmission task, DtTask is the data transmission window corresponding to the ground station, ob is the observation task, and ob.gt.p indicates the priority corresponding to the observation task generated by the observation target;

[0033] Objective 2: The satellite maneuver duration is minimized, then:

[0034] Min(∑Transtime)

[0035] Among them, Transtime is the duration of attitude maneuver;

[0036] Objective 3: The target is observed in the shortest time possible, then:

[0037] Min(∑task.st ob )

[0038] Among them, task.s is the start observation time required by the task, t ob The observation time or appearance time required for the mission objective.

[0039] Preferably, the method of providing a multi-objective deep reinforcement learning method based on a mixed space comprises:

[0040] The Markov decision process model M is constructed as:

[0041] M={S,A,P,R,γ}

[0042] Among them, S is a finite state set, A is an action set, P is a state transition probability matrix, R is a profit equation, γ is a discount factor, γ∈[0,1];

[0043] In the satellite integrated mission planning problem, the finite state set S includes the mission set, satellite state and ground station state.

[0044] S={Obtask,SAT}

[0045] Among them, Obtask is the observation task corresponding to the observation target, and SAT is the satellite that performs the task.

[0046] The action set is a finite decision variable. Here, the action set A is designed as:

[0047] A={No,s}

[0048] Among them, No is the task number selected for execution in this action, s is the start of this task in the time window, and is a normalized [-1,1] parameter;

[0049] The duration d of the observation task is specified by the user, and its observation start time ob.s and observation end time ob.e are:

[0050]

[0051] Among them, gt.vtw.s is the start time of the task window, gt.vtw.e is the end time of the task window, and gt.d is the duration of the observation;

[0052] At each time step ti, the Markov decision process model obtains the environment state s t And make an action t , and then get the revenue r t and the next state s t+1 ; wherein the action a tThe Markov decision process model selects in its action space; the action space includes discrete actions and continuous actions; in the discrete action space, the Markov decision process model selects actions from a limited set of discrete actions, and in the continuous action space, the Markov decision process model selects from a continuous action space;

[0053] The method is used to train a satellite integrated mission planning model based on continuous decision-making, including:

[0054] The multi-objective deep reinforcement learning method based on mixed space is used to process time continuously, set the input as satellite status and task status information, and output as the selected task and the task start time within the time window, so as to train the satellite integrated task planning model based on continuous decision-making; wherein, minibatch is used to divide the training data into several minibatches, one minibatch is run for each training, the gradient is updated once, and a fixed number of targets are randomly selected from the target database for training; a comprehensive reward mechanism is formed by considering multiple different targets at the same time, and the reinforcement learning model is trained according to these reward signals, so that the model can make efficient decisions under the premise of meeting multiple task requirements.

[0055] Preferably, the step of performing mission planning using the trained satellite integrated mission planning model based on continuous decision-making includes:

[0056] All observation tasks are sorted in the order of the task window start time gt.vtw.s, and the sequence number, time window start time and end time, priority and duration, and swing angle of the first n tasks are selected each time as the input of the observation task, and the start time and end time of all data transmission windows and the swing angle are used as the input of the data transmission task; the input of the satellite status includes: static parameters and dynamic parameters, among which the static parameters are the total on-board capacity of the satellite, the observation consumption capacity rate, the data transmission rate, and the maneuvering speed, and the dynamic parameters are the current time, the current maneuvering angle of the satellite, and the remaining capacity; then the task set of input parameters is set to: {No, gt.vtw.s, gt.vtw.e, gt.θ, gt.t}.

[0057] The output includes two parameters, where the first parameter is a number from 0 to n, and one observation task is selected from the n inputs each time; the second parameter is a normalized number in the range of -1 to 1, indicating the start time of the observation;

[0058] After each decision is made, the current benefit is returned, the model weights are updated, the observation tasks are sorted according to the current time point, and the tasks are selected again for input. Through the mixed action space, the first parameter that forms the discrete parameter and the second parameter that forms the continuous parameter are jointly output, realizing the comprehensive planning of data transmission and observation tasks.

[0059] According to another aspect of the present invention, there is provided an observation satellite multi-objective planning system based on a hybrid action space, comprising:

[0060] A mathematical model building module, which is used to describe the problem of agile satellite integrated mission planning and build a satellite integrated mission planning model based on continuous decision-making;

[0061] A mathematical model training module, which is used to provide a multi-objective deep reinforcement learning method based on a mixed space, and uses this method to train a satellite integrated mission planning model based on continuous decision-making;

[0062] The mission planning module uses the trained satellite integrated mission planning model based on continuous decision-making to carry out mission planning.

[0063] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to execute any one of the methods described above in the present invention, or to run the system described above in the present invention.

[0064] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above in the present invention, or to run the system described above in the present invention.

[0065] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0066] The present invention provides an observation satellite multi-objective planning method and system based on hybrid action space, which plans remote sensing satellite missions based on hybrid action space, making the planning process more flexible and improving mission benefits.

[0067] The multi-objective planning method and system for observation satellites based on hybrid action space provided by the present invention adopts a multi-objective reinforcement learning method to obtain significant beneficial effects in the application of agile satellite integrated mission planning problems; wherein, while selecting the execution task through the hybrid action space, the start time of the task execution moment can be selected. Agile satellites usually have a long time window, and the attitude maneuvering angle will change greatly within the time window due to time changes. Therefore, this selection strategy can decide on better planning results. When faced with multiple conflicting mission objectives, a more intelligent mission planning solution is provided by dynamically adjusting and weighing multiple objectives. Multi-objective reinforcement learning can find a balance between different objectives, optimize the weights of multiple objectives, and enable satellite missions to achieve the optimal solution under different constraints. This avoids the situation where a single objective is favored at the expense of other objectives.

[0068] The observation satellite multi-objective planning method and system based on hybrid action space provided by the present invention can allocate satellite resources and time according to the priorities of different tasks, ensuring that key tasks are processed first. At the same time, the use of reinforcement learning algorithms can continuously optimize task planning decisions through repeated training and trial-and-error mechanisms, reduce unnecessary actions and adjustments, and thus significantly improve the efficiency of task planning. The application of hybrid action space allows the system to simultaneously optimize continuous and discrete action selections, making task decisions more refined and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0070] Figure 1 The present invention is a flowchart of a multi-objective planning method for an observation satellite based on a hybrid action space in a preferred embodiment of the present invention.

[0071] Figure 2 The figure is a schematic diagram of the training process of a satellite integrated mission planning model based on continuous decision-making in a preferred embodiment of the present invention.

[0072] Figure 3 The schematic diagram of the planning work of the multi-objective planning method of the observation satellite based on the hybrid action space in a preferred embodiment of the present invention.

[0073] Figure 4 It is a schematic diagram of the component modules of the observation satellite multi-objective planning system based on the hybrid action space in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0075] The existing deep reinforcement learning (DRL) method is not flexible enough when facing the problem of satellite integrated scheduling, and the action space is single, making it difficult to comprehensively schedule multiple tasks. In view of the above problems, an embodiment of the present invention provides a multi-objective planning method for observation satellites based on hybrid action space. The method is oriented to the complex and sudden task conditions of the targets and the characteristics of large solution space and slow search speed for the comprehensive problem of agile satellite task planning. A multi-objective problem model is established based on reinforcement learning, and a reinforcement learning method based on gradient strategy hybrid space (the hybrid proximal policy optimization HPPO) is introduced to solve the problem of satellite integrated task planning. By solving the reinforcement learning method based on gradient strategy hybrid space, discrete choices in continuous time are made. Due to the existence of the hybrid action space, data transmission and observation tasks can be arranged at the same time.

[0076] Specifically, Figure 1 As shown, the observation satellite multi-objective planning method based on the hybrid action space provided in this embodiment may include the following operations:

[0077] S1, describes the problem of agile satellite integrated mission planning and constructs a satellite integrated mission planning model based on continuous decision-making.

[0078] In some preferred implementations, when the agile satellite is passing through, it observes the target according to the user's needs and generates an observation task. The window in which the target can be observed is called the visible time window (VTW). The remote sensing satellite planning problem is to maximize the benefits by reasonably arranging data transmission and observation tasks under the condition of meeting the constraints. It further includes the following operations:

[0079] S11, define the various elements in the agile satellite integrated mission planning problem, as shown in Table 1:

[0080] Table 1

[0081] element meaning <![CDATA[ObTask i ]]> The observation task corresponding to the i-th target <![CDATA[[ObTaskS i ,ObTaskE i ]]]> The visible time window corresponding to the i-th target <![CDATA[[DtTaskS j ,DtTaskE j ]]> The countable transmission window corresponding to the jth ground station <![CDATA[θ i ]]> The slewingangle corresponding to the i-th task t Observation target appearance time <![CDATA[d i ]]> The duration of the i-th target <![CDATA[x i ]]> Whether to execute the i-th goal v Measuring pendulum speed <![CDATA[Transtime i,j ]]> The transfer time between the i-th and j-th tasks maxθ Maximum swing angle MaxM Maximum capacity on board <![CDATA[ΔM i ]]> Energy change caused by the i-th task <![CDATA[e i ]]> The energy consumed by the i-th task E Total energy on the star <![CDATA[p i ]]> The priority of the i-th task Maxv Maximum swing rate

[0082] set up:

[0083] [ST,ET] is the start time and end time of the scenario planned for this task;

[0084] GT={gt,|GT|=ngt} is the set of all observed targets, where n gt is the number of observed targets, and each observed target gt can be expressed as:

[0085] gt={No,lat,lon,alt,alt,p,d,vtw}

[0086] Among them, No is the number of the observed target, gt.{lat,lon,alt} is the location of the observed target, gt.lat, gt.lon, gt.alt are the longitude, latitude and altitude of the target respectively, gt.p is the priority of the observed target, gt.d is the duration of the observation, which is generally specified by the user, and vtw is the visible time window.

[0087] ObTask={ob,|Obtask|=n ob} is the set of all observation tasks generated within the planning time, where n ob is the total number of observation tasks, and each observation task ob can be expressed as:

[0088] ob={No,gt,s,e,θ,d,m}

[0089] Among them, No is the number of the observed target, gt is the target observed in this task, ob.s is the start time of the observation, ob.e is the end time of the observation, ob.θ is the measured pendulum angle required for the observation, ob.d is the duration of the observation, which is determined by the priority of the observed target, and ob.m is the storage space consumed by the observation task.

[0090] S12, set the constraints for the remote sensing satellite planning problem:

[0091] When comprehensively planning the observation targets and data transmission tasks in a certain area, the following constraints need to be considered:

[0092] S121, data transmission and observation task conflict constraint. In scenarios where there is a large installation angle between the data transmission antenna and the camera of the remote sensing satellite or the ground station is remote, distortion will occur when the satellite's pendulum angle is greater than 45°, which will cause the satellite to be unable to observe certain targets when transmitting data to a distant data transmission station.

[0093] S122, time window constraint. Each task has an executable window, and the task execution time must be within the time window:

[0094] ob.s≤ob.gt.vtw.s

[0095] ob.e≤ob.gt.vtw.e

[0096] S123, attitude transfer time constraint. When a satellite completes one mission and executes the next mission, it needs to adjust its attitude and maneuver to a suitable position to perform observation or data transmission tasks. The attitude transfer time is determined by the satellite's maneuvering speed and the difference in angles required between the two tasks:

[0097]

[0098] |θ i |≤Maxθ

[0099] Among them, v is the satellite's pendulum rate, which can be regarded as a constant; Maxθ is the satellite's maximum pendulum angle, which is a constant.

[0100] e i-1 +Transtime i-1,i ≤s i

[0101] Among them, e i-1 For Transtime i-1,i For i The start time of the next task;

[0102] Formula (8) represents the satellite attitude transfer time constraint. The time of satellite attitude maneuver from the current mission to the next mission cannot exceed the difference between the end time of the current mission and the start time of the next one.

[0103] S124, storage constraints and energy constraints. The on-board capacity is limited. When performing observation tasks, the satellite generates data, which is temporarily stored on the satellite and waits for downloading. The size of the temporary data on the satellite must not exceed the total storage space of the satellite. And the energy consumed by performing the task must not exceed the total energy on the satellite:

[0104] ∑ΔM i ≤MaxM

[0105]

[0106] Among them, MaxM is the total capacity on the satellite, and N is the number of missions. i is the energy consumed by mission i, and E is the total energy size on the satellite.

[0107] S13, build task model.

[0108] A time-continuous resource allocation model is provided to solve the problem of agile satellite integrated mission planning. A satellite integrated mission planning model based on continuous decision-making is established, and continuous decision variables are introduced to represent the task start time within the executable time window of each task.

[0109] The optimization objectives of building a satellite integrated mission planning model based on continuous decision-making include the following three objectives:

[0110] Objective 1: The sum of the priorities of the observed targets should be as high as possible:

[0111] Max(∑ dt∈DtTask ∑ ob∈dt ob.gt.p)

[0112] Among them, dt is the corresponding time of the data transmission task, DtTask is the data transmission window corresponding to the ground station, ob is the observation task, ob.gt.p represents the priority corresponding to the observation task generated by the observation target, and the above formula represents the sum of the priorities of all observation and data transmission tasks.

[0113] Goal 2: Keep satellite maneuver duration as short as possible:

[0114] Min(∑Transtime)

[0115] Among them, Transtime is the total duration of posture transfer.

[0116] Objective 3: The observation target is observed as soon as possible:

[0117] Min(∑task.st ob )

[0118] Among them, task.s is the start time of the observation task, t ob The observation time or appearance time required for the mission objective.

[0119] S2, provides a multi-objective deep reinforcement learning method based on mixed space, which is used to train a satellite integrated mission planning model based on continuous decision-making.

[0120] like Figure 2 The figure shows a schematic diagram of the training process. The satellite integrated mission planning model based on continuous decision-making includes an Actor network and a Critic network; among them, the satellite state, ground station information and ground target information are states obtained from the environment, and the discrete and continuous Actor networks take actions, outputting actions on continuous airborne and actions in discrete space respectively, and the environment feeds back the next state and a positive and negative reward based on this action. The Critic network trains a state value function to calculate the value used to update the actor network. In the update process, minibatch is used for updating, and the training data is divided into several minibatches. One minibatch is run at a time, and the gradient is updated once, which can speed up the training process.

[0121] During each training process, the agent will randomly select a fixed number of targets (for example, preferably 100) from the target database for training. Each time the environment is different, so as to train the agent's processing capabilities when facing different complex scenarios.

[0122] like Figure 3 As shown in the figure, mathematical modeling is carried out according to the satellite, target, ground station, observation task and data transmission task in the planning scenario, and constraint formula and optimization target are established. Then, a Markov decision process model is established, and the HPPO algorithm is applied for training. After the training, the algorithm (agent) performs task planning.

[0123] S21, construct a Markov decision process model.

[0124] In deep reinforcement learning, the intelligent agent makes decisions and gains benefits by interacting with the dynamic environment, thereby learning to make the best decision. The Markov decision process model is the basis of reinforcement learning. In this study, the satellite is an intelligent agent, making decisions in the current state, changing to the next state and gaining benefits. The Markov decision process model M is:

[0125] M={S,A,P,R,γ}

[0126] Among them, S is a finite state set, A is an action set, P is a state transition probability matrix, R is a profit equation, γ is a discount factor, γ∈[0,1], which is used to adjust the importance of future rewards in long-term decision optimization.

[0127] In the satellite integrated mission planning problem, the finite state set S includes the task set and satellite state.

[0128] S={Obtask,SAT}

[0129] Among them, SAT is the status information of the satellite.

[0130] The action set is a finite decision variable. Here, the action set A is designed as:

[0131] A={No,s}

[0132] Among them, No is the task number selected for execution in this action, s is the start time of this task within the time window, and s is a normalized [-1,1] parameter.

[0133] The duration d of the observation task is specified by the user, and the task end and start times are:

[0134]

[0135] Among them, gt.vtw.s is the start time of the task window, gt.vtw.e is the end time of the task window, and gt.d is the duration of the observation;

[0136] S22, Multi-objective HPPO optimization based on weighted sum method

[0137] In reinforcement learning, the process of interaction between the agent (i.e., model) and the environment is as follows: at each time step ti, the agent obtains the environment state s ti And make an action ti , and then get the revenue r ti and the next state s ti+1 Action a ti The agent selects in its action space. The types of action space can be divided into discrete actions and continuous actions. In the discrete action space, the agent selects actions from a limited set of discrete actions, and in the continuous action space, the agent selects from a continuous action space. Different problems will have different action spaces and require different algorithms. Traditional reinforcement learning algorithms only optimize one of the action spaces. However, in the satellite integrated mission planning problem, choosing to execute a task is a discrete action, and choosing when the task starts or ends is a continuous action. In the face of such a mixed action space, traditional reinforcement learning methods are not applicable. The embodiment of the present invention introduces a mixed space gradient strategy reinforcement learning method to solve the satellite integrated mission planning problem. Its agent can make discrete and continuous decisions, and can process time in a continuous manner. The input is set as satellite status and mission status information, and the output is the selected task and the task start time within the time window. The network is trained to reach the optimal.

[0138] Traditional reinforcement learning (RL) mostly only proposes optimization solutions for continuous or discrete action spaces, and has a good effect in solving problems with only discrete actions or continuous actions. However, in actual situations, there are more mixed spaces, and traditional RL cannot effectively handle joint optimization in mixed spaces. This algorithm (HPPO) can effectively handle both discrete and continuous spaces.

[0139] Multi-Objective Reinforcement Learning (MORL) aims to solve a problem that is not common in standard RL frameworks: how to balance multiple, sometimes conflicting reward functions. Traditional RL uses a single scalar reward function to guide the learning process, while MORL uses a vector reward function. Each element represents a different goal. In MORL, the reward is no longer a single scalar, but a vector, where each element reflects a different goal. A weighted sum method is used for multi-objective optimization, where the importance of each goal is determined by a preset weight. A weighted sum method, where the importance of each goal is determined by a preset weight. In actual reinforcement learning environments, each behavior or decision is evaluated based on a comprehensive objective function (F). Common reinforcement learning algorithms, such as Q-learning or its variants, can be applied here. The goal of the algorithm is to find a strategy that maximizes the comprehensive reward (F) through exploration and exploitation. In this method, meta-policy-based multi-objective reinforcement learning is used for training. During the training process, a preference vector is defined to clarify the priority of each goal, where the reward Reward is defined as:

[0140]

[0141] Among them, w i is the weight of different objectives, r i The benefits corresponding to each goal.

[0142] r1 corresponds to target 1, and target r1 is:

[0143] r1=ob.gt.p / p m

[0144] Where ob.gt.p is the priority of the target observed in action A in the Markov model, and p m The maximum priority value.

[0145] r2 corresponds to target 2, r2 is:

[0146]

[0147] Among them, Transtime i Indicates the current required posture transfer time. Indicates the maximum posture transfer duration.

[0148] r3 corresponds to target 3, r3 is:

[0149]

[0150] Among them, task.s represents the execution time of the task, t ob Indicates the observation time required by the task.

[0151] The purpose of Meta-Policy is to learn how to generate the best policy under different goal combinations. During the training process, the environment will provide different weight combinations. Meta-Policy adjusts the sub-policies through these weight combinations and optimizes according to the performance of these policies. This method uses a reinforcement learning method to train Meta-Policy.

[0152] S3, uses the trained satellite integrated mission planning model based on continuous decision-making to carry out mission planning.

[0153] S31, input parameter design.

[0154] First, all observation tasks are sorted in the order of the task window start time gt.vtw.s. Each time, the sequence number, time window start time and end time, priority and duration, and swing angle of the first 10 tasks are selected as the input of the observation task. The input of the data transmission task is the start time and end time of all data transmission windows and the swing angle. The satellite status input is divided into static and dynamic. The static parameters are the total on-board capacity of the satellite, the observation consumption capacity rate, the data transmission rate, and the maneuvering speed. The dynamic parameters are the current time, the current maneuvering angle of the satellite, and the remaining capacity.

[0155] The task set of input parameters is set to {No,gt.vtw.s,gt.vtw.e,gt.θ,gt.t}.

[0156] S32, output parameter design.

[0157] The output has 2 parameters. The first parameter is a number from 0 to 10. Each time, one observation task is selected from the ten input tasks.

[0158] The second parameter is a normalized number in the range -1 to 1, indicating the start time of the observation.

[0159] After each decision is made, the current profit is returned, the network weights are updated, the observed tasks are sorted according to the current time point, and the tasks are selected again for input.

[0160] The design of the output parameters reflects the superiority of the HPPO method. Through the mixed action space, the discrete parameters (the first parameter) and the continuous parameters (the second parameter) are jointly output, realizing the comprehensive planning of data transmission and observation tasks.

[0161] In a specific application example, a fire target is taken as an observation target, and the multi-objective planning method of an observation satellite based on a hybrid action space provided by the above embodiment of the present invention is adopted to effectively cope with the characteristics of the sudden target of the fire target and the large solution space and slow search speed of the comprehensive problem of agile satellite mission planning. A multi-objective problem model is established through reinforcement learning, and a deep reinforcement algorithm based on the hybrid space is used to solve it, and discrete choices are made in continuous time to achieve comprehensive planning of data transmission and observation tasks for fire targets.

[0162] Based on the same inventive concept, an embodiment of the present invention also provides an observation satellite multi-objective planning system based on a hybrid action space.

[0163] Specifically, Figure 4 As shown, the observation satellite multi-objective planning system based on the hybrid action space provided in this embodiment may include the following modules:

[0164] A mathematical model building module, which is used to describe the problem of agile satellite integrated mission planning and build a satellite integrated mission planning model based on continuous decision-making;

[0165] A mathematical model training module, which is used to provide a multi-objective deep reinforcement learning method based on a mixed space, and uses this method to train a satellite integrated mission planning model based on continuous decision-making;

[0166] The mission planning module uses the trained satellite integrated mission planning model based on continuous decision-making to carry out mission planning.

[0167] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.

[0168] An embodiment of the present invention further provides a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0169] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0170] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0171] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0172] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0173] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0174] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0175] All matters not covered in the above embodiments of the present invention are well known in the art.

[0176] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A multi-objective planning method for observation satellites based on hybrid action space, characterized in that: include: Describe the problem of agile satellite integrated mission planning and build a satellite integrated mission planning model based on continuous decision-making; A multi-objective deep reinforcement learning method based on mixed space is provided, and a satellite integrated mission planning model based on continuous decision-making is trained by using this method; The trained satellite integrated mission planning model based on continuous decision-making is used for mission planning.

2. The observation satellite multi-objective planning method based on hybrid action space according to claim 1 is characterized in that: The agile satellite integrated mission planning problem is described, and a satellite integrated mission planning model based on continuous decision-making is constructed, including: Define the elements of the agile satellite integrated mission planning problem; Setting constraints for remote sensing satellite planning problems; Based on the elements in the defined agile satellite integrated mission planning problem and the set constraints of the remote sensing satellite planning problem, a satellite integrated mission planning based on continuous decision-making is constructed.

3. The observation satellite multi-objective planning method based on hybrid action space according to claim 2 is characterized in that: The elements of the agile satellite integrated mission planning problem are defined, including: set up: [ST, ET] is the start time and end time of the scenario planned for this task; GT={gt,|GT|=n gt } is the set of all observed targets, where n gt is the number of observed targets, and each observed target gt can be expressed as: gt={No, lat, lon, alt, alt, p, d, vtw} Where No is the number of the observed target, gt.{lat, lon, alt} is the location of the observed target, gt.lat, gt.lon, gt.alt are the longitude, latitude and altitude of the target respectively, gt.p is the priority of the observed target, gt.d is the duration of the observation, and vtw is the visible time window; ObTask={ob,|Obtask|=n ob } is the set of all observation tasks generated within the planning time, where n ob is the total number of observation tasks, and each observation task ob can be expressed as: ob={No, gt, s, e, θ, d, m} (3) Among them, No is the number of the observed target, gt is the target observed in this task, ob.s is the start time of the observation, ob.e is the end time of the observation, ob.θ is the measured pendulum angle required for the observation, ob.d is the duration of the observation, which is determined by the priority of the observed target, and ob.m is the storage space consumed by the observation task.

4. The observation satellite multi-objective planning method based on hybrid action space according to claim 2 is characterized in that: The constraints on the remote sensing satellite planning problem are set, including: The conflict constraint between data transmission and observation tasks is used to ensure that the satellite can still observe the observation target when transmitting data to a data transmission station whose distance is greater than the set distance threshold; Time window constraints are used to ensure that the task execution time always stays within the time window; Attitude transfer time constraint is used to limit the attitude transfer time to ensure that the satellite can maneuver to the set position to perform observation or data transmission tasks after completing one mission and executing the next mission; Storage constraints and energy constraints are used to limit the size of temporary data on board to no more than the total storage space of the satellite.

5. The observation satellite multi-objective planning method based on hybrid action space according to claim 2 is characterized in that: The construction of satellite integrated mission planning based on continuous decision-making includes: A time-continuous resource allocation model is provided to solve the problem of agile satellite integrated mission planning. A satellite integrated mission planning model based on continuous decision-making is established, and continuous decision variables are introduced to represent the task start time within the executable time window of each task. The optimization objectives of the satellite integrated mission planning model based on continuous decision-making are constructed, including: Objective 1: The sum of the priorities of the observed targets is the largest; the sum of the priorities of all observed and transmitted tasks is defined as: Max(∑ dt∈DtTask ∑ ob∈dt ob.gt.p) (11) Among them, dt is the corresponding time of the data transmission task, DtTask is the data transmission window corresponding to the ground station, ob is the observation task, and ob.gt.p indicates the priority corresponding to the observation task generated by the observation target; Objective 2: The satellite maneuver duration is minimized, then: Min(∑ Transtime) Among them, Transtime is the duration of attitude maneuver; Objective 3: The target is observed in the shortest time possible, then: Min(∑task.s-t ob ) Among them, task.s is the start observation time required by the task, t ob The observation time or appearance time required for the mission objective.

6. The observation satellite multi-objective planning method based on hybrid action space according to claim 1 is characterized in that: The invention provides a multi-objective deep reinforcement learning method based on a mixed space, comprising: The Markov decision process model M is constructed as: M={S,A,P,R,γ} Among them, S is a finite state set, A is an action set, P is a state transition probability matrix, R is a profit equation, γ is a discount factor, γ∈[0,1]; In the satellite integrated mission planning problem, the finite state set S includes the mission set, satellite state, and ground station state; S = {Obtask, SAT} Among them, Obtask is the observation task corresponding to the observation target, and SAT is the satellite performing the task; The action set is a finite decision variable. Here, the action set A is designed as: A={No,s} Among them, No is the task number selected for execution in this action, s is the start time of this task in the time window, and is a normalized [-1, 1] parameter; The duration d of the observation task is specified by the user, and its observation start time ob.s and observation end time o be: Among them, gt.vtw.s is the start time of the task window, gt.vtw.e is the end time of the task window, and gt.d is the duration of the observation; At each time step ti, the Markov decision process model obtains the environment state s ti And make an action ti , and then get the revenue r ti and the next state s ti+1 ; wherein the action a ti The Markov decision process model selects in its action space; the action space includes discrete actions and continuous actions; in the discrete action space, the Markov decision process model selects actions from a limited set of discrete actions, and in the continuous action space, the Markov decision process model selects from a continuous action space; The method is used to train a satellite integrated mission planning model based on continuous decision-making, including: The multi-objective deep reinforcement learning method in the hybrid space based on the gradient strategy is used to process time continuously, set the input as the satellite state and task state information, and output as the selected task and the task start time within the time window, so as to train the satellite integrated task planning model based on continuous decision-making; wherein, the training data is divided into several minibatches by minibatch, one minibatch is run for each training, the gradient is updated once, and a fixed number of targets are randomly selected from the target database for training; a comprehensive reward mechanism is formed by considering multiple different targets at the same time, and the reinforcement learning model is trained according to these reward signals, so that the model can make efficient decisions under the premise of meeting multiple task requirements.

7. The observation satellite multi-objective planning method based on hybrid action space according to claim 1 is characterized in that: The method of using the trained satellite integrated mission planning model based on continuous decision-making to perform mission planning includes: All observation tasks are sorted in the order of the task window start time gt.vtw.s, and the sequence number, time window start time and end time, priority and duration, and swing angle of the first n tasks are selected as the input of the observation task each time, and the start time and end time of all data transmission windows and the swing angle are used as the input of the data transmission task; the input of the satellite status includes: static parameters and dynamic parameters, wherein the static parameters are the total on-board capacity of the satellite, the observation consumption capacity rate, the data transmission rate, and the maneuvering speed, and the dynamic parameters are the current time, the current maneuvering angle of the satellite, and the remaining capacity; then the task set of input parameters is set to: {No, gt.vtw.s, gt.vtw.e, gt.θ, gt.t}; The output includes two parameters, where the first parameter is a number from 0 to n, and one observation task is selected from the n inputs each time; the second parameter is a normalized number in the range of -1 to 1, indicating the start time of the observation; After each decision is made, the current benefit is returned, the model weights are updated, the observation tasks are sorted according to the current time point, and the tasks are selected again for input. Through the mixed action space, the first parameter that forms the discrete parameter and the second parameter that forms the continuous parameter are jointly output, realizing the comprehensive planning of data transmission and observation tasks.

8. An observation satellite multi-objective planning system based on hybrid action space, characterized in that: include: A mathematical model building module, which is used to describe the problem of agile satellite integrated mission planning and build a satellite integrated mission planning model based on continuous decision-making; A mathematical model training module, which is used to provide a multi-objective deep reinforcement learning method based on a mixed space, and uses this method to train a satellite integrated mission planning model based on continuous decision-making; The mission planning module uses the trained satellite integrated mission planning model based on continuous decision-making to carry out mission planning.

9. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it can be used to perform the method described in any one of claims 1 to 7, or to run the system described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method described in any one of claims 1 to 7, or to run the system described in claim 8.

Images