Intersatellite link satellite measurement and control scheduling method and system based on double-layer reinforcement learning

Abstract

The invention discloses an inter-satellite link satellite measurement and control scheduling method and system based on double-layer reinforcement learning, and the method comprises the steps: building an inter-satellite link satellite measurement and control scheduling model, and enabling the optimization target of the inter-satellite link satellite measurement and control scheduling model to maximize the total benefit of a task; distributing all tasks to a ground station antenna through an upper layer optimization model according to a supporting relation between the requested task and the ground station antenna; the lower-layer optimization model calculates the task revenue return and the state of the next step under the action through a task distribution scheme provided by the upper-layer optimization model, information feedback interaction of the upper-layer optimization model and the lower-layer optimization model is achieved, and the double-layer optimization process is circulated for multiple times; and a lower-layer single antenna scheduling result is fed back to an upper layer to guide next satellite measurement and control task allocation, and a time window for executing each task is determined. The method is based on a double-layer deep reinforcement learning algorithm, the upper layer optimizes the satellite measurement and control task allocation problem, the lower layer optimizes the single-antenna task scheduling problem, and the solving quality and speed are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of satellite scheduling, and specifically relates to an inter-satellite link satellite measurement and control scheduling method and system based on double-layer reinforcement learning. Background Art

[0002] As the global demand for satellite Internet continues to grow and the number of satellites in orbit continues to increase, satellites play a key role in national defense security, weather forecasting, path navigation and other fields. However, with the increasing number of satellites and the demand for satellite measurement and control, how to effectively manage satellites and achieve efficient execution of measurement and control tasks has become a challenge that needs to be solved urgently. On the other hand, due to the limitations of the regional distribution of ground stations and the sharp increase in the number of satellites, satellite measurement and control tasks face severe challenges. At the same time, the frequent occurrence of satellite emergency tasks requires satellites and ground stations to quickly establish communication links to transmit information. Therefore, the timely completion of remote control and telemetry tasks will face increasing challenges. In order to deal with such challenging tasks, it is necessary to consider intersatellite link technology to achieve rapid response to measurement and control tasks. When a satellite needs to transmit data quickly and there is no optional ground station antenna, an intersatellite link is established through an intermediate satellite to transmit information and quickly transmit the information to the ground. Such as Figure 1 As shown, satellite tracking and control can quickly complete remote control and telemetry tasks through satellite-to-ground and inter-satellite links.

[0003] Link establishment time process for intersatellite links: A satellite can establish intersatellite links with multiple satellites. After the links are established between satellites, a satellite-to-ground connection is established through an intermediate satellite and the ground station antenna. There can be multiple visible time windows between the satellite and the ground. The general steps of satellite measurement and control scheduling based on intersatellite links are: when a satellite needs to transmit data to the ground, the current satellite selects an optional satellite from all satellites in the intersatellite link as an intermediate satellite. Then, a communication connection is established with the ground station antenna through the intermediate satellite. This achieves the purpose of rapid data transmission between the satellite and the ground station.

[0004] Intersatellite link technology has great advantages in dealing with satellite measurement and control scheduling problems, and can quickly achieve remote control and telemetry tasks. However, satellite measurement and control scheduling problems with intersatellite links still face huge challenges:

[0005] (1) Intersatellite link modeling is more complex. The satellite measurement and control scheduling process is easily affected by the geographical location of the ground station. Due to the limitations of the regional distribution of ground stations and the sharp increase in the number of satellites, it is difficult to establish an effective link between the satellite and the ground station antenna. In order to establish satellite communications between satellites and ground stations, establishing communications through intersatellite links is the key to solving satellite measurement and control scheduling problems. However, after using intersatellite link technology, satellites lack visible time windows and need to transmit information through intermediate satellites to increase the user's visible time window. It can even achieve all-day measurement and control to achieve the purpose of rapid measurement and control. Mission modeling needs to consider more constraints, making the model more complex.

[0006] (2) The size of the inter-satellite link solution space increases. Since inter-satellite links can be established between satellites to transmit information, timely communication between satellites and ground station antennas can be achieved. Through inter-satellite link technology, the satellite's time window is significantly increased. This leads to an increase in the size of the solution space for solving satellite measurement and control tasks. The exponential increase in the solution space will bring huge challenges to the scheduling algorithm. Summary of the invention

[0007] In view of the above-mentioned technical problems, the purpose of the present invention is to provide an inter-satellite link satellite measurement and control scheduling method and system based on double-layer reinforcement learning. The method constructs an integer programming model of the inter-satellite link satellite measurement and control scheduling model, and is based on a double-layer deep reinforcement learning algorithm. The upper layer optimizes the satellite measurement and control task allocation problem, and the lower layer optimizes the single antenna task scheduling problem, thereby improving the quality and speed of the solution.

[0008] The technical solution of the present invention is:

[0009] A satellite measurement and control scheduling method for intersatellite links based on double-layer reinforcement learning comprises the following steps:

[0010] S01: establishing an inter-satellite link satellite measurement and control scheduling model, wherein the optimization goal of the inter-satellite link satellite measurement and control scheduling model is to maximize the total benefit of the task;

[0011] S02: Allocate all tasks to the ground station antenna according to the support relationship between the requested tasks and the ground station antenna through the upper-layer optimization model;

[0012] S03: The lower-level optimization model calculates the task benefit return and the next step status under the action through the task allocation plan provided by the upper-level optimization model, realizes the information feedback interaction between the upper-level optimization model and the lower-level optimization model, and the two-layer optimization process is iterated multiple times. The lower-level single antenna scheduling result is fed back to the upper layer to guide the next satellite measurement and control task allocation and determine the time window for the execution of each task.

[0013] In the preferred technical solution, the inter-satellite link satellite measurement and control scheduling model is:

[0014]

[0015] The constraints are as follows:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024]

[0025] in, is the decision variable for whether task m is executed, p m is the revenue value of task m, R is the set of low-orbit satellite measurement and control scheduling tasks, AS is the satellite set, ALS i is the set of satellites that can establish a link with satellite i, AG ij is the set of optional ground stations for satellite j to establish a link, Aa ijk is the antenna set of ground station k, AW mijkn is the set of optional time windows for task m on antenna n, slt ij is the start time of establishing the link between satellite i and satellite j, elt ij is the end time of the inter-satellite link between satellite i and satellite j, is the start time of the lth time window, is the end time of the lth time window, est m is the earliest allowed start time of the task, lst m The latest allowed end time of the task. is the actual start time of the task, is the actual end time of the task, st m is the service time required for task m.

[0026] In the preferred technical solution, step S02 includes modeling the task allocation process of the upper optimization model into a Markov decision process, where the Markov decision process includes a 4-tuple M={S,A,τ,r}, where S represents the state space, A represents the action space, τ represents the state transition rule, and r represents the benefit function;

[0027] In the state space S, the state s at each step is t ∈S represents the time window for task selection;

[0028] Action space A includes selecting tasks and selecting ground station antennas, which is used to select a suitable and executable ground station antenna for each task. A task and an antenna are selected at each step, and the task and antenna need to satisfy a supportable relationship.

[0029] Set the transition rule τ, the next time t state s t+1 According to action a t and the previous state s t Update, that is, s t+1 =τ(s t );

[0030] The benefit r is used to calculate the benefit value of the task after taking a certain action. The benefit in this model is the increment of the total benefit after taking the action. The calculation formula is:

[0031] r t+1 = r(s t+1 ,a t+1 )=pr i

[0032] where pr i It is the revenue of the task.

[0033] In the preferred technical solution, the selection probability of the action in the above steps is selected by using the ε-greedy strategy, and the maximum value of the state is used as the distribution function of the selection probability.

[0034] In the preferred technical solution, the time window for executing each task is determined by a rule-based heuristic algorithm, and the rule-based heuristic algorithm includes:

[0035] S31: Get the task T assigned to the ground station antenna:

[0036] S32: sorting all the tasks to be planned in descending order according to the task benefits;

[0037] S33: If the task satisfies the constraint conditions, the earliest allowed start time of the task is selected, the task is placed in a schedulable scheme, and the execution order and time window of the task are determined.

[0038] In the preferred technical solution, the method for implementing information feedback interaction between the upper optimization model and the lower optimization model in step S03 includes:

[0039] The task allocation schemes corresponding to the single antenna scheduling with benefits higher than the set threshold are recorded, and an elite set is used to store several task allocation schemes with the highest benefits encountered in history. When a scheme appears repeatedly, its weight is increased; in the subsequent satellite measurement and control task allocation, a certain number of allocation schemes are extracted from the elite set according to the weights.

[0040] The present invention also discloses an inter-satellite link satellite measurement, control and scheduling system based on double-layer reinforcement learning, comprising:

[0041] An inter-satellite link satellite measurement and control scheduling model building module is used to establish an inter-satellite link satellite measurement and control scheduling model, wherein the optimization goal of the inter-satellite link satellite measurement and control scheduling model is to maximize the total benefit of the task;

[0042] The upper layer optimization module allocates all tasks to the ground station antenna according to the support relationship between the requested tasks and the ground station antenna through the upper layer optimization model;

[0043] The lower-layer optimization module calculates the task benefit return and the next state of the action through the task allocation plan provided by the upper-layer optimization model, realizes the information feedback interaction between the upper-layer optimization model and the lower-layer optimization model, and the two-layer optimization process is iterated multiple times. The lower-layer single antenna scheduling result is fed back to the upper layer to guide the next satellite measurement and control task allocation and determine the time window for the execution of each task.

[0044] In the preferred technical solution, the upper-level optimization module includes modeling the task allocation process of the upper-level optimization model into a Markov decision process, wherein the Markov decision process includes a 4-tuple M={S,A,τ,r}, where S represents the state space, A represents the action space, τ represents the state transition rule, and r represents the benefit function;

[0045] In the state space S, the state s at each step is t ∈S represents the time window for task selection;

[0046] Action space A includes selecting tasks and selecting ground station antennas, which is used to select a suitable and executable ground station antenna for each task. A task and an antenna are selected at each step, and the task and antenna need to satisfy a supportable relationship.

[0047] Set the transition rule τ, the next time t state s t+1 According to action a t and the previous state s t Update, that is, s t+1 =τ(s t );

[0048] The benefit r is used to calculate the benefit value of the task after taking a certain action. The benefit in this model is the increment of the total benefit after taking the action. The calculation formula is:

[0049] r t+1 = r(s t+1 ,a t+1 )=pr i

[0050] where pr i It is the revenue of the task.

[0051] In the preferred technical solution, the time window for executing each task is determined by a rule-based heuristic algorithm, and the rule-based heuristic algorithm includes:

[0052] S31: Get the task T assigned to the ground station antenna:

[0053] S32: sorting all the tasks to be planned in descending order according to the task benefits;

[0054] S33: If the task satisfies the constraint conditions, the earliest allowed start time of the task is selected, the task is placed in a schedulable scheme, and the execution order and time window of the task are determined.

[0055] The present invention further discloses a computer storage medium on which a computer program is stored. When the computer program is executed, the above-mentioned inter-satellite link satellite measurement, control and scheduling method based on double-layer reinforcement learning is implemented.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] This method constructs an integer programming model for the satellite measurement and control scheduling model of the intersatellite link. Based on a two-layer deep reinforcement learning algorithm, the upper layer optimizes the satellite measurement and control task allocation problem, and the lower layer optimizes the single antenna task scheduling problem to improve the quality and speed of the solution. Combined with the heuristic algorithm theory, the collaborative mechanism and the "reinforcement learning + heuristic" mechanism are studied to build an information interaction mechanism, etc., to maximize the utilization of resources and maximize the overall efficiency of the star cluster, and form a solution framework based on reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0059] Figure 1 Example of building a link between satellite and ground station antenna;

[0060] Figure 2 This is a flow chart of the inter-satellite link satellite measurement and control scheduling method based on double-layer reinforcement learning in this embodiment;

[0061] Figure 3is a block diagram of a two-layer reinforcement learning scheduling model of this embodiment;

[0062] Figure 4a-4c Friedman test for two-layer reinforcement learning and comparison algorithms on different scales. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0064] Example

[0065] like Figure 2 As shown, a satellite measurement and control scheduling method for inter-satellite links based on double-layer reinforcement learning includes the following steps:

[0066] S01: establishing an inter-satellite link satellite measurement and control scheduling model, wherein the optimization goal of the inter-satellite link satellite measurement and control scheduling model is to maximize the total benefit of the task;

[0067] S02: Allocate all tasks to the ground station antenna according to the support relationship between the requested tasks and the ground station antenna through the upper-layer optimization model;

[0068] S03: The lower-level optimization model calculates the task benefit return and the next step status under the action through the task allocation plan provided by the upper-level optimization model, realizes the information feedback interaction between the upper-level optimization model and the lower-level optimization model, and the two-layer optimization process is iterated multiple times. The lower-level single antenna scheduling result is fed back to the upper layer to guide the next satellite measurement and control task allocation and determine the time window for the execution of each task.

[0069] In a preferred embodiment, the satellite measurement and control scheduling model of the inter-satellite link is:

[0070]

[0071] The constraints are as follows:

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] in, is the decision variable for whether task m is executed, p m is the revenue value of task m, R is the set of low-orbit satellite measurement and control scheduling tasks, AS is the satellite set, ALS i is the set of satellites that can establish a link with satellite i, AG ij is the set of optional ground stations for satellite j to establish a link, Aa ijk is the antenna set of ground station k, AW mijkn is the set of optional time windows for task m on antenna n, slt ij is the start time of establishing the link between satellite i and satellite j, elt ij is the end time of the inter-satellite link between satellite i and satellite j, is the start time of the th time window, is the end time of the th time window, est m is the earliest allowed start time of the task, lst m The latest allowed end time of the task. is the actual start time of the task, is the actual end time of the task, st m is the service time required for task m.

[0082] In a preferred embodiment, step S02 includes modeling the task allocation process of the upper optimization model into a Markov decision process, wherein the Markov decision process includes a 4-tuple M={S, A, τ, r}, where S represents the state space, A represents the action space, τ represents the state transition rule, and r represents the benefit function;

[0083] In the state space S, the state s at each step is t ∈S represents the time window for task selection;

[0084] Action space A includes selecting tasks and selecting ground station antennas, which is used to select a suitable and executable ground station antenna for each task. A task and an antenna are selected at each step, and the task and antenna need to satisfy a supportable relationship.

[0085] Set the transition rule τ, the next time t state s t+1 According to action a t and the previous state s tUpdate, that is, s t+1 =τ(s t );

[0086] The benefit r is used to calculate the benefit value of the task after taking a certain action. The benefit in this model is the increment of the total benefit after taking the action. The calculation formula is:

[0087] r t+1 = r(s t+1 , a t+1 )=pr i

[0088] where pr i It is the revenue of the task.

[0089] In a preferred embodiment, the selection probability of the action in the above steps is selected by using the ∈-greedy strategy, and the maximum value of the state is used as the distribution function of the selection probability.

[0090] In a preferred embodiment, the time window for executing each task is determined by a rule-based heuristic algorithm, wherein the rule-based heuristic algorithm includes:

[0091] S31: Get the task T assigned to the ground station antenna:

[0092] S32: sorting all the tasks to be planned in descending order according to the task benefits;

[0093] S33: If the task satisfies the constraint conditions, the earliest allowed start time of the task is selected, the task is placed in a schedulable scheme, and the execution order and time window of the task are determined.

[0094] In a preferred embodiment, the method for implementing information feedback interaction between the upper optimization model and the lower optimization model in step S03 includes:

[0095] The task allocation schemes corresponding to the single antenna scheduling with benefits higher than the set threshold are recorded, and an elite set is used to store several task allocation schemes with the highest benefits encountered in history. When a scheme appears repeatedly, its weight is increased; in the subsequent satellite measurement and control task allocation, a certain number of allocation schemes are extracted from the elite set according to the weights.

[0096] In another embodiment, a computer storage medium stores a computer program, and when the computer program is executed, the intersatellite link satellite measurement and control scheduling method based on double-layer reinforcement learning is implemented. The specific method is consistent with the intersatellite link satellite measurement and control scheduling method based on double-layer reinforcement learning described above, and will not be repeated here.

[0097] Another embodiment, an intersatellite link satellite measurement, control and scheduling system based on double-layer reinforcement learning, includes:

[0098] An inter-satellite link satellite measurement and control scheduling model building module is used to establish an inter-satellite link satellite measurement and control scheduling model, wherein the optimization goal of the inter-satellite link satellite measurement and control scheduling model is to maximize the total benefit of the task;

[0099] The upper layer optimization module allocates all tasks to the ground station antenna according to the support relationship between the requested tasks and the ground station antenna through the upper layer optimization model;

[0100] The lower-layer optimization module calculates the task benefit return and the next state of the action through the task allocation plan provided by the upper-layer optimization model, realizes the information feedback interaction between the upper-layer optimization model and the lower-layer optimization model, and the two-layer optimization process is iterated multiple times. The lower-layer single antenna scheduling result is fed back to the upper layer to guide the next satellite measurement and control task allocation and determine the time window for the execution of each task.

[0101] The following is a detailed description of the workflow of the intersatellite link satellite measurement and control scheduling system based on double-layer reinforcement learning using an optimal example:

[0102] To address the problem of lack of standardized mathematical models and reinforcement learning optimization algorithms for inter-satellite link satellite measurement and control scheduling, 1) an integer programming model for inter-satellite link satellite measurement and control scheduling was established, and 2) a two-layer reinforcement learning scheduling algorithm was provided.

[0103] 1. Establishment of satellite measurement and control scheduling model for intersatellite links based on two-layer reinforcement learning:

[0104] The variables and symbols involved in the satellite tracking and control scheduling model of the inter-satellite link are shown in Table 1.

[0105] Table 1 Symbols and variables for satellite tracking and control scheduling problem of intersatellite links

[0106]

[0107] The optimization goal of the measurement and control of intersatellite links is to maximize the total benefit of the mission. The mathematical model is defined as follows:

[0108]

[0109] The constraints are as follows:

[0110]

[0111] Constraint (2) means that each task can be executed at most once;

[0112] Constraint (3) means that the mission first establishes the inter-satellite link and then starts the measurement and control;

[0113] Constraint (4) indicates that the task needs to be completed before the chain building ends;

[0114] Constraint (5) means that the satellite-ground link is established only after the intersatellite link is established;

[0115] Constraint (6) indicates that the inter-satellite link is completed only after the satellite-ground link is completed.

[0116] Constraint (7) represents the relationship between the task start time and end time.

[0117] Constraint (8) indicates that the task needs to start after the start time of the visible time window.

[0118] Constraint (9) indicates that the task needs to be completed before the end time of the visible time window.

[0119] Constraint (10) represents the value range of the variable.

[0120] The inter-satellite link satellite tracking and control scheduling problem is a complex task planning problem, which not only needs to consider the inter-satellite link establishment between satellites, but also the constraints and task scale. As the task scale expands, the solution space grows exponentially. A new algorithm is needed to improve the quality and speed of the solution. This puts higher requirements and challenges on the solution algorithm. Therefore, this paper proposes a two-layer deep reinforcement learning algorithm to deal with the inter-satellite link satellite tracking and control scheduling problem. The core idea of ​​the algorithm is to decompose the satellite tracking and control scheduling problem into two steps. Since each step has different focuses on solving the problem, different processing strategies need to be adopted.

[0121] The specific modeling is:

[0122] The first step is to assign all tasks to the ground station antennas based on the support relationship between the requested tasks and the ground station antennas. This process can be defined as task allocation, which requires assigning unplanned tasks to specific antennas. In this part of the solution process, deep reinforcement learning is used to achieve the purpose of task allocation. The advantage of using an allocation strategy is that it decomposes large-scale measurement and control scheduling problems into small-scale problems. This not only reduces the difficulty of solving the algorithm, but also reduces the scale of the solution.

[0123] In the second step, all planned tasks are assigned to different ground station antennas to determine which ground station antenna the task is executed on. There are multiple executable time windows on each ground station antenna, which can support the execution of multiple satellite measurement and control scheduling tasks. Therefore, the solution of the second step uses a rule-based heuristic algorithm to determine the time window for each task execution.

[0124] For the first step, the task allocation process is modeled as a Markov decision process (MDP). The Markov decision process consists of a 4-tuple M = {S, A, τ, r}, where S represents the state space, A represents the action space, τ represents the state transition rule, and r represents the profit function. Below I will introduce the definition of each tuple in detail.

[0125] State space S: In the state space S, the state s at each step t ∈S represents the time window for task selection.

[0126] Action space A: Action space A includes selecting tasks and selecting ground station antennas. It is used to select a suitable and executable ground station antenna for each task. A task and an antenna are selected at each step, and the task and antenna need to satisfy a supportable relationship.

[0127] State transition rule τ: According to the transition rule τ, the state s at the next moment t+1 According to action a t and the previous state t Update, that is, s t+1 =τ(s t ).

[0128] Revenue r: Usually used to calculate the revenue value of a task after taking a certain action. The larger the revenue value, the better the action taken. The revenue in this model is the increment of the total revenue after taking the action, and the formula is:

[0129] r t+1 = r (s t+1 , a t+1 ) = pr i (11)

[0130] where pr i It is the revenue of the task.

[0131] Note: The selection probability of the action in the above steps is selected by using the ∈-greedy strategy, that is, the maximum value of the state is the distribution function of the selection probability.

[0132] The scheduling process of two-layer reinforcement learning can be summarized as:

[0133] In the two-layer optimization model, the upper layer optimizes the satellite measurement and control task allocation problem, and the lower layer optimizes the single antenna task scheduling problem. The two-layer optimization process iterates multiple times, and the lower-layer single antenna scheduling results are fed back to the upper layer to guide the next satellite measurement and control task allocation, seeking continuous optimization of the solution to the main problem. The feedback mechanism from the lower layer to the upper layer is the key to the two-layer optimization model. Since the lower-layer single antenna scheduling schemes are all generated based on the upper-layer task allocation schemes, the task allocation schemes corresponding to the high-yield single antenna scheduling can be recorded. An elite set is used to store several task allocation schemes with the highest benefits encountered in history. When a certain scheme appears repeatedly, its weight is increased; when allocating subsequent satellite measurement and control tasks, a certain number of allocation schemes are extracted from the elite set according to the weights. The scheduling process of the two-layer model is as follows: Figure 3 As shown in the figure, first, the satellite scheduling task is modeled as an encoding-decoding structure. According to the characteristics of the task and the characteristics of the antenna, the task is assigned to different antennas, that is, the upper model is modeled as an intelligent agent to realize the measurement and control task allocation and reduce the scale of task solving. When the task is assigned, different antennas can perform different tasks. Then, the heuristic measurement and control scheduling strategy is used to determine the start and end time of the task on each antenna. The main function of the lower model is to model an environment to complete the measurement and control scheduling of the intersatellite link, which needs to meet the constraints and maximize the task benefits. The upper model selects actions based on historical data and value functions and provides a task allocation plan. The lower model uses the heuristic scheduling algorithm to calculate the task benefit return and the next step state under the task allocation plan provided by the upper model, thereby realizing the information feedback interaction between the upper model and the lower model, and quickly finding a better scheduling plan. Therefore, we explore the application of reinforcement learning in intersatellite link satellite measurement, control and scheduling, combine it with the theory of heuristic algorithms, study the collaborative mechanism and the "reinforcement learning + heuristic" mechanism to build an information interaction mechanism, etc., to maximize the utilization of resources and maximize the overall efficiency of the star cluster, and form a solution framework based on reinforcement learning.

[0134] Algorithm 3.2 provides a pseudo code of the heuristic algorithm. This heuristic algorithm first sorts all the tasks to be scheduled in descending order according to the task benefits. The main purpose of this method is to ensure that the tasks with the largest benefits are executed first, so as to obtain a better overall task benefit value. If the task meets the constraints, the earliest start time allowed for the task is selected. Then, the task is placed in the schedulable scheme.

[0135] Using this heuristic algorithm, the execution order and time window of tasks can be determined.

[0136]

[0137] Experimental setup

[0138] The satellite telemetry scheduling experimental scenario for intersatellite links is generated by STK 11.2.0. There are three types of experimental scenarios: small-scale scenarios, medium-scale scenarios, and large-scale scenarios. The task scale of small-scale scenarios is between 100 and 300, the task scale of medium-scale scenarios is between 400 and 600, and the task scale of large-scale scenarios is between 800 and 1000. For each task scenario, two scenarios of low density and high density are set. All low-density scenarios are distributed within 24 hours, small-scale tasks with high-density instances are distributed within 9 hours, medium-scale tasks with high-density instances are distributed within 15 hours; large-scale tasks with high-density instances are distributed within 21 hours. For convenience, the "AB" format is used to represent scenario instances, "A" represents task scale, and "B" represents density. For density, "1" represents low density and "2" represents high density.

[0139] In order to verify the effectiveness of the proposed algorithm, three classic algorithms were selected as comparison algorithms. These three comparison algorithms are: Knowledge-based Evolutionary Algorithm (KBGA), Simulated Annealing (SA) and Adaptive Large Neighborhood Search (ALNS).

[0140] Comparison results with other algorithms

[0141] In order to test the overall performance difference between the DQN / EA algorithm and the three classic algorithms, this paper adopts the Friedman test method to count all task scenarios according to the task scale. The statistical results are as follows: Figure 4a-4c As shown, the lower the ranking, the better the performance of the algorithm. The algorithm rankings of the two-layer reinforcement learning algorithm in small-scale, medium-scale and large-scale scenarios are 1.1, 1.2 and 1 respectively. This shows that the two-layer reinforcement learning algorithm has a greater advantage in processing satellite measurement and control scheduling algorithms. The rankings of the KBGA algorithm in small-scale, medium-scale and large-scale are 2.2, 1.9 and 2 respectively, indicating that the KBGA algorithm ranks second in the performance comparison of the four algorithms. The rankings of the ALNS algorithm in the three task scales are 2.9, 2.9 and 3 respectively, indicating that the ALNS algorithm ranks third in the performance comparison of the four algorithms. The SA algorithm ranks fourth in all task scale scenarios, indicating that SA has the worst performance. Possible reasons are: 1) The two-layer reinforcement learning algorithm uses reinforcement learning to reduce the problem scale, and the heuristic algorithm quickly obtains an approximate solution. The algorithm uses the advantages of the two algorithms and is more effective for the satellite measurement and control scheduling problem of intersatellite links. 2) The reinforcement learning algorithm is used to train the characteristics of the task, so that the execution order of the scheme is more reasonable and the purpose of higher benefits is achieved.

[0142] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.

Claims

1. A satellite measurement and control scheduling method for intersatellite links based on double-layer reinforcement learning, characterized in that: The following steps are involved: S01: establishing an inter-satellite link satellite measurement and control scheduling model, wherein the optimization goal of the inter-satellite link satellite measurement and control scheduling model is to maximize the total benefit of the task; S02: Allocate all tasks to the ground station antenna according to the support relationship between the requested tasks and the ground station antenna through the upper-layer optimization model; S03: The lower-level optimization model calculates the task benefit return and the next step status under the action through the task allocation plan provided by the upper-level optimization model, realizes the information feedback interaction between the upper-level optimization model and the lower-level optimization model, and the two-layer optimization process is iterated multiple times. The lower-level single antenna scheduling result is fed back to the upper layer to guide the next satellite measurement and control task allocation and determine the time window for the execution of each task.

2. The intersatellite link satellite measurement and control scheduling method based on double-layer reinforcement learning according to claim 1 is characterized in that: The intersatellite link satellite measurement and control scheduling model is: The constraints are as follows: in, is the decision variable for whether task m is executed, p m is the revenue value of task m, R is the set of low-orbit satellite measurement and control scheduling tasks, AS is the satellite set, ALS i is the set of satellites that can establish a link with satellite i, AG ij is the set of optional ground stations for satellite j to establish a link, Aa ijk is the antenna set of ground station k, AW mijkn is the set of optional time windows for task m on antenna n, slt ij is the start time of establishing the link between satellite i and satellite j, elt ij is the end time of the inter-satellite link between satellite i and satellite j, is the start time of the lth time window, is the end time of the lth time window, est m is the earliest allowed start time of the task, lst m The latest allowed end time of the task. is the actual start time of the task, is the actual end time of the task, st m is the service time required for task m.

3. The intersatellite link satellite measurement and control scheduling method based on double-layer reinforcement learning according to claim 1 is characterized in that: Step S02 includes modeling the task allocation process of the upper optimization model into a Markov decision process, where the Markov decision process includes a 4-tuple M={S,A,τ,r}, where S represents the state space, A represents the action space, τ represents the state transition rule, and r represents the benefit function; In the state space S, the state s at each step is t ∈S represents the time window for task selection; Action space A includes selecting tasks and selecting ground station antennas, which is used to select a suitable and executable ground station antenna for each task. A task and an antenna are selected at each step, and the task and antenna need to satisfy a supportable relationship. Set the transition rule τ, the next time t state s t+1 According to action a t and the previous state s t Update, that is, s t+1 =τ(s t ); The benefit r is used to calculate the benefit value of the task after taking a certain action. The benefit in this model is the increment of the total benefit after taking the action. The calculation formula is: r t+1 =r(s t+1 ,a t+1 )=pr i where pr i It is the revenue of the task.

4. The intersatellite link satellite measurement and control scheduling method based on double-layer reinforcement learning according to claim 2 is characterized in that: The selection probability of the action in the above steps is selected by using the ∈-greedy strategy, taking the maximum value of the state as the distribution function of the selection probability.

5. The intersatellite link satellite measurement and control scheduling method based on double-layer reinforcement learning according to claim 1 is characterized in that: The time window for executing each task is determined by a rule-based heuristic algorithm, wherein the rule-based heuristic algorithm includes: S31: Get the task T assigned to the ground station antenna: S32: sorting all the tasks to be planned in descending order according to the task benefits; S33: If the task satisfies the constraint conditions, the earliest allowed start time of the task is selected, the task is placed in a schedulable scheme, and the execution order and time window of the task are determined.

6. The intersatellite link satellite measurement and control scheduling method based on double-layer reinforcement learning according to claim 1 is characterized in that: The method for implementing information feedback interaction between the upper optimization model and the lower optimization model in step S03 includes: The task allocation schemes corresponding to the single antenna scheduling with benefits higher than the set threshold are recorded, and an elite set is used to store several task allocation schemes with the highest benefits encountered in history. When a scheme appears repeatedly, its weight is increased; in the subsequent satellite measurement and control task allocation, a certain number of allocation schemes are extracted from the elite set according to the weights.

7. An intersatellite link satellite measurement and control scheduling system based on double-layer reinforcement learning, characterized in that: include: An inter-satellite link satellite measurement and control scheduling model building module is used to establish an inter-satellite link satellite measurement and control scheduling model, wherein the optimization goal of the inter-satellite link satellite measurement and control scheduling model is to maximize the total benefit of the task; The upper layer optimization module allocates all tasks to the ground station antenna according to the support relationship between the requested tasks and the ground station antenna through the upper layer optimization model; The lower-layer optimization module calculates the task benefit return and the next state of the action through the task allocation plan provided by the upper-layer optimization model, realizes the information feedback interaction between the upper-layer optimization model and the lower-layer optimization model, and the two-layer optimization process is iterated multiple times. The lower-layer single antenna scheduling result is fed back to the upper layer to guide the next satellite measurement and control task allocation and determine the time window for the execution of each task.

8. The intersatellite link satellite measurement, control and scheduling system based on double-layer reinforcement learning according to claim 7 is characterized in that: The upper-level optimization module includes modeling the task allocation process of the upper-level optimization model into a Markov decision process, where the Markov decision process includes a 4-tuple M={S,A,τ,r}, where S represents the state space, A represents the action space, τ represents the state transition rule, and r represents the benefit function; In the state space S, the state s at each step is t ∈S represents the time window for task selection; Action space A includes selecting tasks and selecting ground station antennas, which is used to select a suitable and executable ground station antenna for each task. A task and an antenna are selected at each step, and the task and antenna need to satisfy a supportable relationship. Set the transition rule τ, the next time t state s t+1 According to action a t and the previous state s t Update, that is, s t+1 =τ(s t ); The benefit r is used to calculate the benefit value of the task after taking a certain action. The benefit in this model is the increment of the total benefit after taking the action. The calculation formula is: r t+1 =r(s t+1 ,a t+1 )=pr i where pr i It is the revenue of the task.

9. The intersatellite link satellite measurement, control and scheduling system based on double-layer reinforcement learning according to claim 7, characterized in that: The time window for executing each task is determined by a rule-based heuristic algorithm, wherein the rule-based heuristic algorithm includes: S31: Get the task T assigned to the ground station antenna: S32: sorting all the tasks to be planned in descending order according to the task benefits; S33: If the task satisfies the constraint conditions, the earliest allowed start time of the task is selected, the task is placed in a schedulable scheme, and the execution order and time window of the task are determined.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the inter-satellite link satellite measurement, control and scheduling method based on double-layer reinforcement learning described in any one of claims 1 to 6 is implemented.

Images