Relay satellite military-civilian fusion multi-target scheduling method based on deep reinforcement learning

Abstract

The invention discloses a relay satellite military and civilian fusion multi-target scheduling method based on deep reinforcement learning. The method specifically comprises the following steps: step 1, collecting data of a user task set and a relay satellite resource set; step 2, establishing a mathematical model according to the data collected in the step 1 by following a relay satellite scheduling principle; step 3, realizing a process of scheduling the mathematical model established in the step 2 through a Markov decision process; and step 4, optimizing the decision process in the step 3 through deep reinforcement learning to realize relay satellite military-civilian fusion multi-target scheduling. The problems that the scheduling center conflicts in civil and military major project scheduling, user application and execution are not timely, and the utilization rate of relay satellite resources is low are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of relay satellite resource scheduling, and relates to a relay satellite military-civilian integration multi-objective scheduling method based on deep reinforcement learning. Background Art

[0002] Relay satellites, the core hub of space communications, provide communication and control support for numerous spacecraft by establishing long-distance, high-speed data transmission channels between satellites and between satellites and the ground. With advantages such as high network coverage, effective mission execution, strong data transmission capabilities, and high cost-effectiveness, relay satellites have been widely used in aerospace communications, military reconnaissance, disaster response, and other fields. They can also provide connectivity support to remote areas with insufficient infrastructure, enabling real-time communication at any time and from any location.

[0003] The current bottleneck in relay satellite scheduling lies primarily in the conflict between limited relay satellite link resources and the demands of multiple missions. This is manifested in the fierce competition for time, space, and spectrum resources in the scheduling conflicts between major civil and military projects. This creates a significant challenge for the dispatch center in balancing the interests of all parties, while also addressing issues such as delayed execution of user applications and low resource utilization. The dispatch center is primarily concerned with efficiently and fairly allocating relay satellite resources, improving overall scheduling efficiency and resource utilization, and minimizing mission execution lags to meet the diverse needs of different users, all while ensuring the priority completion of major national projects such as military missions.

[0004] However, current research on relay satellite resource scheduling, both domestically and internationally, focuses on improving the efficiency of scheduling schemes, increasing relay satellite resource utilization, and accelerating algorithm convergence. However, this approach oversimplifies the relay satellite scheduling problem model, focusing on the single objective of maximizing system benefits while neglecting important objectives such as balancing the priorities of major national scheduling tasks, such as military missions, with those of civilian missions and user experience. Furthermore, traditional optimization algorithms, such as the simplex method, nonlinear programming, and heuristic algorithms, are highly dependent on precise mathematical models and struggle to handle dynamically changing resource and mission requirements when solving relay satellite scheduling problems. They are prone to falling into local optimal solutions, suffer from slow convergence, and exhibit high computational complexity. Furthermore, they lack the flexibility to adapt to real-time changes and struggle to effectively balance multi-objective optimization requirements, resulting in an inability to efficiently and accurately complete scheduling tasks. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning, which solves the problems of scheduling conflicts between major civil and military projects in the scheduling center, untimely execution of user applications, and low utilization of relay satellite resources.

[0006] The technical solution adopted by the present invention is a multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning, which specifically includes the following steps: Step 1: Collect data of user mission set and relay satellite resource set; Step 2: Establish a mathematical model based on the data collected in step 1 and the relay satellite scheduling principle; Step 3: Implement the scheduling process of the mathematical model established in step 2 through the Markov decision process; Step 4: Optimize the decision-making process in step 3 through deep reinforcement learning to achieve multi-objective scheduling of relay satellites for military-civilian integration.

[0007] The present invention is also characterized in that: The specific process of step 1 is as follows: Step 1.1: The set of all tasks applied by the user is J , is the total number of tasks; i The set of priorities , , ;Task i The income composition of the collection , ;Task i The duration of the collection D , ; User's task i The application start time and deadline constitute the user demand window , ; Step 1.2: Group all the links in the relay satellite into a set , ; The relay satellite scheduling period is , ;link The idle time window set is used express; ; Executable tasks i The set of links is L i , Indicates a task i Can the link k to execute; Step 1.3, Task i and Link k Under theoretical conditions, the visible time window constitutes a set of , ;Task i and Link k The available time windows constitute the set , ;link k The set of transition times between the above tasks is .

[0008] The specific process of step 2 is as follows: Step 2.1: The tasks selected for scheduling form a set , , Indicates a task has been selected, It means the task Not selected; Step 2.2, Task i In the link k If the above is successfully called, Otherwise, use express; Step 2.3, Task i In the link k After being successfully called, the task is executed immediately j ,but ,otherwise ; Step 2.4, Task i In the link k On the w available time windows are successfully executed, then Otherwise, use express; Step 2.5, Task i The constraints satisfied during the execution process are shown in formulas (1) to (9). Indicates a task i Start time: (1) (2) (3) (4) (5) (6) (7) (8) (9) Step 2.6, calculate the sum of the benefits of executing the task, as shown in formula (10): (10) Step 2.7, calculate the lag time of task execution, as shown in formula (11): (11) Step 2.8, the priority of the task changes; Step 2.9, Relay Satellite Link As the scheduling task is executed, its free time window is constantly occupied. changes occur; Step 2.10, judgment task Time to start execution Whether the relay satellite scheduling period has been exceeded. If so, the link selection and task selection are terminated, as shown in formula (12): (12) Step 2.11, if the task set If all tasks in have been selected, the link selection and task selection are terminated, as shown in formula (13): (13).

[0009] The specific process of step 3 is as follows: Step 3.1, input the state data of the Markov decision process, input Time step, the state of the entire space ,in Indicates the status of the link. Indicates the status of the task; Step 3.2, select the link and task. In the process of relay satellite scheduling, as time steps For movement, the agent first selects the execution link according to the link status and task status, and then selects the execution task on the execution link; Step 3.3, calculate the reward value of the selected task , the reward function is shown in formula (14): (14) Step 3.4: As the task is executed, the spatial state changes as shown in formula (15): (15) Step 3.5, Link The set of free time windows Transformed into ,link The collection of executed tasks Transformed into ,link The start time collection of executed tasks Transformed into ,link The sum of the benefits of the executed tasks is given by Transformed into ,link The average delay time of the executed tasks is given by Transformed into ; Step 3.6, Task Profit and execution time As the time step progresses, the task In the link The available time window on Time step Transformed into Time step ,Task The priority is determined by Time step Transformed into Time step .

[0010] The specific process of step 3.1 is as follows: Step 3.1.1, Link Status As shown in formula (16): (16) in, Indicates a link exist The set of free time windows in the time step, ; Indicates Time step, link The set of executed tasks; Indicates Time step, link The set of start times of executed tasks; Indicates Time step, link The sum of the benefits of executed tasks is shown in formula (17): (17) Indicates Time step, link The average delay time of executed tasks is shown in formula (18): (18) Step 3.1.2, Task Status As shown in formula (19): (19) in, Indicates a task exist Time steps and links The set of available time windows on , , Indicates time step, task i and Link k No. w The start time of the available time window, It indicates the end time; Indicates a task Execution time; Indicates a task Execution proceeds; Indicates a task exist Priority of the time step.

[0011] The specific process of step 4 is as follows: Step 4.1, create a memory pool; Step 4.2: Initialize the memory pool. Before training begins, pre-fill the memory pool with a certain amount of initial experience data using a random strategy. Step 4.3, obtain the link status and task status under the current situation from the environment; Step 4.4, select the maximum output of the current network The action a corresponding to the value is learned using greedy learning To balance exploration and exploitation; Step 4.5, execute the selected action a, obtain the reward R and the next state ;use To indicate the status Select an action What you get The value is used as the experience for future action selection, as shown in formula (20): (20) Step 4.6, memory storage; Step 4.7, update of memory pool; Step 4.8: Build a neural network. The estimated neural network has the same structure as the target neural network. Input the spatial state, including the link state and the task state, and output the state of all actions in this state. value; Step 4.9, initialize the estimated network parameters and target network parameters ; Step 4.10, optimize the parameters of the neural network; Step 4.11, randomly select a batch of data sets from the memory pool to train the estimated neural network; Step 4.12, use the Bellman equation to calculate the target The value is shown in formula (21): (twenty one) Step 4.13, calculate the estimated value: ; Step 4.14, calculate the loss value through the loss function, as shown in formula (22): (twenty two) Step 4.15, the loss value As the error is passed back, the parameters are adjusted by gradient descent Make updates; Step 4.16, regularly update the parameters of the target neural network, ; Step 4.17, save the structure and parameters of the target neural network and the estimated neural network; In step 4.18, the new task status and link status are input into the trained neural network to obtain the final scheduling solution.

[0012] In step 4.8, the neural network includes an estimation neural network and a target neural network. The estimation neural network predicts the current Q value of each action based on the current link and task status, and calculates the maximum target Q value through the target neural network. The network structure of the estimation neural network and the target neural network is exactly the same, including 1 input layer, 3 hidden layers and 1 output layer; among them, the input layer outputs the input space state, including the link state and task state ; First hidden layer input , output , whose activation function is ReLU; the second hidden layer input , output , whose activation function is ReLU; the third hidden layer input , output , whose activation function is ReLU; the output layer outputs all actions in this state value .

[0013] In step 4.13, the Bellman equation is used to calculate the target The value is shown in formula (23): (twenty three).

[0014] The beneficial effect of this invention is that it fully considers the dispatch center's concerns about conflicts between civil tasks and major national projects, such as military ones, and the user experience. It designs a mathematical model for scheduling tasks with variable priority and incorporates the lag time of task execution into the objective function. This ensures that major national projects, such as military ones, are prioritized and improves the user experience. It enables the dispatch center to more flexibly allocate relay satellite resources, promoting a virtuous cycle of unified satellite scheduling and control. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A diagram of a relay satellite network scenario for the application of the relay satellite civil-military integration multi-objective scheduling method based on deep reinforcement learning of the present invention; Figure 2 This is a path transfer diagram between different satellites for the link of the multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning of the present invention; Figure 3 This is a Markov decision process diagram of the multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning of the present invention; Figure 4 This is the DQN network algorithm framework of the relay satellite civil-military integration multi-objective scheduling method based on deep reinforcement learning in the present invention; Figure 5 The Gantt chart is adjusted for the final overall task of the relay satellite civil-military integration multi-objective scheduling method based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] The present invention is based on the multi-objective scheduling method of relay satellite civil-military integration based on deep reinforcement learning. The relay satellite network scene diagram is divided into three layers, such as Figure 1 As shown in the figure, the backbone layer is composed of a constellation of relay satellites, responsible for data reception, processing, and global forwarding. The user layer includes various low-orbit satellites, aircraft, rocket launch terminals, and other terminals, which enable cross-regional communication through relay satellites. The ground layer includes ground control centers, receiving stations, and user terminals, responsible for satellite management, data processing, and information distribution. These three layers collaborate to build an efficient global communication network. The advent of relay satellites has not only significantly improved the utilization efficiency of various satellites, but also greatly reduced the number of ground stations, thereby reducing the overall cost of satellite communication networks. The main steps of this method include modeling relay satellite scheduling through a Markov decision process and solving the model based on deep reinforcement learning to obtain the optimal scheduling solution.

[0018] Example 1 The present invention provides a multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning, which comprises the following steps: Step 1: Collect data of user mission set and satellite resource set.

[0019] Step 2: Based on the data collected in Step 1, a mathematical model is established according to the relay satellite scheduling principles. First, antennas are selected, then tasks are selected to determine the task order. When scheduling tasks, high-priority tasks are prioritized, and then execution is performed within available time windows that meet their constraints.

[0020] Step 3: Use the Markov decision process to describe the scheduling process of the mathematical model established in step 2.

[0021] Step 4: Optimize the decision-making process in step 3 through deep reinforcement learning to obtain the optimal scheduling solution.

[0022] Example 2 The specific process of step 1 is: Collect data on user task sets. The set of all tasks applied by the user is J , , is the total number of tasks; i The set of priorities , , p i The larger the value, the more tasks i The higher the priority, ,like , it means that within the user's required window period, the task i must be performed; task i The income composition of the collection , , The larger the value, the greater the benefit of executing this task. i The duration of the collection D , ; User's task i The application start time and deadline constitute the user demand window , , Indicates the user's task i The requested start time, Indicates the user's task i The required deadline. In this method, the user's specific task data is shown in Table 1.

[0023] Collect data from the satellite resource set. In this method, the relay satellite is Tianlian-1 01, and the user spacecraft includes various types of spacecraft such as meteorological, reconnaissance, and navigation. The user spacecraft orbital parameters are shown in Table 2. Assume that the relay satellite has three links, and the idle time windows of the three links are exactly the same. All the links in the relay satellite are grouped into a set , ; The relay satellite scheduling period is , ,In this method, the scheduling time of relay satellite is from April 25, 2025 to April 26, 2025;,Link The idle time window set is used express, ,link No. The idle time window starts at , and the end time is ; Executable tasks i The set of links is , Indicates a task i Can the link k To execute, if the task i Link k To execute, ,otherwise, , as shown in formula (1): (1) Task i and Link k The time period during which stable communication can be maintained during orbital operation constitutes the visible time window set , ,Task i and Link k No. n The start time of the visible time window is , and the end time is In this method, the visible time windows of the relay satellite and the user spacecraft are shown in Tables 3 to 9. i Demand Window ,link k The set of idle time windows and tasks i and Link k The set of visible time windows Composing user tasks i and Link k The available time windows constitute the set , ,Task i and Linkk No. w The available time windows start at , and the end time is ;link k The transition between the above tasks requires a certain amount of adjustment time, such as Figure 2 As shown in FIG, it is the path transfer diagram of the link between different satellites of the present invention, the solid line represents the task execution time period, and the dotted line represents the switching time. k The set of transition times between the above tasks is , columns represent tasks j , rows represent tasks i ,Task i With the task j Between links k Adjustment time: ,in , The expression of is shown in formula (2): (2) Example 3 The specific process of step 2 is as follows: Step 2.1: The tasks selected for scheduling form a set , , Indicates a task i It has been selected, and whether it is successfully scheduled for execution in this set is not of concern. It means the task i Not yet selected.

[0024] Step 2.2, Task i In the link k If the above is successfully called, Otherwise, use express.

[0025] Step 2.3, Task i In the link k After being successfully called, the task is executed immediately j ,but ,otherwise . Note that if the task The first one is executed, then Similarly, if the task The last one is executed, then .

[0026] Step 2.4, Task i In the link k On the w available time windows are successfully executed, then Otherwise, use express.

[0027] Step 2.5, use Indicates a task i The time when execution started.

[0028] Step 2.6, Task i It can only be executed on one link at most. The constraint expression is shown in formula (3): (3) Step 2.7: At the same time step, a link can only serve one task at most. Here, the order of link task execution is used to constrain the execution, that is, on a link, the task i After the execution is completed, execute the task j , such a situation will only occur once at most, and the constraint expression is shown in formula (4): (4) Step 2.8, successfully scheduled tasks i , can only be k An available time window is selected on the link to execute the task. The constraint expression is shown in formula (5): (5) Step 2.9, the task is successfully scheduled i ,exist k Select the first w Available time window execution, task i Start time No less than w The start time of the available time window , and ensure that within this visible time window, the task i Can be executed, the constraint expression is shown in formula (6): (6) Step 2.10, Task i Execution time At least the earliest required start time of the task After that, the constraint expression is shown in formula (7): (7) Step 2.11, Task i Execution time At least at the latest required end time of the task Before, and the time interval meets the task i After the execution is completed, the constraint expression is shown in formula (8): (8) Step 2.12, in the same link k , successively carry out tasks i and tasks j ,Task i The start time is ,Task j The start time is , ensuring that the time difference between the two meets the task i Execution is complete and the link can be k Switch to execution task j In the direction of , the constraint expression is shown in formula (9): (9) Step 2.13, Highest Priority Task i , must be executed within the user demand window. Avoid high-priority tasks being continuously abandoned due to long execution time, and instead frequently execute low-priority tasks with shorter execution time. The obtained collection , is the set of tasks with the highest priority, Indicates the highest priority, otherwise use It is expressed as shown in formula (10), and the constraint expression is shown in formula (11): (10) (11) Step 2.14, calculate the sum of the benefits of the executed tasks, as shown in formula (12): (12) In step 2.15, calculate the lag time of task execution. Considering user satisfaction, the interval between the user's requested time and the task execution end time should be shortened as much as possible. In other words, the timeliness of task execution is used to measure the quality of the scheduling solution, as shown in formula (13): (13) Step 2.16, the change of task priority. If the scheduled task and Link In its first available time window If it is not executed, its priority Add 1, use express, . It can be executed with a higher probability in the next available time window. In addition, there is a buffer zone between the priority of military and other major national project scheduling tasks and the priority of civil low-orbit satellite scheduling tasks. Here, the priority of military and other major national project scheduling tasks is , and the priority of civil low-orbit satellite scheduling tasks is the highest at the initial specified time. .

[0029] Step 2.17, Relay Satellite Link As the scheduling task is executed, its free time window is constantly occupied. To represent the link exist The set of free time windows in the time step, .

[0030] Step 2.18, judgment task Time to start execution Whether it exceeds the relay satellite scheduling period. If it has exceeded, the link selection and task selection are terminated; if it has not exceeded, the link selection and task selection continue, as shown in formula (14): (14) Step 2.19, if the task set If all tasks in have been selected, the link selection and task selection are terminated. If there are unselected tasks, the link selection and task selection continue, as shown in formula (15): (15) Example 4 The Markov decision process of the present invention is as follows Figure 3 shown.

[0031] Step 3.1, input the state data of the Markov decision process. Time step, the state of the entire space ,in Indicates the status of the link. Indicates the status of the task. Proceed as in steps 3.1.1 and 3.1.2.

[0032] Step 3.2, select the link and task. In the process of relay satellite scheduling, as time steps The main action of the agent is to first select the execution link and then select the execution task on the execution link. represents the action space, , Indicates that at time step Select Link To carry out the mission, Indicates that at time step Select Task On the selected link Execute on.

[0033] Step 3.3, calculate the reward value of the selected task The ultimate goal of this method is to maximize the benefits of scheduling tasks and minimize the delay time of task execution within the scheduling cycle. Task scheduling is done with To express, reward To express it, first normalize the two targets separately and then sum them. The reward function is shown in formula (20): (20) In step 3.4, as the task is executed, the state of the link and the state of the task also change, as shown in formula (21).

[0034] (twenty one) Step 3.5: The link status changes. The set of free time windows Transformed into ,link exist The set of free time windows in the time step should remove the link exist The execution time window of the task selected by the time step is shown in formula (22): (twenty two) Step 3.6, Link The collection of executed tasks Transformed into ,link exist The set of executed tasks at a time step needs to be linked exist The task performed at the time step, that is, the index of the task The value at position becomes 1, as shown in formula (23): (twenty three) Step 3.7, Link The start time collection of the executed tasks Transformed into ,link exist The set of executed task start times of the time step should be linked exist The start time of the task executed by the time step, that is, the index of the task The value at position becomes , as shown in formula (24): (twenty four) Step 3.8, Link The sum of the benefits of the executed tasks is given by Transformed into ,link exist The sum of the benefits of the executed tasks in the time step needs to be added The benefit of the task performed in the time step is shown in formula (25): (25) Step 3.9, Link The average delay time of the executed tasks is given by Transformed into ,link exist The average delay time of the executed tasks in the time step needs to be added first The delay time of the tasks executed in the time step is averaged again, as shown in formula (26): (26) Step 3.10, the task status changes, task Profit and execution time As the time step progresses, the task In the link The set of available time windows on ,because ,link The set of free time windows Transformed into , therefore, the task In the link The available time window on Time step Transformed into Time step .Task Priority As time steps The task priority is increased by 1 each time an available task window has not been called. The priority is determined by Time step Transformed into Time step .

[0035] Example 5 The specific process of step 3.1 is: Step 3.1.1, Link Status As shown in formula (27): (27) in Indicates a link exist The set of free time windows in the time step, ; Indicates Time step, link The set of executed tasks; Indicates Time step, link The set of start times of executed tasks; Indicates Time step, link The sum of the benefits of executed tasks is used to describe the quality of the task scheduling solution, as shown in formula (28): (28) Indicates Time step, link The average delay time of executed tasks is shown in formula (29): (29) Step 3.1.2, Task Status As shown in formula (30): (30) in Indicates a task exist Time steps and links The set of available time windows on , , Indicates time step, task i and Link k No. w The start time of the available time window, It indicates the end time; Indicates a task Execution time; Indicates a task Execution proceeds; Indicates a task exist Priority of the time step.

[0036] Example 6 This method specifically uses DQN (Deep Q-Network) to solve the problem. The DQN network algorithm framework of the present invention is as follows: Figure 4 The specific process is as follows: Step 4.1: Create a memory pool to record the benefits obtained by the agent from the selected tasks under different link state environments.

[0037] Step 4.2, initialize the memory pool. Before training begins, a certain amount of initial experience data is pre-filled into the memory pool through a random strategy. This method sets the capacity of the memory pool to 10,000.

[0038] Step 4.3, obtain the environment status: the agent interacts with the environment to obtain the link status and task status under the current circumstances.

[0039] Step 4.4, select action: According to the link status and task status obtained by interacting with the environment in step 4.5, select the maximum output in the current state. The action a corresponding to the value is the task to be performed. In order to avoid falling into the local optimum, greedy learning is used. To balance exploration and utilization. In this method, the exploration rate .

[0040] Step 4.5, execute the action: execute the selected action a, that is, the task to be performed, and obtain the reward value R and the next environmental state , The next step includes the link status and task status. To indicate the status Select an action What you get The value, that is, the reward value R, is used as the experience of future action selection, as shown in formula (31). It is learning efficiency, . is the decay value of future rewards, , The closer it is to 1, the greater the impact of future rewards on the present, that is, the better the agent's vision; conversely, the closer it is to 0, the worse the vision, and the learning rate of this method , attenuation value .

[0041] (31) Step 4.6, memory storage. Stored in the memory pool as experience for future action selection.

[0042] Step 4.7: Update the memory pool. When the memory pool reaches its maximum capacity, it will follow the first-in-first-out principle and cycle back to the oldest stored information, deleting the oldest stored content, and so on.

[0043] Step 4.8, build a neural network. The neural network in this method includes an estimation neural network and a target neural network. The estimation neural network predicts the current Q value of each action based on the current link and task state, and calculates the maximum target Q value through the target neural network. The network structure of the estimation neural network and the target neural network is exactly the same. The network structure is shown in Table 10, which includes 1 input layer, 3 hidden layers and 1 output layer. Among them, the input layer outputs the input space state, including the link state and task state. ; First hidden layer input , output , whose activation function is ReLU; the second hidden layer input , output , whose activation function is ReLU; the third hidden layer input , output , whose activation function is ReLU; the output layer outputs all actions in this state value .

[0044] Step 4.9, initialize the estimated network parameters and target network parameters In this method, both the estimation neural network and the target neural network are initialized using random initialization. At the beginning of the optimization, the network parameters are estimated. and target network parameters Exactly the same, that is .

[0045] Step 4.10: Optimize the parameters of the neural network. The number of training steps designed for this method is 100,000.

[0046] Step 4.11, randomly select a batch of data sets from the memory pool to train the estimated neural network. The batch size in this method is .

[0047] Step 4.12, estimate the neural network value: .

[0048] Step 4.13, get the target through the target neural network Value: This method uses the Bellman equation to calculate the target The value is shown in formula (32): (32) Step 4.14, the estimate calculated by the estimation neural network Value and target Neural network calculation target Value, use the loss function to calculate the loss value, as shown in formula (33): (33) Step 4.15, the loss value As the error is passed back, the parameters are adjusted by gradient descent Before backpropagation, the accumulated gradients are cleared. Adam is selected as the optimizer to optimize the gradients. The learning rate of the optimizer is 0.001, and the exponential decay rates of the first-order moment estimate and the second-order moment estimate are 0.9 and 0.99, respectively.

[0049] Step 4.16, regularly update the parameters of the target neural network. In this method, every 100 steps, the parameters of the estimated neural network are transferred to the target neural network. .

[0050] Step 4.17, save the structure and parameters of the target neural network and the estimated neural network.

[0051] In step 4.18, the new task state and link state are input into the trained neural network to obtain the final scheduling solution, such as Figure 5 As shown, adjust the Gantt chart for the final overall task.

[0052] Table 1 User task data

[0053] Table 2 Orbital parameters

[0054] Table 3 Visibility time window between relay satellite and user spacecraft COSMOS_1976

[0055] Table 4 Visibility time window between the relay satellite and user spacecraft BEIDOU-3_M3

[0056] Table 5 Visibility time window between relay satellite and user spacecraft GPS_2R-14

[0057] Table 6 Visibility time window between relay satellite and user spacecraft HST

[0058] Table 7 Visibility time window between relay satellite and user spacecraft ISS

[0059] Table 8 Visibility time window between relay satellite and user spacecraft LANDSAT_8

[0060] Table 9 Visibility time window between relay satellite and user spacecraft SOYUZ-MS_03

[0061] Table 10 Neural network structure

Claims

1. A multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning, characterized by: The specific steps include: Step 1: Collect data of user mission set and relay satellite resource set; Step 2: Establish a mathematical model based on the data collected in step 1 and the relay satellite scheduling principle; Step 3: Implement the scheduling process of the mathematical model established in step 2 through the Markov decision process; Step 4: Optimize the decision-making process in step 3 through deep reinforcement learning to achieve multi-objective scheduling of relay satellites for military-civilian integration.

2. The multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning according to claim 1 is characterized in that: The specific process of step 1 is as follows: Step 1.1: The set of all tasks applied by the user is J , , is the total number of tasks; i The set of priorities , , ;Task i The income composition of the collection , ; Task i The duration of the collection D , ; User's task i The application start time and deadline constitute the user demand window , ; Step 1.2: Group all the links in the relay satellite into a set , ; The relay satellite scheduling period is , ;link The idle time window set is used express; ; Executable tasks i The set of links is L i , Indicates a task i Can the link k to execute; Step 1.3, Task i and Link k Under theoretical conditions, the visible time window constitutes a set of , ; Task i and Link k The available time windows constitute the set , ;link k The set of transition times between the above tasks is .

3. The multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning according to claim 2 is characterized in that: The specific process of step 2 is as follows: Step 2.1: The tasks selected for scheduling form a set , , Indicates a task has been selected, It means the task Not selected; Step 2.2, Task i In the link k If the above is successfully called, Otherwise, use express; Step 2.3, Task i In the link k After being successfully called, the task is executed immediately j ,but ,otherwise ; Step 2.4, Task i In the link k On the w available time windows are successfully executed, then Otherwise, use express; Step 2.5, Task i The constraints satisfied during the execution process are shown in formulas (1) to (9). Indicates a task i Start time: （1） （2） （3） （4） （5） （6） （7） （8） （9） Step 2.6, calculate the sum of the benefits of executing the task, as shown in formula (10): （10） Step 2.7, calculate the lag time of task execution, as shown in formula (11): （11） Step 2.8, the priority of the task changes; Step 2.9, Relay Satellite Link As the scheduling task is executed, its free time window is constantly occupied. changes occur; Step 2.10, judgment task Time to start execution Whether the relay satellite scheduling period has been exceeded. If so, the link selection and task selection are terminated, as shown in formula (12): （12） Step 2.11, if the task set If all tasks in have been selected, the link selection and task selection are terminated, as shown in formula (13): （13）。 4. The multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning according to claim 3 is characterized by: The specific process of step 3 is as follows: Step 3.1, input the state data of the Markov decision process, input Time step, the state of the entire space ,in Indicates the status of the link. Indicates the status of the task; Step 3.2, select the link and task. In the process of relay satellite scheduling, as time steps For movement, the agent first selects the execution link according to the link status and task status, and then selects the execution task on the execution link; Step 3.3, calculate the reward value of the selected task , the reward function is shown in formula (14): （14） Step 3.4: As the task is executed, the spatial state changes as shown in formula (15): （15） Step 3.5, Link The set of free time windows Transformed into ,link The collection of executed tasks Transformed into ,link The start time collection of executed tasks Transformed into ,link The sum of the benefits of the executed tasks is given by Transformed into ,link The average delay time of the executed tasks is given by Transformed into ; Step 3.6, Task Profit and execution time As the time step progresses, the task In the link The available time window on Time step Transformed into Time step ,Task The priority is determined by Time step Transformed into Time step .

5. The multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning according to claim 4 is characterized in that: The specific process of step 3.1 is as follows: Step 3.1.1, Link Status As shown in formula (16): （16） in, Indicates a link exist The set of free time windows in the time step, ; Indicates Time step, link The set of executed tasks; Indicates Time step, link The set of start times of executed tasks; Indicates Time step, link The sum of the benefits of executed tasks is shown in formula (17): （17） Indicates Time step, link The average delay time of executed tasks is shown in formula (18): （18） Step 3.1.2, Task Status As shown in formula (19): （19） in, Indicates a task exist Time steps and links The set of available time windows on , , Indicates time step, task i and Link k No. w The start time of the available time window, It indicates the end time; Indicates a task Execution time; Indicates a task Execution proceeds; Indicates a task exist Priority of the time step.

6. The multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning according to claim 5 is characterized by: The specific process of step 4 is as follows: Step 4.1, create a memory pool; Step 4.2: Initialize the memory pool. Before training begins, fill the memory pool with initial experience data using a random strategy. Step 4.3, obtain the link status and task status under the current situation from the environment; Step 4.4, select the maximum output of the current network The action a corresponding to the value is learned using greedy learning To balance exploration and exploitation; Step 4.5, execute the selected action a, obtain the reward R and the next state ;use To indicate the status Select an action What you get The value is used as the experience for future action selection, as shown in formula (20): （20） Step 4.6, memory storage; Step 4.7, update of memory pool; Step 4.8: Build a neural network. The estimated neural network has the same structure as the target neural network. Input the spatial state, including the link state and the task state, and output the state of all actions in this state. value; Step 4.9, initialize the estimated network parameters and target network parameters ; Step 4.10, optimize the parameters of the neural network; Step 4.11, randomly select a batch of data sets from the memory pool to train the estimated neural network; Step 4.12, use the Bellman equation to calculate the target The value is shown in formula (21): （21） Step 4.13, calculate the estimated value: ; Step 4.14, calculate the loss value through the loss function, as shown in formula (22): （22） Step 4.15, the loss value As the error is passed back, the parameters are adjusted by gradient descent Make updates; Step 4.16, regularly update the parameters of the target neural network, ; Step 4.17, save the structure and parameters of the target neural network and the estimated neural network; In step 4.18, the new task status and link status are input into the trained neural network to obtain the final scheduling solution.

7. The multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning according to claim 6 is characterized by: In step 4.8, the neural network includes an estimation neural network and a target neural network. The estimation neural network predicts the current Q value of each action based on the current link and task state, and calculates the maximum target Q value through the target neural network. The network structure of the estimation neural network and the target neural network is exactly the same, including 1 input layer, 3 hidden layers and 1 output layer; wherein the input layer outputs the input space state, including the link state and the task state ;The first hidden layer input , output , whose activation function is ReLU; the second hidden layer input , output , whose activation function is ReLU; the third hidden layer input , output , whose activation function is ReLU; the output layer outputs all actions in this state value .

8. The multi-objective scheduling method for relay satellite civil-military integration based on deep reinforcement learning according to claim 7 is characterized by: In step 4.13, the Bellman equation is used to calculate the target The value is shown in formula (23): （23）。

Images