A multi-satellite multi-target observation task planning method based on reinforcement learning

Abstract

A multi-satellite multi-target observation task planning method based on reinforcement learning, the steps are as follows: (1) count the index requirements of all observation targets and the performance indexes of all imaging satellites; (2) based on the index requirements and performance indexes, build an initial deep reinforcement learning model based on reinforcement learning; (3) input the index requirements and performance indexes into the deep reinforcement learning model to obtain an initial task planning sequence set; (4) convert the task planning sequence set into a task sequence matrix, compare, sort and insert the elements in the task sequence matrix, and obtain the optimal task allocation result; (5) allocate the imaging satellites and observation targets according to the optimal task allocation result to complete the task planning; in the above process, it is constantly judged whether the imaging satellite has an emergency, and if the emergency occurs, the results of each step are modified, so that the task is re-planned immediately in the case of sudden failure of the imaging satellite.

Description

Technical Field

[0001] This invention relates to a multi-satellite, multi-target observation mission planning method based on reinforcement learning, belonging to the field of aerospace. Background Technology

[0002] Given limited imaging satellite resources, achieving a high mission completion rate within a specified timeframe is a complex combinatorial optimization problem with multiple constraints and conflicts. How to rationally and efficiently plan missions for the constellation, and how to promptly correct and replan missions after satellite malfunctions, are pressing issues that need to be addressed. Current methods for mission planning in multi-satellite, multi-target observation missions mostly rely on metaheuristic algorithms and deep reinforcement learning algorithms. These methods can only be applied to pre-defined scenarios. When mission information changes, they require recalculation to converge to the optimal solution, which is time-consuming and cannot achieve real-time mission planning. Summary of the Invention

[0003] The technical problem solved by this invention is to overcome the shortcomings of existing technologies and provide a multi-satellite, multi-target observation mission planning method based on reinforcement learning. This method combines deep reinforcement learning with matrix planning algorithms and performs real-time mission replanning in the event of sudden imaging satellite failures, thus solving the problem that existing methods cannot achieve real-time mission planning when mission information changes.

[0004] The technical solution of this invention is: A multi-satellite, multi-target observation task planning method based on reinforcement learning, comprising the following steps: (1) Statistics m The indicator requirements for each observation target, statistics n Performance metrics of each imaging satellite; (2) Based on the indicator requirements of the observation target and the performance indicators of the imaging satellite, build an initial deep reinforcement learning model based on reinforcement learning and set the basic elements of the initial deep reinforcement learning model. (3) Determine if any imaging satellite is in an emergency; if any imaging satellite is in an emergency, adjust the initial deep reinforcement learning model according to the emergency to obtain a deep reinforcement learning model; otherwise, directly use the initial deep reinforcement learning model as the deep reinforcement learning model. (4) m The indicator requirements for each observation target and n The performance metrics of each imaging satellite are input into a deep reinforcement learning model, which outputs an initial task planning sequence set; this initial task planning sequence set represents the... m Each observation target is assigned to n The situation of individual imaging satellites; (5) Determine if any imaging satellite has malfunctioned; if any imaging satellite has malfunctioned, then the initial mission planning sequence set is modified to obtain the mission planning sequence set; otherwise, the initial mission planning sequence set is directly used as the mission planning sequence set. (6) Transform the task planning sequence set into a task sequence matrix, compare, sort and insert the elements in the task sequence matrix to obtain the optimal task allocation result; (7) Allocate imaging satellites and observation targets according to the optimal task allocation results to complete the task planning.

[0005] Furthermore, the basic elements of setting up a deep reinforcement learning model include setting up the environment matrix of the deep reinforcement learning model, setting up the state vector of the deep reinforcement learning model, setting up the actions of the deep reinforcement learning model, and setting up the reward of the deep reinforcement learning model.

[0006] Furthermore, the specific process of setting the environment matrix for the deep reinforcement learning model is as follows: First step, according to m The indicator requirements for each observation target and n The performance metrics of each imaging satellite are used to obtain the observation effect of each imaging satellite on each observation target at different times; The second step is to combine the observation results of each imaging satellite for each target at different times into a three-dimensional visibility matrix. O ; The third step is to apply the visibility matrix. O As the environment matrix for deep reinforcement learning models; The specific steps for setting the state vector of the deep reinforcement learning model are as follows: First step, according to m The indicator requirements for each observation target are determined, including the task priority, theoretical observation time, actual observation time, first imaging satellite, second imaging satellite, and mission completion markers for each observation target. The second step involves constructing a property vector for each observation target by assigning its mission priority, theoretical observation time, actual observation time, first imaging satellite, second imaging satellite, and mission completion marker. The vector is defined as follows: j The property vector of each observed target is { Q j , TN j , TS j , S1 j , S2 j , P j}, Qj , TN j , TS j , S1 j , S2 j and P j The first j The mission priority, theoretical observation time, actual observation time, first imaging satellite, second imaging satellite, and mission completion markers for each observation target; The third step is to arrange the property vectors of all observed targets to obtain the state matrix. U ; The fourth step is to process the state matrix. U Normalization and flattening are performed to obtain the state vector of the deep reinforcement learning model. The formula for normalization is:

[0007] in, The 6th in the state vector i + j One element, Represents the state matrix of the th i Line number j Column elements, State matrix U The Middle i The minimum value of the row. State matrix U The Middle i The maximum value of a row; The specific steps for setting up a deep reinforcement learning model are as follows: The first step is to define the motion vector. A The action vector A elements in Indicates the first i The importance of each satellite mission allocation; The second step is to select the motion vector. A The index corresponding to the maximum value of the element is used as the action of the deep reinforcement learning model at the next moment. a ; The specific reward for setting the deep reinforcement learning model is as follows: Calculate the reward when the deep reinforcement learning model selects an action based on that action. R The calculation formula is:

[0008] in, Select the timing for the action; The first imaging satellite for the selected mission; The observation results of the first imaging satellite for the selected mission; The second imaging satellite for the selected mission; The second imaging satellite observation results for the selected mission; Set the task priority for the selected task; The time required for theoretical observations of the selected task; Set the task completion flag for the selected task.

[0009] Furthermore, the emergency situations that occur to the imaging satellite in step (3) include satellite malfunctions and emergency missions. When a satellite encounters an emergency, the initial deep reinforcement learning model is adjusted by setting the element in the environment matrix of the initial deep reinforcement learning model that represents the observation effect of the faulty satellite on the observation target to 0. When a satellite encounters an emergency mission, the initial deep reinforcement learning model is adjusted as follows: the reward in the initial deep reinforcement learning model is adjusted... R The calculation formula is replaced with:

[0010] in, for t Time of the first k The observation results of the first imaging satellite for an urgent mission. for t Time of the first k The observation results of the second imaging satellite for an urgent mission. For the first k The completion marker for an urgent task. For the first k This is a sign that an urgent task has been completed.

[0011] Furthermore, the deep reinforcement learning model includes a first hidden layer and a second hidden layer; The specific steps for the deep reinforcement learning model to output the task planning sequence in step (4) are as follows: (4.1) Based on m The indicator requirements for each observation target and n Performance metrics of each imaging satellite, and acquisition of the initial mission state vector. X 0; the initial task state vector X 0 indicates the allocation of observation targets; (4.2) Initial task state vector X 0 is input into the deep reinforcement learning model, representing the initial task state vector. XEach element in 0 corresponds to a weight in the first hidden layer of the deep reinforcement learning model. Multiply each product separately, sum the results, and add the first bias value to the sum. b After passing through the first activation function, the output of the first hidden layer is obtained. h 1; First hidden layer output h The formula for calculating 1 is:

[0012] in, The first activation function is the Softmax function. (4.3) Output of the first hidden layer h 1 is used as the input to the second hidden layer; the first hidden layer output is... h Each element in 1 corresponds to a weight in the second hidden layer. Multiply each product separately, sum the results, and add the second bias value to the sum. b The action value is obtained after passing through the second activation function. Q Action value Q The calculation formula is

[0013] in, The second activation function is the Softmax function. (4.4) A greedy algorithm is used to calculate the value of each action. Q Select the first action a 1. Calculate the first action a The first reward of the deep reinforcement learning model corresponding to 1 R 1; (4.5) Based on the first action a 1. Obtain the state vector of the next task. X’ and the second reward corresponding to the next task state vector R’ ; (4.6) Initial task state vector X 0. First action a 1. First Reward R 1. Next task state vector X’ Second reward R’ , forming an aggregate vector { X 0, a 1, R 1, X’ , R’}; (4.7) Repeat steps (4.2) to (4.6) until the preset number of times is reached. LEach time step (4.2) is repeated, the next task state vector obtained in the previous iteration is used. X’ As the initial task state vector X 0; ultimately obtained L One aggregate vector; (4.8) From L Randomly selected from aggregate vectors p The parameters of the deep reinforcement learning model are updated using aggregated vectors; (4.9) Repeat steps (4.2) to (4.8) until the preset number of iterations is reached; each time step (4.2) is executed, the initial task state vector is changed. X 0 is input into the deep reinforcement learning model after the parameter update in the previous iteration; after repeated iterations, all the next task state vectors obtained form the initial task planning sequence set.

[0014] Furthermore, the parameter update of the deep reinforcement learning model in step (4.8) specifically involves: (4.8.1) Based on the extracted p Calculate the quality value corresponding to each aggregation vector; (4.8.2) Based on the quality value of each aggregation vector, calculate the target quality value corresponding to each aggregation vector. The calculation formula is as follows:

[0015] in, For the first i The target quality value of each aggregate vector. For the first i The quality value of each aggregate vector. This is the discount factor, with a value ranging from 0 to 1; (4.8.3) Calculate the root mean square error based on the quality values ​​of all aggregated vectors and the target quality value. MSE The calculation formula is:

[0016] (4.8.4) Based on the mean square error, backpropagation is used to update the parameters of the entire deep reinforcement learning model.

[0017] Furthermore, in step (5), adjusting the initial task planning sequence set to obtain the task planning sequence set specifically involves: modifying the elements in all next task state vectors contained in the initial task planning sequence set to:

[0018] in, For the initial task planning sequence set, the first... iThe next task state vector in the th th j The correction result for each element The set of imaging satellites after removing faulty satellites. For the first b The imaging satellite for the first i The observation effect of each observation target.

[0019] Furthermore, the process of obtaining the optimal task allocation result based on the task planning sequence set in step (6) is as follows: (6.1) Transform the task planning sequence set into a target sequence matrix. H The target sequence matrix H Each element in the matrix represents the allocation result corresponding to the observed target, the target sequence matrix. H elements in For the first i The first imaging satellite to observe the target. For the first i The second imaging satellite for observing targets, For the first i The actual observation time of each observation target; establishing a backup target sequence matrix. B The backup target sequence matrix B All elements in the matrix are 0, and the structure is the same as the target sequence matrix. H Same, backup target sequence matrix B This represents the low-priority tasks that conflict with high-priority tasks in the missions performed by the imaging satellite; a mission execution sequence matrix is ​​established. E The task execution sequence matrix E Represents the actual task allocation result. E The middle element represents all moments. m Each observation target is assigned to n Results from each imaging satellite; proceed to step (6.2); (6.2) Select the target sequence matrix in sequence H For each element in the sequence matrix, if the imaging satellite used in the corresponding observation target allocation result is not in the mission execution sequence matrix... E If an element appears in the sequence, it will be placed into the task execution sequence matrix in order. E If the condition is met, then the element is placed sequentially into the backup target sequence matrix; otherwise, the element is placed sequentially into the backup target sequence matrix. B In the middle; execute step (6.3); (6.3) Determine the task execution sequence matrix E Are there any idle satellites in the sequence matrix? E If there are idle imaging satellites, then the matrix is ​​calculated according to the mission execution sequence. EDuring mission execution, for each observation target, step (6.4) is executed according to the time required for the corresponding imaging satellite to complete the observation; otherwise, the backup target sequence matrix is ​​used. B Select conflict-free observation tasks and add them to E Repeat step (6.2). (6.4) In the task execution sequence matrix E The observation target with the shortest required observation time is selected from the data. This observation target and the corresponding imaging satellite are used as an initial allocation strategy. Simultaneously, this strategy is incorporated into the mission execution sequence matrix. E In the middle, delete the elements related to the observation target; proceed to step (6.5); (6.5) Repeat steps (6.2) to (6.4) until the task execution sequence matrix is ​​reached. E There are no elements in the initial allocation strategy; after repeated iterations, all the initial allocation strategies obtained together constitute the optimal allocation result of the task.

[0020] Secondly, the present invention also proposes a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the aforementioned multi-satellite multi-target observation task planning method based on reinforcement learning.

[0021] Thirdly, the present invention also proposes a processor, characterized in that the processor is used to run a program, wherein the program executes the above-described multi-satellite multi-target observation task planning method based on reinforcement learning.

[0022] The advantages of this invention compared to the prior art are: (1) The present invention obtains the task planning sequence through a deep reinforcement learning model and adjusts the task planning sequence according to the matrix sorting to obtain the optimal task allocation result; if the satellite has an anomaly, the deep reinforcement learning model, the task planning sequence and the optimal task allocation result are corrected, so that the task replanning can be carried out in the event of a sudden failure of the imaging satellite.

[0023] (2) The overall process of this invention is clear and easy to implement. It can react immediately when the imaging satellite suddenly fails, ensuring that the imaging satellite can work normally. Attached Figure Description

[0024] Figure 1 This is a flowchart of a multi-satellite, multi-target observation mission planning method based on reinforcement learning according to the present invention. Detailed Implementation

[0025] The specific embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.

[0026] likeFigure 1 As shown, this invention provides a multi-satellite, multi-target observation task planning method based on reinforcement learning, with the following steps: (1) Statistics m The indicator requirements for each observation target, statistics n Performance metrics of each imaging satellite; (2) Based on the indicator requirements of the observation target and the performance indicators of the imaging satellite, build an initial deep reinforcement learning model based on reinforcement learning and set the basic elements of the initial deep reinforcement learning model. The basic elements of setting up a deep reinforcement learning model include setting the environment matrix, setting the state vector, setting the actions, and setting the rewards.

[0027] The specific process for setting the environment matrix of the deep reinforcement learning model is as follows: First step, according to m The indicator requirements for each observation target and n The performance metrics of each imaging satellite are used to obtain the observation effect of each imaging satellite on each observation target at different times; The second step is to combine the observation results of each imaging satellite for each target at different times into a three-dimensional visibility matrix. O ; The third step is to apply the visibility matrix. O As the environment matrix for deep reinforcement learning models; The specific steps for setting the state vector of the deep reinforcement learning model are as follows: First step, according to m The indicator requirements for each observation target are determined, including the task priority, theoretical observation time, actual observation time, first imaging satellite, second imaging satellite, and mission completion markers for each observation target. The second step involves constructing a property vector for each observation target by assigning its mission priority, theoretical observation time, actual observation time, first imaging satellite, second imaging satellite, and mission completion marker. The vector is defined as follows: j The property vector of each observed target is { Q j , TN j , TS j , S1 j , S2 j , P j}, Q j , TN j, TS j , S1 j , S2 j and P j The first j The mission priority, theoretical observation time, actual observation time, first imaging satellite, second imaging satellite, and mission completion markers for each observation target; The third step is to arrange the property vectors of all observed targets to obtain the state matrix. U ; The fourth step is to process the state matrix. U Normalization and flattening are performed to obtain the state vector of the deep reinforcement learning model. The formula for normalization is:

[0028] in, The 6th in the state vector i + j One element, Represents the state matrix of the th i Line number j Column elements, State matrix U The Middle i The minimum value of the row. State matrix U The Middle i The maximum value of a row; The specific steps for setting up a deep reinforcement learning model are as follows: The first step is to define the motion vector. A The action vector A elements in Indicates the first i The importance of each satellite mission allocation; The second step is to select the motion vector. A The index corresponding to the maximum value of the element is used as the action of the deep reinforcement learning model at the next moment. a ; The specific reward for setting the deep reinforcement learning model is as follows: Based on the actions of the deep reinforcement learning model, calculate the reward when the deep reinforcement learning model selects a particular action. R The calculation formula is:

[0029] in, Select the timing for the action; The first imaging satellite for the selected mission; The observation results of the first imaging satellite for the selected mission; The second imaging satellite for the selected mission; The second imaging satellite observation results for the selected mission; Set the task priority for the selected task; The time required for theoretical observations of the selected task; Set the task completion flag for the selected task.

[0030] (3) Determine if any imaging satellite is in an emergency; if any imaging satellite is in an emergency, adjust the initial deep reinforcement learning model according to the emergency to obtain a deep reinforcement learning model; otherwise, directly use the initial deep reinforcement learning model as the deep reinforcement learning model. The emergency situations that occur to the imaging satellite in step (3) include satellite malfunction and emergency missions. When a satellite encounters an emergency, the initial deep reinforcement learning model is adjusted by setting the element in the environment matrix of the initial deep reinforcement learning model that represents the observation effect of the faulty satellite on the observation target to 0. When a satellite encounters an emergency mission, the initial deep reinforcement learning model is adjusted as follows: the reward in the initial deep reinforcement learning model is adjusted... R The calculation formula is replaced with:

[0031] in, for t Time of the first k The observation results of the first imaging satellite for an urgent mission. for t Time of the first k The observation results of the second imaging satellite for an urgent mission. For the first k The completion marker for an urgent task. For the first k This is a sign that an urgent task has been completed.

[0032] (4) m The indicator requirements for each observation target and n The performance metrics of each imaging satellite are input into a deep reinforcement learning model, which outputs an initial task planning sequence set; this initial task planning sequence set represents the... m Each observation target is assigned to n The situation of individual imaging satellites; The deep reinforcement learning model includes a first hidden layer and a second hidden layer; The specific steps for the deep reinforcement learning model to output the task planning sequence in step (4) are as follows: (4.1) Based on m The indicator requirements for each observation target and n Performance metrics of each imaging satellite, and acquisition of the initial mission state vector. X 0; the initial task state vector X 0 indicates the allocation of observation targets; (4.2) Initial task state vector X 0 is input into the deep reinforcement learning model, representing the initial task state vector. X Each element in 0 corresponds to a weight in the first hidden layer of the deep reinforcement learning model. Multiply each product separately, sum the results, and add the first bias value to the sum. b After passing through the first activation function, the output of the first hidden layer is obtained. h 1; First hidden layer output h The formula for calculating 1 is:

[0033] in, The first activation function is the Softmax function. (4.3) Output of the first hidden layer h 1 is used as the input to the second hidden layer; the first hidden layer output is... h Each element in 1 corresponds to a weight in the second hidden layer. Multiply each product separately, sum the results, and add the second bias value to the sum. b The action value is obtained after passing through the second activation function. Q Action value Q The calculation formula is

[0034] in, The second activation function is the Softmax function. (4.4) A greedy algorithm is used to calculate the value of each action. Q Select the first action a 1. Calculate the first action a The first reward of the deep reinforcement learning model corresponding to 1 R 1; (4.5) Based on the first action a 1. Obtain the state vector of the next task. X’ and the second reward corresponding to the next task state vector R’ ; (4.6) Initial task state vector X 0. First actiona 1. First Reward R 1. Next task state vector X’ Second reward R’ , forming an aggregate vector { X 0, a 1, R 1, X’ , R’}; (4.7) Repeat steps (4.2) to (4.6) until the preset number of times is reached. L Each time step (4.2) is repeated, the next task state vector obtained in the previous iteration is used. X’ As the initial task state vector X 0; ultimately obtained L One aggregate vector; (4.8) From L Randomly selected from aggregate vectors p The parameters of the deep reinforcement learning model are updated using aggregated vectors, and the specific steps are as follows: (4.8.1) Based on the extracted p Calculate the quality value corresponding to each aggregation vector; (4.8.2) Based on the quality value of each aggregation vector, calculate the target quality value corresponding to each aggregation vector. The calculation formula is as follows:

[0035] in, For the first i The target quality value of each aggregate vector. For the first i The quality value of each aggregate vector. This is the discount factor, with a value ranging from 0 to 1; (4.8.3) Calculate the root mean square error based on the quality values ​​of all aggregated vectors and the target quality value. MSE The calculation formula is:

[0036] (4.8.4) Based on the mean square error, backpropagation is used to update the parameters of the entire deep reinforcement learning model.

[0037] (4.9) Repeat steps (4.2) to (4.8) until the preset number of iterations is reached; each time step (4.2) is executed, the initial task state vector is changed. X 0 is input into the deep reinforcement learning model after the parameter update in the previous iteration; after repeated iterations, all the next task state vectors obtained form the initial task planning sequence set.

[0038] (5) Determine if any imaging satellite has malfunctioned; if any imaging satellite has malfunctioned, then the initial mission planning sequence set is modified to obtain the mission planning sequence set; otherwise, the initial mission planning sequence set is directly used as the mission planning sequence set. Furthermore, in step (5), adjusting the initial task planning sequence set to obtain the task planning sequence set specifically involves: modifying the elements in all next task state vectors contained in the initial task planning sequence set to:

[0039] in, For the initial task planning sequence set, the first... i The next task state vector in the th th j The correction result for each element The set of imaging satellites after removing faulty satellites. For the first b The imaging satellite for the first i The observation effect of each observation target.

[0040] (6) Transform the task planning sequence set into a task sequence matrix, compare, sort, and insert the elements in the task sequence matrix to obtain the optimal task allocation result. The specific steps are as follows: (6.1) Transform the task planning sequence set into a target sequence matrix. H The target sequence matrix H Each element in the matrix represents the allocation result corresponding to the observed target, the target sequence matrix. H elements in For the first i The first imaging satellite to observe the target. For the first i The second imaging satellite for observing targets, For the first i The actual observation time of each observation target; establishing a backup target sequence matrix. B The backup target sequence matrix B All elements in the matrix are 0, and the structure is the same as the target sequence matrix. H Same, backup target sequence matrix B This represents the low-priority tasks that conflict with high-priority tasks in the missions performed by the imaging satellite; a mission execution sequence matrix is ​​established. E The task execution sequence matrix E Represents the actual task allocation result. E The middle element represents all moments. m Each observation target is assigned to n Results from each imaging satellite; proceed to step (6.2); (6.2) Select the target sequence matrix in sequence H For each element in the sequence matrix, if the imaging satellite used in the corresponding observation target allocation result is not in the mission execution sequence matrix... E If an element appears in the sequence, it will be placed into the task execution sequence matrix in order. E If the condition is met, then the element is placed sequentially into the backup target sequence matrix; otherwise, the element is placed sequentially into the backup target sequence matrix. B In the middle; execute step (6.3); (6.3) Determine the task execution sequence matrix E Are there any idle satellites in the sequence matrix? E If there are idle imaging satellites, then the matrix is ​​calculated according to the mission execution sequence. E During mission execution, for each observation target, step (6.4) is executed according to the time required for the corresponding imaging satellite to complete the observation; otherwise, the backup target sequence matrix is ​​used. B Select conflict-free observation tasks and add them to E Repeat step (6.2). (6.4) In the task execution sequence matrix E The observation target with the shortest required observation time is selected from the data. This observation target and the corresponding imaging satellite are used as an initial allocation strategy. Simultaneously, this strategy is incorporated into the mission execution sequence matrix. E In the middle, delete the elements related to the observation target; proceed to step (6.5); (6.5) Repeat steps (6.2) to (6.4) until the task execution sequence matrix is ​​reached. E There are no elements in the initial allocation strategy; after repeated iterations, all the initial allocation strategies obtained together constitute the optimal allocation result of the task.

[0041] (7) Allocate imaging satellites and observation targets according to the optimal task allocation results to complete the task planning.

[0042] The results obtained by the above method satisfy four constraints: visibility, time window, on-board resources, and mission conflicts. Specifically: Visibility constraints: This invention aims to ensure the observation target j To achieve the desired observation results, two imaging satellites were selected. i 1. i 2. Simultaneously observe a target j Therefore, the task j At any moment t Observability requires that:

[0043] Time window constraints: Ground target observation tasks require continuous observation over a period of time; therefore, the observation tasks must be performed within the visible time window, i.e.:

[0044] in, For observation target j The initial observation time, For observation j The end of observation time, For the goal j The initial visible moment, For the goal j The time at which the event ends is visible.

[0045] Onboard resource constraints: This invention will use satellites i The power supply, storage capacity, and other resources of a satellite are collectively referred to as on-board resources. These resources are used for performing observation missions. j The resources consumed include motor energy consumption. Energy consumption for observation The available satellite resources before the mission are Execute the task j Need to meet

[0046] Mission conflict: Due to payload limitations, a satellite can only perform a single observation task at any given time, and to avoid wasting resources, the same task can be performed by at most two satellites simultaneously once. Therefore, the following conditions are met:

[0047]

[0048] For satellite i At any moment t The number of tasks executed. For satellite i For the task j The number of times the task is executed.

[0049] Based on the above process, this invention obtains a task planning sequence through a deep reinforcement learning model and adjusts the task planning sequence according to matrix sorting to obtain the optimal task allocation result. Throughout the process, if the satellite experiences an anomaly, the deep reinforcement learning model, task planning sequence, and optimal task allocation result are corrected, thereby enabling real-time task replanning in the event of a sudden imaging satellite failure. In addition, the overall process of this invention is clear, easy to implement, and can react instantly in the event of a sudden imaging satellite failure, ensuring that the imaging satellite can operate normally.

[0050] Secondly, the present invention also proposes a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the aforementioned multi-satellite multi-target observation task planning method based on reinforcement learning.

[0051] Thirdly, the present invention also proposes a processor, characterized in that the processor is used to run a program, wherein the program executes the above-described multi-satellite multi-target observation task planning method based on reinforcement learning.

[0052] The parts of this invention not described in detail are common knowledge to those skilled in the art.

Claims

1. A multi-satellite, multi-target observation task planning method based on reinforcement learning, characterized in that, include: statistics m The indicator requirements for each observation target, statistics n Performance metrics of each imaging satellite; Based on the indicator requirements of the observation target and the performance indicators of the imaging satellite, an initial deep reinforcement learning model based on reinforcement learning is built, and the basic elements of the initial deep reinforcement learning model are set. Determine if any imaging satellite is in an emergency; if any imaging satellite is in an emergency, adjust the initial deep reinforcement learning model according to the emergency situation to obtain a new deep reinforcement learning model. Otherwise, the initial deep reinforcement learning model is used directly as the deep reinforcement learning model; Will m The indicator requirements for each observation target and n The performance metrics of each imaging satellite are input into a deep reinforcement learning model, which outputs an initial task planning sequence set; this initial task planning sequence set represents the... m Each observation target is assigned to n The situation of individual imaging satellites; Determine if any imaging satellite has malfunctioned; if any imaging satellite has malfunctioned, revise the initial mission planning sequence set to obtain the new mission planning sequence set. Otherwise, the initial task planning sequence set is directly used as the task planning sequence set; The task planning sequence set is transformed into a task sequence matrix. The elements in the task sequence matrix are compared, sorted, and interpolated to obtain the optimal task allocation result. Based on the optimal task allocation results, imaging satellites and observation targets are allocated to complete the task planning.

2. The multi-satellite, multi-target observation task planning method based on reinforcement learning according to claim 1, characterized in that: The basic elements of setting up a deep reinforcement learning model include setting the environment matrix, setting the state vector, setting the actions, and setting the rewards.

3. The multi-satellite, multi-target observation task planning method based on reinforcement learning according to claim 2, characterized in that: The specific process for setting the environment matrix of the deep reinforcement learning model is as follows: First step, according to m The indicator requirements for each observation target and n The performance metrics of each imaging satellite are used to obtain the observation effect of each imaging satellite on each observation target at different times; The second step is to combine the observation results of each imaging satellite for each target at different times into a three-dimensional visibility matrix. O ; Observability matrix O elements in ,represent t Time of the first i The satellite for the first j The observation effect of a single observation target; the larger the value, the better the observation effect. The third step is to apply the visibility matrix. O As the environment matrix for deep reinforcement learning models; The specific steps for setting the state vector of the deep reinforcement learning model are as follows: First step, according to m The indicator requirements for each observation target are determined, including the task priority, theoretical observation time, actual observation time, first imaging satellite, second imaging satellite, and mission completion markers for each observation target. The second step involves constructing a property vector for each observation target by assigning its mission priority, theoretical observation time, actual observation time, first imaging satellite, second imaging satellite, and mission completion marker. The vector is defined as follows: j The property vector of each observed target is { Q j , TN j , TS j , S1 j , S2 j , P j }, Q j , TN j , TS j , S1 j , S2 j and P j The first j The mission priority, theoretical observation time, actual observation time, first imaging satellite, second imaging satellite, and mission completion markers for each observation target; The third step is to arrange the property vectors of all observed targets to obtain the state matrix. U ; The fourth step is to process the state matrix. U Normalization and flattening are performed to obtain the state vector of the deep reinforcement learning model. The formula for normalization is: in, The 6th in the state vector i + j One element, Represents the state matrix of the th i Line number j Column elements, State matrix U The Middle i The minimum value of the row. State matrix U The Middle i The maximum value of a row; The specific steps for setting up a deep reinforcement learning model are as follows: The first step is to define the motion vector. A The action vector A elements in Indicates the first i The importance of each satellite mission allocation; The second step is to select the motion vector. A The index corresponding to the maximum value of the element is used as the action of the deep reinforcement learning model at the next moment. a ; The specific reward for setting the deep reinforcement learning model is as follows: Calculate the reward when the deep reinforcement learning model selects an action based on that action. R The calculation formula is: in, Select the timing for the action; The first imaging satellite for the selected mission; The observation results of the first imaging satellite for the selected mission; The second imaging satellite for the selected mission; The second imaging satellite observation results for the selected mission; Set the task priority for the selected task; The time required for theoretical observations of the selected task; Set the task completion flag for the selected task.

4. The multi-satellite, multi-target observation task planning method based on reinforcement learning according to claim 3, characterized in that: Emergency situations involving the imaging satellite include satellite malfunctions and emergency missions. When a satellite encounters an emergency, the initial deep reinforcement learning model is adjusted by setting the element in the environment matrix of the initial deep reinforcement learning model that represents the observation effect of the faulty satellite on the observation target to 0. When a satellite encounters an emergency mission, the initial deep reinforcement learning model is adjusted as follows: the reward in the initial deep reinforcement learning model is adjusted... R The calculation formula is replaced with: in, for t Time of the first k The observation results of the first imaging satellite for an urgent mission. for t Time of the first k The observation results of the second imaging satellite for an urgent mission. For the first k The completion marker for an urgent task. For the first k This is a sign that an urgent task has been completed.

5. The multi-satellite, multi-target observation task planning method based on reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning model includes a first hidden layer and a second hidden layer; The specific steps for the deep reinforcement learning model to output the task planning sequence are as follows: (4.1) Based on m The indicator requirements for each observation target and n Performance metrics of each imaging satellite, and acquisition of the initial mission state vector. X 0; the initial task state vector X 0 indicates the allocation of observation targets; (4.2) Initial task state vector X 0 is input into the deep reinforcement learning model, representing the initial task state vector. X Each element in 0 corresponds to a weight in the first hidden layer of the deep reinforcement learning model. Multiply each product separately, sum the results, and add the first bias value to the sum. b After passing through the first activation function, the output of the first hidden layer is obtained. h 1; First hidden layer output h The formula for calculating 1 is: in, The first activation function is the Softmax function. (4.3) Output of the first hidden layer h 1 is used as the input to the second hidden layer; the first hidden layer outputs... h Each element in 1 corresponds to a weight in the second hidden layer. Multiply each product separately, sum the results, and add the second bias value to the sum. b The action value is obtained after passing through the second activation function. Q Action value Q The calculation formula is: in, The second activation function is the Softmax function. (4.4) A greedy algorithm is used to calculate the value of each action. Q Select the first action a 1. Calculate the first action a The first reward of the deep reinforcement learning model corresponding to 1 R 1; (4.5) Based on the first action a 1. Obtain the state vector of the next task. X’ and the second reward corresponding to the next task state vector R’ ; (4.6) Initial task state vector X 0. First action a 1. First Reward R 1. Next task state vector X’ Second reward R’ , forming an aggregate vector { X 0, a 1, R 1, X’ , R’ }; (4.7) Repeat steps (4.2) to (4.6) until the preset number of times is reached. L Each time step (4.2) is repeated, the next task state vector obtained in the previous iteration is used. X’ As the initial task state vector X 0; ultimately obtained L One aggregate vector; (4.8) From L Randomly selected from aggregate vectors p The parameters of the deep reinforcement learning model are updated using aggregated vectors; (4.9) Repeat steps (4.2) to (4.8) until the preset number of iterations is reached; each time step (4.2) is executed, the initial task state vector is changed. X 0 is input into the deep reinforcement learning model after the parameter update in the previous iteration; after repeated iterations, all the next task state vectors obtained form the initial task planning sequence set.

6. The multi-satellite, multi-target observation task planning method based on reinforcement learning according to claim 5, characterized in that: The parameter update of the deep reinforcement learning model in step (4.8) specifically involves: (4.8.1) Based on the extracted p Calculate the quality value corresponding to each aggregation vector; (4.8.2) Based on the quality value of each aggregation vector, calculate the target quality value corresponding to each aggregation vector. The calculation formula is as follows: in, For the first i The target quality value of each aggregate vector. For the first i The quality value of each aggregate vector. This is the discount factor, with a value ranging from 0 to 1; (4.8.3) Calculate the root mean square error based on the quality values ​​of all aggregated vectors and the target quality value. MSE The calculation formula is: (4.8.4) Based on the mean square error, backpropagation is used to update the parameters of the entire deep reinforcement learning model.

7. The multi-satellite, multi-target observation task planning method based on reinforcement learning according to claim 5, characterized in that: The adjustment of the initial task planning sequence set to obtain the task planning sequence set specifically involves: modifying the elements in all next task state vectors contained in the initial task planning sequence set to: in, For the initial task planning sequence set, the first... i The next task state vector in the th th j The correction result for each element The set of imaging satellites after removing faulty satellites. For the first b The imaging satellite for the first i The observation effect of each observation target.

8. The multi-satellite, multi-target observation task planning method based on reinforcement learning according to claim 1, characterized in that: The process of obtaining the optimal task allocation result based on the task planning sequence set is as follows: (6.1) Transform the task planning sequence set into a target sequence matrix. H The target sequence matrix H Each element in the matrix represents the allocation result corresponding to the observed target, the target sequence matrix. H elements in For the first i The first imaging satellite to observe the target. For the first i The second imaging satellite for observing targets, For the first i The actual observation time of each observation target; Establish a backup target sequence matrix B The backup target sequence matrix B All elements in the matrix are 0, and the structure is the same as the target sequence matrix. H Same, backup target sequence matrix B This represents a low-priority task that conflicts with a high-priority task in the missions performed by the imaging satellite; Establish a task execution sequence matrix E The task execution sequence matrix E Represents the actual task allocation result. E The middle element represents all moments. m Each observation target is assigned to n Results from one imaging satellite; Perform step (6.2); (6.2) Select the target sequence matrix in sequence H For each element in the sequence matrix, if the imaging satellite used in the corresponding observation target allocation result is not in the mission execution sequence matrix... E If an element appears in the sequence, it will be placed into the task execution sequence matrix in order. E If the condition is met, then the element is placed sequentially into the backup target sequence matrix; otherwise, the element is placed sequentially into the backup target sequence matrix. B In the middle; execute step (6.3); (6.3) Determine the task execution sequence matrix E Are there any idle satellites in the sequence matrix? E If there are idle imaging satellites, then the matrix is ​​calculated according to the mission execution sequence. E During mission execution, for each observation target, step (6.4) is executed according to the time required for the corresponding imaging satellite to complete the observation; otherwise, the backup target sequence matrix is ​​used. B Select conflict-free observation tasks and add them to E Repeat step (6.2). (6.4) In the task execution sequence matrix E The observation target with the shortest required observation time is selected from the data. This observation target and the corresponding imaging satellite are used as an initial allocation strategy. Simultaneously, this strategy is incorporated into the mission execution sequence matrix. E In the middle, delete the elements related to the observed target; Perform step (6.5); (6.5) Repeat steps (6.2) to (6.4) until the task execution sequence matrix is ​​reached. E There are no elements in the initial allocation strategy; after repeated iterations, all the initial allocation strategies obtained together constitute the optimal allocation result of the task.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the multi-satellite, multi-target observation mission planning method based on reinforcement learning as described in any one of claims 1 to 8.

10. A processor, characterized in that, The processor is used to run a program, wherein the program executes a multi-satellite multi-target observation mission planning method based on reinforcement learning as described in any one of claims 1 to 8.

Images