Multi-type satellite collaborative observation task planning method based on deep reinforcement learning

Abstract

The invention discloses a multi-type satellite collaborative observation task planning method based on deep reinforcement learning, and belongs to the technical field of satellite task planning. The method comprises the following steps: firstly, acquiring attributes of multiple types of satellites and attribute information of observation tasks such as single-satellite, double-satellite cooperation and three-satellite cooperation; then, constructing a multi-dimensional constraint model; a deep reinforcement learning decision model based on the Seq2Seq architecture is built, a coding layer extracts position information for relative position coding, and a decoding layer outputs a task-satellite allocation scheme and an execution sequence; an Actor-Critic algorithm is adopted to train the model, and minimization of loss of an uncompleted task is taken as a target optimization strategy. The problems that a traditional satellite task planning method is difficult in multi-constraint balance, low in calculation efficiency and weak in generalization ability are solved, and the method has the advantages of being high in multi-constraint fusion ability, high in calculation efficiency, excellent in generalization performance and remarkable in income optimization and can be effectively applied to the fields of agricultural monitoring, disaster emergency, environmental governance and the like.

Description

Technical Field

[0001] This invention relates to the field of spacecraft mission planning technology, specifically a multi-type satellite collaborative observation mission planning method based on deep reinforcement learning. Background Technology

[0002] Earth observation satellites are remote sensing satellites that can observe and acquire image information of Earth targets such as land, ocean, and space from their orbits using optical imaging remote sensors (panchromatic, multispectral, hyperspectral, infrared, etc.) or synthetic aperture radar imaging remote sensors. Imaging using Earth observation satellites has significant advantages such as wide observation range, long observation time, and no spatial or geographical limitations, and has been widely applied in agriculture, disaster reduction, transportation, and other fields.

[0003] With the ever-increasing global demand for satellite remote sensing technology, satellite observation missions have evolved from independent observation by a single satellite to collaborative observation by multiple satellites. Especially in the collaborative operation of multiple types of imaging satellites, different types of sensors (such as optical imaging, radar imaging, and infrared imaging) can complement each other, jointly improving the accuracy and coverage of Earth observation missions. This collaborative observation mode can significantly improve the perception capabilities in areas such as Earth surface changes, natural disasters, and environmental monitoring, providing more accurate data support for global resource monitoring, environmental protection, and climate change research.

[0004] However, with the continuous increase in the types and numbers of satellites and the observation targets, how to efficiently plan missions for multiple types of imaging satellites has become an urgent problem to be solved. Current mission planning methods mostly rely on traditional optimization algorithms and human experience, which have problems such as difficulty in adapting to dynamic environments, rapid changes in mission requirements, and high complexity of observation missions. Especially when multiple satellites work together, the effective scheduling of mission time, space, and sensor resources becomes a critical issue.

[0005] Deep reinforcement learning (DRL), as a cutting-edge technology in the field of artificial intelligence in recent years, has made significant progress in solving complex decision-making problems. By constructing reward mechanisms, DRL can learn and optimize itself in dynamic environments, making it particularly suitable for tasks with high uncertainty, complex constraints, and long-term planning. Applying deep reinforcement learning to the planning of collaborative observation missions for multiple types of imaging satellites can achieve adaptive mission scheduling and optimization in uncertain observation environments, improving the efficiency and accuracy of satellite missions.

[0006] Therefore, considering the complex constraints and dynamic requirements of collaborative observation of multiple types of satellites, it is urgent to propose a task planning method based on deep reinforcement learning to provide technical support for the large-scale application of collaborative observation modes of multiple types of satellites. Summary of the Invention

[0007] The present invention provides a multi-type satellite collaborative observation mission planning method based on deep reinforcement learning, which aims to solve the technical problems of high computational complexity and difficulty in obtaining the global optimal solution when the traditional mission planning method is used to handle multi-type observation satellite collaborative planning, as well as the shortcomings of some heuristic and intelligent optimization algorithms in terms of global and local optimization balance, conflict handling, efficiency and reliability.

[0008] The multi-type satellite collaborative observation mission planning method based on deep reinforcement learning provided by this invention includes the following steps:

[0009] S1: Obtain observation mission simulation data of the imaging satellite constellation and construct a satellite static attribute matrix based on the satellite static attributes;

[0010] S2: Obtain mission information for imaging satellite observation missions from simulation data, and construct a mission static attribute matrix based on the mission information;

[0011] S3: Based on the constructed satellite static attribute matrix and mission static attribute matrix, initialize the satellite dynamic attribute matrix and mission dynamic attribute matrix;

[0012] S4: Construct a deep reinforcement learning network, which includes an encoder, a decoder, and a multi-head attention module. The encoder is used to obtain the relative position information of the attribute matrices of the satellite and the mission, and to map the matrix to a high dimension. The decoder takes the static feature vector of the previously selected mission as input and outputs the corresponding high-dimensional query vector. The multi-head attention module obtains the mission probability distribution and is used to decide the mission to be planned in the next step.

[0013] Construct a reward function and train the deep reinforcement learning network using the attribute matrices constructed from S1 to S3 to obtain the trained deep reinforcement learning network.

[0014] Step 5: Obtain the observation tasks to be planned and the available satellite sequences, and obtain the corresponding satellite static attribute matrix, the static attribute matrix of the observation tasks to be planned, the satellite dynamic attribute matrix, and the task dynamic attribute matrix. Input these into the trained deep reinforcement learning model. Under constraints, the deep reinforcement learning network maximizes the reward of the execution scheme and outputs the planned task execution sequence to complete the collaborative task planning of multiple types of satellites.

[0015] Compared with the prior art, the beneficial effects of the present invention include:

[0016] This invention employs deep reinforcement learning technology, combined with an encoder-decoder and multi-head attention mechanism, to efficiently capture the attribute associations and temporal features between satellites and missions, significantly reducing the planning complexity of collaborative observation of multiple types of satellites and solving the problem that traditional methods struggle to obtain the global optimal solution.

[0017] This invention fully considers multi-dimensional constraints such as mission-satellite adaptation, time window, and power storage. Through dynamic attribute real-time updates and conflict resolution mechanisms, it balances overall planning and local optimization, thereby improving the reliability and execution efficiency of mission scheduling.

[0018] This invention aims to maximize mission rewards. Through the Actor-Critic training mechanism and scientific reward function design, it prioritizes the completion of high-reward and multi-satellite collaborative missions, providing high-quality data support for Earth observation-related fields. Attached Figure Description

[0019] Figure 1 This is a flowchart of the multi-type satellite collaborative mission planning method based on deep reinforcement learning, as described in this invention.

[0020] Figure 2 This is an architecture diagram of a deep reinforcement learning model.

[0021] Figure 3 This is a diagram of the encoder architecture for deep reinforcement learning.

[0022] Figure 4 This is a diagram of the decoder architecture for deep reinforcement learning.

[0023] Figure 5 This is a diagram of the multi-head attention mechanism.

[0024] Figure 6 A flowchart for task planning for deep reinforcement learning models.

[0025] Figure 7 This is a diagram of the Critic network architecture.

[0026] Figure 8 The training iteration data graph.

[0027] Figure 9 Visualize the test results. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] like Figure 1 As shown, the present invention includes the following steps:

[0030] S1: Construction of Satellite Static Attribute Matrix

[0031] To obtain simulated data on the operation of the imaging satellite, in this embodiment, the static attributes of the imaging satellite include six items: satellite number, satellite type, total power, total storage capacity, attitude maneuver speed, and power consumption per unit distance during attitude maneuvering, resulting in a satellite static attribute matrix. .

[0032] S2: Construction of Task Static Attribute Matrix

[0033] The mission information for imaging satellite observation tasks is obtained from simulated data. In this embodiment, the mission information includes 10 static attributes: mission number, corresponding observation satellite number, mission type, observation benefits, power consumption, storage consumption, time window start time, observation duration, side-swing angle, and time window end time. A mission static attribute matrix is ​​then constructed. .

[0034] S3: Based on the constructed satellite static attribute matrix and mission static attribute matrix, initialize the satellite dynamic attribute matrix and mission dynamic attribute matrix. The satellite's dynamic attributes include four items: current remaining satellite battery power, remaining storage capacity, execution time of the previous mission, and the previous mission number. The initial values ​​of the satellite's remaining battery power and remaining storage capacity are obtained by reading the satellite static attribute matrix. The total power and storage capacity are obtained, and the remaining dynamic attributes are initialized to 0. The mission's dynamic attributes include two items: whether it is visible to the satellite and whether it has been observed by the satellite; both are initially set to 0. These are then used to form a satellite dynamic attribute matrix. Task dynamic attribute matrix .

[0035] S4: Construct a deep reinforcement learning network, which includes an encoder, a decoder, and a multi-head attention module. The encoder is used to obtain the relative position information of the attribute matrices of the satellite and the mission, and to map the matrix to a high dimension. The decoder takes the static feature vector of the previously selected mission as input and outputs the corresponding high-dimensional query vector. The multi-head attention module obtains the mission probability distribution and is used to decide the mission to be planned in the next step.

[0036] Construct a reward function and train the deep reinforcement learning network using the attribute matrices constructed in steps 1 to 3 to obtain a trained deep reinforcement learning network.

[0037] S5: Obtain the observation tasks to be planned and the available satellite sequences, and obtain the corresponding static attribute matrix of the satellite sequences. Static attribute matrix of the set of observation tasks to be planned Satellite dynamic attribute matrix Task dynamic attribute matrix The data is then fed into a trained deep reinforcement learning model. The model selects tasks for each satellite in turn, and maximizes the reward of the execution plan under constraints of time window, power consumption, and storage capacity. It then outputs a planned task execution sequence to complete the collaborative task planning for multiple types of satellites.

[0038] The deep reinforcement learning model in this invention includes an encoder, a decoder, and a multi-head attention module, such as... Figure 2 As shown, the details are as follows:

[0039] A1: The encoder consists of two parts: a static encoder and a dynamic encoder. Both parts are composed of linear layers and relative position encoding. The specific structure of the encoder is as follows: Figure 3 As shown. By examining the static attribute matrix respectively , and dynamic attribute matrix , After padding and concatenation to obtain matrices with uniform dimensions, each matrix is ​​input into the encoder. The encoder then transforms these matrices to a higher dimension using a linear layer, followed by relative position encoding to obtain a high-dimensional static vector containing positional information. and dynamic vectors .

[0040] A2: The decoder uses a lightweight Transformer, with the specific structure as follows: Figure 4 As shown, input the static attribute vector of the task selected in the previous step. After passing through n Transformer encoding layers, each layer's specific architecture includes multi-head self-attention, residual and normalization, and feedforward network output, finally outputting the corresponding high-dimensional query vector. This is used by the subsequent attention mechanism to select the next task.

[0041] A3: The specific structure of the multi-head attention mechanism module is as follows: Figure 5 As shown, it receives the high-dimensional matrix of satellite-mission attributes output by the encoder. , After concatenation, the vectors are input into a multi-head attention projection layer to obtain the K and V vectors corresponding to the key-value pairs; these vectors are then input into the query vector generated by the decoder. The high-dimensional query vector Q is obtained through a multi-head attention projection layer. After additive attention calculation, the task probability distribution is obtained. ,in For the task selected in the previous step. Given the sequence of tasks to be planned in the previous step, this probability distribution is used to decide which task should be planned in the next step.

[0042] The deep reinforcement learning model training method in S4 is the Advantage Actor-Critic method, and the specific training method is as follows:

[0043] S41: The encoder + decoder + multi-head attention mechanism module described above is selected as the Actor network for task selection and generating planning sequences; an independent state-value network is constructed as the Critic, with the specific architecture as follows: Figure 7As shown, input static attribute matrix , and dynamic attribute matrix , After being encoded and concatenated, the current state score is calculated using an attention pooling network and a feedforward network. This is used to evaluate the performance of the current plan;

[0044] S42: Construct the reward function Reward, based on the advantage function using the training sample set. Synchronously update the Actor network policy parameters and the Critic network value estimation parameters, where For time step t, the value of the Actor network reward function. The corresponding critic scores the current state.

[0045] The Reward function, whose optimization objective is to minimize the reward for unfinished tasks, is as follows:

[0046]

[0047] in The length of the task sequence. Representative task The benefits, This is an indicator coefficient for mission completion, obtained by traversing the observation sequence of each satellite after mission planning is completed. Specifically, it is defined as follows: when the observation mission is successfully completed, the coefficient is 1; when the mission type is a multi-satellite collaborative mission, if only the observation mission of some satellites is completed, then a partial coefficient is obtained based on the number of satellites that completed the observation; if no satellites complete the mission, then it is 0.

[0048] S43: Iterate training until the reward converges, completing model training.

[0049] The task planning process of the deep reinforcement learning model in S4 is as follows: Figure 6 As shown, the process is described below:

[0050] 1) Input the static attribute matrix and dynamic attribute matrix corresponding to the satellite and the mission to be observed into the model, and output a high-dimensional vector of fused attributes and temporal features through the relative position encoder;

[0051] 2) The task selection mechanism is initiated in turn according to the satellite number order. The current task matching probability distribution is generated by combining the high-bit vector generated by the encoder with the decoder and multi-head attention mechanism module.

[0052] 3) Select the most probable and valid mission as the current satellite mission;

[0053] 4) Update the dynamic attribute matrix of the satellite and the mission to be observed, and filter out the missions that do not meet the constraints after the update, mark them, and avoid including them in the selection later;

[0054] 5) Repeat the above process until all satellites or missions have been assigned.

[0055] The deep reinforcement learning model is subject to several constraints when generating task sequences, as follows:

[0056] Mission-Satellite Type Compatibility Constraints: The satellites performing this mission must be of a type that meets the mission restrictions.

[0057] Mission-Satellite Imaging Constraints: The number of participating satellites in a collaborative mission cannot exceed the mission's limit.

[0058] Time window constraint: The satellite must start executing the mission within the mission's visible time window and remain within the time window after the observation ends;

[0059] Mission transfer time constraint: When the same satellite performs two missions, the start time of the latter mission must be later than the sum of the end time and maneuver time of the former mission;

[0060] Power constraint: When a satellite is performing an observation mission, the sum of the power consumed by imaging and the power consumed by attitude maneuvering shall not exceed the total power of the satellite.

[0061] Storage constraint: When a satellite performs an observation mission, the amount of storage consumed for imaging shall not exceed the satellite's storage space.

[0062] In this embodiment, the training environment is a Python simulation. A training dataset is obtained by simulating a satellite dataset. Through training with a large amount of data, a trained deep reinforcement learning model is obtained. The training process data includes, for example, various parameters, and other parameters. Figure 8 As shown.

[0063] This invention first constructs a training dataset of 150,000 sets by simulating the attribute parameters and observation task datasets of multiple types of satellite constellations. The dataset includes 50 tasks and 3 types and numbers of satellites, ensuring that the model's capabilities meet the requirements of practical applications.

[0064] The deep reinforcement learning model was optimized through iterative training with large datasets. Batch training was employed, with each batch containing 32 samples. The learning rate was dynamically adjusted, and the discount factor was fixed at 0.95. Key performance indicators (KPIs) were monitored in real-time during training, including the planning return rate, constraint satisfaction, and model loss on both the training and validation sets. Figure 8 shows the data for each indicator. The figure illustrates that the model's return rate increased rapidly in the early stages of iteration, rising from 11% to 96%. In the middle stages, the KPIs stabilized, with return rate fluctuations controlled within 3%, and the loss gradually converged to below 0.02. In the later stages, the model fully converged, achieving a final training set return rate of 96.3% and a validation set return rate of 96.0%, with no planning results violating constraints such as time windows, power consumption, or storage limitations.

[0065] After training, the model's performance was tested and validated. The test dataset used new scene data that was not used in the training. The test results show that the model's average inference time for a single set of 50 task sequences is 0.75 seconds, which is tens of times shorter than the traditional genetic algorithm. A graph of one set of training results is shown below. Figure 9 The image shows a visualization of the satellite mission planning sequence results.

[0066] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A multi-type satellite collaborative observation mission planning method based on deep reinforcement learning, characterized in that, Includes the following steps: S1: Obtain observation mission simulation data of the imaging satellite constellation and construct a satellite static attribute matrix based on the satellite static attributes; S2: Obtain mission information for imaging satellite observation missions from simulation data, and construct a mission static attribute matrix based on the mission information; S3: Based on the constructed satellite static attribute matrix and mission static attribute matrix, initialize the satellite dynamic attribute matrix and mission dynamic attribute matrix; S4: Construct a deep reinforcement learning network, which includes an encoder, a decoder, and a multi-head attention module. The encoder is used to obtain the relative position information of the attribute matrices of the satellite and the mission, and to map the matrix to a high dimension. The decoder takes the static feature vector of the previously selected mission as input and outputs the corresponding high-dimensional query vector. The multi-head attention module obtains the mission probability distribution and is used to decide the mission to be planned in the next step. Construct a reward function and train the deep reinforcement learning network using the attribute matrices constructed from S1 to S3 to obtain the trained deep reinforcement learning network. S5: Obtain the observation tasks to be planned and the available satellite sequences, and obtain the corresponding satellite static attribute matrix, the static attribute matrix of the observation tasks to be planned, the satellite dynamic attribute matrix, and the task dynamic attribute matrix. Input these into the trained deep reinforcement learning model. Under constraints, the deep reinforcement learning network maximizes the reward of the execution scheme and outputs the planned task execution sequence to complete the collaborative task planning of multiple types of satellites.

2. The multi-type satellite collaborative observation mission planning method based on deep reinforcement learning according to claim 1, characterized in that, In S1, the static attributes include satellite number, satellite type, total power, total storage capacity, attitude maneuver speed, and power consumption per unit distance of attitude maneuver. In S2, the mission information includes mission number, corresponding observation satellite number, mission type, observation benefits, power consumption, storage consumption, time window start time, observation duration, side swing angle, and time window end time.

3. The multi-type satellite collaborative observation mission planning method based on deep reinforcement learning according to claim 1, characterized in that, The initialization of the satellite's dynamic attribute matrix and the mission's dynamic attribute matrix as described in S3 includes: the satellite's dynamic attributes include the current remaining satellite power, remaining storage capacity, the execution time of the previous mission, and the previous mission number. The initial values ​​of the satellite's remaining power and remaining storage capacity are obtained by reading the satellite's static attribute matrix. The total power and storage capacity are obtained, and the remaining dynamic attributes are initialized to 0. The dynamic attributes of the mission include whether it is visible to the satellite and whether it has been observed by the satellite, and the initial values ​​are all 0.

4. The multi-type satellite collaborative observation mission planning method based on deep reinforcement learning according to claim 1, characterized in that, In S4, the encoder consists of two parts: a static encoder and a dynamic encoder. Both parts are composed of linear layers and relative position encoding. The static and dynamic attribute matrices are filled and concatenated to obtain matrices with uniform dimensions, which are then input into the encoder. The encoder transforms the matrix to a higher dimension through linear layers and then uses relative position encoding to generate a high-dimensional static vector containing positional information. and dynamic vectors .

5. The multi-type satellite collaborative observation mission planning method based on deep reinforcement learning according to claim 4, characterized in that, In S4, the decoder takes the static attribute vector of the selected task as input, passes it through n Transformer encoding layers, and outputs the corresponding high-dimensional query vector. This is used by the subsequent attention mechanism to select the next task.

6. The multi-type satellite collaborative observation mission planning method based on deep reinforcement learning according to claim 5, characterized in that, In S4, the multi-head attention mechanism module receives the high-dimensional static vector output by the encoder. and dynamic vectors After concatenation, the vectors are input into a multi-head attention projection layer to obtain the K and V vectors corresponding to the key-value pairs; these vectors are then input into the query vector generated by the decoder. The high-dimensional query vector Q is obtained through a multi-head attention projection layer. After additive attention calculation, the task probability distribution is obtained. ,in For the task selected in the previous step. Given the sequence of tasks to be planned in the previous step, this probability distribution is used to decide which task should be planned in the next step.

7. The multi-type satellite collaborative observation mission planning method based on deep reinforcement learning according to claim 1, characterized in that, In S4, the deep reinforcement learning model is trained using the Advantage Actor-Critic method, and the specific training method is as follows: S41: An encoder, decoder, and multi-head attention mechanism module are selected as the Actor network to select tasks and generate planning sequences; an independent state value network is constructed as the Critic, which takes static and dynamic attribute matrices as input, and after being encoded and concatenated, is calculated through an attention pooling network and a feedforward network to finally obtain the current state score. This is used to evaluate the performance of the current plan; S42: Construct the reward function Reward, based on the advantage function using the training sample set. Synchronously update the Actor network policy parameters and the Critic network value estimation parameters, where For time step t, the value of the Actor network reward function. The corresponding critic scores the current state. S43: Iterate training until the reward converges, completing model training.

8. The multi-type satellite collaborative observation mission planning method based on deep reinforcement learning according to claim 7, characterized in that, The specific Reward function is as follows: ； in The length of all task sequences, Representative task The benefits, This is an indicator coefficient for mission completion, obtained by traversing the observation sequence of each satellite after mission planning is completed. Specifically, it is defined as follows: when the observation mission is successfully completed, the coefficient is 1; when the mission type is a multi-satellite collaborative mission, if only the observation mission of some satellites is completed, then a partial coefficient is obtained based on the number of satellites that completed the observation; if no satellites complete the mission, then it is 0.

9. The multi-type satellite collaborative observation mission planning method based on deep reinforcement learning according to claim 1, characterized in that, In S5, the deep reinforcement learning network maximizes the reward of the execution plan under constraints, outputting a planned task execution sequence, specifically including: S51: Input the static attribute matrix and dynamic attribute matrix corresponding to the satellite and the mission to be observed into the model, and output a high-dimensional vector of fused attributes and temporal features after passing through the relative position encoder; S52: The task selection mechanism is initiated in turn according to the satellite number order. The current task matching probability distribution is generated by combining the high-bit vector generated by the encoder with the decoder and multi-head attention mechanism module. S53: Select the most probable and valid mission as the current mission for the satellite; S54: Update the dynamic attribute matrix of the satellite and the mission to be observed, and filter out missions that do not meet the constraints after the update, mark them, and avoid including them in the selection later; S55: Repeat S51 to S54 until all satellites or missions have been assigned.

10. The multi-type satellite collaborative observation mission planning method based on deep reinforcement learning according to claim 1, characterized in that, In S5, the constraints specifically include: Mission-Satellite Type Compatibility Constraints: The satellites and satellite types performing this mission must comply with the mission restrictions; Mission-Satellite Imaging Constraints: The number of participating satellites in a collaborative mission cannot exceed the mission's limit. Time window constraint: The satellite must start executing the mission within the mission's visible time window and remain within the time window after the observation ends; Mission transfer time constraint: When the same satellite performs two missions, the start time of the latter mission must be later than the sum of the end time and maneuver time of the former mission; Power constraint: When a satellite is performing an observation mission, the sum of the power consumed by imaging and the power consumed by attitude maneuvering shall not exceed the total power of the satellite. Storage constraint: When a satellite performs an observation mission, the amount of storage consumed for imaging shall not exceed the satellite's storage space.

Images