Star group task planning method based on deep reinforcement learning
Based on the method of deep strengthening learning, the star group task planning model is optimized, and the problems of large -scale star group tasks in existing technologies have a complicated and huge space for solving space, and achieved more efficient and better task planning results.
Patent Information
- Application Number
- CN202411162867.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2025-01-21
AI Technical Summary
When facing large-scale star cluster task planning, the existing technology has complex constraints and huge solution space, resulting in poor optimization of the planning results and low efficiency.
Adopt the method of star group task planning based on deep -enhanced learning, through steps such as state construction, strategic model construction, action selection, reward calculation, status update and model update. Essence
The optimal and efficiency of the results of the star group task planning can be effectively compatible with the task planning needs of different satellites and target numbers, and have strong flexibility and robustness.
Smart Images
Figure CN119338145A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a constellation mission planning method based on deep reinforcement learning, and belongs to the field of satellite mission planning. Background Art
[0002] At present, with the increase in the number of satellites and the diversification of observation requirements, the constellation mission planning method plays an important role in exerting the overall effectiveness of the constellation and improving the efficiency of task completion. The constellation mission planning method aims to effectively allocate satellite resources based on information such as satellite resource capabilities, status, and observation mission requirements, and to solve the allocation relationship between observation tasks and satellite resources and the timing of task execution.
[0003] Traditional constellation mission planning methods mainly include rule-based mission planning methods, heuristic method-based mission planning methods, and contract network-based mission planning methods. The rule-based mission planning method arranges tasks based on simple rules, such as sorting tasks according to priority, and then planning them in sequence, but the optimality of the planning results is generally poor; heuristic methods include genetic, annealing and other methods. The optimality of the mission planning results based on heuristic methods is better than that of rule-based methods, but the iterative solution process reduces the planning efficiency; the contract network-based mission planning allocates tasks through repeated negotiations between satellites. This method has a high planning efficiency for a small number of tasks, but the inter-satellite negotiation process will result in a large amount of communication overhead. Summary of the invention
[0004] The technical problem solved by the present invention is: to overcome the shortcomings of the existing technology, to face the technical challenges of complex constraints and huge solution space in large-scale star cluster mission planning, and to provide a star cluster mission planning method based on deep reinforcement learning, which improves the optimality of planning results and improves planning efficiency.
[0005] The technical solution of the present invention is:
[0006] A constellation mission planning method based on deep reinforcement learning, comprising:
[0007] (1) State construction: Encode and integrate the input information related to the constellation mission planning to form the state used by the strategy model;
[0008] (2) Build a strategy model and an evaluation model: The strategy model input is the current state, and the output is the action, that is, the single-step decision result; the evaluation model input is the current state, and the output is the evaluation value of the current state;
[0009] (3) Action selection: The constructed state is input into the policy model, and the action is output, i.e., the single-step planning result. The single-step planning result includes two actions, namely, selecting the visible window and calculating the observation start time.
[0010] (4) Reward calculation: For the single-step planning results output by the strategy model, the reward is calculated based on the planning target priority and target observation quality;
[0011] (5) Update the state: Based on the single-step planning results and state output by the strategy model, the visible window information whose time is occupied in the state is updated according to the satellite observation time, maneuvering time, and target observation times, and the updated state is output;
[0012] (6) Perform model update: Repeat steps (3) to (5) until the optional visible window in the state is empty, obtain the state update sequence and reward value sequence in the planning process, and store them in the data pool; when the accumulated number of samples reaches the capacity of the data pool, update the policy model and evaluation model based on the proximal policy optimization algorithm according to the state update sequence and reward sequence in the data pool, and complete a round of training; finally, multiple rounds of training are performed until the policy network and evaluation network converge.
[0013] Furthermore, the input information includes the visible window of each satellite for each target, the value of each target, the observation time requirement of each target, the maneuvering time of each satellite to observe each target, the number of remaining observable targets of the satellite, and the target priority.
[0014] Furthermore, the input information is encoded and integrated to form the state s used by the strategy solving model i , specifically:
[0015] s i =[ws n,m,j ,we n,m,j ,ob m ,mv n,m,j ,stor n ,value m ,n,m]
[0016] Among them, ws n,m,j and we n,m,j represent the start and end time of the jth visible window of the nth satellite to the mth target, respectively. m represents the observation time requirement of target m, mv n,m,j represents the lead time required by the nth satellite to observe the mth target in the jth visible window, stor n Indicates the number of remaining observable tasks of satellite n, value m Indicates the priority of target m. The final complete state s = [s 1 ,…,s I ], where I represents the total number of visible windows.
[0017] Furthermore, the strategy model is a Transformer-Encoder-based strategy model, and the model can receive variable-length states as input to adapt to planning scenarios with different numbers of satellites and targets.
[0018] Furthermore, the strategy model is specifically constructed as follows:
[0019] The strategy model consists of two neural networks, which respectively realize the two actions of selecting the visible window and calculating the start time of the task. The two neural networks are in series. The first neural network selects the visible window, and then inputs the visible window selection result into the second neural network to output the actual start time of the task.
[0020] Furthermore, the input of the evaluation model is the state, and the output is a scalar, which represents the score value of the current state; the basic structures of the strategy model and the evaluation model are both based on
[0021] Transformer-Encoder, used to extract features from the state.
[0022] Furthermore, selecting a visible window means selecting a non-empty visible window from the visible window set contained in the state, indicating that the observation behavior planned this time will appear in the visible window, and the action space corresponding to the visible window is selected as the discrete action space;
[0023] Calculating the observation start time means calculating the observation start time from the selected visible window. At this time, the output of the strategy model is the offset time of the observation start time relative to the selected visible window. The action space corresponding to the observation start time is a continuous action space. The observation end time point can be obtained by adding the observation start time point and the target required observation duration.
[0024] Based on the two actions of selecting the visible window and calculating the observation start time, and combining the attributes of the observation window in the state, the complete information of the current plan is obtained, including: the observed target number, the observed satellite number, the observation start time and the observation end time.
[0025] Furthermore, the reward calculation takes into account two indicators: target priority and observation quality. The target priority is:
[0026] f 1 =value m
[0027] The target observation quality is expressed as the degree of displacement of the observation window center relative to the visible window center f 2 :
[0028]
[0029] Among them, osn,m ,oe n,m They represent the start time and end time of observation of the mth target by the nth satellite in the single-step planning results.
[0030] In a second aspect, the present invention further proposes a constellation mission planning system based on deep reinforcement learning, comprising:
[0031] State building module: encodes and integrates the input information related to the constellation mission planning to form the state used by the strategy model;
[0032] Strategy model and evaluation model building module: The input of the strategy model is the current state, and the output is the action, that is, the single-step decision result; the input of the evaluation model is the current state, and the output is the evaluation value of the current state;
[0033] Action selection module: inputs the constructed state into the policy model and outputs the action, i.e., the single-step planning result; the single-step planning result includes two actions, namely, selecting the visible window and calculating the observation start time;
[0034] Reward calculation module: For the single-step planning results output by the policy model, the reward calculation is performed based on the planning target priority and target observation quality;
[0035] State update module: Based on the single-step planning results and states output by the strategy model, the visible window information of the state whose time is occupied is updated according to the satellite observation time, maneuvering time, and target observation times, and the updated state is output;
[0036] Model update module: Repeat the operations of the action selection module, reward calculation module and state update module until the optional visible window in the state is empty, obtain the state update sequence and reward value sequence in the planning process, and store them in the data pool; when the accumulated number of samples reaches the capacity of the data pool, according to the state update sequence and reward sequence in the data pool, based on the proximal policy optimization algorithm, update the policy model and evaluation model to complete a round of training; finally, after multiple rounds of training, the policy network and evaluation network converge.
[0037] In a third aspect, the present invention further proposes a computer program product, which, when executed by a processor, implements the constellation mission planning method based on deep reinforcement learning.
[0038] The beneficial effects of the present invention compared with the prior art are:
[0039] (1) The present invention solves the problem of planning the observation mission of a star cluster to a location target based on a deep reinforcement learning method. On the one hand, the planning model can be optimized through the reinforcement learning training process to improve the optimization performance of the model; on the other hand, the trained model can be directly used for reasoning to improve the efficiency of task planning.
[0040] (2) Traditional star cluster mission planning methods are mostly based on heuristic methods to iteratively solve, which takes a long time to solve and has poor optimality of the solution. The star cluster mission planning method proposed in the present invention optimizes the model in advance based on reinforcement learning for multiple rounds, and directly applies the optimized model to solve the planning scheme. The solution speed and optimality of the solution are better than those of traditional methods.
[0041] (3) The design of the state, model and other elements in this method is scalable. This method can be compatible with mission planning requirements with different numbers of satellites and targets without retraining the model, and has strong flexibility and robustness.
[0042] (4) This method comprehensively considers constraints such as visibility, satellite maneuverability, satellite storage, and energy, and the resulting planning scheme is closer to practical application. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 is a schematic diagram of the model structure;
[0044] Figure 2 Select a schematic diagram for the action;
[0045] Figure 3 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0046] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.
[0047] The present invention proposes a swarm task planning method based on deep reinforcement learning. First, the input information of the swarm task planning method problem is encoded to form an initial state, and then a Transformer-based strategy model is built. Samples are further accumulated through action selection, reward calculation, and state update. Finally, the strategy model is trained through the PPO training method to obtain a strategy model that can be directly used for reasoning. Figure 3 As shown, the specific implementation is as follows:
[0048] Step 1: State construction. Encode and integrate the input information related to the constellation mission planning to form the state used by the strategy model. The input information includes the visible window of each satellite for each target, the value of each target, the observation time requirement of each target, the maneuvering time of each satellite to observe each target, the number of remaining observable targets of the satellite, and the priority of the target.
[0049] Encode and integrate the input information to form the state s used by the strategy solving model i , specifically:
[0050] s i =[ws n,m,j ,we n,m,j,ob m ,mv n,m,j ,stor n ,value m ,n,m]
[0051] Among them, ws n,m,j and we n,m,j represent the start and end time of the jth visible window of the nth satellite to the mth target, respectively. m represents the observation time requirement of target m, mv n,m,j represents the lead time required by the nth satellite to observe the mth target in the jth visible window, stor n Indicates the number of remaining observable tasks of satellite n, value m Indicates the priority of target m. The final complete state s = [s 1 ,…,s I ], where I represents the total number of visible windows.
[0052] Step 2: Model building, building strategy model and evaluation model.
[0053] The input of the strategy model is the current state, that is, the single-step decision result; the output is two actions: selecting the visible window and calculating the observation start time. The model design corresponds to the action design. The selection of the visible window and the calculation of the task start time are respectively realized through two parts of the neural network. The two parts of the model are in series.
[0054] In order to adapt to the reinforcement learning architecture of Actor-Critic, an evaluation model is built in addition to the policy model. The input of the evaluation model is the state, and the output is a scalar, which represents the evaluation value of the current state. The basic structure of both models uses a multi-head attention layer (Transformer-Encoder) to extract features from the global visible window. The model structure is as follows Figure 1 shown.
[0055] Step 3: Action selection, input the constructed state into the policy model and output the action, that is, the single-step planning result; the single-step planning result includes two actions, namely selecting the visible window and calculating the observation start time.
[0056] like Figure 2As shown in the figure, according to the modeling results of the serial solution of the task planning problem, the action of a single decision of the model is divided into two parts: selecting a visible window and calculating the observation start time. Selecting a visible window means selecting a non-empty visible window from the visible window set contained in the state, indicating that the observation behavior planned this time will appear in the visible window. The action space corresponding to this action is a discrete action space. Calculating the observation start time means calculating the start time of the observation from the selected visible window. At this time, the output of the model is the offset time of the observation start time relative to the selected visible window. The action space corresponding to this action is a continuous action space. The observation start time point and the target required observation duration are further added to obtain the observation end time point. Based on the two actions of selecting the visible window and calculating the observation start time, and combined with the attributes of the observation window in the state, the complete information of the current plan can be obtained, including: the observed target number, the observation satellite number, the observation start time and the observation end time.
[0057] Step 4: Reward calculation.
[0058] The reward design mainly considers two indicators: target priority and observation quality. The target priority refers to the priority of the target selected at that time:
[0059] f 1 =value m #(1)
[0060] The target observation quality is expressed as the degree of deviation of the observation window center relative to the visible window center:
[0061]
[0062] Step 5: State update. Based on the single-step planning results and states output by the strategy model, the visible window information whose time is occupied in the state is updated according to the satellite observation time, maneuvering time, and target observation times, and the updated state is output.
[0063] Based on the results of the strategic network planning, the environment updates the state. The state update is mainly a local update. According to the state of the visible window before and after the update, it can be divided into "modification" (including "shortening" and "splitting") and "deletion" of the visible window. The update and deletion of the visible window have the following conditions
[0064] 1) Modification of visible windows: If the planning result is that satellite i observes target t in a certain period of time, all visible windows corresponding to the satellite are deducted from the part that coincides with the period of time (row direction);
[0065] 2) Deletion of visible windows: When the number of observations of target t reaches the requirement, delete all satellites’ visible windows of the target (column direction). When a satellite reaches the maximum number of observations, delete all visible windows corresponding to the satellite (row direction). When the updated observation window length in 1) is shorter than the maneuvering time + observation time, delete the visible window.
[0066] Step 6: Model update.
[0067] Repeat steps 3, 4, and 5 until the optional visible window in the state is empty, and obtain the state update sequence and reward value sequence in the planning process, and store them in the data pool. When the number of accumulated samples reaches the capacity of the data pool, the policy network and the evaluation network are updated based on the proximal policy optimization (PPO) algorithm according to the state update sequence and reward sequence in the data pool, and a round of training is completed. Finally, after multiple rounds of training, the policy network and the evaluation network converge.
[0068] According to the above algorithm, multi-step decision-making is performed. After a certain number of samples are accumulated, the model is trained and updated according to the historical status and rewards. The proximal gradient descent (PPO) method is a reinforcement learning method under the Actor-Critic architecture. It supports the optimization of a hybrid action space composed of a discrete action space and a continuous action space, and the method tuning is relatively simple. To this end, the policy network is trained based on the PPO method. After multiple rounds of training until the policy model converges, a constellation task planning model that can be directly used for reasoning is obtained.
[0069] Parts of the present invention that are not described in detail belong to common knowledge among those skilled in the art.
Claims
1. A constellation mission planning method based on deep reinforcement learning, characterized in that include: (1) State construction: Encode and integrate the input information related to the constellation mission planning to form the state used by the strategy model; (2) Build a strategy model and an evaluation model: The strategy model input is the current state, and the output is the action, that is, the single-step decision result; the evaluation model input is the current state, and the output is the evaluation value of the current state; (3) Action selection: The constructed state is input into the policy model, and the output action is the single-step planning result. The single-step planning result includes two actions, namely, selecting the visible window and calculating the observation start time. (4) Reward calculation: For the single-step planning results output by the strategy model, the reward is calculated based on the planning target priority and target observation quality; (5) Update the state: Based on the single-step planning results and state output by the strategy model, the visible window information whose time is occupied in the state is updated according to the satellite observation time, maneuvering time, and target observation times, and the updated state is output; (6) Model update: Repeat steps (3) to (5) until the optional visible window in the state is empty, obtain the state update sequence and reward value sequence in the planning process, and store them in the data pool; When the number of accumulated samples reaches the capacity of the data pool, the policy model and evaluation model are updated based on the proximal policy optimization algorithm according to the state update sequence and reward sequence in the data pool to complete a round of training; finally, multiple rounds of training are carried out until the policy network and evaluation network converge.
2. The method for constellation mission planning based on deep reinforcement learning according to claim 1, characterized in that: The input information includes the visible window of each satellite for each target, the value of each target, the observation time requirement of each target, the maneuvering time of each satellite to observe each target, the number of remaining observable targets of the satellite, and the target priority.
3. The method for constellation mission planning based on deep reinforcement learning according to claim 2, characterized in that: Encode and integrate the input information to form the state s used by the strategy solving model i , specifically: s i =[ws n,m,j ,we n,m,j ,ob m ,mv n,m,j ,stor n ,value m ,n,m] Among them, ws n,m,j and we n,m,j represent the start and end time of the jth visible window of the nth satellite to the mth target, respectively. m represents the observation time requirement of target m, mv n,m,j represents the lead time required by the nth satellite to observe the mth target in the jth visible window, stor n Indicates the number of remaining observable tasks of satellite n, value m represents the priority of target m. The final complete state s=[s1,…,s I ], where I represents the total number of visible windows.
4. The method for constellation mission planning based on deep reinforcement learning according to claim 1, characterized in that: The strategy model is a Transformer-Encoder-based strategy model, and the model can receive variable-length states as input to adapt to planning scenarios with different numbers of satellites and targets.
5. The method for constellation mission planning based on deep reinforcement learning according to claim 1, characterized in that: The construction strategy model is specifically as follows: The strategy model consists of two neural networks, which respectively realize the two actions of selecting the visible window and calculating the start time of the task. The two neural networks are in series. The first neural network selects the visible window, and then inputs the visible window selection result into the second neural network to output the actual start time of the task.
6. The method for constellation mission planning based on deep reinforcement learning according to claim 5, characterized in that: The input of the evaluation model is the state, and the output is a scalar, which represents the score value of the current state; the basic structures of the strategy model and the evaluation model both use Transformer-Encoder to extract features from the state.
7. The method for constellation mission planning based on deep reinforcement learning according to claim 5, characterized in that: Selecting a visible window means selecting a non-empty visible window from the visible window set contained in the state, indicating that the observation behavior planned this time will appear in the visible window, and the action space corresponding to the visible window is selected as the discrete action space; Calculating the observation start time means calculating the observation start time from the selected visible window. At this time, the output of the strategy model is the offset time of the observation start time relative to the selected visible window. The action space corresponding to the observation start time is a continuous action space. The observation end time point can be obtained by adding the observation start time point and the target required observation duration. Based on the two actions of selecting the visible window and calculating the observation start time, and combining the attributes of the observation window in the state, the complete information of the current plan is obtained, including: the observed target number, the observed satellite number, the observation start time and the observation end time.
8. The method for constellation mission planning based on deep reinforcement learning according to claim 1, characterized in that: The reward calculation mentioned above considers two indicators: target priority and observation quality. Target priority: f1=value m The target observation quality is expressed as the degree of deviation f2 of the observation window center relative to the visible window center: Among them, os n,m ,oe n,m They represent the start time and end time of observation of the mth target by the nth satellite in the single-step planning results.
9. A constellation mission planning system based on deep reinforcement learning, characterized in that include: State building module: encodes and integrates the input information related to the constellation mission planning to form the state used by the strategy model; Strategy model and evaluation model building module: The input of the strategy model is the current state, and the output is the action, that is, the single-step decision result; the input of the evaluation model is the current state, and the output is the evaluation value of the current state; Action selection module: inputs the constructed state into the policy model and outputs the action, i.e., the single-step planning result; the single-step planning result includes two actions, namely, selecting the visible window and calculating the observation start time; Reward calculation module: For the single-step planning results output by the policy model, the reward calculation is performed based on the planning target priority and target observation quality; State update module: Based on the single-step planning results and states output by the strategy model, the visible window information of the state whose time is occupied is updated according to the satellite observation time, maneuvering time, and target observation times, and the updated state is output; Model update module: repeat the operations of action selection module, reward calculation module and state update module until the optional visible window in the state is empty, obtain the state update sequence and reward value sequence in the planning process, and store them in the data pool; When the number of accumulated samples reaches the capacity of the data pool, the policy model and evaluation model are updated based on the proximal policy optimization algorithm according to the state update sequence and reward sequence in the data pool to complete a round of training; finally, multiple rounds of training are carried out until the policy network and evaluation network converge.
10. A computer program product, characterized in that When the computer program product is executed by a processor, the star cluster mission planning method based on deep reinforcement learning described in any one of claims 1 to 8 is implemented.