Autonomous mission planning method and system for deep space probes based on dynamic reward reinforcement learning

Abstract

The present invention discloses a deep space probe autonomous task planning method and system based on dynamic reward reinforcement learning, the method includes defining reinforcement learning parameters for deep space probe task planning; constructing an intelligent agent interaction environment; constructing a policy network; training the policy network; and executing task planning using the trained policy network. The present invention integrates the time constraints, resource constraints and timing constraints in deep space exploration task planning into the training process of the deep space probe task planning reinforcement learning model, which can ensure that multiple constraints are met in the deep space probe task planning process, and enhance the adaptability of the deep space probe intelligent agent to complex environments. At the same time, the dynamic reward algorithm integrates the resource constraints, time constraints and timing constraints to construct an instant reward model, and defines the loss function to update the policy gradient accordingly, which accelerates the convergence speed of the deep space probe autonomous task planning reinforcement learning model under the premise of ensuring multiple constraints, reduces the deviation, and improves the quality of autonomous task planning.

Description

Technical Field

[0001] The invention relates to a deep space probe autonomous task planning method based on dynamic reward reinforcement learning, belonging to the field of aerospace technology. Background Art

[0002] The field of deep space exploration is one of the important technological development areas in the world today. Due to the incompleteness of the known characteristics of the detection objects and the complexity of the environment, the attachment mission is very difficult. In order to successfully complete the scientific exploration mission, deep space probes need to have a high degree of autonomy and be able to independently plan tasks in complex and unknown environments.

[0003] During the attachment process, a deep space probe needs to have the ability to plan various tasks and plan a set of executable action sequences based on various external environments, the status of its own system, and various constraints.

[0004] In previous probe planning research, some scholars proposed a two-stage spacecraft autonomous mission planning algorithm that includes platform and payload mission management, realizing integrated platform and payload autonomous mission management of spacecraft, which can output complete instruction sequences according to high-level mission objectives.

[0005] Many traditional planning issues in the field of deep space exploration are too passive, the detectors are not sufficiently autonomous, and they are unable to respond well to emergencies. Summary of the invention

[0006] The present invention provides a deep space probe autonomous task planning method based on dynamic reward reinforcement learning, which is used to solve the task planning problem of the deep space probe when performing deep space exploration missions.

[0007] The technical solution of the present invention is: a method for autonomous mission planning of deep space probes based on dynamic reward reinforcement learning, comprising:

[0008] S1. Define the reinforcement learning parameters for deep space probe mission planning;

[0009] S2, build an intelligent agent interaction environment;

[0010] S3, build a strategy network;

[0011] S4, training strategy network;

[0012] S5. Use the trained strategy network to perform task planning.

[0013] The definition of the reinforcement learning parameters for deep space probe mission planning includes:

[0014] Definition 1. Task. The task is expressed by formula (1):

[0015] F={I F ,N F ,E FL ,E FP ,t FES ,t FLE} (1)

[0016] Where: F represents the scientific mission that can be performed by the deep space probe; I F Represents the number of task F; N F Represents the name of task F; E FL represents the task value of task F; E FP represents the priority of task F; t FES represents the earliest start time of task F; t FLE Represents the latest start time of task F;

[0017] Definition 2: Resource set: A resource set is a set of resources that a deep space probe will use when performing an action. It is represented by R, as shown in formula (2):

[0018] R={r1,r2,...,r i ,...,r |R|} (2)

[0019]

[0020] Where: r i represents the i-th resource; |R| represents the number of elements in the set R, that is, the number of resource types; Represents the number of the i-th resource; Represents the name of the i-th resource; represents the remaining amount of the i-th resource; Represents the type of the i-th resource, that is, whether the resource is a renewable resource or a non-renewable resource;

[0021] Definition 3: Deep space probe state set. Let S represent the state set of a deep space probe. Assume that the probe has N subsystems. Let S represent the state set of the jth subsystem. j Indicates that its status is |S j | Species; S j It is expressed as formula (4), and S can be expressed as formula (5), which is:

[0022]

[0023] S=S1∪S2∪...S j ...∪S N (5)

[0024] Where: s jkrepresents the kth state of the jth subsystem; |S j | represents the set S j The number of elements of S, that is, the number of states of the j-th subsystem; j represents the state set of the jth subsystem;

[0025] Definition 4: executable action; executable action is the action that deep space probe can perform; use a jh It represents the hth executable action of the jth subsystem, as shown in formula (6):

[0026]

[0027]

[0028] Where: Represents action a jh Serial number; Substitute action a jh Duration of the movement; Represents action a jh The resource set consumed; R represents the resource set of the deep space probe, is a subset of R;

[0029] Definition 5: The executable action set of deep space probes; A is used to represent the executable action set of deep space probes; the executable action set of the jth subsystem is represented by A j Indicates that its types are |A j | Species; A j It is expressed as formula (8), and A is expressed as formula (9):

[0030]

[0031] A=A1∪A2∪...A j ...∪A N (9)

[0032] Where: |A j | represents the number of elements in the set, that is, the number of executable actions of the jth subsystem; A j represents the set of executable actions of the jth subsystem;

[0033] Definition 6, deep space probe state transition set;

[0034] The triple is constructed as G = (S, A, S); where G represents the state transition set of deep space probe states and actions, S represents the deep space probe state set, which has |S| states; A represents the deep space probe executable action set, which has |A| executable actions; the basic form of the triple is (s jk ,a jh ,sjk+1 ), where s jk represents the kth state of the jth subsystem, s jk+1 Indicates that the jth subsystem performs action a jh The k+1th state reached after s; the first element s in the triple G jk From S, the second element a jh From A, the third element s jk+1 From S.

[0035] The constructing of the intelligent agent interaction environment includes: a deep space probe state space library, a deep space probe action space library, and a deep space probe state transfer space library.

[0036] The constructing of the intelligent agent interaction environment includes:

[0037] Deep space detector state space library: The deep space detector is regarded as an intelligent agent. The state of the intelligent agent is defined as the state of each subsystem in the deep space detector. That is, the state of the intelligent agent comes from the deep space detector state space library State composed of the states of each subsystem of the deep space detector, where State = S;

[0038] Deep space probe action space library: Each subsystem of the deep space probe has an executable action set. The executable action set of the jth subsystem is A j , as shown in formula (8), the executable action a in this set jh Can make the subsystem from state s jk Arrivals jk+1 ; When the state is s jk In the case of j A subset of jv , total |A jv | species, as shown in formulas (10) and (11), the deep space detector action space library is defined as Action as shown in formula (12):

[0039]

[0040]

[0041]

[0042] Where: A jv A represents the set of actions that can be performed by the jth subsystem in the vth state. j represents the executable action set of the jth subsystem; since A jv Yes A j A is a subset of jv The elements in come from A j ,|A jv| represents the number of actions that can be performed by the j-th subsystem in the v-th state, represents the jth subsystem in the vth state |A jv | executable actions; A jv represents the set of executable actions in the vth state of the jth subsystem. The deep space probe is divided into N subsystems, among which the Nth subsystem has |S N | kind of state;

[0043] Deep space probe state transfer space library;

[0044] The deep space probe state transfer space library Transfer is defined by triples. The state transfer contained in the library means that the agent will transfer from the current state to another state after performing a certain action. Transfer is expressed by formula (13):

[0045] Transfer = {(s 11 ,a 11 ,s 12 ),...,(s jk ,a jh ,s jk+1 ),...} (13).

[0046] The construction of the strategy network includes:

[0047] Combined with the constructed agent interaction environment, a 4-layer MLP strategy network structure is defined; the input layer of the network is the current state of the deep space probe agent, represented by 2 parameters, the hidden layer has 2 layers, each with 40 nodes, and the output is the probability of the deep space probe agent taking each action under the input layer state;

[0048] Definition of the loss function that integrates time constraints, resource constraints, and timing constraints:

[0049] Time constraint: Establish the action time consumption matrix T, as shown in formulas (14) and (15). This matrix saves the time consumption of executable actions of each subsystem to meet the time constraints in the planning process;

[0050] M=Max{|A1|,|A2|,...,|A j |,...,|A N |} (14)

[0051]

[0052] Where: |A j| is the number of executable actions of the jth subsystem of the deep space probe; M is the largest number of executable actions corresponding to the N subsystems of the deep space probe; T is the action time consumption matrix, t ju is the time required for the uth executable action of the jth subsystem; for any subsystem, if the number of executable actions is less than M, the corresponding time required is -1;

[0053] Resource constraints: Establish the action resource consumption matrix L, as shown in formula (16). This matrix saves the resource consumption of the executable actions of each subsystem to meet the resource constraints in the planning process;

[0054]

[0055] Where: r xy The resources consumed by the yth executable action of the xth subsystem;

[0056] Timing constraints: For deep space probe mission planning, there are timing constraints. Define action a jh The set of preceding actions is The previous action set contains the actions that must be executed before the action is executed, as shown in formula (17):

[0057]

[0058] Where: is a subset of A;

[0059] Taking into account the above resource, time and timing constraints, a dynamic reward strategy is adopted to instantly reward Y m The calculation of is shown in formula (19):

[0060]

[0061]

[0062] Where: E represents the action sequence that the deep space exploration agent has planned when performing an Episode, Y m Indicates that the mth action in E is a jh The instant reward at time t jh 、r jh They represent the time consumed and the resources consumed by the h-th executable action of the j-th subsystem respectively;

[0063] Therefore, the expected reward G of the currently selected executable action is m The definition is as follows:

[0064]

[0065] Where: γ is the discount factor, 0<γ≤1, anm represents the total number of executable actions in the current Episode that needs to be calculated;

[0066] According to the policy gradient descent method, a loss function as shown in formula (21) is constructed;

[0067]

[0068] Where: a gh is the currently selected action, π θ (a jh |s jk ) is the action prediction strategy.

[0069] The training strategy network comprises:

[0070] Define the terminal state s of the task from the deep space probe state space library in the agent interaction environment ow ; Among them, s ow represents the wth state of the oth subsystem;

[0071] Randomly generate an initial state s from the deep space probe state space library in the agent interaction environment pq Then, the constructed policy network is used to plan a set of episodes, and a set of episodes is used as input to update the policy network according to the defined formula (21); until the policy network converges, a trained policy network is obtained; where s pq Represents the qth state of the pth subsystem.

[0072] The step of using the trained strategy network to perform task planning includes:

[0073] According to the scientific task F to be performed, the initial state and the terminal state are input, and the trained strategy network is used for planning until the agent reaches the terminal state and ends the planning; the planning results are drawn into a task Gantt chart of each subsystem of the deep space probe.

[0074] According to another aspect of an embodiment of the present invention, there is also provided a deep space probe autonomous mission planning system based on dynamic reward reinforcement learning, including:

[0075] Definition unit, used to define the reinforcement learning parameters for deep space probe mission planning;

[0076] The first construction unit is used to construct an intelligent agent interaction environment;

[0077] The second building unit is used to build a policy network;

[0078] Training unit, used to train the policy network;

[0079] The execution unit is used to execute task planning using the trained policy network.

[0080] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is used to run a program, wherein when the program is running, the method for autonomous mission planning of deep space probes based on dynamic reward reinforcement learning is executed as described in any one of the above.

[0081] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, which includes a stored program, and when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned dynamic reward-based reinforcement learning deep space probe autonomous mission planning methods.

[0082] The beneficial effects of the present invention are as follows: the present invention defines reinforcement learning parameters for deep space probe mission planning, and constructs a new intelligent agent interactive environment adapted to deep space exploration planning in combination with the learning parameters; based on the interactive environment, the time constraints, resource constraints and timing constraints in deep space exploration mission planning are integrated into the training process of the deep space probe mission planning reinforcement learning model, which can ensure that multiple constraints are met during the deep space probe mission planning process, and enhance the adaptability of the deep space probe intelligent agent to complex environments; at the same time, the dynamic reward algorithm integrates resource constraints, time constraints and timing constraints to construct an instant reward model, and defines the loss function to update the policy gradient accordingly, which accelerates the convergence speed of the deep space probe autonomous mission planning reinforcement learning model under the premise of ensuring multiple constraints, reduces deviations, and improves the quality of autonomous mission planning; simulation experimental results show that the planning success rate of the method of the present invention is significantly improved compared with the static reward policy gradient algorithm, thereby providing a new solution for autonomous planning and decision-making of deep space probes. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 is a flow chart of the present invention;

[0084] Figure 2 This is a schematic diagram of the action output based on the policy gradient reinforcement learning method;

[0085] Figure 3 It is a task Gantt chart of each subsystem of a deep space probe obtained by an example of the present invention;

[0086] Figure 4 This is a comparison chart of the success rates of the method of the present invention and the static reward strategy gradient algorithm. DETAILED DESCRIPTION

[0087] The invention will be further described below in conjunction with the accompanying drawings and embodiments, but the content of the invention is not limited to the scope of the embodiments.

[0088] Example 1: Figure 1-4 As shown, a method for autonomous mission planning of deep space probes based on reinforcement learning of dynamic rewards includes: S1, defining reinforcement learning parameters for deep space probe mission planning; S2, building an intelligent agent interaction environment; S3, building a strategy network; S4, training the strategy network; S5, using the trained strategy network to execute mission planning.

[0089] The definition of the reinforcement learning parameters for deep space probe mission planning includes:

[0090] Definition 1. Task. The task is expressed by formula (1):

[0091] F={I F ,N F ,E FL ,E FP ,t FES ,t FLE} (1)

[0092] Where: F represents the scientific mission that can be performed by the deep space probe; I F Represents the number of task F; N F Represents the name of task F; E FL represents the task value of task F; E FP represents the priority of task F; t FES represents the earliest start time of task F; t FLE Represents the latest start time of task F;

[0093] Definition 2: Resource set: A resource set is a set of resources that a deep space probe will use when performing an action. It is represented by R, as shown in formula (2):

[0094] R={r1,r2,...,r i ,...,r |R|} (2)

[0095]

[0096] Where: r i represents the i-th resource; |R| represents the number of elements in the set R, that is, the number of resource types; Represents the number of the i-th resource; Represents the name of the i-th resource; represents the remaining amount of the i-th resource; Represents the type of the i-th resource, that is, whether the resource is a renewable resource or a non-renewable resource;

[0097] Definition 3: Deep space probe state set. Let S represent the state set of a deep space probe. Assume that the probe has N subsystems. Let S represent the state set of the jth subsystem. j Indicates that its status is |S j | Species; S j It is expressed as formula (4), and S can be expressed as formula (5), which is:

[0098]

[0099] S=S1∪S2∪...S j ...∪S N (5)

[0100] Where: s jk represents the kth state of the jth subsystem; |S j | represents the set S j The number of elements of S, that is, the number of states of the j-th subsystem; j represents the state set of the jth subsystem;

[0101] Definition 4: executable action; executable action is the action that deep space probe can perform; use a jh It represents the hth executable action of the jth subsystem, as shown in formula (6):

[0102]

[0103]

[0104] Where: Represents action a jh Serial number; Substitute action a jh Duration of the movement; Represents action a jh The resource set consumed; R represents the resource set of the deep space probe, is a subset of R;

[0105] Definition 5: The executable action set of deep space probes; A is used to represent the executable action set of deep space probes; the executable action set of the jth subsystem is represented by A j It is represented by a in formula (6) jh Composition, its types are |A j | Species; A j It is expressed as formula (8), and A is expressed as formula (9):

[0106]

[0107] A=A1∪A2∪...A j ...∪AN (9)

[0108] Where: |A j | represents the number of elements in the set, that is, the number of executable actions of the jth subsystem; A j represents the set of executable actions of the jth subsystem;

[0109] Definition 6, deep space probe state transition set;

[0110] The triple is constructed as G = (S, A, S); where G represents the state transition set of deep space probe states and actions, S represents the deep space probe state set, which has |S| states; A represents the deep space probe executable action set, which has |A| executable actions; the basic form of the triple is (s jk ,a jh ,s jk+1 ), where s jk represents the kth state of the jth subsystem, s jk+1 Indicates that the jth subsystem performs action a jh The k+1th state reached later; the first element s in the triple G jk From S, the second element a jh From A, the third element s jk+1 From S.

[0111] The autonomous mission planning of deep space probes is based on the perception and understanding of the space environment, the capabilities and status of the probe itself, and the use of computer software technology, artificial intelligence theoretical methods, etc., based on the mission objectives within a period of time, to infer a number of optional actions, available resources, constraints, etc., and automatically generate a set of time-ordered action sequences. Once the plan is executed, the state of the probe can be successfully transferred to the desired target state. Obviously, the present invention formally defines the tasks, resource sets, deep space probe state sets, executable actions, deep space probe executable action sets, and deep space probe state transition sets in this process, so as to be more in line with the actual situation of deep space exploration.

[0112] The constructing of the intelligent agent interaction environment includes: a deep space probe state space library, a deep space probe action space library, and a deep space probe state transfer space library.

[0113] The constructing of the intelligent agent interaction environment includes:

[0114] Deep space detector state space library: The deep space detector is regarded as an intelligent agent. The state of the intelligent agent is defined as the state of each subsystem in the deep space detector. That is, the state of the intelligent agent comes from the deep space detector state space library State composed of the states of each subsystem of the deep space detector, where State = S;

[0115] Deep space probe action space library: Each subsystem of the deep space probe has an executable action set. The executable action set of the jth subsystem is A j , as shown in formula (8), the executable action a in this set jh This subsystem can be changed from state s jk Arrivals jk+1 ; When the state is s jk In the case of j A subset of jv , total |A jv | species, as shown in formulas (10) and (11), the deep space detector action space library is defined as Action as shown in formula (12):

[0116]

[0117]

[0118]

[0119] Where: A jv A represents the set of actions that can be performed by the jth subsystem in the vth state. j represents the executable action set of the jth subsystem; since A jv Yes A j A is a subset of jv The elements in come from A j ,|A jv | represents the number of actions that can be performed by the j-th subsystem in the v-th state, represents the jth subsystem in the vth state |A jv | executable actions; A jv represents the set of executable actions in the vth state of the jth subsystem. The deep space probe is divided into N subsystems, among which the Nth subsystem has |S N | kind of state;

[0120] Deep space probe state transfer space library;

[0121] The deep space probe state transfer space library Transfer is defined by triples. The state transfer contained in the library means that the agent will transfer from the current state to another state after performing a certain action. It is represented by the triple in Definition 6 in Section 1. Transfer is expressed by formula (13):

[0122] Transfer = {(s 11 ,a 11 ,s 12 ),...,(s jk ,ajh ,s jk+1 ),...} (13).

[0123] The reinforcement learning method requires a learning environment, and the traditional learning environment published by reinforcement learning cannot adapt to the autonomous mission planning of deep space probes. It shows that the deep space probe definition agent interaction environment constructed by the present invention, including the deep space probe state space library, the deep space probe action space library and the deep space probe state transfer space library, is more suitable for subsequent reinforcement learning.

[0124] The construction of the strategy network includes:

[0125] Combined with the constructed agent interaction environment, a 4-layer MLP strategy network structure is defined, as shown in the attached figure. Figure 2 As shown in the figure, the input layer of the network is the current state of the deep space probe agent, which is represented by two parameters. There are two hidden layers, each with 40 nodes. The output is the probability of the deep space probe agent taking each action under the input layer state.

[0126] Definition of the loss function that integrates time constraints, resource constraints, and timing constraints:

[0127] Time constraint: In order to describe the time required for the executable actions of each subsystem of the deep space probe, the action time consumption matrix T is established, as shown in formulas (14) and (15). This matrix saves the time consumption of the executable actions of each subsystem to meet the time constraints in the planning process;

[0128] M=Max{|A1|,|A2|,...,|A j |,...,|A N |} (14)

[0129]

[0130] Where: |A j | is the number of executable actions of the jth subsystem of the deep space probe; M is the largest number of executable actions corresponding to the N subsystems of the deep space probe; T is the action time consumption matrix, t ju is the time required for the uth executable action of the jth subsystem; for any subsystem, if the number of executable actions is less than M, the corresponding time required is -1;

[0131] Resource constraints: In order to describe the resources required for the executable actions of each subsystem of the deep space probe, the action resource consumption matrix L is established, as shown in formula (16). This matrix saves the resource consumption of the executable actions of each subsystem to meet the resource constraints in the planning process;

[0132]

[0133] Where: r xy The resources consumed by the yth executable action of the xth subsystem;

[0134] Timing constraints: For deep space probe mission planning, there are timing constraints. Define action a jh The set of preceding actions is The previous action set contains the actions that must be executed before the action is executed, as shown in formula (17):

[0135]

[0136] Where: is a subset of A;

[0137] Taking into account the above resource, time and timing constraints, a dynamic reward strategy is adopted to instantly reward Y m The calculation of is shown in formula (19):

[0138]

[0139]

[0140] Where: E represents the action sequence that the deep space exploration agent has planned when performing an Episode, Y m Indicates that the mth action in E is a jh This dynamic reward mechanism is used to integrate the time constraints, resource constraints and timing constraints in deep space exploration mission planning into the reinforcement learning training process.

[0141] Therefore, the expected reward G of the currently selected executable action is m The definition is as follows:

[0142]

[0143] Where: γ is the discount factor. When humans make decisions, their preference for immediate rewards will show a cognitive pattern close to exponential growth. The purpose of introducing the discount factor is to punish the uncertainty in future rewards. 0<γ≤1, anm represents the total number of executable actions in the current Episode that needs to be calculated;

[0144] According to the policy gradient descent method, a loss function as shown in formula (21) is constructed for updating the policy network;

[0145]

[0146] Where: a gh is the currently selected action, πθ (a jh |s jk ) is the action prediction strategy.

[0147] The reinforcement learning algorithm updates the policy network using the policy gradient method by constructing a loss function. Unlike general reinforcement learning problems, there are multiple constraints in deep space exploration mission planning, including time constraints, resource constraints, and timing constraints. These three constraints are considered when intelligently completing each episode (i.e., completing a round of planning), thus meeting the multi-constraint requirements of autonomous mission planning for deep space probes.

[0148] The training strategy network;

[0149] Define the terminal state s of the mission from the deep space probe state space library in S2 ow ; Among them, s ow represents the wth state of the oth subsystem;

[0150] Randomly generate an initial state s from the deep space probe state space library in S2 pq Then, the policy network constructed by S3 is used to plan a set of Episodes. The set of Episodes is used as input to update the policy network according to the formula (21) defined by S4 until the policy network converges and the trained policy network is obtained. pq represents the qth state of the pth subsystem;

[0151] The step of using the trained strategy network to perform task planning includes:

[0152] According to the scientific task F to be performed, the initial state and the terminal state are input, and the trained strategy network obtained in S4 is used for planning until the agent reaches the terminal state and ends the planning; the planning results are drawn into a task Gantt chart of each subsystem of the deep space probe.

[0153] According to another aspect of the present invention, there is also provided a deep space probe autonomous mission planning system based on dynamic reward reinforcement learning, comprising:

[0154] Definition unit, used to define the reinforcement learning parameters for deep space probe mission planning;

[0155] The first construction unit is used to construct an intelligent agent interaction environment;

[0156] The second building unit is used to build a policy network;

[0157] Training unit, used to train the policy network;

[0158] The execution unit is used to execute task planning using the trained policy network.

[0159] According to another aspect of the present invention, a processor is provided, which is used to run a program, wherein when the program is running, any one of the above-mentioned dynamic reward-based reinforcement learning deep space probe autonomous mission planning methods is executed.

[0160] According to another aspect of the present invention, a computer-readable storage medium is also provided, which includes a stored program, and when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned dynamic reward-based reinforcement learning deep space probe autonomous mission planning methods.

[0161] Further, the present invention provides the following implementation data:

[0162] The names of the deep space detector subsystems corresponding to the specific instances, the state sets corresponding to each deep space detector subsystem, and the action sets corresponding to each state are as follows:

[0163]

[0164] The planning results are drawn into a task Gantt chart of each subsystem of the deep space probe, such as Figure 3 At the same time, a comparison chart of the success rate between the method of the present invention and the static reward strategy gradient algorithm is given, as shown in Figure 4 ,Comparative experimental results show that the dynamic reward strategy proposed in this ,paper has a significant improvement in the planning success rate compared with the ,general static reward strategy gradient algorithm.

[0165] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A deep space probe autonomous mission planning method based on dynamic reward reinforcement learning, characterized by: include: S1. Define the reinforcement learning parameters for deep space probe mission planning; S2, build an intelligent agent interaction environment; S3, build a strategy network; S4, training strategy network; S5. Use the trained strategy network to perform task planning; The definition of the reinforcement learning parameters for deep space probe mission planning includes: Definition 1. Task. The task is expressed by formula (1): F={I F ,N F ,E FL ,E FP ,t FES ,t FLE } (1) Where: F represents the scientific mission that can be performed by the deep space probe; I F Represents the number of task F; N F Represents the name of task F; E FL represents the task value of task F; E FP represents the priority of task F; t FES represents the earliest start time of task F; t FLE Represents the latest start time of task F; Definition 2: Resource set: A resource set is a set of resources that a deep space probe will use when performing an action. It is represented by R, as shown in formula (2): R＝{r1,r2,...,r i ,...,r |R|} (2) Where: r i represents the i-th resource; |R| represents the number of elements in the set R, that is, the number of resource types; Represents the number of the i-th resource; Represents the name of the i-th resource; represents the remaining amount of the i-th resource; Represents the type of the i-th resource, that is, whether the resource is a renewable resource or a non-renewable resource; Definition 3: Deep space probe state set. Let S represent the state set of a deep space probe. Assume that the probe has N subsystems. Let S represent the state set of the jth subsystem. j Indicates that its status is |S j | Species; S j It is expressed as formula (4), and S can be expressed as formula (5), which is: S j ={s j1 ,s j2 ,...,s jk ,...,S j|Sj| }(4) S=S1∪S2∪...S j ...∪S N (5) Where: s jk represents the kth state of the jth subsystem; |S j | represents the set S j The number of elements of S, that is, the number of states of the j-th subsystem; j represents the state set of the jth subsystem; Definition 4: executable action; executable action is the action that deep space probe can perform; use a jh It represents the hth executable action of the jth subsystem, as shown in formula (6): Where: Represents action a jh Serial number; Substitute action a jh Duration of the movement; Represents action a jh The resource set consumed; R represents the resource set of the deep space probe, is a subset of R; Definition 5: The executable action set of deep space probes; A is used to represent the executable action set of deep space probes; the executable action set of the jth subsystem is represented by A j Indicates that its types are |A j | Species; A j It is expressed as formula (8), and A is expressed as formula (9): A=A1∪A2∪...A j ...∪A N (9) Where: |A j | represents the number of elements in the set, that is, the number of executable actions of the jth subsystem; A j represents the set of executable actions of the jth subsystem; Definition 6, deep space probe state transition set; The triple is constructed as G = (S, A, S); where G represents the state transition set of deep space probe states and actions, S represents the deep space probe state set, which has |S| states; A represents the deep space probe executable action set, which has |A| executable actions; the basic form of the triple is (s jk ,a jh ,s jk+1 ), where s jk represents the kth state of the jth subsystem, s jk+1 Indicates that the jth subsystem performs action a jh The k+1th state reached after s; the first element s in the triple G jk From S, the second element a jh From A, the third element s jk+1 From S; The constructing of the intelligent agent interaction environment includes: a deep space detector state space library, a deep space detector action space library, and a deep space detector state transfer space library; The constructing of the intelligent agent interaction environment includes: Deep space detector state space library: The deep space detector is regarded as an intelligent agent. The state of the intelligent agent is defined as the state of each subsystem in the deep space detector. That is, the state of the intelligent agent comes from the deep space detector state space library State composed of the states of each subsystem of the deep space detector, where State = S; Deep space probe action space library: Each subsystem of the deep space probe has an executable action set. The executable action set of the jth subsystem is A j , as shown in formula (8), the executable action a in this set jh Can make the subsystem from state s jk Arrivals jk+1 ; When the state is s jk In the case of j A subset of jv , total |A jv | species, as shown in formulas (10) and (11), the deep space detector action space library is defined as Action as shown in formula (12): Where: A jv A represents the set of actions that can be performed by the jth subsystem in the vth state. j represents the executable action set of the jth subsystem; since A jv Yes A j A is a subset of jv The elements in come from A j ,|A jv | represents the number of actions that can be performed by the j-th subsystem in the v-th state, represents the jth subsystem in the vth state |A jv | executable actions; A jv represents the set of executable actions in the vth state of the jth subsystem. The deep space probe is divided into N subsystems, among which the Nth subsystem has |S N | kind of state; Deep space probe state transfer space library; The deep space probe state transfer space library Transfer is defined by triples. The state transfer contained in the library means that the agent will transfer from the current state to another state after performing a certain action. Transfer is expressed by formula (13): Transfer={(s 11 ,a 11 ,s 12 ),...,(s jk ,a jh ,s jk+1 ),...}(13); The construction of the strategy network includes: Combined with the constructed agent interaction environment, a 4-layer MLP strategy network structure is defined; the input layer of the network is the current state of the deep space probe agent, represented by 2 parameters, the hidden layer has 2 layers, each with 40 nodes, and the output is the probability of the deep space probe agent taking each action under the input layer state; Definition of the loss function that integrates time constraints, resource constraints, and timing constraints: Time constraint: Establish the action time consumption matrix T, as shown in formulas (14) and (15). This matrix saves the time consumption of executable actions of each subsystem to meet the time constraints in the planning process; M=Max{|A1|,|A2|,...,|A j |,...,|A N |}(14) Where: |A j | is the number of executable actions of the jth subsystem of the deep space probe; M is the largest number of executable actions corresponding to the N subsystems of the deep space probe; T is the action time consumption matrix, t ju is the time required for the uth executable action of the jth subsystem; for any subsystem, if the number of executable actions is less than M, the corresponding time required is -1; Resource constraints: Establish the action resource consumption matrix L, as shown in formula (16). This matrix saves the resource consumption of the executable actions of each subsystem to meet the resource constraints in the planning process; Where: r xy The resources consumed by the yth executable action of the xth subsystem; Timing constraints: For deep space probe mission planning, there are timing constraints. Define action a jh The set of preceding actions is The previous action set contains the actions that must be executed before the action is executed, as shown in formula (17): Where: is a subset of A; Taking into account the above resource, time and timing constraints, a dynamic reward strategy is adopted to instantly reward Y m The calculation of is shown in formula (19): Where: E represents the action sequence that the deep space exploration agent has planned when performing an Episode, Y m Indicates that the mth action in E is a jh The instant reward at time t jh 、r jh They represent the time consumed and the resources consumed by the h-th executable action of the j-th subsystem respectively; Therefore, the expected reward G of the currently selected executable action is m The definition is as follows: Where: γ is the discount factor, 0<γ≤1, anm represents the total number of executable actions in the current Episode that needs to be calculated; According to the policy gradient descent method, a loss function as shown in formula (21) is constructed; Where: a gh is the currently selected action, π θ (a jh |s jk ) is the action prediction strategy; The training strategy network comprises: Define the terminal state s of the task from the deep space probe state space library in the agent interaction environment ow ; Among them, s ow represents the wth state of the oth subsystem; Randomly generate an initial state s from the deep space probe state space library in the agent interaction environment pq Then, the constructed policy network is used to plan a set of episodes, and a set of episodes is used as input to update the policy network according to the defined formula (21); until the policy network converges, a trained policy network is obtained; where s pq Represents the qth state of the pth subsystem.

2. The method for autonomous mission planning of deep space probes based on dynamic reward reinforcement learning according to claim 1, characterized in that: The step of using the trained strategy network to perform task planning includes: According to the scientific task F to be performed, the initial state and the terminal state are input, and the trained strategy network is used for planning until the agent reaches the terminal state and the planning is ended; the planning results are drawn into a task Gantt chart of each subsystem of the deep space probe.

3. A deep space probe autonomous mission planning system based on dynamic reward reinforcement learning for executing the method of claim 1, characterized in that: include: Definition unit, used to define the reinforcement learning parameters for deep space probe mission planning; The first construction unit is used to construct an intelligent agent interaction environment; The second building unit is used to build a policy network; Training unit, used to train the policy network; The execution unit is used to execute task planning using the trained policy network.

4. A processor, characterized in that: The processor is used to run a program, wherein when the program is run, the method for autonomous mission planning of deep space probes based on dynamic reward reinforcement learning as described in any one of claims 1 to 2 is executed.

5. A computer-readable storage medium, the computer-readable storage medium comprising a stored program, characterized in that: When the program is running, the device where the computer-readable storage medium is located is controlled to execute the deep space probe autonomous mission planning method based on dynamic reward reinforcement learning as described in any one of claims 1 to 2.

Images