Observation constellation optimization method based on deep reinforcement learning

Through the improved TD3 algorithm and dynamic strategy noise mechanism, the problem of unstable observation constellations in traditional TD3 technology is solved, the learning and training efficiency and stability are improved, and the observation performance and life of constellations are improved.

CN119004972AActive Publication Date: 2024-11-22SICHUAN UNIV

Patent Information

Application Number
CN202411072310.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-22
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

Traditional TD3 technology causes unstable observation constellations, low learning and training efficiency, and unstable optimization model when there is a large change in noise.

Method used

Adopting the observation constellation optimization method based on deep reinforcement learning, the improved TD3 algorithm introduces priority experience playback and dynamic strategy noise mechanism, establishes a state space model, action space model and reward function, and uses the Actor policy network and the Critic value network for optimization strategy training. , improve the training efficiency and stability of the model.

Benefits of technology

It improves the training efficiency and stability of the zodiac optimization design decision model, improves the zodiac observation performance to the ground, reduces operating costs, and extends service life.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004972A_ABST
    Figure CN119004972A_ABST
Patent Text Reader

Abstract

The invention discloses an observation constellation optimization method based on deep reinforcement learning, and relates to the technical field of satellite intelligence, and the method comprises the following steps: determining the constellation configuration of a to-be-observed constellation, and obtaining the satellite orbit parameters of the to-be-observed constellation; constructing a constellation optimization strategy by using a reinforcement learning algorithm based on the satellite orbit parameters; establishing a state space model and an action space model and designing a reward function; based on the optimal strategy function, the state space model, the action space model and the reward function are processed through an improved TD3 algorithm, and a final optimization strategy is obtained; and executing the final optimization strategy through the to-be-observed constellation to complete optimization of the observation constellation. According to the method, the defects of low learning training efficiency and instability of an optimization model of the traditional TD3 are overcome, the training efficiency and stability of a constellation optimization design decision model are improved by introducing priority experience playback and dynamic strategy noise, and the earth observation performance of the constellation is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of satellite intelligence technology, and in particular to an observation constellation optimization method based on deep reinforcement learning. Background Art

[0002] An observation constellation is a type of satellite constellation that observes targets on the ground, sea, and air. The optimization of an observation constellation refers to a series of technical activities that aim to improve observation efficiency, reduce costs, and expand application areas through comprehensive and integrated analysis and optimization of the satellite constellation system. In constellation design, the application of optimization technology can greatly improve the observation performance of the constellation, and is one of the important technical means to maintain high observation performance and reduce operating costs for satellite constellations. Therefore, it is of great significance to study efficient optimization methods for observation constellations.

[0003] One of the challenges of optimizing the observation constellation is that the observation path needs to be adjusted quickly to meet the changing observation mission requirements and complex environmental constraints. Reinforcement learning, as an intelligent control method, improves the satellite's autonomous decision-making ability by learning the optimal strategy through interaction with the environment. In emergencies, satellites can quickly adjust plans, give priority to observing key areas, and provide real-time information for decision-making. By optimizing the constellation, not only is its adaptability enhanced, but energy consumption is also reduced and its service life is extended. Research on the application of reinforcement learning in the optimization of observation constellations is the key to promoting the intelligence of space technology and improving observation efficiency, and it has important practical significance for national security and environmental protection.

[0004] At present, the experience replay in the traditional TD3 technology is that the intelligent model learns through uniformly distributed sampling, but its high-value samples are diluted by ordinary samples, making the model update ineffective and inefficient. Moreover, the noise added to the action output is Gaussian noise with a stable standard deviation. When the noise changes greatly before and after, it will cause the intelligent model to be unstable. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides an observation constellation optimization method based on deep reinforcement learning, which solves the problem that the traditional TD3 technology may cause unstable observation constellations when the noise changes greatly before and after.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] A method for optimizing an observation constellation based on deep reinforcement learning is provided, which comprises the following steps:

[0008] S1. Determine the constellation configuration of the constellation to be observed and obtain its satellite orbit parameters;

[0009] S2. Based on satellite orbit parameters, a constellation optimization strategy is constructed using reinforcement learning algorithm;

[0010] S3, establish state space model, action space model and design reward function;

[0011] S4. Based on the optimal strategy function, the improved TD3 algorithm is used to process the state space model, action space model and reward function to obtain the final optimization strategy;

[0012] S5. Execute the final optimization strategy through the constellation to be observed to complete the optimization of the observed constellation.

[0013] Furthermore, step S1 includes the following steps:

[0014] S1-1. Determine the constellation configuration of the constellation to be observed; wherein the constellation configuration includes a universal constellation configuration and a Walker constellation configuration;

[0015] S1-2, judging whether the constellation configuration of the constellation to be observed is a universal constellation configuration; if so, proceeding to step S1-3; otherwise, judging that the constellation configuration of the constellation to be observed is a Walker constellation configuration and proceeding to step S1-4;

[0016] S1-3, taking the six first orbital parameters corresponding to each satellite of the universal constellation configuration as satellite orbital parameters, and proceeding to step S2; wherein the six first orbital parameters are respectively the satellite orbit semi-major axis, orbital eccentricity, orbital plane inclination, perigee argument, ascending node right ascension, and true anomaly;

[0017] S1-4. Determine the total number of satellites N in the Walker constellation configuration s As well as the orbital elements, phase factors, orbital plane numbers, orbital semi-major axes, and orbital inclinations of each satellite;

[0018] S1-5. Use the first satellite of the Walker constellation configuration as the seed satellite and obtain the corresponding ascending node right ascension and true anomaly at the initial moment;

[0019] S1-6, according to the formula:

[0020]

[0021] Get the ascending node right ascension raan of the remaining satellites m,n and true anomaly Among them, raan 1,1 represents the right ascension of the ascending node of the seed satellite, P represents the orbital plane number, π represents pi, m represents that the satellite is located in the mth orbital plane, n represents the nth satellite on the orbital plane, represents the true anomaly of the seed satellite at the initial time;

[0022] S1-7. The total number of satellites N in the Walker constellation configuration s , phase factor, orbital plane number, orbital semi-major axis, orbital plane inclination, right ascension of ascending nodes of all satellites, and true anomaly at the initial moment are taken as satellite orbit parameters, and step S2 is entered.

[0023] Further, step S2 includes the following steps:

[0024] S2-1. Take the constellation to be observed as an intelligent agent and determine the optimization task of the constellation to be observed. Construct the corresponding MDP model based on the Markov decision process. The corresponding formula is:

[0025] MDP=(S,A,P,R,γ)

[0026] Among them, S represents the state set, including satellite orbit parameters, A represents the action set, P represents the state transfer function, R represents the immediate reward and punishment, and γ represents the discount factor;

[0027] S2-2. Calculate the cumulative rewards corresponding to all decisions executed by the MDP model;

[0028] S2-3. Calculate the optimal state value function and the optimal state action value function Among them, a represents the action in the action set A;

[0029] S2-4. Based on the optimal state value function and the optimal state action value function Determine constellation optimization strategy.

[0030] Furthermore, the calculation formula for the cumulative reward is:

[0031]

[0032] Among them, G t , G t+1 denote the cumulative rewards corresponding to time t and time t+1, ∑(·) denotes the summation function, and R t+1 , R t+2 , R t+3 , R t+k+1 They represent the instant rewards and punishments corresponding to time t, time t+1, time t+3, and time t+k+1 respectively;

[0033] Optimal state value function The calculation formula is:

[0034]

[0035] Among them, max(·) represents the maximum value function, s' and s represent the state, represents the optimal state value function corresponding to state s', P(s'|s,a) represents the probability of the agent performing action a to transfer to state s' when in state s, and R(s,a) represents the immediate reward or punishment for the agent performing action a when in state s;

[0036] The calculation formula of the optimal state action value function is:

[0037]

[0038] in, Represents the optimal state-action-value function for the agent to perform action a' when in state s'.

[0039] Further, step S3 includes the following steps:

[0040] S3-1, judging whether the constellation configuration of the constellation to be observed is a universal constellation configuration; if so, proceeding to step S3-2; otherwise, merging into step S3-4;

[0041] S3-2. Change the semi-major axis of the satellite orbit of the universal constellation configuration to 0 Set it to 8000km, set the satellite orbit to a circular orbit, set the eccentricity to 0, set the perigee argument to 0, and take the orbital inclination, ascending node right ascension, true anomaly and position parameters as the state, that is, the state space model. The corresponding expression is:

[0042]

[0043] in, Respectively represent the orbital inclination of the first satellite S1 and the second satellite S2 of the constellation to be observed Respectively represent the right ascension of the ascending node of the first satellite S1 and the second satellite S2 of the constellation to be observed They represent the first satellite S1, the second satellite S2, and the Nth satellite S1 of the constellation to be observed. S Satellites True anomaly s lon 、s lat 、s alt They respectively represent the longitude, latitude and altitude of the constellation to be observed in the GPS84 coordinate system;

[0044] S3-3, establish an action space model of the universal constellation configuration, that is, a state space model, and proceed to step S3-6; wherein the expression corresponding to the state space model a of the universal constellation configuration is:

[0045]

[0046] in, They represent the inclination of the orbital plane of the first satellite S1 and the second satellite S2 of the constellation to be observed. Actions, They represent the right ascension of the first satellite S1 and the second satellite S2 of the constellation to be observed respectively. Actions, They represent the first satellite S1, the second satellite S2, and the Nth satellite S3 of the observed constellation. S Satellites At true anomaly Actions;

[0047] S3-4, the semi-major axis of the satellite orbit of the Walker constellation configuration is a constant, and the number of orbital planes, phase factor, orbital plane inclination, ascending node right ascension, true anomaly and position parameters are taken as states, that is, the state space model, and the corresponding expression is:

[0048]

[0049] Among them, s P 、s F 、s inc , They respectively represent the status of the Walker constellation orbital plane number, phase factor, orbital plane inclination, ascending node right ascension, and true anomaly;

[0050] S3-5. Establish an action space model of the Walker constellation configuration, that is, a state space model, and proceed to step S3-6; wherein the expression corresponding to the state space model a of the Walker constellation configuration is:

[0051]

[0052] Among them, a P 、a F 、a inc , They represent the actions on the Walker constellation orbital plane number, phase factor, orbital plane inclination, ascending node right ascension, and true anomaly respectively;

[0053] S3-6, according to the formula:

[0054] R=α1R1+α2R2

[0055] Construct a reward function R; where R1 and R2 represent the target observation performance reward and optimization number reward respectively, α1 and α2 represent coefficients respectively, and α1+α2=1.

[0056] Furthermore, the improved TD3 algorithm adopts a dynamic strategy noise mechanism and includes an Actor strategy network and a Critic value network; wherein the Actor strategy network includes a target strategy network and an online strategy network; the Critic value network includes a target value network and an online value network; the target value network includes a target1 value network and a target2 value network; and the online value network includes an online1 value network and an online2 value network.

[0057] Furthermore, the training process of the improved TD3 algorithm in step S4 includes the following steps:

[0058] S4-1, set the total number of training rounds and the total number of steps per round;

[0059] S4-2, initialize the environment, that is, randomly initialize the training satellite orbit parameters within the preset value range, and obtain the current state s t ;

[0060] S4-3, based on the current state s t , select action a through the online policy network t ;

[0061] S4-4. Execute the action a t , and synchronize the simulation scene based on STK to obtain the corresponding reward R t and the next state s t+1 , and update the optimal strategy of the constellation;

[0062] S4-5, the current state s t , next state s t+1 、Action a t and reward R t As sample data and stored in the experience pool RB;

[0063] S4-6. Randomly sample N sample data from the experience pool RB and select actions through the target strategy network Calculate the loss function L;

[0064] S4-6. Update the parameters of the target value network and the online value network according to the loss function L;

[0065] S4-7. Calculate the policy gradient and update the parameters of the online policy network;

[0066] S4-8, update the target strategy network and the target value network of step S4-6 by soft update;

[0067] S4-9, determine whether the current number of steps reaches the total number of steps in the round; if so, the current round ends and enters step S4-10; otherwise, the current number of steps is increased by 1 and returns to step S4-3; wherein the initial value of the number of steps is 0;

[0068] S4-10. Repeat steps S4-1 to S4-9 until the total number of training rounds is reached.

[0069] Furthermore, the formula of step S4-3 is:

[0070] a t =clip(π(s t |θ μ )+ε,a low , a high ),ε~N(0,σ epi 2 )

[0071] Among them, π(s t |θ μ ) indicates that the online policy network π is in the current network parameter θ μ According to the current state s t The output action, ε represents the policy noise of the current step, a low 、a high Indicates action a t The value range of Ν(0,σ epi 2 ) means the mean is 0 and the standard deviation is σ epi Gaussian distribution, clip(π(s t |θ μ )+ε,a low ,a high ) indicates that the current network parameter is θ μ And in the current state s t The online strategy network π under

[0072] The calculation formula of the loss function L of S4-6 is:

[0073]

[0074] Among them, min(·) represents the minimum function, represents the network parameters of the target value network, Represents the network parameters of the online value network, s i+1 、s i They represent the states of the i+1th sample data and the ith sample data, respectively. q(·) represents the online value network. a i represents the action of the i-th sample data, and n represents the sequence number of the online value network;

[0075] Policy gradient of steps S4-7 The calculation formula is:

[0076]

[0077] in, Indicates that the online value network q has the current network parameters According to the current state s i 、Action a i The value gradient of the output, Indicates that in the online policy network π, the current network parameter is θ π According to the current state s i The output action gradient;

[0078] Furthermore, the experience pool RB adopts a priority experience replay mechanism.

[0079] The beneficial effects of the present invention are as follows: the present method improves the shortcomings of low learning and training efficiency and instability of the optimization model of the traditional TD3, improves the training efficiency and stability of the constellation optimization design decision model by introducing priority experience replay and dynamic strategy noise, and improves the constellation's earth observation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a method flow chart;

[0081] Figure 2 This is the flow chart of the improved TD3 algorithm;

[0082] Figure 3 The reward curve diagram for the training process of Walker constellation configuration;

[0083] Figure 4 A reward curve diagram for the training process of the general constellation configuration;

[0084] Figure 5 This is a comparison chart of optimization results for different constellation configurations;

[0085] Figure 6 This is the result diagram of the 20 particle group parameters;

[0086] Figure 7 This is the result diagram of parameter optimization of 100 particle swarms;

[0087] Figure 8 Optimize observation performance curves for constellations;

[0088] Fig. 9The constellation optimization observation efficiency curve diagram based on the particle swarm algorithm; (a) is the constellation optimization observation efficiency curve diagram when the particle swarm scale is 20, and (b) is the constellation optimization observation efficiency curve diagram when the particle swarm scale is 100;

[0089] Fig.10 It is a comparison histogram based on particle swarm algorithm and this method; among them, (a) is the observation performance comparison histogram based on particle swarm algorithm and this method, and (b) is the time consumption comparison histogram based on particle swarm algorithm and this method. DETAILED DESCRIPTION

[0090] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0091] like Figure 1 As shown, a method for optimizing observation constellations based on deep reinforcement learning includes the following steps:

[0092] S1. Determine the constellation configuration of the constellation to be observed and obtain its satellite orbit parameters;

[0093] Step S1 includes the following steps:

[0094] S1-1. Determine the constellation configuration of the constellation to be observed; wherein the constellation configuration includes a universal constellation configuration and a Walker constellation configuration;

[0095] S1-2, judging whether the constellation configuration of the constellation to be observed is a universal constellation configuration; if so, proceeding to step S1-3; otherwise, judging that the constellation configuration of the constellation to be observed is a Walker constellation configuration and proceeding to step S1-4;

[0096] S1-3, taking the six first orbital parameters corresponding to each satellite of the universal constellation configuration as satellite orbital parameters, and proceeding to step S2; wherein the six first orbital parameters are respectively the satellite orbit semi-major axis, orbital eccentricity, orbital plane inclination, perigee argument, ascending node right ascension, and true anomaly;

[0097] S1-4. Determine the total number of satellites Ns in the Walker constellation configuration 以 And the orbital elements, phase factors, orbital plane numbers, orbital semi-major axes, and orbital inclinations of each satellite;

[0098] S1-5. Use the first satellite of the Walker constellation configuration as the seed satellite and obtain the corresponding ascending node right ascension and true anomaly at the initial moment;

[0099] S1-6, according to the formula:

[0100]

[0101] Get the ascending node right ascension raan of the remaining satellites m,n and true anomaly Among them, raan 1,1 represents the right ascension of the ascending node of the seed satellite, P represents the orbital plane number, π represents pi, m represents that the satellite is located in the mth orbital plane, n represents the nth satellite on the orbital plane, represents the true anomaly of the seed satellite at the initial time;

[0102] S1-7. The total number of satellites N in the Walker constellation configuration s , phase factor, orbital plane number, orbital semi-major axis, orbital plane inclination, right ascension of ascending nodes of all satellites, and true anomaly at the initial moment are taken as satellite orbit parameters, and step S2 is entered.

[0103] Among them, the orbital inclination, the right ascension of the ascending node, and the true anomaly are all periodic.

[0104] S2. Based on satellite orbit parameters, a constellation optimization strategy is constructed using reinforcement learning algorithm;

[0105] Step S2 includes the following steps:

[0106] S2-1. Take the constellation to be observed as an intelligent agent and determine the optimization task of the constellation to be observed. Construct the corresponding MDP model based on the Markov decision process. The corresponding formula is:

[0107] MDP=(S,A,P,R,γ)

[0108] Among them, S represents the state set, including satellite orbit parameters, and any element s∈S represents a state that may exist in the environment; A represents the action set, and any element a∈A represents an action that the agent can choose; P represents the state transition function, R represents immediate rewards and punishments, and γ is a discount factor, that is, a weight, which represents the tendency towards future lasting rewards and current immediate rewards;

[0109] S2-2. Calculate the cumulative rewards corresponding to all decisions executed by the MDP model;

[0110] The calculation formula for cumulative rewards is:

[0111]

[0112] Among them, G t , G t+1denote the cumulative rewards corresponding to time t and time t+1, ∑(·) denotes the summation function, R t+1 , R t+2 , R t+3 , R t+k+1 They represent the instant rewards and punishments corresponding to time t, time t+1, time t+3, and time t+k+1 respectively;

[0113] S2-3. Calculate the optimal state value function and the optimal state action value function Among them, a represents the action in the action set A;

[0114] Optimal state value function and the optimal state-action-value function is given by the state-value function v π (s) and state-action-value function q π (s,a) and Bellman optimality principle.

[0115] State value function v π (s) and state-action-value function q π (s,a) is used to estimate the cumulative reward and evaluate the quality of each state and state-action combination. The corresponding formula is:

[0116] v π (s)=E[G t |s t =s,π]

[0117] q π (s,a)=E[G t |s t =s,a t =a,π]

[0118] Among them, π represents the strategy, s represents the state, E[G t |s t =s,π] represents the expected cumulative reward of the agent in state s when executing strategy π, E[G t |s t =s,a t =a,π] represents the expected cumulative reward of the agent when state s performs action a when executing strategy π;

[0119] According to Bellman's optimality principle, the state value function v π (s) and state-action-value function q π (s,a) can be rewritten using the Bellman equation to obtain:

[0120]

[0121]

[0122] Thus, the optimal state value function The calculation formula is:

[0123]

[0124] Among them, max(·) represents the maximum value function, s' and s represent the state, represents the optimal state value function corresponding to state s', P(s'|s,a) represents the probability of the agent performing action a to transfer to state s' when in state s, and R(s,a) represents the immediate reward or punishment for the agent performing action a when in state s;

[0125] The calculation formula of the optimal state action value function is:

[0126]

[0127] in, Represents the optimal state-action-value function for the agent to perform action a in state s.

[0128] S2-4. Based on the optimal state value function and the optimal state action value function Determine constellation optimization strategy.

[0129] Among them, the constellation optimization strategy must meet the following conditions:

[0130] π(a t |s t )=P[a t |s t ]

[0131] Among them, π(a t |s t ) indicates that the agent is in state s t Take action a according to policy π t The probability, P[a t |s t ] represents the probability that the agent performs action a' in state s'.

[0132] S3, establish state space model, action space model and design reward function;

[0133] Step S3 includes the following steps:

[0134] S3-1, judging whether the constellation configuration of the constellation to be observed is a universal constellation configuration; if so, proceeding to step S3-2; otherwise, proceeding to step S3-4;

[0135] S3-2. Change the semi-major axis of the satellite orbit of the universal constellation configuration to 0Set it to 8000km, set the satellite orbit to a circular orbit, set the eccentricity to 0, set the perigee argument to 0, and take the orbital inclination, ascending node right ascension, true anomaly and position parameters as the state, that is, the state space model. The corresponding expression is:

[0136]

[0137] in, Respectively represent the orbital inclination of the first satellite S1 and the second satellite S2 of the constellation to be observed Respectively represent the right ascension of the ascending node of the first satellite S1 and the second satellite S2 of the constellation to be observed They represent the first satellite S1, the second satellite S2, and the Nth satellite S1 of the constellation to be observed. S Satellites True anomaly s lon 、s lat 、s alt They respectively represent the longitude, latitude and altitude of the constellation to be observed in the GPS84 coordinate system;

[0138] S3-3, establish an action space model of the universal constellation configuration, that is, a state space model, and proceed to step S3-6; wherein the expression corresponding to the state space model a of the universal constellation configuration is:

[0139]

[0140] in, They represent the inclination of the orbital plane of the first satellite S1 and the second satellite S2 of the constellation to be observed. Actions, They represent the right ascension of the first satellite S1 and the second satellite S2 of the constellation to be observed respectively. Actions, They represent the first satellite S1, the second satellite S2, and the Nth satellite S3 of the observed constellation. S Satellites At true anomaly Actions;

[0141] S3-4, the semi-major axis of the satellite orbit of the Walker constellation configuration is a constant, and the number of orbital planes, phase factor, orbital plane inclination, ascending node right ascension, true anomaly and position parameters are taken as states, that is, the state space model, and the corresponding expression is:

[0142]

[0143] Among them, s P 、s F 、s inc , They respectively represent the status of the Walker constellation orbital plane number, phase factor, orbital plane inclination, ascending node right ascension, and true anomaly;

[0144] S3-5. Establish an action space model of the Walker constellation configuration, that is, a state space model, and proceed to step S3-6; wherein the expression corresponding to the state space model a of the Walker constellation configuration is:

[0145]

[0146] Among them, a P 、a F 、a inc , They represent the actions on the Walker constellation orbital plane number, phase factor, orbital plane inclination, ascending node right ascension, and true anomaly respectively;

[0147] Among them, whether it is the Walker constellation configuration or the universal constellation configuration, the corresponding state value needs to be normalized and mapped to the range of 0 to 1; the corresponding action output value needs to be normalized and mapped to the range of -1 to 1. For example, the actual value range of the inclination angle is 0 to 180°, and the initial state value is The state value after the state transfer Relationships and action values Satisfy the following formula:

[0148]

[0149] S3-6, according to the formula:

[0150] R=α1R1+α2R2

[0151] Construct a reward function R; where R1 and R2 represent the target observation performance reward and optimization number reward respectively, α1 and α2 represent coefficients respectively, and α1+α2=1.

[0152] Among them, the relationship between the constellation-to-target observation performance reward R1 and the constellation-to-target observation performance score m obtained by the constellation-to-target observation performance evaluation method based on multi-layer fuzzy comprehensive evaluation is as follows:

[0153]

[0154] The relationship between the optimization reward R2 and the change n of all parameters to be optimized before and after the current round of optimization is:

[0155] Among them, the process of the constellation target observation performance evaluation method based on multi-layer fuzzy comprehensive evaluation is as follows:

[0156] First, construct an evaluation index system. The overall target layer is "constellation observation performance of the target", the intermediate layer is "observation accuracy", "observation time resolution" and "observation stability", and the index layer is the specific observation data or parameters. Then, determine the factor set and comment set. The factor set is a general set composed of various factors that affect the evaluation target. V = {v1, v2, ... v m}, where u1 represents the observation accuracy, u2 represents the observation time resolution, etc. The comment set is a set of various possible results that the evaluator may make on the evaluation target, represented by V = {v1, v2, ... v m}, where v1 represents "excellent", v2 represents "good", etc. Secondly, the weights of each factor are specified, and a fuzzy evaluation matrix is ​​constructed. According to the evaluation levels in the comment set, a fuzzy evaluation matrix is ​​constructed. Each element in the matrix represents the degree of membership of a factor to a certain evaluation level. Finally, multi-layer fuzzy comprehensive evaluation is performed and the results are analyzed. First, starting from the bottom layer, a single-factor fuzzy evaluation is performed on each sub-factor to obtain their respective fuzzy evaluation matrices. Then, a synthesis operation is performed using their respective weights and fuzzy evaluation matrices to obtain the fuzzy evaluation vector of the previous level. This process is carried out layer by layer until the overall target layer is reached, and the final fuzzy evaluation vector is obtained. Quantitative analysis of the final fuzzy evaluation vector can be used to convert it into a comprehensive score or determine the optimal evaluation level, thereby determining the final evaluation level of the evaluation object.

[0157] S4. Based on the optimal strategy function, the improved TD3 algorithm is used to process the state space model, action space model and reward function to obtain the final optimization strategy;

[0158] The improved TD3 algorithm adopts a dynamic strategy noise mechanism and includes an Actor strategy network and a Critic value network; wherein, the Actor strategy network includes a target strategy network and an online strategy network; the Critic value network includes a target value network and an online value network; the target value network includes a target1 value network and a target2 value network; and the online value network includes an online1 value network and an online2 value network.

[0159] Because traditional TD3 adds Gaussian noise with a stable standard deviation when outputting actions, the agent model will become unstable when encountering large changes in noise before and after. Therefore, a dynamic strategy noise mechanism is adopted, and its formula is:

[0160]

[0161] Among them, σ a represents the noise standard deviation of the output action, σ max, σ min They represent the maximum noise standard deviation of the output action and the minimum noise standard deviation of the output action, respectively. all_epi represents the total training rounds, t represents the current number of training rounds, and n early_epi 、n later_epi They represent the number of training rounds corresponding to the boundaries between the early and middle stages and between the middle and late stages of the agent model training phase, n epi Indicates the current training round number. Its purpose is to give the agent the maximum action noise in the early stage of training to enrich the state action sample experience pool used for training; in the middle stage of training, as the number of training times increases, the action noise gradually decreases; in the late stage of training, the agent is given the minimum action noise, so that the agent model gradually stabilizes.

[0162] like Figure 2 As shown, the training process of the improved TD3 algorithm in step S4 includes the following steps:

[0163] S4-1, set the total number of training rounds and the total number of steps per round;

[0164] S4-2, initialize the environment, that is, randomly initialize the training satellite orbit parameters within the preset value range, and obtain the current state s t ;

[0165] S4-3, based on the current state s t , select action a through the online policy network t ;

[0166] The formula of step S4-3 is:

[0167] a t =clip(π(s t |θ μ )+ε,a low , a high ),ε~N(0,σ epi 2 )

[0168] Among them, π(s t |θ μ ) represents, ε represents the strategy noise of the current step, a low 、a high Indicates action a t The value range of Ν(0,σ epi 2 ) means the mean is 0 and the standard deviation is σ epi Gaussian distribution, clip(π(s t |θ μ )+ε,a low ,a high) indicates that the current network parameter is θ μ And in the current state s t The online strategy network π under

[0169] S4-4. Execute the action a t , and synchronize the simulation scene based on STK to obtain the corresponding reward R t and the next state s t+1 , and update the optimal strategy of the constellation;

[0170] S4-5, the current state s t , next state s t+1 、Action a t and reward R t As sample data and stored in the experience pool RB;

[0171] The experience pool RB adopts a priority experience replay mechanism, namely:

[0172] δ t =R t+1 +γmax a q(s t+1 ,a)-q(S t ,A t )

[0173] The absolute value of the TD-error value of the data |δ t | is used as the sampling weight of the data, giving samples with high learning value a larger sampling weight to achieve higher training efficiency. The larger the TD-error, the less accurate the current value prediction, and the more effective the data sample is for learning the agent value model. t+1 ,a) indicates the state is s t+1 The output data of the online value network for executing action a, q(S t ,A t ) indicates the state is S t Next, execute action A t The output data of the online value network.

[0174] S4-6. Randomly sample N sample data from the experience pool RB and select actions through the target strategy network Calculate the loss function L;

[0175] The calculation formula of the loss function L of S4-6 is:

[0176]

[0177] Among them, min(·) represents the minimum function, represents the network parameters of the target value network, Represents the network parameters of the online value network, s i+1 、s i They represent the states of the i+1th sample data and the ith sample data, respectively. q(·) represents the online value network. a i represents the action of the i-th sample data, and n represents the sequence number of the online value network;

[0178] S4-6. Update the parameters of the target value network and the online value network according to the loss function L;

[0179] S4-7. Calculate the policy gradient and update the parameters of the online policy network;

[0180] Policy gradient of steps S4-7 The calculation formula is:

[0181]

[0182] in, Indicates that the online value network q has the current network parameters According to the current state s i 、Action a i The value gradient of the output, Indicates that in the online policy network π, the current network parameter is θ π According to the current state s i The output action gradient;

[0183] S4-8, update the target policy network and the target value network of step S4-6 by soft update. The corresponding formula is:

[0184]

[0185] in, They represent the network parameters of target1 value network, target2 value network and target strategy network respectively, τ represents the soft update coefficient, θ π Respectively represent the network parameters of the online1 value network, online2 value network, and online strategy network. S4-9, determine whether the current step number reaches the total number of steps in the round; if so, the current round ends and enters step S4-10; otherwise, the current step number is increased by 1 and returns to step S4-3; wherein the initial value of the step number is 0;

[0186] S4-10. Repeat steps S4-1 to S4-9 until the total number of training rounds is reached.

[0187] S5. Execute the final optimization strategy through the constellation to be observed to complete the optimization of the observed constellation.

[0188] In one embodiment of the present invention, simulation training is performed based on the constellation optimization design algorithm of the improved TD3, the weight set of the three-level indicators is obtained by the hierarchical analysis method, and the hierarchical analysis method judgment matrix of the three-level indicator weight set is constructed. The values ​​of the calculated three-level evaluation indicator weight set are shown in Table 1.

[0189] Table 1

[0190]

[0191] The main hyperparameter settings for training each strategy network and valuation network based on the TD3 algorithm are shown in Table 2.

[0192] Table 1

[0193]

[0194]

[0195] like Figure 3 and Figure 4 As shown in the figure, the reward curves of the universal constellation configuration and the Walker constellation configuration are analyzed. Compared with the simplified universal constellation configuration, the Walker constellation configuration is basically consistent in state and action dimensions for the same satellite scale, only in the case of a smaller constellation scale design. As the constellation scale increases, the optimization parameters required for the simplified universal constellation configuration are amplified in multiples, that is, the state and action dimensions of the simplified universal constellation configuration are amplified in multiples, which increases the difficulty of network learning. The direct impact is that the training process converges more slowly, while the state and action dimensions of the Walker constellation configuration remain basically unchanged. The simplified universal constellation configuration requires a longer convergence time. In addition, the training results for both the simplified universal constellation and the Walker constellation can achieve convergence, further proving that the constellation optimization algorithm of the improved TD3 algorithm of this method is effective.

[0196] Due to the inconsistency of the state and action dimensions of the two constellation configurations and the different value ranges of the optimization variables, the optimization times in the reward function are not comparable, and the rewards obtained for the environment design are inconsistent. It is meaningless to compare the reward values ​​here. The decision model obtained through training can be used to optimize the constellation. The optimization results are as follows: Figure 5 shown.

[0197] The universal constellation configuration is simulated and the simulation results of 12,000 trainings are compared with those of the particle swarm algorithm. The particle swarm is set to 20 and 100 for comparison.

[0198] like Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 As shown in the figure, the observation efficiency curve of this method is smoother than the two observation efficiency curves of the particle swarm algorithm, and the observation performance value of this method is 71.04, which is 7.81 higher than the observation performance value of the particle swarm of 20 and 2.1 higher than the observation performance value of the particle swarm of 100; the processing time of this method is only 69.39s, while the processing time when the particle swarm is set to 20 and 100 is 384.2s and 1890.64s respectively, which is much longer than the processing time of this method, further indicating that the observation performance of the constellation optimized by the particle swarm algorithm based constellation optimization method is improved with the increase of the particle swarm scale, but the required optimization time is significantly increased. Compared with the constellation optimization method based on the particle swarm algorithm, the constellation optimized by the constellation optimization algorithm proposed in this chapter has basically the same observation performance and shows obvious speed advantages. This is because compared to the particle swarm algorithm, the constellation optimization method based on the improved TD3 algorithm obtains a decision mapping through training, while the constellation optimization method based on the particle swarm algorithm is one-time and requires iterative search from the beginning each time it is used, which proves the superiority of the algorithm in this chapter in efficiency.

[0199] In summary, the present invention improves the shortcomings of low learning and training efficiency and instability of the optimization model of the traditional TD3, improves the training efficiency and stability of the constellation optimization design decision model by introducing priority experience replay and dynamic strategy noise, and improves the constellation's earth observation performance.

Claims

1. A method for optimizing observation constellations based on deep reinforcement learning, Features: The following steps are involved: S1. Determine the constellation configuration of the constellation to be observed and obtain its satellite orbit parameters; S2. Based on satellite orbit parameters, a constellation optimization strategy is constructed using reinforcement learning algorithm; S3, establish state space model, action space model and design reward function; S4. Based on the optimal strategy function, the improved TD3 algorithm is used to process the state space model, action space model and reward function to obtain the final optimization strategy; S5. Execute the final optimization strategy through the constellation to be observed to complete the optimization of the observed constellation.

2. The observation constellation optimization method based on deep reinforcement learning according to claim 1, Features: The step S1 comprises the following steps: S1-1. Determine the constellation configuration of the constellation to be observed; wherein the constellation configuration includes a universal constellation configuration and a Walker constellation configuration; S1-2, judging whether the constellation configuration of the constellation to be observed is a universal constellation configuration; if so, proceeding to step S1-3; otherwise, judging that the constellation configuration of the constellation to be observed is a Walker constellation configuration and proceeding to step S1-4; S1-3, taking the six first orbital parameters corresponding to each satellite of the universal constellation configuration as satellite orbital parameters, and proceeding to step S2; wherein the six first orbital parameters are respectively the satellite orbit semi-major axis, orbital eccentricity, orbital plane inclination, perigee argument, ascending node right ascension, and true anomaly; S1-4. Determine the total number of satellites N in the Walker constellation configuration s As well as the orbital elements, phase factors, orbital plane numbers, orbital semi-major axes, and orbital inclinations of each satellite; S1-5. Use the first satellite of the Walker constellation configuration as the seed satellite and obtain the corresponding ascending node right ascension and true anomaly at the initial moment; S1-6, according to the formula: Get the ascending node right ascension raan of the remaining satellites m,n and true anomaly Among them, raan 1,1 represents the right ascension of the ascending node of the seed satellite, P represents the orbital plane number, π represents pi, m represents that the satellite is located in the mth orbital plane, n represents the nth satellite on the orbital plane, represents the true anomaly of the seed satellite at the initial time; S1-7. The total number of satellites N in the Walker constellation configuration s , phase factor, orbital plane number, orbital semi-major axis, orbital plane inclination, right ascension of ascending nodes of all satellites, and true anomaly at the initial moment are taken as satellite orbit parameters, and step S2 is entered.

3. The observation constellation optimization method based on deep reinforcement learning according to claim 2, Features: The step S2 comprises the following steps: S2-1. Take the constellation to be observed as an intelligent agent and determine the optimization task of the constellation to be observed. Construct the corresponding MDP model based on the Markov decision process. The corresponding formula is: MDP=(S,A,P,R,γ) Among them, S represents the state set, including satellite orbit parameters, A represents the action set, P represents the state transfer function, R represents the immediate reward and punishment, and γ represents the discount factor; S2-2. Calculate the cumulative rewards corresponding to all decisions executed by the MDP model; S2-3. Calculate the optimal state value function and the optimal state action value function Among them, a represents the action in the action set A; S2-4. Based on the optimal state value function and the optimal state action value function Determine constellation optimization strategy.

4. The observation constellation optimization method based on deep reinforcement learning according to claim 3, Features: The calculation formula for the cumulative reward is: Among them, G t , G t+1 denote the cumulative rewards corresponding to time t and time t+1, ∑(·) denotes the summation function, R t+1 , R t+2 , R t+3 , R t+k+1 They represent the instant rewards and punishments corresponding to time t, time t+1, time t+3, and time t+k+1 respectively; The optimal state value function The calculation formula is: Among them, max(·) represents the maximum value function, s' and s represent the state, represents the optimal state value function corresponding to state s', P(s'|s,a) represents the probability of the agent performing action a to transfer to state s' when in state s, and R(s,a) represents the immediate reward or punishment for the agent performing action a when in state s; The calculation formula of the optimal state action value function is: in, Represents the optimal state-action-value function for the agent to perform action a' when in state s'.

5. The observation constellation optimization method based on deep reinforcement learning according to claim 3, Features: The step S3 comprises the following steps: S3-1, judging whether the constellation configuration of the constellation to be observed is a universal constellation configuration; if so, proceeding to step S3-2; otherwise, proceeding to step S3-4; S3-2. Change the semi-major axis of the satellite orbit of the universal constellation configuration to 0 Set it to 8000km, set the satellite orbit to a circular orbit, set the eccentricity to 0, set the perigee argument to 0, and take the orbital inclination, ascending node right ascension, true anomaly and position parameters as the state, that is, the state space model. The corresponding expression is: in, They represent the first satellite S of the constellation to be observed. 1 , the second satellite S 2 The orbital inclination They represent the first satellite S of the constellation to be observed. 1 , the second satellite S 2 Ascending node right ascension They represent the first satellite S of the constellation to be observed. 1 , the second satellite S 2 、Nth S Satellites True anomaly s lon 、s lat 、s alt They respectively represent the longitude, latitude and altitude of the constellation to be observed in the GPS84 coordinate system; S3-3, establish an action space model of the universal constellation configuration, that is, a state space model, and proceed to step S3-6; wherein the expression corresponding to the state space model a of the universal constellation configuration is: in, They represent the first satellite S of the constellation to be observed. 1 , the second satellite S 2 Inclination of orbital plane Actions, They represent the first satellite S of the constellation to be observed. 1 , the second satellite S 2 Right Ascension at the Ascending Node Actions, They represent the first satellite S of the observed constellation. 1 , the second satellite S 2 、Nth S Satellites At true anomaly Actions; S3-4, the semi-major axis of the satellite orbit of the Walker constellation configuration is a constant, and the number of orbital planes, phase factor, orbital plane inclination, ascending node right ascension, true anomaly and position parameters are taken as states, that is, the state space model, and the corresponding expression is: Among them, s P 、s F 、s inc , They respectively represent the status of the Walker constellation orbital plane number, phase factor, orbital plane inclination, ascending node right ascension, and true anomaly; S3-5. Establish an action space model of the Walker constellation configuration, that is, a state space model, and proceed to step S3-6; wherein the expression corresponding to the state space model a of the Walker constellation configuration is: Among them, a P 、a F 、a inc , They represent the actions on the orbital plane number, phase factor, orbital plane inclination, ascending node right ascension, and true anomaly of the Walker constellation respectively; S3-6, according to the formula: R=a 1 R 1 +a 2 R 2 Construct a reward function R; where R 1 , R 2 They represent the target observation performance reward and optimization times reward, α 1 , α 2 denote coefficients respectively, and α 1 +α 2 =1.

6. The observation constellation optimization method based on deep reinforcement learning according to claim 5, Features: The improved TD3 algorithm adopts a dynamic strategy noise mechanism and includes an Actor strategy network and a Critic value network; wherein the Actor strategy network includes a target strategy network and an online strategy network; the Critic value network includes a target value network and an online value network; the target value network includes a target1 value network and a target2 value network; and the online value network includes an online1 value network and an online2 value network.

7. The observation constellation optimization method based on deep reinforcement learning according to claim 6, Features: The training process of the improved TD3 algorithm in step S4 includes the following steps: S4-1, set the total number of training rounds and the total number of steps per round; S4-2, initialize the environment, that is, randomly initialize the training satellite orbit parameters within the preset value range, and obtain the current state s t ; S4-3, based on the current state s t , select action a through the online policy network t ; S4-4. Execute the action a t , and synchronize the simulation scene based on STK to obtain the corresponding reward R t and the next state s t+1 , and update the optimal strategy of the constellation; S4-5, the current state s t , next state s t+1 、Action a t and reward R t As sample data and stored in the experience pool RB; S4-6. Randomly sample N sample data from the experience pool RB and select actions through the target strategy network Calculate the loss function L; S4-6. Update the parameters of the target value network and the online value network according to the loss function L; S4-7. Calculate the policy gradient and update the parameters of the online policy network; S4-8, update the target policy network and the target value network of step S4-6 by soft update; S4-9, determine whether the current number of steps reaches the total number of steps in the round; if so, the current round ends and enters step S4-10; otherwise, the current number of steps is increased by 1 and returns to step S4-3; wherein the initial value of the number of steps is 0; S4-10. Repeat steps S4-1 to S4-9 until the total number of training rounds is reached.

8. The observation constellation optimization method based on deep reinforcement learning according to claim 7, Features: The formula of step S4-3 is: a t =clip(π(s t |θ μ ))+s,a low ,a high ),ε~N(0,σ epi 2 ) Among them, π(s t |θ μ ) indicates that the online policy network π is in the current network parameter θ μ According to the current state s t The output action, ε represents the policy noise of the current step, a low 、a high Indicates action a t The value range of Ν(0,σ epi 2 ) means the mean is 0 and the standard deviation is σ epi Gaussian distribution, clip(π(s t |θ μ )+ε,a low ,a high ) indicates that the current network parameter is θ μ And in the current state s t The online strategy network π under The calculation formula of the loss function L of S4-6 is: Among them, min(·) represents the minimum function, represents the network parameters of the target value network, Represents the network parameters of the online value network, s i+1 、s i They represent the states of the i+1th sample data and the ith sample data, respectively. q(·) represents the online value network. a i represents the action of the i-th sample data, and n represents the sequence number of the online value network; The policy gradient of step S4-7 The calculation formula is: in, Indicates that the online value network q has the current network parameters According to the current state s i 、Action a i The value gradient of the output, Indicates that in the online policy network π, the current network parameter is θ π According to the current state s i The output action gradient.

9. The observation constellation optimization method based on deep reinforcement learning according to claim 7, Features: The experience pool RB adopts a priority experience replay mechanism.

Citation Information

Patent Citations

  • Efficient multi-objective optimization method for satellite constellation

    CN107798187A

  • Satellite constellation configuration method and device based on real-time coverage area evaluation

    CN114826373A

  • LEO inter-satellite link dynamic planning method based on multi-agent reinforcement learning

    CN114928401A

  • Satellite space target collaborative observation distributed planning method based on multi-agent reinforcement learning

    CN116187160A

  • D2D user resource allocation method based on deep reinforcement learning algorithm and storage medium

    CN116456493A

Cited By

  • Satellite scheduling method and device based on reinforcement learning, equipment and storage medium

    CN120342462A

  • Cooperative routing and resource scheduling method and system for low earth orbit satellites

    CN122533639A