A step-by-step intelligent generation method for space orbit game
By using a tiered intelligent generation method, and employing an instant reward function and a multi-level digital simulation environment to train the decision-making model of the red spacecraft, the problems of low efficiency and difficulty in accurate positioning in existing technologies are solved, and efficient generation and rapid development of space orbital games are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-26
- Publication Date
- 2026-03-24
AI Technical Summary
Existing technologies in space orbital games suffer from problems such as low efficiency in intelligent generation, failure of training curves to converge, and difficulty in accurately locating design process errors, resulting in long development cycles.
A tiered intelligent generation method is adopted. By setting up an instant reward function and policy network for the red team's spacecraft, a multi-level digital simulation environment is built. Reinforcement learning algorithms are used for training to gradually improve the spacecraft's intelligent decision-making capabilities, including the design criteria for subjects one to three, and gradually increasing the constraints and information complexity.
It has realized a systematic and standardized intelligent generation process for space orbit game, which has improved generation efficiency, shortened the R&D cycle, and can quickly and accurately locate design problems, making it suitable for reusable development of different space missions.
Smart Images
Figure CN115892514B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a hierarchical intelligent generation method for space orbital game theory, belonging to the field of spacecraft orbital control technology. Background Technology
[0002] Spacecraft orbital game theory is characterized by significant long-term cumulative effects. Furthermore, spacecraft orbital control is a continuous control variable, resulting in an exponential increase in the complexity of its action space with increasing game duration. This leads to low intelligent generation efficiency and even convergence failure. Current intelligent methods for space orbital game theory mostly employ end-to-end one-step generation approaches. However, when faced with complex multi-constraint orbital game problems, these methods suffer from low generation efficiency and even convergence failure of the training curve. On the other hand, one-step generation methods struggle to accurately locate errors in the design process, easily leading to drawbacks such as long development cycles. Summary of the Invention
[0003] The technical problem to be solved by this invention is to overcome the shortcomings of the prior art and solve the problem of intelligent, rapid and effective generation of space orbit game.
[0004] The objective of this invention is achieved through the following technical solutions:
[0005] A hierarchical intelligent generation method for space orbit game theory includes:
[0006] Based on the space orbit game task, an instant reward function is set for the red team's spacecraft to incentivize the generation of the spacecraft's intelligent decision-making model;
[0007] Establish a strategy network and a value network for the Red Force spacecraft;
[0008] Set the hyperparameters of the reinforcement learning algorithm to help the learning and training algorithm converge.
[0009] Based on the design principles of Subject 1, build the first digital simulation environment;
[0010] Based on the red team's immediate reward function and the first simulation environment, the spacecraft intelligent decision-making model is trained to obtain the first spacecraft decision-making model;
[0011] Based on the design principles of Subject 2, build a second digital simulation environment;
[0012] The first spacecraft decision model is loaded, and learning and training are carried out according to the red team's real-time reward function and the second simulation environment to obtain the second spacecraft decision model;
[0013] Based on the design principles of driving test subject three, a third digital simulation environment was built;
[0014] The second spacecraft decision model is loaded, and learning and training are carried out according to the red team's real-time reward function and the third digital simulation environment to obtain the third spacecraft decision model;
[0015] By utilizing a third spacecraft decision model, spacecraft velocity increments are generated based on observations obtained from the environment to control the orbit, thereby conducting a space orbit game mission.
[0016] Preferably, the strategy network autonomously generates spacecraft velocity increments based on spacecraft observations to achieve orbit control.
[0017] Preferably, the value network outputs an estimate of the cumulative reward obtained in the current state based on spacecraft observations.
[0018] Preferably, hyperparameters include cumulative reward discount rate, learning rate, and batch data volume.
[0019] Preferably, the design criteria for Subject 1 include complete information, track constraints, and unilateral rule sets; the design criteria for Subject 2 include complete information, track constraints, fuel constraints, and red-blue game theory; and the design criteria for Subject 3 include incomplete information, track constraints, fuel constraints, and red-blue game theory.
[0020] Preferably, the orbital constraints are as follows: with the spacecraft's initial orbital position as the origin, the x-axis direction as the orbital velocity direction, the z-axis pointing towards the Earth's center, and the y-axis perpendicular to the orbital plane, the direction of which is determined by the right-hand screw rule, a relative orbital coordinate system is established; the spacecraft controls its orbit through pulse thrust.
[0021]
[0022] In the formula, The x, y, and z axes velocities of the spacecraft at time k;
[0023] [Δv x (k) Δv y (k) Δv z (k)] T The xyz velocity increments represent the spacecraft's velocity increments at time k;
[0024] Represents the x, y, and z-axis velocities of the spacecraft after its orbital maneuver;
[0025] The spacecraft's relative orbital dynamics are as follows:
[0026]
[0027] In the formula, [x(k) y(k) z(k)] T The x, y, and z axes represent the spacecraft's position at time k;
[0028] [x(k+1) y(k+1) z(k+1)] T The x, y, and z axes represent the spacecraft's position at time k+1;
[0029] The x, y, and z axes velocities represent the spacecraft's velocity at time k+1;
[0030] n represents the orbital angular velocity, T represents the simulation step size, and c nT and s nT These are abbreviations for cos(nT) and sin(nT), respectively.
[0031] Preferably, the fuel constraint is as follows: a maximum fuel limit is set for the spacecraft, and when the cumulative velocity increment of the spacecraft exceeds this limit, the spacecraft no longer has maneuverability, that is:
[0032]
[0033] [Δv x (k) Δv y (k) Δv z (k)] T This represents the xyz velocity increment of the spacecraft at time k.
[0034] Preferably, the red-blue game is as follows: In the space orbit game task, both the red and blue sides use decision models intelligently generated by deep reinforcement learning methods to carry out left-right mutual learning training, so as to achieve a spiral improvement in the intelligence of both the red and blue sides.
[0035] Preferably, in the space orbital game task, the red side uses a decision model intelligently generated by deep reinforcement learning to carry out learning and training, while the blue side uses differential games and Lambert orbital change methods to determine rules and strategies.
[0036] Preferably, the complete information is: the spacecraft can obtain the position and velocity information of other spacecraft in real time; the incomplete information is: a limit is set on the radar detection range of the spacecraft; when other spacecraft are outside the radar detection range, the spacecraft can only obtain their orientation information through optical sensors. Both position and velocity information are missing.
[0037] Compared with the prior art, the present invention has the following advantages:
[0038] (1) In view of the problem that decision intelligence is difficult to generate quickly and effectively in space orbit game, this invention focuses on the design principle of increasing the number of constraints, increasing the number of environmental / target information, and increasing the complexity of tasks from easy to difficult. Training subjects one to three are designed from easy to difficult, forming a systematic and standardized intelligent generation process for space orbit game, which effectively guides the gradual and stable improvement of the intelligence level of space orbit game.
[0039] (2) This invention can effectively carry out reusable development for different space missions. Most different space missions have many common parts, so the intelligent generation for new space missions does not need to start from scratch, but can start from the pre-defined subjects of the original space mission, thus shortening the research and development cycle and process.
[0040] (3) The present invention systematically, standardizedly and stepwise decomposes the intelligent generation training subjects. The steps are simple and the logic is clear. It meets the application requirements of actual mission models and is more conducive to the actual application of spacecraft in orbit.
[0041] (4) Compared with the end-to-end intelligent generation mode, the tiered intelligent generation process proposed in this invention can more accurately and quickly locate the problems and defects in the design, reduce the number of project iterations, and effectively shorten the R&D cycle. Attached Figure Description
[0042] Figure 1 A smart generation process for space orbital game theory provided for the invention. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0044] A hierarchical intelligent generation method for space orbit game theory includes:
[0045] (1) Based on the space orbit game scenario, an instant reward function is set for the red spacecraft to incentivize the generation of the spacecraft's intelligent decision-making model;
[0046] (2) Set up a strategy network and a value network for the Red Team's spacecraft;
[0047] (3) Set the hyperparameters of the reinforcement learning algorithm to assist in the convergence of the learning and training algorithm;
[0048] (4) Based on the design principles of Subject 1 (complete information, track constraints, unilateral rule body), build a digital simulation environment 1.0;
[0049] (5) Based on the red side's instant reward function and simulation environment 1.0, the spacecraft intelligent decision-making model is trained to obtain spacecraft decision-making model 1.0;
[0050] (6) Based on the design criteria of Subject 2 (complete information, track constraints, fuel constraints, red-blue game), build a digital simulation environment 2.0;
[0051] (7) Load the spacecraft decision model 1.0, and conduct learning and training according to the red team's instant reward function and simulation environment 2.0 to obtain the spacecraft decision model 2.0;
[0052] (8) Based on the design principles of Subject 3 (incomplete information, track constraints, fuel constraints, red-blue game), build a digital simulation environment 3.0;
[0053] (9) Load the spacecraft decision model 2.0, and conduct learning and training according to the red team's instant reward function and simulation environment 3.0 to obtain the spacecraft decision model 3.0;
[0054] (10) Using Spacecraft Decision Model 3.0, spacecraft velocity increments are generated based on observations obtained from the environment to control the orbit and then carry out space orbit game tasks.
[0055] Preferably, the strategy network can autonomously generate spacecraft velocity increments based on spacecraft observations to achieve orbit control, and the value network can output an estimate of the cumulative reward obtainable in the current state based on spacecraft observations.
[0056] Preferably, the hyperparameters include cumulative reward discount rate, learning rate, and batch data volume.
[0057] Preferably, the orbital constraints are as follows: a relative orbital coordinate system is established with the spacecraft's initial orbital position as the origin, the x-axis pointing towards the orbital velocity, the z-axis pointing towards the Earth's center, and the y-axis perpendicular to the orbital plane, with its direction determined by the right-hand screw rule. The spacecraft performs orbital control via pulse thrust.
[0058]
[0059] In the formula, The x, y, and z axes velocities of the spacecraft at time k;
[0060] [Δv x (k) Δv y (k) Δv z (k)] T The xyz velocity increments represent the spacecraft's velocity increments at time k;
[0061] This represents the x, y, and z-axis velocities of the spacecraft after its orbital maneuver.
[0062] The spacecraft's relative orbital dynamics are as follows:
[0063]
[0064] In the formula, [x(k) y(k) z(k)] T The x, y, and z axes represent the spacecraft's position at time k;
[0065] [x(k+1) y(k+1) z(k+1)] TThe x, y, and z axes represent the spacecraft's position at time k+1;
[0066] The x, y, and z axes velocities represent the spacecraft's velocity at time k+1;
[0067] n represents the orbital angular velocity, T represents the simulation step size, and c nT and s nT These are abbreviations for cos(nT) and sin(nT), respectively.
[0068] Preferably, the fuel constraint is: setting a maximum fuel limit for the spacecraft; when the cumulative velocity increment of the spacecraft exceeds this maximum limit, the spacecraft no longer possesses maneuverability, that is:
[0069]
[0070] Preferably, the unilateral rule body is as follows: in the space orbit game task, the red side uses a decision model intelligently generated by deep reinforcement learning to carry out learning and training, while the blue side uses a rule strategy designed by differential game and Lambert orbit change method.
[0071] Preferably, the red-blue game is as follows: in the space orbit game task, both the red and blue sides use decision models intelligently generated by deep reinforcement learning methods to carry out left-right mutual learning training, so as to achieve a spiral improvement in the intelligence of both the red and blue sides.
[0072] Preferably, the complete information is: the spacecraft can obtain the position and velocity information of other spacecraft in real time.
[0073] Preferably, the incomplete information is: setting a radar detection range limit for the spacecraft. When the relative distance between the spacecraft and other spacecraft is less than or equal to the maximum radar detection range, the spacecraft can obtain the position and velocity information of other spacecraft in real time; when the relative distance between the spacecraft and other spacecraft is greater than the maximum radar detection range, the spacecraft can only obtain the orientation information of other spacecraft through optical sensors. Both position and velocity information were lost.
[0074] Example:
[0075] A tiered intelligent generation method for space orbital game theory is proposed, targeting a scenario where spacecraft A approaches spacecraft B for detailed investigation. Following the design principles of increasing constraints, environmental / target information, and task complexity, training subjects one through three are designed from easy to difficult, forming a systematic and standardized intelligent generation process for space orbital game theory. This effectively guides the gradual and stable improvement of the intelligence level in space orbital game theory. The resulting intelligent decision-making model 3.0 can autonomously calculate the x, y, and z-axis velocity increments in orbit based on observations obtained from the environment, successfully completing the task of detailed investigation of spacecraft B's approach. The specific method is as follows:
[0076] (1) Based on the space orbit game scenario, set an instant reward function for the red spacecraft A;
[0077] Instant reward R for Red Team spacecraft A red It is divided into two parts, the first part R r1 R is the negative of the relative distance between the red spacecraft A and the blue spacecraft B. r2 It is the negative of the sum of the absolute values of the velocity increments of the red spacecraft A, specifically:
[0078]
[0079] R r2 =-(|Δv x |+|Δv y |+|Δv z |)
[0080] R red =R r1 +R r2
[0081] In the formula, [x r y r z r ] T Represents the xyz three-axis position of the red spacecraft A, [x b y b z b ] T Represents the xyz axis position of the blue spacecraft B, [Δv x Δv y Δv z ] T The xyz three-axis velocity increments represent the red spacecraft A.
[0082] (2) Set up a strategy network and a value network for Red Spacecraft A;
[0083] The strategy network and value network of Red Spacecraft A adopt the same network structure.
[0084] Policy network structure: 3 hidden layers, with 256 nodes in the first layer and 128 nodes each in the second and third layers.
[0085] Value network structure: 3 hidden layers, with 256 nodes in the first layer and 128 nodes each in the second and third layers.
[0086] Network intermediate layer activation function: ReLU
[0087] Network output layer activation function: Sigmoid
[0088] The expression for the ReLU function is:
[0089] y = max(0,x)
[0090] In the formula, x is the input of the activation function, y is the output of the activation function, and max(0,x) is the maximum value between 0 and x.
[0091] The expression for the Sigmoid function is:
[0092]
[0093] In the formula, x is the input of the activation function, y is the output of the activation function, and e is the natural constant.
[0094] (3) Set the hyperparameters of the reinforcement learning algorithm to assist in the convergence of the learning and training algorithm;
[0095] The cumulative reward discount rate is set to 0.98, the learning rate to 0.0002, and the batch data size to 128.
[0096] (4) Based on the design principles of Subject 1 (complete information, track constraints, unilateral rule body), build a digital simulation environment 1.0;
[0097] The complete information is as follows: Red spacecraft A can obtain the position and velocity information of Blue spacecraft B in real time.
[0098] The orbital constraints are as follows: The initial orbital position of the red spacecraft A is taken as the origin, the x-axis represents the orbital velocity, the z-axis points towards the Earth's center, and the y-axis is perpendicular to the orbital plane, with its direction determined by the right-hand screw rule, establishing a relative orbital coordinate system. The spacecraft uses pulse thrust for orbital control.
[0099]
[0100] In the formula, The x, y, and z axes velocities of the spacecraft at time k;
[0101] [Δv x (k) Δv y (k) Δv z (k)] T The xyz velocity increments represent the spacecraft's velocity increments at time k;
[0102] This represents the x, y, and z-axis velocities of the spacecraft after its orbital maneuver.
[0103] The spacecraft's relative orbital dynamics are as follows:
[0104]
[0105] In the formula, [x(k) y(k) z(k)] T The x, y, and z axes represent the spacecraft's position at time k;
[0106] [x(k+1) y(k+1) z(k+1)] T The x, y, and z axes represent the spacecraft's position at time k+1;
[0107] Let x, y, and z represent the three-axis velocities of the spacecraft at time k+1. Set the initial orbit of the red spacecraft A to a geostationary orbit (GEO) with an orbital angular velocity n of 7.29 × 10⁻⁶. -5 rad / s, simulation step size T is 10 minutes, c nT and s nT These are abbreviations for cos(nT) and sin(nT), respectively.
[0108] The unilateral rule body is as follows: the red spacecraft A uses a decision model intelligently generated by deep reinforcement learning to carry out learning and training, while the blue spacecraft B uses a rule strategy designed by differential game and Lambert orbit change method.
[0109] (5) Based on the red side's instant reward function and simulation environment 1.0, the spacecraft intelligent decision-making model is trained to obtain the decision-making model 1.0 of the red side's spacecraft A;
[0110] (6) Based on the design criteria of Subject 2 (complete information, track constraints, fuel constraints, red-blue game), build a digital simulation environment 2.0;
[0111] The fuel constraint is as follows: the maximum fuel speed of the Red Force spacecraft A is set at 50 m / s. When the cumulative velocity increment of the Red Force spacecraft A exceeds 50 m / s, the Red Force spacecraft A will no longer have maneuverability.
[0112]
[0113] The red-blue game is as follows: both red spacecraft A and blue spacecraft B use decision models intelligently generated by deep reinforcement learning methods to carry out left-right mutual learning training, so as to achieve a spiral improvement in the intelligence of both red and blue sides.
[0114] (7) Load decision model 1.0, and conduct learning and training according to the red side's instant reward function and simulation environment 2.0 to obtain the decision model 2.0 of the red side's spacecraft A;
[0115] (8) Based on the design principles of Subject 3 (incomplete information, track constraints, fuel constraints, red-blue game), build a digital simulation environment 3.0;
[0116] The incomplete information is as follows: The maximum detection range of the radar of the red spacecraft A is set to 50km. When the relative distance between the red spacecraft A and the blue spacecraft B is less than or equal to 50km, spacecraft A can obtain the position and velocity information of spacecraft B in real time; when the relative distance between the two spacecraft A and spacecraft B is greater than 50km, spacecraft A can only obtain the azimuth information of spacecraft B through the optical sensor. Both position and velocity information were lost.
[0117] (9) Load decision model 2.0, and conduct learning and training according to the red side's instant reward function and simulation environment 3.0 to obtain the decision model 3.0 of the red side's spacecraft A;
[0118] (10) Using Decision Model 3.0, Red Spacecraft A generates spacecraft velocity increments based on observations obtained from the environment to perform orbit control and conduct a close-in detailed investigation of Spacecraft B.
[0119] like Figure 1 As shown, the overall design of the intelligent generation process and the design principles for each subject are presented.
[0120] The contents not described in detail in this specification are common knowledge to those skilled in the art.
[0121] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.
Claims
1. A ladder-based intelligent generation method for space orbit game theory, characterized in that, include: Based on the space orbit game task, an instant reward function is set for the red team's spacecraft to incentivize the generation of the spacecraft's intelligent decision-making model; Establish a strategy network and a value network for the Red Force spacecraft; Set the hyperparameters of the reinforcement learning algorithm to help the learning and training algorithm converge. Based on the design principles of Subject 1, build the first digital simulation environment; Based on the red team's instant reward function and the first digital simulation environment, the spacecraft intelligent decision-making model is trained to obtain the first spacecraft decision-making model; Based on the design principles of Subject 2, build a second digital simulation environment; The first spacecraft decision model is loaded, and learning and training are carried out according to the red team's real-time reward function and the second digital simulation environment to obtain the second spacecraft decision model; Based on the design principles of driving test subject three, a third digital simulation environment was built; The second spacecraft decision model is loaded, and learning and training are carried out according to the red team's real-time reward function and the third digital simulation environment to obtain the third spacecraft decision model; Using a third spacecraft decision model, spacecraft velocity increments are generated based on observations obtained from the environment to control the orbit, thereby carrying out space orbit game tasks; The design criteria for Subject 1 include complete information, track constraints, and unilateral rule sets; the design criteria for Subject 2 include complete information, track constraints, fuel constraints, and red-blue game theory; the design criteria for Subject 3 include incomplete information, track constraints, fuel constraints, and red-blue game theory. The complete information is: the spacecraft can obtain the position and velocity information of other spacecraft in real time; the incomplete information is: the spacecraft's radar detection range is limited; when other spacecraft are outside the radar detection range, the spacecraft can only obtain their orientation information through optical sensors. Position and velocity information are both missing; [x(k)y(k)z(k)] T The x, y, and z axes represent the spacecraft's position at time k; The fuel constraint is as follows: a maximum fuel limit is set for the spacecraft. When the cumulative velocity increment of the spacecraft exceeds this limit, the spacecraft loses its maneuverability. [Δv x (k)Δv y (k)Δv z (k)] T This represents the xyz velocity increment of the spacecraft at time k.
2. The step-by-step intelligent generation method according to claim 1, characterized in that, The strategy network autonomously generates spacecraft velocity increments based on spacecraft observations, thereby achieving orbit control.
3. The step-by-step intelligent generation method according to claim 1, characterized in that, The value network outputs an estimate of the cumulative reward obtained in the current state based on spacecraft observations.
4. The step-by-step intelligent generation method according to claim 1, characterized in that, Hyperparameters include cumulative reward discount rate, learning rate, and batch data volume.
5. The step-by-step intelligent generation method according to claim 1, characterized in that, The orbital constraints are as follows: The initial orbital position of the spacecraft is taken as the origin, the x-axis represents the orbital velocity direction, the z-axis points towards the Earth's center, and the y-axis is perpendicular to the orbital plane, with its direction determined by the right-hand screw rule, establishing a relative orbital coordinate system. The spacecraft controls its orbit through pulse thrust. In the formula, The x, y, and z axes velocities of the spacecraft at time k; [Δv x (k)Δv y (k)Δv z (k)] T The xyz velocity increments represent the spacecraft's velocity increments at time k; Represents the x, y, and z-axis velocities of the spacecraft after its orbital maneuver; The spacecraft's relative orbital dynamics are as follows: In the formula, [x(k+1)y(k+1)z(k+1)] T The x, y, and z axes represent the spacecraft's position at time k+1; The x, y, and z axes velocities represent the spacecraft's velocity at time k+1; n represents the orbital angular velocity, T represents the simulation step size, and c nT and s nT These are abbreviations for cos(nT) and sin(nT), respectively.
6. The step-by-step intelligent generation method according to claim 1, characterized in that, Red-Blue Game: In a space orbital game task, both the red and blue sides use decision models intelligently generated by deep reinforcement learning methods to carry out left-right mutual learning training, so as to achieve a spiral improvement in the intelligence of both sides.
7. The step-by-step intelligent generation method according to claim 6, characterized in that, In the space orbital game task, the red team uses a decision model intelligently generated by deep reinforcement learning to carry out learning and training, while the blue team uses differential games and Lambert orbital change methods to determine the rules and strategies.