Maneuvering strategy layered training method oriented to multi-satellite orbit interaction

Abstract

The invention provides a maneuvering strategy hierarchical training method for multi-satellite orbit interaction, and relates to spacecraft control, and the method comprises the steps: building an orbit pursuit interaction model of three satellites according to a preset CW coordinate system; determining first interaction data and a first reward function when the tracking satellite interacts with the escape satellite according to the orbit pursuit and escape interaction model, and second interaction data and a second reward function when the tracking satellite interacts with the defense satellite, and training a preset initial neural network; sequentially obtaining a first tracking network model of a tracking star, a first escape network model of an escape star, a second escape network model of the tracking star and a second tracking network model of a defense star; and determining third interaction data and a third reward function during three-satellite interaction according to the orbit pursuit interaction model, the first escape network model and the second tracking network model, and training the initial decision network of the tracking satellites to obtain a maneuvering strategy selection model. According to the scheme, the intelligent switching of the tracking star between the tracking behavior and the escape behavior is realized.

Description

Technical Field

[0001] The present invention relates to the field of spacecraft control technology, and in particular to a hierarchical training method for maneuvering strategies oriented to multi-satellite orbit interaction. Background Art

[0002] The types and functions of spacecraft are becoming increasingly diverse, and their capabilities are constantly improving. They have gradually developed from early deterministic and standardized tasks to highly dynamic and uncertain tasks. Spacecraft orbital interaction refers to the game between two spacecraft with autonomous maneuvering capabilities with the goal of tracking or escaping. The study of orbital interaction is of great significance to improving the on-orbit space security defense capabilities of spacecraft and responding to the rendezvous threat of unidentified objects such as space debris.

[0003] Traditional orbital interactions only include two roles: pursuer and escaper. However, with the development of spacecraft, orbital interaction tasks have shown multi-role, multi-individual and high-dynamic characteristics. This makes it difficult for existing methods of directly training spacecraft to train tracking strategies that meet mission requirements.

[0004] Based on this, there is an urgent need for a hierarchical training method for maneuvering strategies for multi-satellite orbit interaction to solve the above technical problems. Summary of the invention

[0005] The embodiment of the present invention provides a hierarchical training method for maneuvering strategies for multi-satellite orbit interaction, which can train an intelligent maneuvering strategy that can avoid threats and track predetermined targets at the same time.

[0006] In a first aspect, an embodiment of the present invention provides a hierarchical training method for multi-satellite orbit interaction maneuvering strategy, including: Establishing an orbital pursuit and escape interactive model of three stars according to a preset CW coordinate system; wherein the three stars include three spacecrafts, namely, a tracking star, an escape star and a defense star, and the orbital pursuit and escape interactive model includes a relative orbital motion model, a thrust constraint model, an optimization index and a winning or losing condition; Determine the first interaction data and the first reward function when the tracking star and the escape star interact according to the orbit pursuit interaction model, and train the first initial neural network of the binary star to obtain the first tracking network model of the tracking star and the first escape network model of the escape star; Determine the second interaction data and the second reward function when the tracking star and the defending star interact according to the orbit pursuit interaction model, and train the second initial neural network of the binary star to obtain the second escape network model of the tracking star and the second tracking network model of the defending star; The third interaction data and the third reward function during the three-star interaction are determined according to the orbital pursuit interaction model, the first escape network model and the second tracking network model, and the initial decision network of the tracking star is trained to obtain a maneuver strategy selection model for switching the maneuver strategy of the tracking star.

[0007] In a second aspect, an embodiment of the present invention further provides a maneuver strategy hierarchical training device for multi-satellite orbit interaction, comprising: A modeling module is used to establish an orbital pursuit and escape interaction model of three spacecraft according to a preset CW coordinate system; wherein the three spacecraft include a tracking star, an escape star and a defense star, and the orbital pursuit and escape interaction model includes a relative orbital motion model, a thrust constraint model, an optimization index and a winning and losing condition; A first training module is used to determine the first interaction data and the first reward function when the tracking star and the escape star interact according to the orbit pursuit interaction model, and train the first initial neural network of the binary star to obtain the first tracking network model of the tracking star and the first escape network model of the escape star; The second training module is used to determine the second interaction data and the second reward function when the tracking star and the defending star interact according to the orbit pursuit interaction model, and train the second initial neural network of the binary star to obtain the second escape network model of the tracking star and the second tracking network model of the defending star; The third training module is used to determine the third interaction data and the third reward function during the three-star interaction according to the orbit pursuit interaction model, the first escape network model and the second tracking network model, and to train the initial decision network of the tracking star to obtain a maneuver strategy selection model for switching the tracking star maneuver strategy.

[0008] In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in any embodiment of this specification is implemented.

[0009] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method described in any embodiment of this specification.

[0010] The embodiment of the present invention provides a hierarchical training of maneuvering strategies for multi-satellite orbital interaction. For a multi-role pursuit and escape game system consisting of a tracking star, an escaping star, and a defending star, in order to empower the tracking star, a hierarchical training architecture is adopted, including the first layer training the tracking star's strategy of approaching the escaping star; the second layer training the tracking star's strategy of avoiding the defending star; the third layer training the tracking star's role switching strategy, intelligently switching between pursuit and escape styles; and finally achieving that the tracking star can avoid the defending star and approach the escaping star through intelligent switching of strategies. This method fully considers the task requirements of the tracking star in the three-star orbital pursuit and escape interaction, that is, the tracking star must track the escaping star and avoid the defending star at the same time, accurately models the interactive system's motion model, constraint model, optimization index, winning and losing conditions, etc., proposes a multi-satellite collaborative hierarchical training architecture, and realizes the intelligent switching of the tracking star between tracking and escape behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 This is a flow chart of a hierarchical training method for multi-satellite orbit interaction maneuvering strategy provided by an embodiment of the present invention; Figure 2 is a schematic diagram of the profit curves of three types of spacecraft during the neural network training process provided by an embodiment of the present invention; Figure 3 is a hardware architecture diagram of an electronic device provided by an embodiment of the present invention; Figure 4 It is a structural diagram of a maneuver strategy hierarchical training device for multi-satellite orbit interaction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0014] As mentioned above, existing pursuit-and-escape interaction training usually only takes into account the situation of two stars. This training method makes it difficult for the tracking star to obtain satisfactory training results when facing the situation of multiple star interactions.

[0015] Based on this, the concept of the present invention is to consider both the tracking and escaping roles of the tracking star, and to improve the training efficiency and the flexibility and adaptability of the training results through hierarchical training.

[0016] The specific implementation of the above concept is described below.

[0017] Please refer to Figure 1 The embodiment of the present invention provides a hierarchical training method for maneuvering strategies for multi-satellite orbit interaction, the method comprising: Step 100, establishing a three-star orbit pursuit interaction model according to a preset CW coordinate system; wherein the three stars include three types of spacecraft, namely, a tracking star, an escape star and a defense star, and the orbit pursuit interaction model includes a relative orbit motion model, a thrust constraint model, an optimization index and a winning or losing condition; Step 102, determining the first interaction data and the first reward function when the tracking star and the escape star interact according to the orbit pursuit interaction model, and training the first initial neural network of the binary star to obtain the first tracking network model of the tracking star and the first escape network model of the escape star; Step 104, determining the second interaction data and the second reward function when the tracking star and the defending star interact according to the orbit pursuit interaction model, and training the second initial neural network of the two stars to obtain the second escape network model of the tracking star and the second tracking network model of the defending star; Step 106, determining the third interaction data and the third reward function when three stars interact according to the orbit pursuit interaction model, the first escape network model and the second tracking network model, and training the initial decision network of the tracking star to obtain a maneuver strategy selection model for switching the tracking star maneuver strategy.

[0018] In the embodiment of the present invention, for a multi-role pursuit and escape game system consisting of a tracking star, an escaping star, and a defending star, in order to empower the tracking star, a hierarchical training architecture is adopted, including the first layer training the tracking star's strategy of approaching the escaping star; the second layer training the tracking star's strategy of avoiding the defending star; the third layer training the tracking star's role switching strategy, intelligently switching between pursuit and escape styles; and finally achieving that the tracking star can avoid the defending star and approach the escaping star through intelligent switching of strategies. This method fully considers the task requirements of the tracking star in the three-star orbit pursuit and escape interaction, that is, the tracking star must track the escaping star and avoid the defending star at the same time, accurately models the interactive system's motion model, constraint model, optimization index, winning and losing conditions, etc., proposes a multi-star collaborative hierarchical training architecture, and realizes the tracking star's intelligent switching between tracking and escape behaviors.

[0019] Described below Figure 1 How the various steps are performed.

[0020] First, with respect to step 100, a three-star orbit pursuit interaction model is established according to a preset CW coordinate system.

[0021] In the embodiment of the present invention, the three stars include three spacecrafts: a tracking star, an escape star and a defense star. The established orbit pursuit and escape interaction model includes a relative orbit motion model, a thrust constraint model, optimization indicators and winning and losing conditions.

[0022] Specifically, CW relative orbital dynamics is used to describe the orbital motion of each spacecraft, and a relative orbital dynamics is established with any point in the geosynchronous orbit as the origin and the orbital velocity direction as , the direction of the center of the earth is , for The CW coordinate system of the spacecraft is denoted by ,in p It stands for Pursuer. e stands for evader, d represents the defense star. For the pulse thrust engine, when performing orbit recursion, since the orbit motion time is much longer than the engine jet time, it can be considered that the velocity increment is generated instantaneously, so the relative orbit motion model can be written as: in ; is the single pulse amplitude of the spacecraft at the kth pulse moment, is the position vector of the spacecraft relative to the CW coordinate system; is the velocity of the spacecraft relative to the CW reference frame. From t 0 The state transfer matrix from time to time t is expressed as: in, t 0 The state transfer matrix from the position at time t to the position at time t; t 0 The state transfer matrix from the velocity at time t to the position at time t; t 0 The state transfer matrix from the position at time t to the velocity at time t; t 0 The state transfer matrix from the speed at time t to the speed at time t.

[0023] The expressions of each module of the transfer matrix are: In the formula, is the angle at which the origin of the CW coordinate system rotates relative to the center of the earth.

[0024] Furthermore, the thrust constraint model includes: Single axis pulse constraint: Pulse period constraint: Total pulse constraint: in , , They represent the single pulse amplitude of the X, Y, and Z axes of the i-th spacecraft body in the CW coordinate system, represents the upper limit of the single-axis pulse amplitude of the i-th spacecraft, and represents the time when the i-th spacecraft performs the k+1th pulse and the time when the k-th pulse occurs, represents the time interval between two adjacent pulses of the spacecraft. To facilitate the design and analysis of pulse strategies, it is assumed that in each round of interaction, the time interval between two adjacent pulses of the spacecraft is kept constant. At the same time, the spacecraft attitude motion is not considered, and it is assumed that the spacecraft attitude can be rotated as needed, and the camera optical axis, orbit control thruster, etc. are rotated to the desired position. It represents the upper limit of fuel consumption allowed for the i-th spacecraft in one game of interaction.

[0025] It is worth noting that the total pulse constraint is used to characterize the movement process of the spacecraft, that is, the spacecraft will no longer send pulses after the fuel is exhausted in an interaction round, but the interaction in this round will not stop due to the exhaustion of fuel. Instead, the interaction will end after the victory or defeat of the spacecraft is determined according to the following victory or defeat conditions.

[0026] In the embodiment of the present invention, the optimization index of the tracking star , Optimization index of escape star and optimization indicators of defense stars They are: Tracking Star Victory Conditions: in, and They are the safe defense zone radius of the escape star and the tracking star respectively; is the position of the escape star relative to the pursuit star; is the position of the defense star relative to the tracking star; The moment when the interaction ends; Escape Star victory conditions: in, To track the position of the star relative to the coordinate origin; The maximum allowed range of motion for the escape star; Defense Star Victory Conditions: in, To track the star's own safety defense distance.

[0027] Then, for step 102, the first interaction data and the first reward function when the tracking star and the escaping star interact are determined according to the orbit pursuit interaction model, and the first initial neural network of the binary star is trained to obtain the first tracking network model of the tracking star and the first escape network model of the escaping star.

[0028] After completing the construction of the orbital pursuit and escape interaction model for the three stars, we began to train the neural network for each spacecraft interaction. Considering that during the interaction of the three stars, the tracking star needs to make two decisions, tracking and escaping, according to the requirements, this embodiment adopts a hierarchical training method to train the interaction of two stars, that is, the interaction between the tracking star and the escape star, and the interaction between the tracking star and the defense star are trained separately. This method facilitates multiple people to develop and train different sub-models in parallel, improves the training efficiency for complex tasks, and lays the foundation for the rapid generation and adjustment of strategies for future on-orbit interactions. At the same time, since each sub-module is independent of each other, it can be flexibly increased or decreased according to specific tasks. After increase or decrease, only the top-level mode selection network needs to be retrained.

[0029] In the embodiment of the present invention, the training process for the above-mentioned dual-satellite interaction includes: S11, randomly initializing the initial state information of the spacecraft during the binary star interaction within a preset target range, wherein the state information includes the position and velocity of the spacecraft in the CW coordinate system; S12, using the initial neural network of the spacecraft in the corresponding interactive mode to recursively process the initial state information to obtain the next state information of the spacecraft; wherein the recursive processing includes inputting the initial state information into the corresponding initial neural network, outputting the thruster start time of each of the spacecraft, and recursively calculating the start time using the relative orbital motion model to obtain the next state information of the spacecraft; S13, re-inputting the next moment state information into the initial neural network, and repeating the recursive process until an interaction round ends; S14, inputting the state information at each moment obtained by recursion into the reward function of the corresponding interaction mode, outputting and storing the reward value at each moment, and obtaining the total benefit of one interaction round of the spacecraft in the same interaction mode; S15. Update the network weights of the initial neural network according to the gradient descent method, and use the updated initial neural network to repeat the calculation process of steps S11-S14 until the total revenue of each round converges to a fixed value, and record the neural network at this time as the network model of the corresponding interaction mode.

[0030] In the embodiment of the present invention, the first tracking network model and the first escape network model are trained in the following manner: Building the first reward function for tracking stars : in, and are all constants; Establishing the first reward function for the escape star : in, and are all constants; The first initial neural network is trained using a proximal strategy optimization algorithm, and the first initial neural network weights of the tracking star and the escaping star are updated respectively to obtain a first tracking network model of the tracking star and a first escaping network model of the escaping star.

[0031] For example, first build an interactive scene, let the tracking star, escape star and defense star randomly initialize their positions within 200km around the origin of the CW coordinate system, and ensure that the relative distance between the three stars is greater than 50km. The initial speed relative to the origin of the CW coordinate system is randomly selected from the range of [-10,10] (m / s). Set the winning condition of the tracking star to be at the terminal time t = t f =36000s to enter the 50km range of the escape star, and prevent the defense star from entering the 20km range of itself; the victory condition of the escape star is at the terminal time t = t f =36000s Tracking Star fails to enter 50km range; Defense Star victory condition is at the terminal time t = t f =36000s The tracking star fails to enter the 50km range of the escape star, and it enters the 20km range of the tracking star.

[0032] Then, the initial neural network of Samsung is established as shown in Table 1 below; Table 1 Then the network weights of the neural network are initialized.

[0033] Take the training process of tracking stars and escaped stars as an example: Design the reward function for tracking stars and escaping stars: Randomly initialize the positions and velocities of the tracking star and the escape star within a range of 50-200 km to serve as the initial state of the interaction; The status information of both parties at the current moment is input into their respective neural networks P_pursuit_net0 and E_evasion_net0. The tracking star and escape star networks output their respective thruster startup times. According to the established relative orbital motion model, the status information at the next moment is recursively obtained and stored in the data pool.

[0034] Then substitute the state information at the next moment into the reward function to obtain the reward value of both parties at the current moment and store it in the data pool.

[0035] The state at the next moment is used as the state at the current moment to repeat the above process. The interval between adjacent sampling moments is 10 minutes. Until the end of an interaction round (t=36000s), the reward value in the data pool is accumulated by discount. , discount factor γ=0.95, the total revenue of the tracking star and the escaped star in one round is obtained, and then the gradient descent method is used to update the network weights of the tracking star and the escaped star.

[0036] Finally, the above process is repeated until the variance of the payoffs of both parties in 10 consecutive rounds is less than 1. At this time, the first pursuit network model of the pursuit star and the first escape network model of the escape star are recorded as P_pursuit_net and E_pursuit_net respectively.

[0037] For step 104, the second interaction data and the second reward function when the tracking star and the defending star interact are determined according to the orbital pursuit interaction model, and the second initial neural network of the two stars is trained to obtain the second escape network model of the tracking star and the second tracking network model of the defending star.

[0038] The training process for tracking stars and defending stars is similar to the above steps, except that in this training, the tracking star changes from a tracking role to an escaping role.

[0039] In the embodiment of the present invention, the second escape network model and the second tracking network model are trained in the following manner: Establish the second reward function of the tracking star : in, and are all constants; The maximum allowed range of motion for tracking stars; To track the position of the star relative to the coordinate origin; Establish the second reward function of the defense star : in, and are all constants; The maximum allowed range of motion for the defense star; is the position of the defense star relative to the coordinate origin; The second initial neural network is trained using a proximal strategy optimization algorithm, and the second initial neural network weights of the tracking star and the defending star are updated respectively to obtain a second escape network model of the tracking star and a second tracking network model of the defending star.

[0040] For example, the reward function for tracking stars and defending stars is designed as follows Then, the positions and velocities of the tracking and defending stars are randomly initialized within a range of 50-200 km, which serves as the initial state of a game of interaction.

[0041] The status information of both parties at the current moment is input into their respective neural networks P_evasion_net0 and D_pursuit_net0. The tracking star and defense star networks output the startup time of their respective thrusters. According to the relative orbital motion model, the status information of the next moment is recursively obtained and stored in the data pool.

[0042] Then substitute the state information at the next moment into the reward function to obtain the reward value of both parties at the current moment and store it in the data pool.

[0043] The state at the next moment is used as the state at the current moment, and the above process is repeated until an interaction round ends. The interval between adjacent sampling moments is 10 minutes. Until an interaction round ends (t=36000s), the reward value discount in the data pool is accumulated. , discount factor γ=0.95, accumulate the reward values ​​in the data pool, get the total revenue of the tracking star and the defense star in one round, and then use the gradient descent method to update the network weights of the tracking star and the defense star.

[0044] The above process is repeated until the variance of the payoffs of both parties in 10 consecutive rounds is less than 1. At this time, the second escape network model of the pursuit star and the second pursuit network model of the defense star are recorded as P_evasion_net and D_pursuit_net respectively.

[0045] The training reward value change curves of the three spacecraft in the above process are as follows: Figure 2 shown.

[0046] For step 106, the third interaction data and the third reward function during the three-star interaction are determined according to the orbit pursuit interaction model, the first escape network model and the second tracking network model, and the initial decision network of the tracking star is trained to obtain a maneuver strategy selection model for switching the tracking star maneuver strategy.

[0047] The training process of triple-star interaction is also similar to the above steps, but due to the dual identity of the tracking star during triple-star interaction, it is necessary to switch the motion strategy at all times according to the position and speed relationship between the tracking star and the other two spacecraft, so as to achieve the purpose of tracking the escaped star and avoiding the defending star.

[0048] In the embodiment of the present invention, the maneuver strategy selection model is trained in the following manner: Establish the third reward function : in, and are all constants; is the position of the defense star relative to the tracking star; is the position of the defense star relative to the tracking star; and They are the safe defense zone radius of the escape star and the tracking star respectively; For the constant value of rewards and punishments; The escape model, the tracking model and the initial decision network are trained using a proximal strategy optimization algorithm, and the weight of the initial decision network is updated to obtain the maneuver strategy selection model. The training process includes: S21, randomly initialize the initial state information of three stars within a preset target range, and calculate the initial interaction data of the three stars; wherein the initial interaction data includes the initial position speed of the tracking star relative to the escape star, the initial position speed of the tracking star relative to the defense star, and the initial position speed of the defense star relative to the escape star; the initial state information includes the position and speed of the tracking star, the escape star, and the defense star relative to the origin of the CW coordinate system; S22, inputting the initial interaction data into the initial decision network, and outputting a strategy selection result; wherein the strategy selection result includes a tracking strategy and an escape strategy; S23, performing the recursive processing on the tracking star according to the strategy selection result to obtain the state information of the tracking star at the next moment; wherein, when the strategy selection result is the tracking strategy, the position and velocity of the tracking star relative to the origin of the CW coordinate system are recursively deduced using the first tracking network model to obtain the state information of the tracking star at the next moment; when the strategy selection result is the escape strategy, the position and velocity of the tracking star relative to the origin of the CW coordinate system are recursively deduced using the second escape network model to obtain the state information of the tracking star at the next moment; S24, using the first escape network model to recursively deduce the position and velocity of the escape star relative to the origin of the CW coordinate system to obtain the state information of the escape star at the next moment; S25, using the second tracking network model to recursively deduce the position and velocity of the defense star relative to the origin of the CW coordinate system to obtain the next moment state information of the defense star; S26, according to the next moment state information of the tracking star, the escape star, and the defense star, calculating the position velocity of the tracking star relative to the escape star, the position velocity of the tracking star relative to the defense star, and the position velocity of the defense star relative to the escape star at the next moment; S27, inputting the state information at each moment obtained by recursion into the third reward function, outputting and storing the reward value at each moment, and obtaining the total benefits of one interaction round of the three spacecrafts; S28. Update the network weights of the initial decision network according to the gradient descent method, and use the updated initial neural network to repeat the calculation process of steps S21-S27 until the total revenue of each round converges to a fixed value, and record the decision network at this time as the maneuver strategy selection model.

[0049] For example, the positions and velocities of the tracking star, escape star, and defending star are randomly initialized within a range of 50-200km, which serves as the initial state of a game of interaction.

[0050] The current position velocity of the tracking star relative to the escaping star, the position velocity of the tracking star relative to the defending star, and the position velocity of the defending star relative to the escaping star are input into the initial decision network P_select_net0 of the tracking star, and P_select_net0 outputs 0 or 1.

[0051] If P_select_net0 outputs 0, the tracking strategy is selected, and the position and velocity of the tracking star relative to the escape star are input into the P_pursuit_net network model. The network outputs the start-up time of the tracking star thrusters. According to the relative orbital motion model, the position and velocity of the tracking star at the next moment are recursively obtained and stored in the data pool; if P_select_net0 outputs 1, the escape strategy is selected, and the position and velocity of the tracking star relative to the defense star are input into the P_evasion_net network model. The network outputs the start-up time of the tracking star thrusters. According to the relative orbital motion model, the position and velocity of the tracking star at the next moment are recursively obtained and stored in the data pool.

[0052] The position and velocity of the tracking star relative to the escaping star are input into the escape network E_evasion_net of the escaping star. The network outputs the start-up time of the escaping star thrusters. According to the relative orbital motion model, the position and velocity of the escaping star at the next moment are recursively obtained and stored in the data pool.

[0053] The position and velocity of the tracking star relative to the defense star are input into the defense star's approach network D_pursuit_net. The network outputs the defense star thruster startup time. According to the relative orbital motion model, the position and velocity of the defense star at the next moment are recursively obtained and stored in the data pool.

[0054] According to the position and speed of the three stars at the next moment, the relative position and speed of the three stars are solved and substituted into their respective reward functions to obtain the reward value of the tracking star at the current moment and store it in the data pool.

[0055] The state at the next moment is taken as the state at the current moment, and the above process is repeated. The interval between adjacent sampling moments is 10 minutes, until an interaction round ends (t=36000s), and the reward value discount in the data pool is accumulated. , discount factor γ=0.95, accumulate the reward values ​​in the data pool to get the total revenue of the tracking star in one round, and then use the gradient descent method to update the network weights of the tracking star P_select_net0.

[0056] The above process is repeated until the variance of the benefits of both sides in 10 consecutive rounds is less than 1, and the maneuver strategy selection model of the tracking star at this time is recorded as P_select_net, completing the final training process.

[0057] like Figure 3 , Figure 4 As shown, the embodiment of the present invention provides a multi-satellite orbit interaction maneuver strategy hierarchical training device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From the hardware level, Figure 3As shown, it is a hardware architecture diagram of an electronic device in which a multi-satellite orbit interaction maneuvering strategy hierarchical training device is provided in an embodiment of the present invention. Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, Figure 4 As shown, as a device in a logical sense, the CPU of the electronic device in which it is located reads the corresponding computer program in the non-volatile memory into the memory and runs it. This embodiment provides a multi-satellite orbit interaction maneuver strategy layered training device, including: A modeling module 400 is used to establish an orbital pursuit and escape interaction model of three spacecraft according to a preset CW coordinate system; wherein the three spacecraft include a tracking star, an escape star and a defense star, and the orbital pursuit and escape interaction model includes a relative orbital motion model, a thrust constraint model, an optimization index and a winning or losing condition; The first training module 402 is used to determine the first interaction data and the first reward function when the tracking star and the escape star interact according to the orbit pursuit interaction model, and train the first initial neural network of the binary star to obtain the first tracking network model of the tracking star and the first escape network model of the escape star; The second training module 404 is used to determine the second interaction data and the second reward function when the tracking star and the defending star interact according to the orbit pursuit interaction model, and train the second initial neural network of the binary star to obtain the second escape network model of the tracking star and the second tracking network model of the defending star; The third training module 406 is used to determine the third interaction data and the third reward function during the three-star interaction according to the orbit pursuit interaction model, the first escape network model and the second tracking network model, and to train the initial decision network of the tracking star to obtain a maneuver strategy selection model for switching the tracking star maneuver strategy.

[0058] In the embodiment of the present invention, the winning and losing conditions include: Tracking Star Victory Conditions: in, and They are the safe defense zone radius of the escape star and the tracking star respectively; is the position of the escape star relative to the pursuit star; is the position of the defense star relative to the tracking star; The moment when the interaction ends; Escape Star victory conditions: in, To track the position of the star relative to the coordinate origin; The maximum allowed range of motion for the escape star; Defense Star Victory Conditions: in, To track the star's own safety defense distance.

[0059] In the embodiment of the present invention, the first tracking network model and the first escape network model are trained in the following manner: Establish the first reward function of the tracking star : in, and are all constants; Establish the first reward function of the escape star : in, and are all constants; The first initial neural network is trained using a proximal strategy optimization algorithm, and the first initial neural network weights of the tracking star and the escaping star are updated respectively to obtain a first tracking network model of the tracking star and a first escaping network model of the escaping star.

[0060] In the embodiment of the present invention, the second escape network model and the second tracking network model are trained in the following manner: Establish the second reward function of the tracking star : in, and are all constants; The maximum allowed range of motion for tracking stars; To track the position of the star relative to the coordinate origin; Establish the second reward function of the defense star : in, and are all constants; The maximum allowed range of motion for the defense star; is the position of the defense star relative to the coordinate origin; The second initial neural network is trained using a proximal strategy optimization algorithm, and the second initial neural network weights of the tracking star and the defending star are updated respectively to obtain a second escape network model of the tracking star and a second tracking network model of the defending star.

[0061] In the embodiment of the present invention, the initial neural network is trained using a proximal strategy optimization algorithm to obtain a tracking model and an escape model, including: S11, randomly initializing the initial state information of the spacecraft during the binary star interaction within a preset target range, wherein the state information includes the position and velocity of the spacecraft in the CW coordinate system; S12, using the initial neural network of the spacecraft in the corresponding interactive mode to recursively process the initial state information to obtain the next state information of the spacecraft; wherein the recursive processing includes inputting the initial state information into the corresponding initial neural network, outputting the thruster start time of each of the spacecraft, and recursively calculating the start time using the relative orbital motion model to obtain the next state information of the spacecraft; S13, re-inputting the next moment state information into the initial neural network, and repeating the recursive process until an interaction round ends; S14, inputting the state information at each moment obtained by recursion into the reward function of the corresponding interaction mode, outputting and storing the reward value at each moment, and obtaining the total benefit of one interaction round of the spacecraft in the same interaction mode; S15. Update the network weights of the initial neural network according to the gradient descent method, and use the updated initial neural network to repeat the calculation process of steps S11-S14 until the total revenue of each round converges to a fixed value, and record the neural network at this time as the network model of the corresponding interaction mode.

[0062] In the embodiment of the present invention, the maneuver strategy selection model is trained in the following manner: Establish the third reward function : in, and are all constants; is the position of the defense star relative to the tracking star; is the position of the defense star relative to the tracking star; and They are the safe defense zone radius of the escape star and the tracking star respectively; For the constant value of rewards and punishments; The escape model, the tracking model and the initial decision network are trained using a proximal strategy optimization algorithm, and the weights of the initial decision network are updated to obtain the maneuver strategy selection model.

[0063] In the embodiment of the present invention, the use of the proximal strategy optimization algorithm to train the escape model, the tracking model and the initial decision network to obtain the maneuver strategy selection model includes: S21, randomly initializing the initial state information of three stars within a preset target range, and calculating the initial interaction data of the three stars; wherein the initial interaction data includes the initial position speed of the tracking star relative to the escaping star, the initial position speed of the tracking star relative to the defending star, and the initial position speed of the defending star relative to the escaping star; S22, inputting the initial interaction data into the initial decision network, and outputting a strategy selection result; wherein the strategy selection result includes a tracking strategy and an escape strategy; S23, performing the recursive processing on the tracking star according to the strategy selection result to obtain the state information of the tracking star at the next moment; wherein, when the strategy selection result is the tracking strategy, the position and velocity of the tracking star relative to the origin of the CW coordinate system are recursively deduced using the first tracking network model to obtain the state information of the tracking star at the next moment; when the strategy selection result is the escape strategy, the position and velocity of the tracking star relative to the origin of the CW coordinate system are recursively deduced using the second escape network model to obtain the state information of the tracking star at the next moment; S24, using the first escape network model to recursively deduce the position and velocity of the escape star relative to the origin of the CW coordinate system to obtain the state information of the escape star at the next moment; S25, using the second tracking network model to recursively deduce the position velocity of the defense star relative to the origin of the CW coordinate system to obtain the next moment state information of the defense star; S26, according to the next moment state information of the tracking star, the escape star, and the defense star, calculating the position velocity of the tracking star relative to the escape star, the position velocity of the tracking star relative to the defense star, and the position velocity of the defense star relative to the escape star at the next moment; S27, inputting the state information at each moment obtained by recursion into the third reward function, outputting and storing the reward value at each moment, and obtaining the total benefits of one interaction round of the three spacecrafts; S28. Update the network weights of the initial decision network according to the gradient descent method, and use the updated initial neural network to repeat the calculation process of steps S21-S27 until the total revenue of each round converges to a fixed value, and record the decision network at this time as the maneuver strategy selection model.

[0064] It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on a maneuvering strategy layered training device for multi-satellite orbit interaction. In other embodiments of the present invention, a maneuvering strategy layered training device for multi-satellite orbit interaction may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0065] The information interaction, execution process and other contents between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For the specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0066] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a hierarchical training method for maneuvering strategies for multi-satellite orbit interaction in any embodiment of the present invention is implemented.

[0067] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes a hierarchical training method for maneuvering strategies for multi-satellite orbit interaction in any embodiment of the present invention.

[0068] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0069] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0070] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0071] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0072] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0073] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical factors in the process, method, article or device including the elements.

[0074] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hierarchical training method for maneuvering strategies for multi-satellite orbit interaction, characterized in that: include: Establishing an orbital pursuit and escape interactive model of three stars according to a preset CW coordinate system; wherein the three stars include three spacecrafts, namely, a tracking star, an escape star and a defense star, and the orbital pursuit and escape interactive model includes a relative orbital motion model, a thrust constraint model, an optimization index and a winning or losing condition; Determine the first interaction data and the first reward function when the tracking star and the escape star interact according to the orbit pursuit interaction model, and train the first initial neural network of the binary star to obtain the first tracking network model of the tracking star and the first escape network model of the escape star; Determine the second interaction data and the second reward function when the tracking star and the defending star interact according to the orbit pursuit interaction model, and train the second initial neural network of the binary star to obtain the second escape network model of the tracking star and the second tracking network model of the defending star; The third interaction data and the third reward function during the three-star interaction are determined according to the orbital pursuit interaction model, the first escape network model and the second tracking network model, and the initial decision network of the tracking star is trained to obtain a maneuver strategy selection model for switching the maneuver strategy of the tracking star.

2. The method according to claim 1, characterized in that The winning and losing conditions include: Tracking Star Victory Conditions: in, and They are the safe defense zone radius of the escape star and the tracking star respectively; is the position of the escape star relative to the pursuit star; is the position of the defense star relative to the tracking star; The moment when the interaction ends; Escape Star victory conditions: in, To track the position of the star relative to the coordinate origin; The maximum allowed range of motion for the escape star; Defense Star Victory Conditions: in, To track the star's own safety defense distance.

3. The method according to claim 2, characterized in that The first tracking network model and the first escape network model are trained in the following manner: Establish the first reward function of the tracking star : in, and are all constants; Establish the first reward function of the escape star : in, and are all constants; The first initial neural network is trained using a proximal strategy optimization algorithm, and the first initial neural network weights of the tracking star and the escaping star are updated respectively to obtain a first tracking network model of the tracking star and a first escaping network model of the escaping star.

4. The method according to claim 2, characterized in that: The second escape network model and the second tracking network model are trained in the following manner: Establish the second reward function of the tracking star : in, and are all constants; The maximum allowed range of motion for tracking stars; To track the position of the star relative to the coordinate origin; Establish the second reward function of the defense star : in, and are all constants; The maximum allowed range of motion for the defense star; is the position of the defense star relative to the coordinate origin; The second initial neural network is trained using a proximal strategy optimization algorithm, and the second initial neural network weights of the tracking star and the defending star are updated respectively to obtain a second escape network model of the tracking star and a second tracking network model of the defending star.

5. The method according to claim 3 or 4, characterized in that: The initial neural network is trained using the proximal strategy optimization algorithm to obtain the tracking model and escape model, including: S11, randomly initializing the initial state information of the spacecraft during the binary star interaction within a preset target range, wherein the state information includes the position and velocity of the spacecraft in the CW coordinate system; S12, using the initial neural network of the spacecraft in the corresponding interactive mode to recursively process the initial state information to obtain the next state information of the spacecraft; wherein the recursive processing includes inputting the initial state information into the corresponding initial neural network, outputting the thruster start time of each of the spacecraft, and recursively calculating the start time using the relative orbital motion model to obtain the next state information of the spacecraft; S13, re-inputting the next moment state information into the initial neural network, and repeating the recursive process until an interaction round ends; S14, inputting the state information at each moment obtained by recursion into the reward function of the corresponding interaction mode, outputting and storing the reward value at each moment, and obtaining the total benefit of one interaction round of the spacecraft in the same interaction mode; S15. Update the network weights of the initial neural network according to the gradient descent method, and use the updated initial neural network to repeat the calculation process of steps S11-S14 until the total revenue of each round converges to a fixed value, and record the neural network at this time as the network model of the corresponding interaction mode.

6. The method according to claim 5, characterized in that The maneuver strategy selection model is trained in the following way: Establish the third reward function : in, and are all constants; is the position of the defense star relative to the tracking star; is the position of the defense star relative to the tracking star; and They are the safe defense zone radius of the escape star and the tracking star respectively; For the constant value of rewards and punishments; The escape model, the tracking model and the initial decision network are trained using a proximal strategy optimization algorithm, and the weights of the initial decision network are updated to obtain the maneuver strategy selection model.

7. The method according to claim 6, characterized in that The method of using the proximal strategy optimization algorithm to train the escape model, the tracking model and the initial decision network to obtain the maneuver strategy selection model includes: S21, randomly initializing the initial state information of three stars within a preset target range, and calculating the initial interaction data of the three stars; wherein the initial interaction data includes the initial position speed of the tracking star relative to the escaping star, the initial position speed of the tracking star relative to the defending star, and the initial position speed of the defending star relative to the escaping star; S22, inputting the initial interaction data into the initial decision network, and outputting a strategy selection result; wherein the strategy selection result includes a tracking strategy and an escape strategy; S23, performing the recursive processing on the tracking star according to the strategy selection result to obtain the state information of the tracking star at the next moment; wherein, when the strategy selection result is the tracking strategy, the position and velocity of the tracking star relative to the origin of the CW coordinate system are recursively deduced using the first tracking network model to obtain the state information of the tracking star at the next moment; when the strategy selection result is the escape strategy, the position and velocity of the tracking star relative to the origin of the CW coordinate system are recursively deduced using the second escape network model to obtain the state information of the tracking star at the next moment; S24, using the first escape network model to recursively deduce the position and velocity of the escape star relative to the origin of the CW coordinate system to obtain the state information of the escape star at the next moment; S25, using the second tracking network model to recursively deduce the position and velocity of the defense star relative to the origin of the CW coordinate system to obtain the next moment state information of the defense star; S26, according to the next moment state information of the tracking star, the escape star, and the defense star, calculating the position velocity of the tracking star relative to the escape star, the position velocity of the tracking star relative to the defense star, and the position velocity of the defense star relative to the escape star at the next moment; S27, inputting the state information at each moment obtained by recursion into the third reward function, outputting and storing the reward value at each moment, and obtaining the total benefits of one interaction round of the three spacecrafts; S28. Update the network weights of the initial decision network according to the gradient descent method, and use the updated initial neural network to repeat the calculation process of steps S21-S27 until the total revenue of each round converges to a fixed value, and record the decision network at this time as the maneuver strategy selection model.

8. A maneuver strategy hierarchical training device for multi-satellite orbit interaction, characterized in that: include: A modeling module is used to establish an orbital pursuit and escape interaction model of three spacecraft according to a preset CW coordinate system; wherein the three spacecraft include a tracking star, an escape star and a defense star, and the orbital pursuit and escape interaction model includes a relative orbital motion model, a thrust constraint model, an optimization index and a winning and losing condition; A first training module is used to determine the first interaction data and the first reward function when the tracking star and the escape star interact according to the orbit pursuit interaction model, and train the first initial neural network of the binary star to obtain the first tracking network model of the tracking star and the first escape network model of the escape star; The second training module is used to determine the second interaction data and the second reward function when the tracking star and the defending star interact according to the orbit pursuit interaction model, and train the second initial neural network of the binary star to obtain the second escape network model of the tracking star and the second tracking network model of the defending star; The third training module is used to determine the third interaction data and the third reward function during the three-star interaction according to the orbit pursuit interaction model, the first escape network model and the second tracking network model, and to train the initial decision network of the tracking star to obtain a maneuver strategy selection model for switching the tracking star maneuver strategy.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images