Double-optimization efficient learning training method for on-orbit interaction of double spacecrafts

Abstract

The invention discloses a double-optimization efficient learning training method for on-orbit interaction of double spacecrafts, which aims at a task entering a specified target area, fully considers time-space characteristics of the task, establishes relative orbit dynamics of a tracking star and a target star and a mathematical model of the target area, designs constraint conditions, and establishes win-lose conditions of the tracking star and the target star. In order to improve the training efficiency, a twice optimized efficient learning training method is designed, firstly, a tracking star and a target star respectively adopt a decision-making method based on zero-control miss distance to carry out pursuit interaction, track data in the interaction process of the two parties are collected and analyzed, and a neural network is adopted to fit the action of the tracking star for guiding subsequent training; according to the method, the defect that convergence is difficult due to the fact that training starts from zero can be overcome, the neural network obtained through training can output the optimal thrust of the tracking star, and the tracking star can enter a designated area at the designated moment.

Description

Technical Field

[0001] The present invention relates to the field of spacecraft orbit control technology, and in particular to a dual-optimization efficient learning and training method for dual spacecraft on-orbit interaction. Background Art

[0002] As the intelligence level of spacecraft increases, the uncertainty in space is increasing, and spacecraft are facing severe challenges. In a highly dynamic and highly unknown environment, spacecraft are often required to enter a designated area of ​​irregular shape near the target spacecraft under constraints to provide good orbital conditions for subsequent missions. At the same time, the target spacecraft may be maneuverable, and the two sides form a highly dynamic interactive situation. How to intelligently generate maneuvering strategies in the process of multi-spacecraft interaction, make full use of one's own advantages, and obtain maximum benefits is a major technical problem currently faced. Summary of the invention

[0003] In view of the above technical problems, the present invention provides a dual-optimization efficient learning and training method for dual spacecraft on-orbit interaction to solve the defect that the existing methods are difficult to deal with strong intelligent targets, so as to achieve approach to the target star.

[0004] The technical solution of the present invention is:

[0005] A dual-spacecraft on-orbit interaction dual-optimization efficient learning and training method, comprising:

[0006] 1) Establish orbital dynamics model of high-orbit spacecraft;

[0007] 2) Establish a mathematical model of the target area where the tracking satellite approaches;

[0008] 3) Establish terminal constraint models for tracking and target satellites;

[0009] 4) Design the structure and parameters of the neural network used for learning and training;

[0010] 5) Establish a neural network for tracking star reinforcement learning and a neural network for target star reinforcement learning, perform neural network pre-training, and obtain the optimized tracking star action network weights and target star action network weights;

[0011] 6) Using the optimized tracking star action network weights obtained in step 5) as the initial weights of the tracking star action network, and using the optimized target star action network weights obtained in step 5) as the initial weights of the target star action network; using reinforcement learning to perform secondary training on the neural network used for tracking star reinforcement learning and the neural network used for target star reinforcement learning, obtain the secondary optimized value network and action network of the tracking star and the target star, and complete the learning and training method.

[0012] Preferably, the mathematical model of the target area approached by the tracking star is described by a spherical cone model, where the vertex of the cone surface is the center of mass of the target star, and the vertex of the cone surface coincides with the center of the sphere; the radius of the sphere in the spherical cone is r lim , the semi-apex angle of the cone is θ lim , the generatrix length r of the cone in the spherical cone lim and half cone angle θ lim The size is determined by the input parameters of the interactive task, and the axis of the spherical cone points from the vertex of the cone surface to the center of the earth.

[0013] Preferably, the tracking star terminal constraint model is specifically:

[0014]

[0015] in, represents the terminal time t f , the position vector of the tracking star relative to the target star;

[0016] is the terminal time t input by the superior f The expected position vector of the tracking star relative to the target star;

[0017] θ(t f ) represents the terminal time t f , the angle between the line between the tracking star and the target star and the axis of the spherical cone;

[0018] The target satellite terminal constraint model is as follows:

[0019]

[0020] Among them, r com_lim is the distance threshold input by the superior task, indicating the maximum movement radius allowed by the target star;

[0021] r te (t f ) is the position vector of the target satellite relative to the center of the earth at the terminal moment;

[0022] r te0 is the position vector of the target satellite relative to the center of the earth at the initial moment.

[0023] Preferably, the method of obtaining the optimized tracking star motion network weights and target star motion network weights in step 5) is specifically as follows:

[0024] 51) Perform the two-star interaction process to obtain N motion trajectories of the tracking star and N motion trajectories of the target star;

[0025] 52) Establish a neural network for reinforcement learning of the tracking star, including: a value network and an action network; establish a neural network for reinforcement learning of the target star, including: a value network and an action network; optimize the weights of the tracking star action network and the weights of the target star action network respectively, and obtain the weights of the tracking star action network and the weights of the target star action network after optimization.

[0026] Preferably, the method of obtaining N motion trajectories of the tracking star and N motion trajectories of the target star in step 51) is specifically as follows:

[0027] 511) The tracking star adopts a tracking strategy based on zero-control miss amount, and the target star adopts an escape strategy based on zero-control miss amount to interact; at the current t-th time point, based on the orbital dynamics model established in step 1), the target star generates a thrust direction and magnitude acting on the target star, and the tracking star generates a thrust direction and magnitude acting on the tracking star;

[0028] 512) The target star and the tracking star move one step according to the orbital dynamics model established in step 1), and the time step between two adjacent time points ranges from 1s to 100s, and the relative position vector and relative velocity vector of the target star and the tracking star at time point t+1 are obtained;

[0029] 513) traverses each time point and repeats steps 511) to 512) until the terminal time t f , a round is over; a motion track of the tracking star and a motion track of the target star are obtained in the current round; the data in the motion track include: at each time point, the thrust direction and magnitude generated by the target star on the target star, the thrust direction and magnitude generated by the tracking star on the tracking star, and the relative position vector and relative velocity vector of the tracking star and the target star;

[0030] 514) randomly changes the initial position vector and initial velocity vector of the tracking star and the target star, repeats 511) to 513) N times, and collects N motion trajectories of the tracking star and N motion trajectories of the target star respectively; wherein N is a positive integer not less than 10000.

[0031] Preferably, in step 511), the method in which the target star generates the thrust direction and magnitude acting on the target star and the tracking star generates the thrust direction and magnitude acting on the tracking star is specifically as follows:

[0032] A) Using the orbital dynamics model and the thrust magnitude and direction of the two satellites obtained in the previous shot, the position and velocity of the tracking satellite and the target satellite in the current shot are generated;

[0033] B) using the current positions and velocities of the tracking star and the target star obtained in step A), and making the tracking star adopt a tracking strategy based on the zero-control miss amount to generate the thrust direction and magnitude of the current shot acting on the tracking star; making the target star adopt an escape strategy based on the zero-control miss amount to generate the thrust direction and magnitude of the current shot acting on the target star;

[0034] The tracking star adopts a tracking strategy based on zero control miss distance to generate the thrust direction and magnitude of the current shot acting on the tracking star, specifically:

[0035] a1) Receive the input from the upper level at the terminal time t f The expected position vector of the tracking star relative to the target star

[0036] a2) At the current time point t, the tracking satellite uses the sensors it carries to collect the current relative position vector and velocity vector of the tracking satellite and the target satellite, and estimates the relative position vector and velocity vector of the target satellite at the terminal time t based on the orbital dynamics model. f The position vector of the tracking star relative to the target star is taken as the position vector of the tracking star at the terminal time t f Tracking star position vector estimates;

[0037] a3) according to the expected position vector input by the upper level obtained in step a1) and the estimated value of the tracking star position vector obtained in step a2), based on the orbital dynamics model, the thrust direction and magnitude of the current tracking star at the tth time point, i.e., the current shot;

[0038] The target star adopts an escape strategy based on zero-control miss distance to generate the thrust direction and magnitude of the current shot acting on the target star, specifically:

[0039] b1) At any time point t, the target satellite uses the sensors it carries to collect the current relative position vector and velocity vector of the tracking satellite and the target satellite in real time, and estimates the relative position vector and velocity vector at the terminal time t based on the orbital dynamics model. f The position vector of the target star relative to the tracking star is taken as the position vector of the target star at the terminal time t f The estimated value of the target star position vector;

[0040] b2) The terminal time t obtained according to step b1) f The estimated value of the target satellite position vector determines the terminal time t f The expected position vector of the target star relative to the tracking star; the method for determining the expected position vector of the target star relative to the tracking star is specifically as follows:

[0041] In ||r te (t f )-r te0 || <r com_limSearch for a point within the defined target star's allowed motion range, so that the distance between this point and the estimated value of the target star position vector obtained in step b1) is the farthest, and take this point as the point at the terminal time t f The expected position vector of the target star relative to the tracking star;

[0042] b3) The terminal time t obtained according to step b2) f The expected position vector of the target star relative to the tracking star and the terminal time t obtained in step b1) f The estimated value of the target star position vector is calculated based on the orbital dynamics model to calculate the thrust direction and magnitude of the target star at the current t-th time point.

[0043] Preferably, the method of obtaining the optimized tracking star motion network weights in step 52) is as follows:

[0044] Using the N motion trajectories of the tracking star collected in step 51), the position vector and velocity vector of the tracking star relative to the target star at each time point in the N motion trajectories are used as the input of the tracking star action network, the thrust direction and magnitude of the tracking star at each time point in the N motion trajectories obtained in step a3) are used as the expected output, and the tracking star thrust output by the tracking star action network is used as the actual output; using the expected output and the actual output, the gradient descent method is used for training to obtain the optimized tracking star action network weights.

[0045] Preferably, the method of obtaining the target star action network weight after optimization in step 52) is as follows:

[0046] Using the N motion trajectories of the target star collected in step b51), the position vector and velocity vector of the target star relative to the tracking star at each time point in the N motion trajectories are used as the input of the target star motion network, the thrust direction and magnitude of the target star at each time point in the N motion trajectories obtained in step b3) are used as the expected output, and the target star thrust output by the target star motion network is used as the actual output; using the expected output and the actual output, the gradient descent method is adopted for training to obtain the target star motion network weight after one optimization.

[0047] Preferably, the method of step 6) obtaining the value network and action network after secondary optimization of the tracking star and the target star is specifically as follows:

[0048] 61) Establishing reward functions for the tracking star and the target star according to the terminal constraint model of the tracking star and the target star established in step 3);

[0049] 62) using the optimized tracking star motion network weights obtained in step 5) as the initial weights of the tracking star motion network, and using the optimized target star motion network weights obtained in step 5) as the initial weights of the target star motion network;

[0050] 63) Randomly set the initial position vector and velocity vector of the tracking star and the target star at the initial moment, and obtain the state value function and action value function corresponding to the tracking star and the state value function and action value function corresponding to the target star at each time point as interaction data;

[0051] 64) Repeat step 63) multiple times to obtain M bureau interaction data; wherein M is not less than 200;

[0052] 65) Using the M-game interaction data collected in step 64), a second training is performed on the neural network used for reinforcement learning of the tracking star and the neural network used for reinforcement learning of the target star to obtain the value network and action network of the tracking star and the target star after second optimization.

[0053] Preferably, in step 65), the method of performing secondary training on the neural network for tracking star reinforcement learning and the neural network for target star reinforcement learning is specifically as follows:

[0054] 651) Taking the state value function of the tracking star at each time point obtained in step 63) as the expected value, taking the state value output by the tracking star value network as the estimated value, taking the difference and then squaring them to obtain the optimization index function of the tracking star value network; the optimization index function of the tracking star action network is equal to the average value of the trajectory action advantage function of all time points in the M bureaus, wherein the trajectory action advantage function at each time point is obtained by subtracting the state value function from the action value function corresponding to the tracking star obtained in step 63);

[0055] The state value function of the target star at each time point obtained in step 63) is taken as the expected value, and the state value output by the target star value network is taken as the estimated value. The difference between the two is squared to obtain the optimization index function of the target star value network; the optimization index function of the target star action network is equal to the average value of the trajectory action advantage function of all time points in the M bureaus, wherein the trajectory action advantage function at each time point is obtained by subtracting the state value function from the action value function corresponding to the target star obtained in step 63);

[0056] 652) According to the optimization index function of the tracking star value network and the optimization index function of the tracking star action network, the optimization index function of the target star value network and the optimization index function of the target star action network obtained in 651), the gradient descent method is used to train the value network and the action network of the tracking star and the target star respectively, and the weights of the secondary optimized value network and the secondary optimized action network of the tracking star and the target star are obtained;

[0057] 653) Determine whether the weights of the secondary optimized value network and the secondary optimized action network of the tracking star and the target star converge at the same time. If not, return to step 63) and start a new cycle by changing the initial position vector and initial velocity vector of the tracking star and the target star at the randomly set initial moment; otherwise, stop training to obtain the optimal value network and action network of the tracking star and the target star.

[0058] Preferably, the method for determining whether the weights of the value network have converged in step 653) is specifically:

[0059] According to the value network weights output by two adjacent loops, the square difference of the value network weights is calculated twice. At the same time, according to the action network weights output by two adjacent loops, the square difference of the action network weights is calculated twice. If the square differences of the value network and action network weights are both less than 0.001, it is determined that the current value network weights and action network weights converge at the same time.

[0060] The advantages of the present invention compared with the prior art are:

[0061] 1) The present invention establishes a conical terminal constraint model. The terminal expected position vector in existing orbital interaction research is a fixed value, and the terminal constraint set is rarely considered. When the terminal constraint condition is a set, the flexibility of optimization is increased, the optimization space is expanded, and the design requirements of the decision algorithm are higher. The present invention guides the neural network training by designing a reasonable reward function, so that the control strategy output by the neural network can drive the tracking star to move toward the terminal constraint set;

[0062] 2) The present invention respectively designs an interaction method based on zero-control miss amount and a learning method based on reinforcement learning training. The control strategy based on zero-control miss amount provided by the present invention can effectively guide reinforcement learning training. Existing research generally trains intelligent agents from scratch. For complex tasks with strong uncertainty, the training is time-consuming and computationally inefficient. The present invention introduces the method of zero-control miss amount into the reinforcement learning framework. When facing a target with strong interaction ability, the initial weights of the network can be designed based on the previous interaction data with the target star with weaker ability, so that the tracking star has a certain interaction ability at the initial stage of training, thereby improving the network convergence speed vector. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of the change in relative distance between the tracking star and the target star during the training process provided for the invention.

[0064] Figure 2 A schematic diagram of the changes in the earnings of a tracking star during the training process provided for the invention.

[0065] Figure 3A schematic diagram of the change in relative distance between the tracking star and the target star corresponding to a round provided by the invention.

[0066] Figure 4 Schematic diagram of the change of the angle between the tracking star-target star connection line and the axis of the spherical top cone corresponding to a game in one embodiment of the present invention.

[0067] FIG. 5( a ) is a graph showing the change in the relative distance between the tracking star and the target star over time in one game according to an embodiment of the present invention.

[0068] FIG. 5( b ) is a graph showing how the angle between the tracking star-target star connection line and the target star top cone axis changes over time in one embodiment of the present invention.

[0069] FIG. 6( a ) is a graph showing the change in the relative distance between the tracking star and the target star over time in one game according to an embodiment of the present invention.

[0070] FIG. 6( b ) is a graph showing how the angle between the tracking star-target star connection line and the target star top cone axis changes over time in one embodiment of the present invention.

[0071] FIG. 7( a ) is a graph showing the change in the relative distance between the tracking star and the target star over time in one game according to an embodiment of the present invention.

[0072] FIG. 7( b ) is a graph showing how the angle between the tracking star-target star connection line and the target star top cone axis changes over time in one embodiment of the present invention.

[0073] Figure 8 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0074] In the present invention, for targets with strong interactive capabilities, if reinforcement learning is directly used for left-right interaction training, problems such as difficulty in learning convergence and low training efficiency will be faced. The present invention provides a dual-optimization efficient learning and training method for dual spacecraft on-orbit interaction. Aiming at the problem of entering the conical target area, the temporal and spatial characteristics of the task are fully considered, the relative orbital dynamics of the tracking star and the target star and the mathematical model of the target area are established, and a dual-optimization efficient neural network training method is designed to achieve entering the target area at a specified time. The specific steps are as follows:

[0075] 1) Establish the orbital dynamics model of high-orbit spacecraft, which is described by CW equation.

[0076] 2) Establish a mathematical model of the target area where the tracking satellite approaches. The mathematical model of the target area is described by a spherical cone model, where the vertex of the cone is the center of mass of the target star, and the vertex of the cone coincides with the center of the sphere; the radius of the sphere in the spherical cone is r lim , the semi-apex angle of the cone is θ lim, the generatrix length r of the cone in the spherical cone lim and half cone angle θ lim The size is determined by the input parameters of the interactive task, and the axis of the spherical cone points from the vertex of the cone surface to the center of the earth.

[0077] 3) Establish terminal constraint models for tracking satellites and target satellites.

[0078] For tracking stars, the goal is to enter the target area described in step 2) (i.e., the spherical cone with the center of mass of the target star as the vertex). Therefore, at the terminal time t f , the position vector of the tracking star relative to the target star must satisfy the tracking star terminal constraint model of formula (1).

[0079]

[0080] Wherein, formula (1) represents the terminal time t f , the tracking star is located in the target area described in step 2). is the terminal time t input by the superior f The expected position vector of the tracking star relative to the target star. θ(t f ) represents the terminal time t f , the angle between the line between the tracking star and the target star and the axis of the spherical cone.

[0081] For the target star, the goal is to avoid the tracking star through orbital maneuvers so that at the terminal time t f , the tracking star does not enter the target area described in step 2), and the target star terminal constraint model is shown in formula (2).

[0082]

[0083] Wherein, formula (2) represents the terminal time t f , the tracking star is outside the top cone of the target star established in step 3), and the position vector r of the target star relative to the center of the earth at the terminal moment te (t f ) and the position vector r of the target satellite relative to the center of the earth at the initial moment te0 The distance between them is less than the distance threshold r com_lim .

[0084] Distance threshold r com_lim Input from the parent task.

[0085] Terminal time t f , the angle θ between the line connecting the tracking star and the target star and the axis of the spherical cone (t f ), specifically:

[0086]

[0087] Among them, t f Indicates the terminal time of each interaction, which is given according to the requirements of the interaction task.

[0088] represents the terminal time t f , the position vector of the tracking star relative to the target star;

[0089] r te (t f ) represents the terminal time t f , the position vector of the target satellite relative to the center of the earth. The value of this variable can only be obtained at the terminal moment;

[0090] r ce (t f ) represents the terminal time t f , the position vector of the tracking star relative to the center of the earth, the value of this variable can only be obtained at the terminal moment;

[0091] r te0 Indicates the position vector of the target star relative to the center of the earth at the initial moment;

[0092] r com_lim The distance threshold represents the maximum movement radius allowed by the target star, and its value is determined according to the scope of the pursuit and escape interaction task. com_lim The constraints are input by the parent task.

[0093] 4) Design the structure and parameters of the neural network used for learning and training.

[0094] The structure of the neural network includes: network connection mode, number of network layers and number of neurons in each layer. The connection mode of the neural network is fully connected. The number of network layers and the number of neurons are related to the complexity of the interactive task. Considering the small number of satellites in 1v1 pursuit and escape interaction, the number of network layers is generally between 3 and 10, and the number of neurons in each layer is between 50 and 300.

[0095] The parameters of the neural network include: network activation function. The network activation function is any one of the tanh function, sigmoid function and Relu function.

[0096] 5) Pre-training of neural network to obtain the optimized tracking star motion network weights and target star motion network weights.

[0097] 51) Perform the two-star interaction process and collect data to obtain N motion trajectories of the tracking star and N motion trajectories of the target star; the details are as follows:

[0098] 511) The tracking star adopts a tracking strategy based on zero-control miss distance, and the target star adopts an escape strategy based on zero-control miss distance to interact. At the current t-th time point, based on the orbital dynamics model established in step 1), the target star generates the thrust direction and magnitude acting on the target star, and the tracking star generates the thrust direction and magnitude acting on the tracking star; specifically:

[0099] A) Using the orbital dynamics model and the thrust magnitude and direction of the two satellites obtained in the previous shot, the position and velocity of the tracking satellite and the target satellite in the current shot are generated;

[0100] B) Using the current positions and velocities of the tracking star and the target star obtained in step A), and making the target star adopt an escape strategy based on the zero-control miss amount, the thrust direction and magnitude of the current shot acting on the target star are generated; making the tracking star adopt a tracking strategy based on the zero-control miss amount, and generating the thrust direction and magnitude of the current shot acting on the tracking star.

[0101] 512) The target star and the tracking star move one step according to the orbital dynamics model established in step 1), and the time step between two adjacent time points ranges from 1s to 100s, and the relative position vector and relative velocity vector of the target star and the tracking star at time point t+1 are obtained.

[0102] 513) traverses each time point and repeats steps 511) to 512) until the terminal time t f , a round is over. Obtain a motion track of the tracking star and a motion track of the target star in the current round. The data in the motion track includes: at each time point, the thrust direction and magnitude generated by the target star on the target star, the thrust direction and magnitude generated by the tracking star on the tracking star, and the relative position vector and relative velocity vector of the tracking star and the target star.

[0103] 514) randomly changes the initial position vector and initial velocity vector of the tracking star and the target star, repeats 511) to 513) N times, and collects N motion trajectories of the tracking star and N motion trajectories of the target star respectively. The distance of the tracking star relative to the target star at the initial moment is less than 300 km, and the modulus of the velocity vector of the tracking star relative to the target star at the initial moment is less than 30 m / s. N is a positive integer not less than 10,000.

[0104] The tracking strategy based on the zero-control miss amount described in step 511) generates the thrust direction and magnitude of the current shot acting on the tracking star. The specific steps are:

[0105] a1) Receive the input from the upper level at the terminal time t f The expected position vector of the tracking star relative to the target star It is expressed by formula 1), which means that at the terminal moment, the tracking star is located in the spherical cone established in step 3) with the target star as the vertex. The expected position vector in the Nth interaction process is remain unchanged;

[0106] a2) At the current time point t, the tracking satellite uses the sensors it carries to collect the current relative position vector and velocity vector of the tracking satellite and the target satellite, and estimates the relative position vector and velocity vector of the target satellite at the terminal time t based on the orbital dynamics model (i.e., CW equation). f The position vector of the tracking star relative to the target star is taken as the position vector of the tracking star at the terminal time t f Tracking star position vector estimates;

[0107] a3) According to the expected position vector of the superior input obtained in step a1) and the estimated value of the tracking star position vector obtained in step a2), the thrust direction and magnitude of the tracking star at the current t-th time point (i.e., the current shot) are calculated based on the orbital dynamics model (i.e., the CW equation).

[0108] The escape strategy based on the zero-control miss distance described in step 511) generates the thrust direction and magnitude of the current shot acting on the target star, and the specific steps are:

[0109] b1) At any time point t, the target satellite uses the sensors it carries to collect the current relative position vector and velocity vector of the tracking satellite and the target satellite in real time, and estimates the relative position vector and velocity vector at the terminal time t based on the orbital dynamics model (i.e., CW equation). f The position vector of the target star relative to the tracking star is taken as the position vector of the target star at the terminal time t f The estimated value of the target star position vector;

[0110] b2) The terminal time t obtained according to step b1) f The estimated value of the target satellite position vector determines the terminal time t f The expected position vector of the target star relative to the tracking star; the method for determining the expected position vector of the target star relative to the tracking star is specifically as follows:

[0111] In ||r te (t f )-r te0 || <r com_lim Search for a point within the defined target star's allowed motion range, so that the distance between this point and the estimated value of the target star position vector obtained in step b1) is the farthest, and take this point as the point at the terminal time t f The expected position vector of the target star relative to the tracking star;

[0112] b3) The terminal time t obtained according to step b2) f The expected position vector of the target star relative to the tracking star and the terminal time t obtained in step b1) fThe estimated value of the target star position vector is used to calculate the thrust direction and magnitude of the target star at the current t-th time point based on the orbital dynamics model (i.e., the CW equation).

[0113] 52) Establish a neural network for reinforcement learning of the tracking star, including: a value network and an action network. Establish a neural network for reinforcement learning of the target star, including: a value network and an action network; optimize the weights of the tracking star's action network and the target star's action network respectively, and obtain the weights of the tracking star's action network and the target star's action network after optimization; among which, the initial value of the tracking star's value network weight is randomly initialized by the software Tensorflow; the initial value of the target star's value network weight is randomly initialized by the software Tensorflow.

[0114] Step 52) A method for obtaining the weights of the tracking star motion network after an optimization is as follows:

[0115] Using the N motion trajectories of the tracking star collected in step 51), the position vector and velocity vector of the tracking star relative to the target star at each time point in the N motion trajectories are used as the input of the tracking star motion network, the thrust direction and magnitude of the tracking star at each time point in the N motion trajectories obtained in step a3) are used as the expected output, and the tracking star thrust output by the tracking star motion network is used as the actual output; using the expected output and the actual output, the gradient descent method is used for training to obtain the weights of the tracking star motion network after one optimization;

[0116] Step 52) A method for obtaining the weights of the target star action network after an optimization is as follows:

[0117] Using the N motion trajectories of the target star collected in step b51), the position vector and velocity vector of the target star relative to the tracking star at each time point in the N motion trajectories are used as inputs of the target star motion network, the thrust direction and magnitude of the target star at each time point in the N motion trajectories obtained in step b3) are used as expected outputs, and the target star thrust output by the target star motion network is used as actual output; using the expected output and actual output, a gradient descent method is used for training to obtain the weights of the target star motion network after one optimization;

[0118] 6) Using the optimized tracking star action network weights obtained in step 5) as the initial weights of the tracking star action network, and using the optimized target star action network weights obtained in step 5) as the initial weights of the target star action network; using reinforcement learning to perform secondary training on the neural network used for tracking star reinforcement learning and the neural network used for target star reinforcement learning, obtain the secondary optimized value network and action network of the tracking star and the target star, and complete the learning and training method.

[0119] 61) Design reward function

[0120] According to formula (1) and formula (2) in the terminal constraint model of the tracking star and the target star established in step 3), the reward function of the tracking star and the target star is established.

[0121] The star tracking reward function is:

[0122]

[0123] The reward function of the target star is designed as

[0124]

[0125] Among them, λ1,λ2,λ3,λ4,λ5,λ6 are weight factors, and their value range is (0,100]; δ,δ2,δ4 are constant reward values, and their value range is (0,1000];

[0126] r ba represents the position vector of the tracking star relative to the target star, θ e is the vector r ba The angle between the target star antenna and the target star antenna, v ba =v b -v a is the velocity vector of the tracking star relative to the target star, v b is the velocity vector of the tracking star, v a is the velocity vector of the target star;

[0127] θ v is the vector v ba With vector r ab The angle between index is the penalty term related to the target star motion range, and its value range is (0, 100]. The larger the value, the more conducive it is to keep the target star within the allowable motion range defined by formula (2) || r te (t f )-r te0 || <r com_lim δ3 is the bonus item of the target star. If the first condition in formula (2) If it satisfies the above condition, a positive number δ3 is given, and in this embodiment, δ3=10.

[0128] 62) Using the optimized tracking star motion network weights obtained in step 5) as the initial weights of the tracking star motion network, and using the optimized target star motion network weights obtained in step 5) as the initial weights of the target star motion network.

[0129] 63) Randomly set the initial position vector and velocity vector of the tracking star and the target star at the initial moment, collect data, and obtain the state value function and action value function corresponding to the tracking star and the state value function and action value function corresponding to the target star at each time point as interaction data. The specific process is as follows:

[0130] 631) Randomly set the initial position vector and velocity vector of the tracking star and the target star at the initial moment, and ensure that at the initial moment, the distance of the tracking star relative to the target star is less than 300km, and the velocity vector of the tracking star relative to the target star is less than 30m / s.

[0131] 632) At the current t-th time point, the thrust of each star output by the action network of the tracking star and the target star is used to determine the position vector and velocity vector of the tracking star and the target star at the t+1 time point; at the same time, according to the reward function model of formula (3) and formula (4), the reward R of the tracking star at the current t-th time point is calculated respectively. red and the target star's reward R blue ;

[0132] 633) stores the relative position vector, relative velocity vector, self-output thrust and respective rewards (R) of the tracking star and the target star at each time point. red , R blue ).

[0133] 634) traverses each time point and repeats the above process from 632) to 633) until the terminal time t f ,, the game is over.

[0134] 635) After each round, the reward R of the tracking star at each time point obtained in step 634) is calculated. red and the target star's reward R blue At the same time, according to the reward function model of formula (3) and formula (4), reversely calculate the state value function and action value function corresponding to the tracking star and the state value function and action value function corresponding to the target star at each time point in the game, and store them.

[0135] 64) Repeat step 63) to obtain M rounds of interaction data. The value of M depends on the network training requirements. In the embodiment of the present invention, M is not less than 200. The "state value function and action value function corresponding to each moment" of each round has 4 value functions corresponding to each point, and each round has multiple points, with a total of M rounds.

[0136] 65) Network training: Using the M-game interaction data collected in step 64), the interaction data includes: at each moment, the state value function and action value function corresponding to the tracking star and the target star, and performing secondary training on the neural network used for reinforcement learning of the tracking star and the neural network used for reinforcement learning of the target star to obtain the value network and action network of the tracking star and the target star after secondary optimization.

[0137] Specifically:

[0138] 651) Taking the state value function of the tracking star at each time point obtained in step 63) as the expected value, taking the state value output by the tracking star value network as the estimated value, taking the difference and then squaring them to obtain the optimization index function of the tracking star value network; the optimization index function of the tracking star action network is equal to the average value of the trajectory action advantage function of all time points in the M bureaus, wherein the trajectory action advantage function at each time point is obtained by subtracting the state value function from the action value function corresponding to the tracking star obtained in step 63);

[0139] Similarly, at each time point obtained in step 63), the state value function of the target star is taken as the expected value, and the state value output by the target star value network is taken as the estimated value. The difference between the two is squared to obtain the optimization index function of the target star value network; the optimization index function of the target star action network is equal to the average value of the trajectory action advantage function of all time points in the M games, where the trajectory action advantage function at each time point is obtained by subtracting the state value function from the action value function corresponding to the target star obtained in step 63).

[0140] 652) According to the optimization index function of the tracking star value network and the optimization index function of the tracking star action network, the optimization index function of the target star value network and the optimization index function of the target star action network obtained in 651), the gradient descent method is used to train the value network and action network of the tracking star and the target star respectively, and the weights of the value network and the action network of the tracking star and the target star after reinforcement learning (i.e., quadratic optimization) and the weights of the action network after reinforcement learning (i.e., quadratic optimization) are obtained.

[0141] 653) Determine whether the weights of the secondary optimized value network and the secondary optimized action network of the tracking star and the target star converge at the same time. If not, return to step 63) and start a new cycle by changing the initial position vector and velocity vector of the tracking star and the target star at the random initial moment; otherwise, stop training to obtain the optimal value network and action network of the tracking star and the target star.

[0142] The method for judging whether the weights of the value network converge is as follows:

[0143] According to the value network weights output by two adjacent loops, the square difference of the value network weights is calculated twice, and according to the action network weights output by two adjacent loops, the square difference of the action network weights is calculated twice. If the square differences of the value network and action network weights are both less than the specified value (0.001), it is determined that the current value network weights and action network weights converge at the same time.

[0144] Example

[0145] In this embodiment, if Figure 8 As shown in FIG. 1 , the specific process of the knowledge-guided learning decision-making method for entering a spherical cone with a target star as the vertex is as follows:

[0146] 1) Establishing the orbital dynamics equation of the high-orbit spacecraft, wherein the orbital dynamics is described by a two-body model. Since the present invention considers close-range interaction, the CW equation is used to describe the relative motion of the tracking star and the target star;

[0147]

[0148] 2) Establish the physical and mathematical representation of the target expected entry area model, terminal constraints, optimization indicators and reward functions. The specific steps are as follows:

[0149] 21) Establish the physical and mathematical model of the target star expected area. With the target star as the vertex, construct a conic surface, where the generatrix length of the conic surface is r lim =50km, half cone angle is θ lim =30°;

[0150] 22) Establish a mathematical model of terminal constraints. For the tracking star, it needs to approach the target star conical range described in step 21), and its terminal constraint is shown in formula (6). For the target star, it needs to avoid the tracking star through orbital maneuvers so that the tracking star cannot enter the conical area range described in step 21) at the terminal moment, and its terminal constraint is shown in formula (7).

[0151]

[0152] 23) Establish a mathematical model for the optimization index (reward function). For tracking stars, the optimization index (reward function) is designed as

[0153]

[0154] The optimization index (reward function) of the target star is designed as

[0155]

[0156] Among them, r ba represents the position vector of the tracking star relative to the target star, θe is the vector r ba The angle between the target star and the earth, v ba =v b -v a is the velocity vector of the tracking star relative to the target star, θ v v ba With r ab Vector angle. δ index is the penalty term related to the target star's motion range. In order to maintain the original business, the target star needs to remain within a certain orbital range. Here, the target star needs to remain within its initial position vector r ae0 R com_lim If it exceeds the range, a negative reward is given.

[0157] 3) Establish a simulation model, build a simulation interactive environment in gym for tracking satellites to enter the designated area of ​​the target satellite, and develop visualization software;

[0158] 4) The tracking satellite adopts a tracking strategy based on zero-control miss distance, and the target satellite adopts an escape strategy based on zero-control miss distance to interact. At the current t-th time point, the target satellite and the tracking satellite generate thrust direction and magnitude based on the zero-control miss distance strategy respectively. The thrust acts on the tracking satellite and the target satellite respectively. The two satellites move one step according to the orbital dynamics model established in step 1). The step length of the two time points is in the range of 10s. The position vector and velocity vector of the target satellite and the tracking satellite at t+1 time point are obtained, so as to obtain and store the relative position vector, relative velocity vector and thrust direction of the two satellites at each time point. Traverse each time point and repeat the above process until the terminal time t f , a round is over. Get the current round, the movement trajectory of the tracking star and the movement trajectory of the target star.

[0159] Randomly change the initial position vector and velocity vector of the tracking star and the target star, and ensure that the distance between the tracking star and the target star is less than 300km at the initial moment, and the velocity vector of the tracking star relative to the target star is less than 30m / s at the initial moment. Repeat the above interactive process 10,000 times, collect 10,000 motion trajectories of the tracking star and 10,000 motion trajectories of the target star respectively, use them as training samples, use deep learning algorithm, and obtain the action network weights of the tracking star through gradient descent method.

[0160] 4) Design an empowerment algorithm based on reinforcement learning, conduct empowerment training in the gym simulation environment, obtain the action network and value network of the tracking star and the target star, so that the tracking star and the target star have strong interaction capabilities. The specific method is:

[0161] 41) Establish a training framework and design a training algorithm;

[0162] 42) Design the neural network structure and parameters, including network connection mode, number of network layers, number of neurons in each layer, and network activation function; see Table 1 and Table 2 for parameter settings. The action network weights of the tracking star and the target star obtained in step 3) are used as the initial values ​​of the reinforcement learning training of the two-star action network.

[0163] 43) Design training parameters, including learning rate, weighting factor, discount factor, termination condition, simulation step size and total number of steps per round, and data batch length; see Table 1 and Table 2 for parameter settings.

[0164] Table 1 Network structure and parameters of tracking satellites

[0165]

[0166] Table 2 Network structure and parameters of target satellite

[0167]

[0168]

[0169] The expression of the ReLU function is:

[0170] y(x)=max(0,x)

[0171] The expression of Tanh function is:

[0172]

[0173] 44) Randomly initialize the position vector and velocity vector of the target star and the tracking star, input the current state of the two stars into the neural network, the neural network outputs the control strategy, the two stars move forward according to the orbital dynamics, and calculate their respective reward functions through formula (8) and formula (9). The specific steps are as follows:

[0174] 441) normalizing the relative position vector and velocity vector of the tracking star and the target star;

[0175] 442) The action space is discretized into 20 values, and the thruster startup strategy for each step is used as the output of the action network;

[0176] 443) The action network calculates the three-axis control strategy based on the current network weights and network inputs, outputs the probability value of the discretized action, and uses the ε-greedy strategy to select the action, as shown in formula (10), to ensure that the probability of each action being selected is not zero, ε = 0.1. Finally, the start-up time of each thruster is obtained.

[0177]

[0178] 45) Repeat step 44) and store the position vector, velocity vector and reward of the tracking star and the target star at each moment until the data volume meets the requirements of network training;

[0179] 46) The gradient descent method is used to update the network weights. Based on the stored data, the action value and state value function corresponding to each moment are reversely calculated. This is used as the expected indicator of supervised learning, which is subtracted from the output result of the neural network, and the partial derivative of the network weight is taken to obtain the direction of weight update. After multiplying by the step size, the update amount of the weight at each step is obtained.

[0180] 47) Repeat steps 45)-46) until the network weights converge, that is, the variance of the network weights of two adjacent games is less than the specified value, and the benefits converge to the optimal value, such as Figure 2 shown.

[0181] The embodiments corresponding to different weight factor values ​​in the tracking star reward function in formula (3) are as follows: The larger the relative distance weights λ1, λ3 and the relative angle weights λ2, λ4, the faster the tracking star will enter the target area of ​​the target star. If the relative distance weights λ1, λ3 increase while the relative angle weights λ2, λ4 remain unchanged, the tracking star can quickly approach the target star, but the relative angle does not satisfy the conical area constraint.

[0182] like Figure 1 As shown in , after 1000 rounds of training, the tracking star can approach the cone area of ​​the target star, and the benefit of the tracking star gradually converges to the maximum value, as shown in Figure 2 After the training is completed, the local shooting verification is carried out and one of the games is selected for display. The simulation results are shown in Figure 3 and Figure 4 As shown, after 2000s, the tracking star enters the cone area of ​​the target star, satisfying the relative distance and angle constraints.

[0183] Figures 5(a), 6(a), and 7(a) show the curves of the relative distance between the tracking star and the target star changing over time in a game under different reward function weight factors (corresponding to step 61 above). Figures 5(b), 6(b), and 7(b) show the curves of the angle between the line between the tracking star and the target star and the axis of the cone at the top of the target star changing over time in a game under different reward function weight factors.

[0184] In Figure 5(a)(b),

[0185] In Figure 6(a)(b),

[0186] In Figure 7(a)(b),

[0187] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solution of the present invention by using the above disclosed methods and technical contents without departing from the spirit and scope of the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall belong to the protection scope of the technical solution of the present invention. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0188] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A dual-spacecraft on-orbit interactive dual-optimization efficient learning and training method, characterized in that: include: 1) Establish orbital dynamics model of high-orbit spacecraft; 2) Establish a mathematical model of the target area where the tracking satellite approaches; 3) Establish terminal constraint models for tracking and target satellites; 4) Design the structure and parameters of the neural network used for learning and training; 5) establishing a neural network for reinforcement learning of the tracking star and a neural network for reinforcement learning of the target star, performing neural network pre-training according to the orbital dynamics model of the high-orbit spacecraft described in step 1) and the terminal constraint models of the tracking star and the target star described in step 2), and obtaining the weights of the tracking star motion network and the target star motion network after one optimization; 6) Using the once optimized tracking star action network weights obtained in step 5) as the initial weights of the tracking star action network, and using the once optimized target star action network weights obtained in step 5) as the initial weights of the target star action network; according to the terminal constraint model of the tracking star and the target star described in step 3), using reinforcement learning to perform secondary training on the neural network used for reinforcement learning of the tracking star and the neural network used for reinforcement learning of the target star, obtaining the twice optimized value network and action network of the tracking star and the target star, and completing the learning and training method.

2. According to claim 1, a dual-spacecraft on-orbit interactive dual optimization efficient learning and training method is characterized in that: The mathematical model of the target area where the tracking star approaches is described by a spherical cone model. The vertex of the cone is the center of mass of the target star, and the vertex of the cone coincides with the center of the sphere. The radius of the sphere in the spherical cone is r. lim , the semi-apex angle of the cone is θ lim , the generatrix length r of the cone in the spherical cone lim and half cone angle θ lim The size is determined by the input parameters of the interactive task, and the axis of the spherical cone points from the vertex of the cone surface to the center of the earth.

3. According to claim 2, a dual-spacecraft on-orbit interactive dual optimization efficient learning and training method is characterized in that: The tracking star terminal constraint model is as follows: in, represents the terminal time t f , the position vector of the tracking star relative to the target star; is the terminal time t input by the superior f The expected position vector of the tracking star relative to the target star; θ(t f ) represents the terminal time t f , the angle between the line between the tracking star and the target star and the axis of the spherical cone; The target satellite terminal constraint model is as follows: Among them, r com_lim is the distance threshold input by the superior task, indicating the maximum movement radius allowed by the target star; r te (t f ) is the position vector of the target satellite relative to the center of the earth at the terminal moment; r te0 is the position vector of the target satellite relative to the center of the earth at the initial moment.

4. The dual optimization efficient learning and training method for dual spacecraft on-orbit interaction according to claim 3 is characterized in that: Step 5) A method for obtaining the optimized tracking star motion network weights and target star motion network weights is as follows: 51) Perform the two-star interaction process to obtain N motion trajectories of the tracking star and N motion trajectories of the target star; 52) Establish a neural network for reinforcement learning of the tracking star, including: a value network and an action network; establish a neural network for reinforcement learning of the target star, including: a value network and an action network; optimize the weights of the tracking star action network and the weights of the target star action network respectively, and obtain the weights of the tracking star action network and the weights of the target star action network after optimization.

5. The dual optimization efficient learning and training method for dual spacecraft on-orbit interaction according to claim 4 is characterized in that: Step 51) A method for obtaining N motion trajectories of the tracking star and N motion trajectories of the target star is specifically as follows: 511) The tracking star adopts a tracking strategy based on zero-control miss amount, and the target star adopts an escape strategy based on zero-control miss amount to interact; at the current t-th time point, based on the orbital dynamics model established in step 1), the target star generates a thrust direction and magnitude acting on the target star, and the tracking star generates a thrust direction and magnitude acting on the tracking star; 512) The target star and the tracking star move one step according to the orbital dynamics model established in step 1), and the time step between two adjacent time points ranges from 1s to 100s, and the relative position vector and relative velocity vector of the target star and the tracking star at time point t+1 are obtained; 513) traverses each time point and repeats steps 511) to 512) until the terminal time t f , a round is over; obtain a motion track of the tracking star and a motion track of the target star in the current round; The data in the motion trajectory include: at each time point, the direction and magnitude of the thrust generated by the target star on the target star, the direction and magnitude of the thrust generated by the tracking star on the tracking star, and the relative position vector and relative velocity vector of the tracking star and the target star; 514) randomly changes the initial position vector and initial velocity vector of the tracking star and the target star, repeats 511) to 513) N times, and collects N motion trajectories of the tracking star and N motion trajectories of the target star respectively; wherein N is a positive integer not less than 10000.

6. The dual-spacecraft on-orbit interactive dual-optimization efficient learning and training method according to claim 5 is characterized in that: In step 511), the target star generates the thrust direction and magnitude acting on the target star, and the tracking star generates the thrust direction and magnitude acting on the tracking star, specifically: A) Using the orbital dynamics model and the thrust magnitude and direction of the two satellites obtained in the previous shot, the position and velocity of the tracking satellite and the target satellite in the current shot are generated; B) using the current positions and velocities of the tracking star and the target star obtained in step A), and making the tracking star adopt a tracking strategy based on the zero-control miss amount to generate the thrust direction and magnitude of the current shot acting on the tracking star; making the target star adopt an escape strategy based on the zero-control miss amount to generate the thrust direction and magnitude of the current shot acting on the target star; The tracking star adopts a tracking strategy based on zero control miss distance to generate the thrust direction and magnitude of the current shot acting on the tracking star, specifically: a1) Receive the input from the upper level at the terminal time t f The expected position vector of the tracking star relative to the target star a2) At the current time point t, the tracking satellite uses the sensors it carries to collect the current relative position vector and velocity vector of the tracking satellite and the target satellite, and estimates the relative position vector and velocity vector of the target satellite at the terminal time t based on the orbital dynamics model. f The position vector of the tracking star relative to the target star is taken as the position vector of the tracking star at the terminal time t f Tracking star position vector estimates; a3) according to the expected position vector input by the upper level obtained in step a1) and the estimated value of the tracking star position vector obtained in step a2), based on the orbital dynamics model, the thrust direction and magnitude of the current tracking star at the tth time point, i.e., the current shot; The target star adopts an escape strategy based on zero-control miss distance to generate the thrust direction and magnitude of the current shot acting on the target star, specifically: b1) At any time point t, the target satellite uses the sensors it carries to collect the current relative position vector and velocity vector of the tracking satellite and the target satellite in real time, and estimates the relative position vector and velocity vector at the terminal time t based on the orbital dynamics model. f The position vector of the target star relative to the tracking star is taken as the position vector of the target star at the terminal time t f The estimated value of the target star position vector; b2) The terminal time t obtained according to step b1) f The estimated value of the target satellite position vector determines the terminal time t f The expected position vector of the target star relative to the tracking star; the method for determining the expected position vector of the target star relative to the tracking star is specifically as follows: In ||r te (t f )-r te0 || <r com_lim Search for a point within the defined target star's allowed motion range, so that the distance between this point and the estimated value of the target star position vector obtained in step b1) is the farthest, and take this point as the point at the terminal time t f The expected position vector of the target star relative to the tracking star; b3) The terminal time t obtained according to step b2) f The expected position vector of the target star relative to the tracking star and the terminal time t obtained in step b1) f The estimated value of the target star position vector is calculated based on the orbital dynamics model to calculate the thrust direction and magnitude of the target star at the current t-th time point.

7. The dual-spacecraft on-orbit interactive dual-optimization efficient learning and training method according to claim 6 is characterized in that: Step 52) A method for obtaining the weights of the tracking star motion network after an optimization is as follows: Using the N motion trajectories of the tracking star collected in step 51), the position vector and velocity vector of the tracking star relative to the target star at each time point in the N motion trajectories are used as the input of the tracking star action network, the thrust direction and magnitude of the tracking star at each time point in the N motion trajectories obtained in step a3) are used as the expected output, and the tracking star thrust output by the tracking star action network is used as the actual output; using the expected output and the actual output, the gradient descent method is used for training to obtain the optimized tracking star action network weights.

8. The dual-spacecraft on-orbit interactive dual-optimization efficient learning and training method according to claim 7 is characterized in that: Step 52) A method for obtaining the weights of the target star action network after an optimization is as follows: Using the N motion trajectories of the target star collected in step b51), the position vector and velocity vector of the target star relative to the tracking star at each time point in the N motion trajectories are used as the input of the target star motion network, the thrust direction and magnitude of the target star at each time point in the N motion trajectories obtained in step b3) are used as the expected output, and the target star thrust output by the target star motion network is used as the actual output; using the expected output and the actual output, the gradient descent method is adopted for training to obtain the target star motion network weight after one optimization.

9. A dual-spacecraft on-orbit interactive dual-optimization efficient learning and training method according to any one of claims 4 to 8, characterized in that: Step 6) A method for obtaining the value network and action network after secondary optimization of the tracking star and the target star is as follows: 61) Establishing reward functions for the tracking star and the target star according to the terminal constraint model of the tracking star and the target star established in step 3); 62) using the optimized tracking star motion network weights obtained in step 5) as the initial weights of the tracking star motion network, and using the optimized target star motion network weights obtained in step 5) as the initial weights of the target star motion network; 63) Randomly set the initial position vector and velocity vector of the tracking star and the target star at the initial moment, and obtain the state value function and action value function corresponding to the tracking star and the state value function and action value function corresponding to the target star at each time point as interaction data; 64) Repeat step 63) multiple times to obtain M bureau interaction data; wherein M is not less than 200; 65) Using the M-game interaction data collected in step 64), a second training is performed on the neural network used for reinforcement learning of the tracking star and the neural network used for reinforcement learning of the target star to obtain the value network and action network of the tracking star and the target star after second optimization.

10. The dual-spacecraft on-orbit interactive dual-optimization efficient learning and training method according to claim 9, characterized in that: Step 65) A method for performing secondary training on the neural network used for tracking star reinforcement learning and the neural network used for target star reinforcement learning is specifically as follows: 651) Taking the state value function of the tracking star at each time point obtained in step 63) as the expected value, taking the state value output by the tracking star value network as the estimated value, taking the difference and then squaring them to obtain the optimization index function of the tracking star value network; the optimization index function of the tracking star action network is equal to the average value of the trajectory action advantage function of all time points in the M bureaus, wherein the trajectory action advantage function at each time point is obtained by subtracting the state value function from the action value function corresponding to the tracking star obtained in step 63); The state value function of the target star at each time point obtained in step 63) is taken as the expected value, and the state value output by the target star value network is taken as the estimated value. The difference between the two is squared to obtain the optimization index function of the target star value network; the optimization index function of the target star action network is equal to the average value of the trajectory action advantage function of all time points in the M bureaus, wherein the trajectory action advantage function at each time point is obtained by subtracting the state value function from the action value function corresponding to the target star obtained in step 63); 652) According to the optimization index function of the tracking star value network and the optimization index function of the tracking star action network, the optimization index function of the target star value network and the optimization index function of the target star action network obtained in 651), the gradient descent method is used to train the value network and the action network of the tracking star and the target star respectively, and the weights of the secondary optimized value network and the secondary optimized action network of the tracking star and the target star are obtained; 653) Determine whether the weights of the secondary optimized value network and the secondary optimized action network of the tracking star and the target star converge at the same time. If not, return to step 63) and start a new cycle by changing the initial position vector and initial velocity vector of the tracking star and the target star at the randomly set initial moment; otherwise, stop training to obtain the optimal value network and action network of the tracking star and the target star.

11. The dual-spacecraft on-orbit interactive dual-optimization efficient learning and training method according to claim 10, characterized in that: The method for determining whether the weights of the value network have converged in step 653) is specifically as follows: According to the value network weights output by two adjacent loops, the square difference of the value network weights is calculated twice. At the same time, according to the action network weights output by two adjacent loops, the square difference of the action network weights is calculated twice. If the square differences of the value network and action network weights are both less than 0.001, it is determined that the current value network weights and action network weights converge at the same time.

Images