Dynamic threat-based lgvf-apf-sac spacecraft path planning method, storage medium and electronic device

Abstract

The application discloses a dynamic threat-based LGVF-APF-SAC spacecraft path planning method, a storage medium and an electronic device, and specifically comprises the following steps: firstly, a real-time risk function is established according to a dynamic threat (a radar detection exposure risk), and a warning area is further generated according to the risk function (radar radiation). Based on the warning area, an improved dynamic time-varying LGVF is used to construct a dynamic time-varying LGVF-based guidance potential field, which guides the spacecraft to avoid the warning area and reach a target point. Then, an end-to-end reinforcement learning framework is developed. The framework selects a maximum entropy RL (SAC) algorithm, integrates the constructed guidance potential field into the learning process in the form of a reward function, and evaluates and optimizes the decision action of the spacecraft. The algorithm disclosed by the application can guarantee to meet the spacecraft dynamics overload condition, and improve the dynamic adaptability and reliability of path planning.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of spacecraft guidance, navigation and control (GNC), and in particular to a spacecraft path planning method based on dynamic threats, an LGVF-APF-SAC spacecraft path planning method, a storage medium and an electronic device. BACKGROUND

[0002] At present, in recent years, with the development of automatic control technology, various spacecrafts have been widely applied to industrial, military and medical fields. At the same time, the task scene of the spacecraft becomes more and more complex and variable, and many disturbances, countermeasures and time-varying characteristics are introduced in the actual application, which causes new challenges to the guidance, navigation and control (GNC) of the spacecraft, and attracts wide attention of the academic and industrial circles.

[0003] Path planning, as a basic topic of GNC, has become a research hotspot in the related field. The path planning technology requires the spacecraft to find a collision-free path to safely and efficiently reach the destination. The previous path planning methods can be divided into traditional methods, heuristic algorithms and learning-based methods. The traditional methods have the disadvantages of long search time, high calculation consumption and poor real-time performance. Therefore, they are usually used for global path planning and are difficult to adapt to dynamic environments. The main disadvantage of the heuristic algorithm is that it has a calculation cost and a memory requirement and depends on heuristic or probabilistic rules, which do not always guarantee an optimal solution and can also cause the problem of slow convergence speed. The learning-based method fails to utilize the optimal solution of the control theory in the solving process, does not consider the spacecraft dynamics constraints, and has insufficient generalization modeling for dynamic threat environments, which inevitably brings the problem of unreliable local optimality of the planned path. Therefore, it is necessary to propose a reliable path planning method with dynamic generalization capability. SUMMARY

[0004] The application aims to provide a spacecraft path planning method based on dynamic threats, a storage medium and an electronic device, which can solve two major problems of spacecraft path planning in a dynamic unknown environment. One is that the guidance potential field as an evaluation function compresses the solution space, solving the problem that the deep reinforcement learning network is difficult to converge due to strong environmental dynamics. The other is that the stable and smooth characteristics of LGVF are used to consider the dynamics constraints of the spacecraft while outputting safe yaw instructions, generating a smooth and efficient path.

[0005] The technical scheme adopted by the application is as follows:

[0006] The spacecraft path planning method based on dynamic threats comprises the following steps:

[0007] Step 1: establishing a spacecraft path planning scene model, and the specific process is as follows:

[0008] The present application relates to the spacecraft path planning task in dynamic unknown environment, described as the process of spacecraft avoiding dynamic threats in the process of going to the target point to perform the task, and an optimization function model is established for the problem as shown in formula (1):

[0009]

[0010] Wherein, p t =[x t ,y t ] is the position of the spacecraft at time t; x t and y t are the coordinates of the spacecraft at time t; X represents the threat area, x and y are the coordinate information of the spacecraft in two dimensions; N max =[N Xmax ,N Ymax ] represents the longitudinal and lateral dynamic overload threshold of the spacecraft, and the angular velocity of the spacecraft cannot be greater than the threshold; p0 and p G represent the starting and ending positions respectively; the objective function aims to find a path which not only has the shortest total distance, but also meets the spacecraft dynamics constraints while safely avoiding the threat area, without overload power turning action;

[0011] The dynamic differential model of the spacecraft in continuous time is defined as shown in formula (2):

[0012]

[0013] Wherein x and y are the coordinate information of the spacecraft in two dimensions; v represents the speed of the spacecraft, And u is the differential of the speed, i.e. acceleration u; Represents the heading of the spacecraft, Is the differential of the heading, i.e. angular velocity ψ;

[0014] The condition for the success of the task in this scenario is that the distance between the spacecraft P and the target point E is less than the task range of the spacecraft, as shown in formula (3):

[0015] d PE ≤R P (3)

[0016] Wherein, d PE is the distance between the spacecraft P and the target E, and R P is the task range of the spacecraft P; the condition for the failure of the task in this scenario is that the spacecraft P overlaps with the threat area X, i.e. the spacecraft is detected by the threat radar, as shown in formula (4):

[0017] p t ∈X(4)

[0018] Step 2: According to the spacecraft path planning scene model established in step 1, design the state space model and action space model of the spacecraft;

[0019] Step 3: According to the spacecraft path planning scene model established in step 1, establish a dynamic warning area based on radar detection exposure risk;

[0020] Step 4: According to the dynamic warning area based on radar detection exposure risk established in step 3, establish a dynamic time-varying LGVF guidance potential field based on radar detection threat;

[0021] Step 5: According to the spacecraft path planning scene model established in step 1, propose an improved deep reinforcement learning SAC algorithm to provide real-time action strategy for the spacecraft to cope with unknown dynamic complex scenes;

[0022] Step 6: Integrate the dynamic time-varying LGVF guidance potential field based on radar detection threat described in step 4 into the mobile planner based on deep reinforcement learning algorithm proposed in step 5 in the form of reward function, and optimize the reinforcement learning decision process;

[0023] Step 7: Load the optimized algorithm model proposed in step 6 into the online task simulation scene with partially observable information and unpredictable information, test the test effect of the pursuit spacecraft performing the pursuit task, and perfect the combined optimization method through feedback.

[0024] The step 2 specifically includes the following steps:

[0025] Step 2.1: Design of spacecraft state space model:

[0026] Set the spacecraft on-board GPS device and gyroscope to obtain its position information and speed information; threat radar load device can obtain the position information and speed information of spacecraft within the detection range;

[0027] Let the spacecraft state space model S contain a continuous bounded state space Where x t and y t are the coordinates of the spacecraft at time t, v t and ψ t are the speed and heading, and P target and X t are the position information recorded by the spacecraft to the target and the measurement information of the threat area, respectively;

[0028] Step 2.2: Design of spacecraft action space model, specifically:

[0029] The control input of the pursuit spacecraft is designed as angular velocity and acceleration, assuming that the dynamic equation of the spacecraft sets the spacecraft as uniform accelerated motion, continuous action space a t As shown in equation (5):

[0030] a t = [u t , ψ t ] = [u t-1 + Δu, ψ t-1 + Δψ] T (5)

[0031] Wherein, the maximum angular velocity of the spacecraft is set to be less than 25.5 rad / sec, that is, ψ ∈ [-25.5, 25.5], and the counterclockwise direction in the top view is taken as the positive direction.

[0032] The specific steps of step 3 for designing and establishing a dynamic warning area based on radar detection exposure risk according to the spacecraft pursuit task scene model established in step 1 are as follows:

[0033] In order to simulate the dynamic scanning threat of the radar, the anisotropy of the radar beam is simulated according to the antenna gain at different angles of the radar, so as to establish the warning area of the spacecraft based on the radar detection exposure risk, that is, the modeling of the threat area;

[0034] It is assumed that the center of the radar array is placed at the origin of the spherical coordinate system, wherein there are N elements with uniform spacing d; the element position is shown in equation (6):

[0035]

[0036] The flow vector v k (k) of the radar array is defined as shown in equation (7):

[0037]

[0038] Wherein k is the amplitude of the wave number;

[0039] Then, for the plane wave input with a propagation direction a and a time domain frequency ω, that is, an arc of 0, the frequency-wave number response function of the N-element ULA is shown in equation (8):

[0040]

[0041] Wherein is the frequency domain weighting coefficient of each array

[0042] According to the frequency-wave number response function, the beam pattern function of the radar array can be obtained as shown in equation (9):

[0043]

[0044] The beam pattern of radar array reflects the antenna gain of radar array in different angular directions; in the following study, we will express the antenna gain of radar as G(θ, φ) = |B(ω: θ, φ)| 2

[0045]

[0046] where Pt is the peak transmit power of radar, G is the directional antenna gain, λ is the wavelength, σ is the radar cross section, and R is the distance radius from radar; using directional antenna with gain G, let Smin represent the minimum detectable signal power, then the maximum radar detection distance Rmax is shown in equation (11):

[0047]

[0048] where Pt is the peak transmit power of radar, G is the directional antenna gain, λ is the wavelength, σ is the radar cross section; according to the antenna gain G in different angles, we can obtain the maximum detection distance Rmax of radar in each angle; according to the standard parameter line simulation, we can obtain the detection distance of radar in each angle;

[0049] Considering the dynamics constraints, the radar detection area envelope is processed and approximated as an elliptical continuous area; it can be described by the elliptical equation in polar coordinates ; where a is the major axis of the ellipse, and e is the eccentricity of the ellipse; by minimizing the residual Δθ of the envelope line and the maximum detection distance of radar in each angle, the parameters a and e are solved, and the major axis a and the minor axis of the ellipse can be obtained; at each time, the algorithm will recalculate the parameters of the early warning area to simulate the change of the scanning boundary of radar, so that we can define the boundary of the elliptical area; thus, the dynamic early warning area construction is completed.

[0050] The specific steps of the dynamic time-varying LGVF guidance potential field based on the radar detection threat according to the dynamic early warning area based on the radar detection exposure risk established in step 3 are as follows:

[0051] After obtaining the dynamic early warning area, the original static potential field needs to be stretched based on its dynamic parameters when constructing the potential field; the total potential field expression of the dynamic time-varying LGVF guidance potential field based on the radar detection threat is shown in equation (12):

[0052] U total (x,y)=w att ·U att (x,y)+w rep ​·U rep (x,y) (12)

[0053] where w att and w rep are the weight coefficients of the attractive and repulsive fields respectively; U att and U rep represent the attractive and repulsive fields, the attractive field is constructed based on the relative position between the spacecraft and the target, and its general form is shown in equation (13):

[0054]

[0055] The repulsive field is constructed based on the improved dynamic LGVF; specifically, according to the positions of the radar and the spacecraft and the Lyapunov stability principle, the Lyapunov function is set as shown in equation (14):

[0056]

[0057] (x U ,y U ) and (x R ,y R ) represent the positions of the spacecraft and the radar respectively, R d is the safety distance threshold of the spacecraft from the danger, and is the straight-line distance between the spacecraft and the threat; to simulate the danger model in the real scene, we use the coordinate stretching transformation to take the obtained radar detection risk parameters, i.e., the long and short semi-axes a and b of the radar beam model, as the change factors, and set so that the limit cycle of the vector field changes from a regular circle to an ellipse;

[0058] According to the Lyapunov stability principle, the flight trajectory of the spacecraft will converge to the set hovering area, i.e., the modeled warning area; the constructed vector field will also adapt to the changes of the warning area parameters a and b, giving the spacecraft a vector guide that changes with the dynamic threat;

[0059] In the constructed new potential field, the improved dynamic time-varying LGVF is used as the repulsive field of the guide potential field to guide the spacecraft to avoid danger; the Lyapunov function is used as the repulsive field potential function

[0060] The specific steps of step 5 are as follows:

[0061] The SAC algorithm based on deep reinforcement learning is used to establish a spacecraft mobile planner. The online learning-offline training algorithm can make the model directly used for real-time planning. After learning the dynamic characteristics of the scene, the feedback output of the strategy meets the task requirements, providing the ability to handle dynamic unknown problems and meeting the current need for real-time optimization of the scene. The entropy coefficient proposed according to the different sub-tasks of the pursuit spacecraft improves the learning efficiency of the SAC algorithm and the tracking accuracy of the pursuit spacecraft. Its form is shown in equation (15).

[0062]

[0063] where R(s t ,a t ) represents the reward value obtained by taking action a t in state s t , the goal of the strategy π * is to maximize the expected total reward and the entropy of each output action, α is the entropy coefficient, and H(π(·|s t )) is the entropy term; the purpose of requiring maximum entropy is mainly to randomize the strategy, i.e., to balance the probability of each output action as much as possible, so that the agent needs to explore all possible optimal paths; the algorithm solves the optimal strategy by maximizing the expected reward of the optimal strategy, which can learn the optimal control strategy of the spacecraft and optimize the distribution of the strategy using the entropy term; a large number of sample sampling and backpropagation operations are required during the entire update process, and the control strategy can finally have strong flexibility and robustness.

[0064] The specific process of step 6 is as follows:

[0065] After constructing the dynamic time-varying LGVF guidance potential field based on radar detection exposure risk, the potential field is integrated into the reinforcement learning framework in the form of a reward function, and the potential field is used as the action reference in the offline training-online decision process of the SAC algorithm to evaluate the decision action of reinforcement learning, thereby improving the generation of more optimal actions by the SAC network parameters.

[0066] The form of the designed reward function is shown in equation (16):

[0067] r=r a +r b +r c (16)

[0068] where r represents the total reward, r a represents the sparse reward function, which is the reward given when the pursuit spacecraft successfully reaches the target point and the reward given when the task fails, and its form is shown in equation (17):

[0069]

[0070] r b The gradient descent process of the guide potential field is simulated, and the agent is given a positive reward in the guide potential field gradient descent, so as to guide the spacecraft to find the global minimum point (target point) along the potential energy gradient descent direction of the guide potential field; the form is as shown in formula (18):

[0071]

[0072] Where ω pf is a weight coefficient of the potential field reward, and are the potential energy values of the guide potential field at the previous moment and the current moment respectively;

[0073] r c is a designed guide reward component, and the process of the pursuit spacecraft flying to the escape spacecraft is divided into two stages, and the form is as shown in formula (19):

[0074]

[0075] Where, using and represent the previous distance and the current distance between the spacecraft and the target, and a weight μ1 is introduced to adjust the reward; stage 1 means that the pursuit spacecraft does not perceive the existence of the threat, at this time, the pursuit spacecraft continuously moves to the target under the driving of the gravitational field; stage 2 means that when the pursuit spacecraft points to the path where the threat T exists, a vector field model is established for the threat T according to the Lyapunov guidance vector field algorithm, and the vector field radius is designed as d PT +w P , wherein w P refers to the width of the body of the pursuit spacecraft, and the threat level is expressed in the form of an exponential function; at this time, r c guides the distance between the pursuit spacecraft control and the threat, otherwise the negative reward value is added to the total reward value as a punishment after the weight μ2 controls the reward value dimension.

[0076] The step 7 specifically comprises the following steps:

[0077] Firstly, according to the network model parameters of the running reward and feedback adjustment algorithm and the mobile planner structure, the spacecraft task scene is adapted, and the autonomous motion planning framework in step 5 is perfected;

[0078] Secondly, the potential field model parameters in step 4 are adjusted according to the feedback results, so that the evasive trajectory of the pursuit spacecraft designed by the guide potential field is more stable and more suitable for the spacecraft pursuit task scene;

[0079] Finally, the training network model parameters are continuously adjusted according to the reward function feedback, so as to improve the model training efficiency and the tracking accuracy of the spacecraft pursuit.

[0080] A computer readable storage medium, having stored thereon a computer program, which computer program is executed by a processor to cause an apparatus in which the computer readable storage medium is located to perform the LGVF-APF-SAC spacecraft path planning method based on dynamic threats.

[0081] An electronic device includes a memory and a processor, the memory having stored thereon a program executable on the processor, the processor implementing the LGVF-APF-SAC spacecraft path planning method based on dynamic threats when executing the program.

[0082] The present application models the dynamic scanning threat area of the radar according to the established spacecraft mission scenario, establishes a dynamic time-varying guidance potential field according to the dynamic threat area, proposes an improved SAC algorithm, establishes a spacecraft mobile planner, and introduces the guidance potential field and Lyapunov guidance vector field method to improve the mobile planner of the spacecraft. The present application combines the optimization ability of the traditional model algorithm, the strong perception ability of deep learning, and the macro decision learning ability of reinforcement learning, forms a new stable path planning method, makes new progress in the field of spacecraft guidance, navigation and control, and can be applied to spacecraft path planning tasks under the threat of radar dynamic scanning. The method provides real-time autonomous decision-making ability for the spacecraft, considers the power constraints of the spacecraft, and improves the mission success rate. BRIEF DESCRIPTION OF DRAWINGS

[0083] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0084] Figure 1 The flowchart of the present application;

[0085] Figure 2 The spacecraft path graph under different initial positions;

[0086] Figure 3 The spacecraft path graph under different threat layouts. DETAILED DESCRIPTION

[0087] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0088] As shown in FIGS. Figure 1 , 2 and 3, the present application comprises the following steps:

[0089] Step 1. Establishing a spacecraft path planning scene model, the specific process is as follows:

[0090] The present application relates to the spacecraft path planning task in a dynamic unknown environment, which is described as the process of avoiding dynamic threats in the way during the spacecraft performing a task to a target point. An optimization function model is established for this problem as shown in formula (1):

[0091]

[0092] Wherein, p t =[x t ,y t ] is the position of the spacecraft at each time; X represents a dangerous area; N max =[N Xmax ,N Ymax ] represents the longitudinal and lateral dynamic overload conditions of the spacecraft, and the angular velocity of the spacecraft cannot be greater than the threshold value; p0 and p G represent the starting and ending positions respectively. The objective function aims to find a path which not only has the shortest total distance, but also meets the spacecraft dynamics constraints while safely avoiding the threat area, and no overload power turning action is allowed.

[0093] The dynamic differential model of the spacecraft in continuous time is defined as shown in formula (2):

[0094]

[0095] Wherein x and y are the coordinate information of the spacecraft in two dimensions; v represents the speed of the spacecraft, and u is the differential quantity of the speed, i.e. acceleration u; represents the heading of the spacecraft, is the differential quantity of the heading, i.e. angular velocity ψ.

[0096] The condition for the success of the scene task is set as the distance between the spacecraft P and the target point E being less than the task action range of the spacecraft, as shown in formula (3):

[0097] d PE ≤RP (3)

[0098] where d PE is the distance between spacecraft P and target E, R P is the mission range of spacecraft P;

[0099] The condition of mission failure is that spacecraft P overlaps with the detection range X of the threat, i.e. spacecraft is detected by the threat radar, as shown in equation (4):

[0100] p t ∈X(4):

[0101] Step 2: Design the spacecraft state space model and action space model according to the scenario model established in step 1, which specifically includes the following steps:

[0102] Step 2.1: Design of spacecraft state space model, specifically:

[0103] Set the spacecraft onboard GPS device and gyroscope to obtain its own position information and speed information; threat radar payload device can obtain the position information and speed information of spacecraft within the detection range.

[0104] Let the spacecraft state space model S contain a continuous bounded state space where x t and y t are the coordinates of the spacecraft at time t, v t and ψ t are the speed and heading, and P target and X t are the position information recorded by the spacecraft to the target and the measurement information of the threat radar.

[0105] Step 2.2: Design of spacecraft action space model, specifically:

[0106] The control input of the pursuit spacecraft is designed as angular velocity and acceleration, assuming that the spacecraft dynamics equation sets the spacecraft as uniform accelerated motion, continuous action space a t as shown in equation (5):

[0107] a t =[u t , ψ t ]=[u t-1 +Δu, ψ t-1 +Δψ] T (5)

[0108] where the maximum angular velocity of the spacecraft is set to be less than 25.5 rad / sec, i.e. ψ∈[-25.5, 25.5], taking the counterclockwise direction in the top view as the positive direction.

[0109] Step 3: According to the established spacecraft path planning task scenario, a dynamic warning area based on radar detection risk is established, specifically:

[0110] To simulate the dynamic scanning threat of radar, the anisotropy of radar beam is simulated according to the antenna gain at different angles, so as to establish the warning area of spacecraft based on radar detection exposure risk.

[0111] Let the center of the radar array be placed at the origin of the spherical coordinate system, where there are N elements with uniform spacing d. The element position is shown in formula (6):

[0112]

[0113] Define the flow vector v of the radar array k (k) is shown in formula (7):

[0114]

[0115] where k is the amplitude of the wave number

[0116] Then for the plane wave input with propagation direction a and time domain frequency (radian) 0, the frequency-wave number response function of N-element ULA is shown in formula (8):

[0117]

[0118] where is the frequency domain weighting coefficient of each array;

[0119] According to the frequency-wave number response function, the beam pattern function of the radar array can be obtained as shown in formula (9):

[0120]

[0121] The beam pattern of the radar array reflects the antenna gain of the radar array in different angular directions. In the following research, we will use the directional power function G(θ,φ) = |B(ω:θ,φ) 2 to represent the antenna gain of the radar.

[0122] Using a directional antenna with gain G, the power transmitted by the antenna to the radar signal processor is shown in formula (10):

[0123]

[0124] where Pt is the peak transmit power of the radar, G is the directional antenna gain, λ is the wavelength, σ is the radar cross section, and R is the distance radius from the radar.

[0125] Using a directional antenna with gain G, let Smin represent the minimum detectable signal power, then the maximum radar detection range Rmax is shown in equation (11):

[0126]

[0127] where Pt is the radar peak transmit power, G is the directional antenna gain, λ is the wavelength, and σ is the radar cross section. According to the antenna gain G at different angles, we can obtain the maximum radar detection range Rmax at each angle. According to the standard parameter line simulation, we can obtain the radar detection distance at each angle.

[0128] For the consideration of dynamic constraints, the radar detection area envelope is processed and approximated as an elliptical continuous area. It can be described by the elliptical equation in polar coordinates where a is the major axis of the ellipse and e is the eccentricity of the ellipse. By minimizing the residual Δθ of the envelope line and the maximum radar detection distance at each angle, we can solve the parameters a and e to obtain the major axis a and the minor axis At each time, the algorithm will recalculate the parameters of the warning area to simulate the change of the radar scanning boundary, so that we can define the boundary of the elliptical area. At this point, the dynamic warning area construction is completed.

[0129] Step 4: Based on the dynamic warning area based on radar detection exposure risk established in step 3, a dynamic time-varying LGVF guidance potential field based on radar detection threat is established, which is specifically:

[0130] After obtaining the dynamic warning area, the original static potential field needs to be stretched based on its dynamic parameters when constructing the potential field. The total potential field expression of the dynamic time-varying LGVF guidance potential field based on radar detection threat is shown in equation (12):

[0131] U total (x,y)=w att ·U att (x,y)+w rep ·U rep (x,y) (12)

[0132] where w att and w rep are the weight coefficients of the attractive field and the repulsive field, respectively. U att and U rep represent the attractive field and the repulsive field. The attractive field is constructed based on the relative position of the spacecraft and the target, and its general form is shown in equation (13):

[0133]

[0134] The repulsive field is constructed based on the improved dynamic LGVF. Specifically, according to the positions of the radar and the spacecraft and the Lyapunov stability principle, the Lyapunov function is set as shown in equation (14):

[0135]

[0136] (x U ,y U ) and (x R ,y R ) represent the positions of the spacecraft and the radar respectively, R d is the safety distance threshold of the spacecraft from the danger, is the straight-line distance between the spacecraft and the threat. To simulate the danger model in the real scene, we use the coordinate stretching transformation to take the obtained radar detection risk parameters, i.e., the long and short semi-axes a and b of the radar beam model, as the change factors, and set so that the limit cycle of the vector field changes from a regular circle to an ellipse.

[0137] According to the Lyapunov stability principle, the flight trajectory of the spacecraft will converge to the set hovering area, i.e., the modeled warning area. The constructed vector field will also adapt to the changes of the warning area parameters a and b, giving the spacecraft a vector guide that changes with the dynamic threat.

[0138] In the constructed new potential field, the improved dynamic time-varying LGVF is used as the repulsive field of the guidance potential field to guide the spacecraft to avoid danger. The Lyapunov function is used as the potential function of the repulsive field

[0139] Step 5: Based on the spacecraft path planning scene model established in step 1, an improved deep reinforcement learning SAC algorithm is proposed to provide a mobile planner for the spacecraft, providing real-time action strategies for the spacecraft to cope with unknown dynamic complex scenes. The specific steps are as follows:

[0140] The spacecraft mobile planner is established through the SAC algorithm of deep reinforcement learning. Its online learning-offline training algorithm characteristics can make its model directly used for real-time planning. After learning the dynamic characteristics of the scene, it feedbacks the strategy output that meets the task requirements, providing the ability to handle dynamic unknown problems and meeting the current need for real-time optimization of the scene. The proposed adaptive entropy coefficient according to different sub-tasks of the pursuit spacecraft improves the learning efficiency of the SAC algorithm and the tracking accuracy of the pursuit spacecraft, and its form is shown in equation (15):

[0141]

[0142] where R(s t ,a t ) represents the state st Take action a t The obtained reward value, strategy π * The goal is to maximize the expected total reward value and the entropy of each output action, α is the entropy coefficient, H(π(·|s t )) is the entropy term. The purpose of requiring maximum entropy is mainly to randomize the strategy, that is, to balance the probability of each output action as much as possible, so that the agent needs to explore all possible optimal paths. The algorithm solves the optimal strategy by maximizing the expected reward of the optimal strategy. In this way, the optimal control strategy of the spacecraft can be learned, and the entropy term is used to optimize the distribution of the strategy. A large number of sample sampling and back propagation operations are required during the entire update process, and the control strategy finally has strong flexibility and robustness.

[0143] Step 6: Integrate the dynamic time-varying LGVF guidance potential field based on radar detection threats described in step 4 into the mobile planner based on the deep reinforcement learning algorithm proposed in step 5 in the form of a reward function, and optimize the reinforcement learning decision-making process. Specifically:

[0144] After constructing the dynamic time-varying LGVF guidance potential field based on radar detection exposure risk, the potential field is integrated into the reinforcement learning framework in the form of a reward function, and the potential field is used as the action benchmark in the offline training-online decision-making process of the SAC algorithm. The decision action of reinforcement learning is evaluated to improve the SAC network parameter to generate better actions.

[0145] The reward function is designed as shown in formula (16):

[0146] r = r a + r b + r c (16)

[0147] Where r represents the total reward value, r a represents a sparse reward function, which is the reward given when the pursuit spacecraft successfully reaches the target point and the reward given when the task fails, and its form is shown in formula (17):

[0148]

[0149] r b Simulate the gradient descent process of the guidance potential field, give the agent a positive reward when the gradient of the guidance potential field is descending, and thus guide the spacecraft to find the global minimum point (target point) along the potential energy gradient descent direction of the guidance potential field. Its form is shown in (18):

[0150]

[0151] r cis the designed guidance reward component, the process of pursuit spacecraft flying to the escape spacecraft is divided into two stages, the form is shown as formula (19):

[0152]

[0153] Wherein, using and respectively represent the previous distance and the current distance between the spacecraft and the target, while introducing the weight μ1 of adjusting the reward. Stage 1 means that the pursuit spacecraft does not perceive the existence of threats, at this time the pursuit spacecraft continuously moves to the target under the driving of the gravitational field; stage 2 means that when the pursuit spacecraft points to the path of the target vector, the threat T appears, at this time the vector field model of the threat T is established according to the Lyapunov guidance vector field algorithm, and the vector field radius is designed as d PT +w P , wherein w P refers to the body width of the pursuit spacecraft, and the threat level is expressed in the form of an exponential function; at this time r c The distance between the pursuit spacecraft control and the threat is guided, otherwise the negative reward value generated is added to the total reward value as a punishment after the dimension of the reward value is controlled by the weight μ2.

[0154] Step 7: load the optimized algorithm model proposed in step 6 into the online task simulation scene with observable and unpredictable information part, and test the test effect feedback of the pursuit task performed by the pursuit spacecraft, and perfect the combination optimization method. First, according to the network model parameters and the mobile planner structure of the running reward and feedback adjustment algorithm, adapt to the spacecraft task scene, and perfect the autonomous motion planning framework in step 5; secondly, according to the feedback result, adjust the potential field model parameters in step 4, so that the evasive trajectory of the pursuit spacecraft designed by the guide potential field to the threat is more stable and more suitable for the spacecraft pursuit task scene; finally, according to the reward function feedback, constantly adjust the training network model parameters, improve the model training efficiency and the pursuit accuracy of the pursuit spacecraft.

[0155] With the rapid development of artificial intelligence technology, the traditional spacecraft path planning method gradually exposes limitations in dealing with complex dynamic environments, especially in dealing with uncertain external threats. The method based on mathematical model is difficult to provide real-time and effective solutions. Deep reinforcement learning, with its characteristics of "off-line training and on-line decision", can realize dynamic response and real-time decision in complex environment, and become an ideal way to solve this problem. The present application proposes a LGVF-APF-SAC spacecraft path planning method combining Lyapunov guidance vector field (LGVF) and real-time risk modeling, which effectively improves the path planning accuracy and task success rate of spacecraft in the face of dynamic threats such as radar exposure, makes up for the lack of autonomous decision-making ability of existing technology in complex dynamic scenarios, and has significant application value and practical significance.

[0156] Considering the unpredictability and partial observability of external information in practical applications, it is difficult to establish an accurate mathematical model for solution, and it is difficult to guarantee the optimality of the solution simply relying on non-model learning method. Therefore, the present application proposes a combined optimization method through joint learning to improve the flexibility and reliability of the solution. First, a real-time risk function is constructed according to the dynamic threat (such as radar detection exposure risk), and then a warning area is generated; then, an improved dynamic time-varying Lyapunov guidance vector field (LGVF) is used to guide the spacecraft to avoid the warning area, and finally reach the target point. In order to further improve the adaptive ability of path planning, an end-to-end reinforcement learning framework is developed, and an improved SAC algorithm is used to integrate the guide potential field into the reward function, and optimize the decision-making ability and task execution effect of the spacecraft.

[0157] The LGVF-APF-SAC method proposed in the present application has been verified in simulation environment for many times. The experimental results show that the method can significantly improve the dynamic adaptability and reliability of path planning while meeting the spacecraft dynamics overload condition. Compared with the original SAC algorithm, after many training, the spacecraft based on LGVF-APF-SAC can quickly converge in 70000 steps of training, while the original SAC algorithm is difficult to converge, and often has overfitting problem, affecting the model performance. In addition, in the same simulation test environment, the task success rate of the spacecraft controlled by the LGVF-APF-SAC algorithm is significantly higher than that of the original SAC algorithm.

[0158] In summary, the present application provides the ability of autonomous path planning and task execution in dynamic environment for spacecraft by combining dynamic threat risk modeling and improved SAC algorithm, not only improves the autonomy and task success rate of spacecraft, but also guarantees the control accuracy and dynamic response flexibility.

[0159] Algorithm Success rate Convergence speed LGVF-APF-SAC 93％ 70000 steps Original SAC 32％ Hard to converge, overfitting

[0160] A computer readable storage medium, having stored thereon a computer program, which computer program is executable by a processor to cause a device on which the computer readable storage medium is present to perform the method of dynamic threat based LGVF-APF-SAC spacecraft path planning as described above. The computer program can comprise code, which can be in source code, object code, executable or some intermediate form. The computer readable medium can comprise any entity or medium that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memories.

[0161] An electronic device, comprising: a memory and a processor, the memory having stored thereon a program executable by the processor to implement the method of dynamic threat based LGVF-APF-SAC spacecraft path planning as described above when the processor executes the program.

[0162] The modules / units of the electronic device described in the present application, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the present application can implement all or part of the processes in the above-mentioned embodiment methods, and can also be completed by instructing related hardware devices through a computer program, which can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor.

[0163] Further, the computer readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, etc.; and the data storage area can store data created according to the use of the blockchain node, etc.

[0164] The computer readable storage medium has stored thereon computer readable instructions, which are executed by a processor in an electronic device to implement the method of dynamic threat based LGVF-APF-SAC spacecraft path planning as described in any of the above embodiments.

[0165] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner.

[0166] Any technical features in the above-described embodiments can be combined in any manner, and for the sake of brevity, not all possible combinations are described, but it is understood that the scope of the present disclosure encompasses all possible combinations.

[0167] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be construed as limiting the scope of the patent application. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

[0168] It should be noted that the terms "comprise" and "have" and any variations thereof in the specification and claims of the present application are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0169] Note that the above is only the preferred embodiment of the present application and the application of technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the specific embodiments described herein, and can include more other effective embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A dynamic threat-based LGVF-APF-SAC spacecraft path planning method, characterized by: The steps include: Step 1: Establish a spacecraft path planning scenario model. The specific process is as follows: The spacecraft path planning task in a dynamic unknown environment involved in the present invention is described as the process of a spacecraft avoiding dynamic threats on the way to a target point to perform a pursuit mission. The optimization function model for this problem is shown in formula (1): Among them, p t =[x t ,y t ] is the position of the spacecraft at time t; x t and y t is the coordinate of the spacecraft at time t, X represents the threat area, x and y are the coordinate information of the spacecraft in two dimensions; N max =[N Xmax ,N Ymax ] represents the longitudinal and lateral dynamic overload thresholds of the spacecraft, and the angular velocity of the spacecraft cannot be greater than this threshold; p0 and p G Represent the starting and ending positions respectively; the objective function aims to find a path that not only has the shortest total distance but also meets the spacecraft dynamic constraints while safely avoiding the threat area and cannot have overload power turning maneuvers; The definition of the spacecraft dynamic differential model under continuous time is shown in formula (2): Where x and y are the coordinate information of the spacecraft in two dimensions; v represents the speed of the spacecraft. and u is the differential of the velocity, i.e., acceleration u; represents the heading of the spacecraft, is the differential component of heading change, i.e. angular velocity ψ; The condition for the success of the mission in this scenario is that the distance between the spacecraft P and the target point E is less than the mission range of the spacecraft, as shown in formula (3): d PE ≤R P (3) Among them, d PE is the distance between spacecraft P and target E, R P It refers to the mission scope of spacecraft P; The mission failure condition for this scenario is that the spacecraft P overlaps with the threat area X, that is, the spacecraft is detected by the threat radar, as shown in formula (4): p t ∈X(4) Step 2: Based on the spacecraft path planning scenario model established in step 1, design the spacecraft state space model and action space model; Step 3: Based on the spacecraft path planning scenario model established in Step 1, establish a dynamic warning area based on radar detection exposure risk; Step 4: Based on the dynamic warning area based on radar detection exposure risk established in step 3, a dynamic time-varying LGVF guidance potential field based on radar detection threat is established; Step 5: Based on the spacecraft path planning scenario model established in Step 1, an improved deep reinforcement learning (SAC) algorithm is proposed to provide real-time action strategies for the spacecraft to cope with unknown dynamic and complex scenarios. Step 6: Integrate the dynamic time-varying LGVF guidance potential field based on radar threat detection described in Step 4 into the spacecraft path planning scenario model based on the deep reinforcement learning algorithm proposed in Step 5 in the form of a reward function to optimize the reinforcement learning decision-making process; Step 7: Load the optimized algorithm model proposed in step 6 into an online mission simulation scenario where the information is partially observable and unpredictable for testing, and improve the spacecraft path planning scenario model through the test feedback of the pursuit spacecraft performing the pursuit mission.

2. The dynamic threat-based LGVF-APF-SAC spacecraft path planning method according to claim 1 is characterized in that: The step 2 specifically includes the following steps: Step 2.1: Design of spacecraft state space model: Set the spacecraft's onboard GPS equipment and gyroscope to obtain its own position and speed information; the threat radar payload equipment can obtain the position and speed information of spacecraft within the detection range; Assume that the spacecraft state space model S contains a continuous bounded state space where x t and y t is the coordinate of the spacecraft at time t, v t and ψ t are the speed and heading, and P target and X t They are the spacecraft's position information record of the target and the measurement information of the threat area; Step 2.2: Design of the spacecraft motion model, specifically: The control input of the pursuit spacecraft is designed to be angular velocity and acceleration. It is assumed that the dynamic equation of the spacecraft sets the spacecraft to uniform acceleration motion. The continuous action space a t As shown in formula (5): a t =[u t ,ψ t ]=[u t-1 +D u ,ψ t-1 +Dψ] T (5) The maximum angular velocity of the spacecraft is set to be less than 25.5 rad / sec, that is, ψ∈[-25.5,25.5], and the counterclockwise direction of the top view is taken as the positive direction.

3. The dynamic threat-based LGVF-APF-SAC spacecraft path planning method according to claim 1 is characterized in that: The specific steps of step 3 to design and establish a dynamic warning area based on radar detection exposure risk based on the spacecraft pursuit mission scenario model established in step 1 are as follows: To simulate the dynamic scanning threat of radar, the anisotropy of the radar beam was simulated based on the antenna gain at different radar angles. This allowed the establishment of a warning area for the spacecraft's exposure risk based on radar detection, i.e., modeling of the threat area. Assume that the center of the radar array is placed at the origin of the spherical coordinate system, where there are N array elements with uniform spacing d; the array element positions are shown in formula (6): Define the popularity vector v of the radar array k (k) is as shown in formula (7): where k is the amplitude of the wave number; Then, for a plane wave input with a propagation direction of a and a time domain frequency of radian 0 of ω, the frequency-wavenumber response function of the N-element ULA is shown in formula (8): in is the frequency domain weighting coefficient of each array According to the frequency-wave number response function, the beam pattern function of the radar array can be obtained as shown in formula (9): The beam pattern of the radar array reflects the antenna gain of the radar array in different angular directions. In the following study, we will use the directional power function G(θ,φ)=|B(ω:θ,φ)| 2 To express the antenna gain of the radar; Using a directional antenna with a gain of G, the power delivered by the antenna to the radar signal processor is shown in formula (10): Where Pt is the radar peak transmit power, G is the directional antenna gain, λ is the wavelength, σ is the radar cross-sectional area, and R is the distance radius from the radar. Using a directional antenna with a gain of G, let Smin represent the minimum detectable signal power, then the maximum radar detection range Rmax is shown in formula (11): Where Pt is the radar peak transmit power, G is the directional antenna gain, λ is the wavelength, and σ is the radar cross-sectional area. Based on the antenna gain G at different angles, we can obtain the maximum detection range Rmax of the radar at each angle. By performing simulations based on standard parameters, we can obtain the detection range of the radar at each angle. Considering the dynamic constraints, the radar detection area envelope is processed and approximated as an elliptical continuous area; the ellipse equation in polar coordinates can be used to Description; where a is the major axis of the ellipse and e is the eccentricity of the ellipse; by minimizing the residual Δθ between the envelope and the maximum detection range of the radar at each angle, the parameters a and e can be solved to obtain the major axis a and minor axis of the ellipse. At each moment, the algorithm will recalculate the parameters of the warning area to simulate the changes in the radar's scanning boundary, allowing us to define the boundaries of the elliptical area; at this point, the dynamic warning area is constructed.

4. The dynamic threat-based LGVF-APF-SAC spacecraft path planning method according to claim 1, characterized in that: The specific steps of step 4 for establishing the dynamic warning area based on radar detection exposure risk in step 3 and the dynamic time-varying LGVF guidance potential field based on radar detection threat are as follows: After obtaining the dynamic warning area, the potential field needs to be constructed based on its dynamic parameters and the original static potential field needs to be stretched. The total potential field expression of the dynamic time-varying LGVF guidance potential field based on radar detection threat is shown in formula (12): U total (x,y)=w att ·U att (x,y)+w rep ·U rep (x,y) (12) Among them, w att and w rep are the weight coefficients of the gravitational field and the repulsive field respectively; U att with U rep Represents the gravitational field and the repulsive field. The gravitational field is constructed based on the relative position of the spacecraft and the target. Its general form is shown in formula (13): The repulsive field is constructed based on the improved dynamic LGVF. Specifically, according to the position of the radar and the spacecraft and the Lyapunov stability principle, the Lyapunov function is as shown in formula (14): Use (x U ,y U ) and (x R ,y R ) represent the spacecraft and radar positions respectively, R d is the safe distance threshold between the spacecraft and the danger, is the straight-line distance between the spacecraft and the threat; to simulate the danger model in the real scene, we use the obtained radar detection risk parameters, i.e., the major and minor semi-axes a and b of the radar beam modeling, as the variation factors based on the coordinate stretching transformation, and let The limit cycle of the vector field changes from a regular circle to an ellipse; According to the Lyapunov stability principle, the spacecraft's flight trajectory will converge to the set hovering area, which is the modeled warning area. The constructed vector field will also readjust with changes in the warning area parameters a and b, providing the spacecraft with vector guidance that changes with dynamic threats. In the new potential field constructed, the improved dynamic time-varying LGVF is used as the repulsive field of the guiding potential field to guide the spacecraft to avoid danger; the Lyapunov function As a repulsive potential function 5. The dynamic threat-based LGVF-APF-SAC spacecraft path planning method according to claim 1 is characterized in that: The specific steps of step 5 are as follows: The spacecraft movement planner is established through the SAC algorithm of deep reinforcement learning. Its online learning-offline training algorithm characteristics enable the model to be directly used for real-time planning. After learning the dynamic characteristics of the scene, it feedbacks the policy output that meets the task requirements, provides the ability to handle dynamic unknown problems, and meets the current need for real-time optimization of the scene. The proposed entropy coefficient that adapts to the different subtasks of the pursuit spacecraft improves the learning efficiency of the SAC algorithm and the tracking accuracy of the pursuit spacecraft. Its form is shown in formula (15). Among them, R(s t ,a t ) represents the state s t Take action a t The reward value obtained, strategy π * The goal is to maximize the expected total return value and maximize the entropy of each output action, α is the entropy coefficient, H(π(·|s t )) is the entropy term; the purpose of requiring the maximum entropy is mainly to randomize the strategy, that is, the probability of each output action is as balanced as possible, so that the agent needs to explore all possible optimal paths; the algorithm solves the optimal strategy by maximizing the expected return of the optimal strategy, so that the optimal control strategy of the spacecraft can be learned, and the entropy term is used to optimize the distribution of the strategy; a large number of sample sampling and backpropagation operations are required in the entire update process, and the algorithm can ultimately make the control strategy more flexible and robust.

6. The dynamic threat-based LGVF-APF-SAC spacecraft path planning method according to claim 1, characterized in that: The specific process of step 6 is as follows: After constructing a dynamic, time-varying LGVF guidance potential field based on radar detection exposure risk, this potential field was integrated into the reinforcement learning framework in the form of a reward function. This potential field was used as an action benchmark in the offline training and online decision-making process of the SAC algorithm, evaluating the decision-making actions of reinforcement learning, thereby improving the SAC network parameters to generate more optimal actions. The design reward function is shown in formula (16): r＝r a +r b +r c (16) Among them, r represents the total return value, r a represents the sparse reward function, specifically the reward given when the pursuit spacecraft successfully reaches the target point and the reward given when the mission fails. Its form is shown in formula (17): r b Simulate the gradient descent process of the guidance potential field and give the agent a positive reward when the gradient of the guidance potential field decreases, thereby guiding the spacecraft to find the global minimum point (target point) along the potential energy gradient descent direction of the guidance potential field; its form is shown in (18): where ω pf is the weight coefficient of potential field reward, and are the potential energy values ​​of the guiding potential field at the previous moment and the current moment respectively; r c is the designed guided reward component. The process of the chasing spacecraft flying towards the escaping spacecraft is divided into two stages, as shown in formula (19): Among them, use and represents the previous distance and current distance between the spacecraft and the target, respectively, and introduces the weight μ1 to adjust the reward; stage 1 refers to the pursuit spacecraft not sensing the existence of a threat. At this time, the pursuit spacecraft continues to move towards the target driven by the gravitational field; stage 2 refers to when a threat appears in the path of the pursuit spacecraft's vector pointing to the target. At this time, a vector field model is established for the threat T according to the Lyapunov guidance vector field algorithm, and the vector field radius is designed to be d PT +w P , where w P refers to the width of the pursuit spacecraft, and the threat level is expressed in the form of an exponential function; at this time, r c The pursuit spacecraft is guided to control the distance between it and the threat. Otherwise, the negative reward value generated is added to the total reward value as a penalty after controlling the dimension of the reward value through the weight μ2.

7. The dynamic threat-based LGVF-APF-SAC spacecraft path planning method according to claim 1, characterized in that: The step 7 specifically includes the following steps: First, adjust the network model parameters and mobility planner structure of the algorithm based on the running rewards and feedback to adapt to the spacecraft mission scenario and improve the spacecraft path planning scenario model described in step 5; Secondly, adjust the potential field model parameters described in step 4 based on the feedback results, so that the pursuit spacecraft's evasive trajectory designed by the guidance potential field is more stable and more suitable for the spacecraft pursuit mission scenario; Finally, the training network model parameters are continuously adjusted based on the reward function feedback to improve the model training efficiency and the tracking accuracy of the spacecraft.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the device where the computer-readable storage medium is located executes the dynamic threat-based LGVF-APF-SAC spacecraft path planning method according to any one of claims 1 to 7.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a program that can be run on the processor, and when the processor executes the program, the dynamic threat-based LGVF-APF-SAC spacecraft path planning method according to any one of claims 1 to 7 is implemented.

Images