Dynamic threat-based LGVF-APF-SAC spacecraft path planning method, storage medium and electronic equipment

Abstract

The invention discloses a dynamic threat-based LGVF-APF-SAC spacecraft path planning method, a storage medium and an electronic device, and the method specifically comprises the following steps: firstly, building a real-time risk function according to a dynamic threat (radar detection exposure risk), and further generating an early warning region according to the risk function (radar radiation); based on the early warning area, an improved dynamic time-varying LGVF is used to construct a guide potential field based on the dynamic time-varying LGVF, so that the spacecraft is guided to avoid the early warning area to reach a target point. Then, an end-to-end reinforcement learning framework is developed; according to the framework, a maximum entropy RL (SAC) algorithm is selected, and a constructed guide potential field is integrated into a learning process in a reward function form so as to evaluate and optimize decision actions of a spacecraft. According to the algorithm provided by the invention, the dynamic adaptive capacity and reliability of path planning can be improved while the dynamic overload condition of the spacecraft is ensured to be met.

Description

Technical Field

[0001] The present invention relates to the technical field of spacecraft guidance, navigation and control (GNC), and in particular to a LGVF-APF-SAC spacecraft path planning method, a storage medium and an electronic device based on dynamic threats. Background Art

[0002] At present, with the development of automatic control technology in recent years, various spacecraft have been widely used in the fields of industry, military and medicine. At the same time, the mission scenarios of spacecraft have become more and more complex and changeable. Many interferences, confrontations and time-varying characteristics have been introduced in practical applications, and its guidance, navigation and control (GNC) has caused new challenges, which has attracted widespread attention from academia and industry.

[0003] As a basic topic of GNC, path planning has become a research hotspot in related fields. Path planning technology requires finding a collision-free path for the spacecraft to reach the destination safely and efficiently. Previous path planning methods can be divided into traditional methods, heuristic algorithms, and learning-based methods. The disadvantages of traditional methods are long search time, high computational consumption, and poor real-time performance. Therefore, they are usually used for global path planning and are difficult to adapt to dynamic environments. The main disadvantages of heuristic algorithms are that they have computational costs and memory requirements, and rely on heuristic or probabilistic rules, which do not always guarantee the optimal solution and may also lead to slow convergence. Most of the learning-based method solution processes fail to utilize the optimal solution of control theory, do not consider the spacecraft dynamic constraints, and lack generalization modeling of dynamic threat environments, which inevitably leads to the unreliability of the local optimality of the planned path. Therefore, it is necessary to propose a reliable path planning method with dynamic generalization capabilities. Summary of the invention

[0004] The purpose of the present invention is to provide a dynamic threat-based LGVF-APF-SAC spacecraft path planning method, storage medium and electronic device, which can solve two major problems of spacecraft path planning in a dynamic unknown environment. First, the guidance potential field is used as an evaluation function to compress the solution space to solve the problem that the environment is highly dynamic and the deep reinforcement learning network is difficult to converge; second, the stable and smooth characteristics of LGVF are used to consider the dynamic constraints of the spacecraft while outputting safe yaw commands to generate a smooth and efficient path.

[0005] The technical solution adopted by the present invention is:

[0006] The LGVF-APF-SAC spacecraft path planning method based on dynamic threats includes the following steps:

[0007] Step 1: Establish a spacecraft path planning scenario model. The specific process is as follows:

[0008] The spacecraft path planning task in a dynamic unknown environment involved in the present invention is described as the process of the spacecraft avoiding dynamic threats on the way to the target point to perform the mission. The optimization function model for this problem is established as shown in formula (1):

[0009]

[0010] Among them, p t =[x t ,y t ] is the position of the spacecraft at time t; x t and t is the coordinate of the spacecraft at time t, X represents the threat area, x and y are the coordinate information of the spacecraft in two dimensions; N max =[N Xmax ,N Ymax ] represents the longitudinal and lateral dynamic overload thresholds of the spacecraft, and the angular velocity of the spacecraft cannot be greater than this threshold; p 0 and p G Represent the starting and ending positions respectively; the objective function aims to find a path that not only has the shortest total distance, but also meets the spacecraft dynamic constraints while safely avoiding the threat area, and cannot have overload power turning actions;

[0011] The dynamic differential model of the spacecraft in continuous time is defined as shown in formula (2):

[0012]

[0013] Where x and y are the coordinate information of the spacecraft in two dimensions; v represents the speed of the spacecraft. and u is the differential of velocity, i.e. acceleration u; represents the heading of the spacecraft, is the differential component of heading change, i.e. angular velocity ψ;

[0014] The condition for the success of the mission in this scenario is that the distance between the spacecraft P and the target point E is less than the mission range of the spacecraft, as shown in formula (3):

[0015] d PE ≤R P (3)

[0016] Among them, d PE is the distance between the spacecraft P and the target E, R P It refers to the mission scope of spacecraft P. The mission failure condition of this scenario is set as the overlap between spacecraft P and threat area X, that is, the spacecraft is detected by the threat radar, as shown in formula (4):

[0017] p t ∈X(4)

[0018] Step 2: Based on the spacecraft path planning scenario model established in step 1, design the state space model and action space model of the spacecraft;

[0019] Step 3: Based on the spacecraft path planning scenario model established in step 1, establish a dynamic warning area based on radar detection exposure risk;

[0020] Step 4: According to the dynamic warning area based on radar detection exposure risk established in step 3, a dynamic time-varying LGVF guidance potential field based on radar detection threat is established;

[0021] Step 5: Based on the spacecraft path planning scenario model established in step 1, an improved deep reinforcement learning SAC algorithm is proposed to provide real-time action strategies for the spacecraft to cope with unknown dynamic and complex scenarios;

[0022] Step 6: Integrate the dynamic time-varying LGVF guidance potential field based on radar detection threat described in step 4 into the mobile planner based on deep reinforcement learning algorithm proposed in step 5 in the form of reward function to optimize the reinforcement learning decision process;

[0023] Step 7: Load the optimized algorithm model proposed in step 6 into an online mission simulation scenario where the information is partially observable and unpredictable for testing, and improve the combined optimization method through the test effect feedback of the pursuit spacecraft performing the pursuit mission.

[0024] The step 2 specifically includes the following steps:

[0025] Step 2.1: Design of spacecraft state space model:

[0026] Set the spacecraft's onboard GPS equipment and gyroscope to obtain its own position information and speed information; the threat radar payload equipment can obtain the spacecraft's position information and speed information within the detection range;

[0027] Assume that the spacecraft state space model S contains a continuous bounded state space where x t and t is the coordinate of the spacecraft at time t, v t and ψ t are the speed and heading, and P target and X t They are the spacecraft's position information record of the target and the measurement information of the threat area;

[0028] Step 2.2: Design of the spacecraft motion space model, specifically:

[0029] The control input of the pursuit spacecraft is designed to be angular velocity and acceleration. It is assumed that the dynamic equation of the spacecraft sets the spacecraft to uniformly accelerated motion. The continuous action space a t As shown in formula (5):

[0030] a t =[u t , ψ t ]=[u t-1 +Δu,ψ t-1 +Δψ] T (5)

[0031] The maximum angular velocity of the spacecraft is set to be less than 25.5 rad / sec, that is, ψ∈[-25.5,25.5], and the counterclockwise direction of the top view is taken as the positive direction.

[0032] The specific steps of step 3 to design and establish a dynamic warning area based on radar detection exposure risk according to the spacecraft pursuit mission scenario model established in step 1 are as follows:

[0033] To simulate the dynamic scanning threat of radar, the anisotropy of radar beam was simulated according to the antenna gain at different angles of radar, so as to establish the early warning area of ​​spacecraft exposure risk based on radar detection, that is, the modeling of threat area;

[0034] Assume that the center of the radar array is placed at the origin of the spherical coordinate system, where there are N array elements with uniform spacing d; the array element positions are shown in formula (6):

[0035]

[0036] Define the popularity vector v of the radar array k (k) is as shown in formula (7):

[0037]

[0038] where k is the amplitude of the wave number;

[0039] Then, for a plane wave input with a propagation direction of a and a time domain frequency of radian 0 of ω, the frequency-wave number response function of the N-element ULA is as shown in formula (8):

[0040]

[0041] in is the frequency domain weighting coefficient of each array

[0042] According to the frequency-wave number response function, the beam pattern function of the radar array can be obtained as shown in formula (9):

[0043]

[0044] The beam pattern of the radar array reflects the antenna gain of the radar array at different angles; in the following study, we will use the directional power function G(θ,φ)=|B(ω:θ,φ)| 2 To represent the antenna gain of the radar; using a directional antenna with a gain of G, the power transmitted to the radar signal processor by the antenna is shown in formula (10):

[0045]

[0046] Where Pt is the radar peak transmit power, G is the directional antenna gain, λ is the wavelength, σ is the radar cross-sectional area, and R is the distance radius from the radar. Using a directional antenna with a gain of G, let Smin represent the minimum detectable signal power, then the maximum radar detection distance Rmax is as shown in formula (11):

[0047]

[0048] The value of the transmission power, G is the directional antenna gain, λ is the wavelength, and σ is the radar cross-sectional area; according to the antenna gain G at different angles, we can obtain the maximum detection distance Rmax of the radar at each angle; according to the standard parameters, simulation can be performed to obtain the detection distance of the radar at each angle;

[0049] Considering the dynamic constraints, the radar detection area envelope is processed and approximated as an elliptical continuous area; the ellipse equation in polar coordinates can be used Description; where a is the major axis of the ellipse, and e is the eccentricity of the ellipse; by minimizing the residual Δθ between the envelope and the maximum detection distance of the radar at each angle, the parameters a and e can be solved to obtain the major axis a and minor axis of the ellipse. At each moment, the algorithm will recalculate the parameters of the warning area to simulate the changes in the radar's scanning boundaries, allowing us to define the boundaries of the elliptical area; at this point, the dynamic warning area is constructed.

[0050] The specific steps of the step 4 of establishing the dynamic warning area based on radar detection exposure risk in step 3 and the dynamic time-varying LGVF guidance potential field based on radar detection threat are as follows:

[0051] After obtaining the dynamic warning area, the potential field needs to be constructed based on its dynamic parameters and the original static potential field needs to be stretched. The total potential field expression of the dynamic time-varying LGVF guidance potential field based on radar detection threat is shown in formula (12):

[0052] U total (x,y)=w att ·U att (x,y)+w rep ·Urep (x,y) (12)

[0053] Among them, w att and w rep are the weight coefficients of the gravitational field and the repulsive field respectively; U att with U rep represents the gravitational field and the repulsive field. The gravitational field is constructed based on the relative position of the spacecraft and the target. Its general form is shown in formula (13):

[0054]

[0055] The repulsive field is constructed based on the improved dynamic LGVF. Specifically, according to the position of the radar and the spacecraft and the Lyapunov stability principle, the Lyapunov function is as shown in formula (14):

[0056]

[0057] Use (x U ,y U ) and (x R ,y R ) represent the spacecraft and radar positions respectively, R d is the safe distance threshold between the spacecraft and the danger, is the straight-line distance between the spacecraft and the threat; to simulate the danger model in the real scene, we use the obtained radar detection risk parameters, i.e., the major and minor semi-axes a and b of the radar beam modeling, as the variation factors based on the coordinate stretching transformation, and let The limit cycle of the vector field changes from a regular circle to an ellipse;

[0058] According to the Lyapunov stability principle, the flight trajectory of the spacecraft will converge to the set hovering area, that is, the modeled warning area; the constructed vector field will also readapt with the changes of the warning area parameters a and b, giving the spacecraft vector guidance that changes with dynamic threats;

[0059] In the new potential field constructed, the improved dynamic time-varying LGVF is used as the repulsive field of the guiding potential field to guide the spacecraft to avoid danger; the Lyapunov function As a repulsive potential function

[0060] The specific steps of step 5 are as follows:

[0061] The spacecraft movement planner is established through the SAC algorithm of deep reinforcement learning. Its online learning-offline training algorithm characteristics enable its model to be directly used for real-time planning. After learning the dynamic characteristics in the scene, it feeds back the policy output that meets the task requirements, provides the ability to handle dynamic unknown problems, and meets the current need for real-time optimization of the scene. The proposed entropy coefficient that is adaptive to different subtasks of the pursuit spacecraft improves the learning efficiency of the SAC algorithm and the tracking accuracy of the pursuit spacecraft. Its form is shown in formula (15);

[0062]

[0063] Among them, R(s t ,a t ) represents the state s t Take action a t The return value obtained, strategy π * The goal is to maximize the expectation of the total return value and maximize the entropy of each output action. α is the entropy coefficient, H(π(·|s t )) is the entropy term; the purpose of requiring the maximum entropy is mainly to randomize the strategy, that is, the probability of each output action is as balanced as possible, so that the agent needs to explore all possible optimal paths; the algorithm solves the optimal strategy by maximizing the expected return of the optimal strategy, so that the optimal control strategy of the spacecraft can be learned, and the entropy term is used to optimize the distribution of the strategy; a large number of sample sampling and back propagation operations are required in the entire update process, and the algorithm can ultimately make the control strategy more flexible and robust.

[0064] The specific process of step 6 is as follows:

[0065] After constructing a dynamic time-varying LGVF guidance potential field based on radar detection exposure risk, the potential field is integrated into the reinforcement learning framework in the form of a reward function. The potential field is used as the action benchmark in the offline training-online decision-making process of the SAC algorithm to evaluate the decision-making actions of reinforcement learning, thereby improving the SAC network parameters to generate better actions.

[0066] The design reward function is shown in formula (16):

[0067] r=r a +r b +r c (16)

[0068] Among them, r represents the total return value, r a represents the sparse reward function, which is the reward given when the pursuit spacecraft successfully reaches the target point and the reward given when the mission fails. Its form is shown in formula (17):

[0069]

[0070] r b The gradient descent process of the guidance potential field is simulated, and a positive reward is given to the agent when the gradient of the guidance potential field decreases, thereby guiding the spacecraft to find the global minimum point (target point) along the direction of the potential energy gradient descent of the guidance potential field; its form is shown in (18):

[0071]

[0072] where ω pf is the weight coefficient of potential field reward, and are the potential energy values ​​of the guiding potential field at the previous moment and at the current moment respectively;

[0073] r c is the designed guiding reward component. The process of the chasing spacecraft flying towards the escaping spacecraft is divided into two stages, as shown in formula (19):

[0074]

[0075] Among them, use and Represent the previous distance and current distance between the spacecraft and the target respectively, and introduce the weight μ to adjust the reward 1 ; Stage 1 refers to the pursuit spacecraft not sensing the existence of a threat. At this time, the pursuit spacecraft continues to move toward the target driven by the gravitational field; Stage 2 refers to when a threat appears in the path of the vector pointing to the target by the pursuit spacecraft. At this time, a vector field model is established for the threat T according to the Lyapunov guidance vector field algorithm, and the vector field radius is designed to be d PT +w P , where w P refers to the fuselage width of the pursuit spacecraft, and the threat level is expressed in the form of an exponential function; at this time r c Guide the pursuit spacecraft to control the distance between it and the threat, otherwise the negative reward value generated is passed through the weight μ 2 After controlling the dimension of the reward value, it is added to the total reward value as a penalty.

[0076] The step 7 specifically includes the following steps:

[0077] First, adjust the network model parameters and the mobile planner structure of the algorithm according to the running reward and feedback, adapt to the spacecraft mission scenario, and improve the autonomous motion planning framework described in step 5;

[0078] Secondly, the potential field model parameters described in step 4 are adjusted according to the feedback results, so that the evasive trajectory of the pursuit spacecraft designed by the guidance potential field is more stable and more suitable for the spacecraft pursuit mission scenario;

[0079] Finally, the training network model parameters are continuously adjusted according to the reward function feedback to improve the model training efficiency and the tracking accuracy of the spacecraft.

[0080] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the device where the computer-readable storage medium is located executes the LGVF-APF-SAC spacecraft path planning method based on dynamic threats.

[0081] An electronic device comprises: a memory and a processor, wherein the memory stores a program that can be run on the processor, and the processor implements the dynamic threat-based LGVF-APF-SAC spacecraft path planning method when executing the program.

[0082] The present invention models the dynamic scanning threat area of ​​the radar according to the established spacecraft mission scenario; establishes a dynamic time-varying guidance potential field according to the dynamic threat area; proposes an improved SAC algorithm to establish a spacecraft motion planner; and introduces the guidance potential field and Lyapunov guidance vector field method to improve the spacecraft motion planner. The present invention combines the optimization ability of the traditional model algorithm, the powerful perception ability of deep learning, and the macro decision-making learning ability of reinforcement learning to form a new stable path planning method, making new progress in the field of spacecraft guidance, navigation and control, and can be applied to spacecraft path planning tasks under the threat of radar dynamic scanning. The method provides real-time autonomous decision-making capabilities for spacecraft, takes into account the power constraints of the spacecraft, and improves the mission success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0084] Figure 1 is a flow chart of the present invention;

[0085] Figure 2 is the spacecraft path diagram at different initial positions;

[0086] Figure 3 Figure 2 shows the spacecraft path diagram under different threat layouts. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] like Figure 1 , 2 As shown in 3, the present invention includes the following steps:

[0089] Step 1. Establish a spacecraft path planning scenario model. The specific process is as follows:

[0090] The spacecraft path planning task in a dynamic unknown environment involved in the present invention is described as the process of the spacecraft avoiding dynamic threats on the way to the target point to perform the mission. The optimization function model for this problem is established as shown in formula (1):

[0091]

[0092] Among them, p t =[x t ,y t ] is the position of the spacecraft at each moment; X represents the danger zone; N max =[N Xmax ,N Ymax ] represents the longitudinal and lateral dynamic overload conditions of the spacecraft, and the angular velocity of the spacecraft cannot be greater than this threshold; p 0 and p G Represent the start and end positions respectively. The objective function aims to find a path that not only has the shortest total distance, but also meets the spacecraft dynamic constraints while safely avoiding the threat area and without overload power turning actions.

[0093] The dynamic differential model of the spacecraft in continuous time is defined as shown in formula (2):

[0094]

[0095] Where x and y are the coordinate information of the spacecraft in two dimensions; v represents the speed of the spacecraft. and u is the differential of velocity, i.e. acceleration u; represents the heading of the spacecraft, is the differential component of heading change, namely the angular velocity ψ.

[0096] The condition for the success of the mission in this scenario is that the distance between the spacecraft P and the target point E is less than the mission range of the spacecraft, as shown in formula (3):

[0097] dPE ≤R P (3)

[0098] Among them, d PE is the distance between the spacecraft P and the target E, R P It refers to the mission scope of spacecraft P;

[0099] The condition for mission failure in this scenario is that the spacecraft P overlaps with the threat detection range X, that is, the spacecraft is detected by the threat radar, as shown in formula (4):

[0100] p t ∈X(4):

[0101] Step 2: According to the scenario model established in step 1, design the spacecraft state space model and action space model. Step 2 specifically includes the following steps:

[0102] Step 2.1: Design of spacecraft state space model, specifically:

[0103] By setting the spacecraft's onboard GPS equipment and gyroscope, it can obtain its own position information and speed information; the threat radar payload equipment can obtain the spacecraft's position information and speed information within the detection range.

[0104] Assume that the spacecraft state space model S contains a continuous bounded state space where x t and t is the coordinate of the spacecraft at time t, v t and ψ t are the speed and heading, and P target and X t They are the spacecraft's position information record of the target and the radar's measurement information of the threat detection.

[0105] Step 2.2: Design of the spacecraft motion space model, specifically:

[0106] The control input of the pursuit spacecraft is designed to be angular velocity and acceleration. It is assumed that the dynamic equation of the spacecraft sets the spacecraft to uniformly accelerated motion. The continuous action space a t As shown in formula (5):

[0107] a t =[u t , ψ t ]=[u t-1 +Δu,ψ t-1 +Δψ] T (5)

[0108] Among them, the maximum angular velocity of the spacecraft is set to be less than 25.5 rad / sec, that is, ψ∈[-25.5,25.5], and the counterclockwise direction of the top view is taken as the positive direction;

[0109] Step 3: Based on the established spacecraft path planning mission scenario, establish a dynamic warning area based on radar detection risk, specifically:

[0110] In order to simulate the dynamic scanning threat of radar, the anisotropy of the radar beam was simulated according to the antenna gain at different angles of the radar, thereby establishing an early warning area for spacecraft exposure risks based on radar detection.

[0111] Assume that the center of the radar array is placed at the origin of the spherical coordinate system, where there are N array elements with uniform spacing d. The array element positions are shown in formula (6):

[0112]

[0113] Define the popularity vector v of the radar array k (k) is as shown in formula (7):

[0114]

[0115] where k is the amplitude of the wave number

[0116] Then, for a plane wave input with a propagation direction of a and a time domain frequency (radian) 0 of ω, the frequency-wave number response function of the N-element ULA is as shown in formula (8):

[0117]

[0118] in is the frequency domain weighting coefficient of each array;

[0119] According to the frequency-wave number response function, the beam pattern function of the radar array can be obtained as shown in formula (9):

[0120]

[0121] The beam pattern of the radar array reflects the antenna gain of the radar array at different angles. In the following study, we will use the directional power function G(θ,φ)=|B(ω:θ,φ)| 2 To represent the radar antenna gain.

[0122] Using a directional antenna with a gain of G, the power delivered by the antenna to the radar signal processor is shown in formula (10):

[0123]

[0124] Among them, Pt is the radar peak transmit power, G is the directional antenna gain, λ is the wavelength, σ is the radar cross-sectional area, and R is the distance radius from the radar.

[0125] Using a directional antenna with a gain of G, let Smin represent the minimum detectable signal power, then the maximum radar detection distance Rmax is as shown in formula (11):

[0126]

[0127] Among them, Pt is the radar peak transmission power, G is the directional antenna gain, λ is the wavelength, and σ is the radar cross-sectional area. According to the antenna gain G at different angles, we can obtain the maximum detection distance Rmax of the radar at each angle. According to the standard parameters, the detection distance of the radar at each angle can be obtained by simulation.

[0128] Considering the dynamic constraints, the radar detection area envelope is processed and approximated as an elliptical continuous area. The ellipse equation in polar coordinates can be used to Description. Where a is the major axis of the ellipse, and e is the eccentricity of the ellipse. By minimizing the residual Δθ between the envelope and the maximum detection distance of the radar at each angle, the parameters a and e can be solved to obtain the major axis a and minor axis of the ellipse. At each moment, the algorithm will recalculate the parameters of the warning area to simulate the change of the radar scanning boundary, so that we can define the boundary of the elliptical area. At this point, the dynamic warning area is constructed.

[0129] Step 4: According to the dynamic warning area based on radar detection exposure risk established in step 3, a dynamic time-varying LGVF guidance potential field based on radar detection threat is established, specifically:

[0130] After obtaining the dynamic warning area, the potential field needs to be constructed based on its dynamic parameters and the original static potential field needs to be stretched. The total potential field expression of the dynamic time-varying LGVF guidance potential field based on radar detection threat is shown in formula (12):

[0131] U total (x,y)=w att ·U att (x,y)+w rep ·U rep (x,y) (12)

[0132] Among them, w att and w rep are the weight coefficients of the gravitational field and the repulsive field respectively. att with U rep Represents the gravitational field and the repulsive field. The gravitational field is constructed based on the relative position of the spacecraft and the target, and its general form is shown in formula (13):

[0133]

[0134] The repulsive field is constructed based on the improved dynamic LGVF. Specifically, according to the position of the radar and the spacecraft and the Lyapunov stability principle, the Lyapunov function is as shown in formula (14):

[0135]

[0136] Use (x U ,y U ) and (x R ,y R ) represent the spacecraft and radar positions respectively, R d is the safe distance threshold between the spacecraft and the danger, is the straight-line distance between the spacecraft and the threat. To simulate the danger model in the real scene, we use the radar detection risk parameters obtained, i.e., the major and minor semi-axes a and b of the radar beam modeling, as the variation factors based on the coordinate stretching transformation, and let This changes the limit cycle of the vector field from a regular circle to an ellipse.

[0137] According to the Lyapunov stability principle, the flight trajectory of the spacecraft will converge to the set hovering area, that is, the modeled warning area. The constructed vector field will also readapt with the changes of the warning area parameters a and b, giving the spacecraft vector guidance that changes with dynamic threats.

[0138] In the new potential field constructed, the improved dynamic time-varying LGVF is used as the repulsive field of the guiding potential field to guide the spacecraft to avoid danger. As a repulsive potential function

[0139] Step 5: Based on the spacecraft path planning scenario model established in step 1, an improved deep reinforcement learning SAC algorithm is proposed to provide a mobile planner to provide real-time action strategies for the spacecraft to cope with unknown dynamic and complex scenarios. The specific steps are:

[0140] The spacecraft movement planner is established through the SAC algorithm of deep reinforcement learning. Its online learning-offline training algorithm characteristics enable its model to be directly used for real-time planning. After learning the dynamic characteristics in the scene, it feeds back the policy output that meets the task requirements, provides the ability to handle dynamic unknown problems, and meets the current need for real-time optimization of the scene. The proposed entropy coefficient that is adaptive to different subtasks of the pursuit spacecraft improves the learning efficiency of the SAC algorithm and the tracking accuracy of the pursuit spacecraft. Its form is shown in formula (15);

[0141]

[0142] Among them, R(s t ,a t ) represents the state s t Take action a t The return value obtained, strategy π * The goal is to maximize the expectation of the total return value and maximize the entropy of each output action. α is the entropy coefficient, H(π(·|s t )) is the entropy term. The purpose of requiring the maximum entropy is mainly to randomize the strategy, that is, the probability of each action output is as balanced as possible, so that the agent needs to explore all possible optimal paths. The algorithm solves the optimal strategy by maximizing the expected return of the optimal strategy. In this way, the optimal control strategy of the spacecraft can be learned, and the entropy term can be used to optimize the distribution of the strategy. A large number of sample sampling and back-propagation operations are required in the entire update process. The algorithm can ultimately make the control strategy more flexible and robust.

[0143] Step 6: Integrate the dynamic time-varying LGVF guidance potential field based on radar detection threat described in step 4 into the mobile planner based on deep reinforcement learning algorithm proposed in step 5 in the form of reward function to optimize the reinforcement learning decision process. Specifically:

[0144] After constructing a dynamic time-varying LGVF guidance potential field based on radar detection exposure risk, the potential field is integrated into the reinforcement learning framework in the form of a reward function. The potential field is used as the action benchmark in the offline training-online decision-making process of the SAC algorithm to evaluate the decision-making actions of reinforcement learning, thereby improving the SAC network parameters to generate better actions.

[0145] The design reward function is shown in formula (16):

[0146] r=r a +r b +r c (16)

[0147] Among them, r represents the total return value, r a represents the sparse reward function, which is the reward given when the pursuit spacecraft successfully reaches the target point and the reward given when the mission fails. Its form is shown in formula (17):

[0148]

[0149] r b The gradient descent process of the guidance potential field is simulated, and a positive reward is given to the agent when the gradient of the guidance potential field decreases, thereby guiding the spacecraft to find the global minimum point (target point) along the direction of the potential energy gradient descent of the guidance potential field. Its form is shown in (18):

[0150]

[0151] r c is the designed guiding reward component. The process of the chasing spacecraft flying towards the escaping spacecraft is divided into two stages, as shown in formula (19):

[0152]

[0153] Among them, use and Represent the previous distance and current distance between the spacecraft and the target respectively, and introduce the weight μ to adjust the reward 1 Stage 1 refers to the pursuit spacecraft not sensing the threat. At this time, the pursuit spacecraft continues to move towards the target driven by the gravitational field. Stage 2 refers to the threat appearing in the path of the vector pointing to the target by the pursuit spacecraft. At this time, a vector field model is established for the threat T according to the Lyapunov guidance vector field algorithm. The vector field radius is designed to be d PT +w P , where w P refers to the fuselage width of the pursuit spacecraft, and the threat level is expressed in the form of an exponential function; at this time r c Guide the pursuit spacecraft to control the distance between it and the threat, otherwise the negative reward value generated is passed through the weight μ 2 After controlling the dimension of the reward value, it is added to the total reward value as a penalty.

[0154] Step 7: Load the optimized algorithm model proposed in step 6 into an online mission simulation scenario where the information is partially observable and unpredictable for testing, and improve the combined optimization method through the test effect feedback of the pursuit spacecraft performing the pursuit mission. First, adjust the network model parameters and mobile planner structure of the algorithm according to the running reward and feedback to adapt to the spacecraft mission scenario and improve the autonomous motion planning framework described in step 5; secondly, adjust the potential field model parameters described in step 4 according to the feedback results, so that the pursuit spacecraft designed by the guidance potential field has a more stable avoidance trajectory for threats and is more suitable for the spacecraft pursuit mission scenario; finally, continuously adjust the training network model parameters according to the reward function feedback to improve the model training efficiency and the tracking accuracy of the pursuit spacecraft.

[0155] With the rapid development of artificial intelligence technology, traditional spacecraft path planning methods have gradually exposed their limitations when dealing with complex dynamic environments. In particular, when dealing with uncertain external threats, methods based on mathematical models are difficult to provide real-time and effective solutions. Deep reinforcement learning, with its characteristics of "offline training and online decision-making", can achieve dynamic response and real-time decision-making in complex environments, and has become an ideal way to solve this problem. The present invention proposes a LGVF-APF-SAC spacecraft path planning method that combines the Lyapunov guidance vector field (LGVF) with real-time risk modeling, which effectively improves the path planning accuracy and mission success rate of spacecraft when facing dynamic threats such as radar exposure, and makes up for the lack of autonomous decision-making capabilities of existing technologies in complex dynamic scenes. It has significant application value and practical significance.

[0156] Taking into account the unpredictability and partial observability of external information in practical applications, it is very difficult to establish an accurate mathematical model for solving the problem, and it is difficult to guarantee the optimality of the solution by relying solely on non-model learning methods. Therefore, the present invention proposes a combined optimization method through joint learning to improve the flexibility and reliability of the solution. First, a real-time risk function is constructed based on dynamic threats (such as radar detection exposure risks) to generate a warning area; then, an improved dynamic time-varying Lyapunov guidance vector field (LGVF) is used to guide the spacecraft to avoid the warning area and finally reach the target point. In order to further improve the adaptive capability of path planning, the present invention develops an end-to-end reinforcement learning framework, uses an improved SAC algorithm, integrates the guidance potential field into a reward function, and optimizes the decision-making ability and mission execution effect of the spacecraft.

[0157] The LGVF-APF-SAC method proposed in the present invention has been verified many times in a simulation environment. Experimental results show that this method can significantly improve the dynamic adaptability and reliability of path planning while meeting the spacecraft dynamic overload conditions. By comparing with the original SAC algorithm, after multiple trainings, the spacecraft based on LGVF-APF-SAC can converge quickly in 70,000 steps of training, while the original SAC algorithm is not only difficult to converge, but also often has overfitting problems, affecting the model performance. In addition, in the same simulation test environment, the mission success rate of the spacecraft controlled by the LGVF-APF-SAC algorithm is significantly higher than that of the original SAC algorithm.

[0158] In summary, the present invention provides spacecraft with the ability to autonomously plan paths and execute missions in a dynamic environment by combining dynamic threat risk modeling with an improved SAC algorithm. This not only improves the autonomy and mission success rate of the spacecraft, but also ensures the control accuracy and flexibility of dynamic response.

[0159] algorithm Success rate Convergence speed LGVF-APF-SAC 93％ 70000 steps Original SAC 32％ Difficult to converge, overfitting

[0160] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the device where the computer-readable storage medium is located executes the LGVF-APF-SAC spacecraft path planning method based on dynamic threats as described above. The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory.

[0161] An electronic device comprises: a memory and a processor, wherein the memory stores a program that can be run on the processor, and when the processor executes the program, the LGVF-APF-SAC spacecraft path planning method based on dynamic threats as described above is implemented.

[0162] If the module / unit integrated in the electronic device described in this application is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0163] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0164] Computer-readable instructions are stored in the computer-readable storage medium, and the computer-readable instructions are executed by a processor in an electronic device to implement the dynamic threat-based LGVF-APF-SAC spacecraft path planning method described in any of the above embodiments.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0166] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

[0168] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0169] Note that the above are only preferred embodiments of the present invention and the principles of the application technology. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention is described in more detail through the above embodiments, the present invention is not limited to the specific embodiments described herein, and may include more other effective embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. The LGVF-APF-SAC spacecraft path planning method based on dynamic threats is characterized by: The steps include: Step 1: Establish a spacecraft path planning scenario model. The specific process is as follows: The spacecraft path planning task in a dynamic unknown environment involved in the present invention is described as the process of the spacecraft avoiding dynamic threats on the way to the target point to perform the mission. The optimization function model for this problem is established as shown in formula (1): Among them, p t =[x t ,y t ] is the position of the spacecraft at time t; x t and t is the coordinate of the spacecraft at time t, X represents the threat area, x and y are the coordinate information of the spacecraft in two dimensions; N max =[N Xmax ,N Ymax ] represents the longitudinal and lateral dynamic overload thresholds of the spacecraft, and the angular velocity of the spacecraft cannot be greater than this threshold; p0 and p G Represent the starting and ending positions respectively; the objective function aims to find a path that not only has the shortest total distance, but also meets the spacecraft dynamic constraints while safely avoiding the threat area, and cannot have overload power turning actions; The dynamic differential model of the spacecraft in continuous time is defined as shown in formula (2): Where x and y are the coordinate information of the spacecraft in two dimensions; v represents the speed of the spacecraft. and u is the differential of velocity, i.e. acceleration u; represents the heading of the spacecraft, is the differential component of heading change, i.e. angular velocity ψ; The condition for the success of the mission in this scenario is that the distance between the spacecraft P and the target point E is less than the mission range of the spacecraft, as shown in formula (3): d PE ≤R P (3) Among them, d PE is the distance between the spacecraft P and the target E, R P It refers to the mission scope of spacecraft P. The mission failure condition of this scenario is set as the overlap between spacecraft P and threat area X, that is, the spacecraft is detected by the threat radar, as shown in formula (4): p t ∈X(4) Step 2: Based on the spacecraft path planning scenario model established in step 1, design the state space model and action space model of the spacecraft; Step 3: Based on the spacecraft path planning scenario model established in step 1, establish a dynamic warning area based on radar detection exposure risk; Step 4: According to the dynamic warning area based on radar detection exposure risk established in step 3, a dynamic time-varying LGVF guidance potential field based on radar detection threat is established; Step 5: Based on the spacecraft path planning scenario model established in step 1, an improved deep reinforcement learning SAC algorithm is proposed to provide real-time action strategies for the spacecraft to cope with unknown dynamic and complex scenarios; Step 6: Integrate the dynamic time-varying LGVF guidance potential field based on radar detection threat described in step 4 into the mobile planner based on deep reinforcement learning algorithm proposed in step 5 in the form of reward function to optimize the reinforcement learning decision process; Step 7: Load the optimized algorithm model proposed in step 6 into an online mission simulation scenario where the information is partially observable and unpredictable for testing, and improve the combined optimization method through the test effect feedback of the pursuit spacecraft performing the pursuit mission.

2. The LGVF-APF-SAC spacecraft path planning method based on dynamic threats according to claim 1 is characterized in that: The step 2 specifically includes the following steps: Step 2.1: Design of spacecraft state space model: Set the spacecraft's onboard GPS equipment and gyroscope to obtain its own position information and speed information; the threat radar payload equipment can obtain the spacecraft's position information and speed information within the detection range; Assume that the spacecraft state space model S contains a continuous bounded state space where x t and t is the coordinate of the spacecraft at time t, v t and ψ t are the speed and heading, and P target and X t They are the spacecraft's position information record of the target and the measurement information of the threat area; Step 2.2: Design of the spacecraft motion space model, specifically: The control input of the pursuit spacecraft is designed to be angular velocity and acceleration. It is assumed that the dynamic equation of the spacecraft sets the spacecraft to uniformly accelerated motion. The continuous action space a t As shown in formula (5): a t =[u t ,ψ t ]=[u t-1 +Δu,ψ t-1 +Dψ] T (5) The maximum angular velocity of the spacecraft is set to be less than 25.5 rad / sec, that is, ψ∈[-25.5,25.5], and the counterclockwise direction of the top view is taken as the positive direction.

3. The LGVF-APF-SAC spacecraft path planning method based on dynamic threats according to claim 1, characterized in that: The specific steps of step 3 to design and establish a dynamic warning area based on radar detection exposure risk according to the spacecraft pursuit mission scenario model established in step 1 are as follows: To simulate the dynamic scanning threat of radar, the anisotropy of radar beam was simulated according to the antenna gain at different angles of radar, so as to establish the early warning area of ​​spacecraft exposure risk based on radar detection, that is, the modeling of threat area; Assume that the center of the radar array is placed at the origin of the spherical coordinate system, where there are N array elements with uniform spacing d; the array element positions are shown in formula (6): Define the popularity vector v of the radar array k (k) is as shown in formula (7): where k is the amplitude of the wave number; Then, for a plane wave input with a propagation direction of a and a time domain frequency of radian 0 of ω, the frequency-wave number response function of the N-element ULA is as shown in formula (8): in is the frequency domain weighting coefficient of each array According to the frequency-wave number response function, the beam pattern function of the radar array can be obtained as shown in formula (9): The beam pattern of the radar array reflects the antenna gain of the radar array at different angles; in the following study, we will use the directional power function G(θ,φ)=|B(ω:θ,φ)| 2 To represent the antenna gain of the radar; using a directional antenna with a gain of G, the power transmitted to the radar signal processor by the antenna is shown in formula (10): Where Pt is the radar peak transmit power, G is the directional antenna gain, λ is the wavelength, σ is the radar cross-sectional area, and R is the distance radius from the radar. Using a directional antenna with a gain of G, let Smin represent the minimum detectable signal power, then the maximum radar detection distance Rmax is as shown in formula (11): Among them, Pt is the radar peak transmission power, G is the directional antenna gain, λ is the wavelength, and σ is the radar cross-sectional area; according to the antenna gain G at different angles, we can obtain the maximum detection distance Rmax of the radar at each angle; according to the standard parameters, simulation can be performed to obtain the detection distance of the radar at each angle; Considering the dynamic constraints, the radar detection area envelope is processed and approximated as an elliptical continuous area; the ellipse equation in polar coordinates can be used Description; where a is the major axis of the ellipse, and e is the eccentricity of the ellipse; by minimizing the residual Δθ between the envelope and the maximum detection distance of the radar at each angle, the parameters a and e can be solved to obtain the major axis a and minor axis of the ellipse. At each moment, the algorithm will recalculate the parameters of the warning area to simulate the changes in the radar's scanning boundaries, allowing us to define the boundaries of the elliptical area; at this point, the dynamic warning area is constructed.

4. The LGVF-APF-SAC spacecraft path planning method based on dynamic threats according to claim 1, characterized in that: The specific steps of the step 4 according to the dynamic warning area based on radar detection exposure risk established in step 3 and the dynamic time-varying LGVF guidance potential field based on radar detection threat are as follows: after obtaining the dynamic warning area, when constructing the potential field, it is necessary to stretch the original static potential field based on its dynamic parameters; the total potential field expression of the dynamic time-varying LGVF guidance potential field based on radar detection threat is shown in formula (12): U total (x,y)=w att ·U att (x,y)+w rep ·U rep (x,y) (12) Among them, w att and w rep are the weight coefficients of the gravitational field and the repulsive field respectively; U att with U rep represents the gravitational field and the repulsive field. The gravitational field is constructed based on the relative position of the spacecraft and the target. Its general form is shown in formula (13): The repulsive field is constructed based on the improved dynamic LGVF. Specifically, according to the position of the radar and the spacecraft and the Lyapunov stability principle, the Lyapunov function is as shown in formula (14): Use (x U ,y U ) and (x R ,y R ) represent the spacecraft and radar positions respectively, R d is the safe distance threshold between the spacecraft and the danger, is the straight-line distance between the spacecraft and the threat; to simulate the danger model in the real scene, we use the obtained radar detection risk parameters, i.e., the major and minor semi-axes a and b of the radar beam modeling, as the variation factors based on the coordinate stretching transformation, and let The limit cycle of the vector field changes from a regular circle to an ellipse; According to the Lyapunov stability principle, the flight trajectory of the spacecraft will converge to the set hovering area, that is, the modeled warning area; the constructed vector field will also readapt with the changes of the warning area parameters a and b, giving the spacecraft vector guidance that changes with dynamic threats; In the new potential field constructed, the improved dynamic time-varying LGVF is used as the repulsive field of the guiding potential field to guide the spacecraft to avoid danger; the Lyapunov function As a repulsive potential function 5. The LGVF-APF-SAC spacecraft path planning method based on dynamic threats according to claim 1, characterized in that: The specific steps of step 5 are as follows: The spacecraft movement planner is established through the SAC algorithm of deep reinforcement learning. Its online learning-offline training algorithm characteristics enable its model to be directly used for real-time planning. After learning the dynamic characteristics in the scene, it feeds back the policy output that meets the task requirements, provides the ability to handle dynamic unknown problems, and meets the current need for real-time optimization of the scene. The proposed entropy coefficient that is adaptive to different subtasks of the pursuit spacecraft improves the learning efficiency of the SAC algorithm and the tracking accuracy of the pursuit spacecraft. Its form is shown in formula (15); Among them, R(s t ,a t ) represents the state s t Take action a t The return value obtained, strategy π * The goal is to maximize the expectation of the total return value and maximize the entropy of each output action. α is the entropy coefficient, H(π(·|s t )) is the entropy term; The purpose of requiring the maximum entropy is mainly to randomize the strategy, that is, the probability of each output action is as balanced as possible, so that the intelligent agent needs to explore all possible optimal paths; the algorithm solves the optimal strategy by maximizing the expected return of the optimal strategy, so that the optimal control strategy of the spacecraft can be learned, and the entropy term is used to optimize the distribution of the strategy; a large number of sample sampling and back-propagation operations are required in the entire update process, and the algorithm can ultimately make the control strategy more flexible and robust.

6. The LGVF-APF-SAC spacecraft path planning method based on dynamic threats according to claim 1, characterized in that: The specific process of step 6 is as follows: After constructing a dynamic time-varying LGVF guidance potential field based on radar detection exposure risk, the potential field is integrated into the reinforcement learning framework in the form of a reward function. The potential field is used as the action benchmark in the offline training-online decision-making process of the SAC algorithm to evaluate the decision-making actions of reinforcement learning, thereby improving the SAC network parameters to generate better actions. The design reward function is shown in formula (16): r＝r a +r b +r c (16) Among them, r represents the total return value, r a represents the sparse reward function, which is the reward given when the pursuit spacecraft successfully reaches the target point and the reward given when the mission fails. Its form is shown in formula (17): r b Simulate the gradient descent process of the guidance potential field, give the agent a positive reward when the guidance potential field gradient decreases, and guide the spacecraft along the potential energy gradient descent direction of the guidance potential field to find the global minimum point, i.e., the target point; Its form is shown in (18): where ω pf is the weight coefficient of potential field reward, and are the potential energy values ​​of the guiding potential field at the previous moment and at the current moment respectively; r c is the designed guiding reward component. The process of the chasing spacecraft flying towards the escaping spacecraft is divided into two stages, as shown in formula (19): Among them, use and represents the previous distance and current distance between the spacecraft and the target, respectively, and introduces the weight μ1 to adjust the reward; stage 1 refers to the pursuit spacecraft not sensing the existence of a threat, at which time the pursuit spacecraft continues to move toward the target driven by the gravitational field; stage 2 refers to when a threat appears in the path of the vector pointing to the target by the pursuit spacecraft, at this time, a vector field model is established for the threat T according to the Lyapunov guidance vector field algorithm, and the vector field radius is designed to be d PT +w P , where w P refers to the fuselage width of the pursuit spacecraft, and the threat level is expressed in the form of an exponential function; at this time r c Guide the pursuit spacecraft to control the distance between it and the threat, otherwise the negative reward value generated will be added to the total reward value as a penalty after controlling the reward value dimension through the weight μ2.

7. The LGVF-APF-SAC spacecraft path planning method based on dynamic threats according to claim 1, characterized in that: The step 7 specifically includes the following steps: First, adjust the network model parameters and the mobile planner structure of the algorithm according to the running reward and feedback, adapt to the spacecraft mission scenario, and improve the autonomous motion planning framework described in step 5; Secondly, adjust the potential field model parameters described in step 4 according to the feedback results, so that the pursuit spacecraft's avoidance trajectory designed by the guidance potential field is more stable and more suitable for the spacecraft pursuit mission scenario; finally, continuously adjust the training network model parameters according to the reward function feedback to improve the model training efficiency and the pursuit spacecraft tracking accuracy.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the device where the computer-readable storage medium is located executes the dynamic threat-based LGVF-APF-SAC spacecraft path planning method described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a program that can be run on the processor, and when the processor executes the program, the dynamic threat-based LGVF-APF-SAC spacecraft path planning method as described in any one of claims 1-7 is implemented.

Images