Interception guidance strategy acquisition method and device, equipment and storage medium

Abstract

The invention provides an interception guidance strategy obtaining method and device, equipment and a storage medium, and the method comprises the steps: obtaining a first Bellman equation which is a separation strategy integral reinforcement learning Bellman equation obtained based on a nonlinear interception kinetic equation, a game value function and an iteration control strategy; acquiring control input data and state output data; and iteratively solving the first Bellman equation based on the control input data and the state output data to obtain an optimal interception guidance control strategy. Thus, under the condition of not depending on specific dynamic information, the optimal control strategy is found in a data driving mode, so that the purpose that the pursuit party performs high-precision interception on the high-maneuverability escape party is achieved through the optimal control strategy.

Description

Technical Field

[0001] The present application relates to the field of aircraft interception and guidance technology, and in particular to an interception and guidance strategy acquisition method, device, equipment and storage medium. Background Art

[0002] As a key part of the air defense system, interception and guidance strategy involves a series of complex technical operations and strategic decisions to accurately track and intercept incoming targets. Its core goal is to ensure that the missile system can quickly and accurately lock on to the target and complete the interception mission in the shortest possible time. Faced with the increasing mobility and uncertainty of targets in modern warfare, the importance of interception and guidance strategy has become increasingly prominent.

[0003] During the interception process, the missile not only needs to track the target according to its initial trajectory, but also needs to adjust the interception decision in real time according to the target's maneuvering strategy. For example, when the target suddenly changes its flight direction or speed, the interceptor missile must be able to respond quickly and adjust its own flight trajectory and speed to ensure that it always remains in the best interception position.

[0004] However, traditional interception and guidance strategies have the problem of insufficient guidance accuracy when facing highly maneuverable targets. Therefore, how to achieve high-precision interception of missiles when facing highly maneuverable targets has become a top priority. Summary of the invention

[0005] In view of this, the present application provides an interception guidance strategy acquisition method, device, equipment and storage medium to solve the technical problems raised by the background technology.

[0006] In a first aspect, an embodiment of the present application provides a method for acquiring an interception guidance strategy, including: obtaining a first Bellman equation, which is an off-strategy integral reinforcement learning Bellman equation obtained based on a nonlinear interception dynamics equation, a game value function, and an iterative control strategy; obtaining control input data and state output data; based on the control input data and the state output data, iteratively solving the first Bellman equation to obtain an optimal interception guidance control strategy.

[0007] In the second aspect, an embodiment of the present application provides an interception guidance strategy acquisition device, including: a first equation acquisition module, used to obtain a first Bellman equation, wherein the first Bellman equation is an off-strategy integral reinforcement learning Bellman equation obtained based on a nonlinear interception dynamics equation, a game value function, and an iterative control strategy; an input and output data acquisition module, used to obtain control input data and state output data; a control strategy generation module, used to iteratively solve the first Bellman equation based on the control input data and the state output data to obtain an optimal interception guidance control strategy.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory storing computer program instructions, wherein the computer program instructions, when executed by the processor, implement the steps of the interception guidance strategy acquisition method described in any embodiment of the present application.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer executable program is stored, wherein when the computer executable program is executed by a processor, the steps of the interception guidance strategy acquisition method described in any embodiment of the present application are implemented.

[0010] In summary, the interception guidance strategy acquisition method, device, equipment and storage medium provided by the embodiment of the present application have at least the following beneficial effects: the embodiment of the present application first obtains the first Bellman equation, which is an off-policy integral reinforcement learning Bellman equation obtained based on the nonlinear interception dynamics equation, the game value function and the iterative control strategy, and then obtains the control input data and the state output data, so as to iteratively solve the first Bellman equation based on the control input data and the state output data to obtain the optimal interception guidance control strategy. The embodiment of the present application can obtain the first Bellman equation based on the nonlinear interception dynamics equation, the game value function and the iterative control strategy, and realizes the generation of the optimal control strategy in a data-driven manner without relying on specific dynamics information, so that the control input is optimal, and the optimal control strategy can better realize the guided interception task of the maneuvering escape party, and the interception accuracy is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A schematic diagram showing a flow chart of a method for acquiring an interception guidance strategy provided by an embodiment of the present application;

[0013] Figure 2 A schematic diagram showing a flow chart of another interception guidance strategy acquisition method provided by an embodiment of the present application;

[0014] Figure 3 The present application provides a method for evaluating the weight parameters of a neural network. Convergence plot;

[0015] Figure 4The weight parameters of a pursuit neural network provided by the present application are shown as follows: Convergence plot;

[0016] Figure 5 The weight parameters of the neural network of the escape party provided in the embodiment of the present application are shown as follows: Convergence plot;

[0017] Figure 6 A three-dimensional schematic diagram showing a missile interception target implemented according to an optimal interception control strategy provided by an embodiment of the present application;

[0018] Figure 7 A schematic diagram showing the structure of an interception guidance strategy acquisition device provided in an embodiment of the present application is shown;

[0019] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] In order to make the above and other features and advantages of the present application more clear, the present application is further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0021] In the following description, many specific details are set forth to provide a thorough understanding of the present application. However, it is apparent to those skilled in the art that specific details need not be adopted to practice the present application. In other cases, well-known steps or operations are not described in detail to avoid blurring the present application.

[0022] Traditional guidance methods have the following problems when facing highly maneuverable targets: (1) The tracking guidance method cannot achieve omnidirectional interception of the target and is generally used for missiles attacking slow-moving or stationary targets; (2) The implementation of the parallel approach method requires accurate measurement of the state information of the missile and the target, strictly satisfying the motion relationship of the parallel approach method, which places strict requirements on the guidance system and is difficult to achieve in engineering; (3) The proportional guidance method has poor guidance accuracy when the target is fast, maneuvering, and launched at a large off-axis angle; (4) The traditional guidance method lacks performance indicators during the guidance process, and the control input is not optimal; (5) The optimal guidance method is optimal, but the solution process requires the accurate dynamic model of both parties to be known.

[0023] In response to the above problems, the inventors proposed a method for acquiring an interception guidance strategy, which is a one-to-one interception guidance strategy acquisition method that combines reinforcement learning and differential game. The method can be applied to scenarios where interceptors intercept highly maneuverable targets, thereby overcoming the shortcomings of various existing guidance methods, such as insufficient guidance accuracy when facing highly maneuverable targets, non-optimality, and the need for a known and accurate dynamic model, and improving the missile interception guidance performance.

[0024] It should be noted that in the embodiments of the present application, the interception scenario of the pursuer can be regarded as a pursuit scenario of the pursuer and the escaper, the pursuer can be regarded as the pursuer, and the escaper can be regarded as the escaper.

[0025] Before introducing the interception guidance strategy acquisition method provided in the embodiment of the present application, the coordinate system involved in the interception guidance strategy acquisition method is first described. This application adopts the ground inertial coordinate system Body coordinate system And the velocity coordinate system (airflow coordinate system) To describe the movements of the pursuer and the fleeing party.

[0026] Ground inertial coordinate system E I The origin O is the launching point of the pursuing party. I , The axis points to the projection of the launch direction in the horizontal plane, that is, the intersection of the ballistic plane and the horizontal plane. The axis is located at the In the vertical plane of the axis and perpendicular to The axis is vertically upward, The axis orientation is determined according to the right-hand rule.

[0027] Body coordinate system E B Origin O B is the center of mass of the pursuer or escaper, The axis points forward along the longitudinal axis of the pursuer or escaping party, and The axis is vertical and points upwards, The axis orientation is determined according to the right-hand rule.

[0028] Velocity coordinate system (airflow coordinate system) E A Origin O A Consistent with the body coordinate system, The axis points to the direction of the center of mass velocity. The axis is located in the longitudinal symmetry plane of the pursuer or escaper and is perpendicular to The axis points upward. The axis orientation is determined according to the right-hand rule.

[0029] On the one hand, an embodiment of the present application provides an interception guidance strategy acquisition method, which can be implemented by an interception guidance strategy acquisition device. Figure 1 A flow chart of a method for acquiring an interception guidance strategy provided by an embodiment of the present application is shown as follows: Figure 1 As shown, the interception guidance strategy acquisition method may include the following steps.

[0030] S11, obtain the first Bellman equation.

[0031] In one embodiment, the first Bellman equation is an off-policy integral reinforcement learning Bellman equation obtained based on a nonlinear intercept dynamics equation, a game value function, and an iterative control strategy. The first Bellman equation describes the relationship between the update of the game value function and the current reward and subsequent state values. The first Bellman equation can be used to obtain the optimal control input of the game under the condition of completely unknown system dynamics parameters.

[0032] The nonlinear interception dynamics equation describes the nonlinear relationship between the state change of the pursuer and the escaper and the control input and the current state. The game value function can be equal to the game performance function constructed according to zero-sum differential game theory. The iterative control strategy can include an update strategy for the interceptor control strategy and an update strategy for the escaper control strategy.

[0033] S12, obtaining control input data and state output data.

[0034] In one embodiment, the control input data may include the initial control strategy of the interceptor and the escaper, expressed as The state output data may be multi-dimensional interception state space data, including but not limited to: the position vector difference and velocity vector difference between the pursuer and the escaper, the speed of the pursuer, the track inclination of the pursuer, the track azimuth of the pursuer, the speed of the escaper, the track inclination of the escaper, and the track azimuth of the escaper, etc. The state output data may be system output state data recorded at a time interval T, wherein T may be set according to system requirements.

[0035] S13, based on the control input data and the state output data, iteratively solve the first Bellman equation to obtain the optimal interception guidance control strategy.

[0036] In one embodiment, iterative solution may refer to starting from any initial control strategy, repeatedly applying the first Bellman equation based on state output data to update the game value function of each state and the control strategy of both parties until convergence to the optimal control strategy. The optimal interception guidance control strategy may refer to the optimal control input strategy that satisfies the Nash equilibrium condition. The optimal interception guidance control strategy has optimality.

[0037] Among them, the Nash equilibrium condition can be expressed as:

[0038]

[0039] Among them, J represents the game performance function, V * (x(t)) represents the optimal game value function.

[0040] It should be noted that the convergence of the control strategies of both parties to the optimal control strategy can be determined based on the convergence condition. The convergence condition can be that the game value function of the kth time converges to the optimal game value function, or the positions of the two parties are close, that is, the distance between the two parties is less than a preset threshold. Among them, k is not less than 0.

[0041] In the above embodiment, the first Bellman equation is obtained, which is an off-policy integral reinforcement learning Bellman equation obtained based on the nonlinear interception dynamics equation, the game value function and the iterative control strategy, so that the interception problem can be more accurately described and solved by using the first Bellman equation based on the nonlinear interception dynamics equation, the game value function and the iterative control strategy, and a framework suitable for off-policy integral reinforcement learning is formed. The control input data and the state output data are obtained, and the first Bellman equation is iteratively solved based on the control input data and the state output data to obtain the optimal interception guidance control strategy, so that the off-policy integral reinforcement learning method can be used to find the optimal control strategy in a data-driven manner without relying on specific dynamics information, so that the control input is optimal, and the optimal control strategy can better achieve the guided interception task of the maneuvering escape party, and the interception accuracy is high.

[0042] In some embodiments, the first Bellman equation can be expressed as the following equation.

[0043]

[0044] Among them, u M and u T represents the initial input control strategy, and is the optimal input control strategy for the kth iteration, and is the k+1th optimal input control strategy, x represents the state output data, T represents the time interval, α represents the angle of attack, t represents the start time of state sampling, Q C represents the augmented state weight coefficient, R M represents the first control weight coefficient, R T represents the second control weight coefficient, and τ represents the integration time.

[0045] In the above formula (2), the dynamic information f,g is not included T ,g M , which only contains the fixed control input u of the system M ,u TAnd the output state information x, by solving formula (2) can simultaneously update the game value function V k and the optimal control input of both parties The initial control input u in the equation M ,u T With updated control strategies Can be different. The initial control strategy needs to be a stable control strategy with exploration noise. The initial control strategy is used to collect different state output data.

[0046] The off-policy integral reinforcement learning algorithm proposed by the inventor can be divided into two stages. First, in the first stage, a fixed initial exploratory control strategy is applied to the system, and the system output state data at time intervals T are collected, and the output state data is processed into a data matrix form. Secondly, in the second stage, without the need for system dynamics knowledge, the information collected in the first stage is repeatedly used to update the control strategy sequence until it converges to the optimal control strategy. In addition, since the first Bellman equation is a scalar equation, after collecting sufficient system data, it can be solved by a linear matrix solution method.

[0047] Therefore, in some embodiments, S13, iteratively solving the first Bellman equation based on the control input data and the state output data to obtain the optimal interception guidance control strategy may include: converting the input control strategy and the state output data into a first data matrix and a second data matrix respectively; fitting the game value function to be updated, the pursuit party control strategy and the escape party control strategy in the first Bellman equation based on the evaluation neural network, the pursuit party neural network and the escape party neural network respectively to obtain the fitted game value function to be updated, the fitted pursuit party control strategy and the fitted escape party control strategy; converting the first Bellman equation into a fitting equation based on the fitted game value function to be updated, the fitted pursuit party control strategy and the fitted escape party control strategy; converting the fitting equation into a linear matrix equation based on the first data matrix, the second data matrix, the fitted game value function to be updated, the fitted pursuit party control strategy and the fitted escape party control strategy; solving the linear matrix equation to obtain the optimal interception guidance control strategy.

[0048] Here, the first data matrix can be a matrix representation of the control input strategy, which can include the control data matrix of the pursuer and the control data matrix of the escaper, and can be represented by the basis function of the neural network of the pursuer or the basis function of the neural network of the escaper. The first data matrix can be specifically represented by the following formula.

[0049]

[0050] in, represents the control data matrix of the pursuer, represents the control data matrix of the escaped party, e represents the index, t1 to t q represents the start or end time of the integration of each element in the data matrix, τ represents the integration time, φ(x(·)) represents the basis function of the pursuit neural network, represents the basis function of the escape neural network, u M represents the control strategy of the pursuer, u T The control strategy of the escapee.

[0051] The second data matrix may be a matrix representation of the state output data and may include δ xx ,I xx ,I φφ and It can be represented by the basis function of the evaluation neural network, the basis function of the pursuit neural network or the basis function of the escape neural network. The second data matrix can be represented by the following formula.

[0052]

[0053] Among them, δ xx represents the first component of the second data matrix, I xx represents the second component of the second data matrix, I φφ represents the third component of the second data matrix, represents the fourth component of the second data matrix, α represents the angle of attack, T represents the time interval, φ(x(·)) represents the basis function of the neural network of the pursuer, represents the basis function of the escape neural network, and σ(x(·)) represents the basis function of the evaluation neural network.

[0054] The "evaluation neural network", "pursuer neural network" and "escaper neural network" involved in the embodiment of the present application are all neural networks designed according to specific task requirements. In the embodiment of the present application, the game value function V to be updated in the first Bellman equation is approximately fitted by three neural networks, namely, the evaluation neural network, the pursuit neural network and the escaper neural network. k , Pursuer Control Strategy And escape control strategy Get the fitted game value function to be updated Fitted pursuer control strategy And the fitted escape control strategy

[0055] Fitted game value function to be updated It can be expressed as formula (9).

[0056]

[0057] Fitted pursuer control strategy It can be expressed as formula (10).

[0058]

[0059] Fitting escape control strategy It can be expressed as formula (11).

[0060]

[0061] Among them, σ(x), φ(x), are the basis functions of the evaluation neural network, the pursuit neural network, and the escape neural network, respectively. They are the constant weight matrices of the evaluation neural network, the pursuit neural network, and the escape neural network, and are the objects updated in the iterative process.

[0062] The "fitting equation" involved in the embodiment of the present application can be a game value function to be updated based on the fitting: Fitted pursuer control strategy And the fitted escape control strategy The Bellman equation corresponding to the first Bellman equation is represented by . That is, by substituting formula (9) to formula (11) into the first Bellman equation, the following fitting equation can be obtained.

[0063]

[0064] Among them, each term in the fitting equation can be expressed by Krone product.

[0065]

[0066] Among them, R T represents the first control weight coefficient, R M represents the second control weight coefficient, I φn represents the data matrix of φ(x), express , vec(·) represents a vector.

[0067] The "linear matrix equation" involved in the embodiment of the present application can be represented by the first data matrix, the second data matrix, the fitted game value function to be updated, the fitted pursuit party control strategy, and the fitted escape party control strategy. Substituting the first data matrix, the second data matrix, the fitted game value function to be updated, the fitted pursuit party control strategy, and the fitted escape party control strategy into the fitting equation, the following linear equation can be obtained.

[0068]

[0069] Among them, Η ik Matrix and Γ ik The matrices can be expressed as follows.

[0070]

[0071] In the embodiment of the present application, since formula (18) is a scalar equation, it can be solved by a linear matrix equation after collecting sufficient system data. It should be noted that there are many methods for solving the linear matrix equation, such as the least squares method, ridge regression and other linear regression solving methods.

[0072] In the above embodiment, by combining the Bellman equation with three neural networks, complex nonlinear problems and uncertainties can be handled, making the interception guidance strategy acquisition method more flexible and universal. The solution method of the linear matrix equation can significantly improve the computational efficiency, especially when processing large-scale data, which can significantly reduce the computing time and resource consumption and reduce the difficulty of solving the Bellman equation. Through data-driven methods, it can automatically adapt to changes in different scenarios and conditions, and there is no need to perform cumbersome manual adjustments to the system model. Through iterative optimization and neural network fitting, the optimal solution can be gradually approached to improve the accuracy and performance of the interception guidance strategy.

[0073] In some embodiments, before obtaining the first Bellman equation at S11, the interception guidance strategy acquisition method may further include: constructing a nonlinear interception dynamics equation based on the pursuit party motion model and the escape party motion model, and obtaining a game value function of the zero-sum game theory.

[0074] In one embodiment, the pursuit party motion model and the escape party motion model can both be six-degree-of-freedom dynamic models. The pursuit party motion model and the escape party motion model can be expressed by a general formula. The general formula can be expressed as the following formula (20).

[0075]

[0076] The subscript i can be the pursuer M or the escaper T, T i represents the constant thrust of the pursuer or escaper, V i represents the speed of the pursuer or the escapee, γi represents the inclination of the track of the pursuer or the escapee, ψi represents the azimuth of the track of the pursuer or the escapee, αi represents the angle of attack of the pursuer or the escapee, φi represents the sideslip angle of the pursuer or the escapee, mi represents the mass parameter of the pursuer or the escapee, and g represents the local gravitational angular velocity constant.

[0077] L i and D iThey represent the lift or drag of the pursuer or escaping party respectively, and can be expressed as the following formula (21).

[0078]

[0079] Among them, c Li is the lift coefficient, c Di is the drag coefficient, ρ is the atmospheric density, s i Indicates the reference cross-sectional area.

[0080] Let p Represents the position vector of the pursuer or escaper, represents the velocity vector of the pursuer or escaper, so the interception state space is as follows: Formula (22).

[0081]

[0082] Then, the interception guidance control strategy u i It can be expressed as formula (23).

[0083]

[0084] Among them, T i represents the constant thrust of the pursuer or escaper, α i represents the angle of attack of the pursuer or escaper, φ i Indicates the sideslip angle of the pursuing or escaping party.

[0085] The nonlinear intercept dynamics equation can be a nonlinear equation, which can be expressed as formula (24).

[0086]

[0087] in, They represent the control strategies of the pursuer and the escaper after k iterations. f represents the function f(x), g T Represents the function g T (x), and g M Indicates g M (x).

[0088]

[0089]

[0090] in, It represents the atmospheric density used by the pursuer to calculate the drag. It represents the atmospheric density used by the escape party to calculate the drag. represents the atmospheric density used by the escape party to calculate thrust, Indicates the atmospheric density used by the pursuer to calculate drag.

[0091] The "game value function" involved in the implementation of this application can be equal to the game performance function of zero-sum game theory. The game performance function can be expressed as formula (28).

[0092]

[0093] V(x(t))=J(u M ,u T ) (29)

[0094] Among them, V(x(t)) represents the game value function, Q represents the state weight matrix, R M represents the first control weight coefficient, R T represents the second control weight coefficient, and α represents the discount factor.

[0095] In some embodiments, constructing a nonlinear interception dynamics equation based on the pursuit party motion model and the escape party motion model may include: constructing an initial nonlinear interception dynamics equation based on the pursuit party motion model and the escape party motion; converting the initial nonlinear interception dynamics equation into a nonlinear interception dynamics equation based on the time differential of the initial nonlinear interception dynamics equation.

[0096] Here, the initial nonlinear interception dynamics equation can be derived through formula (20) to formula (22), and the initial nonlinear interception dynamics equation can be expressed as the following formula (30).

[0097]

[0098] Where f(x) can be expressed as formula (25), g T (x) can be expressed as formula (26), and g M (x) can be expressed as formula (27), to Represent each term in formula (22) respectively.

[0099] The initial nonlinear interception dynamics equation is time differentiated to transform the initial nonlinear interception dynamics equation (30) into the nonlinear interception dynamics equation (24).

[0100] In the above embodiment, by constructing the initial nonlinear interception dynamics equation and converting it into the final nonlinear interception dynamics equation, an important theoretical basis and mathematical model can be provided for the subsequent interception guidance control strategy design. This helps to achieve more accurate and efficient interception tasks and improve the performance and reliability of the interception guidance strategy.

[0101] In some embodiments, S11, obtaining the first Bellman equation, may include: obtaining the second Bellman equation based on the nonlinear intercept dynamics equation and the game value function; and converting the second Bellman equation into the first Bellman equation based on an iterative control strategy.

[0102] In some embodiments, the second Bellman equation can be obtained by differentiating the game value function and substituting the nonlinear intercept dynamics equation into the differentiated game value function. The game value function can be a game value function of zero-sum game theory.

[0103] The second Bellman equation can be expressed as follows

[0104]

[0105] The meanings of the parameters in formula (30) are the same as those in the first Bellman equation.

[0106] The iterative control strategy may refer to a k-th interception control strategy and a k+1-th interception control strategy variation strategy. The iterative control strategy may be expressed as the following formula (32).

[0107]

[0108] in, represents the control input strategy of the k+1th escape party, It represents the control input strategy of the pursuer at the k+1th time. Represents the k-th game value function.

[0109] In some embodiments, based on an iterative control strategy, the second Bellman equation is converted into a first Bellman equation, including: based on an iterative control strategy, the second Bellman equation is converted into an intermediate equation; multiplying both sides of the intermediate equation by an exponential function and integrating it to obtain the first Bellman equation.

[0110] In some embodiments, substituting the iterative control strategy into the second Bellman equation can obtain an intermediate equation, where the intermediate equation can be expressed as formula (33).

[0111]

[0112] The “exponential function” involved in the embodiments of the present application can be expressed as e -α(τ-t) , the integral can refer to the integral of the differentiated game value function within the time interval T. In this way, both sides of the intermediate equation are multiplied by e -α(τ-t) , and integrate over the time interval to obtain the second Bellman equation.

[0113] It should be noted that the first Bellman equation is a key equation in the off-policy integral reinforcement learning framework, which combines the advantages of nonlinear intercept dynamics, game theory and iterative control strategy. Compared with the second Bellman equation, the first Bellman equation focuses more on finding the optimal control strategy in the iterative process and describes the relationship between state transition and expected return in an integral form.

[0114] In the above embodiment, by converting the second Bellman equation into the first Bellman equation, an important theoretical basis and mathematical model can be provided for subsequent control strategy design and optimization.

[0115] In order to fully understand the interception guidance strategy acquisition method provided by the embodiment of the present application, the embodiment of the present application provides another interception guidance strategy acquisition method on the other hand. Figure 2 A flow chart of another interception guidance strategy acquisition method provided by an embodiment of the present application is shown as follows: Figure 2 The interception guidance strategy acquisition method may include the following steps.

[0116] S21, initializing parameters, using two admissible control strategies as initial control strategies.

[0117] S22, adding exploration noise to the initial control strategy and inputting it into the system, and collecting the control input data and state output data of the system.

[0118] The system may refer to the nonlinear intercept dynamics equation constructed based on the above.

[0119] S23, using the collected control input data and state output data to iteratively solve the first Bellman equation, and simultaneously update the game value function and the optimal input control strategy of both parties.

[0120] S24, judging whether the convergence condition is satisfied. If the condition is satisfied, output the kth optimal input control strategy, if the condition is not satisfied, set k=k+1, take the kth optimal input control strategy as the initial control strategy, and return to S22.

[0121] In the above embodiment, the online off-policy integral reinforcement learning algorithm is used to iteratively solve the first Bellman equation of the model-free off-policy integral reinforcement learning, thereby obtaining the optimal input control strategy that can achieve the Nash equilibrium condition. In this way, compared with the traditional guidance control method, the process of obtaining the interception guidance strategy of the present application has the advantage of being model-free, and the interception guidance strategy obtained is optimal.

[0122] On the other hand, an embodiment of the present application provides a simulation experiment for verifying the effectiveness of the optimal interception guidance control strategy obtained in the aforementioned embodiment of the present application.

[0123] The initial simulation parameters include: the initial position of the missile Initial Speed Initial track inclination Initial track deviation Initial position of the target Initial Speed Initial track inclination Initial track deviation And dynamic parameters, missile: mass parameter m M =150kg, lift coefficient c LM =35rad, drag coefficient c DM =0.74, reference cross-sectional area s M =0.0324m 2 , thrust T M =5880N; Target: mass parameter m T =7500kg, lift coefficient c LT =4.01rad, drag coefficient c DT =0.0169, reference cross-sectional area s T =26m 2 , thrust T T =65000N, local gravity acceleration constant g = 9.8m / s 2 , atmospheric density ρ = 0.9096 kg / m 3 The performance index function parameters are Q = 100I, R M =5I,R T =I, where I is the identity matrix.

[0124] The output matrix is ​​defined as follows:

[0125] The exploration noise is set to the sum of complex trigonometric function signals. The neural network basis function is defined as φ(x)=x, The initial value of the neural network constant weight matrix is

[0126]

[0127] The simulation results are as follows Figures 3 to 6 As shown. Among them, Figure 3 The present application provides a method for evaluating the weight parameters of a neural network. Convergence diagram, Figure 4 The weight parameters of a pursuit neural network provided by the present application are shown as follows: Convergence plot. Figure 5 The weight parameters of the neural network of the escape party provided in the embodiment of the present application are shown as follows: Convergence diagram. The horizontal axis represents the number of iterations, and the vertical axis represents the convergence value of the weight parameter. Figures 3 to 5 It can be seen that all three neural networks converge at 15 iterations, so that the optimal interception control strategy can be output at the 15th iteration. In this way, the purpose of quickly outputting the optimal interception control strategy can be achieved through the off-policy integral reinforcement learning algorithm.

[0128] Figure 6 A three-dimensional schematic diagram of a missile interception target implemented according to an optimal interception control strategy provided by an embodiment of the present application is shown. Figure 6 As shown, the missile and the target fly based on the optimal interception control strategy, which can make the final position of the missile close to the position of the target, and the distance between the two is within the allowable error range, thereby verifying the accuracy of the optimal interception control strategy.

[0129] In the embodiment of the present application, first, by modeling the relative motion model of the interception scene, a nonlinear interception dynamics equation is obtained, and a differential game modeling is performed based on the relative motion model, and a zero-sum game under the interception problem is defined. And the theoretical optimal control strategy of the game is solved by the minimum principle. Secondly, in order to solve the optimal interception strategy of the game under the condition that the dynamic information of the relevant system is unknown, a reinforcement learning framework is introduced, and the model-free off-strategy integral reinforcement learning Bellman equation is obtained by theoretical derivation, and a corresponding online off-strategy integral reinforcement learning algorithm is proposed. Finally, the system output state information is collected by stabilizing the control input plus the exploration noise, and the game value function, the pursuer strategy, and the escaper strategy are fitted by using three neural networks, and the integral reinforcement learning Bellman equation is iteratively solved until the neural network converges, thereby obtaining the optimal interception guidance control strategy, and the effectiveness of the neural network parameters obtained by the reinforcement learning algorithm and the effectiveness of the optimal interception guidance control strategy are verified by simulation experiments.

[0130] Another aspect of the embodiments of the present application provides an interception guidance strategy acquisition device. Figure 7 A schematic diagram of the structure of an interception guidance strategy acquisition device provided in an embodiment of the present application is shown as follows: Figure 7 As shown, the interception guidance strategy acquisition device 70 may include the following modules.

[0131] The first equation acquisition module 71 is used to acquire the first Bellman equation, where the first Bellman equation is an off-policy integral reinforcement learning Bellman equation obtained based on a nonlinear intercept dynamics equation, a game value function, and an iterative control strategy.

[0132] An input and output data acquisition module 72, used to acquire control input data and state output data;

[0133] The control strategy generation module 73 is used to iteratively solve the first Bellman equation based on the control input data and the state output data to obtain an optimal interception guidance control strategy.

[0134] In the above embodiment, the first Bellman equation is obtained, which is an off-policy integral reinforcement learning Bellman equation obtained based on a nonlinear interception dynamics equation, a game value function, and an iterative control strategy, so that the interception problem can be more accurately described and solved by using the first Bellman equation based on the nonlinear interception dynamics equation, the game value function, and the iterative control strategy, and a framework suitable for off-policy integral reinforcement learning is formed. The control input data and the state output data are obtained, and the first Bellman equation is iteratively solved based on the control input data and the state output data to obtain the optimal interception guidance control strategy, so that the off-policy integral reinforcement learning method can be used to find the optimal control strategy in a data-driven manner without relying on a specific system model, so that the control input is optimal, and the optimal control strategy can better achieve the guided interception task of the maneuvering escape party, and the interception accuracy is high.

[0135] In some embodiments, the control strategy generation module 73 may include the following sub-modules.

[0136] The data matrix conversion submodule is used to convert the control input data and the state output data into a first data matrix and a second data matrix respectively.

[0137] The fitting submodule is used to fit the game value function to be updated, the pursuit party control strategy and the escape party control strategy in the first Bellman equation based on the evaluation neural network, the pursuit party neural network and the escape party neural network, respectively, to obtain the fitted game value function to be updated, the fitted pursuit party control strategy and the fitted escape party control strategy.

[0138] The fitting equation conversion submodule is used to convert the first Bellman equation into a fitting equation based on the fitted game value function to be updated, the fitted pursuit party control strategy and the fitted escape party control strategy, wherein each term in the fitting equation is represented by a Krone product.

[0139] The linear matrix conversion submodule is used to convert the fitting equation into a linear matrix equation based on the first data matrix, the second data matrix, the fitted game value function to be updated, the fitted pursuit party control strategy and the fitted escape party control strategy.

[0140] The linear equation solving submodule is used to solve the linear matrix equations to obtain the optimal interception guidance control strategy.

[0141] In some embodiments, the first equation acquisition module 71 may include several sub-modules.

[0142] A second equation obtaining submodule, used for obtaining a second Bellman equation based on the nonlinear intercept dynamics equation and the game value function;

[0143] The equation conversion submodule is used to convert the second Bellman equation into the first Bellman equation based on an iterative control strategy.

[0144] In some embodiments, the equation conversion submodule may include the following units.

[0145] An intermediate equation conversion unit, used for converting the second Bellman equation into an intermediate equation based on an iterative control strategy;

[0146] The first equation obtains a unit for multiplying both sides of the intermediate equation by an exponential function and integrating the equation to obtain the first Bellman equation.

[0147] In some embodiments, the interception guidance strategy acquisition device 70 may also include the following modules.

[0148] A nonlinear equation building module is used to build nonlinear interception dynamics equations based on the pursuit party motion model and the escape party motion model;

[0149] Function acquisition module, used to obtain the game value function of zero-sum game theory.

[0150] In some embodiments, the nonlinear equation building module may include the following sub-modules.

[0151] The initial equation construction submodule is used to construct the initial nonlinear interception dynamics equation based on the pursuit party motion model and the escape party motion model.

[0152] The nonlinear equation sub-conversion module is used for converting the initial nonlinear interception dynamics equation into the nonlinear interception dynamics equation based on the time differentiation of the initial nonlinear interception dynamics equation.

[0153] In some embodiments, the first Bellman equation is:

[0154]

[0155] Among them, u M and u T represents the initial input control strategy, and is the optimal input control strategy for the kth iteration, and is the k+1th optimal input control strategy, and x represents the output state information.

[0156] It should be understood that the specific features, operations and details described hereinabove with respect to the method of the present application may also be similarly applied to the device and system of the present application, or vice versa. In addition, each step of the method of the present application described above may be performed by a corresponding component or unit of the device or system of the present application.

[0157] It should be understood that each module / unit of the device of the present application can be implemented in whole or in part by software, hardware, firmware or a combination thereof. Each module / unit can be embedded in the processor of the electronic device or independent of the processor in the form of hardware or firmware, or can be stored in the memory of the electronic device in the form of software for the processor to call to perform the operation of each module / unit. Each module / unit can be implemented as an independent component or module, or two or more modules / units can be implemented as a single component or module.

[0158] According to another aspect of the embodiments of the present application, an electronic device is provided. Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. Figure 8 As shown, the electronic device 80 includes a processor 81 and a memory 82 storing computer program instructions. When the processor 81 executes the computer program instructions, each step of the above-mentioned method for implementing the dressing scene is implemented. The electronic device 80 can be generally a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities.

[0159] In one embodiment, the electronic device 80 may include a processor, a memory, a network interface, a communication interface, etc. connected through a system bus. The processor of the electronic device 80 may be used to provide necessary computing, processing and / or control capabilities. The memory of the electronic device 80 may include a non-volatile storage medium and an internal memory. The non-volatile storage medium may store an operating system, a computer program, etc. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the electronic device 80 may be used to connect and communicate with external devices through a network. When the computer program is executed by the processor, the steps of the interception guidance strategy acquisition method of the present application are executed.

[0160] The present application provides a computer-readable storage medium on which an interception guidance strategy program is stored. When the vehicle-mounted conference management program is executed by a processor, the steps of the interception guidance strategy acquisition method provided in any of the above embodiments are implemented.

[0161] Those skilled in the art will appreciate that the method steps of the present application can be performed by instructing related hardware such as the electronic device 80 or a processor through a computer program, and the computer program can be stored in a non-temporary computer-readable storage medium, and the computer program causes the steps of the present application to be performed when it is executed. Depending on the circumstances, any reference to memory, storage, or other media herein may include non-volatile or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0162] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for acquiring an interception guidance strategy, characterized in that: include: Obtaining a first Bellman equation, where the first Bellman equation is an off-policy integral reinforcement learning Bellman equation obtained based on a nonlinear intercept dynamics equation, a game value function, and an iterative control strategy; Obtain control input data and status output data; Based on the control input data and the state output data, the first Bellman equation is iteratively solved to obtain an optimal interception guidance control strategy.

2. The method for acquiring the interception guidance strategy according to claim 1, characterized in that: The iteratively solving the first Bellman equation based on the control input data and the state output data to obtain an optimal interception guidance control strategy includes: Converting the control input data and the state output data into a first data matrix and a second data matrix respectively; Based on the evaluation neural network, the pursuit party neural network and the escape party neural network, the game value function to be updated, the pursuit party control strategy and the escape party control strategy in the first Bellman equation are fitted respectively to obtain the fitted game value function to be updated, the fitted pursuit party control strategy and the fitted escape party control strategy; Based on the fitted game value function to be updated, the fitted pursuit party control strategy and the fitted escape party control strategy, converting the first Bellman equation into a fitting equation, wherein each term in the fitting equation is represented by a Krone product; Based on the first data matrix, the second data matrix, the fitted game value function to be updated, the fitted pursuit party control strategy, and the fitted escape party control strategy, converting the fitting equation into a linear matrix equation; Solve the linear matrix equation to obtain the optimal interception guidance control strategy.

3. The interception guidance strategy acquisition method according to claim 1, characterized in that: The obtaining of the first Bellman equation comprises: Based on the nonlinear intercept dynamics equation and the game value function, a second Bellman equation is obtained; Based on an iterative control strategy, the second Bellman equation is transformed into the first Bellman equation.

4. The method for acquiring the interception guidance strategy according to claim 3, characterized in that: The step of converting the second Bellman equation into the first Bellman equation based on an iterative control strategy comprises: Based on an iterative control strategy, the second Bellman equation is converted into an intermediate equation; Multiplying both sides of the intermediate equation by an exponential function and integrating the equation yields the first Bellman equation.

5. The method for acquiring interception guidance strategy according to claim 1, characterized in that: Before obtaining the first Bellman equation, the method further includes: Based on the pursuit party motion model and the escape party motion model, the nonlinear interception dynamics equation is constructed; Get the game value function for zero-sum game theory.

6. The method for acquiring the interception guidance strategy according to claim 5, characterized in that: The nonlinear interception dynamics equation is constructed based on the pursuit party motion model and the escape party motion model, including: Based on the pursuit party motion model and the escape party motion model, the initial nonlinear interception dynamics equation is constructed; Based on the time differentiation of the initial nonlinear interception dynamics equation, the initial nonlinear interception dynamics equation is transformed into a nonlinear interception dynamics equation.

7. The method for acquiring an interception guidance strategy according to any one of claims 1 to 6, characterized in that: The first Bellman equation is: Among them, u M and u T represents the initial input control strategy, and is the optimal input control strategy for the kth iteration, and is the k+1th optimal input control strategy, x represents the output state information, Q c represents the augmented state weight coefficient, R M represents the first control weight coefficient, R T represents the second control weight coefficient, α represents the discount factor, T represents the time interval, V k (x(t)) represents the k-th game value function, and τ represents the integration time.

8. An interception guidance strategy acquisition device, characterized in that: include: A first equation acquisition module is used to acquire a first Bellman equation, where the first Bellman equation is an off-policy integral reinforcement learning Bellman equation obtained based on a nonlinear intercept dynamics equation, a game value function, and an iterative control strategy; Input and output data acquisition module, used to acquire control input data and state output data; A control strategy generation module is used to iteratively solve the first Bellman equation based on the control input data and the state output data to obtain an optimal interception guidance control strategy.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory storing computer program instructions, wherein the computer program instructions, when executed by the processor, implement the steps of the interception guidance strategy acquisition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer-executable program, wherein when the computer-executable program is executed by a processor, the steps of the interception guidance strategy acquisition method according to any one of claims 1 to 7 are implemented.

Images