Model training method and planning guidance method based on maximum entropy reinforcement learning algorithm

Abstract

The present invention provides a model training method and planning guidance method based on maximum entropy reinforcement learning algorithm, including: constructing an action review model, the action review model includes an action network module and a review network module; the action review model obtains a decision process vector group according to a state vector, and the state vector is obtained from a preset environment; according to the decision process vector group, the action network module and the review network module are updated under the principle of maximum entropy, and the optimal strategy result is output. Under the principle of maximum entropy, the present invention avoids calculating the overall space, greatly reducing the complexity of learning.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model training method and a planning guidance method based on a maximum entropy reinforcement learning algorithm. Background Art

[0002] The terminal guidance algorithm is mainly used at the end of the interception process of the interceptor to the target. The planning guidance method gives the control amount acting on the interceptor to control the position of the interceptor to approach the position of the target, thereby achieving the interception of the target. Giving the interceptor the ability to attack the target at the desired angle can increase the success rate of intercepting the target. In an ideal environment, when the guidance law trained by deep learning is applied to the actual interceptor to intercept the target, the interceptor is basically unable to effectively intercept the target. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides a model training method and a planning guidance method based on a maximum entropy reinforcement learning algorithm.

[0004] The training method is characterized by comprising:

[0005] S100: constructing an action review model, wherein the action review model includes an action network module and a review network module;

[0006] S200: The action review model obtains a decision process vector group according to a state vector, where the state vector is obtained from a preset environment;

[0007] S300: According to the decision process vector group, the action network module and the comment network module are updated under the principle of maximizing entropy, and the optimal strategy result is output.

[0008] According to an embodiment of the present invention, the above-mentioned construction of the action review model includes:

[0009] Constructing the above-mentioned action network module, wherein the above-mentioned action network module includes an action sub-network and an action configuration storage sub-network, and the above-mentioned action configuration storage sub-network is used to store parameters of the above-mentioned action sub-network;

[0010] The comment network module is constructed, and the comment network module includes a first comment sub-network, a second comment sub-network and a comment configuration storage sub-network, and the comment configuration storage sub-network is used to store parameters of the first comment sub-network and the second comment sub-network.

[0011] According to an embodiment of the present invention, the action review model obtains a decision process vector group according to the state vector, including:

[0012] Obtain the above state vector from the above preset environment;

[0013] The action sub-network obtains an action vector according to the state vector;

[0014] The action sub-network obtains a reward vector according to the state vector and the action vector;

[0015] According to the above state vector, the above action vector and the above reward vector, the above decision process vector group is obtained.

[0016] According to an embodiment of the present invention, the above-mentioned action network module and the above-mentioned comment network module are updated under the principle of maximizing entropy according to the above-mentioned decision process vector group, and the output of the optimal strategy result includes:

[0017] Update the entropy temperature coefficient according to the above decision process vector group ;

[0018] According to the above entropy temperature coefficient Update the above action network module and the above comment network module, and output the above optimal strategy result.

[0019] According to an embodiment of the present invention, the entropy temperature coefficient is updated according to the decision process vector group. include:

[0020] The above action network module obtains the suboptimal strategy in the strategy iteration process according to the above decision process vector group , where i=0,1,2,…, i is the number of iterations in the strategy iteration process;

[0021] The above review network module is based on the above suboptimal strategy , update the above entropy temperature coefficient .

[0022] According to an embodiment of the present invention, the above-mentioned action network module obtains the suboptimal strategy in the strategy iteration process according to the above-mentioned decision process vector group. include:

[0023]

[0024] Among them, the suboptimal strategy Indicates in status Find a strategy for Action distribution and distribution Minimum, It is the state quantity obtained from the training environment in each simulation step during the AUTO-SAC algorithm training process. For each simulation step, the agent is based on the current state The action network performs decision-making output actions, is the KL divergence, i.e. the distribution and The highest similarity is is the state-action value function under the maximum entropy principle, is the entropy temperature coefficient, is the strategy used in the current state.

[0025] According to an embodiment of the present invention, the comment network module uses the above suboptimal strategy , update the above entropy temperature coefficient Also includes:

[0026] Update entropy temperature coefficient The loss functions include:

[0027]

[0028] The entropy temperature coefficient is updated during the training process by the stochastic gradient descent method. , is the target entropy, Represents the current state Next Pass Strategy Select Action The probability of Indicates the policy in its current state.

[0029] According to an embodiment of the present invention, the above entropy temperature coefficient Update the above action network module and the above comment network module, and output the above optimal strategy results including:

[0030] Calculate the above Suboptimal strategy Perform strategy evaluation and update the above comment network module parameters to the above suboptimal strategy Converging to the optimal strategy of comprehensive cumulative reward and maximum entropy principle , where the optimal strategy The entropy temperature coefficient updated based on the updated comment network module obtained;

[0031] Among them, the comprehensive cumulative rewards include time rewards, distance rewards and sight angle rewards.

[0032] According to another aspect of the present invention, there is provided a planning and guidance method according to the above-mentioned model based on the maximum entropy reinforcement learning algorithm, comprising:

[0033] S400: Obtaining an initial state vector of the interception body and an initial state vector of the target body;

[0034] S500: The model based on the maximum entropy reinforcement learning algorithm outputs an optimal strategy result according to the initial state vector of the interceptor body and the initial state vector of the target body.

[0035] According to another aspect of the present invention, there is provided a planning guidance device, comprising:

[0036] An acquisition module, used to acquire an initial state vector of the interception body and an initial state vector of the target body; and

[0037] The output module outputs the optimal strategy result according to the initial state vector of the interceptor body and the initial state vector of the target body through a model based on the maximum entropy reinforcement learning algorithm.

[0038] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0039] A model training method based on the maximum entropy deep reinforcement learning algorithm is proposed. By obtaining the state vector, the deep learning network that constitutes the action comment model can focus on finding spatial information related to the state more quickly. Further, under the principle of maximizing entropy, it avoids calculating the overall space, greatly reducing the complexity of learning and accelerating the output of the optimal strategy results.

[0040] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flow chart of the model training method based on the maximum entropy reinforcement learning algorithm provided by the present invention.

[0043] Figure 2 It is an algorithm schematic diagram of the model training method based on the maximum entropy reinforcement learning algorithm provided by the present invention.

[0044] Figure 3 It is a relative motion relationship diagram between the interception body and the target body provided by the present invention.

[0045] Figure 4 It is a flow chart of the planning guidance method provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0047] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0048] The proportional guidance method is limited by its principle and cannot meet the accuracy requirements of the terminal guidance interception stage. This method requires accurate information on the maneuvering of the target body and has limited application in practical applications.

[0049] The guidance method based on sliding mode control models the collision angle constraint between the interceptor and the target as an error, and derives a complex guidance algorithm. In practical applications, it will place high demands on the control actuator of the interceptor, and may also be unable to be applied due to the complex mathematical form of the guidance algorithm.

[0050] The guidance method based on deep learning has problems such as difficulty in strategy convergence, huge amount of samples required for training, and low training efficiency.

[0051] In view of this, the present invention provides a model training method and a planning guidance method based on a maximum entropy reinforcement learning algorithm.

[0052] Figure 1 It is a flow chart of the model training method based on the maximum entropy reinforcement learning algorithm provided by the present invention.

[0053] like Figure 1 As shown, the training method includes operations S100 to S300.

[0054] In operation S100 , an action review model is constructed, and the action review model includes an action network module and a review network module.

[0055] In operation S200, the action review model obtains a decision process vector group according to a state vector, where the state vector is obtained from a preset environment.

[0056] In operation S300, according to the decision process vector group, the action network module and the comment network module are updated under the principle of maximizing entropy, and the optimal strategy result is output.

[0057] According to an embodiment of the present invention, the state vector may be a vector representing the state of the interception body obtained according to the position and motion parameters of the interception body in a preset state space.

[0058] According to an embodiment of the present invention, the preset environment may be an environment in which the state information of the interception body and the space where the interception body is located is stored after the interception body and the space where the interception body is located are digitized.

[0059] According to the embodiment of the present invention, the action network module can obtain multiple states of the interception body after the state vector according to the input state vector, thereby obtaining the motion trajectory of the interception body.

[0060] According to an embodiment of the present invention, the comment network module can predict the relative state of the motion trajectory of the intercepting body and the motion trajectory of the target body based on the motion trajectory of the intercepting body, thereby obtaining a result of whether the intercepting body can contact the target body.

[0061] According to an embodiment of the present invention, by iteratively calculating the motion trajectory of the intercepting body and obtaining the result of whether the intercepting body can contact the target body, the action comment model can be trained to adjust the motion trajectory of the intercepting body according to the state vector to obtain an optimal motion trajectory, so that when the intercepting body executes the optimal motion trajectory, it can achieve the result of contacting the target body with the least investment, that is, output the optimal strategy result.

[0062] According to an embodiment of the present invention, a model training method based on a maximum entropy deep reinforcement learning algorithm is proposed. By acquiring the state vector, the deep learning network constituting the action comment model can focus on finding state-related spatial information more quickly, and further avoid calculating the overall space under the principle of maximizing entropy, which greatly reduces the complexity of learning and speeds up the output of the optimal strategy results.

[0063] Figure 2 It is an algorithm schematic diagram of the model training method based on the maximum entropy reinforcement learning algorithm provided by the present invention.

[0064] like Figure 2As shown in Figure 1, the action-critic model can be obtained by adopting the improved Actor-Critic architecture. The action-critic model supports the calculation of the deep reinforcement learning algorithm (AUTO Soft Actor Critic, AUTO-SAC) with the maximum entropy strategy.

[0065] According to an embodiment of the present invention, constructing an action review model includes: constructing an action network module, the action network module includes an action sub-network and an action configuration storage sub-network, and the action configuration storage sub-network is used to store the parameters of the action sub-network. Constructing a review network module, the review network module includes a first review sub-network, a second review sub-network and a review configuration storage sub-network, and the review configuration storage sub-network is used to store the parameters of the first review sub-network and the second review sub-network.

[0066] According to an embodiment of the present invention, the actor network module may include an actor sub-network (Actor), wherein the Actor trains a pair of networks , It can represent the policy results of the maximum entropy deep reinforcement learning algorithm iteration .

[0067] According to an embodiment of the present invention, the mobile network module may include a mobile configuration storage sub-network (target_A), and target_A may store parameters of the mobile sub-network in a soft update strategy.

[0068] According to an embodiment of the present invention, the comment network module may include a first comment sub-network, a second comment sub-network and a comment configuration storage sub-network.

[0069] According to an embodiment of the present invention, the first comment sub-network can train a pair of networks , the second comment sub-network can train a pair of networks ,in, A state-action-value function of a neural network that can represent the iteration of the maximum entropy deep reinforcement learning algorithm .

[0070] According to an embodiment of the present invention, the comment configuration storage sub-network (target_C) may store parameters of the first comment sub-network and the second comment sub-network in a soft update strategy.

[0071] According to an embodiment of the present invention, the deep reinforcement learning algorithm of the maximum entropy strategy can enable the algorithm to quickly learn a relatively reasonable way to approach the target through the exploration of the state space and the action space, so that in the subsequent training process, it can focus on finding the solution space of the angle constraint of the collision angle more quickly, avoiding aimless exploration in the possible solution space, and reducing the complexity of learning and the amount of data required.

[0072] Figure 3 It is a relative motion relationship diagram between the interception body and the target body provided by the present invention.

[0073] like Figure 3 As shown, the interception body is Missile; the target body is Target; the line MT between the interception body and the target body is the line of sight; is the distance between the interceptor and the target; is the interceptor velocity; is the target body speed; is the normal acceleration of the intercept body; is the normal acceleration of the target body; is the sight angle, which is the angle between the extension line of R and the x-axis of the preset coordinate system; is the intercept body velocity inclination angle, that is The angle with the x-axis of the preset coordinate system; is the target velocity inclination angle, that is The angle with the x-axis of the preset coordinate system; is the leading angle of the interceptor; is the lead angle of the target body. Based on the physical and geometric relationship between the relative motion of the interceptor and the target body, the relative motion equation is established as follows (1):

[0074] (1)

[0075] According to an embodiment of the present invention, aerodynamic force is added to the simulation scene to further approach the actual scene. , With gravity .

[0076] , , (2)

[0077] In formula (2), is the interceptor weight; is the reference area; is the aerodynamic derivative of lift; is the intermediate quantity of aerodynamic force; is the zero lift drag coefficient; is the induced drag coefficient; the aerodynamic parameters are the atmospheric parameters at different altitudes obtained by interpolation based on the standard atmospheric parameter table.

[0078] According to an embodiment of the present invention, the deep reinforcement learning algorithm of the maximum entropy strategy is suitable for obtaining an optimal strategy result, so that the interception body intercepts the target body at the desired collision angle using the optimal strategy result as the motion trajectory, that is, when the relative motion relationship between the interception body and the target body reaches the interception condition that is satisfied, the interception body contacts the target body. At the moment of contact with the target body, if the ballistic inclination angle of the interception body and the ballistic inclination angle of the target body meet the expected value, it is determined that the interception body successfully intercepts the target body.

[0079] According to an embodiment of the present invention, the determination that the interception body successfully intercepts the target body can be defined as the following formula (3):

[0080] (3)

[0081] in, is the preset maximum simulation time; Formula (3) indicates the expected At the moment of impact, the interceptor and the target The relative distance Less than the error range , while the velocity inclination of the interceptor The velocity inclination angle with the target The difference is in the expected value nearby.

[0082] According to an embodiment of the present invention, in an application, the measurement information of the interceptor on the target body may be the line of sight angle and the line of sight angular rate information, and the line of sight angle and the line of sight angular rate are usually used as important guidance information to design the guidance law. Therefore, the collision angle constraint can be converted into a line of sight angle constraint problem, and the connection between the line of sight angle and the collision angle is established by using a parallel approach collision geometry model, and the ideal collision angle can be indirectly controlled by controlling the ideal line of sight angle.

[0083] According to an embodiment of the present invention, the conversion relationship between the ideal sight angle and the ideal collision angle can be defined as the following formula (4):

[0084] (4)

[0085] in, is the ideal sight angle; is the ideal collision angle.

[0086] According to an embodiment of the present invention, the policy iteration in the deep reinforcement learning algorithm may be composed of two parts, policy evaluation and policy improvement, performed alternately. The policy iteration process may be defined as the following formula (5):

[0087] (5)

[0088] In the formula Represents the suboptimal strategy during the policy iteration process; represents the optimal strategy, that is, the best strategy to solve the problem; For strategy The value of represents the policy evaluation estimate of executing the policy in a certain state The average return obtained when investing represents the quality of the strategy.

[0089] According to an embodiment of the present invention, strategy improvement is to adjust the strategy to execute actions with greater rewards in the current state. According to the strategy improvement theorem, when the state space and action space are limited, strategy iteration can obtain the optimal strategy within a limited time.

[0090] According to an embodiment of the present invention, the optimal strategy can be defined as the following formula (6):

[0091] (6)

[0092] For the status Adopting strategies Selected Action The rewards that can be obtained; For the strategy In this case, the agent will encounter state-action pairs The distribution it obeys.

[0093] According to an embodiment of the present invention, maximum entropy reinforcement learning introduces the maximum entropy principle into the standard reinforcement learning algorithm objective to optimize the optimal strategy of the RL algorithm. Entropy is a measure of the uncertainty of probability distribution and can be defined as the following formula (7):

[0094] (7)

[0095] According to an embodiment of the present invention, entropy has a dependent variable The uncertainty of is positively correlated with the reward function, and the reward function can be defined as follows:

[0096] (8)

[0097] in, is the entropy temperature coefficient, which determines the emphasis on entropy maximization.

[0098] According to an embodiment of the present invention, the maximum entropy reinforcement learning algorithm target can be obtained by combining formula (6), and the target can be defined as the following formula (9):

[0099] (9)

[0100] in, is the state action value function, which represents the current state Execute an action The expected return obtained when .

[0101] According to an embodiment of the present invention, It can be defined as the following formula (10):

[0102] (10)

[0103] According to an embodiment of the present invention, It can be defined as the following formula (11): (11)

[0104] in, is the discount factor, which reflects the decay rate of the subsequent reward at the current moment along the Markov process. Formula (11) is the calculation method of the strategy evaluation process in formula (5).

[0105] According to the embodiment of the present invention, by substituting formula (8) into formula (10), the following formula (12) can be obtained:

[0106] (12)

[0107] in, For current strategy The value of .

[0108] According to an embodiment of the present invention, according to the value function With the state value function The relationship between them can be obtained as follows:

[0109] (13)

[0110] in, It is the calculation result based on the maximum entropy principle.

[0111] According to an embodiment of the present invention, the current strategy is calculated based on the strategy evaluation method of formula (12): of , solve the following optimization problem, and you can get a better strategy than the current one A better new strategy can be obtained as follows:

[0112] (14)

[0113] Among them, the suboptimal strategy For the status Find a strategy for Action distribution and distribution Minimum, In the process of training the agent for the AUTO-SAC algorithm, in each simulation step, the observation (17) is made from the training environment. The state quantity obtained is: , and Similarly, for each simulation step, the agent is based on the current state , the action network performs decision output, in formula (16) there is , For every moment ,Right now , and every moment The sum together constitutes the guidance law at this moment , is the KL divergence, i.e. the distribution and The highest similarity is is the state-action value function under the maximum entropy principle, is the entropy temperature coefficient, is the strategy used in the current state.

[0114] According to an embodiment of the present invention, the entropy temperature coefficient It can determine the degree to which the algorithm attaches importance to maximizing entropy. Too small 𝛼 will cause the algorithm to degenerate back to the ordinary Actor-Critic algorithm, and too large 𝛼 will cause the algorithm to degenerate back to the ordinary Actor-Critic algorithm. This will make the algorithm focus only on maximizing entropy and ignore the environmental reward signal. Therefore, choosing an appropriate Very important.

[0115] According to an embodiment of the present invention, a target entropy may be set , when the entropy of the strategy is greater than the target entropy, reduce , so that the maximum entropy reinforcement learning algorithm based on the maximum entropy reduces the degree of maximizing entropy, thereby reducing the entropy of the current strategy; when the entropy of the strategy is less than the target entropy, increase , so that the degree of maximizing entropy based on the maximum entropy reinforcement learning algorithm increases, thereby increasing the entropy of the current strategy.

[0116] According to an embodiment of the present invention, the entropy loss function can be defined as the following formula (15):

[0117] (15)

[0118] Among them, the stochastic gradient descent (SGD) method can be used to update the entropy temperature coefficient during the training process. .

[0119] in, is the target entropy, which is an artificially set algorithm parameter. is the entropy temperature coefficient, which determines the algorithm's emphasis on entropy maximization (exploration). Represents the current state Next Pass Strategy Select Action The probability of Indicates the strategy in the current state. as well as It is represented as a probability distribution in the code implementation.

[0120] According to an embodiment of the present invention, the current state can be Input to the action network module, the action network module according to the current state Output Action , according to the current state and actions Interacting with the environment is rewarded and , the resulting Markov decision process tuple is Stored in the experience replay buffer and calculated by the comment network module Evaluate the current strategy and update the network parameters of the comment network module. Then update the network parameters of the comment network module based on formula (14), and then update the entropy temperature coefficient After obtaining the updated entropy temperature coefficient In the case of , get back to the current strategy A better new strategy is then evaluated based on the better new strategy and the network parameters of the comment network module are updated. Finally, after iterative updates, the strategy of the final algorithm can be obtained according to the strategy improvement theorem, and converged to the optimal strategy of comprehensive cumulative rewards and maximum entropy principle. .

[0121] According to the embodiment of the present invention, the proportional guidance has the characteristics of simple form, easy implementation and good robustness. The prior knowledge can be combined with the classical guidance idea of ​​proportional guidance. A proportional guidance framework with collision angle constraint bias (AUTO-SACBPNG) driven by deep reinforcement learning data is proposed. AUTO-SACBPNG is defined as the following formula (16):

[0122] (16)

[0123] in, is a pure proportional guidance term, bias term Given by the agent's policy network, the control amount given by the proportional guidance item is adjusted to make the collision angle meet the angle constraint. Therefore, the action space of the designed agent is the bias item .

[0124] According to an embodiment of the present invention, in the initial stage of model training, the proportional guidance term Driven by , AUTO-SACBPNG can quickly learn a relatively reasonable way to approach the target, so that in the subsequent training process it can focus on finding the solution space for the collision angle constraint more quickly, avoiding aimless exploration in the possible solution space, and reducing the complexity of learning and the amount of data required.

[0125] According to an embodiment of the present invention, by introducing inherent and universal rules or knowledge, the trained model is improved so that it can better generalize when facing new, unseen interception volume data, target volume data or environmental data. Experiments have shown that AUTO-SACBPNG can still have high accuracy even in scenarios that have not appeared during the training process.

[0126] According to an embodiment of the present invention, the introduction of prior knowledge into the model may result in certain subjective biases or assumptions based on prior knowledge when making strategy selections, thus falling into local optimality. However, by iteratively optimizing algorithm training based on the maximum entropy reinforcement learning algorithm, the state-action space is fully and effectively explored while maintaining the maximum cumulative reward, thereby alleviating the occurrence of this phenomenon.

[0127] According to an embodiment of the present invention, in the guidance problem with angle constraints, the state space needs to fully describe the motion state of the interception body and the physical relationship between the interception body and the target body, while paying attention to the constraint conditions of the angle constraint guidance. On this basis, the element scales in the state space are unified, and the state space can be designed as the following formula (17):

[0128] (17)

[0129] in, is the maximum relative distance.

[0130] According to an embodiment of the present invention, based on the angle constraint guidance problem, the interception guidance problem is regarded as a time-related sequence, and a reward function is designed by combining sparse rewards with reshaping rewards.

[0131] According to an embodiment of the present invention, when the interceptor fails to hit the target within the specified maximum simulation time, the guidance sequence process ends ( ) The time reward is:

[0132] (18)

[0133] When the interceptor successfully hits the target, There are two situations:

[0134] (19)

[0135] Giving rewards only at the end of a sequence is too sparse and not conducive to training, so rewards are set during the guided sequence to guide learning.

[0136] When the relative distance In the case of , it indicates that the interceptor aircraft is moving away from the target. Distance bonus given .

[0137] (20)

[0138] When the sight angle Deviates from the expected sight angle obtained by equation (6) In the case of each sequence step The sight angle bonus is given in:

[0139] (twenty one)

[0140] in, , are all constants greater than 0.

[0141] Combining the three rewards designed above, the final reward function is:

[0142] (twenty two)

[0143] According to an embodiment of the present invention, by training a reinforcement learning agent to add a bias term on the basis of traditional proportional guidance, the problem of hitting a moving target with a desired collision angle is solved, and the advantages of the AUTO-SACBPNG algorithm and proportional guidance are combined to obtain a collision angle constraint guidance algorithm for intercepting aircraft facing a maneuvering target in a standard atmosphere model with less hyperparameter adjustment, fast model training speed, and strong generalization. AUTO-SACBPNG is superior to BPNG in terms of interception collision angle control accuracy and miss distance accuracy, and the designed guidance law has stronger adaptability to target maneuvers.

[0144] According to an embodiment of the present invention, based on the biased proportional guidance law of maximum entropy deep reinforcement learning, the problem of hitting a moving target with a desired collision angle is solved by training a reinforcement learning agent to add a bias term on the basis of traditional proportional guidance, and the advantages of the AUTO-SACBPNG algorithm and proportional guidance are combined to develop a collision angle constraint guidance algorithm for intercepting aircraft facing a maneuvering target in a standard atmosphere model with less hyperparameter adjustment, fast model training speed, effective mitigation of local optimality, and strong generalization. AUTO-SACBPNG is superior to BPNG in terms of interception collision angle control accuracy and miss amount accuracy, and the designed guidance law has stronger adaptability to the interception target body.

[0145] (twenty three)

[0146] According to an embodiment of the present invention, specifically, an aircraft interception guidance training and simulation environment can be established. The specific experimental parameters of the training environment are shown in Table 1. The parameters with a subscript of 0 in the table represent the initial state of the simulation:

[0147] Table 1 Training environment parameters

[0148]

[0149] According to an embodiment of the present invention, specifically, the neural network in the Actor module and the Critic module of the interception aircraft agent in AUTO-SACBPNG can be set using the network parameters shown in Table 2:

[0150] Table 2 Intelligent network parameter settings

[0151]

[0152] According to an embodiment of the present invention, specifically, the learning rate of the action network module can be ; The learning rate of the comment network module can be ; The maximum capacity of the experience playback area is ; The learning rate of the adaptive entropy temperature coefficient is ; The initial value of the adaptive entropy temperature coefficient is ;Simulations and experiments can be performed on the NVIDIA GeForce RTX 3080Ti graphics card.

[0153] According to the embodiment of the present invention, specifically, when the target maneuvers, BPNG fails to hit the target and causes a miss, while the angle error of AUTO-SACBPNG is 1.283° and the miss distance is 9.85m. The above simulation experiments show that AUTO-SACBPNG has better guidance performance than BPNG, and the miss distance and collision angle error of AUTO-SACBPNG are significantly reduced.

[0154] The following describes the effectiveness of the guidance law proposed by the present invention in intercepting a maneuvering target at a desired collision angle in an aerodynamic environment in combination with Table 3 and Table 4.

[0155] The Monte Carlo simulation parameters are shown in Table 3.

[0156] Table 3: Monte Carlo simulation environment parameters

[0157]

[0158] The mean and variance of the target miss distance and collision angle error are shown in Table 4. It shows the performance of the target under different maneuvering amplitudes.

[0159] Table 4: Mean and variance of target miss distance and collision angle error

[0160]

[0161] According to the embodiment of the present invention, the parameters not shown in Table 3 can be consistent with the parameters provided in the above embodiment in the specification. The target maneuvers in the Monte Carlo simulation process take different values, and Table 4 is the mean and variance of the target miss amount and collision angle error. The Monte Carlo simulation experiment shows the effectiveness of the guidance law proposed by the present invention in intercepting maneuvering targets at the expected collision angle in an aerodynamic environment.

[0162] According to an embodiment of the present invention, the collision angle constraint guidance problem is converted into a line of sight angle constraint problem based on a geometric model, and a reward function is designed to guide the interceptor missile to hit and satisfy the collision angle constraint by using the relative motion state of the missile and the target and the expected line of sight angle error during the guidance process. Finally, an AUTO-SAC biased proportional guidance law (AUTO-SACBPNG) is obtained by training reinforcement learning. Numerical simulations are carried out under the conditions of steady target motion and maneuvering, and the Monte Carlo simulation results show that the proposed deep reinforcement learning guidance law AUTO-SACBPNG is significantly better than the biased proportional guidance law BPNG with angle constraint, and can effectively reduce the miss amount and collision angle error at the moment of collision, and its excellent robustness and generalization are proved through Monte Carlo experiments.

[0163] Figure 4 It is a flow chart of the planning guidance method provided by the present invention.

[0164] like Figure 4 As shown, the planning guidance method includes operations S400 to S500.

[0165] In operation S400, an initial state vector of an intercept body and an initial state vector of a target body are acquired.

[0166] In operation S500, the model based on the maximum entropy reinforcement learning algorithm outputs an optimal strategy result according to the initial state vector of the intercept body and the initial state vector of the target body.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method based on maximum entropy reinforcement learning algorithm, characterized in that: include: S100: constructing an action review model of a terminal guidance algorithm, wherein the action review model includes an action network module and a review network module; S200: the action review model of the terminal guidance algorithm obtains a decision process vector group according to a state vector, wherein the state vector is obtained from a preset environment; The action review model for constructing the terminal guidance algorithm includes: Constructing the action network module, wherein the action network module includes an action sub-network and an action configuration storage sub-network, wherein the action configuration storage sub-network is used to store parameters of the action sub-network; Constructing the comment network module, wherein the comment network module includes a first comment sub-network, a second comment sub-network and a comment configuration storage sub-network, wherein the comment configuration storage sub-network is used to store parameters of the first comment sub-network and the second comment sub-network; The step of updating the action network module and the comment network module according to the decision process vector group under the principle of maximizing entropy and outputting the optimal strategy result includes: Update the entropy temperature coefficient according to the decision process vector group ; According to the entropy temperature coefficient Updating the action network module and the comment network module, and outputting the optimal strategy result; The entropy temperature coefficient is updated according to the decision process vector group include: The action network module obtains the suboptimal strategy in the strategy iteration process according to the decision process vector group , where i=0,1,2,…, i is the number of iterations in the strategy iteration process; The review network module uses the suboptimal strategy , update the entropy temperature coefficient ; The action network module obtains the suboptimal strategy in the strategy iteration process according to the decision process vector group include: Among them, the suboptimal strategy For the status Find a strategy for Action distribution and distribution Minimum, It is the state quantity obtained from the training environment in each simulation step during the AUTO-SAC algorithm training process. For each simulation step, the agent is based on the current state The action network performs decision output actions, the actions is the bias term at each moment , the bias term Given by the agent’s policy network, is the KL divergence, i.e. the distribution and The highest similarity is is the state-action value function under the maximum entropy principle, is the entropy temperature coefficient, is the strategy used in the current state; S300: According to the decision process vector group, the action network module and the comment network module are updated under the principle of maximizing entropy, and the optimal strategy result is output.

2. The model training method based on the maximum entropy reinforcement learning algorithm according to claim 1, characterized in that: The action review model obtains a decision process vector group according to the state vector, including: Obtaining the state vector from the preset environment; The action sub-network obtains an action vector according to the state vector; The action sub-network obtains a reward vector according to the state vector and the action vector; The decision process vector group is obtained according to the state vector, the action vector and the reward vector.

3. The model training method based on the maximum entropy reinforcement learning algorithm according to claim 1, characterized in that: The review network module uses the suboptimal strategy , update the entropy temperature coefficient Also includes: Update entropy temperature coefficient The loss functions include: Among them, the entropy temperature coefficient is updated during the training of the agent by stochastic gradient descent method , is the target entropy, Represents the current state Next Pass Strategy Select Action The probability of Indicates the policy in its current state.

4. The model training method based on the maximum entropy reinforcement learning algorithm according to claim 1, characterized in that: According to the entropy temperature coefficient Updating the action network module and the comment network module, and outputting the optimal strategy result includes: Calculate the Suboptimal strategy Perform strategy evaluation and update the review network module parameters to the suboptimal strategy Converging to the optimal strategy of comprehensive cumulative reward and maximum entropy principle , where the optimal strategy The entropy temperature coefficient updated based on the updated comment network module obtained; Among them, the comprehensive cumulative rewards include time rewards, distance rewards and sight angle rewards.

5. A planning and guidance method using the model based on the maximum entropy reinforcement learning algorithm as described in any one of claims 1 to 4, characterized in that: include: S400: Obtaining an initial state vector of the interception body and an initial state vector of the target body; S500: The model based on the maximum entropy reinforcement learning algorithm outputs an optimal strategy result according to the initial state vector of the interceptor body and the initial state vector of the target body.

Images