A pre-training method, training method and system for a collaborative guidance law model

Abstract

The present application relates to the technical field of control, and in particular to a pre-training method, training method and system for a collaborative guidance law model. The pre-training method includes: obtaining pre-training samples and a pre-training model from a missile, and then performing offline reinforcement learning training on the pre-training model based on the pre-training samples to obtain a pre-trained collaborative guidance law model, and using the network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation target network in the pre-trained collaborative guidance law model as the initial network parameters of the evaluation network to be trained and the initial network parameters of the evaluation target to be trained in the training process of the collaborative guidance law model, which can simplify the actual training process, improve training efficiency, and obtain the collaborative guidance law model quickly and efficiently.

Description

Technical Field

[0001] The present application relates to the technical field of control, and in particular to a pre-training method, a training method and a system for a collaborative guidance law model. Background Art

[0002] During a confrontation, drones must continuously adjust their operational strategies based on the enemy aircraft's position, status, and tactics, ensuring they can accurately strike the enemy along a defined path or trajectory. When multiple drones engage or launch multiple missiles, they must collaborate to track and strike the target. This can be achieved through a multi-missile time-coordinated guidance law, whereby a master missile guides and slave missiles track, introducing residual time coordination compensation instructions. This allows all slave missiles to actively follow and coordinate their attack on the target.

[0003] In related technologies, samples in the experience pool are directly used to train the neural network, but the training process requires a large amount of data and online model training takes a long time. Summary of the Invention

[0004] The purpose of this application is to provide a pre-training method, training method and system for a collaborative guidance law model, which can improve training efficiency.

[0005] The above-mentioned application objective 1 of this application is achieved through the following technical solutions:

[0006] In a first aspect, a pre-training method for a collaborative guidance law model is provided, comprising:

[0007] Obtain pre-training samples from the bullet, wherein each pre-training sample includes: an environment state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environment state at a third moment;

[0008] Acquire a pre-trained model, the pre-trained model comprising: the pre-trained network and the pre-trained target network, the pre-trained network comprising: a pre-trained action network and a pre-trained evaluation network, the pre-trained target network comprising: a pre-trained action target network and a pre-trained evaluation target network;

[0009] According to the pre-training samples, the pre-training model is subjected to reinforcement learning training until a stopping condition is reached to obtain a pre-trained collaborative guidance law model.

[0010] The network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation target network in the pre-trained collaborative guidance law model serve as the initial network parameters of the evaluation network to be trained and the initial network parameters of the evaluation target to be trained in the collaborative guidance law model training process.

[0011] In a possible implementation, obtaining a pre-training sample from a bullet includes:

[0012] Acquire an environmental state at a first moment, and obtain an action value corresponding to the first moment using a cooperative guidance law according to the environmental state at the first moment;

[0013] According to the action value corresponding to the first moment, the training environment is simulated to obtain the environment state at the second moment and the reward value corresponding to the first moment;

[0014] According to the environmental state at the second moment and the cooperative guidance law, a second action value corresponding to the second moment is obtained; and according to the action value corresponding to the second moment, a training environment is simulated to obtain the environmental state at the third moment and the second reward value corresponding to the second moment.

[0015] In one possible implementation, simulating the training environment based on the action value corresponding to the first moment to obtain the environment state at the second moment and the reward value corresponding to the first moment includes:

[0016] Simulating the training environment according to the action value corresponding to the first moment to obtain the environment state at the second moment;

[0017] determining, based on the environmental state at the second moment, a lead angle of the missile, a relative distance between the missile and the target, and a relative speed between the missile and the target corresponding to the missile at the second moment, wherein the missile includes a main missile and a slave missile;

[0018] When the missile hits, determining a first reward value corresponding to the first moment according to the main missile hitting moment and the secondary missile hitting moment;

[0019] When the missile misses, the predicted hit moment corresponding to the main missile and the predicted hit moment corresponding to the slave missile are determined based on the corresponding lead angle of the missile, the relative distance between the missile and the target, and the relative speed between the missile and the target; and the first reward value corresponding to the first moment is determined based on the predicted hit moment corresponding to the main missile and the predicted hit moment corresponding to the slave missile.

[0020] In a possible implementation, performing reinforcement learning training on the pre-trained model according to the pre-trained samples until a stopping condition is reached to obtain a pre-trained collaborative guidance law model includes:

[0021] Obtaining an evaluation value according to the environmental state at the first moment and the pre-trained network;

[0022] Obtaining an initial evaluation target value using a pre-trained target network according to the environmental state at the third moment;

[0023] determining an evaluation target value according to the initial evaluation target value, the first reward value, and the second reward value;

[0024] The pre-training model is iteratively trained according to the evaluation target value, the evaluation value and the pre-training sample until a stopping condition is reached to obtain a pre-trained collaborative guidance law model.

[0025] In a possible implementation, determining the evaluation target value according to the initial evaluation target value, the first reward value, and the second reward value includes:

[0026] Determining an intermediate evaluation target value according to the initial evaluation target value, the attenuation discount coefficient, and the second reward value;

[0027] The evaluation target value is determined according to the intermediate evaluation target value, the decay discount coefficient and the first reward value.

[0028] In a possible implementation, iteratively training the pre-trained model according to the evaluation target value, the evaluation value, and the pre-trained samples until a stopping condition is met to obtain the pre-trained collaborative guidance law model includes:

[0029] After completing the calculation of the pre-trained model for the preset number of samples, determining the evaluation network parameters according to the evaluation target values ​​and the evaluation values ​​corresponding to the preset number of samples, and updating the pre-trained evaluation network according to the evaluation network parameters;

[0030] After the pre-trained evaluation network is updated for a first preset period, the pre-trained evaluation target network is updated according to the evaluation network parameters determined after the first preset period;

[0031] Performing policy gradient calculation on the pre-trained action network based on the updated pre-trained evaluation network to obtain an action loss value, and updating the network parameters of the pre-trained action network based on the action loss value;

[0032] When the pre-trained action network is updated for a second preset period, updating the network parameters of the pre-trained action target network according to the action loss value determined after the second preset period;

[0033] Iterative training is performed until the stopping condition is reached to obtain the pre-trained collaborative guidance law model.

[0034] In a possible implementation, the pre-training sample further includes: an action value corresponding to the first moment,

[0035] The strategy gradient calculation is performed on the pre-trained action network according to the updated pre-trained evaluation network to obtain the action loss value, including:

[0036] Evaluate the action value corresponding to the first moment to obtain the offline expert evaluation value corresponding to the first moment;

[0037] determining a correction value according to the offline expert evaluation value and the evaluation value corresponding to the first moment;

[0038] Among them, when the offline expert evaluation value corresponding to the first moment is greater than the evaluation value corresponding to the first moment, the correction value is the two-norm of the action value corresponding to the first moment and the predicted action value corresponding to the first moment; otherwise, the correction value is 0; the predicted action value corresponding to the first moment is obtained based on the environment state at the first moment and the pre-trained action network.

[0039] The policy gradient calculation is performed on the pre-trained action network according to the correction value and the updated pre-trained evaluation network to obtain the action loss value.

[0040] In a second aspect, the present application provides a training method for a collaborative guidance law model, comprising:

[0041] Acquire multiple training samples, where each training sample includes: an environment state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environment state at a third moment;

[0042] Performing iterative reinforcement learning training on the model to be trained based on the multiple training samples to obtain a collaborative guidance law model;

[0043] The model to be trained includes a network to be trained and a target network to be trained. The initial parameters of the evaluation network to be trained in the network to be trained and the target evaluation network to be trained in the target network to be trained are corresponding parameters in a pre-trained collaborative guidance law model. The pre-trained collaborative guidance law model is obtained by reinforcement learning training based on multiple pre-training samples.

[0044] Thirdly, a pre-training system for a collaborative guidance law model is provided, including:

[0045] A first acquisition module is used to acquire pre-training samples from the bullet, wherein each pre-training sample includes: an environmental state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environmental state at a third moment;

[0046] A second acquisition module is used to acquire a pre-trained model, wherein the pre-trained model includes: the pre-trained network and the pre-trained target network, the pre-trained network includes: a pre-trained action network and a pre-trained evaluation network, and the pre-trained target network includes: a pre-trained action target network and a pre-trained evaluation target network;

[0047] The first training module is used to perform reinforcement learning training on the pre-training model according to the pre-training samples until a stopping condition is reached to obtain a pre-trained collaborative guidance law model.

[0048] The network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation target network in the pre-trained collaborative guidance law model serve as the initial network parameters of the evaluation network to be trained and the initial network parameters of the evaluation target to be trained in the collaborative guidance law model training process.

[0049] In a fourth aspect, a training system for a collaborative guidance law model is provided, comprising:

[0050] A third acquisition module is used to acquire multiple training samples, wherein each training sample includes: an environment state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environment state at a third moment;

[0051] A second training module is configured to perform iterative reinforcement learning training on the training model based on the multiple training samples to obtain a collaborative guidance law model;

[0052] The model to be trained includes a network to be trained and a target network to be trained. The initial parameters of the evaluation network to be trained in the network to be trained and the target evaluation network to be trained in the target network to be trained are corresponding parameters in a pre-trained collaborative guidance law model. The pre-trained collaborative guidance law model is obtained by reinforcement learning training based on multiple pre-training samples.

[0053] According to a fifth aspect, an electronic device is provided, including:

[0054] one or more processors;

[0055] Memory;

[0056] One or more applications, wherein one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: perform operations corresponding to the method shown in any possible implementation of the first aspect.

[0057] According to a sixth aspect, another electronic device is provided, comprising:

[0058] one or more processors;

[0059] Memory;

[0060] One or more applications, wherein the one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: perform operations corresponding to the method shown in the second aspect.

[0061] In the seventh aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method as shown in any possible implementation method of the first aspect.

[0062] In an eighth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method shown in the second aspect.

[0063] In summary, this application includes at least one of the following beneficial technical effects:

[0064] This solution obtains pre-trained samples and pre-trained models from the missile, and then performs offline reinforcement learning training on the pre-trained model based on the pre-trained samples to obtain a pre-trained collaborative guidance law model, and uses the network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation target network in the pre-trained collaborative guidance law model as the initial network parameters of the evaluation network to be trained and the initial network parameters of the evaluation target to be trained in the training process of the collaborative guidance law model, which can simplify the actual training process, improve training efficiency, and obtain the collaborative guidance law model quickly and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a schematic diagram of a master-slave bullet information interaction provided by an embodiment of the present application;

[0066] Figure 2 1 is a flow chart of a pre-training method for a collaborative guidance law model provided in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of pre-training provided in an embodiment of the present application;

[0068] Figure 4 A flowchart of a training method for a collaborative guidance law model provided in an embodiment of the present application;

[0069] Figure 5 A flowchart of a specific training process provided in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of the structure of a pre-training system for a collaborative guidance law model provided in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of the structure of a training system for a collaborative guidance law model provided in an embodiment of the present application;

[0072] Figure 8A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Implementation Method

[0073] The following is combined with Figures 1-8 This application is described in further detail.

[0074] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed, but as long as they are within the scope of the claims of the present application, they are protected by the patent law.

[0075] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0076] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0077] Coordinated guidance law design is a crucial factor influencing guided missile interception and strike accuracy. Faced with increasingly complex combat scenarios and missions, the military need for coordinated multi-missile strikes is becoming increasingly urgent. Coordinated guidance laws are often derived analytically from formulas for estimating residual guidance time. These laws make assumptions about small-angle relationships and ignore high-order terms in the residual time expansion. By controlling the overload commands of individual missiles, the residual time differences between them are minimized, thus enabling coordinated strikes.

[0078] The goal of coordinated guidance is to ensure that all missiles have the same remaining flight time. If there is a discrepancy between the remaining flight times, the motion values ​​are adjusted. When all missiles have the same remaining flight time, they can achieve a coordinated strike on the target. However, the coordinated guidance law is difficult to implement in engineering applications.

[0079] The embodiment of the present application designs a solution for a multi-missile collaborative guidance system, using reinforcement learning technology to design a collaborative guidance law model to achieve coordinated strikes of all missiles on targets. In order to enable rapid training of the collaborative guidance law model, the model is first appropriately pre-trained offline to obtain network parameters, which are used as initial parameters in the online training process to enable fast and efficient iterative learning.

[0080] In the embodiments of the present application, the pre-training process and the training process both include: a master-slave collaborative information topology corresponding to information interaction, a multi-missile motion model, and a collaborative guidance law.

[0081] Specifically, the master-slave system information topology is further explained.

[0082] The time coordination problem of N missiles attacking the target can be converted into the time coordination problem of N-1 slave missiles attacking the main missile, that is, the main missile guidance and the slave missile tracking. The remaining time coordination compensation instruction method can be introduced into the basic proportional guidance law of the slave missile to adjust the time when the slave missile reaches the target, thereby achieving the purpose of all the slave missiles following the main missile to coordinate the attack on the target. The main missile guidance law only uses its own detection information to achieve guidance, but the slave missile needs to receive the status of the main missile and the detection information and combine its own status for guidance. There is no need for information exchange between the slave missiles. The entire missile group belongs to a star topology structure. For details, please refer to Figure 1 .

[0083] Specifically, the multi-bomb motion model is further elaborated. Taking the horizontal plane as an example, the design can be carried out, including: ,

[0084] Among them, i=0,1,2…,n, among all missiles, i=0 is the main missile, i=1,2…,n is the slave missile.

[0085] is the relative speed of the i-th missile to the target; is the target's speed; is the lead angle of the target; is the speed of the i-th missile; is the lead angle of the i-th missile; is the change in the sight angle between the i-th missile and the target; is the relative distance from the i-th missile to the target; is the sight angle between the i-th missile and the target, is the heading angle of the target; is the heading angle of the i-th missile, is the change in the heading angle of the i-th missile; is the lateral acceleration acting on the i-th missile.

[0086] Specifically, the framework of the collaborative guidance law is further elaborated.

[0087] In this application, the cooperative guidance law includes a basic proportional guidance law and / or a cooperative correction term.

[0088] In one possible implementation, the cooperative guidance law framework (training action network) is composed of a basic proportional guidance law and cooperative correction terms during training. The guidance law framework (pre-trained action network) can be used as the basis for the proportional guidance law during the pre-training process.

[0089] Where i is the missile number. The basic proportional guidance law is composed of feedback from the missile's own state and the target state, which is used to guide the missile to fly to and attack the target. For the main missile (i=0), the basic proportional guidance law is , the basic proportional guidance law for the missile is , where N is the proportional guidance coefficient, is the lead angle, K is the time coordination term correction coefficient, For the remaining time estimate, The difference between the estimated remaining time of the main projectile and the estimated remaining time of the main projectile; is the relative distance from the missile to the target.

[0090] It is the time cooperative control term of the guidance law, i.e., the cooperative correction term, which is used to adjust the remaining time for the slave missile to hit the target. Reinforcement learning method can be used to make decisions. It is a representation of the neural network compensation term to achieve the purpose of minimizing the time difference between the main missile and the slave missile hitting the target, that is, ,in, Indicates the main projectile's impact time under the guidance law. represents the time when the i-th missile hits.

[0091] In order to improve the efficiency of training, the embodiment of the present application performs offline pre-training to obtain the network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation target network as the initial network parameters of the evaluation network to be trained and the initial network parameters of the evaluation target to be trained in the training process of the collaborative guidance law model.

[0092] Specifically, the present invention provides a method for pre-training a collaborative guidance law model. Figure 2As shown, the method provided in the embodiments of the present application can be executed by a training system, which can be a server or a terminal device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application. The method includes:

[0093] Step S110: Obtain pre-training samples from the bullet, wherein each pre-training sample includes: an environment state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environment state at a third moment;

[0094] A preset number of pre-training samples are randomly selected from the experience pool.

[0095] Step S120: Obtain a pre-trained model, the pre-trained model includes: a pre-trained network and a pre-trained target network, the pre-trained network includes: a pre-trained action network and a pre-trained evaluation network, and the pre-trained target network includes: a pre-trained action target network and a pre-trained evaluation target network;

[0096] The pretrained model used in the embodiments of the present application includes a pretrained action network, a pretrained evaluation network, a pretrained action target network, and a pretrained evaluation target network. The pretrained evaluation network and the pretrained evaluation target network can each include two evaluation networks. Since the evaluation network may overestimate in practical applications, selecting smaller evaluation values ​​from the two evaluation networks can reduce overestimation. After training, the action network (pretrained action network and pretrained action target network) can be used as the initial parameters of the neural network controller to be trained based on the pretrained parameters. Furthermore, the trained model can be used to implement missile attitude control.

[0097] The action network takes the environment state as input and outputs the action value. The evaluation network (pre-trained evaluation network and pre-trained evaluation target network) takes the environment state and action value as input and outputs the corresponding evaluation value. The purpose of the action network is to output the action value that maximizes the evaluation value based on the state. The evaluation network evaluates the environment state and action value and outputs the evaluation value.

[0098] It can be understood that the pre-trained action network and the pre-trained action target network have the same structure, and the pre-trained evaluation network and the pre-trained evaluation target network have the same structure.

[0099] In the embodiment of the present application, the pre-trained action network may include a basic proportional guidance term and / or a time coordination correction term, wherein the time coordination correction term is g i Function , the decision can be made by adjusting the remaining time from the missile hitting the target. Preferably, the pre-trained action network is composed of basic proportional guidance law items.

[0100] Step S130: Perform reinforcement learning training on the pre-training model according to the pre-training samples until the stopping condition is reached to obtain the pre-trained collaborative guidance law model.

[0101] The network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation target network in the pre-trained collaborative guidance law model serve as the initial network parameters of the evaluation network to be trained and the initial network parameters of the evaluation target to be trained in the training process of the collaborative guidance law model.

[0102] During the pre-training process, the pre-trained model can be initialized with random parameters, and then reinforcement learning is performed on the initialized pre-trained model based on the pre-training samples until a stopping condition is reached. The stopping condition can be that the difference in the time between the main and secondary projectiles hitting the target is no greater than a preset threshold. The preset threshold can be set according to actual needs. The shorter the time between the main and secondary projectiles hitting the target, the better the model effect. Preferably, the preset threshold is 0.

[0103] It can be seen that in the embodiment of the present application, pre-trained samples and pre-trained models of the missile are obtained, and then the pre-trained model is subjected to offline reinforcement learning training based on the pre-trained samples to obtain a pre-trained collaborative guidance law model, and the network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation target network in the pre-trained collaborative guidance law model are used as the initial network parameters of the evaluation network to be trained and the initial network parameters of the evaluation target to be trained in the training process of the collaborative guidance law model, which can simplify the actual training process, improve training efficiency, and obtain the collaborative guidance law model quickly and efficiently.

[0104] In one feasible solution, the method of obtaining the pre-training samples from the bullet includes:

[0105] Obtaining the environmental state at the first moment, and using the cooperative guidance law to obtain the action value corresponding to the first moment according to the environmental state at the first moment;

[0106] According to the action value corresponding to the first moment, the training environment is simulated to obtain the environment state at the second moment and the reward value corresponding to the first moment;

[0107] According to the environmental state at the second moment and the cooperative guidance law, a second action value corresponding to the second moment is obtained; according to the action value corresponding to the second moment, the environmental state at the third moment and the second reward value corresponding to the second moment are obtained by simulating the training environment.

[0108] Among them, before collecting training samples, in order to realize the training of the collaborative guidance law algorithm based on reinforcement learning, it can also include: first, establishing the following Markov decision model for the slave missile , s is the current state of the environment, a is the action value corresponding to the current moment, R is the reward function corresponding to the current moment, s' is the next moment state corresponding to the current moment, and the observation state is defined as: ,in, The speed of the target; the goal of the training design is to minimize the error in the time of impact between the slave and main projectiles, such as:

[0109] ,in, The duration of the main bullet's hit when it hits. is the duration of the bullet’s impact when it hits the target; design the reward R for the agent.

[0110] Specifically, when collecting pre-training samples, the samples can be obtained through cooperative guidance law interception simulation. The main missile uses the traditional proportional guidance law to directly attack the target. The guidance law corresponding to the main missile is , the missile adopts a classic time-coordinated guidance law, the guidance law is .

[0111] Specifically, given the environment state S0 at the first moment, the action network (i.e., the collaborative guidance law) is used to predict the corresponding action value a0, and then the action value a0 is input into the training environment for simulation to obtain the environment state S1 at the second moment and the reward value R corresponding to the first moment. 00 Then, according to the environment state S1 at the second moment and the cooperative guidance law, the second action value a1 corresponding to the second moment is obtained; according to the action value a1 corresponding to the second moment, the training environment is simulated to obtain the environment state S2 at the third moment and the second reward value R corresponding to the second moment. 11 , so we get a sample (S0, R 00 , R 11 , S2), and put it into the experience pool. Of course, the sample can also be (S0, a0, R 00 , S1, a1, R 11 , S2, a2).

[0112] The experience pool can eliminate the correlation between samples. This is because the previous and subsequent actions in reinforcement learning are usually strongly correlated. Putting them into the experience pool and randomly selecting a batch of samples during subsequent training can make the neural network training more accurate.

[0113] It can be seen that in the embodiment of the present application, the action value corresponding to the first moment can be obtained by using the action network according to the environmental state at the first moment, and the reward value corresponding to the first moment can be obtained by simulation through the training environment; further, the action value corresponding to the second moment can be obtained by using the action network according to the environmental state at the second moment, and the reward value corresponding to the third moment can be obtained by simulation through the training environment to generate pre-training samples from the bullet, and the sample sequence of three consecutive moments can be used to reduce the error of the evaluation network under large-scale random initialization conditions during pre-training.

[0114] In the embodiment of the present application, the process of determining the reward value corresponding to the first moment and the reward value corresponding to the second moment are the same, and the two can be compared with each other.

[0115] In this embodiment of the present application, the reward R of the design reward agent can be:

[0116] ,

[0117] in This is the bonus for hitting a target, which is only related to the time difference between the master and slave projectiles:

[0118] ,in It is the proportional coefficient, which can be customized and the value is greater than 0.

[0119] The reward before hitting the target and the predicted missile hitting time related:

[0120] ,in It is the proportional coefficient, which can be customized and the value is greater than 0. is the predicted main missile impact time, is the predicted time of impact of the missile.

[0121] A possible implementation of the embodiment of the present application is: ,in,

[0122] and The function of the term is to constrain the magnitude of the reward output to avoid excessive or saturated instructions. a The value of is greater than 0.

[0123] Then, according to the action value corresponding to the first moment, the training environment is simulated to obtain the environment state at the second moment and the reward value corresponding to the first moment, including:

[0124] According to the action value corresponding to the first moment, the training environment is simulated to obtain the environment state at the second moment;

[0125] determining, based on the environmental state at the second moment, a lead angle of the missile, a relative distance between the missile and the target, and a relative speed between the missile and the target corresponding to the missile at the second moment, wherein the missile includes a main missile and a slave missile;

[0126] When the missile hits, a first reward value corresponding to the first moment is determined according to the main missile hit moment and the secondary missile hit moment;

[0127] When the missile misses, the predicted hit time of the main missile and the predicted hit time of the slave missile are determined based on the corresponding lead angle of the missile, the relative distance between the missile and the target, and the relative speed between the missile and the target; and the first reward value corresponding to the first moment is determined based on the predicted hit time of the main missile and the predicted hit time of the slave missile.

[0128] Specifically, the environmental state at the second moment can be obtained through the action corresponding to the first moment. The environmental state at the second moment includes: the relative distance from the main missile to the target, the missile-target line-of-sight angle, heading angle, and speed of the main missile and the target, as well as the relative distance from the slave missile to the target, the missile-target line-of-sight angle, heading angle, and speed of the slave missile and the target, and the speed of the target; the relative speed of the slave missile and the target is calculated using the formula of the environmental state at the second moment and the multi-missile motion model.

[0129] It is determined whether the next moment is the final state. If it is the final state, it is determined that the missile has hit. Otherwise, it is determined that the missile has not hit.

[0130] Then, when the missile hits, according to the main missile hitting time and the secondary missile hitting time, use , determine the first reward value corresponding to the first moment, the main bullet hitting moment and the slave bullet hitting moment are both moments obtained by simulation.

[0131] When the missile misses, the missile is used according to the corresponding lead angle, the relative distance between the missile and the target, and the relative speed between the missile and the target. , determine the predicted hitting time corresponding to the main bullet and the predicted hitting time corresponding to the slave bullet, and then use the predicted hitting time corresponding to the main bullet and the predicted hitting time corresponding to the slave bullet to , determine the first reward value corresponding to the first moment.

[0132] Another possible implementation of the embodiment of the present application is to design ,

[0133] When the missile misses, the missile is guided according to the corresponding lead angle, the relative distance between the missile and the target, the relative speed between the missile and the target, and the proportional guidance coefficient. , calculate the predicted hitting moment corresponding to the main bullet and the predicted hitting moment corresponding to the slave bullet; and determine the first reward value corresponding to the first moment according to the predicted hitting moment corresponding to the main bullet and the predicted hitting moment corresponding to the slave bullet.

[0134] It can be seen that after the embodiment of the present application simulates the environmental state at the second moment according to the action value corresponding to the first moment, the state of the main missile and the slave missile at the second moment is determined according to the second environmental state. Then, when the missile hits the target, the first reward value is determined according to the moment when the main missile hits and the moment when the slave missile hits. When the missile misses the target, the hit moment is predicted based on the lead angle corresponding to the missile, the relative distance between the missile and the target, and the relative speed between the missile and the target, and the first reward value is determined based on the prediction result, which can improve the accuracy of the reward value determination.

[0135] Furthermore, in an embodiment of the present application, step S130 performs reinforcement learning training on the pre-trained model according to the pre-trained samples until a stopping condition is reached to obtain a pre-trained collaborative guidance law model, including: obtaining an evaluation value according to the environmental state at the first moment and the pre-trained network; obtaining an initial evaluation target value according to the environmental state at the third moment using the pre-trained target network; determining the evaluation target value according to the initial evaluation target value, the first reward value and the second reward value; iteratively training the pre-trained model according to the evaluation target value, the evaluation value and the pre-trained samples until the stopping condition is reached to obtain the pre-trained collaborative guidance law model.

[0136] Please refer to Figure 3 , Figure 3 A schematic diagram of a pre-training process provided in an embodiment of the present application, wherein an action is predicted by a pre-trained action network in a pre-training network based on an environmental state at a first moment, and a predicted action value corresponding to the first moment is obtained. The predicted action value corresponding to the first moment and the environmental state at the first moment are then input into a first pre-trained evaluation network and a second pre-trained evaluation network to obtain a first evaluation value and a second evaluation value, with the smaller evaluation value being used as the final evaluation value.

[0137] According to the environmental state at the third moment, the action is predicted by the pre-trained action target network in the pre-trained target network to obtain the predicted target action value corresponding to the third moment, and then the predicted target action value corresponding to the third moment and the environmental state at the third moment are input into the first pre-trained evaluation target network and the second pre-trained evaluation target network to obtain the first initial evaluation value and the second initial evaluation value, and the smaller initial evaluation value is used as the final initial evaluation value; wherein, in one possible case, the predicted target action value is the value directly output by the pre-trained action target network, and in another possible case, the predicted target action value is the sum of the value output by the pre-trained action target network and noise, which can encourage exploration to obtain a more accurate initial evaluation value;

[0138] Then, the intermediate evaluation target value is determined based on the initial evaluation target value, the decay discount coefficient, and the second reward value. The evaluation target value is then determined based on the intermediate evaluation target value, the decay discount coefficient, and the first reward value. The decay discount coefficient is introduced to reduce the multi-partial difference error in a step-by-step manner, making the final evaluation target value more accurate.

[0139] Specifically, in order to further reduce the error of the evaluation network under a wide range of random initialization conditions, a multi-step differential error is introduced in the update estimation of the pre-trained evaluation network according to the step-by-step attenuation. That is, the update process of the pre-trained evaluation network can use samples at three consecutive moments. , in the calculation Target value of the pre-trained target evaluation network at each moment When using Rewards of the moment as well as State of the moment , specifically, the initial evaluation target value is calculated , and then use formula (1) to get the evaluation target value y:

[0140] Formula (1);

[0141] Among them, the subscript represents the pre-trained evaluation target network, is the reward attenuation discount coefficient, Actions in state and Actions in state It can be calculated using formula (2) and formula (3) respectively, which are the values ​​after adding exploratory perturbations to the output action of the pre-trained action target network in the corresponding state.

[0142] Formula (2);

[0143] Formula (3);

[0144] Among them, the subscript It is represented as a pre-trained action target network with low update frequency and amplitude. Indicates that it conforms to the normal distribution and is affected Amplitude-constrained random exploration perturbations.

[0145] As can be seen, in this embodiment, exploration noise is added after the output signal of the action goal network to avoid local convergence of the agent, eliminate overfitting of the cumulative evaluation estimate, and enhance the agent's ability to generalize to the environment. Using a smaller evaluation value estimate to calculate the expected cumulative reward prevents overestimation of the cumulative reward estimate and promotes training convergence.

[0146] For the embodiment of the present application, the initial evaluation target, the first reward value and the second reward value determined by the environmental state at the third moment are combined to jointly determine the evaluation target value, which can improve the accuracy of each update and thereby improve the training efficiency of pre-training.

[0147] In one feasible implementation, the pre-trained model is iteratively trained according to the evaluation target value, the evaluation value, and the pre-trained samples until a stopping condition is reached to obtain a pre-trained collaborative guidance law model, including:

[0148] After completing the calculation of the pre-trained model for a preset number of samples, determining the evaluation network parameters according to the evaluation target values ​​and evaluation values ​​corresponding to the preset number of samples, and updating the pre-trained evaluation network according to the evaluation network parameters;

[0149] Among them, the evaluation value can be The evaluation network parameters are determined according to the formula (4) with the calculated evaluation target value y. , and then based on Update the parameters of the pre-trained evaluation network:

[0150] Formula (4);

[0151] Wherein, M is the number of samples, i.e. the preset number.

[0152] When the pre-trained evaluation network is updated to the first preset period, the pre-trained evaluation target network is updated according to the evaluation network parameters determined after the first preset period;

[0153] The update frequency and update amplitude of the pre-trained evaluation target network are low, that is, the evaluation network parameters of the pre-trained evaluation network are small. Each update of the first preset period is recorded as times, and the parameters corresponding to the pre-trained evaluation target network are updated once.

[0154] In one possible scenario, the update method of the pre-trained evaluation target network is ,in Much less than 1.

[0155] In another possible scenario, the network parameters of the pre-trained evaluation target network can be directly set to the network parameters of the pre-trained evaluation network corresponding to the current moment.

[0156] Perform policy gradient calculation on the pre-trained action network based on the updated pre-trained evaluation network to obtain the action loss value, and update the network parameters of the pre-trained action network based on the action loss value;

[0157] It is necessary to use the evaluation network to solve the policy gradient of the action network and update the policy network according to the policy gradient definition.

[0158] When the pre-trained action network is updated for a second preset period, updating the network parameters of the pre-trained action target network according to the action loss value determined after the second preset period;

[0159] Specifically, the first preset period may be consistent with or different from the second preset period, and this is not limited in the embodiments of the present application.

[0160] In one possible scenario, the pre-trained action-goal network is updated as ,in Much less than 1.

[0161] In another possible scenario, the network parameters of the pre-trained action target network can be directly set to the network parameters of the pre-trained action network corresponding to the current moment.

[0162] Iterative training is performed until the stopping condition is reached to obtain the pre-trained collaborative guidance law model.

[0163] In the embodiment of the present application, a double-delay deterministic deep policy gradient algorithm is adopted to solve the problem that the evaluation network estimation deviation may be large.

[0164] Furthermore, in the embodiment of the present application, the pre-training sample further includes: an action value corresponding to the first moment,

[0165] The policy gradient calculation is performed on the pre-trained action network based on the updated pre-trained evaluation network to obtain the action loss value, including:

[0166] Evaluate the action value corresponding to the first moment to obtain the offline expert evaluation value corresponding to the first moment;

[0167] Determine a correction value based on the offline expert evaluation value and the evaluation value corresponding to the first moment;

[0168] Among them, when the offline expert evaluation value corresponding to the first moment is greater than the evaluation value corresponding to the first moment, the correction value is the bi-norm of the action value corresponding to the first moment and the predicted action value corresponding to the first moment, otherwise, the correction value is 0; the predicted action value corresponding to the first moment is obtained based on the environment state at the first moment and the pre-trained action network.

[0169] The policy gradient of the pre-trained action network is calculated based on the correction value and the updated pre-trained evaluation network to obtain the action loss value.

[0170] In the embodiment of the present application, the evaluation network of reinforcement learning is pre-trained. The basic framework of the pre-training algorithm is consistent with the TD3 algorithm. In the embodiment of the present application, the pre-training evaluation network Evaluation network parameters The determination of can be obtained by formula (2), formula (5), and formula (6):

[0171] Formula (5);

[0172] Formula (6);

[0173] in The state, action, reward and next state transition obtained by the cooperative guidance law interception simulation are: is a random exploration action near the action obtained after policy inference, and They represent the pre-training target evaluation network parameters and the pre-training target evaluation network parameters respectively, and N represents the number of samples from the missile.

[0174] Pre-training action network in pre-training algorithm The pre-trained action network parameters Update according to the policy gradient. For the embodiment of the present application, the difference between the evaluation value of the network output action and the action of the offline expert strategy is introduced as the correction value Based on the revised value Correct the policy gradient.

[0175] Specifically, the offline expert evaluation value of the action value corresponding to the first moment is calculated based on the action network (i.e., the collaborative guidance law) , calculate the correction value according to formula (7), specifically, the offline expert evaluation value corresponding to the first moment and the evaluation value corresponding to the first moment under the pre-trained action network For comparison, if Greater than ,but is the bi-norm of the two actions (the action value corresponding to the first moment in the pre-trained sample and the predicted action value corresponding to the first moment obtained according to the environment state at the first moment and the pre-trained action network), otherwise is 0. Then calculate the loss function of the policy gradient according to formula (8) , the calculation formula is as follows:

[0176] Formula (7);

[0177] Formula (8);

[0178] in, is the adjustment parameter.

[0179] It can be seen that in the embodiment of the present application, adding a correction term to correct the parameters of the action network can improve the training efficiency.

[0180] Specifically, the present application embodiment provides a training method for a collaborative guidance law model, such as Figure 4 As shown, it includes: step S410-step S420, wherein:

[0181] Step S410: Acquire multiple training samples, where each training sample includes: an environment state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environment state at a third moment;

[0182] Step S420: performing iterative reinforcement learning training on the training model based on multiple training samples to obtain a collaborative guidance law model;

[0183] Among them, the model to be trained includes the network to be trained and the target network to be trained. The initial parameters of the evaluation network to be trained in the network to be trained and the target evaluation network to be trained in the target network to be trained are the corresponding parameters in the pre-trained collaborative guidance law model. The pre-trained collaborative guidance law model is obtained by reinforcement learning training based on multiple pre-training samples.

[0184] In one possible scenario, only the network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation action target network are used as the initial network parameters of the evaluation network to be trained and the initial network parameters of the action evaluation target network to be trained in the training process of the collaborative guidance law model. The performance of the exploration space of the trained initial model is better and the training efficiency can be improved.

[0185] In another possible scenario, the network parameters of the pre-trained network and the network parameters of the pre-trained target network after pre-training are used as the initial network parameters of the model to be trained. In this case, the network model to be trained converges faster during training.

[0186] Specifically, the training process of the embodiment of the present application is compared with the above-mentioned pre-training process, and the relevant training content will not be repeated in the embodiment of the present application.

[0187] In the embodiment of the present application, the training evaluation network is used to solve the policy gradient of the training action network, and the training action network is updated according to the definition of the policy gradient. The policy gradient calculation formula (9) can be:

[0188] Formula (9);

[0189] Specifically, using sampling strategy gradient Let the expected value update the network parameters of the training action network, where is the gradient of the output action of the training evaluation network relative to the training action network, is the gradient of the training action network output action with respect to the training action network parameters, For the training action network with input s, the input is s and the output is a, To train the network parameters of the action network; the chain rule is used for derivation: theoretically, the indicator function J (revenue) should be proportional to the training action network parameters To find the derivative, use the chain rule, first let the training evaluation network Q to find the derivative of action a, and then use action a to find the derivative of the action network parameters In addition, since the goal of training the action network is to maximize the evaluation value, the average evaluation value is used as the benefit.

[0190] Furthermore, the action network in the to-be-trained network and the action target network in the to-be-trained target network include: a basic proportional guidance law term and / or a collaborative correction term.

[0191] In one achievable manner, the action network in the to-be-trained network and the action-target network in the to-be-trained target network include: a basic proportional guidance term and a time coordination correction term, such as, .

[0192] It is understandable that the coordinated control term of time in the guidance law is a complex multi-parameter term, making it difficult to obtain a control strategy in a short period of time. It requires training and solving it using reinforcement learning methods to minimize the time difference between the main and slave missiles hitting the target. Because the action strategy is continuous, and to prevent overestimation and enable the intelligent agent to be trained quickly and effectively, a double-delayed deterministic deep policy gradient algorithm is used to train the intelligent agent. Furthermore, the embodiments of the present application use the parameters of the evaluation network in pre-training as the initial parameters to ensure that the intelligent agent can carry out fast and efficient learning and training iterations, thereby solving the problem of demanding conditions in engineering applications of the coordinated guidance law.

[0193] Based on any of the above embodiments, this application embodiment provides a specific training process, please refer to Figure 5 ,include:

[0194] Establish a mathematical model of cooperative guidance law;

[0195] According to the collaborative guidance law model, the adversarial environment is simulated to determine the agent's observed environment state and action value to obtain pre-training samples and training samples;

[0196] Pre-training process: Pre-training is performed offline to obtain an evaluation network after pre-training is completed; in order to improve the exploration efficiency of the trained collaborative guidance law in the reinforcement learning process and speed up the training process, the embodiment of the present application adopts a pre-training method framework for offline reinforcement learning based on LBC.

[0197] The training process: The parameters of the evaluation network after pre-training are used as the initial parameters of the evaluation network during training; the parameters of the action network after pre-training are used as the initial parameters of the action network during training; then the current environment state is obtained and an action value is generated based on the trained action network, and an evaluation value is obtained based on the two evaluation networks; the target value is obtained using the updated environment and reward value. After a sequence of evaluation and target values ​​is collected, the training evaluation network and the training action network are updated to determine whether the end condition has been met. If so, the training is determined to be completed and a trained model is obtained; if not, the next sequence of samples is used for training. In other words, the pre-trained evaluation network and action network are first used as the network initial parameters; then training is performed: the environment state is input to the action network, and the action network outputs an action. The action and environment state are input to the evaluation network, and the evaluation network outputs an evaluation value. The environment is then updated to determine the actual reward; and the network is then updated. In summary, the embodiment of the present application is based on an improved guidance law structure. During the training process, in order to enable the new guidance algorithm to quickly adapt to larger strike combat scenarios, the action network is first appropriately pre-trained, and then combined with a double-delayed deep deterministic policy gradient algorithm to carry out fast and efficient learning and training iterations. The reinforcement learning collaborative guidance law trained by the designed framework has the obvious advantages of a wider range of applications and higher time coordination accuracy.

[0198] In the above embodiment, a pre-training method of a collaborative guidance law model is introduced from the perspective of method flow. The following embodiment introduces a pre-training system of a collaborative guidance law model from the perspective of a module or unit. For details, please refer to the following embodiment.

[0199] The embodiment of the present application provides a pre-training system for a collaborative guidance law model, such as Figure 6 As shown, the system may include:

[0200] A first acquisition module 610 is configured to acquire pre-training samples from the missile, wherein each pre-training sample includes: an environment state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environment state at a third moment;

[0201] A second acquisition module 620 is used to acquire a pre-trained model, the pre-trained model includes: a pre-trained network and a pre-trained target network, the pre-trained network includes: a pre-trained action network and a pre-trained evaluation network, and the pre-trained target network includes: a pre-trained action target network and a pre-trained evaluation target network;

[0202] The first training module 630 is used to perform reinforcement learning training on the pre-trained model according to the pre-trained samples until the stopping condition is reached to obtain the pre-trained collaborative guidance law model.

[0203] The network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation target network in the pre-trained collaborative guidance law model serve as the initial network parameters of the evaluation network to be trained and the initial network parameters of the evaluation target to be trained in the training process of the collaborative guidance law model.

[0204] In one possible implementation,

[0205] When executing the acquisition of the pre-training sample from the bullet, the first acquisition module 610 is used to:

[0206] Obtaining the environmental state at the first moment, and using the cooperative guidance law to obtain the action value corresponding to the first moment according to the environmental state at the first moment;

[0207] According to the action value corresponding to the first moment, the training environment is simulated to obtain the environment state at the second moment and the reward value corresponding to the first moment;

[0208] According to the environmental state at the second moment and the cooperative guidance law, a second action value corresponding to the second moment is obtained; according to the action value corresponding to the second moment, the environmental state at the third moment and the second reward value corresponding to the second moment are obtained by simulating the training environment.

[0209] In one possible implementation, when the first acquisition module 610 performs a simulation based on the action value corresponding to the first moment through the training environment to obtain the environment state at the second moment and the reward value corresponding to the first moment, it is configured to:

[0210] According to the action value corresponding to the first moment, the training environment is simulated to obtain the environment state at the second moment;

[0211] determining, based on the environmental state at the second moment, a lead angle of the missile, a relative distance between the missile and the target, and a relative speed between the missile and the target corresponding to the missile at the second moment, wherein the missile includes a main missile and a slave missile;

[0212] When the missile hits, a first reward value corresponding to the first moment is determined according to the main missile hit moment and the secondary missile hit moment;

[0213] When the missile misses, the predicted hit time of the main missile and the predicted hit time of the slave missile are determined based on the corresponding lead angle of the missile, the relative distance between the missile and the target, and the relative speed between the missile and the target; and the first reward value corresponding to the first moment is determined based on the predicted hit time of the main missile and the predicted hit time of the slave missile.

[0214] In one possible implementation, when performing reinforcement learning training on the pre-trained model based on the pre-trained samples until a stopping condition is reached to obtain the pre-trained collaborative guidance law model, the first training module 630 is configured to:

[0215] According to the environment state at the first moment and the pre-trained network, an evaluation value is obtained;

[0216] According to the environmental state at the third moment, the pre-trained target network is used to obtain the initial evaluation target value;

[0217] Determining an evaluation target value based on the initial evaluation target value, the first reward value, and the second reward value;

[0218] The pre-trained model is iteratively trained according to the evaluation target value, evaluation value and pre-training samples until the stopping condition is reached to obtain the pre-trained collaborative guidance law model.

[0219] In one possible implementation, when determining the evaluation target value based on the initial evaluation target value, the first reward value, and the second reward value, the first training module 630 is configured to:

[0220] Determine an intermediate evaluation target value based on the initial evaluation target value, the attenuation discount coefficient, and the second reward value;

[0221] An evaluation target value is determined based on the intermediate evaluation target value, the attenuation discount coefficient, and the first reward value.

[0222] In one possible implementation, the first training module 630, when iteratively training the pre-trained model according to the evaluation target value, the evaluation value, and the pre-training samples until a stopping condition is reached to obtain the pre-trained collaborative guidance law model, is configured to:

[0223] After completing the calculation of the pre-trained model for the preset number of samples, determining the evaluation network parameters according to the evaluation target values ​​and the evaluation values ​​corresponding to the preset number of samples, and updating the pre-trained evaluation network according to the evaluation network parameters;

[0224] After the pre-trained evaluation network is updated for a first preset period, the pre-trained evaluation target network is updated according to the evaluation network parameters determined after the first preset period;

[0225] Performing policy gradient calculation on the pre-trained action network based on the updated pre-trained evaluation network to obtain an action loss value, and updating the network parameters of the pre-trained action network based on the action loss value;

[0226] When the pre-trained action network is updated for a second preset period, updating the network parameters of the pre-trained action target network according to the action loss value determined after the second preset period;

[0227] Iterative training is performed until the stopping condition is reached to obtain the pre-trained collaborative guidance law model.

[0228] In a possible implementation, the pre-training sample further includes: an action value corresponding to the first moment,

[0229] When the first training module 630 performs policy gradient calculation on the pre-trained action network based on the updated pre-trained evaluation network to obtain the action loss value, it is used to:

[0230] Evaluate the action value corresponding to the first moment to obtain the offline expert evaluation value corresponding to the first moment;

[0231] Determine a correction value based on the offline expert evaluation value and the evaluation value corresponding to the first moment;

[0232] Among them, when the offline expert evaluation value corresponding to the first moment is greater than the evaluation value corresponding to the first moment, the correction value is the bi-norm of the action value corresponding to the first moment and the predicted action value corresponding to the first moment, otherwise, the correction value is 0; the predicted action value corresponding to the first moment is obtained based on the environment state at the first moment and the pre-trained action network.

[0233] The policy gradient of the pre-trained action network is calculated based on the correction value and the updated pre-trained evaluation network to obtain the action loss value.

[0234] The pre-training system for a collaborative guidance law model provided in an embodiment of the present application is applicable to the pre-training method embodiment of the collaborative guidance law model described above, which will not be described in detail here.

[0235] In the above embodiment, a training method for a collaborative guidance law model is introduced from the perspective of a method flow. The following embodiment introduces a training system for a collaborative guidance law model from the perspective of a module or unit. For details, please refer to the following embodiment.

[0236] The embodiment of the present application provides a training system for a collaborative guidance law model, such as Figure 7 As shown, the system may include:

[0237] The third acquisition module 710 is used to acquire multiple training samples, wherein each training sample includes: an environment state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environment state at a third moment;

[0238] A second training module 720 is configured to perform iterative reinforcement learning training on the training model based on multiple training samples to obtain a collaborative guidance law model;

[0239] Among them, the model to be trained includes the network to be trained and the target network to be trained. The initial parameters of the evaluation network to be trained in the network to be trained and the target evaluation network to be trained in the target network to be trained are the corresponding parameters in the pre-trained collaborative guidance law model. The pre-trained collaborative guidance law model is obtained by reinforcement learning training based on multiple pre-training samples.

[0240] The training system for a collaborative guidance law model provided in an embodiment of the present application is applicable to the training method embodiment of the collaborative guidance law model described above, which will not be described in detail here.

[0241] An electronic device is provided in an embodiment of the present application, such as Figure 8 As shown, Figure 8 The electronic device 800 shown includes a processor 801 and a memory 803. The processor 801 and the memory 803 are connected, for example, via a bus 802. Optionally, the electronic device 800 may further include a transceiver 804. It should be noted that in actual applications, the number of transceivers 804 is not limited to one, and the structure of the electronic device 800 does not constitute a limitation on the embodiments of the present application.

[0242] Processor 801 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 801 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0243] Bus 802 may include a path for transmitting information between the aforementioned components. Bus 802 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 802 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0244] The memory 803 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0245] The memory 803 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 801. The processor 801 is used to execute the application code stored in the memory 803 to implement the content shown in the above method embodiment.

[0246] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0247] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0248] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0249] The above are only some of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A pre-training method for a collaborative guidance law model, characterized in that: include: Obtain pre-training samples from the bullet, wherein each pre-training sample includes: an environment state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environment state at a third moment; Obtain a pre-trained model, the pre-trained model comprising: a pre-trained network and a pre-trained target network, the pre-trained network comprising: a pre-trained action network and a pre-trained evaluation network, the pre-trained target network comprising: a pre-trained action target network and a pre-trained evaluation target network; According to the pre-training samples, the pre-training model is subjected to reinforcement learning training until a stopping condition is reached to obtain a pre-trained collaborative guidance law model. The network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation target network in the pre-trained collaborative guidance law model are used as the initial network parameters of the evaluation network to be trained and the initial network parameters of the evaluation target to be trained in the collaborative guidance law model training process; The obtaining of pre-training samples from the missile includes: Acquire an environmental state at a first moment, and obtain an action value corresponding to the first moment using a cooperative guidance law according to the environmental state at the first moment; According to the action value corresponding to the first moment, the training environment is simulated to obtain the environment state at the second moment and the reward value corresponding to the first moment; According to the environmental state at the second moment and the cooperative guidance law, a second action value corresponding to the second moment is obtained; according to the action value corresponding to the second moment, a training environment is simulated to obtain the environmental state at a third moment and a second reward value corresponding to the second moment; The step of simulating the training environment according to the action value corresponding to the first moment to obtain the environment state at the second moment and the reward value corresponding to the first moment includes: Simulating the training environment according to the action value corresponding to the first moment to obtain the environment state at the second moment; determining, based on the environmental state at the second moment, a lead angle of the missile, a relative distance between the missile and the target, and a relative speed between the missile and the target corresponding to the missile at the second moment, wherein the missile includes a main missile and a slave missile; When the missile hits, determining a first reward value corresponding to the first moment according to the main missile hitting moment and the secondary missile hitting moment; When the missile misses, the predicted hit moment corresponding to the main missile and the predicted hit moment corresponding to the slave missile are determined based on the corresponding lead angle of the missile, the relative distance between the missile and the target, and the relative speed between the missile and the target; and the first reward value corresponding to the first moment is determined based on the predicted hit moment corresponding to the main missile and the predicted hit moment corresponding to the slave missile.

2. The pre-training method of the collaborative guidance law model according to claim 1, characterized in that: The method of performing reinforcement learning training on the pre-trained model according to the pre-trained samples until a stopping condition is reached to obtain a pre-trained collaborative guidance law model includes: Obtaining an evaluation value according to the environmental state at the first moment and the pre-trained network; Obtaining an initial evaluation target value using a pre-trained target network according to the environmental state at the third moment; determining an evaluation target value according to the initial evaluation target value, the first reward value, and the second reward value; The pre-training model is iteratively trained according to the evaluation target value, the evaluation value and the pre-training sample until a stopping condition is reached to obtain a pre-trained collaborative guidance law model.

3. The pre-training method of the collaborative guidance law model according to claim 2, characterized in that: The determining of the evaluation target value according to the initial evaluation target value, the first reward value, and the second reward value includes: Determining an intermediate evaluation target value according to the initial evaluation target value, the attenuation discount coefficient, and the second reward value; The evaluation target value is determined according to the intermediate evaluation target value, the decay discount coefficient and the first reward value.

4. The pre-training method of the collaborative guidance law model according to claim 2, characterized in that: The iterative training of the pre-trained model according to the evaluation target value, the evaluation value and the pre-trained sample until a stopping condition is reached to obtain a pre-trained collaborative guidance law model includes: After completing the calculation of the pre-trained model for the preset number of samples, determining the evaluation network parameters according to the evaluation target values ​​and the evaluation values ​​corresponding to the preset number of samples, and updating the pre-trained evaluation network according to the evaluation network parameters; After the pre-trained evaluation network is updated for a first preset period, the pre-trained evaluation target network is updated according to the evaluation network parameters determined after the first preset period; Performing policy gradient calculation on the pre-trained action network based on the updated pre-trained evaluation network to obtain an action loss value, and updating the network parameters of the pre-trained action network based on the action loss value; When the pre-trained action network is updated for a second preset period, updating the network parameters of the pre-trained action target network according to the action loss value determined after the second preset period; Iterative training is performed until the stopping condition is reached to obtain the pre-trained collaborative guidance law model.

5. The pre-training method of the collaborative guidance law model according to claim 4, characterized in that: The pre-training sample also includes: the action value corresponding to the first moment, The strategy gradient calculation is performed on the pre-trained action network according to the updated pre-trained evaluation network to obtain the action loss value, including: Evaluate the action value corresponding to the first moment to obtain the offline expert evaluation value corresponding to the first moment; determining a correction value according to the offline expert evaluation value and the evaluation value corresponding to the first moment; Among them, when the offline expert evaluation value corresponding to the first moment is greater than the evaluation value corresponding to the first moment, the correction value is the two-norm of the action value corresponding to the first moment and the predicted action value corresponding to the first moment; otherwise, the correction value is 0; the predicted action value corresponding to the first moment is obtained based on the environment state at the first moment and the pre-trained action network. The policy gradient calculation is performed on the pre-trained action network according to the correction value and the updated pre-trained evaluation network to obtain the action loss value.

6. A training method for a collaborative guidance law model, characterized in that: include: Acquire multiple training samples, where each training sample includes: an environment state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environment state at a third moment; Performing iterative reinforcement learning training on the model to be trained based on the multiple training samples to obtain a collaborative guidance law model; The model to be trained includes a network to be trained and a target network to be trained. Initial parameters of a training evaluation network in the training network and a training target evaluation network of the target network to be trained are corresponding parameters in a pre-trained collaborative guidance law model. The pre-trained collaborative guidance law model is obtained by reinforcement learning training based on multiple pre-training samples. The obtaining of multiple training samples includes: Acquire an environmental state at a first moment, and obtain an action value corresponding to the first moment using a cooperative guidance law according to the environmental state at the first moment; According to the action value corresponding to the first moment, the training environment is simulated to obtain the environment state at the second moment and the reward value corresponding to the first moment; According to the environmental state at the second moment and the cooperative guidance law, a second action value corresponding to the second moment is obtained; according to the action value corresponding to the second moment, a training environment is simulated to obtain the environmental state at a third moment and a second reward value corresponding to the second moment; The step of simulating the training environment according to the action value corresponding to the first moment to obtain the environment state at the second moment and the reward value corresponding to the first moment includes: Simulating the training environment according to the action value corresponding to the first moment to obtain the environment state at the second moment; determining, based on the environmental state at the second moment, a lead angle of the missile, a relative distance between the missile and the target, and a relative speed between the missile and the target corresponding to the missile at the second moment, wherein the missile includes a main missile and a slave missile; When the missile hits, determining a first reward value corresponding to the first moment according to the main missile hitting moment and the secondary missile hitting moment; When the missile misses, the predicted hit moment corresponding to the main missile and the predicted hit moment corresponding to the slave missile are determined based on the corresponding lead angle of the missile, the relative distance between the missile and the target, and the relative speed between the missile and the target; and the first reward value corresponding to the first moment is determined based on the predicted hit moment corresponding to the main missile and the predicted hit moment corresponding to the slave missile.

7. A pre-training system for a collaborative guidance law model, characterized in that: include: A first acquisition module is used to acquire pre-training samples from the bullet, wherein each pre-training sample includes: an environmental state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environmental state at a third moment; A second acquisition module is used to acquire a pre-trained model, wherein the pre-trained model includes a pre-trained network and a pre-trained target network, wherein the pre-trained network includes a pre-trained action network and a pre-trained evaluation network, and wherein the pre-trained target network includes a pre-trained action target network and a pre-trained evaluation target network; The first training module is used to perform reinforcement learning training on the pre-training model according to the pre-training samples until a stopping condition is reached to obtain a pre-trained collaborative guidance law model. The network parameters of the pre-trained evaluation network and the network parameters of the pre-trained evaluation target network in the pre-trained collaborative guidance law model are used as the initial network parameters of the evaluation network to be trained and the initial network parameters of the evaluation target to be trained in the collaborative guidance law model training process; The first acquisition module is specifically configured to acquire an environmental state at a first moment, obtain an action value corresponding to the first moment using a collaborative guidance law based on the environmental state at the first moment; perform simulations in a training environment based on the action value corresponding to the first moment to obtain an environmental state at a second moment and a reward value corresponding to the first moment; obtain a second action value corresponding to the second moment based on the environmental state at the second moment and the collaborative guidance law; and perform simulations in the training environment based on the action value corresponding to the second moment to obtain an environmental state at a third moment and a second reward value corresponding to the second moment; When executing the simulation based on the action value corresponding to the first moment through the training environment to obtain the environment state at the second moment and the reward value corresponding to the first moment, the first acquisition module is used to: simulate the training environment based on the action value corresponding to the first moment to obtain the environment state at the second moment; determine the lead angle corresponding to the missile, the relative distance between the missile and the target, and the relative speed between the missile and the target at the second moment based on the environment state at the second moment, the missile including a main missile and a slave missile; when the missile hits, determine the first reward value corresponding to the first moment based on the hit moment of the main missile and the hit moment of the slave missile; when the missile misses, determine the predicted hit moment corresponding to the main missile and the predicted hit moment corresponding to the slave missile based on the lead angle corresponding to the missile, the relative distance between the missile and the target, and the relative speed between the missile and the target; and determine the first reward value corresponding to the first moment based on the predicted hit moment corresponding to the main missile and the predicted hit moment corresponding to the slave missile.

8. A training system for a collaborative guidance law model, characterized in that: include: A third acquisition module is used to acquire multiple training samples, wherein each training sample includes: an environment state at a first moment, a first reward value corresponding to the first moment, a second reward value corresponding to the second moment, and an environment state at a third moment; A second training module is configured to perform iterative reinforcement learning training on the training model based on the multiple training samples to obtain a collaborative guidance law model; The model to be trained includes a network to be trained and a target network to be trained. Initial parameters of a training evaluation network in the training network and a training target evaluation network of the target network to be trained are corresponding parameters in a pre-trained collaborative guidance law model. The pre-trained collaborative guidance law model is obtained by reinforcement learning training based on multiple pre-training samples. The third acquisition module is specifically configured to acquire an environmental state at a first moment, obtain an action value corresponding to the first moment using a collaborative guidance law based on the environmental state at the first moment, perform simulations in a training environment based on the action value corresponding to the first moment, obtain an environmental state at a second moment and a reward value corresponding to the first moment, obtain a second action value corresponding to the second moment based on the environmental state at the second moment and the collaborative guidance law, and obtain an environmental state at a third moment and a second reward value corresponding to the second moment based on the action value corresponding to the second moment using a simulation in the training environment. The third acquisition module, when executing the simulation based on the action value corresponding to the first moment through the training environment to obtain the environment state at the second moment and the reward value corresponding to the first moment, is used to: simulate the training environment based on the action value corresponding to the first moment to obtain the environment state at the second moment; determine the lead angle corresponding to the missile, the relative distance between the missile and the target, and the relative speed between the missile and the target at the second moment based on the environment state at the second moment, the missile including a main missile and a slave missile; when the missile hits, determine the first reward value corresponding to the first moment based on the hit moment of the main missile and the hit moment of the slave missile; when the missile misses, determine the predicted hit moment corresponding to the main missile and the predicted hit moment corresponding to the slave missile based on the lead angle corresponding to the missile, the relative distance between the missile and the target, and the relative speed between the missile and the target; and determine the first reward value corresponding to the first moment based on the predicted hit moment corresponding to the main missile and the predicted hit moment corresponding to the slave missile.

Images